Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th November 2002, 04:14   #1  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
FluxSmooth

Here's a new spatio-temporal smoother I wrote today based on ideas that came to me developing NoMoSmooth.

The source code is available. Here's the description from the documentation:

Quote:
One of the fundamental properties of noise is that it's random. One of the fundamental properties of motion is that it's not. This is the premise behind FluxSmooth, which examines each pixel and compares it to the corresponding pixel in the previous and last frame. Smoothing occurs if both the previous frame's value and the next frame's value are greater, or if both are less, than the value in the current frame.

I like to call this a "fluctuating" pixel, then I like to wipe that pixel from existence by averaging it with its neighbours. This is (by default) done in a spatio-temporal manner, in that for each fluctuating pixel its 8 immediate spatial neighbours as well as its 2 temporal neighbours (the abovementioned corresponding pixel from the previous and next frames) are considered for inclusion in the average. If the value of each pixel is within the specified threshold, it is included. If not, it isn't.

This filter seems to remove almost all noise from low-noise sources (such as DVD) and a lot of noise from high-noise sources (such as cable TV captures), while maintaining a good amount of detail.

The speed at which it operates depends upon the amount of noise in the clip.
As usual your mileage may vary. This filter is unrelated to flux capacitors, which enable one to travel temporally.

source: http://www.kvcd.net/sansgrip/avisynth/

latest version: http://bengal.missouri.edu/~kes25c/FluxSmooth-1.1b.zip
__________________
Regards,
SansGrip

Last edited by Wilbert; 10th November 2006 at 21:43.
SansGrip is offline   Reply With Quote
Old 17th November 2002, 08:25   #2  |  Link
jorel
Guest
 
Posts: n/a
Re: FluxSmooth

Quote:
Originally posted by SansGrip
Here's a new spatio-temporal smoother I wrote today based on ideas that came to me developing NoMoSmooth.

The source code is available. Here's the description from the documentation:



As usual your mileage may vary. This filter is unrelated to flux capacitors, which enable one to travel temporally.
thanks!........
the download works here,but can't open the page.
  Reply With Quote
Old 17th November 2002, 09:04   #3  |  Link
High Speed Dubb
Registered User
 
Join Date: Jan 2002
Posts: 283
The noise detection sounds similar to a temporal median filter, though your averaging method is different. You might want to check Tom’s STMedianFilter at
http://www.trbarry.com/STMedianFilter.zip
with docs at
http://www.trbarry.com/Readme_STMedianFilter.txt
__________________
9:) Lindsey Dubb
High Speed Dubb is offline   Reply With Quote
Old 17th November 2002, 09:05   #4  |  Link
JuanC
Registered User
 
Join Date: May 2002
Posts: 220
Re: FluxSmooth

Quote:
Originally posted by SansGrip
... One of the fundamental properties of motion is that it's not. This is the premise behind FluxSmooth, which examines each pixel and compares it to the corresponding pixel in the previous and last ¿next? frame. ...
It looks like your web page is unavailable.
JuanC is offline   Reply With Quote
Old 17th November 2002, 16:35   #5  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
All links working fine for me. Must have been just a temporary thing...
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 17th November 2002, 16:49   #6  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
The noise detection sounds similar to a temporal median filter, though your averaging method is different. You might want to check Tom’s STMedianFilter

From what I can gather reading the docs and source, STMedianFilter doesn't specifically target "fluctuating" pixels but instead clips pixels to its nearest neighbours.

I think what makes FluxSmooth different (though I've not checked every smoother out there ) is that the temporal sequence 2, 0, -1 will not get averaged, but 2, 0, 1 will.

Perhaps this has already been implemented. What we need is some central filter algorithm repository we can check before beginning coding .
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 17th November 2002, 19:07   #7  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
New version 0.2

Here's verion 0.2, incorporating a minor bugfix, some optimizations, and an Avisynth 2.5/YV12 version.

Testing appreciated .

Edit: Removed links to old version.
__________________
Regards,
SansGrip

Last edited by SansGrip; 19th November 2002 at 17:42.
SansGrip is offline   Reply With Quote
Old 17th November 2002, 22:43   #8  |  Link
Rrrough
Registered User
 
Rrrough's Avatar
 
Join Date: Aug 2002
Location: Avantasia
Posts: 177
hi, I'd love to test this filter as well as the new version of NoMoSmooth, but unfortunately I can't load your page...
troublesome provider ? can you attach it to a post ?

cheers
Rrrough is offline   Reply With Quote
Old 17th November 2002, 22:46   #9  |  Link
jorel
Guest
 
Posts: n/a
Quote:
Originally posted by Rrrough
hi, I'd love to test this filter as well as the new version of NoMoSmooth, but unfortunately I can't load your page...
troublesome provider ? can you attach it to a post ?

cheers
....but unfortunately I can't load your page too...
troublesome provider ? can you attach it to a post ?
thanks!
  Reply With Quote
Old 18th November 2002, 00:36   #10  |  Link
High Speed Dubb
Registered User
 
Join Date: Jan 2002
Posts: 283
Sansgrip,

That’s how a median works, too. With 2,0, -1, the median is 0, so the middle pixel’s value isn’t changed. With 2, 0, 1, the median is 1, so the middle pixel is changed.
__________________
9:) Lindsey Dubb
High Speed Dubb is offline   Reply With Quote
Old 18th November 2002, 01:24   #11  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
hi, I'd love to test this filter as well as the new version of NoMoSmooth, but unfortunately I can't load your page...

Very odd. The site works fine for me. Can you PM me a traceroute?

can you attach it to a post ?

I did. It's still waiting to be modded.
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 18th November 2002, 01:31   #12  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
That’s how a median works, too.

I'm beginning to think I should pick up a Statistics 101 textbook instead of reinventing it myself .

Ah well. Say, what would be your response to the idea of a spatio-temporal smoother with skin detection?
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 18th November 2002, 03:07   #13  |  Link
High Speed Dubb
Registered User
 
Join Date: Jan 2002
Posts: 283
I wouldn’t toss the idea just because there’s a related filter out there. (Well, actually there are two related filters. Simon’s Sandwich filter at
http://forum.doom9.org/showthread.php?s=&threadid=33817
also has some similarities.)

Medians are a good way to go for noise reduction — They’re usually a little inefficient, but they can be very robust. In other words, they don’t make the best possible use of the data, but the also don’t get messed up by unexpected stuff like interference. That’s because a median will mostly ignore the tails of a distribution, and just look at its center. (There are some sporting events which get judged with medianish filters — I think diving always tosses the low and high ranking before averaging the rest.) Medians will also do a good job with edge problems (i.e., when Y = black), while means will be biased.

I haven’t tried FluxSmooth yet, but the averaging you’re using when fluctuation is detected sounds like a good idea — It’ll probably avoid most of the usual median artifacts, like problems with very fast motion.

Skin detection sounds like it’s worth a try. Are you planning to look for specific colors, and avoid smoothing those too much? Just be sure not to claim that it “discriminates based on skin color” in the docs.
__________________
9:) Lindsey Dubb

Last edited by High Speed Dubb; 18th November 2002 at 05:30.
High Speed Dubb is offline   Reply With Quote
Old 18th November 2002, 05:03   #14  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
Medians are a good way to go for noise reduction — They’re usually a little inefficient, but they can be very robust.

I'd love to find a more efficient way of implementing the "fluctuation detection" than what I'm using right now. Basically it's:

Code:
int prev_diff = prev_luma - curr_luma,
    next_diff = next_luma - curr_luma;
if((prev_diff < 0 && next_diff < 0) ||
   (prev_diff > 0 && next_diff > 0))
{
    // Do the smoothing
}
I've optimized everything else about as far as it'll go (algorithmically that is), and this is the one remaining "if" in the main loop. I'd like to make it as efficient as possible, since I'm not able to remove it.

That’s because a median will mostly ignore the tails of a distribution, and just look at its center.

Interesting, I've never thought about averages in terms of distributions before. The more I get into this stuff the more I want to visit one of the local universities and see about a math course...

I haven’t tried FluxSmooth yet, but the averaging you’re using when fluctuation is detected sounds like a good idea — It’ll probably avoid most of the usual median artifacts, like problems with very fast motion.

I'm ashamed to admit I've not had time to try your fruity smoothers yet either. I'll try yours if you'll try mine...

Skin detection sounds like it’s worth a try. Are you planning to look for specific colors, and avoid smoothing those too much?

Yes, since I find that temporal smoothing, while overall very successful, is particularly destructive to skin tones and the slight details that make it look "real". It seems sensible to me to try to reduce this effect since skin is what we spend a great deal of time looking at in movies (some movies more than others, he added predictably ).

Inspection of colour range is one of the approaches I'd use. A little reading on the web reveals a lot of material relating to skin detection. Most of it is above my head (a significant amount was wrt neural nets), but I did pick up that skin tones, regardless of race, fall within a fairly specific colour range. Not only that, but by checking the overall variance of a 3x3 block it's possible to get a good idea of whether we're dealing with smooth skin or some other kind of similarly coloured texture.

I'm kinda worried it would pick up things like smooth pink walls. I'm still not sure how to differentiate between the two. Any ideas?

Just be sure not to claim that it “discriminates based on skin color” in the docs.

heheh yes, it would need to be worded delicately .
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 18th November 2002, 09:44   #15  |  Link
OUTPinged_
MooPolice 1st division
 
OUTPinged_'s Avatar
 
Join Date: Dec 2001
Location: VIlnius,LT
Posts: 448
SansGrip, you can't do much about skin smoothing unless you will update your motion engine so it would pick pixels according to motion vectors.

All these "smart" algorithms don't help a big deal and they get real slow very fast.

NoMoSmooth is about as far as it would get with current motion engine. You may see some improvement with that algo but it wont be drastical.

I know i sound like a whiner, but maybe try to concentrate on new ME engine. This will solve more problems than just "blended faces".

..Did a quick check and it works about same as nomosmooth, but a bit slower :-(
__________________
___________________MooPolice is watching you!____.o/________
OUTPinged_ is offline   Reply With Quote
Old 18th November 2002, 16:37   #16  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
SansGrip, you can't do much about skin smoothing unless you will update your motion engine so it would pick pixels according to motion vectors.

Well, that's not quite true. It should be possible to detect skin even without motion vectors, though how much else that is similarly coloured and textured you'd also detect is another question.

NoMoSmooth is about as far as it would get with current motion engine. You may see some improvement with that algo but it wont be drastical.

The reason I'm toying with the idea of skin detection is because I find skin to be where temporal smoothing is most noticible. If one could avoid over-smoothing skin then more aggressive settings could be used on the rest of the frame.

I know i sound like a whiner, but maybe try to concentrate on new ME engine. This will solve more problems than just "blended faces".

You don't sound like a whiner, you sound like someone giving valuable input . I'm currently looking around for a simple explanation/C(++) implementation of one of the fast motion estimation algorithms. If I find one I'll try to code it up and see what kind of results I get.

..Did a quick check and it works about same as nomosmooth, but a bit slower :-(

Now that is surprising, since in Flux the averaging is totally table-based while in NoMo it's arithmetical. I'll have to do some benchmarks/profiling and check out what's going on.

Thanks for your continued testing, it really is appreciated .
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 18th November 2002, 17:34   #17  |  Link
slk001
Registered User
 
Join Date: Apr 2002
Posts: 399
Seems to me that the LAST thing you want to do with a noise filer for television is to AVERAGE it with its SPATIAL neighbors. Let me explain. First, since TV is a serially transmitted signal, any noise "blip" is not likely to be only one pixel long, but many pixels in a horizontal line. Assume, for arguments sake, that it is 5 pixels long horizontally, and only one pixel high vertically. Then, a tiny snapshot of the viewing screen would look something like this:

55555555555
55567842555
55555555555.

In this example, the "5" is the "real" signal, while anything else is noise. If we spatially average this noise, we will still have the noise - just averaged. But since we have identified this part as noise, why don't we try to eliminate it altogether? PROBLEM: What do we replace the noise with? Well, the most LIKELY candidates are its TEMPORAL neighbors. As an example, here is the same "noise" section with its two temporal neighbors:

55555555555
55555555555
55555555555 (PREVIOUS FRAME)

55555555555
55567842555
55555555555 (CURRENT FRAME, WITH NOISE)

55555555555
55555555555
55555555555 (NEXT FRAME)

In this scenario, if we endeavour to REPLACE the noise with a suitable substitute, then a very good approximation is the value of the pixels either before, or after the noise - or, the average of these two values NOTE: The noise itself is NOT included in the average, since we have assumed it to be an INVALID value, so, in this case, we are replacing a value based on a calculation in which the original value plays no part!!

As you can see in this simplistic scenario, the noise present in the current frame would be processed out entirely.

Now, progress to the NEXT FRAME. To prevent GHOSTING, the PREVIOUS FRAME is the frame that we PROCESSED OUT THE NOISE EARLIER. As a matter of fact, the ONLY frames that will be passed through unmodified are the very first frame in a video (since it has no "previous neighbor") and the very last frame (since it has no "next neighbor").

PROBLEMS:
1) Scene changes. This can be resolve by setting a WINDOW that the PREVIOUS pixel and the NEXT pixel have to be in before the filter kicks in.
2) Rapid, one frame movements (like a sword slashing through a scene). This can be resolved by setting a WINDOW based on the SPATIAL information of pixels ABOVE and BELOW the current test pixel (if the text pixel is within the window of EITHER the pixel ABOVE or BELOW, then filter ignores the current pixel under test - ie, doesn't change).

Now here are the assumptions that I am making regarding the noise:
1) 98% of the noise "blips" will not be over or under another separate noise "blip". Or, they will not do this (SPATIALLY):

55555555555
55567842555
55555567842
55555555555.

2) The noise will never appear temporally "sync'ed". (Ie, two noise "blips" will never occur at the same spot in two adjacent frames.)

For assumption 1), a WIDTH parameter could be added, where, instead of testing spatially one line above, testing two or three lines above or below the current pixel under test. For obvious reasons, the WIDTH parameter should be limited to a maximum of two or three.

For assumption 2), the filter should ignore the noise as signal, since any algorithm devised to correct this would most likely cause ghosting.
slk001 is offline   Reply With Quote
Old 18th November 2002, 21:41   #18  |  Link
High Speed Dubb
Registered User
 
Join Date: Jan 2002
Posts: 283
SansGrip,

By saying that medians are inefficient I just meant that they don&rsquo;t make use of all the available information. I didn&rsquo;t mean to imply anything about speed.

Beyond your idea of checking the local variance, I can&rsquo;t think of any way to prevent the occasional smooth skin-toned feature from being identified as skin. As mistakes go, though, that might not look too bad &mdash; It&rsquo;ll just mean an occasional extra amount of noise. Considering that it will be mostly in smooth areas, it might even improve the picture.

There&rsquo;s also the usual threshold problem to watch out for. The boundary between &ldquo;skin&rdquo; and &ldquo;not skin&rdquo; might show some artifacts. But if the reduction of smoothing is subtle enough, the boundaries might be hard to spot.

slk001,

It looks like you&rsquo;re assuming that noise consists of small horizontal noise patches on a noiseless background (aka &ldquo;salt and pepper&rdquo; noise). That&rsquo;s a fair description of certain kinds of interference. But it isn&rsquo;t what I&rsquo;ve seen from broadcast video. Instead, there is noise everywhere. Because everything has noise, combining a whole bunch of pixels can improve the accuracy.

But you are right that noise is horizontally correlated (i.e., &ldquo;red&rdquo; noise) in broadcast material. So far I&rsquo;ve found that this can be ignored, but it is worth keeping in mind for the noise filters.
__________________
9:) Lindsey Dubb
High Speed Dubb is offline   Reply With Quote
Old 19th November 2002, 03:36   #19  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
PROBLEM: What do we replace the noise with? Well, the most LIKELY candidates are its TEMPORAL neighbors.

Assuming they aren't noise too . Rather than thinking of noise as a completely invalid value, I tend to think of it as simply a corrupted one. Just because the pixel was originally a 5 and is now an 8 doesn't make that pixel's value useless -- we just need to correct it, and that's the hard part. Averaging, of course, isn't perfect.

But you're right in that there's an argument to be made over whether spatial smoothing is even a good idea. That's why I allowed both temporal and spatial smoothing to be separately disabled by setting the threshold to zero.


As an example, here is the same "noise" section with its two temporal neighbors:


55555555555
55555555555
55555555555 (PREVIOUS FRAME)

55555555555
55567842555
55555555555 (CURRENT FRAME, WITH NOISE)

55555555555
55555555555
55555555555 (NEXT FRAME)


If noise really looked like this, it would be very easy to detect and remove . Unfortunately it's usually very very hard (if not impossible) to make the distinction between noise and fine details. What we're all trying to achieve is an algorithm that is, as it were, tough on dirt yet gentle on your hands .

Now, progress to the NEXT FRAME. To prevent GHOSTING, the PREVIOUS FRAME is the frame that we PROCESSED OUT THE NOISE EARLIER.

This is one approach, but it has the disadvantage that it tends not only to reduce more noise but also destroy more details.

2) Rapid, one frame movements (like a sword slashing through a scene).

I briefly considered this when playing with algorithms, but decided that any movement that lasts for a single frame is so quick that the amount of processing applied to it barely matters. In fact, such a fast-moving thing would be a candidate for more smoothing, not less.

2) The noise will never appear temporally "sync'ed". (Ie, two noise "blips" will never occur at the same spot in two adjacent frames.)

Unfortunately since noise is inherently random, we can't be sure of anything like that.
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Old 19th November 2002, 03:42   #20  |  Link
SansGrip
Noise is your friend
 
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
Beyond your idea of checking the local variance, I can&rsquo;t think of any way to prevent the occasional smooth skin-toned feature from being identified as skin. As mistakes go, though, that might not look too bad &mdash; It&rsquo;ll just mean an occasional extra amount of noise. Considering that it will be mostly in smooth areas, it might even improve the picture.

I had a brief play with it and using the chroma range I settled on, along with a very simple variance measure, there were a lot of false hits in my source material -- including things like static walls that you really want aggressive temporal smoothing on. What might work better is a chroma histogram instead of a simple range, fed with test data from a bunch of different scenes.

But you are right that noise is horizontally correlated

You mean distributed over several horizontal pixels?
__________________
Regards,
SansGrip
SansGrip is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:34.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.