Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
17th November 2002, 04:14 | #1 | Link | |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
FluxSmooth
Here's a new spatio-temporal smoother I wrote today based on ideas that came to me developing NoMoSmooth.
The source code is available. Here's the description from the documentation: Quote:
source: http://www.kvcd.net/sansgrip/avisynth/ latest version: http://bengal.missouri.edu/~kes25c/FluxSmooth-1.1b.zip
__________________
Regards, SansGrip Last edited by Wilbert; 10th November 2006 at 21:43. |
|
17th November 2002, 08:25 | #2 | Link | |
Guest
Posts: n/a
|
Re: FluxSmooth
Quote:
the download works here,but can't open the page. |
|
17th November 2002, 09:04 | #3 | Link |
Registered User
Join Date: Jan 2002
Posts: 283
|
The noise detection sounds similar to a temporal median filter, though your averaging method is different. You might want to check Tom’s STMedianFilter at
http://www.trbarry.com/STMedianFilter.zip with docs at http://www.trbarry.com/Readme_STMedianFilter.txt
__________________
9:) Lindsey Dubb |
17th November 2002, 09:05 | #4 | Link | |
Registered User
Join Date: May 2002
Posts: 220
|
Re: FluxSmooth
Quote:
|
|
17th November 2002, 16:49 | #6 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
The noise detection sounds similar to a temporal median filter, though your averaging method is different. You might want to check Tom’s STMedianFilter
From what I can gather reading the docs and source, STMedianFilter doesn't specifically target "fluctuating" pixels but instead clips pixels to its nearest neighbours. I think what makes FluxSmooth different (though I've not checked every smoother out there ) is that the temporal sequence 2, 0, -1 will not get averaged, but 2, 0, 1 will. Perhaps this has already been implemented. What we need is some central filter algorithm repository we can check before beginning coding .
__________________
Regards, SansGrip |
17th November 2002, 19:07 | #7 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
New version 0.2
Here's verion 0.2, incorporating a minor bugfix, some optimizations, and an Avisynth 2.5/YV12 version.
Testing appreciated . Edit: Removed links to old version.
__________________
Regards, SansGrip Last edited by SansGrip; 19th November 2002 at 17:42. |
17th November 2002, 22:46 | #9 | Link | |
Guest
Posts: n/a
|
Quote:
troublesome provider ? can you attach it to a post ? thanks! |
|
18th November 2002, 01:24 | #11 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
hi, I'd love to test this filter as well as the new version of NoMoSmooth, but unfortunately I can't load your page...
Very odd. The site works fine for me. Can you PM me a traceroute? can you attach it to a post ? I did. It's still waiting to be modded.
__________________
Regards, SansGrip |
18th November 2002, 01:31 | #12 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
That’s how a median works, too.
I'm beginning to think I should pick up a Statistics 101 textbook instead of reinventing it myself . Ah well. Say, what would be your response to the idea of a spatio-temporal smoother with skin detection?
__________________
Regards, SansGrip |
18th November 2002, 03:07 | #13 | Link |
Registered User
Join Date: Jan 2002
Posts: 283
|
I wouldn’t toss the idea just because there’s a related filter out there. (Well, actually there are two related filters. Simon’s Sandwich filter at
http://forum.doom9.org/showthread.php?s=&threadid=33817 also has some similarities.) Medians are a good way to go for noise reduction — They’re usually a little inefficient, but they can be very robust. In other words, they don’t make the best possible use of the data, but the also don’t get messed up by unexpected stuff like interference. That’s because a median will mostly ignore the tails of a distribution, and just look at its center. (There are some sporting events which get judged with medianish filters — I think diving always tosses the low and high ranking before averaging the rest.) Medians will also do a good job with edge problems (i.e., when Y = black), while means will be biased. I haven’t tried FluxSmooth yet, but the averaging you’re using when fluctuation is detected sounds like a good idea — It’ll probably avoid most of the usual median artifacts, like problems with very fast motion. Skin detection sounds like it’s worth a try. Are you planning to look for specific colors, and avoid smoothing those too much? Just be sure not to claim that it “discriminates based on skin color” in the docs.
__________________
9:) Lindsey Dubb Last edited by High Speed Dubb; 18th November 2002 at 05:30. |
18th November 2002, 05:03 | #14 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
Medians are a good way to go for noise reduction — They’re usually a little inefficient, but they can be very robust.
I'd love to find a more efficient way of implementing the "fluctuation detection" than what I'm using right now. Basically it's: Code:
int prev_diff = prev_luma - curr_luma, next_diff = next_luma - curr_luma; if((prev_diff < 0 && next_diff < 0) || (prev_diff > 0 && next_diff > 0)) { // Do the smoothing } That’s because a median will mostly ignore the tails of a distribution, and just look at its center. Interesting, I've never thought about averages in terms of distributions before. The more I get into this stuff the more I want to visit one of the local universities and see about a math course... I haven’t tried FluxSmooth yet, but the averaging you’re using when fluctuation is detected sounds like a good idea — It’ll probably avoid most of the usual median artifacts, like problems with very fast motion. I'm ashamed to admit I've not had time to try your fruity smoothers yet either. I'll try yours if you'll try mine... Skin detection sounds like it’s worth a try. Are you planning to look for specific colors, and avoid smoothing those too much? Yes, since I find that temporal smoothing, while overall very successful, is particularly destructive to skin tones and the slight details that make it look "real". It seems sensible to me to try to reduce this effect since skin is what we spend a great deal of time looking at in movies (some movies more than others, he added predictably ). Inspection of colour range is one of the approaches I'd use. A little reading on the web reveals a lot of material relating to skin detection. Most of it is above my head (a significant amount was wrt neural nets), but I did pick up that skin tones, regardless of race, fall within a fairly specific colour range. Not only that, but by checking the overall variance of a 3x3 block it's possible to get a good idea of whether we're dealing with smooth skin or some other kind of similarly coloured texture. I'm kinda worried it would pick up things like smooth pink walls. I'm still not sure how to differentiate between the two. Any ideas? Just be sure not to claim that it “discriminates based on skin color” in the docs. heheh yes, it would need to be worded delicately .
__________________
Regards, SansGrip |
18th November 2002, 09:44 | #15 | Link |
MooPolice 1st division
Join Date: Dec 2001
Location: VIlnius,LT
Posts: 448
|
SansGrip, you can't do much about skin smoothing unless you will update your motion engine so it would pick pixels according to motion vectors.
All these "smart" algorithms don't help a big deal and they get real slow very fast. NoMoSmooth is about as far as it would get with current motion engine. You may see some improvement with that algo but it wont be drastical. I know i sound like a whiner, but maybe try to concentrate on new ME engine. This will solve more problems than just "blended faces". ..Did a quick check and it works about same as nomosmooth, but a bit slower :-(
__________________
___________________MooPolice is watching you!____.o/________ |
18th November 2002, 16:37 | #16 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
SansGrip, you can't do much about skin smoothing unless you will update your motion engine so it would pick pixels according to motion vectors.
Well, that's not quite true. It should be possible to detect skin even without motion vectors, though how much else that is similarly coloured and textured you'd also detect is another question. NoMoSmooth is about as far as it would get with current motion engine. You may see some improvement with that algo but it wont be drastical. The reason I'm toying with the idea of skin detection is because I find skin to be where temporal smoothing is most noticible. If one could avoid over-smoothing skin then more aggressive settings could be used on the rest of the frame. I know i sound like a whiner, but maybe try to concentrate on new ME engine. This will solve more problems than just "blended faces". You don't sound like a whiner, you sound like someone giving valuable input . I'm currently looking around for a simple explanation/C(++) implementation of one of the fast motion estimation algorithms. If I find one I'll try to code it up and see what kind of results I get. ..Did a quick check and it works about same as nomosmooth, but a bit slower :-( Now that is surprising, since in Flux the averaging is totally table-based while in NoMo it's arithmetical. I'll have to do some benchmarks/profiling and check out what's going on. Thanks for your continued testing, it really is appreciated .
__________________
Regards, SansGrip |
18th November 2002, 17:34 | #17 | Link |
Registered User
Join Date: Apr 2002
Posts: 399
|
Seems to me that the LAST thing you want to do with a noise filer for television is to AVERAGE it with its SPATIAL neighbors. Let me explain. First, since TV is a serially transmitted signal, any noise "blip" is not likely to be only one pixel long, but many pixels in a horizontal line. Assume, for arguments sake, that it is 5 pixels long horizontally, and only one pixel high vertically. Then, a tiny snapshot of the viewing screen would look something like this:
55555555555 55567842555 55555555555. In this example, the "5" is the "real" signal, while anything else is noise. If we spatially average this noise, we will still have the noise - just averaged. But since we have identified this part as noise, why don't we try to eliminate it altogether? PROBLEM: What do we replace the noise with? Well, the most LIKELY candidates are its TEMPORAL neighbors. As an example, here is the same "noise" section with its two temporal neighbors: 55555555555 55555555555 55555555555 (PREVIOUS FRAME) 55555555555 55567842555 55555555555 (CURRENT FRAME, WITH NOISE) 55555555555 55555555555 55555555555 (NEXT FRAME) In this scenario, if we endeavour to REPLACE the noise with a suitable substitute, then a very good approximation is the value of the pixels either before, or after the noise - or, the average of these two values NOTE: The noise itself is NOT included in the average, since we have assumed it to be an INVALID value, so, in this case, we are replacing a value based on a calculation in which the original value plays no part!! As you can see in this simplistic scenario, the noise present in the current frame would be processed out entirely. Now, progress to the NEXT FRAME. To prevent GHOSTING, the PREVIOUS FRAME is the frame that we PROCESSED OUT THE NOISE EARLIER. As a matter of fact, the ONLY frames that will be passed through unmodified are the very first frame in a video (since it has no "previous neighbor") and the very last frame (since it has no "next neighbor"). PROBLEMS: 1) Scene changes. This can be resolve by setting a WINDOW that the PREVIOUS pixel and the NEXT pixel have to be in before the filter kicks in. 2) Rapid, one frame movements (like a sword slashing through a scene). This can be resolved by setting a WINDOW based on the SPATIAL information of pixels ABOVE and BELOW the current test pixel (if the text pixel is within the window of EITHER the pixel ABOVE or BELOW, then filter ignores the current pixel under test - ie, doesn't change). Now here are the assumptions that I am making regarding the noise: 1) 98% of the noise "blips" will not be over or under another separate noise "blip". Or, they will not do this (SPATIALLY): 55555555555 55567842555 55555567842 55555555555. 2) The noise will never appear temporally "sync'ed". (Ie, two noise "blips" will never occur at the same spot in two adjacent frames.) For assumption 1), a WIDTH parameter could be added, where, instead of testing spatially one line above, testing two or three lines above or below the current pixel under test. For obvious reasons, the WIDTH parameter should be limited to a maximum of two or three. For assumption 2), the filter should ignore the noise as signal, since any algorithm devised to correct this would most likely cause ghosting. |
18th November 2002, 21:41 | #18 | Link |
Registered User
Join Date: Jan 2002
Posts: 283
|
SansGrip,
By saying that medians are inefficient I just meant that they don’t make use of all the available information. I didn’t mean to imply anything about speed. Beyond your idea of checking the local variance, I can’t think of any way to prevent the occasional smooth skin-toned feature from being identified as skin. As mistakes go, though, that might not look too bad — It’ll just mean an occasional extra amount of noise. Considering that it will be mostly in smooth areas, it might even improve the picture. There’s also the usual threshold problem to watch out for. The boundary between “skin” and “not skin” might show some artifacts. But if the reduction of smoothing is subtle enough, the boundaries might be hard to spot. slk001, It looks like you’re assuming that noise consists of small horizontal noise patches on a noiseless background (aka “salt and pepper” noise). That’s a fair description of certain kinds of interference. But it isn’t what I’ve seen from broadcast video. Instead, there is noise everywhere. Because everything has noise, combining a whole bunch of pixels can improve the accuracy. But you are right that noise is horizontally correlated (i.e., “red” noise) in broadcast material. So far I’ve found that this can be ignored, but it is worth keeping in mind for the noise filters.
__________________
9:) Lindsey Dubb |
19th November 2002, 03:36 | #19 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
PROBLEM: What do we replace the noise with? Well, the most LIKELY candidates are its TEMPORAL neighbors.
Assuming they aren't noise too . Rather than thinking of noise as a completely invalid value, I tend to think of it as simply a corrupted one. Just because the pixel was originally a 5 and is now an 8 doesn't make that pixel's value useless -- we just need to correct it, and that's the hard part. Averaging, of course, isn't perfect. But you're right in that there's an argument to be made over whether spatial smoothing is even a good idea. That's why I allowed both temporal and spatial smoothing to be separately disabled by setting the threshold to zero. As an example, here is the same "noise" section with its two temporal neighbors: 55555555555 55555555555 55555555555 (PREVIOUS FRAME) 55555555555 55567842555 55555555555 (CURRENT FRAME, WITH NOISE) 55555555555 55555555555 55555555555 (NEXT FRAME) If noise really looked like this, it would be very easy to detect and remove . Unfortunately it's usually very very hard (if not impossible) to make the distinction between noise and fine details. What we're all trying to achieve is an algorithm that is, as it were, tough on dirt yet gentle on your hands . Now, progress to the NEXT FRAME. To prevent GHOSTING, the PREVIOUS FRAME is the frame that we PROCESSED OUT THE NOISE EARLIER. This is one approach, but it has the disadvantage that it tends not only to reduce more noise but also destroy more details. 2) Rapid, one frame movements (like a sword slashing through a scene). I briefly considered this when playing with algorithms, but decided that any movement that lasts for a single frame is so quick that the amount of processing applied to it barely matters. In fact, such a fast-moving thing would be a candidate for more smoothing, not less. 2) The noise will never appear temporally "sync'ed". (Ie, two noise "blips" will never occur at the same spot in two adjacent frames.) Unfortunately since noise is inherently random, we can't be sure of anything like that.
__________________
Regards, SansGrip |
19th November 2002, 03:42 | #20 | Link |
Noise is your friend
Join Date: Sep 2002
Location: Ontario, Canada
Posts: 554
|
Beyond your idea of checking the local variance, I can’t think of any way to prevent the occasional smooth skin-toned feature from being identified as skin. As mistakes go, though, that might not look too bad — It’ll just mean an occasional extra amount of noise. Considering that it will be mostly in smooth areas, it might even improve the picture.
I had a brief play with it and using the chroma range I settled on, along with a very simple variance measure, there were a lot of false hits in my source material -- including things like static walls that you really want aggressive temporal smoothing on. What might work better is a chroma histogram instead of a simple range, fed with test data from a bunch of different scenes. But you are right that noise is horizontally correlated You mean distributed over several horizontal pixels?
__________________
Regards, SansGrip |
Thread Tools | Search this Thread |
Display Modes | |
|
|