improvement of bilateral filtering

[ambrotos]

Reading some papers I have found this

http://graphics.cs.ucf.edu/RXU/IEEECGA2004.pdf

It's a major improvement of the well-known bilateral filter (cf. the work of trictical and Mfa) :
Until now, the bilateral filtering considered a pixel-noise (one pixel alone, very different from is neighbors) as an edge. And so it could'nt remove it.
Here by adding a little preprocessing the authors remove this limitation.

Is someone interesting in porting the code to avisynth ? :D (you will find a link to their source in the pdf)


Fortunately, standard bilateral filtering can be extended to suppress both outliers and inter-pixel incoherence while keeping edges intact.

[tritical]

It is kinda interesting. I could implement the pre-processing gaussian blur into tbilateral, but I think instead of building in a gaussian blur it would be easier, and open up more possibilities, to have it take a second blurred clip as input and optionally use pixel values from that. That way you could use a gaussian blur, by feeding it a clip filtered with the gaussianblur in tsp's variableblur filter for example, or any other pre-processing you wanted to try. Plus, it makes it much easier and faster to do :).

[tritical]

Hm, seems I read over that a little too fast before. They weren't proposing a completely unrelated pre-processing step, but a gaussian blur with standard deviation equal to the spatial (domain) deviation being used in the bilateral filtering step. In other words, obtaining a pixel value estimate from filtering with only the domain weights and then using that value when calculating the range weights to the neighbors. Anyways, it doesn't really matter since you can always setup the externally produced clip to match the bilateral parameters, but it would be interesting to have an integrated version that would recalculate the estimates as the sDev settings are changed in the gui. Might try to do that sometime in the future.

I did manage to add in the ability to use pixels from a second input clip as estimates. It definitely helps with removing outliers, but also starts to produce unwanted artifacts around fine details, such as single pixel wide lines, thin black outlines around white text, etc... That is to be expected when trying to remove that type of noise though. It might be interesting to substitute the gaussian blur with median filtering and see what happens.

[ambrotos]

Indeed it sound very interesting to disconnect the pre-processing step in order to test other settings and pre-filtering methods. As they say, they tried to do it with more or less sofisticated algorithms.

I didn't finish to understand every details yet, but I would also try a straight implementation of there trick.

But thank you for catching the ball
I downloaded your source, I will also try to experiment on my own...

[tritical]

Make any progress? I don't think I had put up the version with support for using a second clip until after your last post. Anyways, the code is not exactly filled with comments, but shouldn't be too horrible I hope. One problem that did pop up is when using an estimate for the pixel value and using very small Dev settings sometimes the final sum/weights will end up being so small that it is out of double precision range. In such cases I simply had it use the estimated pixel value.

[ambrotos]

i'm sorry,
i'm a bit busy at present (my coach scold at me because I was'nt fast enough ^^)

Here I follow a master course in japan studying sound signal processing, and it's also a little time consuming.
I still plan to test the raw algorithm but first under matlab


and if NTT could hurry up and install internet in my room !!

But I don't give up ^^

[ambrotos]

I have coded an equivalent of the bilateral filter under matlab, for still images.
why always matlab ? because I work on it every day and I know it (and when I work on the filter at work nobody notice me ^^) and it's really easy to check immediatly the result and to tweak it intuitively
The main drawbak is : slowness... (on my old computer it takes 2-3 min for one small frame)

Whatever, I tested the bilateral filter with a min/max preprocess on a real picture (with two "out-of-range" pixel and a scar added)

As you can see I'm desperatly looking for material ^^
not your holidays picture, but good grainy anime would help, please send a PM ^^

the original one
http://img292.echo.cx/img292/6589/intergalactic5mj.th.jpg (http://img292.echo.cx/my.php?image=intergalactic5mj.jpg)

extract with two tweaked pixels (50,190 and 150,150 from the top left corner, set to black) and one scar (two parallel lines, one pixel width, and another perpendicular)
http://img204.echo.cx/img204/8999/intergalacticscar6kc.th.png (http://img204.echo.cx/my.php?image=intergalacticscar6kc.png)

after filtering :
radius = 2 (size of the processing square, 2 pixels around)
sigma_d = 1.5
sigma_r = 1.5
http://img228.echo.cx/img228/2874/intergalacticscarfiltered9jm.th.png (http://img228.echo.cx/my.php?image=intergalacticscarfiltered9jm.png)

After processing, the colors are hideous ^^ (but I process in RGB, plan-by-plane so...), the oil-painting effect is very strong whereas details are more or less preserved, but the more important is that every single abnormal pixel are removed while the scar remain untouched (so text incrustation should behave on the same way)

so you know everything ^^

(I think I will work in a luma/chroma space from now to see if colors are better)

[insanedesio]

Check this thread (http://forum.doom9.org/showthread.php?t=96015) . I read over the paper you posted originally and with a little help from tritical I got it to where it is now (it incorporates that paper plus another). I still have one more change I need to make when I get a chance, though, to make it faster.

[tritical]

Nice work on the filter and testing, but after thinking about it... what is the point of having bilateral filtering in your example at all? Simple min/max neighbor clipping, such as you used for pre-processing, would by itself remove those two stray pixels, leave the scar, and cause a lot less damage to the image. In fact, I now don't think 8 neighbor min/max clipping would work all that well for pre-processing in combination with bilateral filtering simply because any pixels it would change could simply be filtered with min/max clipping alone (i.e. the effect wouldn't be much (hardly any) different than doing min/max clipping and then afterwards doing bilateral filtering). In other words, it would only help for single pixel (no neighbors are corrupted) noise, which min/max clipping alone would be better at removing than bilteral filtering or bilateral filtering with preprocessing. Not sure that made any sense :D.

[ambrotos]

I know I know ^^

It's the first and only picture I had on my computer (my avatar). So obviously it's NOT a good example :)

But i was happy of this first positive result : the blur is strong (a min/max filter do not blur so hard), the "bad" pixels disapeared, and the scar is protected.

This pict is the result of 2,348,940,000 failed tests so please do not underestimate the work :p

[ambrotos]

oki just for the fans,
this is the first result of my buggylateral filter....

The original screenshot (borrowed from http://www.animemusicvideos.org/guides/avtech/), just the luma plane
http://img221.echo.cx/img221/2356/utena1pngradius6sigmad4sigmar1.th.png (http://img221.echo.cx/my.php?image=utena1pngradius6sigmad4sigmar1.png)

And the "all-gaussian" filtered
http://img73.echo.cx/img73/2356/utena1pngradius6sigmad4sigmar1.th.png (http://img73.echo.cx/my.php?image=utena1pngradius6sigmad4sigmar1.png)
radius of the kernel = 6
sigma_d = 4 (for the geometric estimator)
sigma_r = 12 (for the color distance estimator)
preprocess : min/max filter

Nothing original until now but, the next step is to test with a "kind of low pass filter" estimation function for the color.
(I'll also add manually some pattern, known for generating artifacts in the other filters... if you have ideas
- small letters ?
- dots...)