yet another denoise filter

[ambrotos]

It is just a prototype... I'm optimizing the code under matlab now

The goal is to remove the noise of flat colored areas and to keep details (a quest for the holy grail ?)

My method is not based on a mask but on an adaptative filter which has shown good results (even if you could improve the speed)

I designed this filter to deal with high quality animations from DVD sources (mainly films from GHIBLI studio) which are covered with a constant noise.
Whereas a simple blurring effect makes the rain disapear on a grey background, this adaptative filter readjust for each pixel the parameters of the smoothing.

please have a look to the attachement : an extract of the first image cleaned by this new method. Taken from "the grave of the fireflies" by isao takahata (french DVD arte edition)
It requires some optimizations but the noise in the sky and in the lane has faded and some details seems sharpened - like the firebombs...

(ok sent to you wilbert ^^)

or here http://tinyurl.com/662n8

[Wilbert]

A mod has to approve your attachments.

But, you can't attach 2.6 Mb files. If needed i can upload it for you (wilbertdijkhof at yahoo.com).

[Didée]

Oh c'mon! Bitmap! There is PNG, or JPEG@100% ... quarters the size, still shows everything.

[ambrotos]

ok ok i'll make PNG file just for you ^_^ (and JPG too if you like.. even if making a JPEG@100% whitout jpeg2000 seems difficult ^^ but not impossible)

before : ClipBoard-5
after : ClipBoard-5_001

it's just the beginning. I hope I could show you something more "spectacular" soon...

[morsa]

looks really good.
How does it look on natural images or video?

[Mug Funky]

nice results there. let's see how the finished filter goes.

[ambrotos]

if you want me to test on specific picture (natural pictures etc...)
please send them to moteuchi_y "at" hotmail.com
(zipped .bmp please)

I hope I could release beta matlab sources soon...

[Wilbert]

I don't get it. It's a matlab filter, not an AviSynth one?

[Didée]

Looks good so far indeed. In particular, the PNGs are much more beautiful than the BMPs ;)

Would you mind telling a little more about the process?
I ask because shown result reminds me much of the N.R. that my current script project uses internally - sort of "shifting the noiselevel down" through a 2- or 3-step edge modulated Gaussian blur. The N.R. solely looks like

http://img227.exs.cx/img227/9528/clipboard5eplg3tap8un.png

Quite similar, isn't it ;)

[ambrotos]

yes.

I wrote the prototype as a matlab function it's more easy to debug etc...

But I intend to use it as a avisynth filter.
Before I code the filter I would like to gather every opinion and people on doom9 forum are perhaps the best suited ones to make a review of my job ^^

[ambrotos]

basically the process is :

for each pixel of the picture a take a square of pixel around
(a 7x7 matrix centered on the pixel)

for each case of the matrix I calculate the distance from the center in 3D colorspace...

ie :
for a grey picture (the initial picture matrix dimension is 'height' x 'width' x '1')
if I take a pixel (the center of my square) and a second one (into the 7x7 matrix) at a distance of 3 pixel on the right side :
(values are from 0-black to 255-white)

[121] [122] [103] [118]
^
center ---------> border

in a 3D space (where color represents the third dimension) the distance between the center and the third pixel is approximately :
(where abs(a-b) represent the distance between a and b... or the absolute value)

(abs(121-122) + abs(122-103) + abs(103-118))
(with some coefficients to do the trick but... not so much)

If I do this for all the pixel from the center to the border
center => 0
1st pixel => abs(121-122) = 1
2nd pixel => abs(121-122) + abs(122-103) = 1+19 = 20
3rd pixel => abs(121-122) + abs(122-103) + abs(103-118) = 35
etc... for all the points of the matrix

After that I use a function (like a lowpass filter function)
which put to '1' every value inferior to a limit
and decrease to zero as the value increase

for our sequence : [0][1][20][35]
we could get : [1][0.999][0.7][0.01] <- wheight

the last numbers are used as wheight to calculate the mean value

any question ? :)

I'll try to bring some illustrations ^^

[ambrotos]

same URL

rep /ie

each file shows :
top-left corner an extract of 7 by 7 pixels of the picture. The center is ... at the center.

top-right corner the result after process or the new color of the center pixel

bottom-left a view of the distance from the center (from 'deep blue -> very close' to 'deep-purple-red -> very far') normalized

bottom-right a view of the coefficients used for the wheigthed mean calculation. (from 'deep-blue -> 0' to 'deep-red -> 1')

images are taken from clipboard_006 :
sky_001 -> the sky (we can see that the blurring effect is very strong, the matrix of wheights is red on a large area)

tree_001 -> at the top of the trees on the left side

detail_001 -> detail near one of the firebomb

vert_001 -> the rooftop on the background (we can see that the blurring effect act only following the vertical axis, so remove the noise whitout interfering with the pattern of tiles. Indeed this filter is able to follow any shape and to adapt even to bended pattern)

vert_002 -> another exemple near the first wood-post

diag_001 -> near the face and the cap of the boy

[ambrotos]

ok I tried some natural photos...

the result is not so good ^_^'

I will release matlab source soon...
I'm waiting your view or opinion about this experimental filter

always the same url
directory -> /ie

[Dali Lama]

Hi ambrotos,

I still see considerable noise in the blue sky. Otherwise, a lot of the grain is gone and visible blur is minimal. Although from your cat picture, it looks like color noise is not well removed (is that implemented there?). Overall, I'd say it looks good.

One thing to note is when you remove constant grain noise, if you don't remove it all, under compression it tends to block up.

All the best,

Dali

[ambrotos]

indeed,

the first implemented version was dealing only with B&W pictures. I just extended it by doing an average (R+B+G)/3 for each pixels so the chroma noise is not detected (?) yet...

that's the main problem. As I was thinking to use it in a YUV or YV12 colorspace... but I need to change it a bit.

First, I need to understand better how to blur in a colorspace where chrominance is coded every 2 pixel ? ^^ and then I need to find how to deal with YV12 by an avs plug-in...

[tsp]

you could just treat each plane as a BW picture and do the filtering for YV12.

[ambrotos]

it is close to what I am doing now. But since the filter is an adaptative one if there is only a chromatic noise (giving an uniform grey in B & W) it doesn't detect it correctly so it doesn't treat it.

BTW I have found some very interesting web pages and threads about colorspace and interlaced YV12 and ...

I think I will ask shOdan some details about ebmp format... and work again

[sh0dan]

Richard Berg wrote that part. It seems like he writes a "BITMAPFILEHEADER" and then a "BITMAPINFOHEADER" struct and then writes the raw frame data. (Y then followde by U and V). The source that reads and write it is in
ImageSeq.cpp (http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/sources/ImageSeq.cpp?view=auto&rev=1.11&sortby=date&only_with_tag=MAIN).

[Richard Berg]

ebmp (Extended BMP) is something I made up a couple years ago when I decided it would be nice for Avisynth to be able to save raw frames in all of its native formats. The spec is very simplistic if you've seen BMPs before.

[ambrotos]

Thank you very much ^_^

Let's continue

I know that this post it's a bit childish ... but it helps me keep my head clear

if I want to calculate the mean value of two color pixels
in RGB24 color space : pixel_1{r1, g1, b1} and pixel_2{r2, g2, b2}
the result will be : pixel{(r1+r2)/2, (g1+g2)/2, (b1+b2)/2}

I guess that in the case of a weighted mean it is very important to keep the same weigths for each R, G and B values (to avoid chromatic aberations). That's why I used a black and white picture to calculate the weights. If we take the weight 'k1' for the pixel_1 and 'k2' for the pixel_2
then the result is
pixel{(k1*r1+k2*r2)/(k1+k2), (k1*g1+k2*g2)/(k1+k2), (k1*b1+k2*b2)/(k1+k2)}

if the transformation from RGB to YUV colorspace is a linear mapping I should use the same argument and calcutate the value like that
for pixel_1{y1, u1, v1} and pixel_2{y2, u2, v2}
the resutl is pixel{(k1*y1+k2*y2)/(k1+k2), (k1*u1+k2*u2)/(k1+k2), (k1*v1+k2*v2)/(k1+k2)}

But in YUV colorspace luma is separated from chroma. So, could we separate the processing of the Y layer and the two chroma layers U and V ?
If we take the weigths 'k1' and 'k2' for Y
and take the weigths 'k3' and 'k4' for U,V then
pixel{(k1*y1+k2*y2)/(k1+k2), (k3*u1+k4*u2)/(k3+k4), (k3*v1+k4*v2)/(k3+k4)}

As soon as my holidays are over I will test... and see

[ambrotos]

looking for a way to deal separatly with luma noise and chroma noise, I designed a small matlab script which convert from RGB -> YUV and display each channel as a monochromatic picture.

we can notice that there is very few luma noise and that all the noise is concentrated on chroma channels

http://tinyurl.com/662n8
follow /ie/yuv_colorspace

So now I can continue and improve my initial filter...

[ambrotos]

I found !

the filter I was reinventing was an improvement of a bilateral filter (like the trictical's one) but more precisely it was based on a discrete beltrami flow which use the geodesic distance (in 3 dimensions - two in the plane and the last one in the colorspace) from the center pixel to calculate a weighted mean

http://www.cs.technion.ac.il/~ron/belt-html/node1.html
http://www-sccm.stanford.edu/~elad/Lectures/Bilateral_Filter.ppt


so I will continue 'cause I found no beltrami filters for avisynth and it could be very usefull for anime