Dust - a noise remover

[SansGrip]

Originally posted by MaTTeR
You may have noticed more and more users on here with dualies though, they are way too cheap these days;) I nearly got a dual system last time I upgraded (a few months ago) but couldn't quite afford it. Next time :D.

If the filter's already written in MMX, though, there's not that much scope for optimization other than algorithmic. This might be a good candidate to include in an encoder, since it's doing ME anyway...

[soulfx]

I must say, nice work on this filter.

I ran a couple tests and found it does a great job at de-noising without sacraficing detail. I did notice the blocking during motion, but it's not really too bad and with C3D it's even less noticable.

I've been workin with Peach for a while, but really like where Dust is heading. Peach does a great job for the speed it goes at, but for really taking out the noise, this filter's great. Dust drops my fps down to 1-4, but hey, my computer doesn't mind the work.

By itself Dust doesn't handle interlaced content very well. I had to seperate the fields, filter, and weave, in order to get the results I was looking for. Maybe something could be worked in to future versions were it can handle this content better by itself?

*Sigh* I too dream of a day in which I have a MP machine.

Keep up the great work, Steady.

[Acaila]

I am seeing something I can only describe as "checkerboard-patterns" appearing in the frames directly before and directly after scene changes. It is more pronounced with Pixiedust than with Faerydust.

I'm using a bare-bones avisynth script with only BicubicResizing (before resizing) and am using it with Avisynth 2.07.

If you're interested I can attach a picture later.

Btw, are there special requirements regarding width and height MOD for these filters? I get the above with cropping to width MOD 4 and height MOD 8.

EDIT: Also, both filters create a black line of about 2-3 pixels wide at the right side of each frame.

[bb]

@Steady:
I wonder whether it would be possible for your filter to use XviD's or DivX5's mv hint files to work faster. I'm asking myself if it's really necessary to do the motion search twice (encoder and denoiser).

bb

[FuPP]

aren't you assuming that all people encode with divx or xvid ? What about mpeg1 or mpeg2 encodes ? Or may be I didn't get you...

[MfA]

XviD cant efficiently use hintfiles at the moment, when it can it might be usefull for dust to write hint-files (that would be better than reading them, for one it would work with 1-pass coding too and also XviD's motion search tries to minimize rate instead of finding "true" motion). It is not an issue at the moment though.

FuPP, which MPEG1/2 encoders can use motion hint files?

[onesoul]

Originally posted by SansGrip
Would that be useful for those with uniprocessor machines? Hi. The new processor of intel the Pentium IV with hyper threading can work on two separate threads of an aplication, running as a dual processor does. Only the processor of 3.06 ghz or higher support HT and requires a specific chipset and also an OS (Windows Xp supports it). The bad thing is that is expensive, here at Portugal the PIV costs 846€. The board has a "normal" price.

[Steady]

First let me stress that this is a very early release. It will be continue to develop not in a matter of days or even weeks but months. I had not really intended to release it for a couple of months but was so surprised at how effective this "first generation" version is that I decided to put it out. It is all to easy to get 'tunnel vision' when writing, and the feedback helps me keep the priorities straight.
bb asks
I wonder whether it would be possible for your filter to use XviD's or DivX5's mv hint files to work faster. I'm asking myself if it's really necessary to do the motion search twice (encoder and denoiser).
The short answer is no. On the surface, motion search for an encoder and for noise removal would seem the same. As you get into the details they become more different. It starts with different goals: An encoder wants to find the closest match to minimize the differences it needs to encode. My filter needs to find the same area in the screen; which may or may not be the the closest match. In low-detail areas with heavy noise it will almost certainly not be the closest match (lowest SAD). The presence of noise greatly complicates a motion search (and without noise there is no point). There are many more differences. My subpixel precision needs to be much better than an encoder, which it isn't yet. Also an encoder uses 16x16 blocks. I use 8x8 blocks. Down the road I may start with 16x16 blocks for the wide search, 8x8 for local search, and 4x4 for subpixel interpolation.
Acaila mentioned
I am seeing something I can only describe as "checkerboard-patterns" appearing in the frames directly before and directly after scene changes. It is more pronounced with Pixiedust than with Faerydust.

Btw, are there special requirements regarding width and height MOD for these filters? I get the above with cropping to width MOD 4 and height MOD 8.

I am guessing that is with fairly dark scenes?
The scene change detector will need tuning. I need to adjust the threshold to the overall screen brightness.

I should have mentioned the size needs to be a multiple of 8. Any remainder after dividing by 8 will not be filtered on the right and bottom edges (though it will be used for the motion search).


I am currently working on the "blockiness with motion" issue. It is turning out to be be a deeper and trickier issue than I had originally thought. It will probably require another level of pre-filtering which will slow things down further (sigh). If it works, a pleasant side effect will be a much improved ability to deal with changes in brightness.

[bb]

Steady,

thanks for clarifying. I somehow had the feeling it wouldn't be that easy...

bb

[MfA]

Originally posted by Steady
The short answer is no. On the surface, motion search for an encoder and for noise removal would seem the same. As you get into the details they become more different. It starts with different goals: An encoder wants to find the closest match to minimize the differences it needs to encode.

Not entirely true, it tries to find the match which takes the least bits to encode ... with modern ME this takes into account the amount of bits needed for the motion vector. The end effect is not too much unlike the smoothness constraint you would find in "true" ME, although especially at higher levels of quantization it tends to smooth out the motion field more and ignore small changes (since sometimes its easier to just encode the differences than to encode the "correct" motion vector).

My filter needs to find the same area in the screen; which may or may not be the the closest match. In low-detail areas with heavy noise it will almost certainly not be the closest match (lowest SAD). The presence of noise greatly complicates a motion search (and without noise there is no point). There are many more differences.

Still the found motion vectors would provide great candidates for a new motion search, even if that one is rate optimal. Especially at motion-discontinuities/edges having a MV candidate which allowed you to skip most of the gradient search for the new surface inside the encoder might provide a decent speedup.

XviD cant use hints in this way at the moment, it just reuses them as is, but it might in the future.

My subpixel precision needs to be much better than an encoder, which it isn't yet.

Have you read Thomas Wedi's stuff on wiener interpolation filters (http://www.tnt.uni-hannover.de/~wedi/papers-html/EI01-TW.html)?

Additionally you might be interested in better matching criteria, changes in illumination arent the only problem ... motion discontinuities cause large motion vector errors. Dirk Farin has some nice pictures in one of his papers (http://www.informatik.uni-mannheim.de/informatik/pi4/publications/library/Farin2001c.pdf) showing the effect when using the MSE criteria, I doubt SAD does any better. His solution to the problem is rather compute intensive though. I have seen ME which performed matching in horizontiallly and vertically sobel filtered images, simply summing SADs for both, which seems more practical (this is also mostly insensitive to illumination changes since it ignores the DC component).

Also an encoder uses 16x16 blocks. I use 8x8 blocks. Down the road I may start with 16x16 blocks for the wide search, 8x8 for local search, and 4x4 for subpixel interpolation.

Are you sure that is wise? In XviD a coarse to fine search makes sense, for 4MV, since most for most MBs the fine search is never done. If all MVs have to be eventually found at the finest level Im not sure if it still makes sense though, I doubt the coarse MV is a better predictor than the median predictor at the finest level ... you might even end up having to do more work.

Marco

[DVD__GR]

I tried it in interlaced source and use smartdeinterlace with blend denoise and feild only difference,i can tell it cleaned some shitty areas and improved a bit the compressibility (i encoded to svcd via cce) but ITS TERRIBLY SLOW.!!
nevertheless its pretty good..
Congratulation for your work
(could we expect more speed in future??)

[digitize]

Im sure Steady is trying his hardest to optimize this filter, but would you rather speed and mediocre (sp?) results, or great results and not too good speed, personally I'd go for the latter. But like I said Im sure Steady is working his hardest, good work Steady ;)

[Boulder]

Seems like a very powerful denoiser, but unfortunately too slow for me as I can't keep my computer running at nighttime. I'll have to wait for the speed version (AngelDust?) then;) The quality is really excellent!

[Steady]

Originally posted by MfA
Have you read Thomas Wedi's stuff on wiener interpolation filters (http://www.tnt.uni-hannover.de/~wedi/papers-html/EI01-TW.html)?
1)
Additionally you might be interested in better matching criteria, changes in illumination arent the only problem ... motion discontinuities cause large motion vector errors. Dirk Farin has some nice pictures in one of his papers (http://www.informatik.uni-mannheim.de/informatik/pi4/publications/library/Farin2001c.pdf) showing the effect when using the MSE criteria, I doubt SAD does any better. His solution to the problem is rather compute intensive though. I have seen ME which performed matching in horizontiallly and vertically sobel filtered images, simply summing SADs for both, which seems more practical (this is also mostly insensitive to illumination changes since it ignores the DC component).

Those are some interesting papers thank-you. (Don't you just hate those postscript to .pdf files?) Do you know offhand where I can find a paper on wiener filters - preferably less calculus intensive (I could care less about proofs and such). I am going to start out with a 4-tap catmull-rom filter. The weights for tau=0.5 are -1/16, 9/16,9/16,-1/16. I may be able to do this using just shifts and adds without even unpacking to word. Later I will look at filters with greater extent. I also see I how will implement the subpixel search. I will use a 'logarithmic diamond search' that starts with the 4 half-pels left,right,top,bottom; then does 1/4-pel search around the best; down to 1/8 pel. This should be a good comprimise between speed and accuracy (though not all possible positions can be reached - particularly on the diagonals).

edit: Yesterday I looked at adding highpass filtering to remove the DC component (just a quick and dirty IIR filter that simulates a 1-capacitor, 1-resistor filter). It didn't seem to help any. I thought about adding 7x7 or so general convolution FIR filter. But that would be quite slow for a minor improvement. (I already use a 3x3 1 2 1 filter to remove the worst of the noise and improve neighbor-pixel similarity)

edit edit: Stupid me, I forgot you can't shift bytes in mmx.

[trbarry]

I don't know the filter math but can't you remove the DC component just by subtracting the average?

- Tom

[Steady]

Yeah, but how do you get the average? That is where the 7x7 would come in - average 48 pixels together and remove that from the current pixel.

[MfA]

Originally posted by Steady
Those are some interesting papers thank-you. (Don't you just hate those postscript to .pdf files?)

As long as I can do text search and its mostly readable I dont care much, it should look ok when printed.

Do you know offhand where I can find a paper on wiener filters - preferably less calculus intensive (I could care less about proofs and such).

No ... but the filter taps are in the paper, calculus wise that is about as low intensity as it is going to get :)

I am going to start out with a 4-tap catmull-rom filter. The weights for tau=0.5 are -1/16, 9/16,9/16,-1/16. I may be able to do this using just shifts and adds without even unpacking to word.

I doubt it will be faster than simply using multiplies with MMX.

Yesterday I looked at adding highpass filtering to remove the DC component (just a quick and dirty IIR filter that simulates a 1-capacitor, 1-resistor filter). It didn't seem to help any.

The more usual approach is to simply take the average of the block and subtract it (you can use a summed area table to quickly find the average for an arbitrary block during ME). One problem is that the DC component plays a big role when you are doing a gradient search though.

Just removing the influence of illumination on ME is only half the problem though, what are you going to do after you find the correct motion? Is simply calculating a scale factor for luma before applying the temporal filtering enough?

Marco

PS. on second thought, for 8x8 blocks it is probably better to just calculate the average on the fly.

[Shamballa]

what's going on on Version 4 ??? see that the first is v3 version et the second v4 version ^^


http://www.lance-dragon.com/images/dustv3.jpg




http://www.lance-dragon.com/images/dustv4.jpg


i precise a have this litle white point in a lot of darknes scene is exactly the same problem i had with the refdivx lumimasking when is buging (is fixed now) i hope i can help you in this beautiful filter's

[SansGrip]

Originally posted by Shamballa
i precise a have this litle white point in a lot of darknes scene Looks like overflow to me. I've never seen this with v4.

[Shamballa]

the ref lumi masquing is based on wavelet code is my memory is good if that can help i have tested it on another anime and i have it too. you can see the image is more crapi too with the v4 i precise i use the parametre fairies() (is have tried pixies too is the bug)

[Steady]

Originally posted by Shamballa
I have these little white points in a lot of dark scenes.
Can you post a link to a very small clip that shows this problem? (10 frames or so). I am on dial-up so I cannot download large files.

I am close to releasing version 5 which will reduce blockiness. The fix will probably be in that version.

[Shamballa]

mm i was removed all other filter i forget only one filter i can imagine have effect on the denoiser is a filter of color (coloryuy2) if i load the video with only coloryuy2 no problem when i load with only dustv4 no problem but when i put this 2 filter i have this white atrifact but when i put dustv3 and coloryuy i have no problem so it's only a problem of cohabitation in the dustv4 ^^ sorry for not have found sooner .....:(

if you want tested :

http://www.lance-dragon.com/images/sample.avi

[trbarry]

edit: Yesterday I looked at adding highpass filtering to remove the DC component (just a quick and dirty IIR filter that simulates a 1-capacitor, 1-resistor filter). It didn't seem to help any. I thought about adding 7x7 or so general convolution FIR filter. But that would be quite slow for a minor improvement. (I already use a 3x3 1 2 1 filter to remove the worst of the noise and improve neighbor-pixel similarity)

FWIW, the psadbw instruction calcs the sum of abs byte differences. So when compared to zeros it calcs the sum of 8 unsigned bytes. Thus it's fast for calculating area avgs (8 wide). But for MMXSSE only.

- Tom

[SansGrip]

Originally posted by trbarry
But for MMXSSE only. Isn't it iSSE?

[trbarry]

SansGrip -

Dunno. Is this just terminology or do I have the wrong collection of machines. I always just called it SSE suport until I realized that I only use integer SSE, which is okay with Athlons. And code like Xvid calls it SSEMMX or MMXSSE so I started to also.

But maybe I have the completely wrong generation of support here. ??

- Tom

[MfA]

Oops I just realized that the link to the Dirk Farin paper I gave was a different file than I had on my own HD ... I converted the PS version also available on the web myself, in which case you do get a text searchable version.

[Steady]

Originally posted by trbarry
FWIW, the psadbw instruction calcs the sum of abs byte differences. So when compared to zeros it calcs the sum of 8 unsigned bytes. Thus it's fast for calculating area avgs (8 wide). But for MMXSSE only.

- Tom
I use psadbw heavily in the search algorithm. With everyone complaining about how slow it is, I was hoping to avoid another pre-filtering/analysis pass, but it looks like I am going to need the average luma for several different things.

I think of psadbw as MMX+ or extended MMX. I think the confusion comes from Intel and AMD putting the same instructions in different class names. AMD is the main culprit here, and their documentation is almost non-existant (at least on their public website). I have to use Intel docs for instruction set reference. I can find no complete reference anywhere on AMD's site.

I know people want more speed from this filter, but it is going to be a long time coming. It is under heavy development/experimentation. I don't want to waste a lot of time optimizing routines for speed when I will just have to throw them out later when that routine gets superceeded.

[Steady]

Shamballa: I cannot recreate the problem. Can you post your exact script? Also is you CPU Intel or AMD?

[sh0dan]

Look for "MMX Extensions" on AMD's site - or go to my CVS-bin site (see sig.) I put up a lot of assembler documentation there. Download the "3D Now MMX extensions.pdf".

[SansGrip]

Originally posted by trbarry
I always just called it SSE suport until I realized that I only use integer SSE, which is okay with Athlons. To me MMX means either simply "multimedia extensions" or refers to the first version of MMX that was introduced with the Pentium Pro (I think). SSE is the next step up (Pentium IIs?) and iSSE next from there (Pentium IIIs), and finally we have SSE2 (Pentium IVs).

The only reason I ask is because if that instruction is from iSSE it'll only work on Pentium IIIs and above. Probably not a big concern, but there you go ;).

[vlad59]

@SansGrip

MMX -> Pentium MMX, Pentium II, K6, K6II, K6III and later
iSSE -> Pentium III, all Duron (called 3DNow extension), all Athlon (called 3DNow extension)
SSE -> Pentium III, Duron (core Morgan), Athlon XP and later
SSE2 -> PIV

I hope I'm right ;)

[Shamballa]

i have put the detail in pm for personale reason ^^ for shodan are you still interested to know what is the interpolation in coloryuy site ? i can try translate it if you want ;)

[d4Rk Ch1Ld]

i wonder will there be any version of Dust for avisynth 2.5a ?
it would be interesting if someone converts it...

[Koke]

Originally posted by d4Rk Ch1Ld
i wonder will there be any version of Dust for avisynth 2.5a ?
it would be interesting if someone converts it...

So it doesn`t work on 2.5a?

Hi Junior!

[d4Rk Ch1Ld]

Originally posted by Koke
So it doesn`t work on 2.5a?

Hi Junior!

just because i mostly observe and read post and dont take a part of discussion in them doesnt mean i am junior ;)

and yap it crashes 2.5a :( tried with 2.07... it does some great job on noisy material but its damn slow i'll rather use in that case c3d or smoothflux and undot with 2.5a and optimized mpeg2dec3, get same results with faster processing...

but its up to the author if he is going to put version of 2.5a version of plugin... catch ja on h.r.d ;)

[Koke]

Originally posted by d4Rk Ch1Ld
just because i mostly observe and read post and dont take a part of discussion in them doesnt mean i am junior ;)

Ups!
I forgot to draw a :p !
I was kiddin, of course...

I`m encoding Gattaca right now.
17 hrs first pass done soon :eek:.
It flies at 2-3 fps on celeron@900.
It flies like hell :).
So I set priority on lowest and
my PC is still usable.

Adio!

[d4Rk Ch1Ld]

Originally posted by Koke
Ups!
I forgot to draw a :p !
I was kiddin, of course...

I`m encoding Gattaca right now.
17 hrs first pass done soon :eek:.
It flies at 2-3 fps on celeron@900.
It flies like hell :).
So I set priority on lowest and
my PC is still usable.

Adio!

i tried it... got 20 fps on black screens (Credits and intro ;))
and then fps dropped around 8-12 fps

got p4 1.8 GHz(northwood) @ 2.4 GHz(533fsb)

[Steady]

Version 5 posted

This version should greatly reduce the blockiness with motion issue. It has many internal changes.

Two new filters introduced.
GoldDust Heavy filtering for more serious problems. This one will definately blur the image, so it recommended you use it before resizing smaller.

SpaceDust This fast filter is spatial only. For those complaining about how slow Dust it:)

[Koke]

Man, you are fast!
2 hrs more to finish my Gattaca 2nd pass vith dust4,
and I have a new dust to try.
Me happy.:)
Thanks

[Candock]

:( just used 8 hours to finish the filtering of a incredible noisy 42min-movie with dust4.

[mikeson]

@Steady:

Many thanks for new version of this great filter!

May I ask what about AviSynth 2.5 support?

[lancer]

nice filter.

came across this yesterday on the forum. in combination with msharpen and convolution if gets rid of some nasty grain.

god, disc 1 side 1 and 2 on star wars are bloody awful. dead soft, lot of grain. side 2 disc 1 on empire is not great either.

convolution was a lot better on color areas than 2dcleaner but I still had spotty grain. if I lofted it's variables too high the detail disappeared. but faerydust is all over them.

neat

[onesoul]

I have to share my thoughts cause I am just a noob :)
I am ashamed but I admit that first when Steady appeared with this dust filter I was in doubt of it (there are so many filters and not always do what we would like), and not even have ever heard of Steady it doubled my doubts. I was wrong.
I see now dust as a pearl and I belive it will become the ultimate denoise filter.
I have not done any really encodings, just tests, and here is what I have done: I grabbed a divx(movie on 700mb) and put as the source on avscompare (the ones who didn't try it yet, please do, it's great) and I compared fluxsmooth(15,10), convolution3d(preset="movieLQ"), nomosmooth(), dnr2(), pixiedust() and no filter at 1 one frame. With no doubt pixiedust was the winner, and when I zoomed the picture I was amazed by the quality achieved (not blocky, fine detail), none of the other filters or no filter at all would match pixiedust.

I will use dust from now on. Great work Steady. :)

[^^-+I4004+-^^]

and now add some peaches and grapes....,and test again...

[digitize]

@Steady
Perhaps this has been asked before, but I haven't seen it so I'll go ahead and ask. What is the spacial matrix for the spacial cleaning part of this filter? Im guessing it is 3x3 like C3D, but I could be wrong. And keep up the good work ;)

[Steady]

Originally posted by digitize
@Steady
Perhaps this has been asked before, but I haven't seen it so I'll go ahead and ask. What is the spacial matrix for the spacial cleaning part of this filter? Im guessing it is 3x3 like C3D, but I could be wrong. And keep up the good work ;)
It starts with a notch filter at the Nyquist frequency (a 3x3 1 2 1 filter). Then it blends this with the raw image on a pixel by pixel basis depending on local detail. The 1 2 1 filter is also good at reducing residual interlace artifacts.

[digitize]

Ah I see, thank you for the response.

[onesoul]

A litte update to my tests (I hope someone finds it useful):
With noisy sources pixelized: pixiedust() is the one to try.
With clean sources: try fluxsmooth(15,10).
Suggestions?

[iago]

With clean sources: try fluxsmooth(15,10)

With clean sources: first, try no filtering at all ;).

[rubic42]

I'm not certain what is going on, but this is what I'm seeing:

The following code produces video with one field interlaced with a completely black field in each frame:


SeparateFields()
e = SelectEven().GoldDust()
o = SelectOdd().GoldDust()
Interleave(e,o)
Weave()


The following code does not have the black field in each frame, looks normal:


SeparateFields()
e = SelectEven().GoldDust()
o = SelectOdd()
Interleave(e,o)
Weave()


What's going on? Is this the proper technique for applying GoldDust() to separate fields in an interlaced video?

Thanks.