Dust - a noise remover

[onesoul]

Originally posted by iago
With clean sources: first, try no filtering at all ;). [/B]I sure had it coming :). But are there any real "clean" sources? I do a little magnifying at avscompare and I see noise and honestly the fluxsmooth gave me the better result with "clean" sources (mostly DVD's).

@rubic42

Have you tried pixiedust()? Goldust is stated by Steady to be experimental. Maybe it is the reason.

[Steady]

Originally posted by rubic42
What's going on? Is this the proper technique for applying GoldDust() to separate fields in an interlaced video?

Well it should be but ... This is related to the 'only one instance per filter' issue. I am not sure why (something about how it interacts with avisynth). But for now if you have more than one (dust) filter in a script you get odd behavior.

For those asking about avisynth 2.5 version ... Of course there will be a avs2.5 version eventually. But Dust is constantly changing, and avisynth2.5 is constantly changing. And I just don't need another source of weird bugs to track down. In a month or 2 when things have stabilized a bit.

Please remember that this is an Alpha version not a Beta. I consider an alpha as something that you are not even sure what it does yet. Beta is more where you make it work stable and fast. If you you this, you may get odd results, crash your computer, or martians may come cut off your head and put it on backwards. Almost certainly you will regret using it on something permanent when the much improved Dust17 comes out down the road.

One other thing I keep forgetting to mention. The height and width should be a multiple of 8 or you may get strange artifacts.

[rubic42]

Well it should be but ... This is related to the 'only one instance per filter' issue. I am not sure why (something about how it interacts with avisynth). But for now if you have more than one (dust) filter in a script you get odd behavior.

I remember reading that caveat, but somehow it didn't connect for me that that would be causing my problems. Thanks for the info.

Otherwise, the filter looks great! Even when it's not operating on interlaced video the right way, it looks awesome. Keep it up.

[onesoul]

Originally posted by Steady
One other thing I keep forgetting to mention. The height and width should be a multiple of 8 or you may get strange artifacts. Thanks for the warning, maybe you could put that at the readme file :)
I got a question, maybe a dumb one: Applying this filter or other filters to a picture with and without black borders (supposing both height and width are multiple of 8) will make any difference to the final quality of the picture?

[Rhaegar Targaryen]

Originally posted by onesoul
A litte update to my tests (I hope someone finds it useful):
With noisy sources pixelized: pixiedust() is the one to try.
With clean sources: try fluxsmooth(15,10).
Suggestions?

Have a look through the original FluxSmooth thread started by the program author. Many people seem to think that even the default values of FluxSmooth (7,7) are too strong for clean DVD's - me included. A more conservative setting such as (5,5) is what we find more appropriate.

[Assault]

@ onesoul

Fluxsmooth is a great filter but I agree with Rhaegar Targaryen and many other forum members here. Fluxsmooth (15,10) is definately too high for a clean dvd source. Try convolution3d moviehq and compare the result to fluxsmooth (15,10). You'll see there's much more detail kept by convolution3d. Perhaps this convinces you... ;)

Regards
Assault

[onesoul]

Originally posted by Assault
Fluxsmooth is a great filter but I agree with Rhaegar Targaryen and many other forum members here. Fluxsmooth (15,10) is definitly too high for a clean dvd source. Try convolution3d moviehq and compare the result to fluxsmooth (15,10). You'll see there's much more detail kept by convolution3d. Perhaps this convinces you... ;) I respect your opinions, but as I said before I tried convolution3d(preset=movieHQ") but I found it ruins too much the picture for me. Some thoughts on which filters applie can be found here (http://forum.doom9.org/showthread.php?s=&threadid=44264). Personally I didn't find much differences of fluxsmooth(7,7) with fluxsmooth(15,15) when applied after pixiedust. Keep in mind that I don't want to sell anything, I respect all authors who put their time on creating their filters and I thank them, I am only trying to find how to take the best out of them.

Cheers

[molerus]

Hi!

For a some time I'm fiddling with Pixie and Gold Dust. I encouraged my brother to using it, but he doesn't have so much time as I. So I wrote the ultimate script to use this filters with all kind of video. But since Dust requires both width and height of video to be divisible by 16 I'm adding a frame prior to filtering and after that I crop back. Since I use filtering in interlaced video I downloaded and modified the filter ViewFields. This filter separates the fields and puts them one under the another, and after Pixie or Gold Dust it interleaves them back.
But when I crop the image horizontally (vertically everything's ok) the filter fails. Although the width of the picture in VDub is ok, my filter recognizes it wrongfully and interleaving fails. I doscovered, that when I add borders with 0 width everything backs to normal.
Has anyone had the same problem?

If you want me I can enclose my de- and interleaving filter.

[Koke]

We want filter...
We want filter...
:)
Please...

[molerus]

Hi!

I enclose my filter (in fact there are two filters in it - ViewFields to de-interlace and MergeFields to interlace back), along with the script and source code. I added some not-so-wise comments, pointing the two errors in AviSynth.
Firstly, as you will probably notice, there is a line "AddBorders (0, 0, 0, 0)". Althouth it looks silly it's the only way that MergeFields could work correclty. Otherwise it scrambles the picture.

The second thing is using AviSource with "RGB24" option and then ConvertToYUY2. I use capture card to record from TV and this card uses the MJPEG codec. When AviSource is used without "RGB24" option it "wraps" the picture, putting the last line first, but only with MJPEG (I use Morgan codec) :p .

Uff!! So many errors - so little time ;)

However: here is the filter
Have fun

[Jellygoose]

Hi Steady!

Me and many others at KVCD.net are anxiously waiting for the next relase of your excellent filter. I just post here to show that I really appreciate the work you're doing! Keep it up!

Regards
Jellygoose

[Si]

@a moderator

I can't down load the attachment. (can now - thanks)

I'd like to look at it since I can't make any sense of the posts and see if there is a problem with my filters or molerus's "filter" (script? :confused: )

Could one of you look into it please.
regards
Simon

[Si]

@molerus
WT* :angry:

If you are going to modify a filter - I'd suggest you change the name to reflect the changes you've made otherwise people (not least the author :angry: ) are going to get confused :(

Another thing that comes to mind is that this is a thread about Steady's Dust filter and to publish another filter in it is a bit rude :(

Normal etiquette would be to email (or PM) the author if they are currently active (which I am :( ) and suggest changes to their filter so everyone can benefit and saves unneccesary proliferation.

And on a personal note what does MergeFields do that UnViewFields doesn't? (but don't actually bother replying to this question as that would contravene point 2 :( )

yours sincerely
S Walters :mad:
PS sorry Steady </rant>

[molerus]

Ok!

So here is the thing. I encoded "The Terminator" with the combination of PixieDust and Convolution3D. The whole film looks just astonishing (2CD + AC3 sound), but some parts became totally incompressible. Scenes where there is a lot of smearing smoke got horrible blocky. At first I thought it's the fault of lack of the luma noise correction. So I had ported NanDub's luma noise estimation routines into a VirtualDub filter, and then computed the noise. Next I muxed the computed values into XviD stat's file and compressed bitrate curve with GKnot, using luma correction. But it appeared that it didn't help, blocks were still there.
I encoded the problematic scene with constant quant 2 and the same happen - blocks and blocks. Only when I used quant 1 (ridiculous I know :p) the block have gone. Is there way to force XviD to use quant 1 for such a problematic scenes?? If it will be 1% of the film I can live with that. If not what are the other solutions??
Maybe adding dust with the use of information about luma noise would be useful?? Maybe someone has made such a filter??

P.S. If someone got angry by my recent post then please accept my sincere apology.

[MfA]

Id start a new thread for that question molerus, instead of tacking it on a 6 page existing thread specifically concerning an avisynth filter which has nothing to do with your question anymore. I suggest the xvid forum.

[Didée]

molerus,

you should read again my answer to your question in your "XviD new luma correction" thread.

:rolleyes:

[crOOk]

originally posted by Steady
82% of the time is spent on motion search
First of all: This filter really blows my mind. Even in this early stage of developement it already does a perfect job (besides the scene changes). Another thing that blew my mind was it's speed... :o
Not that I want to sound demanding, but for me (and I think I'm not the only one there) who does LOTS of encodes (several ones each day) it is just too slow to be used on a regular rip, but I guess you already knew that. ;)
Since the motion search takes such a lot of encoding power and 'most' people do two passes in their encodes, wouldn't it be possible to only do the ME once? You could write the results of the whole process into a file (like XviD does) and reuse this information in the second pass, couldn't you? This could save hours of encoding time. Most likely you have already thought about this, so let me just ask you if there are plans to implement such a feature in a future version. Keep up the brilliant work!

[MasterYoshidino]

would not make any sense to diasable a filter for comp test since the enviroment is different.

your bitrate distribution will be incorrect, as it will be like using TemporalSmoother() for the first pass and FaeryDust() for the second. You have to do both passes with exact same script and settings for a proper 2 pass encode.

you can't expect better compressability and speed :\

[crOOk]

I'm not talking about applying the filter in one of the passes only. I'm just saying that the motion estimation process is done twice. How does the ME affect the quality of the uncompressed video??? The result for both passes would still be the same. Even the script could stay the same. The filter would just need to check if there already is a file that includes the results of the first motion search (similar to the mvh files that XviD can create), so they can be reused in the second pass (If there is no file the filter would create one). This way the motion search does not have to be done twice...
@MasterYoshidino
Either you have not read my post carefully enough or I am terribly wrong about this whole thing. I just don't understand what you're saying. Have I missed the whole point of Motion Estimation?
@Steady
In case I have not missed the whole point of the ME, how much time could be saved by reading the ME results from a file instead of recalculating them all over again? You mentioned that the ME takes about 82% of the processing power needed by your filter.

[wingphil]

yep, he didn't read your post properly ;)

if steady is even still developing this filter he's more likely to be working on a yv12 version i would hope (hint hint :D)

[crOOk]

Thx for being so kind and clearing this up (MasterYoshidino doesn't seem to visit this forum regularly). Actually I've got to admit: I'd love a YV12 version, too. Especially because there'd sure be some speed increase in it as well.
Whatever, I'm using the filter on most encodes right now and am getting around 8 fps (XP2700+). The 'trick' is that I'm applying Dust after resizing and it is still doing a very good job with removing the noise and increasing the compressability without blurring too much.
I love it! Steady, please don't give up your project!

[wingphil]

it's a brilliant filter, and i used to use it every time i did a 1cd encode, but now i've changed to yv12 it's just not worth the slowdown any more.

come on steady :D

[Holomatrix]

Just wondering if we are any closer to a YV12 version? I'm running some tests on clean DVD material and want to be able to test PixieDust YV12. I'm taking Fluxsmooth over Convolution3D and would like to test Flux against Dust in YV12. Thanks

[MasterYoshidino]

hah !
lol :( a yv12 would be nice though as this is the only temporal filter that seems to do the job no noisy anime (yv12 has little need for spatial filtering but I notice I need some temporal filtering still :D )
doh why do I have to stick wtih 2.07 :p ( conversion from YUV2 to YV12 for direct stream encoding causes color errors in encoding MPEG4's)

[sh0dan]

Unless you are using captured material in huffYUV YUY2, you are still having the same number of colorspace conversions - they are just at different placed.

2.0:
MPEG2DEC (YV12) -> MPEG2DEC (YUY2) -> AviSynth (YUY2) -> MPEG4 (YUY2) -> MPEG4 (YV12).

Conversion are YV12 -> YUY2 -> YV12

2.5:
MPEG2DEC (YV12) -> AviSynth (YV12) -> AviSynth (YUY2) -> AviSynth (YV12) -> MPEG4 (YV12).

Conversions are YV12 -> YUY2 -> YV12. Now it's just done within AviSynth instead.

[Richard Berg]

I emailed Steady a couple weeks ago offering to compile Dust for v2.5. Has anyone heard from him recently?

Oh, and forgot to mention, a friend of mine and I have found a pretty serious Crop bug.

[troy]

Can someone tell me how I can use this filter with an avi interlaced source. I read that molerus had some script to use but I could not find it to download. Has anyone made script yet to use for an interlaced avi source.

[Richard Berg]

Look at the bottom of page 1.

[troy]

Sorry I looked on this page. I can not see anything to answer my question?????

[scharfis_brain]

that's easy:



separatefields()
yourfavouritedenoiser(Parameters)
weave()

[ulfschack]

Nope that wont cut it entirely.

You have to separate the the evens from the odds by inserting:
separatefields()
even=selecteven().yourfavouritedenoiser(Parameters)
odd=selectodd().yourfavouritedenoiser(Parameters)
interleave(even,odd) #... possibly the other way around (odd, even)
weave()

Because if you have a 50 fps stream of 288 lines every other picture will be relatively moved half a scanline thus not providing the right information for a temporal denoise filter.

cheers

[Boulder]

@ulfschack: You can't use the filter in more than one instance in your script for some reason. I doubt that the script would work, I'm not 100% sure though;)

[ulfschack]

@boulder

You're absolutely right. In fact I even found this out by myself a few weeks back, but managed to forget :)

Even so, by not separating the evens and the odds be prepared for a much more blurred picture, as the filter with confuse pixels from different locations as being from the same.

If I remember correctly I abandoned Dust in favour of Peach for this reason alone.

cheers

[scharfis_brain]

separatefields()
yourfavouritedenoiser(Parameters)
weave()

but while using this script, you'll have a better temporal noise removal.

But the denoiser may have some trouble with the sharp flickering edges (or is this flickering positive for the denoiser???)

[Loul]

Originally posted by Richard Berg
I emailed Steady a couple weeks ago offering to compile Dust for v2.5.

I'm eagerly waiting for a 2.5x Dust version.

Dust is among the best (and maybe the best) denoisers and it would be so disappointing it would die from not being updated :'(

[trance]

I would to try Dust as well but have been unable because of AVS 2.5 (which I would imagine most people have installed).

[DoW]

Sigh... use loadpluginex.dll from the warpsharp package to load Avisynth 2.0x plugins (like dust). Just remember to put it before the loading of dustv5.dll and make sure you pass YUY2 to dust, and your set.
See: Plugin Chaos (http://forum.doom9.org/showthread.php?s=&threadid=48160&highlight=loadpluginex)

[FredThompson]

Any progress on a 2.5 compile? dustv5 and loadpluginex aren't playing friendly with the latext CVS Avs 2.5.2

[toshi]

Originally posted by trance
I would to try Dust as well but have been unable because of AVS 2.5 (which I would imagine most people have installed).

You had better use LoadPluginEx.dll. You can use Avisynth 2.0 plugin on Avisynth 2.5 by using this plugin.

[FredThompson]

As mentioned in another thread, it's not so simple. The 2.5.2 CVS build doesn't seem to play well and the question is where to find a version of loadpluginex.dll which does.

[Loul]

What's up with Dust ?

Has developpment stopped ? :(

It's my favorite denoiser but I'd like a 2.5x Avisynth version of it and we don't have any feedback :(

If someone has some news let us know please !

[digitalman]

So many people keep asking about it, I don't think there will ever be a 2.5 version. Kind of like certain great games on Playstation or Nintendo 64 were not brought over to PS2 or the Gamecube. Or certain movies that were on VHS, took years before they were on DVD.

[Loul]

That would be terrible.

What has happened to Steady ?

Can't find a post of him that would be of less than 5 months, though he's still in the moderator list of this forum...

[WarpEnterprises]

http://forum.doom9.org/showthread.php?s=&threadid=60703

[FredThompson]

I've been trying to gen up some development of scratch/dropout/dust filters for 2.5.x. There are some good technical articles in Donald Graft's library section of this website. Also, the SpotRemover site has some good theory explained. (Wish Konstantin would release an AviSynth version...) A few experimental filters exist but nothing that's really complete yet.

steady does seem to have gone on an extended sabbatical...

[kempfand]

Dust is basically the only AVS filter which helps with some of my problematic DV-avi conversions. In my experience and situation, I mostly use SpaceDust(). PixieDust(limit=5) is a bit better but too slow :eek: .

Only other filters that help is MipSmooth(preset = "MovieLQ"), or brute-force blur(0,1) / CCE LowPass 4-6.

I really would hope that someone can port Dost to AviSynth 2.5+ and add YV12 ;)

Kind regards,

Andreas

[WarpEnterprises]

someone can port
it is closed source :(

[crOOk]

it is closed source
That's what I don't get. No offense, but if Steady is too busy to develope it, why doesn't he make it open source?

[FredThompson]

steady has, apparently, disappeared. At one time, he was hoping to make dust a commercial product and nothing seems to have happened.

[FredThompson]

Originally posted by kempfand
Dust is basically the only AVS filter which helps with some of my problematic DV-avi conversions. In my experience and situation, I mostly use SpaceDust(). PixieDust(limit=5) is a bit better but too slow :eek: .

Only other filters that help is MipSmooth(preset = "MovieLQ"), or brute-force blur(0,1) / CCE LowPass 4-6.
PAL or NTSC? What, exactly, are the quality issues you are trying to filter?