RemoveDirt

[kassandro]

Originally posted by SILICON
One temporal filter, check the previous and the next frame (frames -1, 0 and 1).
If no motion for frame -1 to frame 1, you filtes think thae the motion in frame 0 are noise. Isn`t it?

That is basicly correct. However, as motion may be restricted to parts of a frame, this strategy is carried out for small 8x8 pixel blocks and not for entire frames.


In the scene switch, we don´t have previous frame, because the previous freme are very diferent and the temporal filter don´t work.

That is exactly the problem.


But the motion are in several frames, not only in two frames. In scene switch, you can check the two next frames.

Earlier in this thread in response to a contribution by krieger2005 I proposed a similar solution to the scene switch problem. But then I realised that this was a broader problem for many temporal filters and I came up with the "cheating" idea underlying the filter ImproveSceneSwitch. If that filter is used conservatively, then the viewer can hardly detect the frame doubling, which clearly gives also the best compression results. In fact with frame doubling codecs can easier find the right I-frames.


In resume, This metod will make lees artifact the the actual SceneSwitch. Can you test it?
One cannot simply take the same code for -1,0,1 and use it for 0,1,2 or -2,-1,0 and each time I change the algorithm for 0,1,2 I would have to change not only the code for -1,0,-1 but also for 0,1,2. Thus I would not have only substantial work once but more work each time I make changes and I think that this is not worthwhile. Thus I will probably stick with ImproveSceneSwitch.

[SILICON]

Hi Kassandro,

One simplest filter proposal.

In very dark images we can see the details. You can test the lumma value of ane point and, if are below of one threshold, high blurring the lumma and croma whith the near points.

Make this, you can see any visual diference, but the compresor need less bitrate. Less bloks in dark images.

For example:

For (x=1; x<ImageHeihgt-1; x++) ; Not process the border lines
For (y=1; y<ImageHeihgt-1; y++)
If Luma(x,y) < LumaThreshold
; blur the luma
BlurLuma = Luma(x-1,y-1)+Luma(x-1,y)+Luma(x-1,y+1)
BlurLuma = BlurLuma + Luma( x,y-1)+Luma( x,y+1)
BlurLuma = BlurLuma + Luma(x+1,y-1)+Luma(x+1,y)+Luma(x+1,y+1)
Luma(x,y) = (Luma(x,y) + BlurLuma) /2
End
Next
Next


¿can you coded it?

[kassandro]

Originally posted by SILICON
Hi Kassandro,

One simplest filter proposal.

In very dark images we can see the details. You can test the lumma value of ane point and, if are below of one threshold, high blurring the lumma and croma whith the near points.

Make this, you can see any visual diference, but the compresor need less bitrate. Less bloks in dark images.

For example:

For (x=1; x<ImageHeihgt-1; x++) ; Not process the border lines
For (y=1; y<ImageHeihgt-1; y++)
If Luma(x,y) < LumaThreshold
; blur the luma
BlurLuma = Luma(x-1,y-1)+Luma(x-1,y)+Luma(x-1,y+1)
BlurLuma = BlurLuma + Luma( x,y-1)+Luma( x,y+1)
BlurLuma = BlurLuma + Luma(x+1,y-1)+Luma(x+1,y)+Luma(x+1,y+1)
Luma(x,y) = (Luma(x,y) + BlurLuma) /2
End
Next
Next


¿can you coded it?
This has nothing to do with RemoveDirt.
What you really want, is to blur dark areas, because you can't see details anyway in the darkness. However, the above meta code blurs all dark pixels and not only blurs dark areas. That's too much! For instance, if you have a thin black line on a white background only a slight shadows remains if the above filter is applied. You clearly don't want that. Consequently, one has also to check the neighboring pixels, which are used for blurring, whether they are below the threshold. Then such a filter makes sense and can be easily coded. However, it is not so easy to get a fast SSE based version. In addition one would like to blur also the chroma in dark areas and this is more complicated because the chroma is much thinner than the luma. Furthermore, a smart mixture of spatial and temporal rather than purely spatial blurring will give significantly better compression ratios. If more people find such a filter useful, I may think about making such a filter.

[SoonUDie]

You can already simulate such a filter directly in avisynth by masking the dark areas and overlaying a spatio-temporal softened version of the original image onto the unsoftened one. For example:



#--------------------------------------------
#Version NOT using Masktools (slower, YV12 only (because of deen, but you can change it) ):
#--------------------------------------------
function SmoothDarkAreas (clip "aClip")
{
mask = DarkMask(aClip)

#Specify your smoother here:
smooth = aClip.deen("a2d", 4, 6, 8)


return Overlay(aClip, smooth, mask=mask)

}

function DarkMask(clip "c", int "low", int "high", float "gamma")
{
low = default(low, 10)
high = default(high, 20)
gamma = default(gamma, 1)

return c.Sharpen(1).levels(low, gamma, high, 0, 255).invert().blur(0.3)
}


#--------------------------------------------
#Version using Masktools (faster, YV12 only):
#--------------------------------------------

function SmoothDarkAreasMT (clip "aClip")
{
mask = DarkMaskMT(aClip)

#Specify your smoother here:
smooth = aClip.deen("a2d", 4, 6, 8)


return MaskedMerge(aClip, smooth, mask, y=3,u=2,v=2, usemmx=true)

}

function DarkMaskMT (clip "c")
{
#fixed values of 10,20,1

return c.YV12Convolution(
\ "-1, 4, -1", "-1, 4, -1", automatic=true, usemmx=true, Y=3, U=1, V=1)
\ .YV12LUT("x 10 - 20 10 - / 1 1 / ^ 255 0 - * 0 +", "x", "x")
\ .invert().YV12Convolution("1 6 1", "1 6 1", automatic=true, Y=3, U=1, V=1)
}


Calling DarkMask() will give you a clip with spatially blurred shadows.

EDIT: changed to functions, uploaded masktools optimized version.

[SoonUDie]

Another filter: Filter both dark AND bright areas.

Masktools only, at the moment (normal Overlay isn't letting me add the light and dark masks for some reason)


function SmoothLightAndDarkMT (clip "c")
{
cSharp = c.YV12Convolution(
\ "-1, 4, -1", "-1, 4, -1", automatic=true, usemmx=true, Y=3, U=1, V=1)

dark = cSharp.YV12LUT("x 8 - 14 8 - / 1 1 / ^ 255 0 - * 0 +", "x", "x")

light = cSharp.Levels(240, 1, 255, 0, 255)
#can someone translate this to YV12LUT format for me?

LDmask = YV12Layer(Dark, Light, "add", chroma=false, Y=3, u=1, v=1)
\ .invert().YV12Convolution("1 6 1", "1 6 1", automatic=true, Y=3, U=1, V=1)


#--------------------------------------
# Specify your smoother here:
#--------------------------------------
smooth = c.deen("a2d", 4, 6, 8)


return MaskedMerge(c, smooth, LDmask, y=3,u=2,v=2, usemmx=true)

}

[cwolf]

The range file is a problem in VirtualDub-MPEG2. I don't know how to workaround it.
SetMemoryMax (200)
input= avisource("f:\falcao maltes.avi")
removedirt (input, pthreshold=30, grey=true, mthreshold=175, athreshold=75,
\ range1="intro.rmd", mthreshold1=500, athreshold1=500)
#removedirt(input,neighbour=firstpass)
#ImproveSceneSwitch(cleaned,input)
-----------------------------
This script loads the movie into VirtualDub, but when reaches the first frame on "intro.rdm" it fails with "Avisynth read error: Avisynth: caught an access violation at 0x00a5aa1, attempting to read from 0x00000000". If I take the range out it works. I put 10-270 and the error happened when reached frame 10 in "preview input".
I searched DOOM9 and found SETMEMORYMAX in a thread to solve this error, but didn't work. Tried the official VirtualDub, too.
I have WINDOWS XP HOME SP1 and Avisynth 2.5.5.

Do you know what could it be?

Thanks in advance.
Cecilia
P.S. could it be related to DLLs? I had trouble yesterday trying to install DUSTV5 for the first time, it took me hours to find out what was going on (putting plugins in different directories, MSVCP71.DLL, etc. What a pain in the *ss :mad: )

[kassandro]

As you describe it, it sounds very much like a RemoveDirt bug. On the other hand, the following script

SetMemoryMax (200)
input= avisource("Elke.avi")
removedirt (input, pthreshold=30, grey=true, mthreshold=175, athreshold=75,range1="intro.rmd", mthreshold1=500, athreshold1=500)

caused no problems with vdubmod. Of course I had to take a different avi. Is your problem dependent on the avi file?

[killerhis]

Hi guys.. I've been following this filter for a while.. keeps getting beter every time :)

I got a small issue, maybe you guys can give me some advice. I've been trying to get a clean motion detection, even during high motion. I got here a anime source (funanimation dbz) which has the famous white cut edges on the top/bottom during a scenechange. Unfortinally, it has also some fast motion, which Removedirt also sees as a scenechange, generating a strobeeffect motion.

Are there any settings you guys advice me that I can try/use? I know its impossible that I cant get a perfect scene detection, but how better the detection, how happier I am :). The other highmotion parts I could fix using trim :D

With regards


KillerHIS

[cwolf]

It seems that using rangefile in VDUBMOD is ok, but I had to fight with it about the pause/play of preview, I'm used to the space bar in VirtualDub but it doesn't work the same way in VDUBMOD. Another issue is the .VCF file with my editings, if I don't save it before refreshing (PF5) I lose them (the reload discards all the editings). I posted in Everwicked forum about it, don't know if somebody replyed.
Another problem is the huge number of frames with rainbows in my capture (dirty VHS): I think every 2 or 3 in 5 have rainbows. Removedirt does an excellent job in removing them, with the default settings it removes the rainbows in the areas where there's no movement. But the eyes and mouths of the persons keep the rainbow, which gives an awfull look to them (kind of they're going to a halloween party :p ). I don't know whether I increase the settings or make a range file with these hundreds (thousands?) of frames (what a work!). I'll find a balance between them, but I thought it could be improved in Removedirt (giving it more intelligence), as these rainbows are a specific kind of dirt, they do have a pattern. They are always stripes of violet, blue, green or yellow color, although may vary in height. Kassandro, do you know them?

[kassandro]

Originally posted by killerhis
I got a small issue, maybe you guys can give me some advice. I've been trying to get a clean motion detection, even during high motion. I got here a anime source (funanimation dbz) which has the famous white cut edges on the top/bottom during a scenechange.

From an entertainment perspextive I am not interested in anime films. However, I am interested in anime specific problems as far as digital video processing is concerned. It would be nice if you could provide a three frame clip of such a scene change with these "white cut edges on the top/bottom". Instead of using Huffyuv I recommend to use direct stream copy mode in vdubmod with winrar compression afterwards. This creates 10-20% smaller files and avoids any color space conversion. The frame, which RemoveDirt should handle should be in the middle, i.e. the second frame.


Unfortinally, it has also some fast motion, which Removedirt also sees as a scenechange, generating a strobeeffect motion.

As you said scene switch detection can't be perfect for the time being. Even sophisticated codecs, which make a much more thorough motion analysis than ImproveSceneSwitch, have problems to choose the right frames as I-frames. On the other, ImproveSceneSwitch is smarter than simply diagnosing a scene switch, if the difference between two frame exceedes a certain threshold. If the differences of the frames before are similarily large - this is usually the case with fast motion - then it does not diagnose a scene switch. Unfortunately even the differences may oscillate strongly during some kind of motion and then ImproveSceneSwitch falsely detects scene switches. I have two ideas for further improvement without sacrificinf too much speed. Firstly I compute the average of the luma and the two chroma planes. During motion these averages shouldn't change much less than during a scene switch. Secondly, if ImproveSceneSwitch detects one scene switch very shortly after another, it should discard the second scene switch.

[kassandro]

Originally posted by cwolf
Another problem is the huge number of frames with rainbows in my capture (dirty VHS): I think every 2 or 3 in 5 have rainbows.

While I know the rainbow problem from my own VHS past, I can't remember it as so prevalent. I have heard about rainbow filters/scripts in this forum, which you might try. Since about two years I record the digital DVB stream directly. The quality is much higher in any respect and rainbows are quite rare and small. hence I do not care much about it.


Removedirt does an excellent job in removing them, with the default settings it removes the rainbows in the areas where there's no movement. But the eyes and mouths of the persons keep the rainbow, which gives an awfull look to them (kind of they're going to a halloween party :p ). I don't know whether I increase the settings or make a range file with these hundreds (thousands?) of frames (what a work!). I'll find a balance between them, but I thought it could be improved in Removedirt (giving it more intelligence), as these rainbows are a specific kind of dirt, they do have a pattern. They are always stripes of violet, blue, green or yellow color, although may vary in height. Kassandro, do you know them?
I am not very optimistic about solving this problem, but if you provide me with a typical three frame clip (as in the previous posting) I will certainly make a look at it. Rainbows are obviously an RGB problem, while color tv works with YUV (in fact YUV was invented to make color tv backward compatible with b&w tv). That makes rainbows a bit of a mystery for me, because YUV perturbations should not creat RGB typical effects.

[Boulder]

This thread has some ideas for rainbow removal (sh0dan's posts):

http://forum.doom9.org/showthread.php?s=&threadid=62873

There seems to be a small error in the docs supplied with v0.61..it says that for B&W clips "...there cannot be any chroma postprocessing, pthreshold should be somewhat lower, say 30 or 40."

The default is 20 so what should the value be for B&W clips?

Thank you for a great filter, it's a must in my scripts!

[kassandro]

Originally posted by Boulder
There seems to be a small error in the docs supplied with v0.61..it says that for B&W clips "...there cannot be any chroma postprocessing, pthreshold should be somewhat lower, say 30 or 40."

The default is 20 so what should the value be for B&W clips?

Thank you for pointing out this incorrectness. Initially the default value for pthreshold was 50. That lead to a slight blockiness in areas with flat contrast and I lowered the default value to 20. For black&white I am using now 10.

[audioslave]

@kassandro
Seems like a very interesting filter, indeed :) !
Sorry for this newbie question, but what settings would you recommend for a scene like this:

http://hea.port5.com/TCM1.jpg

:confused:

It's from the first minute of "The Texas Chainsaw Massacre". I've tried most settings (I could think of) but I still haven't been able to get the scene right... The scene looks awful encoded as it is now. Almost like Lego! :D

Thanks! ;)

[kassandro]

Originally posted by audioslave
@kassandro
Seems like a very interesting filter, indeed :) !
Sorry for this newbie question, but what settings would you recommend for a scene like this:

picture as above

:confused:

It's from the first minute of "The Texas Chainsaw Massacre". I've tried most settings (I could think of) but I still haven't been able to get the scene right... The scene looks awful encoded as it is now. Almost like Lego! :D

Thanks! ;)

Use RemoveDirt(tolerance=100) to check the maximal amount of cleaning possible with RemoveDirt. Of course, you cannot process an entire clip this way, because of massive artifacts in motion scenes.
To judge a specific frame, I not only need the frame itself, but also the predeccor and the successor frames. Ideally you mark the three frames in vdubmod and save them uncompressed to an avi with Directstreamcopy. If the 3 frame avi is compressed further with winrar it is usually smaller than a Huffyuv avi and it doesn't suffer from a color space conversion to yuy2.

[SILICON]

@Kasandro

You can use a simple motion search for increase your filter performance.

The most common motion are left to right (or right to left).

You can search the blocks as is:

1 - Search the block in the next frame in the same place
2 - If don´t locate in step 1, search the block to left and rigth.
3 - If don´t locate, process the next block in step 1
4 - If locate in step 2, save the motion found
5 - For all blocks in this frame, search only with motion of step 4 and with no motion.

The motion is usally place in several frames. You can begin the search in one fame with the motion of the last frame.

I think that make this search, take a small amount of time.

[kassandro]

Originally posted by SILICON

The most common motion are left to right (or right to left).

I agree.

1 - Search the block in the next frame in the same place
2 - If don´t locate in step 1, search the block to left and rigth.
3 - If don´t locate, process the next block in step 1
4 - If locate in step 2, save the motion found
5 - For all blocks in this frame, search only with motion of step 4 and with no motion.

With this kind of motion compensation, one can only capture global motion to the left or right by 8 pixels (the width of a block) and there is no reason why 8 pixel motion should be more likely than 6 or 3 pixel motion. On the other hand, XviD can do (I haven't looked at it) a motion analysis on a quarter pixel basis. Thus, with a crude 8 pixel motion analysis I can capture only a small fraction of horizontal motion.


I think that make this search, take a small amount of time.
As described the speed penalty is very limited but the possible gain is even more limited. There is another problem: currently RemoveDirt matches blocks in the previous frame with blocks in the subsequent frame. The frame to be cleaned, which is just in the middle, is ignored as far as motion analysis is concerned (the reason is explained in the RemoveDirt documentation). Now if a can match blocks with different spatial location, then these blocks can only be used for cleaning a block, whose spatial location is just the arithmetic mean of the spatial location of the two matching blocks and such a location may very well be outside the current 8 pixel grid causing a whole punch of other problems.

[BBugsBunny]

First of all - kassandro thank you very much for that filter. It can enhance quite a lot some captures of mine from analogue source.

I've been testing the filter a little bit and found a scene where the filter has got some problems - or I simply can't find good settings for it - I've already been playing around with the settings a bit.

In the described scene on the bottom left brightness is going up and down. On some frames dark blocks appear that are visible quite well.

So my question is if there are settings that still remove dirt and don't let these artefacts appear, or is this not possible with this filter?

I've uploaded a small sample clip, which is part of a longer scene that shows the artefacts a few times.

http://members.chello.at/nagiller/test/test.avi

This is basically my script:
AVISource("test.avi")
dein=RemoveDirt(dist=1,mthreshold=150,athreshold=50,pthreshold=20,tolerance=12,mode=2)
ImproveSceneSwitch(dein, dein)

[kassandro]

Originally posted by BBugsBunny
I've uploaded a small sample clip, which is part of a longer scene that shows the artefacts a few times.

What you have uploaded, is this the original clip or is this the output after applying RemoveDirt? I really hope, that this messy clip is not a result of RemoveDirt. In general, RemoveDirt can only clean a spot on a frame, if it is correct on the previous and the subsequent frame and if there is no motion. This implies that the previous and the subsequent frame do not deviate much from each other. In your clip, the frames don't fit together at all. It is obviously interlaced and the changes are so fast that not even the two fields of a frame fit together. This you can see by separating the fields with

avisource("test.avi")
SeparateFields()

I am sorry, but for this clip RemoveDirt can't do anything good, but with your rather standard setting it doesn't cause destruction either. On the other hand, ImproveSceneSwitch may cause problems here, because the clip is so discontinuous such that ImproveSceneSwitch may regard any frame of this clip as a scene switch.

[BBugsBunny]

Thanks kassandro for having a look at my file.
Actually the source file was a DV file that I captured from TV with my Hollywood DV Bridge.
I then deinterlaced it with TomsMoComp (0,5,1) and stored it as a Huffyuv 2.2.0 avi. That's what I actually uploaded.
Now I've uploaded/replaced it with the original DV file. (In case anyone still wants to give it a try)

The problem as well could be, that the original is a japanese anime that was converted to pal, so even with deinterlacing some artefacts stay. I've tried inverse telecine and restore 24 fps (that got me an error in avisynth) but I really don't know how and if the original frames can be restored from this pal capture.
If that's be possible perhaps I would get a better result since you say that the source has to be 100% progressive...

But since I do not really know where to start with that problem I think I'll stay with my wavelet/temporal/2d cleaning in VirtualDub although the results are not as good as with your filter in general, but on the other hand the dark blocky artefacts don't appear either.

[kassandro]

Hi BBugsBunny,
I have now downloaded the bigger original clip. Unfortunately
it uses codec with fourcc "dvsd", which I do not know, whence
it is rejected by avisource. It is also rejected by vdubmod if
I load your avi directly. On the other hand, mpc plays the clip
just fine. It now looks a lot better. I can't believe that
tomsmocomp, which is a very fine filter, can creat such a garbage
from it, which I have seen in your first clip. However, to analyse
the clip I have to load it with avisource. So please tell me, which
kind of codec I need to accomplish this. I thought that only huffyuv
and mjpeg (bad for non-progressive material) is used for analogous
capturing and I have installed both with ffdshow.

[Leak]

Originally posted by kassandro
However, to analyse the clip I have to load it with avisource. So please tell me, which kind of codec I need to accomplish this. I thought that only huffyuv and mjpeg (bad for non-progressive material) is used for analogous capturing and I have installed both with ffdshow.

Actually, you should be able to use any recent version of ffdshow to open it via AVISource; just start ffdshow's "VFW configuration" from it's start menu folder and set "DV" to libavcodec on the Codec page in the decoder tab.

np: Autechre - IV VV IV VV VIII (Draft 7.30)

[kassandro]

Originally posted by Leak
Actually, you should be able to use any recent version of ffdshow to open it via AVISource; just start ffdshow's "VFW configuration" from it's start menu folder and set "DV" to libavcodec on the Codec page in the decoder tab.

I have done this, but no success!

[Leak]

Originally posted by kassandro
I have done this, but no success!

Strange - it works for me, now that I've downloaded all of the clip.

Have you tried configuring ffdshow's VFW codec using VirtualDub's compression config dialog? Is the codec ("ffdshow Video Codec") even listed there?

If it isn't, try reinstalling ffdshow to a path that contains no spaces, that should fix it...

np: Autechre - Lowride (Incunabula)

[Leak]

Originally posted by BBugsBunny
In the described scene on the bottom left brightness is going up and down. On some frames dark blocks appear that are visible quite well.

Ummm... exactly where are the dark blocks supposed to appear? Decoding the file with ffdshow I can't seem to find any...

If you want I can extract the same scene from my US-DVDs of Hellsing for comparison - just not now; it's waaay too late already... :)

np: Lamb - Trans Fatty Acid (Lamb)

[kassandro]

Originally posted by Leak
Strange - it works for me, now that I've downloaded all of the clip.

Have you tried configuring ffdshow's VFW codec using VirtualDub's compression config dialog? Is the codec ("ffdshow Video Codec") even listed there?

No it isn't! But as ffdshow can't encode, it doesn't make sense to list ffdshow in the compression dialog. Actually in my case ffdshow was installed by the Matroska all in one package (and is now in a Matroska subdir). While all directshow apps find the codec, vfw apps like vdub, avisource or avicodec don't. Instead of messing around with a new installation of ffdshow, I now load the avi with DirectShowSource and it now works.
Now back to BBugsBunny's clip. As BBugsBunny already said, the light eminating from the bottom left corner is the critical part of the clip. It changes with every field rather than with every frame. Hence a deinterlacer is necessary. However, Tomsmocomp(0,5,1) does an excellent job here and like Leak I can't find any of this black blocks created by RemoveDirt (you may also lower pthreshold slightly).
Obviously something else must have gone wrong the clip, which I downloaded first, because it was really pure garbage. RemoveDirt removes quite a bit of noise of this clip, even after deinterlacing. What may be irritating though, is, that in the bottom left corner sometimes the noise is cleaned and sometimes not depending how fast the light goes up and down.

[Leak]

Originally posted by kassandro
No it isn't! But as ffdshow can't encode, it doesn't make sense to list ffdshow in the compression dialog.

Oh, ffdshow *definitely* can encode, as it's been merged with ffvfw some months back.

Of course, that would mean you have to install a more recent version of it (http://athos.leffe.dnsalias.com/) - the latest version of ffdshow is 4 days old. :)

np: Lamb - Just Is (What Sound)

[FredThompson]

Originally posted by kassandro
As BBugsBunny already said, the light eminating from the bottom left corner is the critical part of the clip. It changes with every field rather than with every frame. Hence a deinterlacer is necessary.Deinterlace or decomb? Deinterlace will give you trash. If you must operate on individual fields, split into 2 streams, filter each, then recombine. Alternately, you could make a 2x frame rate version and clean it.

[kassandro]

Thank you, Leak for your informations and the link. It is good to know that ffdshow is continued. My version is obvious the most recent from the sourceforge web site, which is quite old.

Originally posted by FredThompson
Deinterlace or decomb? Deinterlace will give you trash. If you must operate on individual fields, split into 2 streams, filter each, then recombine. Alternately, you could make a 2x frame rate version and clean it.
I think the output looks quite good and wouldn't call it trash. If you only want to clean, then at your advice to separate the fields and clean them separately is optimal. However, there is the slight danger, that a piece of dirt is cleaned only on one field. But if you deinterlace afterwards - and BBugsBunny's clip should be deinterlaced - then this problem is taken care of as well. Thus the best strategy for denoising interlaced material should be : separate fields, denoise them separately, weave them and finally deinterlace them. I will add this to the documentation. Thank you, Fred.

[FredThompson]

My reply was too short. I should have explained better.

If the "true" source has a different frame rate than the interlaced version, filtering will probably yield better results by decombing to recover those original progressive frames.

Deinterlacing doesn't always do this properly. It can result in some strange artifacts.

If the "true" source is interlaced, splitting and filtering would be the "best" idea because it's really 2 different streams of images which are interleaved.

Yes, you are correct. Splitting and filtering could possibly result in some dirt being missed by the filter. Maybe that seems worse than the actual results because this should only happen in single frames, right?

[kassandro]

Originally posted by FredThompson
If the "true" source has a different frame rate than the interlaced version, filtering will probably yield better results by decombing to recover those original progressive frames.

Deinterlacing doesn't always do this properly. It can result in some strange artifacts.


I agree with you. If there is telecining or even field blending going on, especially good deinterlacers like tomsmocomp can creat strange artifacts. In this special clip, however, there is so little motion, the man with the child is walking very slowly, hence the blinking light is the only motion, that you really can't tell, what's going on. On the other hand, for anime videos true interlacing doesn't really make sense, but I am not an anime expert.



Yes, you are correct. Splitting and filtering could possibly result in some dirt being missed by the filter. Maybe that seems worse than the actual results because this should only happen in single frames, right?

I think, that dirt detection should be better with split filtering, if the input clip is not truely progressive, but it is negative for compression and looks bad, if a dirty spot is only removed on one fields (i.e. you end up with a combed stain).

[krieger2005]

Hi,

i use your filter often but one thing is very bad:
Your filter works fine when you look on one single pic. It is clean.

But in motion it produce a "mask"-Like effect. I had the same effect when i'm use "Blockbuster" with method="dither". It is like an overlay obeout some frames.
This is the result of the definition of threshold (and this is the user's fault), BUT: If a movie have something like stones in the picture, where the color and luma is quite identical and the camera move, than you need a very low threshold. But with a low threshold one does not get the result, which the filter should bring.

I know that there is a tradeoff: Do you want to remove the noise and dirt from the picture (and maybe get a "washed" picture) or you let the noise.

Another bad effect is, if there is an area, quite same color, moving camera. Becuase your filter operate on Blocks (8x8 i think) one block were cleaned (let's say) 5 frames and another 4 frames. But because the color is only quite the same one overstep the threshold in frames 150-155 and another 153-157. In this case the picture get blocky.

I know... This is tuning of the threshold... But as every know, when you tune some thresholds for a scene you get bad results on another and so on.

I think there are only two solutions how to solve such problems:
- A Buildin Anti-Wash/Anti-Blocky filter: If the source is not blocky between two blocks the result should it be too
- Cleaning with motion compensation

It's a hard subject matter and i don't know how to solve it at all. At the moment i use your filter with use of masks (motion-masks).

[kassandro]

Originally posted by krieger2005
i use your filter often but one thing is very bad:
Your filter works fine when you look on one single pic. It is clean.

But in motion it produce a "mask"-Like effect. I had the same effect when i'm use "Blockbuster" with method="dither". It is like an overlay obeout some frames.
This is the result of the definition of threshold (and this is the user's fault), BUT: If a movie have something like stones in the picture, where the color and luma is quite identical and the camera move, than you need a very low threshold. But with a low threshold one does not get the result, which the filter should bring.

If mthreshold and athreshold are too high, then even single frames should look bad. The question is: can something go wrong with the video if single frames look fine. Sometimes, I think so, but then motion often looks unnatural with 25 fps and theoretically it doesn't make much sense, because RemoveDirt either leaves a pixel unchanged or uses the same pixel from the previous or the subsequent frame.


I know that there is a tradeoff: Do you want to remove the noise and dirt from the picture (and maybe get a "washed" picture) or you let the noise.

As RemoveDirt doesn't blur, "washed" is probably not the right work. If it would only be looked washed, then one could choose more aggressive thresholds. The artifacts can be significantly more terrible. For instance, try tolerance=100 in the presence of moving fine objects.


Another bad effect is, if there is an area, quite same color, moving camera. Becuase your filter operate on Blocks (8x8 i think) one block were cleaned (let's say) 5 frames and another 4 frames. But because the color is only quite the same one overstep the threshold in frames 150-155 and another 153-157. In this case the picture get blocky.

blockiness should not occure, if pthreshold is sufficiently small. I gradually reduced pthreshold down to 10 and I can't see any blockiness left. Lowering pthreshold has not such a negative effect as lowering mthreshold (I usually set athreshold=300 in order to ignore the athreshold variable). In fact the basic philosophy of RemoveDirt is, that only one block of a moving area has to be caught and the rest is taken care by postprocessing, which strongly depends on pthreshold. So you can have a lot of cleaning with pthreshold=10.


- A Buildin Anti-Wash/Anti-Blocky filter: If the source is not blocky between two blocks the result should it be too

I think that I have exactly done this, but I only look for oscillation between cleaned and uncleaned blocks. Thus I do not look for oscillation between cleaned blocks and it would not be correct to do so. For instance, if you have a white spot overlapping two blocks, then if both these blocks are cleaned the oscillation would probably increase substantially, but I should certainly not reject the cleaning. Only the oscillation between cleaned and uncleaned blocks is critical, because uncleaned blocks indicate motion and if the oscillation between and uncleaned and a cleaned block is higher than the oscillation between the original blocks, the guess is that the motion extends to the cleaned block and the cleaning of this block will be undone by postprocessing. Now the formerly cleaned block becomes an uncleaned block and its cleaned neighbours are again checked, whether cleaning has raised oscillation. That's how RemoveDirt works.


- Cleaning with motion compensation

If motion compensation would only be nearly as good as some claim it to be! As far as I know motion prediction only works well for codecs and why does it work well for codecs? A codec proceeds roughly as follows. For a block in the current frame it makes a motion prediction from the previous frame. It looks at the difference between the original block and the predicted block, does a DCT and drops the high frequency part(depending on the quantiser) from the difference of the two blocks. If the motion prediction was good enough, then very little low frequency stuff is left and compression is very effective. If the motion prediction was bad, then substantial low frequency stuff is left and more bytes are needed for compression, but a bad prediction has no direct effect on visual quality. The situation is very different, if you use motion prediction for other tasks. The most difficult task is frame interpolation, i.e. you want to creat a new frame between two old frames without blending. That blending is still so heavily used, shows that frame interpolation doesn't work well despite some published claims by Tourapis and collaborators. Now for RemoveDirt the situation is more hopeful: One looks for motion vectors between the previous and the subsequent frame. Cleaning will only be done if the prediction is sufficently good (in the case of frame interpolation you always must do something, even if motion prediction is very bad) and postprocessing will also greeatly help to remove artifacts created by false motion prediction. If cleaned blocks do not fit to uncleaned blocks, cleaning must be undone (you can't do postprocessing for frame interpolation either). I am thinking about motion compensation for RemoveDirt, but RemoveDirt will slow down a lot if this feature is implemented and activated.

[mkanel]

Thanks kassandro for the great filter, it works really well for my captured analog tapes. Also thanks for the great documentation, it really helps when experimenting with your filter.

In your documentation you ask for feedback on extrapolation in ImproveSceneSwitch. I tried Extrapolate=true and for what it's worth I get much better results with the default value of False the extrapolated frame appeared oversharp, it was much more noticeably wrong than a duplicate frame.

The default cleaning mode 2 also works best for me, it provides for a much smoother playback than modes 0 or 1.

When I capture from an analog source I get a number of duplicate frames. By design removedirt does no cleaning on these frames. Would there be any way around this that wouldn't cost too much processing time? I have two ideas, I don't know if either is practical.

1) If remove dirt itself could recognize frame duplication it could compare the duplicates to the frames before and after the duplication and perform cleaning on the duplicates based on those frames.

2) Maybe it would be easier for ImproveSceneSwitch to detect duplicates while it searches for scene changes and then perform a similar substitution of frames so that duplicate1 would be replaced by duplicate1-1 and duplicate2 would be replaced by duplicate2+1.

3) Maybe I just need to keep tweaking to try to banish dropped/added frames from my captures.

Thanks. Mike.

[kassandro]

Thanks for your friendly comment and your interesting suggestions.

Originally posted by mkanel

I tried Extrapolate=true and for what it's worth I get much better results with the default value of False the extrapolated frame appeared oversharp, it was much more noticeably wrong than a duplicate frame.

I came to the same conclusion and don't use it anymore. In fact, in the 0.6.1 there was the possibility to compile a filter which did extrapolate any frame except the first two. Using this filter I then watched clips in vdub and quite a few frames were truely bad. It was an interesting SSE excercise, which, however, turned out to be not useful. The extrapolate version will be phased out, starting with version 0.7.1. First the variable the variable will simply be ignored and later it will be completely removed, just like the doscillation and soscillation variables in RemoveDirt.


The default cleaning mode 2 also works best for me, it provides for a much smoother playback than modes 0 or 1.

Again I agree with you. The mode variable will be phased out and mode=2 will be the sole mode.


When I capture from an analog source I get a number of duplicate frames. By design removedirt does no cleaning on these frames. Would there be any way around this that wouldn't cost too much processing time? I have two ideas, I don't know if either is practical.

1) If remove dirt itself could recognize frame duplication it could compare the duplicates to the frames before and after the duplication and perform cleaning on the duplicates based on those frames.

2) Maybe it would be easier for ImproveSceneSwitch to detect duplicates while it searches for scene changes and then perform a similar substitution of frames so that duplicate1 would be replaced by duplicate1-1 and duplicate2 would be replaced by duplicate2+1.


You have a very good feeling, how things should be implemented in a most efficient way. Indeed the low level frame comparison routine for detecting scene switches is exactly the same as the one for detecting duplicates and I spend some time to avoid that these costly comparisons are not repeated unnecessarily and of course it would be more than smart, to use the output of this routine not only for scene switch detection but also for duplicates. There is, however, a severe problem to do it this way, because in the standard avisynth design a filter can only control the input or the output of another filter, but not both. Currently ImproveSceneSwitch controls the output of RemoveDirt: at a scene switch the frames are duplicated rather than passed through to RemoveDirt. On the other hand, the duplicate problem arises primarily on the input level of RemoveDirt: RemoveDirt should simply not receive duplicates for cleaning. Of course later on duplicated input frames must be duplicated again after cleaning. One needs a filter which completely controls another filter. I think such are already done with conditional filtering.
The same duplicate frame problems arises in deinterlacing and it is even more severe. Deinterlacers which adhere to my "no motion no change" simply leave such frames unchanged, which is of course very bad.
Currently, I am very busy with version 0.7, which will see major changes in many ways. It should be released by the end of the onth or eary July and then I will have more time for such a much more sophisticated ImproveSceneSwitch. Not only analog source, but also dv cam source and anime have a duplicate frame problem. Thus it is really worthwhile to deal with it in such a optimal way, if it is really possible.

[mkanel]

kassandro,

Thanks for bringing up conditional filtering. It was a good reason to try using this feature of Avisynth. I made a working script that takes care of cleaning duplicate frames. It has a couple of restrictions.

It won't work if the duplicate frames are at a scene change.

It doesn't work if you have three or more identical scenes, although you could tweak this script to handle that.

This is meant to find and clean truly identical scenes, that kind that a capture program might insert into an avi. The only time the non-cleaned duplicates stand out is during low movement scenes, this will handle those situations.



#Clean Duplicate Frames

#YV12 needed for frame comparison, add one dup to test, two dups results in no cleaning
dirty=AVISource("D:\captured.avi").ConvertToYv12 #.duplicateframe(50) #.duplicateframe(50)
cleaned=removedirt(dirty) #make sure setting match removedirt below

CleanDup= dirty.ScriptClip("dirty.DeleteFrame(current_frame).removedirt")
CopyCleanDup= dirty.SCriptClip("CleanDup.DuplicateFrame(current_frame-1)")

#If previous/next frames not duplicates uses cleaned frame, otherwise frame is passed to CopyCleanDup
#I don't know what YDifference is but it's always .000003 on my duplicate frames. Try true
IsSecondDup=ConditionalFilter(dirty,CopyCleanDup,cleaned,"YDifferencefromprevious()","<",".000005" ,false)

#following checks if next frame is a duplicate, if yes first dup is cleaned, otherwise
#IsSecondDup (line above) checks if the previous frame is a duplicate. Try true to see more info
ConditionalFilter(dirty,CleanDup,IsSecondDup,"YDifferencetonext()","<",".000005",false)

#Uncomment following line from Avisynth documentation to see YDifference info.
#ScriptClip(dirty, "Subtitle(String(YDifferenceFromPrevious))")
converttoyuy2


Obviously you have to put in the proper path and file name for your avi. RemoveDirt appears twice in the script, make sure you put your desired parameters in both places.
You can uncomment the "#.duplicateframe(50)" to test it. Or uncomment the line near the bottom ScriptClip(dirty, "Subtitle(String(YDifferenceFromPrevious))") to make sure the test conditions "<.000005" are right for your avi.

If this is useful to anyone maybe someone could design a cleaner .avsi

Mike.

[kassandro]

Yes, your script does the job. I am surprised that your script is reasonably fast, because ScriptClip must generate a clip for each frame (of course this is only done for duplicate frames). Here is a script based on the same idea, which uses ScriptClip only once

dirty=mpeg2source("C:\video\input.d2v")
normal_clean=RemoveDirt(dirty)
ScriptClip(dirty, "DeleteFrame(current_frame).RemoveDirt()")
ConditionalFilter(dirty,last,normal_clean,"YDifferencetonext()","<",".000005",false)
#take the previous frame of last clip, if the current frame of the dirty clip is a duplicate of the previous one
ConditionalFilter(dirty,last.DuplicateFrame(0),last,"YDifferencefromprevious()","<",".000005" ,false)


It would be relatively easy to extend ImproveSceneSwitch such that it takes care of duplicate frames, if I would add a third clip argument, to which the user has to supply the clip
ScriptClip(dirty, "DeleteFrame(current_frame).RemoveDirt()")
Multiple difference computations could then be avoided (in the above script, the frame difference is computed twice unless Avisynth is extremely smart). However, I would like to avoid ScriptClip, which will probably get costly if there are many duplicate frames.

ScriptClip is really an awfully inefficient "filter". Each time a frame is called from a ScriptClip clip it has to creat and destroy the filters in the ScriptClip script and I haven't optimised at all RemoveDirt for fast initialisation. If RemoveDirt would have a memory leak, the sytem memory would quickly be exhausted and the sript would crash. Thus the above two scripts are a good robustness test for RemoveDirt, but the problem should really be solved without ScriptClip, which should only be used in an emergency.

[mkanel]

kassandro,

Thanks for the cleaner script. It is faster than the one I was using. If I understand correctly the only way to avoid using ScriptClip is to have the frame duplication detection directly inside RemoveDirt, which would probably lead to ImproveSceneSwitch moving inside RD as well which I think is not what you had in mind for the use of these filters. I don't know how many RD users care about frame duplication cleaning. If you implement a duplicate frame cleaning option in ISS or RD itself I'll gladly use it. If not I'm happy using the script you posted.

Thanks. Mike.

[kassandro]

Mike,
Duplicate frames are a problem for most temporal filters. Thus one should not go for a special solution inside RemoveDirt. I think that this can be done efficiently, but at the moment I have no time to get concret with it. I am currently finishing version 0.7 of RemoveDirt. There will be a lot of changes with respect to version 0.6.1, especially for the yuy2 color space. There will now also be chroma post processing for this color space, which is used for analog capture. Consequently, blockiness should then be gone also for this color space.

[cwolf]

I'm using three range files and would like to include a fourth file, but avisynth says "open failure: script error: the named argument "pthreshold4" was passed more than once to removedirt".
Here's my script:

mpeg2source("f:\dali\dali.d2v").killaudio().freezeframe(2106,2106,2105)
assumetff()
separatefields()
#showframenumber(scroll=false)
f3of3 = selectevery(3,2)
selectevery(3,0,1).weave().assumeframebased().assumetff()
f12of3=last
cleanneig=removedirt(f12of3,pthreshold =75, tolerance =18,athreshold =80, mthreshold =155,
\ range1="f:\dali\rddirty.rmd",pthreshold1=200, tolerance1=45,athreshold1=260,mthreshold1=280,
\ range2="f:\dali\rddontclean.rmd", pthreshold2=25, tolerance2=12,athreshold2=40, mthreshold2=48, cthreshold2=22,
\ range3="f:\dali\rdlessmove.rmd",pthreshold3=160,tolerance3=33,athreshold3=155,mthreshold3=177,
\ range4="f:\dali\rddontclean2.rmd", pthreshold4=25, tolerance4=12,athreshold4=40, mthreshold4=100)

removedirt(f12of3,neighbour=cleanneig, pthreshold=75, tolerance =18,athreshold =80, mthreshold =155,
\ range1="f:\dali\rddirty.rmd",pthreshold1=200, tolerance1=45,athreshold1=260,mthreshold1=280,
\ range2="f:\dali\rddontclean.rmd", pthreshold2=25, tolerance2=12,athreshold2=40, mthreshold2=48, cthreshold2=22,
\ range3="f:\dali\rdlessmove.rmd",pthreshold3=160,tolerance3=33,athreshold3=155,mthreshold3=177,
\ range4="f:\dali\rddontclean2.rmd", pthreshold4=25, tolerance4=12,athreshold4=40, mthreshold4=100)
clean=last
conditionalfilter(clean,clean.swapfields(),clean,"current_frame%2","=","1")
separatefields(last)
interleave(selecteven(),selectodd(),f3of3)
weave()
assumeframebased()
conditionalfilter(last,last.swapfields(),last,"current_frame%3","=","2")


It looks odd but, believe me, this DVD I'm cleaning, which is interlaced, has dirty spread through 3 fields, that is, 2 frames!
Alignfields was spreading the spot/scratch/etc to the 4th field thinking it was motion, so I have to clean before deinterlacing. After some trials & errors, I decided to make a clip with the fields 0 1 3 4 6 7 9 10..., clean it, join again the third field, and then clean this (partially cleaned) clip (apply removedirt again). That's a lot of work but I think it's worthwhile (it's a pity there's no smilie drying the sweat of the face).
I made a sample of 7 frames for you to see:
dali scratch 7frames.avi (382Kb) (http://www.ceciliawolf.pop.com.br/dali scratch 7frames.avi) (do complementparity before separatefields)

Another thing: I tried to use the "removedirt.ini" but it didn't work, saying Avisynth open failure: Removedirt: IO error with f:\dali\rddirty.rmd" and listing the lines after that range file.
My file contains:
dali
athreshold1 = 100
pthreshold1 = 100
mthreshold1 = 500
tolerance1 = 20
range1= "f:\dali\rddirty.rmd"
pthreshold2 = 100
athreshold2 = 100
mthreshold2 = 160
tolerance2 = 40
range2 = "f:\dali\rddontclean.rmd"
pthreshold3 = 5000
athreshold3 = 5000
mthreshold3 = 5000
tolerance3 = 90
range3 = "f:\dali\rdlessmove.rmd"
Nevermind the values, I know they don't match the script.
I'm using VirtualDub-MPEG2.
Thanks in advance.
Cecília
P.S.: link to Amazon about this DVD. (http://www.amazon.com/exec/obidos/tg/detail/-/6305701288/102-7449946-6426503?v=glance) I agree with the last comment.

[kassandro]

cwolf,
I think your script seems to be correct. "open failure: script error: the named argument "pthreshold4" was passed more than once to removedirt" is a message from Avisynth not from RemoveDirt. Avisynth issues this error message, if you use a named option twice, which obviously is not the case here. We had this problem already once in this thread and it has not yet been tracked down. RemoveDirt has a lot of named option and these named options are replicated 10 times to handle also up to 9 range files. Because of its enormous size the option string is not a constant string at compile time. Rather it is created at run time, when the dll is initialised. Either Avisynth has problems with such a big option string or there is something wrong with RemoveDirt.
"Removedirt: IO error with f:\dali\rddirty.rmd" is an error message from RemoveDirt (all its error messages begin with "Removedirt:". Either it could not open the file for reading or it could not read the entire file in one stroke. How big is "f:\dali\rddirty.rmd"?
I will try to reproduce the problems using your script. Of course some adjustments have to be done.

[cwolf]

I made a simplified removedirt.ini with just one rangefile with one range in it, didn't work (same removedirt error).
I'm using "removedirt.dll" 04/18/2004 21:59 34kb. I remember I had two dll's (I had also removedirt2.dll), and I removed the bigger in April or May.

[kassandro]

Originally posted by cwolf
I made a simplified removedirt.ini with just one rangefile with one range in it, didn't work (same removedirt error).
I'm using "removedirt.dll" 04/18/2004 21:59 34kb. I remember I had two dll's (I had also removedirt2.dll), and I removed the bigger in April or May.
It would be nice if you could simplify the context, in which RemoveDirt fails, even further. Also it is very important to know, if the problem is independent of the clip. If I can reproduce the problem, it should be easy to resolve it.

[kassandro]

@cwolf
I just reproduced an corrected the RemoveDirt.ini bug.

Turning to the range4 bug. It can be simplified a lot: RemoveDirt(pthreshold4=25) always fails.

Investigating this problem, I found that Avisynth allows only a maximum of 60 variables for a filter. pthreshold4 seems to be number 61. Thus this is a limitation of Avisynth - not a bug of RemoveDirt. You can still use these variables by packing them into the RemoveDirt.ini file. I will upload a preliminary binary of version 0.7, for which the RemoveDirt,ini bug is fixed, probably tomorrow.

[Serg Belyansky]

kassandro, thanks a lot for such a wonderful filter.

While I generally satisfied with its result I am still scratching my head with question: is it possible to change the amount of "cleaning" with more precision than just "process/ don't process this block"?

I have a source with some amount of negative dirt (white marks, hair and streaks). Using RemoveDirt, I am able to eliminate most of this dirt. But while removing the dirt, it smoothes out all film grain, which is in my case undesirable.

Please tell me if I mistaken and RemoveDirt is not suitable for "dirt only" operation without any "temporal smoothing" effect. For now I have to use Random Noise Remover, which requires a lot of manual post-processing and range applying (its motion blocks exclusion is very crude) and is Virtualdub-only.

[cwolf]

I made some tests with a very simplified script reading a small avi and two lines of removedirt with removedirt.ini, without success both in VDUBMOD and Virtualdub-MPEG2 (removedirt I/O error), switching the removedirt.dll and removedirts.dll in the "pluging" folder and putting the rangefile in the script's folder, the avisynth's folder and root of f:, removing the path of the rangefile [rangefile1="rddirty.rmd"]. I'll try to run avisynth in my working PC at my job (Pentium II) and do the same tests.

[kassandro]

I have just uploaded new binaries to the web site. This is an unofficial release though. It should fix RemoveDirt.ini bug detected by cwolf. Actually this essentially version 0.7. Unfortunately the documentation has not yet been updated. There quite a few new features, some old features have been removed or changed (in particular the show mode is very different). On the other, in most cases it can be used as version 0.6.

Originally posted by cwolf
I made some tests with a very simplified script reading a small avi and two lines of removedirt with removedirt.ini, without success both in VDUBMOD and Virtualdub-MPEG2 (removedirt I/O error), switching the removedirt.dll and removedirts.dll in the "pluging" folder and putting the rangefile in the script's folder, the avisynth's folder and root of f:, removing the path of the rangefile [rangefile1="rddirty.rmd"]. I'll try to run avisynth in my working PC at my job (Pentium II) and do the same tests.

Thanks. The RemoveDirt.ini bug has been fixed. Unfortunately, the pthreshold4 problem is not RemoveDirt bug. Rather it is an Avisynth limitation which allows only a maximum of 60 variables and RemoveDirt has about 140. I will post a request to the AVS 2.5.5 thread for a higher limit. Thus pthreshold4 and all the variables for rangeX with X >4 can only be set through RemoveDirt.ini currently.


Originally posted by Serg Belyansky

While I generally satisfied with its result I am still scratching my head with question: is it possible to change the amount of "cleaning" with more precision than just "process/ don't process this block"?

One can do the following: instead of not cleaning, clean as usual but limit the amount of change. The temporal component of PixieDust seems to persue this strategy and despite making only quite small changes (but very many), PixieDust achieves fairly good compression gains. That let me to write the filter ReduceFluctuation (include in the new RemoveDirt plugin), which uses the same cleaning technique as RemoveDirt, but instead of sophisticated artifact protection, it simply limits the amount of change. For instance ReduceFluctuations(limit=5) cleans everywhere, but the change is limited by 5. This is a purely temporal filter. Thus it can be used for any kind of video. As far as compression gain is concerned, ReduceFluctuations(limit=5) is inferior to PixieDust(5), but in combination with a good spatial denoiser (PixieDust has a spatial component as well) it can beat PixieDust, which for me despite its undisputed quality is hardly usable because of its unacceptable speed.


I have a source with some amount of negative dirt (white marks, hair and streaks). Using RemoveDirt, I am able to eliminate most of this dirt. But while removing the dirt, it smoothes out all film grain, which is in my case undesirable.

Please tell me if I mistaken and RemoveDirt is not suitable for "dirt only" operation without any "temporal smoothing" effect.

While RemoveDirt was intended primarily for real dirt and scratches and eliminates temporal noise as well and film grain is temporal (actual it is the result of a chemical process, which behaves randomly on each pic). On the other hand small temporal noise forces the user to use higher values for mthreshold and that implies a higher artifact risk. Moreover, tiny even invisable noise, can be taken care quite well by spatial denoisers. That's why I created RemoveGrain and it can be used as a precleaner for RemoveDirt to lower mthreshold.

[Fizick]

Serg Belyansky,
I think, that RemoveDirt is not suitable for "dirt only" operation without any "temporal smoothing" effect.
(Sorry, Kassandro) ;)
In place of Random Noise Remover, which requires a lot of manual post-processing and range applying (its motion blocks exclusion is very crude) and is Virtualdub-only
, you may try DeSpot plugin for Avisynth for a source with some amount of negative dirt (white marks, hair and streaks).
But it is almost offtopic, please go to DeSpot thread if you want.

Removedirt may be apllyed as post-processing after DeSpot
(Sorry, Kassandro once more) :)

BTW, I wonder, how you can be register in 2002 and have 1 post?

[Mug Funky]

hmm. removedirt seems to be behaving oddly under the new avisynth. i haven't tracked the problem down completely, but i get "IO error" on loading scripts that call it. by commenting out various other plugins it seems to work, but obviously that's not a good solution (and it seems to be a rather large selection of plugins that don't work with removedirt)

the message after IO error is different every time (and can occasionally list every function in avisynth, or sometimes just those in masktools, but usually it's gibberish of some kind.)

[Boulder]

I've not had such problems..but I have had "RemoveDirt doesn't have an argument mthreshold" (or similar) a couple of time when I've set mthreshold in my script. This doesn't happen nearly every time which makes it hard to reproduce. Once I had it by setting the two passes of an XviD encode, then postponing the second one (I wanted to run it the next day). I ran the first pass and then decided to run the second one as well. This gave me the error. After a reboot, the second pass ran normally.