RemoveDirt

[kassandro]

Originally posted by Mug Funky
hmm. removedirt seems to be behaving oddly under the new avisynth. i haven't tracked the problem down completely, but i get "IO error" on loading scripts that call it. by commenting out various other plugins it seems to work, but obviously that's not a good solution (and it seems to be a rather large selection of plugins that don't work with removedirt)

the message after IO error is different every time (and can occasionally list every function in avisynth, or sometimes just those in masktools, but usually it's gibberish of some kind.)

cwolf has found an important bug, which caused "IO error" messages. However, this bug was not a conflict with other plugins. Rather it was a terrible programming mistake in the RemoveDirt.ini part. This bug has been fixed and the corrected binary is now contained in the binary archive of the web site. cwolf found another bug, which however was not a bug of RemoveDirt. Rather it was a limitation of Avisynth 2.54, which only allows at most about 60 variables and if you set the 61st variable Avisynth produces idiotic error messages.
Mug Funky, have you tried the new binary. If the bug persists, could you please post a script as simple as possible with which I can reproduce the problem.

[Mug Funky]

many thanks, kassandro :)

btw, is there a good quick way to avoid a sheared image in cases of earthquake motion? this is the only thing preventing me from using removedirt in all my scripts :)

i suppose there's applyrange, but it seems to me that in some situations one doesn't have time to fully watch the material one is encoding.

by "earthquake motion" i mean the frame jumping up and down in such a way that removedirt removes half a good frame, so you get a torn-up image and a bizarre mix of moving and non-moving parts.

[scharfis_brain]

@mugfunky: maybe an idea

- stabilize the video before feeding it to removedirt. use depanstabilize
- apply removedirt
- turn the video by 180 degrees
- apply depanstabilize again, to restore the earthquake
- turn the video by 180 degrees

(I restored some old films using that technique exept the 3 last steps)

[kassandro]

Originally posted by Mug Funky
btw, is there a good quick way to avoid a sheared image in cases of earthquake motion? this is the only thing preventing me from using removedirt in all my scripts :)

i suppose there's applyrange, but it seems to me that in some situations one doesn't have time to fully watch the material one is encoding.

by "earthquake motion" i mean the frame jumping up and down in such a way that removedirt removes half a good frame, so you get a torn-up image and a bizarre mix of moving and non-moving parts.

The motion complaints about RemoveDirt are not new and if I look at videos processed by RemoveDirt, I think to myself that there must be something wrong. On the other hand, RemoveDirt has a three way technique to prevent it from mistreating motion. If n is the number of the current frame, then RemoveDirt measures the SAD distance between blocks in frame n-1 and n+1. If the distance exceeds a threshold, the block is registered as a motion block. Of course, on a small block the difference on frames n-1 and n+1 may be small, while the difference on frames n-1,n and n,n+1 may both be large. This happens if there is dirt on the block in frame n, but also if there is a fluctuative motion. Now the first idea is that this motion should show up also between frames n-1, n+1, if not on the block itself, then on some neighbour blocks. The "dist" variable is exactly for this purpose. If motion between n-1, n+1 is found on some neighbour blocks (the distance depends on the value of dist), then the block is also prohibited from cleaning. Finally, if there are still some motion blocks, which are not yet caught, the cleaned block probably won't fit to its untouched neighbours. The postprocessing routine checks whether a cleaned block fits to its uncleaned neighbours. If it is doesn't fit, it is restored and becomes itself an uncleaned block, which then is used on the next postprocessing loop for checking, whether cleaned blocks fit to their uncleaned neighbours. The postprocessing starts a new loop as long as it has restored at least one block on the previous loop. Usually 2-3 loops are enough, but I have seen frames with 20 or more loops. The advantage of this kind of postprocessing is twofolds. Firstly it makes block artifacts impossible (at least if pthreshold is sufficiently low, say, 10). Secondly, it needs to detect only one small block of a moving object to capture the entire moving object. Thus removing blockiness should also greatly help to remove false cleaning artifacts. Now the feedback and my own experience tell me that for sufficiently fast global motion (typicly scharfis_brain suggestion only concerns global motion) this very sophisticated strategy is not sophisticated enough.

My future plan is to modularise RemoveDirt. Firstly, there will be a simple filter, which creates a block mask clip, this is a yuy2 clip with 1/8 of the height and 1/8 of the rowsize of the original clip. Then there will be an equally simple filter which provides masked cleaning in terms of such a block mask clip and finally there will be a postprocessing filter, the only non-trivial filter. There will also be additional filters for debug output etc. That modularisation certainly adds some overhead (I expect the modular version to be about 5-15 fps slower than the monolithic version on my whimpy pc), but adds an enormous amount of flexibility. For instance, one may use a more sophisticated block mask generator or may replace the simple masked clenser by something more sophisticated. Also the postprocessing routine doesn't need to compare/restore with the original clip. Thus the RemoveDirt plugin should become a tool box for builting cleaners possibly using also third party filters. I plan to use it for a cleaner, which uses a Clense + RemoveGrain in static blocks and Clense + Repair + RemoveGrain (see the RemoveGrain thread for more details) for the remaining blocks to achieve a maximum amount of compression without cheap blurring. With a modular design it is also easier to explore and test new ideas, which may ultimately lead to a resolution of the earth quake motion problem.

[Mug Funky]

ah. good to hear.

i've tried making a scriptclip to check the difference between the source and removedirt'ed clips, and when a spike is found it uses the source. it worked on a few frames, but then i quickly realized it stopped working very quickly and simply gave me the original source for almost every frame.

i'll experiment with using more sophisticated methods (it's rather like scene-change detection), but i suspect each better-working solution will slow things down more.

right now i'm using a masking approach that is slow as hell but guarantees a shear-free clip.

[kassandro]

Originally posted by Mug Funky
i've tried making a scriptclip to check the difference between the source and removedirt'ed clips, and when a spike is found it uses the source. it worked on a few frames, but then i quickly realized it stopped working very quickly and simply gave me the original source for almost every frame.

I don't know, whether it would help, but at least one could avoid ScriptClip.I could easily write a filter, which does the following: it compares for each frame the two input clips. If the SAD difference does not exceed a threshold specified as a variable, the frame of the first clip is chosen otherwise the frame of the second clip. Thanks to the psadbw instruction this can be done very fastly (cetainly more than 100 fps on my 1.3 GHZ Celeron). Probably the Scriptclip version is below 1 fps.

[Mug Funky]

surprisingly the scriptclip version ran a total of about 7 fps on my very modest system (p3 733). it just didn't work properly, which is why i abandoned it. but the idea isn't completely mined out though, and i might have another go at it later.

[edit]

tried replacing "averageluma" with "ydifferencefromprevious+ydifferencetonext" and got some more reliable results - now less noise is being let through, and more actual problems are being spotted.

runs at about 8fps (this is my total script speed, which includes smartdecimate and some slow fallback options for it which don't come up much).


function carefulDirt (clip c, int "th", bool "show")
{

th = default(th,16)
show = default(show,false)

dirtless=c.removedirt()

maskd=yv12lutxy(c,dirtless,yexpr="x y - abs 16 > 255 0 ?", Y=3,U=-128,V=-128)
maskd=maskd.expand().reduceby2().reduceby2().expand(Y=3,U=-128,V=-128)

dirtless = (show==true)? dirtless.subtitle("removedirt-ed") : dirtless

conditionalfilter(maskd,c,dirtless,
\ "Ydifferencefromprevious()+Ydifferencetonext()","greaterthan",string(th))

#scriptclip(maskd,"subtitle(string(Ydifferencefromprevious()+Ydifferencetonext()))")

}

[kassandro]

Originally posted by Mug Funky
surprisingly the scriptclip version ran a total of about 7 fps on my very modest system (p3 733). it just didn't work properly, which is why i abandoned it. but the idea isn't completely mined out though, and i might have another go at it later.

As I can see carefulDirt uses ScriptClip only for debugging. If you would really use ScriptClip, you couldn't obtain 7 fps, because ScriptClip has to run for each frame the Avisynth initialisation of the script, which is executed by ScriptClip. If the ScriptClip script uses filters with lengthy initialisation like RemoveDirt, it really becomes a paine.

[Chainmax]

So what's in store for RemoveDirt and RemoveGrain? Will they be integrated into the new RemoveDust? BTW, will RemoveDust be able to remove black spots like RemoveDirt does?

[kassandro]

Originally posted by Chainmax
So what's in store for RemoveDirt and RemoveGrain? Will they be integrated into the new RemoveDust? BTW, will RemoveDust be able to remove black spots like RemoveDirt does?
As said above I will modularise RemoveDirt, the actual cleaners then will be script functions, which put the pieces together. At a later stage, it is probably reasonable to put all the plugins together into single cleaner plugin, but as long as things change a lot it is preferable to keep them separate.

[doxville]

Hi!

I came over following problem last night when i was about to use the removedirt-filter: Without having parameters in the brackets, everything was fine, but as soon as i put some parameters (like 'tolerance=48') into the brackets, TMPGEnc won't accept audio any more. I could not import the avi as an audio source, because of TMPGEnc's 'cannot open, or unsupported'-error. The script was like this:
AVISource("D:\video.avi")
delayaudio(1)
Trim(50,43856)
unfilter(75,50).LanczosResize(448,544,8,4,704,568)
removegrain(7).removegrain(6).removedirt(tolerance=48)
colorYUV(levels="PC->TV")
tweak(sat=1.2,cont=1.2,bright=5)
AddBorders(16,16,16,16)
separatefields()
selecteven()
weave()
I changed the script and used temporalsoften instead, and everything was ok:
AVISource("D:\video.avi")
delayaudio(1)
Trim(50,43856)
unfilter(75,50).LanczosResize(448,544,8,4,704,568)
removegrain(7).removegrain(6).temporalsoften(2,3,5,6,2)
colorYUV(levels="PC->TV")
tweak(sat=1.2,cont=1.2,bright=5)
AddBorders(16,16,16,16)
separatefields()
selecteven()
weave()

Can anyone tell me what went wrong..??

[kassandro]

Can you load the first script into Vdubmod? I don't have Tmpgenc. You may process the audio separately with avs2avi and an mp2 encoder and mux the audio and video later. On the other hand, such mysterious problems should be investigated.

[doxville]

The script doesn't produce problems neither with VDubMod (i initially edited the script with the VdubMod-script-editor) nor with MediaplayerClassic (which shows me a normal preview including video AND audio). So until now i can't tell you how to reproduce the situation without using TMPGEnc.:( There's a free version of TMPGEnc, maybe give it a try: http://download.pegasys-inc.com/download_files/TMPGEnc-2.521.58.169-Free.zip
Of course i could have processed the audio seperately, then all would be fine, it's just that i couldn't believe that a denoiser affects the audio in a script...:confused:

btw: i'm using the latest Avisynth 2.5.5

[kassandro]

Before I download the free version of Tempgenc, could you please test the following:
1. If you replace video.avi by another avi, does the Tmpgenc problem persist?
2. If so, does the problem still persist, if you load an mpeg2 video with MPEG2Source/MPASource instead of AVISource?

A video only filter, like all my filters, just passes through the audio. The only explanation for the problem is that RemoveDirt touches forbidden memory areas, which are only important for Tmpgenc.

[doxville]

Okay, here's what i tested:

1. The same problem occurs with another AVI-file as a source

2. A m2v-file imported with either 'directshowsource()' as well as with 'mpegsource()' lead both to the same problem i am having with the AVI...

[kassandro]

Thank you for testing. I should be able to reproduce it. It shows that the problem has nothing to do with the origin of video and audio. That problem may also be related with other mysterious problems reported here. I am leaving now for a short vacation a will be back on october 4th. Your problem is sufficiently concrete to be investigated first then.

[doxville]

Thank you! Enjoy your vacation! :)

[Major]

Hi Men

I have used removedirt in classic films (black and white)
my setting are:
removedirt(grey= true, pthreshold= 10, mthreshold=180) I have not put cthreshold because is ignored like it is written in the help pf the removedirt web page.

I have noticed a slight cleaning and i have two questions:

1. My setting are good??

2. if I apply removegrain with removedirt, will be better for cleaning the movie??

Thanks

[iradic]

hi

is it possible to use writefile func to create range files?
i was thinking using quantified motion function (its in avisynth doc) for that...

will resulting file be ok for removedirt or do i have to modify it?

is this ok (output of writfile):
--------------------------------
0
1
2
3
...
100

or:
---
0-100

this way i dont need to preview the whole movie before applying removedirt...

thanks

[kassandro]

@Major:
I too encode a lot of b&w movies and your settings look very similar too mine, for me mthreshold=180 is a little bit too high. I use values <= 150. That should still take out a lot of dirt. You may use RemoveGrain(mode=8, modeU=-1) for precleaning before RemoveDirt and then you can lower mthreshold somewhat more. I you merely want better compression, you may use one of the two RemoveDust functions discussed on the RemoveGrain web site. The first one yields very compression however with prehaps too much motion blurring. The second one - currently my personal favorite for progressive material -, somewhat slower with less compression but with only very mild motion blurring. However, RemoveDust, unlike RemoveDirt, can only remove dust particles and grain but no bigger dirt. On the other, these small particles are much more frequent and have therefore a much more significant impact than the big very visible dirt spots.

@iradic:
both ways are correct to describe a range from 0 to 100 in a RemoveDirt range file.

[Boulder]

kassandro,

could you give your opinion on this: http://www.kvcd.net/forum/viewtopic.php?t=14141 . Are the thresholds just too high (apparently the default ones) or is that loss of fine lines in the arc inevitable?

[kassandro]

Originally posted by Boulder
kassandro,

could you give your opinion on this: http://www.kvcd.net/forum/viewtopic.php?t=14141 . Are the thresholds just too high (apparently the default ones) or is that loss of fine lines in the arc inevitable?

The artifacts displayed in these three pictures are not caused by too high thresholds though pthreshold=0 may resolve the problem without harming cleaning to much. It doesn't make sense to solve this problem by lowering pthreshold below 0 or lowering mthreshold a lot, because this would essentially disable cleaning.
Firstly the dark curve marking the lower end of the arc looks like a scratch to RemoveDirt. If it would be nearly identical on at least one adjacent frame RemoveDirt wouldn't touch it. Unfortunately this is not the case here. Either the film camera was a little bit shaky or more likely the DVD mastering was poor. Even a one pixel motion conpensation would probably not resolve this problem completely and a two pixel motion compensation is simply too expensive in many ways.
If you look at this video in full speed instead of single frames, the artifacts, if realized at all, show up as a little bit of flicker. It is the price to pay for real cleaning. The suggested alternative Deen or PixieDust are not cleaners. If I understand the description of the algorithm of these filters correctly they avoids temporal blurring, if the temporal differences exceed a certain threshold. Such filters should be called smart compression improvers but not cleaners. Though I have only seen a few frames, the video is probably too clean for RemoveDirt and you pay the price of artifacts for very little cleaning. Thus for these kind of videos compression improvers like Deen, Dust or RemoveDust (which does some real cleaning and may also have problems with the arc) are more suitable. But make a comparison with material from the thirties and fourties and you won't mind paying the price of some slight artifacts.

[doxville]

I had the same effects in my last TV-capture (using removedirt without parameters):
http://people.freenet.de/doxville/01.jpghttp://people.freenet.de/doxville/02.jpghttp://people.freenet.de/doxville/03.jpghttp://people.freenet.de/doxville/04.jpg

These are screenshots of 4 consecutive frames, but look what it did to the marks on the street...:eek: Are these marks already too small to get recognized as details of the picture and therefore get filtered as 'dirt'?

After lowering the mthreshold down to 50 instead of 150, the target was getting better.

[kassandro]

You probably mean that RemoveDirt darkens some of the white street marks. As far as the first one from the left is concerned that may very well be the case, but again these artifacts are hardly visable at 25 fps. However, the sharp black spots on the second white stripe are certainly not caused by RemoveDirt (these spots are probably persons walking on the street). Here it is necessary to have the original frames. I consider mthreshold=50 too low and would first lower pthreshold or increase dist to 2.

[LeoH]

It looks like the white dashes are disappearing from frame to frame. This same sort of thing was happening to me with various scenes - I had to lower the threshold so low that it became useless. :(

~LeoH

[kassandro]

With RemoveDirt artifacts like in the above two impressive examples cannot be completely avoided, I can do something about it, though because of missing information I am not absolutely certain about the concrete cause here.
RemoveDirt spends most of its time with comparing 8x8 blocks from the previous and the subsequent frame. Thus maximum speed is very desirable for this most common operation. Hence I use the psadbw SSE instruction for this purpose. psadbw is even more necessary for doing any kind of motion vector search. However, psadbw has one serious drawback: one cannot ignore a certain amount of "small" invisable noise, very common in chemical film, which forces one to raise the mthreshold variable to achieve a reasonable amount of cleaning.

Until today I didn't want to change anything with RemoveDirt before modularisation next year. But now I see the necessity to do so. There will be one further new version this year with a new variable "noise".
If it is not set, then RemoveDirt will behave as before. If it is set however, it will use a significantly slower but better block comparison function, which will ignore pixel differences <= the value of noise. Overall I expect the frame rate of Removedirt to fall by 15-40%, if noise is set, but the likelihood of the above artifacts should diminish.
Personally I think the motion distortion reported by Mug Funky and others is a more severe problem than the above artifacts, which are hardly visable at full speed.

[oo_void]

Originally posted by doxville
I had the same effects in my last TV-capture (using removedirt without parameters):

These are screenshots of 4 consecutive frames, but look what it did to the marks on the street...:eek: Are these marks already too small to get recognized as details of the picture and therefore get filtered as 'dirt'?

After lowering the mthreshold down to 50 instead of 150, the target was getting better.
Not intending to detract from kassandro's fine work... But I've been having a lot of success doing spot removal with Fizick's (http://bag.hotmail.ru/) DeSpot, especially when combined with his motion estimation in the Depan plugin. Something like...

i = last
d = depanestimate(trust = 3)
depaninterleave(i, data = d)
despot(p1 = 30, p2 = 15, pwidth = 702, pheight = 276, mthres = 20, motpn = true, dilate = 1, seg = 1)
selectevery(3, 1)

The above example was pulled from the docs, so you'll need to tweak for your own purposes. Fizick's stuff is definately a bit more compilcated to tweak, but with the proper amount of effort, I've completely elimated all the false positives that I was getting with RemoveDirt. When I last tried RemoveDirt, it had a helluva time with shadows and wrinkles on white clothing (Princess Leia's dress in Star Wars for example).

------
oo_void

[Boulder]

Any news regarding the development? The filter's still on top in my analogue capture toolbox:)

[kassandro]

Originally posted by Boulder
Any news regarding the development? The filter's still on top in my analogue capture toolbox:)
Currently, there is no development going on. I just made the switch from my good old 1.3 GHZ Tualatin Celeron to a 2.66 GHZ Prescott Celeron. Unfortunately the system is getting instable, when I use my Technotrend SAT-TV card. I probably will replace it by a Skystar 2. Until all this are sorted out I cannot return to do some programming at home. In fact, I have not yet installed any development system. My first priority is to sort out the hardware problems. When I finally close the computer case, the first thing will be the completion of the SSE2 transition of my RemoveGrain package (I now have even SSE3). Probably by the end of February there should be a new version of this package.

[kingmob]

Originally posted by kassandro
Currently, there is no development going on. I just made the switch from my good old 1.3 GHZ Tualatin Celeron to a 2.66 GHZ Prescott Celeron. Unfortunately the system is getting instable, when I use my Technotrend SAT-TV card. I probably will replace it by a Skystar 2. Until all this are sorted out I cannot return to do some programming at home. In fact, I have not yet installed any development system. My first priority is to sort out the hardware problems. When I finally close the computer case, the first thing will be the completion of the SSE2 transition of my RemoveGrain package (I now have even SSE3). Probably by the end of February there should be a new version of this package.
Just want to add i can't wait. This filter is providing me with some excellent results and speed improvements are always welcome :).
Good job!

[Boulder]

Oh, and if you play with the filter, the presets need some work. I mean, they should definitely work so that every preset is separate.

For example, removedirt.ini with


capture # for TV-caps
mthreshold=100
pthreshold=20
cthreshold=30

bw # for b/w caps
mthreshold=100
pthreshold=10
grey=true


should work so that when 'capture' is called, it sets mthreshold to 100, pthreshold to 20 and cthreshold to 30. Everything else should remain at their default values. Otherwise the presets are pretty much useless, although using a removedirt.ini file with only one preset and calling that is a workaround to the bug where RemoveDirt gives an error message saying it doesn't have the argument mthreshold (or cthreshold etc.)

[kassandro]

@kingmob: thank you for your kind remarks.

@Boulder:
The default values for mthreshold have to be lowered anyway. As I have announced earlier I plan to refine the routine, which detects motion blocks. Essentially, pixel differences which are below the value of the new noise variable are ignored. The new RemoveDirt will behave as the old one, if noise=0. If noise > 0, then RemoveDirt will be somewhat slower but still reasonably fast. I hope to do this until March. Also some stability problems, for instance in combination with tmpgenc, should be fixed.
For the more distant future, I would like to dismantle RemoveDirt into several smaller filters, such that the old RemoveDirt becomes an AVS script function. The advantage of this modular approach compared with the old monolithic approach, is that additional interesting filters can be obtained as AVS script functions. I have demonstrated this already in the current RemoveGrain package and the next one will contain more in this direction.

[Boulder]

Sounds good indeed. Fortunately RemoveGrain+RemoveDirt combo already works very well so I can wait:)

[AVIL]

Hi

I've been testing removedirt over a clip using a cleaned (smoothed) version of the clip for the parameter neighbour. I been thinking that cleaned clip serves solely for motion detection, but the artifacts in this cleaned clip appears in the original clip. ¿Is it a bug or i am the bug?.

The script is :

setmemorymax(128)
a=avisource("bve_filtro.avi").crop(192,0,256,0,align=true).assumebff()
u=a.TDeint(order=-1,mode=1,field=1,type=0,sharp=true,mtnmode=0,mthreshL=3,mthreshC=3,cthresh=3)
v=u.removedirt(mthreshold=100,athreshold=50,pthreshold=19,neighbour=u.fft3dfilter(sigma=20,bt=3,measure=true))
stackhorizontal(u,v,mvex(u,v))

Function mvex(clip c ,clip d){

YV12LUTxy(converttoyv12(c), converttoyv12(d),yexpr="x y - 4 * 128 +" , uexpr="128", vexpr="128",u=3,v=3 ).converttoyuy2()

}

I've attached a frame with trhee parts. The first is the original, the second is the result of removedirt and the third is the difference of luma (four times magnified). It shows the tipical pattern of the filter FF3DFILTER with too high values for sigma. But i have uses this filter for the "neighbour" clip, not for the original clip.

Bye.

[Wilbert]

@AVIL,

I don't see any attachments pending :confused:

[AVIL]

Sorry:

I`ve hosted the image (click to enlarge). I must practise more in attaching files.


http://img225.exs.cx/img225/9330/removedirt9nn.th.jpg (http://img225.exs.cx/my.php?loc=img225&image=removedirt9nn.jpg)

[kassandro]

@AVIL:
As you already said the luma difference "picture" on the right side, shows a block pattern. The blocks are larger than the usual 8x8 blocks used by DCT codecs like XviD or a filter like RemoveDirt. Thus they must stem from FFT3D, which uses a variable block size. Now, if you use a clip filtered with FFT3D as a neighbour clip, it is used for cleaning, whence FFT3D blockiness propagates to the output frames of RemoveDirt. Of course, only in those areas where RemoveDirt cleans. This is normal.

I am currently testing a completely rewritten new version of RemoveDirt. Firstly the new plugin contains motion detection routines, which can ignore a specified level of noise. Secondly, the filter is modularised such that it can be combined with RemoveGrain/Repair, to achieve a maximum amount of cleaning. Actually the filter RemoveDirt has been replaced by RestoreMotionBlocks and RemoveDirt became an AVS script function involving RestoreMotionBlocks, Clense and various instances of RemoveGrain and Repair. The basic philosophy is to clean the static areas similar to the old RemoveDirt and the non-static areas with a much sharper variant of RemoveDust, which avoids motion blurring. Thirdly, support for interleaved YUY2 is replaced by support for planar YUY2, which bodes well for combining it with RemoveGrain.

I hope to release the new version next weekend.

[grua]

Hi,

I use RemoveDirt 0.6.1 with my interlaced analog captures and it does a great work - thanks for this great filter!

I use it in this way:
import("D:\Programme\AviSynth\Lib\SmartBob\mvBob\mvbob.avs")
LoadPlugin("D:\Programme\AviSynth\plugins25\Cnr2.dll")
LoadPlugin("D:\Programme\AviSynth\plugins25\RemoveDirt.dll")
LoadPlugin("D:\Programme\AviSynth\plugins25\Depan.dll")
avisource("test.avi",true,"YUY2")
crop(4,0,-4,-4)
addborders(4,0,4,4)
assumetff()
mvbob()
ConvertToYV12()
data=depanestimate(last,pixaspect=1.094)
depaninterleave(last,data,pixaspect=1.094,prev=2,next=2)
Cnr2(mode="oxx",scdthr=10,ln=35,lm=192,un=60,um=255,vn=60,vm=255,log=false,sceneChroma=false)
RemoveDirt(mthreshold=50,pthreshold=20)
selectevery(5,2)
AssumeTFF.ConvertToYUY2()
SeparateFields(a).SelectEvery(4,0,3).Weave()
When I open this script in VDubMod, everything works fine. But if I open it in Canopus Procoder Express, then there is the AviSynth-Error-Message, that there ist no parameter called mthreshold. Boulder has mentioned an simmilar problem in this thread a view posts above.
I had the same problem some days ago with other source-video and with other encoders too. But in VDub it always works.

Any idea?

[Boulder]

A workaround to the problem is to create a preset with those values in it and then call it in RD. Read the docs, creating presets is described there. This works if you always use those values you wrote or if you change them in the ini file when necessary. Multiple presets in the ini file apparently doesn't work as it should.

[kassandro]

Originally posted by grua

I had the same problem some days ago with other source-video and with other encoders too. But in VDub it always works.

Any idea?
Strange! As I use only VdubMod, I never observed this. There was a problem, when many range files were used, because Avisynth doesn't allow more than about 60 variables for a filter. Here, presets wera lso the only solution. The new RemoveDirt has neither presets nor range files anymore, but I can add it again in later versions, if there is sufficient demand or even necessity, but let's hope this problems are gone.

[grua]

Thanks - an other way ist to save the AVS-Output with VDub as AVI (lostless Huffy, fast recompress) and afterwards load this AVI in the encoder.

[Seed]

Originally posted by grua
Hi,

When I open this script in VDubMod, everything works fine. But if I open it in Canopus Procoder Express, then there is the AviSynth-Error-Message, that there ist no parameter called mthreshold. Boulder has mentioned an simmilar problem in this thread a view posts above.
I had the same problem some days ago with other source-video and with other encoders too. But in VDub it always works.

Any idea?
I don't think it is limited to Removedirt.

I'm quite sure that you have Intel P4 with hyperthreading enabled, and your avi is likely to be Huffyuv-compressed. I encountered this problem long time ago (with simple Avisynth script without any filter), and am quite surprised that there has been hardly any mention of it in the forum. I have since done some digging and found:

(A) In fact it happens to Avisynth in general, even just an one-liner "avisource("myvideo.avi")" script. Version 2.5x for sure, I can't remember if version 2.08 gives the same problem.

(B) For this Avisynth "exception error" to occur, 2 things must be present together:
(i) P4 CPU with hyperthreading enabled, and
(ii) frameserving to/via a Direct Show/Direct-X application (e.g. Media Player Classic, Windows Media Player, Procoder Express)

Therefore, you can circumvent the problem either by
a) disabling hyperhtreading, or
b) frameserving to a VfW (Video for Windows) application, e.g Virtualdub. I also use CCE Basic, and found that CCE Basic appears to accept Avisynth frameseving via VfW and not Direct-X, and thus does not give this "exception error" problem.

(C) When it happens, some codecs are more problematic than others. I use Huffyuv 2.1.1 most often, and with hyperthreading enabled, MPC will invariably crash with Avisynth exception error on playing 50-100 frames or on clicking 2-3 times on the seek bar.

Interestingly, it is much less "exception error" prone if ffdshow is used to playback Avisynth-scripted huffyuv-compressed avi files in MPC. Also, ffdshow's modified version of Huffyuv (FFVH) is also more stable. IIRC, uncompressed avi is not exempted from this "exception error" problem, but can be "stablised" too by using ffdshow as a direct-x filter.

This stabilization by ffdshow is not good enough for proper encoding job in Procoder Express (one garbled frame is too many), but makes it watchable in MPC.

[kassandro]

Originally posted by Seed
I don't think it is limited to Removedirt.

I'm quite sure that you have Intel P4 with hyperthreading enabled, and your avi is likely to be Huffyuv-compressed. I encountered this problem long time ago (with simple Avisynth script without any filter), and am quite surprised that there has been hardly any mention of it in the forum. I have since done some digging and found:

(A) In fact it happens to Avisynth in general, even just an one-liner "avisource("myvideo.avi")" script. Version 2.5x for sure, I can't remember if version 2.08 gives the same problem.

(B) For this Avisynth "exception error" to occur, 2 things must be present together:
(i) P4 CPU with hyperthreading enabled, and
(ii) frameserving to/via a Direct Show/Direct-X application (e.g. Media Player Classic, Windows Media Player, Procoder Express)


Interesting observations. As I have a CeleronD, I can't confirm it. If the application requests two frames simultaneously (in different threads) from Avisynth, garbage or even a crash is almost inevitable. However, I don't know, whether this really is the case here. My DVB program ALT-DVB does not work with hyperthreading as well. Also for video processing hyperthreading would not be useful anyway. Thus my decision to go for a poor man's P4 was probably right. Also the CeleronD can be easier overclocked (because of the slow 133 MHZ FSB).

[kassandro]

I just have uploaded a new version of RemoveDirt to a new server (Arcor, Germany). The link stays the same. The old Austrian server is down too often. The old RemoveDirt 0.6 will stay on the old server, but the links to it are currently incorrect.
The new version has been completely rewritten. RemoveDirt is now an Avisynth script function rather than a binary filter. The RemoveDirt plugin contains the filter RestoreMotionBlocks, from which together with the RemoveGrain package (you need version 0.8 at least) the script is built. RemoveDirt will be improved further with the new RemoveGrain 0.9.

I have done soemthing against the motion distortion observed first by Mug Funky, but I can't really confirm this until now. The tests I have done so far were done with a faulty version. I am doing the first real encoding test right now, but the test results with the faulty version were encouraging. As I have 19 inch (slow) TFT display which generates similar motion artifacts as the old RemoveDirt, I can only say something definite, after burning the results on DVD and watchin them on TV with may DVD player.

A lot of things have changed and one has yet to find the right default values. Presets and range files are currently missing, but I can include them in later version, if there is sufficient demand.

The new documentation is somewhat incomplete. Suggestions for improvement are wellcome.

[grua]

Originally posted by Seed
I don't think it is limited to Removedirt.

I'm quite sure that you have Intel P4 with hyperthreading enabled, and your avi is likely to be Huffyuv-compressed.Sorry, but I use an AMD Athlon Thunderbird 1,4 GHz and PicVideo MJPEG. Also I had this error really only with ReoveDirt and with no other.

But like I said above: no real problem for me, because I can export the AVS in Huffy with VDub and open this Huffy-Avi in Procoder Express.

cu, grua

[AVIL]

Hi:

@Kassandro

I have read carefully the guide about the new version of Removedirt. I'm not have understood the role of neighbour2. May you, please, explain how this parameter is handled.

Thanks in advance.

[kassandro]

Originally posted by AVIL

I have read carefully the guide about the new version of Removedirt. I'm not have understood the role of neighbour2. May you, please, explain how this parameter is handled.

I initially wanted to include the script function MCRemoveDirt in the documentation, but I didn't have the time. The neighbour2 variable was intended for using motion compensation, but I have now another application.
In order to decide, whether a block at time n should be marked as a motion block, RestoreMotionBlocks looks at the difference between this block in the frame neighbour(n-1) and in the frame neighbour(n+1). It doesn't look at the block in frame neighbour(n), because then dirt at time n would distort the differences (the neighbour clip normally is the original unclensed clip) and we really want to clean the dirt at time n.
Now if you specifiy neighbour2, then RestoreMotionBlocks behaves very differently. It compares the block in frame neighbour2(n) and in the frame neighbour(n). In particular, this implies that both clips are different (otherwise there would be no motion blocks at all). Let us look at the following interesting script function

function RMBDenoise(clip input, bool "_grey", int "repmode")
{
clmode=17
clensed=Clense(input, grey=_grey)
restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode )
corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, neighbour2=clensed, gmthreshold=70, dist=0, debug=true, noise=12, noisy=12, grey=_grey)
return RemoveGrain(corrected, mode=clmode, modeU = _grey ? -1 : clmode )
}

It looks very much like the RemoveDirt script function. neighbour2=clensed is the only difference. Now a block is marked as a motion block in the first phase, if the difference between the clensed block and the unclensed original block in the frame input(n) are sufficiently different. This makes this script function more defensive than RemoveDirt. It can no more take out big spots, though it still can take out scratches, if the value of noisy is large enough. On the other hand the artifact risk is lower as well. To make the above denoiser a little bit more aggressive, you may also use neighbour2=restore instead of neighbour2=clensed. Actually, if you use neighbour2=clensed there may be an unnecessary reprocessing, because I did configure the cache too tightly for this situation. This will be fixed in version 0.8.1.

By the way, I did decrease the value of noise from 12 to 8, because there were some rare artifacts at this level.

[kassandro]

The new version 0.9 can be found on the web site (http://www.RemoveDirt.de.tf).
I have added the SCSelect to the plugin. It is primarily for handling sharp changes in the RemoveDirt script function, which has been updated appropriately. As it is now, it seems to be a bit too aggressive. I just did copy it from my test script and so far I like what I see. The most recent RemoveGrain 0.9 is also needed for this script function.
The second major change concerns the second phase of RestoreMotionBlocks. There is now the new mode variable dmode. With dmode=0 (the default) RestoreMotionBlocks should behave as in version 0.8. Unfortunately it doesn't, because there was a bug in version 0.8.
dmode=1,2 are the other possible modes. The number of motion blocks declines as dmode is chosen higher. dmode=2 is quite aggressive and dmode=1 is probably the best compromise.
When dist=0 (no second phase for RestoreMotionBlocks), there was a simple but more severe bug in the third phase, the postprocessing phase. This has been fixed. The documentation has also been cleaned and extended subdtantially.
I hope that version 0.9 is more mature than 0.8.

[AVIL]

Hi.

The link above points to removegrain's page, not to removedirt's one

Regards

[kassandro]

Thanks, I have corrected it.

Alpennic, the company which provides the domain for free in exchange for a popup window, seems to block browsers with a popup blocker for a while.