RemoveDirt [Archive] - Doom9's Forum

kassandro

17th February 2004, 04:21

Originally posted by Bogalvator
1) "Never crop after RemoveDust" - not even if you resize afterwards to a mod 16 resolution? Does it really effect the codec compression performance?
[/B]
That was only reasonable guess of mine. A codec expert has really to answer this question. Or one has to run a lot of tests. On the other hand, I know that all high quality codecs divide the frame into a grid of small blocks and on these blocks the dct = discret cosine transform, a variant of the classical Fourier transform, is applied, to separate the important from the neglible things. That is the reason why too much compression makes a video visibly blocky. As far as I know the dct already appears in jpeg compression. Now RemoveDirt uses also such a grid of blocks and the output gets also blocky, though the blockiness is hardly visable, if pthreshold is not too high. On the other hand, the dct is more sensitve than the human eye, and it should react badly if the edge of a RemoveDirt block is right in the middle of a dct block. To avoid this, any edge of a RemoveDirt block should be the edge of a dct block. That is at least my guess. Now, if you crop on each side by a multiple of 8 that should be ok. But if you crop on the right hand side by 4 and on the left hand side by 12, then both grids are out of sync. The RemoveDirt edges are exactly in the middle of the dct blocks.
A further Remark about blockiness is in order. Because RemoveDirt bases the decision, whether to clean or not to clean, solely on the luma, there should not be any really visible blockiness as far as the luma is concerned. However, theoretically the chroma alone can cause blockiness, though this should be quite unlikely in a normal video. But obviously, as reported by Kintaro, more likely for anime videos. If chroma blockiness becomes a severe problem I can solve it at the expense of execution speed. Also the filter gets a lot more complicated to use, because then the chroma needs its own set of variables.

Originally posted by Bogalvator
2) With that in mind - what about colour space conversions? I would like to use Pixiedust after you filter, should I ConvertToYUY2() before using your filter, or will it be safe to use it in YV12 first then convert afterwards?
[/B]

As far as quality is concerned, it should not matter much, but RemoveDirt is certainly faster in YV12 than YUY2, at least if you use the default mode=2.

Originally posted by Bogalvator
Also, when I added removedust.dll to my plugins directory, both VirtualDub and TMPGEnc requested a file, msvcr70.dll, on startup (before loading AVS script). Naturally this message went away when I copied a version of it into both the VirtualDub and TMPGEnc directories, but do you know why this message came up. Anybody else have this problem?
[/B]
That is a problem with all my filters! To get a rather small dll, I use dynamic linking. In other words, various windows support function are not getting put into the dll. They are in msvcr70.dll and that is the reason why this dynamic link library is needed. Dynamic linking also saves memory during execution, because many other parts of the program can share msvcr70.dll, if they too use dynamic linking. But, if you know how windows wastes memory that shouldn't really matter. Dynamic linking may also be a tiny bit faster. There is one further problem: if your version of msvcr70.dll is not compatible (I really don't understand why this happens), you get execution problems. Perhaps, this is the reason for the crashes reported by Kintaro. With a staticly linked dll, you would not have these problems, because all the support routines are put into the dll. But if you use many staticly linked filters the same support routines may be linked many times into the executable. Nevertheless, I should perhaps release also a staticly linked version.

kassandro

17th February 2004, 04:40

Originally posted by Kintaro

I think its somehow related to avisource, tried opening all avifiles with directshowsource to confirm that -> and not a sinlgle crash occured.

You got it! Excellent debugging! I have reproduced exactly, what you said. RemoveDirt and avisource do not like each other. I can now investigate this myself. But first I have to sleep a little.

kassandro

17th February 2004, 05:42

I couldn't resist to track down the bug detected by Kintaro. I have now a do nothing filter (it simply passes through all the frames) with only a few lines of source code which exhibits the same hostility to avisource. I will post it later to the AVS developer forum. It remains to be seen whether I am doing something forbidden during dll initialisation or whether this a Avisynth bug. Fortunately, this bug has nothing to do with the filter itself.

Mug Funky

17th February 2004, 17:46

to go back to the question about not cropping after removedirt... it will affect compressibility for the DCT components (block edges are shifted and mistaken for high frequencies. high frequencies compress badly because of the VLC coding of blocks after quantization). this probably wont affect motion vectors, so there will still be a gain on your original source.

of course, you could run a deblocker like "blindPP(quant=2,cpu=4)" after removedirt and you might be able to crop after that, so long as the deblocking does it's thing.

[edit]

it'd be really nifty to have hard-coded presets for this (if it's not too much to ask). i ask because although yesterday i knew what the settings do, i've forgotten now... presets would be very cool and could handle 90% of situations flawlessly i think.

kassandro

17th February 2004, 21:52

Originally posted by Mug Funky
to go back to the question about not cropping after removedirt... it will affect compressibility for the DCT components (block edges are shifted and mistaken for high frequencies. high frequencies compress badly because of the VLC coding of blocks after quantization). this probably wont affect motion vectors, so there will still be a gain on your original source.
Thank you for this confirming comment. You obviously know more about the subject.

Originally posted by Mug Funky
it'd be really nifty to have hard-coded presets for this (if it's not too much to ask). i ask because although yesterday i knew what the settings do, i've forgotten now... presets would be very cool and could handle 90% of situations flawlessly i think. [/B]
Currently the plugin is not yet mature enough. But if some agreed presets emerge, then I certainly would implement this presets for quick usage.

Bogalvator

18th February 2004, 01:57

Thank you Kas and Mug for the answers regarding the cropping issue, they were helpful. I think I see why directly cropping the video would have a negative effect on codec performance now.

Look forward to future versions of the filter.

leadman584

18th February 2004, 19:04

Very nice little filter ya got here. Definitely useful for compressibility gain with minimal speed loss. Getting very good results. Keep up the good work.

acrespo

19th February 2004, 07:46

I am a user of MipSmooth in analog anime content. After did some tests with this new filter I am impressed with results. I will do more tests to compare with mipsmooth but I think my new filter is RemoveDirt ;)

Keep good work :)

scharfis_brain

19th February 2004, 18:59

Always getting

AVISynth open failure:
Loadplugin: unable to load "removedirt.dll"

any ideas?

Dreassica

19th February 2004, 19:12

no problems here, using it in autoload environment.

kassandro

19th February 2004, 22:28

Originally posted by scharfis_brain
Always getting

AVISynth open failure:
Loadplugin: unable to load "removedirt.dll"

any ideas?
This is probably due to a missing msvcr70.dll library, which should be put into C:\windows\system32, such that every piece of code, which is dynamicaly linked like RemoveDirt.dll, can find it there. If all applications would be dynamicaly linked, we wouldn't have this problem and little bit more free disk space. I may also provide a staticly linked version, though, to avoid these problems. Please don't put several copies of msvcr70.dll in different application dirs, because then dynamic linking would cost even more disk space and memory than staticly linked binaries. C:\windows\system32 should be the only place.

scharfis_brain

19th February 2004, 22:48

Thanks a lot!

It is working now!

EDIT: please remark this issue to your documentation!

krieger2005

26th February 2004, 20:06

Is there maybe a version without SSE-optimazion. Not all people has CPUs with SSE (like me).

kassandro

26th February 2004, 23:56

Originally posted by krieger2005
Is there maybe a version without SSE-optimazion. Not all people has CPUs with SSE (like me).
In theory, I could provide a non-SSE version, because all low routines have not only been written in SSE assembly code, but also in C. Then I have compared whether the C and the SSE version yield the same result. Checking the validity of the SSE version was the only reason for the C version. In real life the C version would be unusable, because it is at least by a factor of 10 slower than the SSE version. SSE allows not only to handle 8 pixels at a time, but also the programming is virtually jump free. Together with instruction pairing this keeps the cpu pipelines full almost all the time. That makes it possible for RemoveDirt to be slightly faster than the well known mpeg2dec3, that means about 80 fps for a 720x576 clip. Now any cpu without SSE is much slower than my 1.3 GHZ Tualatin Celeron. So it really doesn't make sense to me.

Fizick

27th February 2004, 05:33

Removedirt is needed in Integer SSE or Full (Integer and Float) SSE?

yaz

27th February 2004, 10:12

i've just tested rd on some heavy sources (monty python trilogy, blade runner & so) great stuff. the member of the crew, definitely.
thx for it!
y

kassandro

27th February 2004, 10:23

Originally posted by Fizick
Removedirt is needed in Integer SSE or Full (Integer and Float) SSE?
Only integer SSE is used. But is there any cpu with integer SSE and without floating point SSE? If that is the case, I will certainly fix it in the next version, which will be released next sunday.

Fizick

28th February 2004, 00:45

SSE: New commands present in P3, P4, newer Celerons, Athlon XP, MP.

Integer SSE: A subset of the SSE command set, mostly used for video processing. These instructions are also present in AMD Athlon (all versions), AMD Duron (all versions).

See:
http://www.avisynth.org/index.php?page=IntegerSSE

kassandro

28th February 2004, 07:42

Originally posted by Fizick
SSE: New commands present in P3, P4, newer Celerons, Athlon XP, MP.

Integer SSE: A subset of the SSE command set, mostly used for video processing. These instructions are also present in AMD Athlon (all versions), AMD Duron (all versions).

See:
http://www.avisynth.org/index.php?page=IntegerSSE
Sorry, I didn't know that and will fix it today. I thought the older Athlons and Durons have only 3DNOW. Clearly Integer SSE is much simpler than full SSE, because it does not require the 128 Bit xmm registers. It only implements a few important instructions which were unfortunately missing in the original mmx instruction set.

kassandro

29th February 2004, 00:16

I just have put version 0.5 onto the website. The binary package contains now also a staticly linked version. Please put only one dll into the plugin directory. The following changes have been made:
1. Chroma postprocessing has been added for YV12 clips. By default RemoveDirt does not only luma postprocessing but also chroma postprocessing. It is controlled by the new variable cthreshold. By default, cthreshold=pthreshold. To avoid distortion by the chroma, it is now more important to use "grey=true" for black&white video (in version 0.4 this option was only good for a slight speed increase). This option disables chroma postprocessing. Chroma postprocessing is also disabled if cthreshold > 8*255. Then version 0.5 operates the same way as version 0.4 and is somewhat faster. For b&w videos I recommend to lower pthreshold to 30 or even 20 to avoid slight blockiness in areas with very flat contrast. Chroma postprocessing is yet only implemeted for YV12. Unfortunately, in YUY2 the chroma is vertically twice as dense as it is horizontally. An implementation would therefore require a vertical and horizontal cthreshold variable.
2. If RemoveDirt is used jointly with AVIsource, VirtualdubMod did crash upon leaving (many thanks to Kintaro for detecting this problem). In version 0.5 this problem has been fixed. However, if RemoveDirt stops during initialisation, because something has gone wrong, it stills messes up VirtualDubMod, if it is jointly used with AVIsource. This is an Avisynth bug. Many other filters share the same problem, if they stop during initialisation and are used jointly with AVIsource. Hopefully this problem is fixed with the next version of Avisynth.
3. I always thought, that upon loading a new script, Avisynth starts with a fresh heap and therefore never released allocated memory. Because RemoveDirt allocates one byte for each frame to handle range files, that led to a loss of 135000 Bytes for a 90 minute PAL-video each time the script was reloaded. Together with a slight Avisynth design flaw the memory leak was also responsible for the above VirtualDubMod crash. The memory leak has been fixed in version 0.5.
4. Version 0.4 rejected any cpu, which did not have full SSE capability, although RemoveDirt only uses Integer SSE. As I learned recently (thank you Fizzik), old Athlons and Durons have Integer SSE but not full SSE. Version 0.5 now works with these cpus as well.
5. In version 0.4, the "grey" option was not handled properly for the first and the last frame within range files. 0.5 fixes this problem.
6. The dirt oscillation check of RemoveDirt has been disabled by default. Unfortunately this feature prevented a lot of dirt from being removed, while doing little for artifact protection. For instance, damages caused by film projector heat are black in the interior but have a tiny white boundary, whence this frequent kind of "dirt" has a very high oscillation. RemoveDirt is also faster without oscillation dirt check. Putting doscillation=50 restores the old default mode. In subsequent versions the oscillation check may be removed completely.
7. Various updates for the documentation.

Following a suggestion of Mug Funky I am preparing a general preset mechanism for version 0.6. If now RemoveDirt is used with RemoveDirt(input, default="anime", ...) then RemoveDirt looks for a file named RemoveDirt.ini in the script directory and in this configuration file it looks for preset values associated with anime and replaces the builtin defaults by the anime defaults. The syntax of RemoveDirt.ini will be simple and efficient. In fact, the basic class to achieve the preset mechanism is already in the source code (it can be easily used for other filters as well), but is yet untested, has to be connected with RemoveDirt and there is not yet any documentation.

Thanks for the friendly comments!

krieger2005

1st March 2004, 17:10

Hi,

now i can use RemoveDirt too (seems that i have an old AMD ;)).

I tried your filter and its works very fine. But I have some questions:

First
I tried now to use the range-Options. Here the Script:

ConvertToYV12().RemoveDirt(range1= "1084 2971",
\ mode=2, dist=3, tolerance=50)

As you can see i took first some frames. When i try to open this Script in VDubMod i get this error-message:
"IO Error with 1084 2971"
I thought that this was the cause of use of "avisource". Tried then open my file with "mpeg2source" (i have of this movie an avi-file and a DVD). But the same error.

Second
I have first many more frames. When i tried to open the script i get always this error: "the named argument "range6" was passed more than once to RemoveDirt"

I have used range6 only one time in the hole script.
Here the code:

ConvertToYV12().RemoveDirt(range1= "1084 2971",
\ range2= "60930 61216 61492 62123 62233 62596 62650 62899 62976 63265 63292 63321 63496 63783 63807 64374 64563 65021 65070 65568 66001 66035 66074 66151 66176 66438 67149 67185 67262 67429 67470 68033 68078 68224 68399 68833 68993 69086 69125 69348 69516 69844 69940 70297 71116 71343 71488 71664 71981 72015 73006 73031 73927 73976 74131 74350 74644 74748 75036 75153 75356 75402 75431 75540 75559 76518 76602 76764 76974 77014 77111 77200 77372 77744 77803 78156 78679 78751 78963 79061 79121 79205 79838 79881 79951 80078 81531 81616 81997 82212 82391 82526 82836 83056 83113 83244 83391 83471 83552 83568 83615 84162 84417 84677 84803 84973 85023 85807 85869 85942 85984 86041 86259 86353 86425 86554 86629 86771 86924 86958 87138 87253 87315 87368 87405 87453 87721 87745 87821 87920 87966 88095 88181 88564 88745 89111 89523 89761 90417 90618 91494 92227 92425 92689 92794 92839 93098 93265 93349 93389 94218 94243 94612 94992 95352 95615 96007 96120",
\ range6= "100248 100484 100540 100773 101062 101116 101646 101918 101996 102051 102303 102545 102934 103108 103189 103315 103335 103746 103824 104009 104220 104599 104632 104815 105143 105222 105943 106094 106329 107809 108084 108297 108981 109212 109616 111192 111704 111795 111923 112117 112389 112591 112685 112909 112923 112947 113383 113617 113707 113881 114103 114137 114170 114246 114432 115131 115154 115274 115377 115464 115506 115621 115669 115720 116017 116080 116126 116185 116372 116658 117990 118224 118696 119028 119247 119450 119494 119546 119660 119721 119816 119899 120040 120340 120469 120710 120829 120923 121163 ",
\ range5= "62597 62651 62900 62977 63266 63293 63322 63497 63784 63808 64375 64564 65022 65071 65569 66002 66036 66075 66152 66177 66439 67150 67186 67263 67430 67471 68034 68079 68225 68400 68834 68994 69087 69126 69349 69517 69845 69941 70298 71117 71344 71489 71665 71982 72016 73007 73032 73928 73977 74132 74351 74645 74749 75037 75154 75357 75403 75432 75541 75560 76519 76603 76765 76975 77015 77112 77201 77373 77745 77804 78157 78680 78752 78964 79062 79122 79206 79839 79882 79952 80079 81532 81617 81998 82213 82392 82527 82837 83057 83114 83245 83392 83472 83553 83616 84163 84418 84678 84804 84974 85024 85808 85870 85943 85985 86042 86260 86354 86426 86555 86630 86772 86925 86959 87139 87254 87316 87369 87406 87454 87722 87746 87822 87921 87967 88096 88182 88565 88746 89112 89524 89762 90418 90619 91495 92228 92426 92690 92795 92840 93099 93266 93350 93390 94219 94244 94613 94993 95353 95616 96008 96121 96416 96837 98620 98676 99029 99111 99158 99569 99647 99837 100116 100168 ",
\ mode=2, dist=3, tolerance=50)

What could this error mean?

Third
The Filter works very bad at scene-changes. Is there something in your mind to solve this problem. This is why i have so many frames above. All these frames were at scenechange and were cleaned bad.

But at all i must say, that this is at the moment one of the best Filter to restorate old Movies...

Thank you

kassandro

1st March 2004, 23:47

now i can use RemoveDirt too (seems that i have an old AMD ).
Glad to hear, that the plugin works now for the older AMD's as well.

ConvertToYV12().RemoveDirt(range1= "1084 2971",
\ mode=2, dist=3, tolerance=50)

There is a misunderstanding. range1, range2, etc. expect file names as arguments, not frame sequences. Thus write a text file with "1084 2971" in it, but please without the quotes and let's call it, say, "range1.rd" and in your script replace the above line by

ConvertToYV12().RemoveDirt(range1= "range1.rd",
\ mode=2, dist=3, tolerance=50)

Then the script should work as intended.

IO Error with 1084 2971
That is an error message of RemoveDirt. It indicates an io error with the file named "1084 2971". In this case the filter couldn't simply open the file.

the named argument "range6" was passed more than once to RemoveDirt
This is an error message of Avisynth, not of RemoveDirt. It usually happens if a named argument is used more than once. Clearly you haven't done that. I will investigate this. Perhaps Avisynth has problems with the crazy length of the strings, which you used because of the above misunderstanding. Nevertheless Avisynth should be able to handle even longer strings. Now write for each range1, range2, range5, range6 a range file as above and make the appropriate changes to your script (hopefully the file names are shorter than the frame sequences).

The Filter works very bad at scene-changes. Is there something in your mind to solve this problem.
I wouldn't say bad. RemoveDirt simply can't clean at a sharp scene switch. I would say bad, if it would creat severe artifacts. That is truely the worst. You can force RemoveDirt to clean with tolerance=100. If there is not much motion the damage will be limited but I can't recommend it. It is somewhat better if you use the two pass method, as decribed in the documentation, at scene switches, but I can hardly recommend cleaning at scene switches. Whay can't RemoveDirt clean at scene switches? The basic idea behind any temporal cleaner is to compare frame n-1 and frame n+1. If their difference is much smaller in a certain area than the difference of frame n-1,n and the difference of n, n + 1, then one may suppose dirt in this area of frame n (unfortunately that is only half the truth). Now at a sharp scene switch the difference of frames n-1 and n+1 is unusually large, whence there will only be cleaning in very few random blocks. Nevertheless the basic idea can be modified to work at scene switches. Two cases have to be distinguished: 1. frame n is the last frame of a scene. 2. frame n is the first frame of a scene. In the first case, frame n+1 doesn't fit at all and has to be replaced by frame n-2. In the second case, frame n-1 doesn't fit and has to be replaced by frame n+2. However, one has to be very certain about a scene switch and the above modifications should only be used at scene switches. Obviously this special handling of scene switches has not yet been implemented in RemoveDirt and is of low priority because it concerns only a few frames. It is easier to creat a special filter, which only cleans at scene switches. My main priority is to improve cleaning in the presence of motion, without increasing artifact risk.

Heini011

17th March 2004, 09:28

Hi,

thanks very much for this great filter! i came to good results on
a movie with not very much noise:

RemoveDirt(pthreshold=25,cthreshold=40,mthreshold=200,athreshold=80,show=0)

so i think the default values for motion are quite conservative, but
for dirt detection they could become a litte bit more conservative...

the value of soscillation seems to have no influence at all - maybe
a bug ?!

greetings, Heini011.

kassandro

17th March 2004, 16:24

Originally posted by Heini011
the value of soscillation seems to have no influence at all - maybe
a bug ?!

greetings, Heini011.
since the last version I have raised the default value of doscillation to 260, which disables oscillation checking. You can reactivate oscillation checking by setting doscillation to a value well below 255. Then also soscillation beomes important again. As I wrote earlier, oscillation checking has turned out to be not very useful. On one hand, it does very little for artifact protection. On the other hand, it prevents a lot of dirt from being cleaned.

Heini011

22nd March 2004, 14:47

Hi,

one good automatic solution for high-motion scenes:

FrameEvaluate("mot_level = (YDifferenceFromPrevious+YDifferenceToNext)/AverageLuma")

vid_mo=last
vid_st=RemoveDirt(pthreshold=25,cthreshold=40,mthreshold=160,athreshold=50,show=0)

ConditionalFilter(vid_st, vid_mo, "mot_level", "<", "0.07")

kassandro

23rd March 2004, 13:43

Originally posted by Heini011
Hi,

one good automatic solution for high-motion scenes:

FrameEvaluate("mot_level = (YDifferenceFromPrevious+YDifferenceToNext)/AverageLuma")

vid_mo=last
vid_st=RemoveDirt(pthreshold=25,cthreshold=40,mthreshold=160,athreshold=50,show=0)

ConditionalFilter(vid_st, vid_mo, "mot_level", "<", "0.07")
Nice, my forthcoming filter ImproveSceneSwitch, part of RemoveDirt 0.6 due out next sunday, is based on the same idea. However, it will be a lot faster and it will handle both sides of a sharp scene switch. The last frame of a scene will be replaced by the penultimate frame, while the first frame of the new scene will be replaced by the second frame of the new scene.

kassandro

1st April 2004, 07:46

I just put up a new version of RemoveDirt to the web site. Actually RemoveDirt itself is almost unchanged. However two major features were added to the plugin. Firstly I have added a flexible preset facility to RemoveDirt, such that any user can creat his very own presets. Secondly I have added the the filter ImproveSceneSwitch, which is useful for most temporal filters to cope better with sharp scene switches. Thus it can not only be used with RemoveDirt but also with my deinterlacer AlignFields. For more details I refer to the updated documentation.

Mug Funky

1st April 2004, 11:16

thanks! i really dig this filter...

FredThompson

2nd April 2004, 05:22

How does the output of RemoveDirt compare to Peach's output?

kassandro

2nd April 2004, 08:49

Originally posted by FredThompson
How does the output of RemoveDirt compare to Peach's output?
I cannot find any source code for Peachsmoother. Thus I can only make guesses. As the name indicates Peachsmoother is blurring dirt, if it detects it. RemoveDirt tries to completely remove detected dirt. The advantage of blurring is that artifacts are less severe, if motion is confused with dirt. Thus RemoveDirt requires very sophisticated motion detection and artifact removal.
Peachsmoother is a temporal and a spatial filter. RemoveDirt is 98% temporal. It has a spatial aspect only, because all the processing decisions are based on 8x8 pixel blocks instead of single pixels.
Actually I recommend to use a slight softener after RemoveDirt, to deal with dirt, which could not be handled by RemoveDirt.
RemoveDirt is specifically aimed at dirt and scratches of photographic film, in particular the clips should be progressive. If the clip is telecined, it should be inverse telecined before using RemoveDirt, but best results are obtained with genuinely progressive material. RemoveDirt is not a general purpose noise filter.

FredThompson

2nd April 2004, 10:29

OK, sounds great.

Any known problems filtering split interlaced source then weaving?

This might go a long way towards helping remove tape dropouts.

scharfis_brain

2nd April 2004, 12:43

@fred:

If you want to filter VHS-dropouts, try this:

ibob() #do not use the internal bob. use ibob.dll
removedirt(params...)
separatefields()
selectevery(4,0,3) #or (4,1,2) choose the cripser one. depends on Fieldorder
weave()

I didn't tested this. Its just theoretically.
I think that removedirt could get some problems with the heigh-detail interlacing-shimmer, caused due to the bobbing.

I have no drop-out-sequences to test on.

iradic

2nd April 2004, 14:17

hi...

i have this scene in movie with ship passing by - ship has some black vertical lines which are constantly removed/not-removed with removedirt

what settings should i tweak to avoid this?

thanks

Mug Funky

2nd April 2004, 15:34

@ scharfi: avisynth's internal bob is lossless so long as the "a" parameter is zero. (a = "blurring", b = "ringing" parameters of the Mit-Net filter)

ie: bob() is not lossless as a=1/3 and b=1/3

but bob(0,.5) is lossless, as is bob(0,1)

you can verify with subtract(last, last.bob(0,.5).selectevery(4,0,3).weave)

you'll get pure grey...

(also, ibob() doesn't work in the latest binaries, unless i wasn't holding my mouth right)

kassandro

2nd April 2004, 18:38

Originally posted by iradic
hi...

i have this scene in movie with ship passing by - ship has some black vertical lines which are constantly removed/not-removed with removedirt

what settings should i tweak to avoid this?

thanks
By the very nature of the algorithm, RemoveDirt can only clean dirt on single frames. If two subsequent frames have the same dirt at the same position it cannot be cleaned. Hoping that this is not the case,
the problem is the moving ship. With reasonable values of mthreshold etc. RemoveDirt does not clean in the presence of motion to avoid artifacts. However, you can force RemoveDirt to clean everything, by setting mthreshold=5000 and athreshold=5000. However, this should be done only for a selected number of frames and all the selected frames have to be checked for artifacts. The selected frame numbers or frame range should be written to a file, say, "clensing" as described in the documentation. Then
RemoveDirt(range1="clensing", mthreshold1=5000, athreshold1=5000, ...)
should totally clean the frames specified in the file clensing, while the other frames are processed as usual.

iradic

2nd April 2004, 21:29

sorry i wasnt clear...

you are explaining situation where i want to remove these black lines right?
well i dont want to (my wrong words - "lines" - are part of the ship)...
anyway i will try to work with those tresholds but in opposite values - trying to save these "lines"...

thanks for help

kassandro

3rd April 2004, 10:23

Originally posted by iradic
sorry i wasnt clear...

you are explaining situation where i want to remove these black lines right?
well i dont want to (my wrong words - "lines" - are part of the ship)...
anyway i will try to work with those tresholds but in opposite values - trying to save these "lines"...

thanks for help
I see, you want to preserve the black lines. If they are only partially erased, then pthreshold and cthreshold are too low. Try pthreshold=20 or 10, cthreshold shouldn't matter much, because it seems to more a luma problem. The default value pthreshold=50 seems to be too aggressive in general. This was already remarked by other users.

krieger2005

5th April 2004, 11:46

Can You explain how the extrapolation-method works. I think does not seen the description about.

I tried the function for now. At Scenechange i need a ratio of "3" (at my movie). Interesting is that at ratio=2 the Function found other Scene-changes but not the same as with ratio=3 (with ratio=3 were not all scene-changes found). While the frame-doubling generate "clean" frames the extrapolation generate an "oversharp"-shadow frame. But as you are write at your description, the frame-double-method is bad, when an "fast-scene" is detected as "scene-change". I had seen such situation in my test-movie (with frame-doubling) and does not notice any bad artifacts (while see the movie at TV).

So for now the extrapolation-method is not my preferntial because it generate an oversharp frames and this look in the movie like an artiface (but not realy noticeable)

kassandro

5th April 2004, 19:02

Originally posted by krieger2005
Can You explain how the extrapolation-method works. I think does not seen the description about.

To explain (linear) extrapolation, let me first recall (linear) interpolation. If c[x,y,t] denotes the luma value of the pixel with coordinates x,y at time t, then (c[x,y,0] + c[x,y,2])/2 = c[x,y,0] + (c[x,y,2] - c[x,y,0])/2 is the interpolated luma value at time 1. If we have the values at time 0 and 1 and want to extrapolate the value at time 2, then c[x,y,0] + (c[x,y,1] - c[x,y,0])*2 is the natural analogue of interpolation, but because the parameter 2 is now outside the interval [0,1], this kind of procedure is now called extrapolation instead of interpolation. There one problem with extrapolation: while the interpolated value is always between the two values, this is no more the case for extrapoaltion. In fact, if c[x,y,0] and c[x,y,1] are within the "byte" range 0-256, the extrapolated value may be outside the byte range. Fortunately there are the SSE instructions paddusb and psubusb, which keep the extrapolated value within the byte range. Thus extrapolation can be implemented nicely with SSE.

Interesting is that at ratio=2 the Function found other Scene-changes but not the same as with ratio=3 (with ratio=3 were not all scene-changes found).

Any scene switch found with ratio=3 must be also be found with ratio=2. The lower the value of ratio the more the number of scene switches. There is one exception, though: arithmetic overflow. If the frames of a clip are made of 400,000 pixels, the maximal difference in the yv12 color space is 400,000*255. Now if ratio*400000*255 exceedes 1<<32 (approximately = 4000,000,000), then arithmetic overflow is possible and this may already happen with ratio=40. Now if the color space is RGB32, then the maximal difference is 400,000*255*4 and arithmetic overflow becomes possible already at ratio=10. Of course, I could have avoided arithmetic overflow altogether, by using division instead of multiplication. However, it is a good programmer's rule to avoid division as much as possible, because division, unlike multiplication, is a very slow cpu instruction.
Dirt may distort massively the frame differences. Taking this into account, better scene switch detection with higher ratio value can be obtained for RemoveDirt with

dein=RemoveDirt(input)
ImproveSceneSwitch(dein, dein)

instead of

dein=RemoveDirt(input)
ImproveSceneSwitch(dein, input)

as recommended in the documentation. However, for deinterlacing such a changed should not even be considered.

While the frame-doubling generate "clean" frames the extrapolation generate an "oversharp"-shadow frame. But as you are write at your description, the frame-double-method is bad, when an "fast-scene" is detected as "scene-change". I had seen such situation in my test-movie (with frame-doubling) and does not notice any bad artifacts (while see the movie at TV).

If two many frames are doubled, the motion may look jerky, while with extrapolation it should be smoother. I also noted the oversharpness of extrapolated frames.

So for now the extrapolation-method is not my preferntial because it generate an oversharp frames and this look in the movie like an artiface (but not realy noticeable)
I hope for further feedback. Extrapolation can be nicely implemented with SSE, that was the primary reason, why I did it. However, if it doesn't turn out to be useful, I will not hesitate to remove it.

I also observed that after encoding with ImproveSceneSwitch VdubMod is crashing upon exit. This is probably due to a missing emms instruction and will hopefully be corrected soon.

SILICON

19th April 2004, 22:24

I test the filter and i have a great result

The compresibility gain are very good in a good quality film (MINORITY_REPORT).

In films with spots, remove it.

Can you add a presets for noise and clean film?
Can you put ImproveSceneSwitch inside of RemoveDirt (as switch)?

Thank for you great pluging.

kassandro

21st April 2004, 01:46

Originally posted by SILICON
Can you add a presets for noise and clean film?

Thanks for your kind comments. Starting with version 0.6 there is a preset facility built into the plugin. However, you have to creat a the file RemoveDirt.ini yourself. Then you can simply use RemoveDirt(default="noise") or RemoveDirt(default="clean"), if you have created presets named "noise" and "clean".

Can you put ImproveSceneSwitch inside of RemoveDirt (as switch)?

I think that this would not be a very good idea. Firstly, for easy maintenance, it is better to keep different things separate. Secondly, the scene switch problem is a problem for most temporal filters. It is more severe for advanced deinterlacers than for RemoveDirt.

kassandro

21st April 2004, 02:06

I just put up version 0.6.1 to the web site. Besides correcting an emms problem for ImproveSceneSwitch, RemoveDirt has undergone substantial changes. Firstly I have removed the oscillation code. While theoretically nice the underlying idea has turned out to be poor in practice. Consequently, the variables doscillation and soscillation are now obsolete. I also lowered the default value of pthreshold from 50 to 20. If tolerance=0, then there should be no difference between version 0.6 and 0.6.1. If tolerance > 0 only blocks near the margins may be handled differently by the two versions. Of course, the oscillation code has to be disabled (as is the deefault) in version 0.6 for comparison with version 0.6.1. Version 0.6.1 should be slightly faster than version 0.6, however the speed increase should hardly be measurable.
Why these changes? They are in preparation of a major change in version 0.7. At least I hope so. By providing an alternative motion detection, I hope for cleaning in the presence of a certain type of motion. Most of the work is already in the source code, but I have first to check whether my idea really works in practice.

SILICON

21st April 2004, 18:38

One temporal filter, check the previous and the next frame (frames -1, 0 and 1).
If no motion for frame -1 to frame 1, you filtes think thae the motion in frame 0 are noise. Isn`t it?
In the scene switch, we don´t have previous frame, because the previous freme are very diferent and the temporal filter don´t work.

But the motion are in several frames, not only in two frames. In scene switch, you can check the two next frames. If one block don´t have motion for the frame 1 to 2, you can estimate that haven't motion for the frame 0 to Frame 1 (The change are noise, and not motion.)

If you are wrong, the artifact are the same that the actual SceneSwitch, but you cannot wrong for every blocks.

In resume, This metod will make lees artifact the the actual SceneSwitch. Can you test it?

kassandro

22nd April 2004, 01:45

Originally posted by SILICON
One temporal filter, check the previous and the next frame (frames -1, 0 and 1).
If no motion for frame -1 to frame 1, you filtes think thae the motion in frame 0 are noise. Isn`t it?

That is basicly correct. However, as motion may be restricted to parts of a frame, this strategy is carried out for small 8x8 pixel blocks and not for entire frames.

In the scene switch, we don´t have previous frame, because the previous freme are very diferent and the temporal filter don´t work.

That is exactly the problem.

But the motion are in several frames, not only in two frames. In scene switch, you can check the two next frames.

Earlier in this thread in response to a contribution by krieger2005 I proposed a similar solution to the scene switch problem. But then I realised that this was a broader problem for many temporal filters and I came up with the "cheating" idea underlying the filter ImproveSceneSwitch. If that filter is used conservatively, then the viewer can hardly detect the frame doubling, which clearly gives also the best compression results. In fact with frame doubling codecs can easier find the right I-frames.

In resume, This metod will make lees artifact the the actual SceneSwitch. Can you test it?
One cannot simply take the same code for -1,0,1 and use it for 0,1,2 or -2,-1,0 and each time I change the algorithm for 0,1,2 I would have to change not only the code for -1,0,-1 but also for 0,1,2. Thus I would not have only substantial work once but more work each time I make changes and I think that this is not worthwhile. Thus I will probably stick with ImproveSceneSwitch.

SILICON

24th April 2004, 14:51

Hi Kassandro,

One simplest filter proposal.

In very dark images we can see the details. You can test the lumma value of ane point and, if are below of one threshold, high blurring the lumma and croma whith the near points.

Make this, you can see any visual diference, but the compresor need less bitrate. Less bloks in dark images.

For example:

For (x=1; x<ImageHeihgt-1; x++) ; Not process the border lines
For (y=1; y<ImageHeihgt-1; y++)
If Luma(x,y) < LumaThreshold
; blur the luma
BlurLuma = Luma(x-1,y-1)+Luma(x-1,y)+Luma(x-1,y+1)
BlurLuma = BlurLuma + Luma( x,y-1)+Luma( x,y+1)
BlurLuma = BlurLuma + Luma(x+1,y-1)+Luma(x+1,y)+Luma(x+1,y+1)
Luma(x,y) = (Luma(x,y) + BlurLuma) /2
End
Next
Next

¿can you coded it?

kassandro

24th April 2004, 23:45

Originally posted by SILICON
Hi Kassandro,

One simplest filter proposal.

In very dark images we can see the details. You can test the lumma value of ane point and, if are below of one threshold, high blurring the lumma and croma whith the near points.

Make this, you can see any visual diference, but the compresor need less bitrate. Less bloks in dark images.

For example:

For (x=1; x<ImageHeihgt-1; x++) ; Not process the border lines
For (y=1; y<ImageHeihgt-1; y++)
If Luma(x,y) < LumaThreshold
; blur the luma
BlurLuma = Luma(x-1,y-1)+Luma(x-1,y)+Luma(x-1,y+1)
BlurLuma = BlurLuma + Luma( x,y-1)+Luma( x,y+1)
BlurLuma = BlurLuma + Luma(x+1,y-1)+Luma(x+1,y)+Luma(x+1,y+1)
Luma(x,y) = (Luma(x,y) + BlurLuma) /2
End
Next
Next

¿can you coded it?
This has nothing to do with RemoveDirt.
What you really want, is to blur dark areas, because you can't see details anyway in the darkness. However, the above meta code blurs all dark pixels and not only blurs dark areas. That's too much! For instance, if you have a thin black line on a white background only a slight shadows remains if the above filter is applied. You clearly don't want that. Consequently, one has also to check the neighboring pixels, which are used for blurring, whether they are below the threshold. Then such a filter makes sense and can be easily coded. However, it is not so easy to get a fast SSE based version. In addition one would like to blur also the chroma in dark areas and this is more complicated because the chroma is much thinner than the luma. Furthermore, a smart mixture of spatial and temporal rather than purely spatial blurring will give significantly better compression ratios. If more people find such a filter useful, I may think about making such a filter.

SoonUDie

25th April 2004, 01:49

You can already simulate such a filter directly in avisynth by masking the dark areas and overlaying a spatio-temporal softened version of the original image onto the unsoftened one. For example:

#--------------------------------------------
#Version NOT using Masktools (slower, YV12 only (because of deen, but you can change it) ):
#--------------------------------------------
function SmoothDarkAreas (clip "aClip")
{
mask = DarkMask(aClip)

#Specify your smoother here:
smooth = aClip.deen("a2d", 4, 6, 8)

return Overlay(aClip, smooth, mask=mask)

}

function DarkMask(clip "c", int "low", int "high", float "gamma")
{
low = default(low, 10)
high = default(high, 20)
gamma = default(gamma, 1)

return c.Sharpen(1).levels(low, gamma, high, 0, 255).invert().blur(0.3)
}

#--------------------------------------------
#Version using Masktools (faster, YV12 only):
#--------------------------------------------

function SmoothDarkAreasMT (clip "aClip")
{
mask = DarkMaskMT(aClip)

#Specify your smoother here:
smooth = aClip.deen("a2d", 4, 6, 8)

return MaskedMerge(aClip, smooth, mask, y=3,u=2,v=2, usemmx=true)

}

function DarkMaskMT (clip "c")
{
#fixed values of 10,20,1

return c.YV12Convolution(
\ "-1, 4, -1", "-1, 4, -1", automatic=true, usemmx=true, Y=3, U=1, V=1)
\ .YV12LUT("x 10 - 20 10 - / 1 1 / ^ 255 0 - * 0 +", "x", "x")
\ .invert().YV12Convolution("1 6 1", "1 6 1", automatic=true, Y=3, U=1, V=1)
}

Calling DarkMask() will give you a clip with spatially blurred shadows.

EDIT: changed to functions, uploaded masktools optimized version.

SoonUDie

25th April 2004, 14:29

Another filter: Filter both dark AND bright areas.

Masktools only, at the moment (normal Overlay isn't letting me add the light and dark masks for some reason)

function SmoothLightAndDarkMT (clip "c")
{
cSharp = c.YV12Convolution(
\ "-1, 4, -1", "-1, 4, -1", automatic=true, usemmx=true, Y=3, U=1, V=1)

dark = cSharp.YV12LUT("x 8 - 14 8 - / 1 1 / ^ 255 0 - * 0 +", "x", "x")

light = cSharp.Levels(240, 1, 255, 0, 255)
#can someone translate this to YV12LUT format for me?

LDmask = YV12Layer(Dark, Light, "add", chroma=false, Y=3, u=1, v=1)
\ .invert().YV12Convolution("1 6 1", "1 6 1", automatic=true, Y=3, U=1, V=1)

#--------------------------------------
# Specify your smoother here:
#--------------------------------------
smooth = c.deen("a2d", 4, 6, 8)

return MaskedMerge(c, smooth, LDmask, y=3,u=2,v=2, usemmx=true)

}

cwolf

25th April 2004, 18:24

The range file is a problem in VirtualDub-MPEG2. I don't know how to workaround it.
SetMemoryMax (200)
input= avisource("f:\falcao maltes.avi")
removedirt (input, pthreshold=30, grey=true, mthreshold=175, athreshold=75,
\ range1="intro.rmd", mthreshold1=500, athreshold1=500)
#removedirt(input,neighbour=firstpass)
#ImproveSceneSwitch(cleaned,input)
-----------------------------
This script loads the movie into VirtualDub, but when reaches the first frame on "intro.rdm" it fails with "Avisynth read error: Avisynth: caught an access violation at 0x00a5aa1, attempting to read from 0x00000000". If I take the range out it works. I put 10-270 and the error happened when reached frame 10 in "preview input".
I searched DOOM9 and found SETMEMORYMAX in a thread to solve this error, but didn't work. Tried the official VirtualDub, too.
I have WINDOWS XP HOME SP1 and Avisynth 2.5.5.

Do you know what could it be?

Thanks in advance.
Cecilia
P.S. could it be related to DLLs? I had trouble yesterday trying to install DUSTV5 for the first time, it took me hours to find out what was going on (putting plugins in different directories, MSVCP71.DLL, etc. What a pain in the *ss :mad: )

kassandro

25th April 2004, 19:07

As you describe it, it sounds very much like a RemoveDirt bug. On the other hand, the following script

SetMemoryMax (200)
input= avisource("Elke.avi")
removedirt (input, pthreshold=30, grey=true, mthreshold=175, athreshold=75,range1="intro.rmd", mthreshold1=500, athreshold1=500)

caused no problems with vdubmod. Of course I had to take a different avi. Is your problem dependent on the avi file?

killerhis

4th May 2004, 11:23

Hi guys.. I've been following this filter for a while.. keeps getting beter every time :)

I got a small issue, maybe you guys can give me some advice. I've been trying to get a clean motion detection, even during high motion. I got here a anime source (funanimation dbz) which has the famous white cut edges on the top/bottom during a scenechange. Unfortinally, it has also some fast motion, which Removedirt also sees as a scenechange, generating a strobeeffect motion.

Are there any settings you guys advice me that I can try/use? I know its impossible that I cant get a perfect scene detection, but how better the detection, how happier I am :). The other highmotion parts I could fix using trim :D

With regards

KillerHIS

cwolf

4th May 2004, 16:46

It seems that using rangefile in VDUBMOD is ok, but I had to fight with it about the pause/play of preview, I'm used to the space bar in VirtualDub but it doesn't work the same way in VDUBMOD. Another issue is the .VCF file with my editings, if I don't save it before refreshing (PF5) I lose them (the reload discards all the editings). I posted in Everwicked forum about it, don't know if somebody replyed.
Another problem is the huge number of frames with rainbows in my capture (dirty VHS): I think every 2 or 3 in 5 have rainbows. Removedirt does an excellent job in removing them, with the default settings it removes the rainbows in the areas where there's no movement. But the eyes and mouths of the persons keep the rainbow, which gives an awfull look to them (kind of they're going to a halloween party :p ). I don't know whether I increase the settings or make a range file with these hundreds (thousands?) of frames (what a work!). I'll find a balance between them, but I thought it could be improved in Removedirt (giving it more intelligence), as these rainbows are a specific kind of dirt, they do have a pattern. They are always stripes of violet, blue, green or yellow color, although may vary in height. Kassandro, do you know them?

kassandro

5th May 2004, 01:03

Originally posted by killerhis
I got a small issue, maybe you guys can give me some advice. I've been trying to get a clean motion detection, even during high motion. I got here a anime source (funanimation dbz) which has the famous white cut edges on the top/bottom during a scenechange.

From an entertainment perspextive I am not interested in anime films. However, I am interested in anime specific problems as far as digital video processing is concerned. It would be nice if you could provide a three frame clip of such a scene change with these "white cut edges on the top/bottom". Instead of using Huffyuv I recommend to use direct stream copy mode in vdubmod with winrar compression afterwards. This creates 10-20% smaller files and avoids any color space conversion. The frame, which RemoveDirt should handle should be in the middle, i.e. the second frame.

Unfortinally, it has also some fast motion, which Removedirt also sees as a scenechange, generating a strobeeffect motion.

As you said scene switch detection can't be perfect for the time being. Even sophisticated codecs, which make a much more thorough motion analysis than ImproveSceneSwitch, have problems to choose the right frames as I-frames. On the other, ImproveSceneSwitch is smarter than simply diagnosing a scene switch, if the difference between two frame exceedes a certain threshold. If the differences of the frames before are similarily large - this is usually the case with fast motion - then it does not diagnose a scene switch. Unfortunately even the differences may oscillate strongly during some kind of motion and then ImproveSceneSwitch falsely detects scene switches. I have two ideas for further improvement without sacrificinf too much speed. Firstly I compute the average of the luma and the two chroma planes. During motion these averages shouldn't change much less than during a scene switch. Secondly, if ImproveSceneSwitch detects one scene switch very shortly after another, it should discard the second scene switch.

kassandro

5th May 2004, 01:25

Originally posted by cwolf
Another problem is the huge number of frames with rainbows in my capture (dirty VHS): I think every 2 or 3 in 5 have rainbows.

While I know the rainbow problem from my own VHS past, I can't remember it as so prevalent. I have heard about rainbow filters/scripts in this forum, which you might try. Since about two years I record the digital DVB stream directly. The quality is much higher in any respect and rainbows are quite rare and small. hence I do not care much about it.

Removedirt does an excellent job in removing them, with the default settings it removes the rainbows in the areas where there's no movement. But the eyes and mouths of the persons keep the rainbow, which gives an awfull look to them (kind of they're going to a halloween party :p ). I don't know whether I increase the settings or make a range file with these hundreds (thousands?) of frames (what a work!). I'll find a balance between them, but I thought it could be improved in Removedirt (giving it more intelligence), as these rainbows are a specific kind of dirt, they do have a pattern. They are always stripes of violet, blue, green or yellow color, although may vary in height. Kassandro, do you know them?
I am not very optimistic about solving this problem, but if you provide me with a typical three frame clip (as in the previous posting) I will certainly make a look at it. Rainbows are obviously an RGB problem, while color tv works with YUV (in fact YUV was invented to make color tv backward compatible with b&w tv). That makes rainbows a bit of a mystery for me, because YUV perturbations should not creat RGB typical effects.

Boulder

5th May 2004, 10:25

This thread has some ideas for rainbow removal (sh0dan's posts):

http://forum.doom9.org/showthread.php?s=&threadid=62873

There seems to be a small error in the docs supplied with v0.61..it says that for B&W clips "...there cannot be any chroma postprocessing, pthreshold should be somewhat lower, say 30 or 40."

The default is 20 so what should the value be for B&W clips?

Thank you for a great filter, it's a must in my scripts!

kassandro

6th May 2004, 00:39

Originally posted by Boulder
There seems to be a small error in the docs supplied with v0.61..it says that for B&W clips "...there cannot be any chroma postprocessing, pthreshold should be somewhat lower, say 30 or 40."

The default is 20 so what should the value be for B&W clips?

Thank you for pointing out this incorrectness. Initially the default value for pthreshold was 50. That lead to a slight blockiness in areas with flat contrast and I lowered the default value to 20. For black&white I am using now 10.

audioslave

6th May 2004, 02:20

@kassandro
Seems like a very interesting filter, indeed :) !
Sorry for this newbie question, but what settings would you recommend for a scene like this:

http://hea.port5.com/TCM1.jpg

:confused:

It's from the first minute of "The Texas Chainsaw Massacre". I've tried most settings (I could think of) but I still haven't been able to get the scene right... The scene looks awful encoded as it is now. Almost like Lego! :D

Thanks! ;)

kassandro

6th May 2004, 08:38

Originally posted by audioslave
@kassandro
Seems like a very interesting filter, indeed :) !
Sorry for this newbie question, but what settings would you recommend for a scene like this:

picture as above

:confused:

It's from the first minute of "The Texas Chainsaw Massacre". I've tried most settings (I could think of) but I still haven't been able to get the scene right... The scene looks awful encoded as it is now. Almost like Lego! :D

Thanks! ;)

Use RemoveDirt(tolerance=100) to check the maximal amount of cleaning possible with RemoveDirt. Of course, you cannot process an entire clip this way, because of massive artifacts in motion scenes.
To judge a specific frame, I not only need the frame itself, but also the predeccor and the successor frames. Ideally you mark the three frames in vdubmod and save them uncompressed to an avi with Directstreamcopy. If the 3 frame avi is compressed further with winrar it is usually smaller than a Huffyuv avi and it doesn't suffer from a color space conversion to yuy2.

SILICON

10th May 2004, 15:37

@Kasandro

You can use a simple motion search for increase your filter performance.

The most common motion are left to right (or right to left).

You can search the blocks as is:

1 - Search the block in the next frame in the same place
2 - If don´t locate in step 1, search the block to left and rigth.
3 - If don´t locate, process the next block in step 1
4 - If locate in step 2, save the motion found
5 - For all blocks in this frame, search only with motion of step 4 and with no motion.

The motion is usally place in several frames. You can begin the search in one fame with the motion of the last frame.

I think that make this search, take a small amount of time.

kassandro

10th May 2004, 22:54

Originally posted by SILICON

The most common motion are left to right (or right to left).

I agree.

1 - Search the block in the next frame in the same place
2 - If don´t locate in step 1, search the block to left and rigth.
3 - If don´t locate, process the next block in step 1
4 - If locate in step 2, save the motion found
5 - For all blocks in this frame, search only with motion of step 4 and with no motion.

With this kind of motion compensation, one can only capture global motion to the left or right by 8 pixels (the width of a block) and there is no reason why 8 pixel motion should be more likely than 6 or 3 pixel motion. On the other hand, XviD can do (I haven't looked at it) a motion analysis on a quarter pixel basis. Thus, with a crude 8 pixel motion analysis I can capture only a small fraction of horizontal motion.

I think that make this search, take a small amount of time.
As described the speed penalty is very limited but the possible gain is even more limited. There is another problem: currently RemoveDirt matches blocks in the previous frame with blocks in the subsequent frame. The frame to be cleaned, which is just in the middle, is ignored as far as motion analysis is concerned (the reason is explained in the RemoveDirt documentation). Now if a can match blocks with different spatial location, then these blocks can only be used for cleaning a block, whose spatial location is just the arithmetic mean of the spatial location of the two matching blocks and such a location may very well be outside the current 8 pixel grid causing a whole punch of other problems.

BBugsBunny

23rd May 2004, 19:51

First of all - kassandro thank you very much for that filter. It can enhance quite a lot some captures of mine from analogue source.

I've been testing the filter a little bit and found a scene where the filter has got some problems - or I simply can't find good settings for it - I've already been playing around with the settings a bit.

In the described scene on the bottom left brightness is going up and down. On some frames dark blocks appear that are visible quite well.

So my question is if there are settings that still remove dirt and don't let these artefacts appear, or is this not possible with this filter?

I've uploaded a small sample clip, which is part of a longer scene that shows the artefacts a few times.

http://members.chello.at/nagiller/test/test.avi

This is basically my script:
AVISource("test.avi")
dein=RemoveDirt(dist=1,mthreshold=150,athreshold=50,pthreshold=20,tolerance=12,mode=2)
ImproveSceneSwitch(dein, dein)

kassandro

23rd May 2004, 21:32

Originally posted by BBugsBunny
I've uploaded a small sample clip, which is part of a longer scene that shows the artefacts a few times.

What you have uploaded, is this the original clip or is this the output after applying RemoveDirt? I really hope, that this messy clip is not a result of RemoveDirt. In general, RemoveDirt can only clean a spot on a frame, if it is correct on the previous and the subsequent frame and if there is no motion. This implies that the previous and the subsequent frame do not deviate much from each other. In your clip, the frames don't fit together at all. It is obviously interlaced and the changes are so fast that not even the two fields of a frame fit together. This you can see by separating the fields with

avisource("test.avi")
SeparateFields()

I am sorry, but for this clip RemoveDirt can't do anything good, but with your rather standard setting it doesn't cause destruction either. On the other hand, ImproveSceneSwitch may cause problems here, because the clip is so discontinuous such that ImproveSceneSwitch may regard any frame of this clip as a scene switch.

BBugsBunny

23rd May 2004, 22:39

Thanks kassandro for having a look at my file.
Actually the source file was a DV file that I captured from TV with my Hollywood DV Bridge.
I then deinterlaced it with TomsMoComp (0,5,1) and stored it as a Huffyuv 2.2.0 avi. That's what I actually uploaded.
Now I've uploaded/replaced it with the original DV file. (In case anyone still wants to give it a try)

The problem as well could be, that the original is a japanese anime that was converted to pal, so even with deinterlacing some artefacts stay. I've tried inverse telecine and restore 24 fps (that got me an error in avisynth) but I really don't know how and if the original frames can be restored from this pal capture.
If that's be possible perhaps I would get a better result since you say that the source has to be 100% progressive...

But since I do not really know where to start with that problem I think I'll stay with my wavelet/temporal/2d cleaning in VirtualDub although the results are not as good as with your filter in general, but on the other hand the dark blocky artefacts don't appear either.

kassandro

23rd May 2004, 23:22

Hi BBugsBunny,
I have now downloaded the bigger original clip. Unfortunately
it uses codec with fourcc "dvsd", which I do not know, whence
it is rejected by avisource. It is also rejected by vdubmod if
I load your avi directly. On the other hand, mpc plays the clip
just fine. It now looks a lot better. I can't believe that
tomsmocomp, which is a very fine filter, can creat such a garbage
from it, which I have seen in your first clip. However, to analyse
the clip I have to load it with avisource. So please tell me, which
kind of codec I need to accomplish this. I thought that only huffyuv
and mjpeg (bad for non-progressive material) is used for analogous
capturing and I have installed both with ffdshow.

Leak

23rd May 2004, 23:44

Originally posted by kassandro
However, to analyse the clip I have to load it with avisource. So please tell me, which kind of codec I need to accomplish this. I thought that only huffyuv and mjpeg (bad for non-progressive material) is used for analogous capturing and I have installed both with ffdshow.

Actually, you should be able to use any recent version of ffdshow to open it via AVISource; just start ffdshow's "VFW configuration" from it's start menu folder and set "DV" to libavcodec on the Codec page in the decoder tab.

np: Autechre - IV VV IV VV VIII (Draft 7.30)

kassandro

23rd May 2004, 23:51

Originally posted by Leak
Actually, you should be able to use any recent version of ffdshow to open it via AVISource; just start ffdshow's "VFW configuration" from it's start menu folder and set "DV" to libavcodec on the Codec page in the decoder tab.

I have done this, but no success!

Leak

24th May 2004, 00:52

Originally posted by kassandro
I have done this, but no success!

Strange - it works for me, now that I've downloaded all of the clip.

Have you tried configuring ffdshow's VFW codec using VirtualDub's compression config dialog? Is the codec ("ffdshow Video Codec") even listed there?

If it isn't, try reinstalling ffdshow to a path that contains no spaces, that should fix it...

np: Autechre - Lowride (Incunabula)

Leak

24th May 2004, 01:00

Originally posted by BBugsBunny
In the described scene on the bottom left brightness is going up and down. On some frames dark blocks appear that are visible quite well.

Ummm... exactly where are the dark blocks supposed to appear? Decoding the file with ffdshow I can't seem to find any...

If you want I can extract the same scene from my US-DVDs of Hellsing for comparison - just not now; it's waaay too late already... :)

np: Lamb - Trans Fatty Acid (Lamb)

kassandro

24th May 2004, 12:49

Originally posted by Leak
Strange - it works for me, now that I've downloaded all of the clip.

Have you tried configuring ffdshow's VFW codec using VirtualDub's compression config dialog? Is the codec ("ffdshow Video Codec") even listed there?

No it isn't! But as ffdshow can't encode, it doesn't make sense to list ffdshow in the compression dialog. Actually in my case ffdshow was installed by the Matroska all in one package (and is now in a Matroska subdir). While all directshow apps find the codec, vfw apps like vdub, avisource or avicodec don't. Instead of messing around with a new installation of ffdshow, I now load the avi with DirectShowSource and it now works.
Now back to BBugsBunny's clip. As BBugsBunny already said, the light eminating from the bottom left corner is the critical part of the clip. It changes with every field rather than with every frame. Hence a deinterlacer is necessary. However, Tomsmocomp(0,5,1) does an excellent job here and like Leak I can't find any of this black blocks created by RemoveDirt (you may also lower pthreshold slightly).
Obviously something else must have gone wrong the clip, which I downloaded first, because it was really pure garbage. RemoveDirt removes quite a bit of noise of this clip, even after deinterlacing. What may be irritating though, is, that in the bottom left corner sometimes the noise is cleaned and sometimes not depending how fast the light goes up and down.

Leak

24th May 2004, 12:57

Originally posted by kassandro
No it isn't! But as ffdshow can't encode, it doesn't make sense to list ffdshow in the compression dialog.

Oh, ffdshow *definitely* can encode, as it's been merged with ffvfw some months back.

Of course, that would mean you have to install a more recent version of it (http://athos.leffe.dnsalias.com/) - the latest version of ffdshow is 4 days old. :)

np: Lamb - Just Is (What Sound)

FredThompson

24th May 2004, 13:04

Originally posted by kassandro
As BBugsBunny already said, the light eminating from the bottom left corner is the critical part of the clip. It changes with every field rather than with every frame. Hence a deinterlacer is necessary.Deinterlace or decomb? Deinterlace will give you trash. If you must operate on individual fields, split into 2 streams, filter each, then recombine. Alternately, you could make a 2x frame rate version and clean it.

kassandro

24th May 2004, 19:53

Thank you, Leak for your informations and the link. It is good to know that ffdshow is continued. My version is obvious the most recent from the sourceforge web site, which is quite old.

Originally posted by FredThompson
Deinterlace or decomb? Deinterlace will give you trash. If you must operate on individual fields, split into 2 streams, filter each, then recombine. Alternately, you could make a 2x frame rate version and clean it.
I think the output looks quite good and wouldn't call it trash. If you only want to clean, then at your advice to separate the fields and clean them separately is optimal. However, there is the slight danger, that a piece of dirt is cleaned only on one field. But if you deinterlace afterwards - and BBugsBunny's clip should be deinterlaced - then this problem is taken care of as well. Thus the best strategy for denoising interlaced material should be : separate fields, denoise them separately, weave them and finally deinterlace them. I will add this to the documentation. Thank you, Fred.

FredThompson

24th May 2004, 20:18

My reply was too short. I should have explained better.

If the "true" source has a different frame rate than the interlaced version, filtering will probably yield better results by decombing to recover those original progressive frames.

Deinterlacing doesn't always do this properly. It can result in some strange artifacts.

If the "true" source is interlaced, splitting and filtering would be the "best" idea because it's really 2 different streams of images which are interleaved.

Yes, you are correct. Splitting and filtering could possibly result in some dirt being missed by the filter. Maybe that seems worse than the actual results because this should only happen in single frames, right?

kassandro

24th May 2004, 20:57

Originally posted by FredThompson
If the "true" source has a different frame rate than the interlaced version, filtering will probably yield better results by decombing to recover those original progressive frames.

Deinterlacing doesn't always do this properly. It can result in some strange artifacts.

I agree with you. If there is telecining or even field blending going on, especially good deinterlacers like tomsmocomp can creat strange artifacts. In this special clip, however, there is so little motion, the man with the child is walking very slowly, hence the blinking light is the only motion, that you really can't tell, what's going on. On the other hand, for anime videos true interlacing doesn't really make sense, but I am not an anime expert.

Yes, you are correct. Splitting and filtering could possibly result in some dirt being missed by the filter. Maybe that seems worse than the actual results because this should only happen in single frames, right?

I think, that dirt detection should be better with split filtering, if the input clip is not truely progressive, but it is negative for compression and looks bad, if a dirty spot is only removed on one fields (i.e. you end up with a combed stain).

krieger2005

31st May 2004, 14:43

Hi,

i use your filter often but one thing is very bad:
Your filter works fine when you look on one single pic. It is clean.

But in motion it produce a "mask"-Like effect. I had the same effect when i'm use "Blockbuster" with method="dither". It is like an overlay obeout some frames.
This is the result of the definition of threshold (and this is the user's fault), BUT: If a movie have something like stones in the picture, where the color and luma is quite identical and the camera move, than you need a very low threshold. But with a low threshold one does not get the result, which the filter should bring.

I know that there is a tradeoff: Do you want to remove the noise and dirt from the picture (and maybe get a "washed" picture) or you let the noise.

Another bad effect is, if there is an area, quite same color, moving camera. Becuase your filter operate on Blocks (8x8 i think) one block were cleaned (let's say) 5 frames and another 4 frames. But because the color is only quite the same one overstep the threshold in frames 150-155 and another 153-157. In this case the picture get blocky.

I know... This is tuning of the threshold... But as every know, when you tune some thresholds for a scene you get bad results on another and so on.

I think there are only two solutions how to solve such problems:
- A Buildin Anti-Wash/Anti-Blocky filter: If the source is not blocky between two blocks the result should it be too
- Cleaning with motion compensation

It's a hard subject matter and i don't know how to solve it at all. At the moment i use your filter with use of masks (motion-masks).

kassandro

31st May 2004, 19:36

Originally posted by krieger2005
i use your filter often but one thing is very bad:
Your filter works fine when you look on one single pic. It is clean.

But in motion it produce a "mask"-Like effect. I had the same effect when i'm use "Blockbuster" with method="dither". It is like an overlay obeout some frames.
This is the result of the definition of threshold (and this is the user's fault), BUT: If a movie have something like stones in the picture, where the color and luma is quite identical and the camera move, than you need a very low threshold. But with a low threshold one does not get the result, which the filter should bring.

If mthreshold and athreshold are too high, then even single frames should look bad. The question is: can something go wrong with the video if single frames look fine. Sometimes, I think so, but then motion often looks unnatural with 25 fps and theoretically it doesn't make much sense, because RemoveDirt either leaves a pixel unchanged or uses the same pixel from the previous or the subsequent frame.

I know that there is a tradeoff: Do you want to remove the noise and dirt from the picture (and maybe get a "washed" picture) or you let the noise.

As RemoveDirt doesn't blur, "washed" is probably not the right work. If it would only be looked washed, then one could choose more aggressive thresholds. The artifacts can be significantly more terrible. For instance, try tolerance=100 in the presence of moving fine objects.

Another bad effect is, if there is an area, quite same color, moving camera. Becuase your filter operate on Blocks (8x8 i think) one block were cleaned (let's say) 5 frames and another 4 frames. But because the color is only quite the same one overstep the threshold in frames 150-155 and another 153-157. In this case the picture get blocky.

blockiness should not occure, if pthreshold is sufficiently small. I gradually reduced pthreshold down to 10 and I can't see any blockiness left. Lowering pthreshold has not such a negative effect as lowering mthreshold (I usually set athreshold=300 in order to ignore the athreshold variable). In fact the basic philosophy of RemoveDirt is, that only one block of a moving area has to be caught and the rest is taken care by postprocessing, which strongly depends on pthreshold. So you can have a lot of cleaning with pthreshold=10.

- A Buildin Anti-Wash/Anti-Blocky filter: If the source is not blocky between two blocks the result should it be too

I think that I have exactly done this, but I only look for oscillation between cleaned and uncleaned blocks. Thus I do not look for oscillation between cleaned blocks and it would not be correct to do so. For instance, if you have a white spot overlapping two blocks, then if both these blocks are cleaned the oscillation would probably increase substantially, but I should certainly not reject the cleaning. Only the oscillation between cleaned and uncleaned blocks is critical, because uncleaned blocks indicate motion and if the oscillation between and uncleaned and a cleaned block is higher than the oscillation between the original blocks, the guess is that the motion extends to the cleaned block and the cleaning of this block will be undone by postprocessing. Now the formerly cleaned block becomes an uncleaned block and its cleaned neighbours are again checked, whether cleaning has raised oscillation. That's how RemoveDirt works.

- Cleaning with motion compensation

If motion compensation would only be nearly as good as some claim it to be! As far as I know motion prediction only works well for codecs and why does it work well for codecs? A codec proceeds roughly as follows. For a block in the current frame it makes a motion prediction from the previous frame. It looks at the difference between the original block and the predicted block, does a DCT and drops the high frequency part(depending on the quantiser) from the difference of the two blocks. If the motion prediction was good enough, then very little low frequency stuff is left and compression is very effective. If the motion prediction was bad, then substantial low frequency stuff is left and more bytes are needed for compression, but a bad prediction has no direct effect on visual quality. The situation is very different, if you use motion prediction for other tasks. The most difficult task is frame interpolation, i.e. you want to creat a new frame between two old frames without blending. That blending is still so heavily used, shows that frame interpolation doesn't work well despite some published claims by Tourapis and collaborators. Now for RemoveDirt the situation is more hopeful: One looks for motion vectors between the previous and the subsequent frame. Cleaning will only be done if the prediction is sufficently good (in the case of frame interpolation you always must do something, even if motion prediction is very bad) and postprocessing will also greeatly help to remove artifacts created by false motion prediction. If cleaned blocks do not fit to uncleaned blocks, cleaning must be undone (you can't do postprocessing for frame interpolation either). I am thinking about motion compensation for RemoveDirt, but RemoveDirt will slow down a lot if this feature is implemented and activated.

mkanel

14th June 2004, 17:18

Thanks kassandro for the great filter, it works really well for my captured analog tapes. Also thanks for the great documentation, it really helps when experimenting with your filter.

In your documentation you ask for feedback on extrapolation in ImproveSceneSwitch. I tried Extrapolate=true and for what it's worth I get much better results with the default value of False the extrapolated frame appeared oversharp, it was much more noticeably wrong than a duplicate frame.

The default cleaning mode 2 also works best for me, it provides for a much smoother playback than modes 0 or 1.

When I capture from an analog source I get a number of duplicate frames. By design removedirt does no cleaning on these frames. Would there be any way around this that wouldn't cost too much processing time? I have two ideas, I don't know if either is practical.

1) If remove dirt itself could recognize frame duplication it could compare the duplicates to the frames before and after the duplication and perform cleaning on the duplicates based on those frames.

2) Maybe it would be easier for ImproveSceneSwitch to detect duplicates while it searches for scene changes and then perform a similar substitution of frames so that duplicate1 would be replaced by duplicate1-1 and duplicate2 would be replaced by duplicate2+1.

3) Maybe I just need to keep tweaking to try to banish dropped/added frames from my captures.

Thanks. Mike.

kassandro

14th June 2004, 21:10

Thanks for your friendly comment and your interesting suggestions.

Originally posted by mkanel

I tried Extrapolate=true and for what it's worth I get much better results with the default value of False the extrapolated frame appeared oversharp, it was much more noticeably wrong than a duplicate frame.

I came to the same conclusion and don't use it anymore. In fact, in the 0.6.1 there was the possibility to compile a filter which did extrapolate any frame except the first two. Using this filter I then watched clips in vdub and quite a few frames were truely bad. It was an interesting SSE excercise, which, however, turned out to be not useful. The extrapolate version will be phased out, starting with version 0.7.1. First the variable the variable will simply be ignored and later it will be completely removed, just like the doscillation and soscillation variables in RemoveDirt.

The default cleaning mode 2 also works best for me, it provides for a much smoother playback than modes 0 or 1.

Again I agree with you. The mode variable will be phased out and mode=2 will be the sole mode.

When I capture from an analog source I get a number of duplicate frames. By design removedirt does no cleaning on these frames. Would there be any way around this that wouldn't cost too much processing time? I have two ideas, I don't know if either is practical.

1) If remove dirt itself could recognize frame duplication it could compare the duplicates to the frames before and after the duplication and perform cleaning on the duplicates based on those frames.

2) Maybe it would be easier for ImproveSceneSwitch to detect duplicates while it searches for scene changes and then perform a similar substitution of frames so that duplicate1 would be replaced by duplicate1-1 and duplicate2 would be replaced by duplicate2+1.

You have a very good feeling, how things should be implemented in a most efficient way. Indeed the low level frame comparison routine for detecting scene switches is exactly the same as the one for detecting duplicates and I spend some time to avoid that these costly comparisons are not repeated unnecessarily and of course it would be more than smart, to use the output of this routine not only for scene switch detection but also for duplicates. There is, however, a severe problem to do it this way, because in the standard avisynth design a filter can only control the input or the output of another filter, but not both. Currently ImproveSceneSwitch controls the output of RemoveDirt: at a scene switch the frames are duplicated rather than passed through to RemoveDirt. On the other hand, the duplicate problem arises primarily on the input level of RemoveDirt: RemoveDirt should simply not receive duplicates for cleaning. Of course later on duplicated input frames must be duplicated again after cleaning. One needs a filter which completely controls another filter. I think such are already done with conditional filtering.
The same duplicate frame problems arises in deinterlacing and it is even more severe. Deinterlacers which adhere to my "no motion no change" simply leave such frames unchanged, which is of course very bad.
Currently, I am very busy with version 0.7, which will see major changes in many ways. It should be released by the end of the onth or eary July and then I will have more time for such a much more sophisticated ImproveSceneSwitch. Not only analog source, but also dv cam source and anime have a duplicate frame problem. Thus it is really worthwhile to deal with it in such a optimal way, if it is really possible.

mkanel

19th June 2004, 04:35

kassandro,

Thanks for bringing up conditional filtering. It was a good reason to try using this feature of Avisynth. I made a working script that takes care of cleaning duplicate frames. It has a couple of restrictions.

It won't work if the duplicate frames are at a scene change.

It doesn't work if you have three or more identical scenes, although you could tweak this script to handle that.

This is meant to find and clean truly identical scenes, that kind that a capture program might insert into an avi. The only time the non-cleaned duplicates stand out is during low movement scenes, this will handle those situations.

#Clean Duplicate Frames

#YV12 needed for frame comparison, add one dup to test, two dups results in no cleaning
dirty=AVISource("D:\captured.avi").ConvertToYv12 #.duplicateframe(50) #.duplicateframe(50)
cleaned=removedirt(dirty) #make sure setting match removedirt below

CleanDup= dirty.ScriptClip("dirty.DeleteFrame(current_frame).removedirt")
CopyCleanDup= dirty.SCriptClip("CleanDup.DuplicateFrame(current_frame-1)")

#If previous/next frames not duplicates uses cleaned frame, otherwise frame is passed to CopyCleanDup
#I don't know what YDifference is but it's always .000003 on my duplicate frames. Try true
IsSecondDup=ConditionalFilter(dirty,CopyCleanDup,cleaned,"YDifferencefromprevious()","<",".000005" ,false)

#following checks if next frame is a duplicate, if yes first dup is cleaned, otherwise
#IsSecondDup (line above) checks if the previous frame is a duplicate. Try true to see more info
ConditionalFilter(dirty,CleanDup,IsSecondDup,"YDifferencetonext()","<",".000005",false)

#Uncomment following line from Avisynth documentation to see YDifference info.
#ScriptClip(dirty, "Subtitle(String(YDifferenceFromPrevious))")
converttoyuy2

Obviously you have to put in the proper path and file name for your avi. RemoveDirt appears twice in the script, make sure you put your desired parameters in both places.
You can uncomment the "#.duplicateframe(50)" to test it. Or uncomment the line near the bottom ScriptClip(dirty, "Subtitle(String(YDifferenceFromPrevious))") to make sure the test conditions "<.000005" are right for your avi.

If this is useful to anyone maybe someone could design a cleaner .avsi

Mike.

kassandro

20th June 2004, 11:52

Yes, your script does the job. I am surprised that your script is reasonably fast, because ScriptClip must generate a clip for each frame (of course this is only done for duplicate frames). Here is a script based on the same idea, which uses ScriptClip only once

dirty=mpeg2source("C:\video\input.d2v")
normal_clean=RemoveDirt(dirty)
ScriptClip(dirty, "DeleteFrame(current_frame).RemoveDirt()")
ConditionalFilter(dirty,last,normal_clean,"YDifferencetonext()","<",".000005",false)
#take the previous frame of last clip, if the current frame of the dirty clip is a duplicate of the previous one
ConditionalFilter(dirty,last.DuplicateFrame(0),last,"YDifferencefromprevious()","<",".000005" ,false)

It would be relatively easy to extend ImproveSceneSwitch such that it takes care of duplicate frames, if I would add a third clip argument, to which the user has to supply the clip
ScriptClip(dirty, "DeleteFrame(current_frame).RemoveDirt()")
Multiple difference computations could then be avoided (in the above script, the frame difference is computed twice unless Avisynth is extremely smart). However, I would like to avoid ScriptClip, which will probably get costly if there are many duplicate frames.

ScriptClip is really an awfully inefficient "filter". Each time a frame is called from a ScriptClip clip it has to creat and destroy the filters in the ScriptClip script and I haven't optimised at all RemoveDirt for fast initialisation. If RemoveDirt would have a memory leak, the sytem memory would quickly be exhausted and the sript would crash. Thus the above two scripts are a good robustness test for RemoveDirt, but the problem should really be solved without ScriptClip, which should only be used in an emergency.

mkanel

21st June 2004, 09:06

kassandro,

Thanks for the cleaner script. It is faster than the one I was using. If I understand correctly the only way to avoid using ScriptClip is to have the frame duplication detection directly inside RemoveDirt, which would probably lead to ImproveSceneSwitch moving inside RD as well which I think is not what you had in mind for the use of these filters. I don't know how many RD users care about frame duplication cleaning. If you implement a duplicate frame cleaning option in ISS or RD itself I'll gladly use it. If not I'm happy using the script you posted.

Thanks. Mike.

kassandro

21st June 2004, 22:23

Mike,
Duplicate frames are a problem for most temporal filters. Thus one should not go for a special solution inside RemoveDirt. I think that this can be done efficiently, but at the moment I have no time to get concret with it. I am currently finishing version 0.7 of RemoveDirt. There will be a lot of changes with respect to version 0.6.1, especially for the yuy2 color space. There will now also be chroma post processing for this color space, which is used for analog capture. Consequently, blockiness should then be gone also for this color space.

cwolf

22nd July 2004, 06:57

I'm using three range files and would like to include a fourth file, but avisynth says "open failure: script error: the named argument "pthreshold4" was passed more than once to removedirt".
Here's my script:

mpeg2source("f:\dali\dali.d2v").killaudio().freezeframe(2106,2106,2105)
assumetff()
separatefields()
#showframenumber(scroll=false)
f3of3 = selectevery(3,2)
selectevery(3,0,1).weave().assumeframebased().assumetff()
f12of3=last
cleanneig=removedirt(f12of3,pthreshold =75, tolerance =18,athreshold =80, mthreshold =155,
\ range1="f:\dali\rddirty.rmd",pthreshold1=200, tolerance1=45,athreshold1=260,mthreshold1=280,
\ range2="f:\dali\rddontclean.rmd", pthreshold2=25, tolerance2=12,athreshold2=40, mthreshold2=48, cthreshold2=22,
\ range3="f:\dali\rdlessmove.rmd",pthreshold3=160,tolerance3=33,athreshold3=155,mthreshold3=177,
\ range4="f:\dali\rddontclean2.rmd", pthreshold4=25, tolerance4=12,athreshold4=40, mthreshold4=100)

removedirt(f12of3,neighbour=cleanneig, pthreshold=75, tolerance =18,athreshold =80, mthreshold =155,
\ range1="f:\dali\rddirty.rmd",pthreshold1=200, tolerance1=45,athreshold1=260,mthreshold1=280,
\ range2="f:\dali\rddontclean.rmd", pthreshold2=25, tolerance2=12,athreshold2=40, mthreshold2=48, cthreshold2=22,
\ range3="f:\dali\rdlessmove.rmd",pthreshold3=160,tolerance3=33,athreshold3=155,mthreshold3=177,
\ range4="f:\dali\rddontclean2.rmd", pthreshold4=25, tolerance4=12,athreshold4=40, mthreshold4=100)
clean=last
conditionalfilter(clean,clean.swapfields(),clean,"current_frame%2","=","1")
separatefields(last)
interleave(selecteven(),selectodd(),f3of3)
weave()
assumeframebased()
conditionalfilter(last,last.swapfields(),last,"current_frame%3","=","2")

It looks odd but, believe me, this DVD I'm cleaning, which is interlaced, has dirty spread through 3 fields, that is, 2 frames!
Alignfields was spreading the spot/scratch/etc to the 4th field thinking it was motion, so I have to clean before deinterlacing. After some trials & errors, I decided to make a clip with the fields 0 1 3 4 6 7 9 10..., clean it, join again the third field, and then clean this (partially cleaned) clip (apply removedirt again). That's a lot of work but I think it's worthwhile (it's a pity there's no smilie drying the sweat of the face).
I made a sample of 7 frames for you to see:
dali scratch 7frames.avi (382Kb) (http://www.ceciliawolf.pop.com.br/dali scratch 7frames.avi) (do complementparity before separatefields)

Another thing: I tried to use the "removedirt.ini" but it didn't work, saying Avisynth open failure: Removedirt: IO error with f:\dali\rddirty.rmd" and listing the lines after that range file.
My file contains:
dali
athreshold1 = 100
pthreshold1 = 100
mthreshold1 = 500
tolerance1 = 20
range1= "f:\dali\rddirty.rmd"
pthreshold2 = 100
athreshold2 = 100
mthreshold2 = 160
tolerance2 = 40
range2 = "f:\dali\rddontclean.rmd"
pthreshold3 = 5000
athreshold3 = 5000
mthreshold3 = 5000
tolerance3 = 90
range3 = "f:\dali\rdlessmove.rmd"
Nevermind the values, I know they don't match the script.
I'm using VirtualDub-MPEG2.
Thanks in advance.
Cecília
P.S.: link to Amazon about this DVD. (http://www.amazon.com/exec/obidos/tg/detail/-/6305701288/102-7449946-6426503?v=glance) I agree with the last comment.

kassandro

22nd July 2004, 17:13

cwolf,
I think your script seems to be correct. "open failure: script error: the named argument "pthreshold4" was passed more than once to removedirt" is a message from Avisynth not from RemoveDirt. Avisynth issues this error message, if you use a named option twice, which obviously is not the case here. We had this problem already once in this thread and it has not yet been tracked down. RemoveDirt has a lot of named option and these named options are replicated 10 times to handle also up to 9 range files. Because of its enormous size the option string is not a constant string at compile time. Rather it is created at run time, when the dll is initialised. Either Avisynth has problems with such a big option string or there is something wrong with RemoveDirt.
"Removedirt: IO error with f:\dali\rddirty.rmd" is an error message from RemoveDirt (all its error messages begin with "Removedirt:". Either it could not open the file for reading or it could not read the entire file in one stroke. How big is "f:\dali\rddirty.rmd"?
I will try to reproduce the problems using your script. Of course some adjustments have to be done.

cwolf

25th July 2004, 17:18

I made a simplified removedirt.ini with just one rangefile with one range in it, didn't work (same removedirt error).
I'm using "removedirt.dll" 04/18/2004 21:59 34kb. I remember I had two dll's (I had also removedirt2.dll), and I removed the bigger in April or May.

kassandro

25th July 2004, 18:33

Originally posted by cwolf
I made a simplified removedirt.ini with just one rangefile with one range in it, didn't work (same removedirt error).
I'm using "removedirt.dll" 04/18/2004 21:59 34kb. I remember I had two dll's (I had also removedirt2.dll), and I removed the bigger in April or May.
It would be nice if you could simplify the context, in which RemoveDirt fails, even further. Also it is very important to know, if the problem is independent of the clip. If I can reproduce the problem, it should be easy to resolve it.

kassandro

26th July 2004, 21:31

@cwolf
I just reproduced an corrected the RemoveDirt.ini bug.

Turning to the range4 bug. It can be simplified a lot: RemoveDirt(pthreshold4=25) always fails.

Investigating this problem, I found that Avisynth allows only a maximum of 60 variables for a filter. pthreshold4 seems to be number 61. Thus this is a limitation of Avisynth - not a bug of RemoveDirt. You can still use these variables by packing them into the RemoveDirt.ini file. I will upload a preliminary binary of version 0.7, for which the RemoveDirt,ini bug is fixed, probably tomorrow.

Serg Belyansky

27th July 2004, 15:44

kassandro, thanks a lot for such a wonderful filter.

While I generally satisfied with its result I am still scratching my head with question: is it possible to change the amount of "cleaning" with more precision than just "process/ don't process this block"?

I have a source with some amount of negative dirt (white marks, hair and streaks). Using RemoveDirt, I am able to eliminate most of this dirt. But while removing the dirt, it smoothes out all film grain, which is in my case undesirable.

Please tell me if I mistaken and RemoveDirt is not suitable for "dirt only" operation without any "temporal smoothing" effect. For now I have to use Random Noise Remover, which requires a lot of manual post-processing and range applying (its motion blocks exclusion is very crude) and is Virtualdub-only.

cwolf

27th July 2004, 17:00

I made some tests with a very simplified script reading a small avi and two lines of removedirt with removedirt.ini, without success both in VDUBMOD and Virtualdub-MPEG2 (removedirt I/O error), switching the removedirt.dll and removedirts.dll in the "pluging" folder and putting the rangefile in the script's folder, the avisynth's folder and root of f:, removing the path of the rangefile [rangefile1="rddirty.rmd"]. I'll try to run avisynth in my working PC at my job (Pentium II) and do the same tests.

kassandro

28th July 2004, 01:11

I have just uploaded new binaries to the web site. This is an unofficial release though. It should fix RemoveDirt.ini bug detected by cwolf. Actually this essentially version 0.7. Unfortunately the documentation has not yet been updated. There quite a few new features, some old features have been removed or changed (in particular the show mode is very different). On the other, in most cases it can be used as version 0.6.

Originally posted by cwolf
I made some tests with a very simplified script reading a small avi and two lines of removedirt with removedirt.ini, without success both in VDUBMOD and Virtualdub-MPEG2 (removedirt I/O error), switching the removedirt.dll and removedirts.dll in the "pluging" folder and putting the rangefile in the script's folder, the avisynth's folder and root of f:, removing the path of the rangefile [rangefile1="rddirty.rmd"]. I'll try to run avisynth in my working PC at my job (Pentium II) and do the same tests.

Thanks. The RemoveDirt.ini bug has been fixed. Unfortunately, the pthreshold4 problem is not RemoveDirt bug. Rather it is an Avisynth limitation which allows only a maximum of 60 variables and RemoveDirt has about 140. I will post a request to the AVS 2.5.5 thread for a higher limit. Thus pthreshold4 and all the variables for rangeX with X >4 can only be set through RemoveDirt.ini currently.

Originally posted by Serg Belyansky

While I generally satisfied with its result I am still scratching my head with question: is it possible to change the amount of "cleaning" with more precision than just "process/ don't process this block"?

One can do the following: instead of not cleaning, clean as usual but limit the amount of change. The temporal component of PixieDust seems to persue this strategy and despite making only quite small changes (but very many), PixieDust achieves fairly good compression gains. That let me to write the filter ReduceFluctuation (include in the new RemoveDirt plugin), which uses the same cleaning technique as RemoveDirt, but instead of sophisticated artifact protection, it simply limits the amount of change. For instance ReduceFluctuations(limit=5) cleans everywhere, but the change is limited by 5. This is a purely temporal filter. Thus it can be used for any kind of video. As far as compression gain is concerned, ReduceFluctuations(limit=5) is inferior to PixieDust(5), but in combination with a good spatial denoiser (PixieDust has a spatial component as well) it can beat PixieDust, which for me despite its undisputed quality is hardly usable because of its unacceptable speed.

I have a source with some amount of negative dirt (white marks, hair and streaks). Using RemoveDirt, I am able to eliminate most of this dirt. But while removing the dirt, it smoothes out all film grain, which is in my case undesirable.

Please tell me if I mistaken and RemoveDirt is not suitable for "dirt only" operation without any "temporal smoothing" effect.

While RemoveDirt was intended primarily for real dirt and scratches and eliminates temporal noise as well and film grain is temporal (actual it is the result of a chemical process, which behaves randomly on each pic). On the other hand small temporal noise forces the user to use higher values for mthreshold and that implies a higher artifact risk. Moreover, tiny even invisable noise, can be taken care quite well by spatial denoisers. That's why I created RemoveGrain and it can be used as a precleaner for RemoveDirt to lower mthreshold.

Fizick

28th July 2004, 17:19

Serg Belyansky,
I think, that RemoveDirt is not suitable for "dirt only" operation without any "temporal smoothing" effect.
(Sorry, Kassandro) ;)
In place of Random Noise Remover, which requires a lot of manual post-processing and range applying (its motion blocks exclusion is very crude) and is Virtualdub-only
, you may try DeSpot plugin for Avisynth for a source with some amount of negative dirt (white marks, hair and streaks).
But it is almost offtopic, please go to DeSpot thread if you want.

Removedirt may be apllyed as post-processing after DeSpot
(Sorry, Kassandro once more) :)

BTW, I wonder, how you can be register in 2002 and have 1 post?

Mug Funky

3rd September 2004, 09:25

hmm. removedirt seems to be behaving oddly under the new avisynth. i haven't tracked the problem down completely, but i get "IO error" on loading scripts that call it. by commenting out various other plugins it seems to work, but obviously that's not a good solution (and it seems to be a rather large selection of plugins that don't work with removedirt)

the message after IO error is different every time (and can occasionally list every function in avisynth, or sometimes just those in masktools, but usually it's gibberish of some kind.)

Boulder

3rd September 2004, 09:37

I've not had such problems..but I have had "RemoveDirt doesn't have an argument mthreshold" (or similar) a couple of time when I've set mthreshold in my script. This doesn't happen nearly every time which makes it hard to reproduce. Once I had it by setting the two passes of an XviD encode, then postponing the second one (I wanted to run it the next day). I ran the first pass and then decided to run the second one as well. This gave me the error. After a reboot, the second pass ran normally.

kassandro

3rd September 2004, 20:32

Originally posted by Mug Funky
hmm. removedirt seems to be behaving oddly under the new avisynth. i haven't tracked the problem down completely, but i get "IO error" on loading scripts that call it. by commenting out various other plugins it seems to work, but obviously that's not a good solution (and it seems to be a rather large selection of plugins that don't work with removedirt)

the message after IO error is different every time (and can occasionally list every function in avisynth, or sometimes just those in masktools, but usually it's gibberish of some kind.)

cwolf has found an important bug, which caused "IO error" messages. However, this bug was not a conflict with other plugins. Rather it was a terrible programming mistake in the RemoveDirt.ini part. This bug has been fixed and the corrected binary is now contained in the binary archive of the web site. cwolf found another bug, which however was not a bug of RemoveDirt. Rather it was a limitation of Avisynth 2.54, which only allows at most about 60 variables and if you set the 61st variable Avisynth produces idiotic error messages.
Mug Funky, have you tried the new binary. If the bug persists, could you please post a script as simple as possible with which I can reproduce the problem.

Mug Funky

5th September 2004, 15:15

many thanks, kassandro :)

btw, is there a good quick way to avoid a sheared image in cases of earthquake motion? this is the only thing preventing me from using removedirt in all my scripts :)

i suppose there's applyrange, but it seems to me that in some situations one doesn't have time to fully watch the material one is encoding.

by "earthquake motion" i mean the frame jumping up and down in such a way that removedirt removes half a good frame, so you get a torn-up image and a bizarre mix of moving and non-moving parts.

scharfis_brain

5th September 2004, 19:21

@mugfunky: maybe an idea

- stabilize the video before feeding it to removedirt. use depanstabilize
- apply removedirt
- turn the video by 180 degrees
- apply depanstabilize again, to restore the earthquake
- turn the video by 180 degrees

(I restored some old films using that technique exept the 3 last steps)

kassandro

5th September 2004, 21:35

Originally posted by Mug Funky
btw, is there a good quick way to avoid a sheared image in cases of earthquake motion? this is the only thing preventing me from using removedirt in all my scripts :)

i suppose there's applyrange, but it seems to me that in some situations one doesn't have time to fully watch the material one is encoding.

by "earthquake motion" i mean the frame jumping up and down in such a way that removedirt removes half a good frame, so you get a torn-up image and a bizarre mix of moving and non-moving parts.

The motion complaints about RemoveDirt are not new and if I look at videos processed by RemoveDirt, I think to myself that there must be something wrong. On the other hand, RemoveDirt has a three way technique to prevent it from mistreating motion. If n is the number of the current frame, then RemoveDirt measures the SAD distance between blocks in frame n-1 and n+1. If the distance exceeds a threshold, the block is registered as a motion block. Of course, on a small block the difference on frames n-1 and n+1 may be small, while the difference on frames n-1,n and n,n+1 may both be large. This happens if there is dirt on the block in frame n, but also if there is a fluctuative motion. Now the first idea is that this motion should show up also between frames n-1, n+1, if not on the block itself, then on some neighbour blocks. The "dist" variable is exactly for this purpose. If motion between n-1, n+1 is found on some neighbour blocks (the distance depends on the value of dist), then the block is also prohibited from cleaning. Finally, if there are still some motion blocks, which are not yet caught, the cleaned block probably won't fit to its untouched neighbours. The postprocessing routine checks whether a cleaned block fits to its uncleaned neighbours. If it is doesn't fit, it is restored and becomes itself an uncleaned block, which then is used on the next postprocessing loop for checking, whether cleaned blocks fit to their uncleaned neighbours. The postprocessing starts a new loop as long as it has restored at least one block on the previous loop. Usually 2-3 loops are enough, but I have seen frames with 20 or more loops. The advantage of this kind of postprocessing is twofolds. Firstly it makes block artifacts impossible (at least if pthreshold is sufficiently low, say, 10). Secondly, it needs to detect only one small block of a moving object to capture the entire moving object. Thus removing blockiness should also greatly help to remove false cleaning artifacts. Now the feedback and my own experience tell me that for sufficiently fast global motion (typicly scharfis_brain suggestion only concerns global motion) this very sophisticated strategy is not sophisticated enough.

My future plan is to modularise RemoveDirt. Firstly, there will be a simple filter, which creates a block mask clip, this is a yuy2 clip with 1/8 of the height and 1/8 of the rowsize of the original clip. Then there will be an equally simple filter which provides masked cleaning in terms of such a block mask clip and finally there will be a postprocessing filter, the only non-trivial filter. There will also be additional filters for debug output etc. That modularisation certainly adds some overhead (I expect the modular version to be about 5-15 fps slower than the monolithic version on my whimpy pc), but adds an enormous amount of flexibility. For instance, one may use a more sophisticated block mask generator or may replace the simple masked clenser by something more sophisticated. Also the postprocessing routine doesn't need to compare/restore with the original clip. Thus the RemoveDirt plugin should become a tool box for builting cleaners possibly using also third party filters. I plan to use it for a cleaner, which uses a Clense + RemoveGrain in static blocks and Clense + Repair + RemoveGrain (see the RemoveGrain thread for more details) for the remaining blocks to achieve a maximum amount of compression without cheap blurring. With a modular design it is also easier to explore and test new ideas, which may ultimately lead to a resolution of the earth quake motion problem.

Mug Funky

6th September 2004, 11:47

ah. good to hear.

i've tried making a scriptclip to check the difference between the source and removedirt'ed clips, and when a spike is found it uses the source. it worked on a few frames, but then i quickly realized it stopped working very quickly and simply gave me the original source for almost every frame.

i'll experiment with using more sophisticated methods (it's rather like scene-change detection), but i suspect each better-working solution will slow things down more.

right now i'm using a masking approach that is slow as hell but guarantees a shear-free clip.

kassandro

6th September 2004, 23:54

Originally posted by Mug Funky
i've tried making a scriptclip to check the difference between the source and removedirt'ed clips, and when a spike is found it uses the source. it worked on a few frames, but then i quickly realized it stopped working very quickly and simply gave me the original source for almost every frame.

I don't know, whether it would help, but at least one could avoid ScriptClip.I could easily write a filter, which does the following: it compares for each frame the two input clips. If the SAD difference does not exceed a threshold specified as a variable, the frame of the first clip is chosen otherwise the frame of the second clip. Thanks to the psadbw instruction this can be done very fastly (cetainly more than 100 fps on my 1.3 GHZ Celeron). Probably the Scriptclip version is below 1 fps.

Mug Funky

7th September 2004, 06:03

surprisingly the scriptclip version ran a total of about 7 fps on my very modest system (p3 733). it just didn't work properly, which is why i abandoned it. but the idea isn't completely mined out though, and i might have another go at it later.

[edit]

tried replacing "averageluma" with "ydifferencefromprevious+ydifferencetonext" and got some more reliable results - now less noise is being let through, and more actual problems are being spotted.

runs at about 8fps (this is my total script speed, which includes smartdecimate and some slow fallback options for it which don't come up much).

function carefulDirt (clip c, int "th", bool "show")
{

th = default(th,16)
show = default(show,false)

dirtless=c.removedirt()

maskd=yv12lutxy(c,dirtless,yexpr="x y - abs 16 > 255 0 ?", Y=3,U=-128,V=-128)
maskd=maskd.expand().reduceby2().reduceby2().expand(Y=3,U=-128,V=-128)

dirtless = (show==true)? dirtless.subtitle("removedirt-ed") : dirtless

conditionalfilter(maskd,c,dirtless,
\ "Ydifferencefromprevious()+Ydifferencetonext()","greaterthan",string(th))

#scriptclip(maskd,"subtitle(string(Ydifferencefromprevious()+Ydifferencetonext()))")

}

kassandro

8th September 2004, 00:45

Originally posted by Mug Funky
surprisingly the scriptclip version ran a total of about 7 fps on my very modest system (p3 733). it just didn't work properly, which is why i abandoned it. but the idea isn't completely mined out though, and i might have another go at it later.

As I can see carefulDirt uses ScriptClip only for debugging. If you would really use ScriptClip, you couldn't obtain 7 fps, because ScriptClip has to run for each frame the Avisynth initialisation of the script, which is executed by ScriptClip. If the ScriptClip script uses filters with lengthy initialisation like RemoveDirt, it really becomes a paine.

Chainmax

9th September 2004, 03:50

So what's in store for RemoveDirt and RemoveGrain? Will they be integrated into the new RemoveDust? BTW, will RemoveDust be able to remove black spots like RemoveDirt does?

kassandro

9th September 2004, 08:28

Originally posted by Chainmax
So what's in store for RemoveDirt and RemoveGrain? Will they be integrated into the new RemoveDust? BTW, will RemoveDust be able to remove black spots like RemoveDirt does?
As said above I will modularise RemoveDirt, the actual cleaners then will be script functions, which put the pieces together. At a later stage, it is probably reasonable to put all the plugins together into single cleaner plugin, but as long as things change a lot it is preferable to keep them separate.

doxville

14th September 2004, 07:16

Hi!

I came over following problem last night when i was about to use the removedirt-filter: Without having parameters in the brackets, everything was fine, but as soon as i put some parameters (like 'tolerance=48') into the brackets, TMPGEnc won't accept audio any more. I could not import the avi as an audio source, because of TMPGEnc's 'cannot open, or unsupported'-error. The script was like this:
AVISource("D:\video.avi")
delayaudio(1)
Trim(50,43856)
unfilter(75,50).LanczosResize(448,544,8,4,704,568)
removegrain(7).removegrain(6).removedirt(tolerance=48)
colorYUV(levels="PC->TV")
tweak(sat=1.2,cont=1.2,bright=5)
AddBorders(16,16,16,16)
separatefields()
selecteven()
weave()
I changed the script and used temporalsoften instead, and everything was ok:
AVISource("D:\video.avi")
delayaudio(1)
Trim(50,43856)
unfilter(75,50).LanczosResize(448,544,8,4,704,568)
removegrain(7).removegrain(6).temporalsoften(2,3,5,6,2)
colorYUV(levels="PC->TV")
tweak(sat=1.2,cont=1.2,bright=5)
AddBorders(16,16,16,16)
separatefields()
selecteven()
weave()

Can anyone tell me what went wrong..??

kassandro

14th September 2004, 10:44

Can you load the first script into Vdubmod? I don't have Tmpgenc. You may process the audio separately with avs2avi and an mp2 encoder and mux the audio and video later. On the other hand, such mysterious problems should be investigated.

doxville

14th September 2004, 11:39

The script doesn't produce problems neither with VDubMod (i initially edited the script with the VdubMod-script-editor) nor with MediaplayerClassic (which shows me a normal preview including video AND audio). So until now i can't tell you how to reproduce the situation without using TMPGEnc.:( There's a free version of TMPGEnc, maybe give it a try: http://download.pegasys-inc.com/download_files/TMPGEnc-2.521.58.169-Free.zip
Of course i could have processed the audio seperately, then all would be fine, it's just that i couldn't believe that a denoiser affects the audio in a script...:confused:

btw: i'm using the latest Avisynth 2.5.5

kassandro

16th September 2004, 07:55

Before I download the free version of Tempgenc, could you please test the following:
1. If you replace video.avi by another avi, does the Tmpgenc problem persist?
2. If so, does the problem still persist, if you load an mpeg2 video with MPEG2Source/MPASource instead of AVISource?

A video only filter, like all my filters, just passes through the audio. The only explanation for the problem is that RemoveDirt touches forbidden memory areas, which are only important for Tmpgenc.

doxville

16th September 2004, 13:02

Okay, here's what i tested:

1. The same problem occurs with another AVI-file as a source

2. A m2v-file imported with either 'directshowsource()' as well as with 'mpegsource()' lead both to the same problem i am having with the AVI...

kassandro

16th September 2004, 22:19

Thank you for testing. I should be able to reproduce it. It shows that the problem has nothing to do with the origin of video and audio. That problem may also be related with other mysterious problems reported here. I am leaving now for a short vacation a will be back on october 4th. Your problem is sufficiently concrete to be investigated first then.

doxville

17th September 2004, 07:05

Thank you! Enjoy your vacation! :)

Major

17th September 2004, 23:36

Hi Men

I have used removedirt in classic films (black and white)
my setting are:
removedirt(grey= true, pthreshold= 10, mthreshold=180) I have not put cthreshold because is ignored like it is written in the help pf the removedirt web page.

I have noticed a slight cleaning and i have two questions:

1. My setting are good??

2. if I apply removegrain with removedirt, will be better for cleaning the movie??

Thanks

iradic

27th September 2004, 02:58

hi

is it possible to use writefile func to create range files?
i was thinking using quantified motion function (its in avisynth doc) for that...

will resulting file be ok for removedirt or do i have to modify it?

is this ok (output of writfile):
--------------------------------
0
1
2
3
...
100

or:
---
0-100

this way i dont need to preview the whole movie before applying removedirt...

thanks

kassandro

27th September 2004, 17:59

@Major:
I too encode a lot of b&w movies and your settings look very similar too mine, for me mthreshold=180 is a little bit too high. I use values <= 150. That should still take out a lot of dirt. You may use RemoveGrain(mode=8, modeU=-1) for precleaning before RemoveDirt and then you can lower mthreshold somewhat more. I you merely want better compression, you may use one of the two RemoveDust functions discussed on the RemoveGrain web site. The first one yields very compression however with prehaps too much motion blurring. The second one - currently my personal favorite for progressive material -, somewhat slower with less compression but with only very mild motion blurring. However, RemoveDust, unlike RemoveDirt, can only remove dust particles and grain but no bigger dirt. On the other, these small particles are much more frequent and have therefore a much more significant impact than the big very visible dirt spots.

@iradic:
both ways are correct to describe a range from 0 to 100 in a RemoveDirt range file.

Boulder

27th September 2004, 21:16

kassandro,

could you give your opinion on this: http://www.kvcd.net/forum/viewtopic.php?t=14141 . Are the thresholds just too high (apparently the default ones) or is that loss of fine lines in the arc inevitable?

kassandro

28th September 2004, 22:47

Originally posted by Boulder
kassandro,

could you give your opinion on this: http://www.kvcd.net/forum/viewtopic.php?t=14141 . Are the thresholds just too high (apparently the default ones) or is that loss of fine lines in the arc inevitable?

The artifacts displayed in these three pictures are not caused by too high thresholds though pthreshold=0 may resolve the problem without harming cleaning to much. It doesn't make sense to solve this problem by lowering pthreshold below 0 or lowering mthreshold a lot, because this would essentially disable cleaning.
Firstly the dark curve marking the lower end of the arc looks like a scratch to RemoveDirt. If it would be nearly identical on at least one adjacent frame RemoveDirt wouldn't touch it. Unfortunately this is not the case here. Either the film camera was a little bit shaky or more likely the DVD mastering was poor. Even a one pixel motion conpensation would probably not resolve this problem completely and a two pixel motion compensation is simply too expensive in many ways.
If you look at this video in full speed instead of single frames, the artifacts, if realized at all, show up as a little bit of flicker. It is the price to pay for real cleaning. The suggested alternative Deen or PixieDust are not cleaners. If I understand the description of the algorithm of these filters correctly they avoids temporal blurring, if the temporal differences exceed a certain threshold. Such filters should be called smart compression improvers but not cleaners. Though I have only seen a few frames, the video is probably too clean for RemoveDirt and you pay the price of artifacts for very little cleaning. Thus for these kind of videos compression improvers like Deen, Dust or RemoveDust (which does some real cleaning and may also have problems with the arc) are more suitable. But make a comparison with material from the thirties and fourties and you won't mind paying the price of some slight artifacts.

doxville

29th September 2004, 13:48

I had the same effects in my last TV-capture (using removedirt without parameters):
http://people.freenet.de/doxville/01.jpghttp://people.freenet.de/doxville/02.jpghttp://people.freenet.de/doxville/03.jpghttp://people.freenet.de/doxville/04.jpg

These are screenshots of 4 consecutive frames, but look what it did to the marks on the street...:eek: Are these marks already too small to get recognized as details of the picture and therefore get filtered as 'dirt'?

After lowering the mthreshold down to 50 instead of 150, the target was getting better.

kassandro

29th September 2004, 23:11

You probably mean that RemoveDirt darkens some of the white street marks. As far as the first one from the left is concerned that may very well be the case, but again these artifacts are hardly visable at 25 fps. However, the sharp black spots on the second white stripe are certainly not caused by RemoveDirt (these spots are probably persons walking on the street). Here it is necessary to have the original frames. I consider mthreshold=50 too low and would first lower pthreshold or increase dist to 2.

LeoH

30th September 2004, 04:31

It looks like the white dashes are disappearing from frame to frame. This same sort of thing was happening to me with various scenes - I had to lower the threshold so low that it became useless. :(

~LeoH

kassandro

3rd October 2004, 23:47

With RemoveDirt artifacts like in the above two impressive examples cannot be completely avoided, I can do something about it, though because of missing information I am not absolutely certain about the concrete cause here.
RemoveDirt spends most of its time with comparing 8x8 blocks from the previous and the subsequent frame. Thus maximum speed is very desirable for this most common operation. Hence I use the psadbw SSE instruction for this purpose. psadbw is even more necessary for doing any kind of motion vector search. However, psadbw has one serious drawback: one cannot ignore a certain amount of "small" invisable noise, very common in chemical film, which forces one to raise the mthreshold variable to achieve a reasonable amount of cleaning.

Until today I didn't want to change anything with RemoveDirt before modularisation next year. But now I see the necessity to do so. There will be one further new version this year with a new variable "noise".
If it is not set, then RemoveDirt will behave as before. If it is set however, it will use a significantly slower but better block comparison function, which will ignore pixel differences <= the value of noise. Overall I expect the frame rate of Removedirt to fall by 15-40%, if noise is set, but the likelihood of the above artifacts should diminish.
Personally I think the motion distortion reported by Mug Funky and others is a more severe problem than the above artifacts, which are hardly visable at full speed.

oo_void

4th October 2004, 21:28

Originally posted by doxville
I had the same effects in my last TV-capture (using removedirt without parameters):

These are screenshots of 4 consecutive frames, but look what it did to the marks on the street...:eek: Are these marks already too small to get recognized as details of the picture and therefore get filtered as 'dirt'?

After lowering the mthreshold down to 50 instead of 150, the target was getting better.
Not intending to detract from kassandro's fine work... But I've been having a lot of success doing spot removal with Fizick's (http://bag.hotmail.ru/) DeSpot, especially when combined with his motion estimation in the Depan plugin. Something like...

i = last
d = depanestimate(trust = 3)
depaninterleave(i, data = d)
despot(p1 = 30, p2 = 15, pwidth = 702, pheight = 276, mthres = 20, motpn = true, dilate = 1, seg = 1)
selectevery(3, 1)

The above example was pulled from the docs, so you'll need to tweak for your own purposes. Fizick's stuff is definately a bit more compilcated to tweak, but with the proper amount of effort, I've completely elimated all the false positives that I was getting with RemoveDirt. When I last tried RemoveDirt, it had a helluva time with shadows and wrinkles on white clothing (Princess Leia's dress in Star Wars for example).

------
oo_void

Boulder

11th January 2005, 15:11

Any news regarding the development? The filter's still on top in my analogue capture toolbox:)

kassandro

12th January 2005, 16:02

Originally posted by Boulder
Any news regarding the development? The filter's still on top in my analogue capture toolbox:)
Currently, there is no development going on. I just made the switch from my good old 1.3 GHZ Tualatin Celeron to a 2.66 GHZ Prescott Celeron. Unfortunately the system is getting instable, when I use my Technotrend SAT-TV card. I probably will replace it by a Skystar 2. Until all this are sorted out I cannot return to do some programming at home. In fact, I have not yet installed any development system. My first priority is to sort out the hardware problems. When I finally close the computer case, the first thing will be the completion of the SSE2 transition of my RemoveGrain package (I now have even SSE3). Probably by the end of February there should be a new version of this package.

kingmob

12th January 2005, 22:10

Originally posted by kassandro
Currently, there is no development going on. I just made the switch from my good old 1.3 GHZ Tualatin Celeron to a 2.66 GHZ Prescott Celeron. Unfortunately the system is getting instable, when I use my Technotrend SAT-TV card. I probably will replace it by a Skystar 2. Until all this are sorted out I cannot return to do some programming at home. In fact, I have not yet installed any development system. My first priority is to sort out the hardware problems. When I finally close the computer case, the first thing will be the completion of the SSE2 transition of my RemoveGrain package (I now have even SSE3). Probably by the end of February there should be a new version of this package.
Just want to add i can't wait. This filter is providing me with some excellent results and speed improvements are always welcome :).
Good job!

Boulder

12th January 2005, 23:16

Oh, and if you play with the filter, the presets need some work. I mean, they should definitely work so that every preset is separate.

For example, removedirt.ini with

capture # for TV-caps
mthreshold=100
pthreshold=20
cthreshold=30

bw # for b/w caps
mthreshold=100
pthreshold=10
grey=true

should work so that when 'capture' is called, it sets mthreshold to 100, pthreshold to 20 and cthreshold to 30. Everything else should remain at their default values. Otherwise the presets are pretty much useless, although using a removedirt.ini file with only one preset and calling that is a workaround to the bug where RemoveDirt gives an error message saying it doesn't have the argument mthreshold (or cthreshold etc.)

kassandro

13th January 2005, 19:57

@kingmob: thank you for your kind remarks.

@Boulder:
The default values for mthreshold have to be lowered anyway. As I have announced earlier I plan to refine the routine, which detects motion blocks. Essentially, pixel differences which are below the value of the new noise variable are ignored. The new RemoveDirt will behave as the old one, if noise=0. If noise > 0, then RemoveDirt will be somewhat slower but still reasonably fast. I hope to do this until March. Also some stability problems, for instance in combination with tmpgenc, should be fixed.
For the more distant future, I would like to dismantle RemoveDirt into several smaller filters, such that the old RemoveDirt becomes an AVS script function. The advantage of this modular approach compared with the old monolithic approach, is that additional interesting filters can be obtained as AVS script functions. I have demonstrated this already in the current RemoveGrain package and the next one will contain more in this direction.

Boulder

13th January 2005, 21:51

Sounds good indeed. Fortunately RemoveGrain+RemoveDirt combo already works very well so I can wait:)

AVIL

9th April 2005, 02:31

Hi

I've been testing removedirt over a clip using a cleaned (smoothed) version of the clip for the parameter neighbour. I been thinking that cleaned clip serves solely for motion detection, but the artifacts in this cleaned clip appears in the original clip. ¿Is it a bug or i am the bug?.

The script is :

setmemorymax(128)
a=avisource("bve_filtro.avi").crop(192,0,256,0,align=true).assumebff()
u=a.TDeint(order=-1,mode=1,field=1,type=0,sharp=true,mtnmode=0,mthreshL=3,mthreshC=3,cthresh=3)
v=u.removedirt(mthreshold=100,athreshold=50,pthreshold=19,neighbour=u.fft3dfilter(sigma=20,bt=3,measure=true))
stackhorizontal(u,v,mvex(u,v))

Function mvex(clip c ,clip d){

YV12LUTxy(converttoyv12(c), converttoyv12(d),yexpr="x y - 4 * 128 +" , uexpr="128", vexpr="128",u=3,v=3 ).converttoyuy2()

}

I've attached a frame with trhee parts. The first is the original, the second is the result of removedirt and the third is the difference of luma (four times magnified). It shows the tipical pattern of the filter FF3DFILTER with too high values for sigma. But i have uses this filter for the "neighbour" clip, not for the original clip.

Bye.

Wilbert

9th April 2005, 13:04

@AVIL,

I don't see any attachments pending :confused:

AVIL

9th April 2005, 13:26

Sorry:

I`ve hosted the image (click to enlarge). I must practise more in attaching files.

http://img225.exs.cx/img225/9330/removedirt9nn.th.jpg (http://img225.exs.cx/my.php?loc=img225&image=removedirt9nn.jpg)

kassandro

9th April 2005, 21:45

@AVIL:
As you already said the luma difference "picture" on the right side, shows a block pattern. The blocks are larger than the usual 8x8 blocks used by DCT codecs like XviD or a filter like RemoveDirt. Thus they must stem from FFT3D, which uses a variable block size. Now, if you use a clip filtered with FFT3D as a neighbour clip, it is used for cleaning, whence FFT3D blockiness propagates to the output frames of RemoveDirt. Of course, only in those areas where RemoveDirt cleans. This is normal.

I am currently testing a completely rewritten new version of RemoveDirt. Firstly the new plugin contains motion detection routines, which can ignore a specified level of noise. Secondly, the filter is modularised such that it can be combined with RemoveGrain/Repair, to achieve a maximum amount of cleaning. Actually the filter RemoveDirt has been replaced by RestoreMotionBlocks and RemoveDirt became an AVS script function involving RestoreMotionBlocks, Clense and various instances of RemoveGrain and Repair. The basic philosophy is to clean the static areas similar to the old RemoveDirt and the non-static areas with a much sharper variant of RemoveDust, which avoids motion blurring. Thirdly, support for interleaved YUY2 is replaced by support for planar YUY2, which bodes well for combining it with RemoveGrain.

I hope to release the new version next weekend.

grua

14th April 2005, 20:30

Hi,

I use RemoveDirt 0.6.1 with my interlaced analog captures and it does a great work - thanks for this great filter!

I use it in this way:
import("D:\Programme\AviSynth\Lib\SmartBob\mvBob\mvbob.avs")
LoadPlugin("D:\Programme\AviSynth\plugins25\Cnr2.dll")
LoadPlugin("D:\Programme\AviSynth\plugins25\RemoveDirt.dll")
LoadPlugin("D:\Programme\AviSynth\plugins25\Depan.dll")
avisource("test.avi",true,"YUY2")
crop(4,0,-4,-4)
addborders(4,0,4,4)
assumetff()
mvbob()
ConvertToYV12()
data=depanestimate(last,pixaspect=1.094)
depaninterleave(last,data,pixaspect=1.094,prev=2,next=2)
Cnr2(mode="oxx",scdthr=10,ln=35,lm=192,un=60,um=255,vn=60,vm=255,log=false,sceneChroma=false)
RemoveDirt(mthreshold=50,pthreshold=20)
selectevery(5,2)
AssumeTFF.ConvertToYUY2()
SeparateFields(a).SelectEvery(4,0,3).Weave()
When I open this script in VDubMod, everything works fine. But if I open it in Canopus Procoder Express, then there is the AviSynth-Error-Message, that there ist no parameter called mthreshold. Boulder has mentioned an simmilar problem in this thread a view posts above.
I had the same problem some days ago with other source-video and with other encoders too. But in VDub it always works.

Any idea?

Boulder

15th April 2005, 05:18

A workaround to the problem is to create a preset with those values in it and then call it in RD. Read the docs, creating presets is described there. This works if you always use those values you wrote or if you change them in the ini file when necessary. Multiple presets in the ini file apparently doesn't work as it should.

kassandro

15th April 2005, 08:22

Originally posted by grua

I had the same problem some days ago with other source-video and with other encoders too. But in VDub it always works.

Any idea?
Strange! As I use only VdubMod, I never observed this. There was a problem, when many range files were used, because Avisynth doesn't allow more than about 60 variables for a filter. Here, presets wera lso the only solution. The new RemoveDirt has neither presets nor range files anymore, but I can add it again in later versions, if there is sufficient demand or even necessity, but let's hope this problems are gone.

grua

15th April 2005, 15:31

Thanks - an other way ist to save the AVS-Output with VDub as AVI (lostless Huffy, fast recompress) and afterwards load this AVI in the encoder.

Seed

16th April 2005, 17:30

Originally posted by grua
Hi,

When I open this script in VDubMod, everything works fine. But if I open it in Canopus Procoder Express, then there is the AviSynth-Error-Message, that there ist no parameter called mthreshold. Boulder has mentioned an simmilar problem in this thread a view posts above.
I had the same problem some days ago with other source-video and with other encoders too. But in VDub it always works.

Any idea?
I don't think it is limited to Removedirt.

I'm quite sure that you have Intel P4 with hyperthreading enabled, and your avi is likely to be Huffyuv-compressed. I encountered this problem long time ago (with simple Avisynth script without any filter), and am quite surprised that there has been hardly any mention of it in the forum. I have since done some digging and found:

(A) In fact it happens to Avisynth in general, even just an one-liner "avisource("myvideo.avi")" script. Version 2.5x for sure, I can't remember if version 2.08 gives the same problem.

(B) For this Avisynth "exception error" to occur, 2 things must be present together:
(i) P4 CPU with hyperthreading enabled, and
(ii) frameserving to/via a Direct Show/Direct-X application (e.g. Media Player Classic, Windows Media Player, Procoder Express)

Therefore, you can circumvent the problem either by
a) disabling hyperhtreading, or
b) frameserving to a VfW (Video for Windows) application, e.g Virtualdub. I also use CCE Basic, and found that CCE Basic appears to accept Avisynth frameseving via VfW and not Direct-X, and thus does not give this "exception error" problem.

(C) When it happens, some codecs are more problematic than others. I use Huffyuv 2.1.1 most often, and with hyperthreading enabled, MPC will invariably crash with Avisynth exception error on playing 50-100 frames or on clicking 2-3 times on the seek bar.

Interestingly, it is much less "exception error" prone if ffdshow is used to playback Avisynth-scripted huffyuv-compressed avi files in MPC. Also, ffdshow's modified version of Huffyuv (FFVH) is also more stable. IIRC, uncompressed avi is not exempted from this "exception error" problem, but can be "stablised" too by using ffdshow as a direct-x filter.

This stabilization by ffdshow is not good enough for proper encoding job in Procoder Express (one garbled frame is too many), but makes it watchable in MPC.

kassandro

17th April 2005, 15:03

Originally posted by Seed
I don't think it is limited to Removedirt.

I'm quite sure that you have Intel P4 with hyperthreading enabled, and your avi is likely to be Huffyuv-compressed. I encountered this problem long time ago (with simple Avisynth script without any filter), and am quite surprised that there has been hardly any mention of it in the forum. I have since done some digging and found:

(A) In fact it happens to Avisynth in general, even just an one-liner "avisource("myvideo.avi")" script. Version 2.5x for sure, I can't remember if version 2.08 gives the same problem.

(B) For this Avisynth "exception error" to occur, 2 things must be present together:
(i) P4 CPU with hyperthreading enabled, and
(ii) frameserving to/via a Direct Show/Direct-X application (e.g. Media Player Classic, Windows Media Player, Procoder Express)

Interesting observations. As I have a CeleronD, I can't confirm it. If the application requests two frames simultaneously (in different threads) from Avisynth, garbage or even a crash is almost inevitable. However, I don't know, whether this really is the case here. My DVB program ALT-DVB does not work with hyperthreading as well. Also for video processing hyperthreading would not be useful anyway. Thus my decision to go for a poor man's P4 was probably right. Also the CeleronD can be easier overclocked (because of the slow 133 MHZ FSB).

kassandro

17th April 2005, 15:17

I just have uploaded a new version of RemoveDirt to a new server (Arcor, Germany). The link stays the same. The old Austrian server is down too often. The old RemoveDirt 0.6 will stay on the old server, but the links to it are currently incorrect.
The new version has been completely rewritten. RemoveDirt is now an Avisynth script function rather than a binary filter. The RemoveDirt plugin contains the filter RestoreMotionBlocks, from which together with the RemoveGrain package (you need version 0.8 at least) the script is built. RemoveDirt will be improved further with the new RemoveGrain 0.9.

I have done soemthing against the motion distortion observed first by Mug Funky, but I can't really confirm this until now. The tests I have done so far were done with a faulty version. I am doing the first real encoding test right now, but the test results with the faulty version were encouraging. As I have 19 inch (slow) TFT display which generates similar motion artifacts as the old RemoveDirt, I can only say something definite, after burning the results on DVD and watchin them on TV with may DVD player.

A lot of things have changed and one has yet to find the right default values. Presets and range files are currently missing, but I can include them in later version, if there is sufficient demand.

The new documentation is somewhat incomplete. Suggestions for improvement are wellcome.

grua

18th April 2005, 12:45

Originally posted by Seed
I don't think it is limited to Removedirt.

I'm quite sure that you have Intel P4 with hyperthreading enabled, and your avi is likely to be Huffyuv-compressed.Sorry, but I use an AMD Athlon Thunderbird 1,4 GHz and PicVideo MJPEG. Also I had this error really only with ReoveDirt and with no other.

But like I said above: no real problem for me, because I can export the AVS in Huffy with VDub and open this Huffy-Avi in Procoder Express.

cu, grua

AVIL

18th April 2005, 23:07

Hi:

@Kassandro

I have read carefully the guide about the new version of Removedirt. I'm not have understood the role of neighbour2. May you, please, explain how this parameter is handled.

Thanks in advance.

kassandro

19th April 2005, 07:09

Originally posted by AVIL

I have read carefully the guide about the new version of Removedirt. I'm not have understood the role of neighbour2. May you, please, explain how this parameter is handled.

I initially wanted to include the script function MCRemoveDirt in the documentation, but I didn't have the time. The neighbour2 variable was intended for using motion compensation, but I have now another application.
In order to decide, whether a block at time n should be marked as a motion block, RestoreMotionBlocks looks at the difference between this block in the frame neighbour(n-1) and in the frame neighbour(n+1). It doesn't look at the block in frame neighbour(n), because then dirt at time n would distort the differences (the neighbour clip normally is the original unclensed clip) and we really want to clean the dirt at time n.
Now if you specifiy neighbour2, then RestoreMotionBlocks behaves very differently. It compares the block in frame neighbour2(n) and in the frame neighbour(n). In particular, this implies that both clips are different (otherwise there would be no motion blocks at all). Let us look at the following interesting script function

function RMBDenoise(clip input, bool "_grey", int "repmode")
{
clmode=17
clensed=Clense(input, grey=_grey)
restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode )
corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, neighbour2=clensed, gmthreshold=70, dist=0, debug=true, noise=12, noisy=12, grey=_grey)
return RemoveGrain(corrected, mode=clmode, modeU = _grey ? -1 : clmode )
}

It looks very much like the RemoveDirt script function. neighbour2=clensed is the only difference. Now a block is marked as a motion block in the first phase, if the difference between the clensed block and the unclensed original block in the frame input(n) are sufficiently different. This makes this script function more defensive than RemoveDirt. It can no more take out big spots, though it still can take out scratches, if the value of noisy is large enough. On the other hand the artifact risk is lower as well. To make the above denoiser a little bit more aggressive, you may also use neighbour2=restore instead of neighbour2=clensed. Actually, if you use neighbour2=clensed there may be an unnecessary reprocessing, because I did configure the cache too tightly for this situation. This will be fixed in version 0.8.1.

By the way, I did decrease the value of noise from 12 to 8, because there were some rare artifacts at this level.

kassandro

7th May 2005, 18:33

The new version 0.9 can be found on the web site (http://www.RemoveDirt.de.tf).
I have added the SCSelect to the plugin. It is primarily for handling sharp changes in the RemoveDirt script function, which has been updated appropriately. As it is now, it seems to be a bit too aggressive. I just did copy it from my test script and so far I like what I see. The most recent RemoveGrain 0.9 is also needed for this script function.
The second major change concerns the second phase of RestoreMotionBlocks. There is now the new mode variable dmode. With dmode=0 (the default) RestoreMotionBlocks should behave as in version 0.8. Unfortunately it doesn't, because there was a bug in version 0.8.
dmode=1,2 are the other possible modes. The number of motion blocks declines as dmode is chosen higher. dmode=2 is quite aggressive and dmode=1 is probably the best compromise.
When dist=0 (no second phase for RestoreMotionBlocks), there was a simple but more severe bug in the third phase, the postprocessing phase. This has been fixed. The documentation has also been cleaned and extended subdtantially.
I hope that version 0.9 is more mature than 0.8.

AVIL

7th May 2005, 19:32

Hi.

The link above points to removegrain's page, not to removedirt's one

Regards

kassandro

7th May 2005, 19:54

Thanks, I have corrected it.

Alpennic, the company which provides the domain for free in exchange for a popup window, seems to block browsers with a popup blocker for a while.

nanoflower

29th May 2005, 06:48

Ugh.. I'm sure the new form is much more powerful, but the old RemoveDirt() is so much easier to work with. Perhaps it's just because I'm tired, but after working with new RestoreMotionBlock function I had to give up because I couldn't get it to work. I'm sure it's something simple that I'm not setting correctly, but the old RemoveDirt function is easy to setup.

Fizick

5th June 2005, 14:57

kassandro,
can you consider the some renaming of your new Removedirt to prevent user confusing?
For example, we recently made russian translation of it, and do not understand, what is the right.
(I think both versions are exellent an will be used)
So, what about renaming to RemoveDirt2 for example?
Sorry for disturb.

FredThompson

5th June 2005, 17:56

It would be nice if the filters didn't force VirtualDubMod to crash, too. Happens with all of them, RemoveGrain, etc.

zilog jones

5th June 2005, 20:39

I think that's an issue with AviSynth filters in general...

FredThompson

5th June 2005, 20:44

Possibly. Every filter in the RemoveDirt/RemoveGrain packages makes it choke on my machines. So does TDeint. I'm guessing there's something common to K's filters which causes this to happen, hence, the reason I bring it up.

Calca_P

22nd June 2005, 09:15

I use Removedirt v9 with MPEG2 cleanup (mostely old VHS to DVD releases)
and I like the end product.
I have spent time adjusting parameters but am unable to get past the blur,
(end product is not as defined as the original though much cleaner to the eye)
Is it possible to get a blurless setting with this filter, or is it not possible yet?
I understand that a certain amount of blur is usually an effect though none
of course would be nice. I want too much? possibly.

Thanks

Didée

22nd June 2005, 10:14

@ Calca_P

First off, I've never seriously worked with RemoveDirt, so I know almost nothing about its parameters, and almost nothing about its side-effects.

However, what ~might~ be useful is a lowpass filter. The noise you want to filter by RemoveDirt probably is of high spatial frequency, and unwanted blurriness often is caused by changes in lower spatial frequencies.

Try this: (needs MaskTools (http://manao4.free.fr/MaskTools-v1.5.6.zip))

[...]
#------------------------------------------
function m4(float x) {return( x<16?16:int(round(x/4.0)*4) ) }
#------------------------------------------

before = last

RemoveDirt( [..parameters..] )

after = last

radius = 2.0 # Try 1.25 to 3.0

diff = yv12lutxy(before,after,"x y - 128 +","x y - 128 +","x y - 128 +",U=3,V=3)
diff2= diff.bicubicresize( m4(diff.width/radius), m4(diff.height/radius) )
\ .RemoveGrain(4).bicubicresize( diff.width, diff.height, 1,0 )
diff = yv12lutxy(diff,diff2,"x y 128 - -","x y 128 - -","x y 128 - -",U=3,V=3)

lowpassed = yv12lutxy( before, diff, "x y 128 - -","x y 128 - -","x y 128 - -",U=3,V=3)

lowpassed

Might work in your case, or might not. You'll see.

Calca_P

22nd June 2005, 10:50

Thanks Didee, much apreciation.

Ill give it a go. Yes, I do like Kasandro's your scripts, they do a good job for my likes.

Thanks

EDIT:

Didee, would you please explain to me this, "[...]"
also, Avisynth doesnt accept the function, "m4" on the "diff2=" line

Didée

22nd June 2005, 14:09

Ooops! I didn't notice your edit ... but right now I suddenly remembered that I better should include that tiny "m4" modulo-4 function if I make use of it ...
Corrected the scriptlet above. Sorry.

The [...] means: any avisynth code you have in your original script before calling RemoveDirt().

Calca_P

23rd June 2005, 08:34

Didee,

The script works now and tried it out.
To the eyes, it doesnt seem to change the blur, just moves it a little in places
and creates a darker grainyness throughout the picture.

Ill keep testing, thanks

yaz

23rd June 2005, 12:17

@didée
i see u're a big fun of yv12lutxy, so maybe u can explain what's in your script :Dyv12lutxy(diff,diff2,"x y 128 - -","x y 128 - -","x y 128 - -",U=3,V=3) can u tell me what does this function exactly do ?
it's a kinda polish notation, isn't it ? so, i'd rather put ..."x y - 128 -" ... instead. or am i completely wrong here ?
thx
y

Didée

23rd June 2005, 14:13

@yaz

You're completely right. It's inverse polish notation, and instead of "x y 128 - -" one could write "x y - 128 +" just as well.

Personally, I use both terms, in places. The alternative use of both terms just deals to indicate myself what kind of operation I'm actually doing - either creating an offset plane, or applying an offsetted plane. Personal habit, nothing else.

BTW, this (simple) kind of lowpass protection deals pretty good with temporal smoothers. Try the aboveposted scriptlet with e.g. temporalsoften:

before = last

# make a temporalsoften /w big thresholds, limit result to +-3 ...
LIM = "x 3 + y < x 3 + x 3 - y > x 3 - x 49 * y 51 * + 100 / ? ?"
ts = before.temporalsoften(2,23,23,23,2)
yv12lutxy(before,ts,LIM,LIM,LIM,U=3,V=3)

after = last

radius = 2.0 # Try 1.25 to 3.0

# ... and apply a lowpass to "the effect" to avoid smearing
diff = yv12lutxy(before,after,"x y - 128 +","x y - 128 +","x y - 128 +",U=3,V=3)
diff2= diff.bicubicresize( m4(diff.width/radius), m4(diff.height/radius) )
\ .RemoveGrain(4).bicubicresize( diff.width, diff.height, 1,0 )
diff = yv12lutxy(diff,diff2,"x y 128 - -","x y 128 - -","x y 128 - -",U=3,V=3)

lowpassed = yv12lutxy( before, diff, "x y 128 - -","x y 128 - -","x y 128 - -",U=3,V=3)

lowpassed

Should work well with any filter that does sort of temporal smoothing. I've tested with (limited) TemporalSoften, (limited) Clense, and LRemoveDust.

yaz

24th June 2005, 09:18

whoa, man, u gave me the feeling of 'lutting' :p i've made such lowpass protection several times w/a very complicated multifold masking. your script is not simply more elegant but much quicker. i see i should sentence some time to this 'lutting' business. i wish i had :(
LIM = "x 3 + y < x 3 + x 3 - y > x 3 - x 49 * y 51 * + 100 / ? ?" i luv it. decryption took some time but now i get it (i guess)
thx
y

Didée

24th June 2005, 10:57

Well, the term simply reads

If: difference(original,filtered) <= 3
then: apply 49% of it
else: limit the change to +- 3

An alternative term doing the very same could have been:

LIM = "x y - abs 3 > x 3 x y - x y - abs / * - x 49 * y 51 * + 100 / ?"

[read: abs(x-y) > 3 ? x - 3*sign(x-y) : (49*x + 51*y) / 100 ]

When starting out, the inverse polish notation might be hard to get used to. (I can tell you, I've been there before) ;)
But once you got grip to it, you can write terms "several meters long" , and it doesn't even require much thinking. :)

yaz

24th June 2005, 11:39

When starting out, the inverse polish notation might be hard to get used to. (I can tell you, I've been there before) ;)
But once you got grip to it, you can write terms "several meters long" , and it doesn't even require much thinking. :)yeah. i'd been there too. in the golden age (i.e. right before m$ ;) ) i made device controll, data collection and evalution in 'forth' (does this name tell anything to anyone still?) and then we did write terms approaching 1 meter. that language allowed direct, invers (reverse?) and advanced 'polishing'. that'd been the days, man (deeeep sigh :) )
LIM = "x y - abs 3 > x 3 x y - x y - abs / * - x 49 * y 51 * + 100 / ?"no, i luv the former more. that's more hard to read :D (just think of the double-conditional)

anyway, what really did amaze me is the potential of 'lut'. i've never thought of the way u use it. it prompted me a lot of ideas. (completing, maybe, when retired ;) )

thx again
y

elwoodps

14th September 2005, 16:24

I'm doing an old black & white movie (12 O'clock High) using DVD-RB Pro (avisynth 2.5, CCE Basic 2.70, RemoveDirt 0.61) . This movie has a lot of dirt and scratches, so I'd think it would be substantialy improved by the proper use of RemoveDirt.

I've tried using it with settings similar to those kassandro and Major discussed for this type of movie earlier in this thread: RemoveDirt(grey=true, pthreshold=10, mthreshold=150)

But when I ran it with these settings the output video just displayed a black screen with the following message on it in big red letters: Script error:RemoveDirt does not have a named argument "grey"

So I tried it again without the grey=true argument so the line just read: RemoveDirt(pthreshold=10, mthreshold=150)

The result was the same except the message now reads:Script error:RemoveDirt does not have a named argument "pthreshold"

This kind of makes it look like RemoveDirt isn't accepting any variables. The filter seems to work when I use RemoveDirt() with no variables, but I don't suppose the defaults are optimal for this type of movie.

Any idea why this is happening and what I can do to fix it?

Elwood

Boulder

14th September 2005, 16:29

This is an old problem. You can avoid it by using a preset.

mg262

14th September 2005, 17:16

Also: the place to get support from the author is http://videoprocessing.11.forumer.com/index.php

elwoodps

14th September 2005, 19:17

@ mg262: Thanks! I didn't know that he had moved. The guide that was in the zip with the filter said to post questions in this thread.

@ Boulder: Thanks for your response. I'm not sure I understand what you mean when you say to use a "preset". Is that where you setup the desired variables under a name in the RemoveDirt.ini file and then access them using RemoveDirt(default=name)?

Elwood

Boulder

14th September 2005, 19:28

@ Boulder: Thanks for your response. I'm not sure I understand what you mean when you say to use a "preset". Is that where you setup the desired variables under a name in the RemoveDirt.ini file and then access them using RemoveDirt(default=name)?

Yes, that's right. The presets just work in a funny way in which the one that's later in the ini file will take over if it alters the same parameters as some preset before it - at least that's how I recall it.

That is,

some preset
pthreshold=20
mthreshold=100

some other preset
pthreshold=40

(note that I don't remember the exact syntax of the ini file, but you should get the idea)
When you call "some preset" in RemoveDirt, the parameters will actually be pthreshold=40,mthreshold=100.

There is a newer version of RemoveDirt at www.removedirt.de.tf but I don't know how well it works. Apparently it has a completely new design.

elwoodps

14th September 2005, 19:45

Many thanks Boulder! There's an example of how to setup the RemoveDirt.ini file in the guide that came with the filter.

Elwood

Boulder

14th September 2005, 19:47

If you create just one preset, there won't be any problems, of course:) The latest RemoveDirt version should tackle the jerky motion problem so it might be worth updating.

Heini011

26th February 2006, 19:12

hi folks,

i'am very satisfied with this quite simple RemoveDirtMC function. it is quite fast, does good denoising and yields very good compressibility. it is optimized for slight denoising of dvd-sources and avoiding block-artefacts.

you need: RemoveDirt, RemoveGrain, MVTools 1.1.1

function RemoveDirt(clip input, int limit, bool _grey)
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey)
}

function RemoveDirtMC(clip,int limit, bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)
bvec = clip.MVAnalyse(isb=false, blksize=8, delta=1, pel=2, truemotion=true, idx=1)
fvec = clip.MVAnalyse(isb=true, blksize=8, delta=1, pel=2, truemotion=true, idx=1)
backw = clip.MVFlow(bvec)
forw = clip.MVFlow(fvec)
clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

isb=false/true is in opposite to the mvtools-readme, but the results are correct this way
MVFlow in MVTools 1.1.1 is buggy for idx=1 at pel=2

Boulder

26th February 2006, 19:18

isb=false/true is in opposite to the mvtools-readme, but the results are correct this way
MVFlow in MVTools 1.1.1 is buggy for idx=1 at pel=2
Have you notified Fizick about this? If it's true, all the MC stuff is screwed up.

Did you try v1.2.1?

Fizick

26th February 2006, 22:50

Heini011,
1) isb works as always.
2) Are you "very satisfied" with "buggy" MVTools? :)

(but please post bug report in MVTools thread http://forum.doom9.org/showthread.php?t=102071&page=7)

Heini011

26th February 2006, 23:49

i have MVTools 1.1.1 and older versions only. so i can't say something to the newer ones.

idx=1 can't be used for MVFlow @ pel=2 -> bug!

if i remove SelectEvery(3,1) i get unexpected results at a scenechange with the isb setting according to the readme. same behavior in mvtools 0.985.

Fizick

27th February 2006, 00:46

please, post your bug report to MVTools thread with shortest script and more info.

Mug Funky

27th February 2006, 10:10

@ Heini011:

you might want to add some kind of highpass protection to that. i tried the exact same script a while ago and it behaved somewhat oddly on flicker and fades. if you subtract (or mt_makediff) a blurred clip from the original, grab motion on the result, then apply the compensation to the original, you can avoid strangenesses a bit. but they still will show up in a few cases. i have yet to find a good solution to this, but fizick's overlapped block mvanalyze looks to be part of it (thanks fizick!).

Didée

27th February 2006, 10:46

isb=false/true is in opposite to the mvtools-readme, but the results are correct this way
---
if i remove SelectEvery(3,1) i get unexpected results at a scenechange with the isb setting according to the readme. same behavior in mvtools 0.985.
Usually I stumble over this every time, too. :)
However, the behaviour IS like stated in the docu:

isb : allows to choose between a forward search ( between the current frame and the previous one ) [...]
isb = false means forward search

"is backward" means: search for backward-directed vectors. Making a compensation [n]<--[n+1] means that the [forward] frame n+1 is compensated to fit the [current] frame n, by means of backwards directed vectors.

So all is fine. Have

[n-1] --> [n] <-- [n+1]

in mind, and think of the vector directions, not of the frame ordering.

Heini011

2nd March 2006, 03:00

what you get and what you lose...

anton_foy

2nd March 2006, 23:22

Strange...I was just watching ""The Fearless Vampire Killers on DVD and a few minutes after I read this topic:)

I would really like to know what script you applied on these pics, it looks almost too good to be true. Im having the same noise/grain pattern in my footage so I would like to try that.

Thanks!

Heini011

3rd March 2006, 02:20

Hi anton_foy,

give this hq_filter a try, it is still a bit better ;-)

function RemoveTempGrain(clip input, int _mode)
{
rg = RemoveGrain(input, mode=_mode)
return TemporalRepair(rg, input)
}

function RemoveDirt(clip input, int limit, bool "_grey")
{
_grey=default(_grey, false)
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=16,grey=_grey)
}

function SRemoveDirtMC(clip,int limit, bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)
bvec = clip.MVAnalyse(isb=false, blksize=8, delta=1, pel=2, truemotion=true, idx=1)
fvec = clip.MVAnalyse(isb=true, blksize=8, delta=1, pel=2, truemotion=true, idx=1)
backw = clip.MVFlow(bvec)
forw = clip.MVFlow(fvec)
clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
dnc=clp.RemoveTempGrain(1)
clp=clp.SelectEvery(3,1)
dnc=dnc.SelectEvery(3,1)
clp=clp.SeeSaw(dnc,Sstr=0.26,Szp=12,SdampHi=20,bias=40)
return clp
}

function hq_filter(clip c)
{
c=c.SRemoveDirtMC(6)
c=c.YlevelsS(0,1.05,255,0,255)
c=c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=1,wide=false)
return c
}

SBaT

21st March 2006, 11:33

Thankyou for the nice script example, really usefull when I'm trying to grasp the use of MvTools and RemoveDirt with avs. However I'm quite confused atm after reading the end of RemoDirt manual as far if croping should be done before or after with a script like yours. As I'm encoding my 1080i.ts to 720p I have to crop atleast 2, 0, -4, -8 to get rid of black sides.

canuckerfan

12th September 2006, 03:06

Heini011, will your script wortk with mvtools 1.4.11?

superuser

7th November 2006, 05:44

Edited

dnaxx

6th April 2007, 17:24

Heini011, thanks for the script. But I don't get it running because "SeeSaw" is missing. Can you help me please?

Boulder

6th April 2007, 17:56

Why not use the search, it would give you the answer in no time.

rkalwaitis

14th November 2008, 17:35

Anyone a simple question.

So this script

function RemoveTempGrain(clip input, int _mode)
{
rg = RemoveGrain(input, mode=_mode)
return TemporalRepair(rg, input)
}

function RemoveDirt(clip input, int limit, bool "_grey")
{
_grey=default(_grey, false)
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=16,grey=_grey)
}

function SRemoveDirtMC(clip,int limit, bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)
bvec = clip.MVAnalyse(isb=false, blksize=8, delta=1, pel=2, truemotion=true, idx=1)
fvec = clip.MVAnalyse(isb=true, blksize=8, delta=1, pel=2, truemotion=true, idx=1)
backw = clip.MVFlow(bvec)
forw = clip.MVFlow(fvec)
clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
dnc=clp.RemoveTempGrain(1)
clp=clp.SelectEvery(3,1)
dnc=dnc.SelectEvery(3,1)
clp=clp.SeeSaw(dnc,Sstr=0.26,Szp=12,SdampHi=20,bias=40)
return clp
}

function RMBDenoise(clip input, bool "_grey", int "repmode")
{
clmode=17
clensed=Clense(input, grey=_grey)
restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode )
corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, neighbour2=clensed, gmthreshold=70, dist=0, debug=true, noise=12, noisy=12, grey=_grey)
return RemoveGrain(corrected, mode=clmode, modeU = _grey ? -1 : clmode )
}

function hq_filter(clip c)
{
c=c.SRemoveDirtMC(6)
c=c.YlevelsS(0,1.05,255,0,255)
c=c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=1,wide=false)
return c
}

and the one above are simple place as they are into the avs file? Correct? Of course I have to have the appropriate plugins.

R3Z

17th November 2008, 04:03

Anyone a simple question.

and the one above are simple place as they are into the avs file? Correct? Of course I have to have the appropriate plugins.

It will work that way and you can rename the .avs file to .avsi and it will autoload into memory without being needed to be Loaded manually.

rkalwaitis

17th November 2008, 07:51

R3Z

So I import the avsi file that I made. How do I call the function to do the work . RemoveDirt()? Like this?

Adub

17th November 2008, 19:52

So I import the avsi file that I made.

What do you mean? The whole point of the avsi file is that you just drop it into your plugins directory and you don't have to import anything.

Basically, save those functions above in a text file with ".avsi" as the extension. Drag this to your Avisynth plugins folder.

If you ever want to use those functions, just add something like "hq_filter()" or "RMBDenoise(_grey=false)" to the script that you are currently working on.

There is no need to import anything, you just call the functions like you would any normal plugin.

rkalwaitis

18th November 2008, 19:33

Merlin,

Thanks, that is what I meant. Now I get it. Thanks again.

Lyris

28th June 2009, 01:14

Heini011: thank you so much for this function. The accuracy is INCREDIBLE compared to other dirt and scratch removal plugins I've used!

The only issue I'm having is that it's difficult to keep film grain AND get rid of the dirt/scratches. I've resorted to degraining, using your function, and then re-graining the output to keep the original look.

Once again, thank you.

turnipzoink

18th September 2009, 19:27

Damn I'm jealous.
This is the only plugin I can't get to work? I tried email the author at gmx.de, not replying.

I tried each dll and none of them work. Avisynth is obviously loading them, but can't find the function. What am I doing wrong?

10298

videoFred

21st September 2009, 08:03

I tried each dll and none of them work. Avisynth is obviously loading them, but can't find the function. What am I doing wrong?

If you are using the Heini011 function, you must use HQ_filter() instead of RemoveDirt().

Off cource all dll's must be there, and Heini's function also.

PS: you need RemoveDirt.avs too.

Fred.

Paazabel

1st November 2009, 14:54

To the author: Thanks for this filter. I tried a bunch of different stuff that did not work on this clip. I'm sure many people have used it, but I wanted to make sure you got a "thanks" from me.

Sampling, stacked before-and-after (Lagarith, YV12, ~100M): http://www.sgt.cc/vid/Less-spotty.avi

(side note: Software rights can be a pain in the backside; would be nice if a great script could be "packaged" with all the other plugins that make it work. I know, I'm just lazy, but I spent about 2 hours debugging differences in plugin versions. It was worth it, but for an AVIsynth newb, it would have been improbable for them to stick with it.)

Sample frames:
I was particularly impressed with the second one since there are "real" dark spots in that frame and nothing else came remotely close to finding them. The first frame shows smoke rising over the hill which is in each frame and not processed as "noise." Very cool.
http://www.sgt.cc/pix/d/49030-1/despots-1.jpg

http://www.sgt.cc/pix/d/49033-1/despots-2.jpg

Not as interesting, but ... getting my money's worth out of the ol' computer!
http://www.sgt.cc/pix/d/49036-1/cpu-pegged.jpg
http://www.sgt.cc/pix/d/49038-1/vdub.jpg

Lyris

7th November 2009, 00:57

Question: is it possible to limit RemoveDirt to be colour-specific? That is, to only have it detect black (or nearly black) and white (or nearly white) areas as spots? It would improve accuracy...

LaTo

7th November 2009, 09:09

Question: is it possible to limit RemoveDirt to be colour-specific? That is, to only have it detect black (or nearly black) and white (or nearly white) areas as spots? It would improve accuracy...

Yes it is possible, without testing:

source = last
filtered = source.RemoveDirt()

dark = 32
bright = 196
mask = mt_lut(source,"x "+string(dark)+" < 255 x "+string(bright)+" > 255 0 ? ?",U=1,V=1).removegrain(19,-1,-1)

mt_merge(source,filtered,mask,luma=true)

Adjust dark (0 = do nothing in black areas) & bright (255 = do nothing in bright areas) to your taste...

Another way is to limit RemoveDirt where changes are really big:

source = last
filtered = source.RemoveDirt()

min = 16
max = 255
mask = mt_lutxy(source,filtered,"x y - abs "+string(min)+" < 0 x y - abs "+string(max)+" > 0 255 ? ?",U=1,V=1).removegrain(19,-1,-1)

mt_merge(source,filtered,mask,luma=true)
"min" is the minimum required and "max" is the maximum allowed

If you want to merge the two method:
source = last
filtered = source.RemoveDirt()

dark = 32
bright = 196
mask1 = mt_lut(source,"x "+string(dark)+" < 255 x "+string(bright)+" > 255 0 ? ?",U=1,V=1).removegrain(19,-1,-1)

min = 16
max = 255
mask2 = mt_lutxy(source,filtered,"x y - abs "+string(min)+" < 0 x y - abs "+string(max)+" > 0 255 ? ?",U=1,V=1).removegrain(19,-1,-1)

mt_merge(source,filtered,mt_logic(mask1,mask2,"min"),luma=true)

This one should be the best after tweaking the 4 parameters ;)

Paazabel

7th November 2009, 17:48

LaTo

8th November 2009, 14:50

Given the temporal nature of the filter (comparing one frame vs another), I'm not sure this would "improve performance" on most dirt. The comparison is the same frame-by-frame and it even removes blue dirt, from what I can tell, as well as grain and noise that are not near those thresholds.

Which is to say - academically, yes, I think you can do that, but I'm not sure it will have any meaningful "improvement" in either accuracy or speed.

Hum, your method remove spots... but also grain & details : oversmoothed effect! (artificial look)
It isn't really good to me :p

Some screenshots of my purpose:

Screenshot 1:
Original frame (http://img525.imageshack.us/img525/9092/1orig.jpg)
The filtered clip (Your) (http://img204.imageshack.us/img204/8137/1your.jpg)
The code above (Mine) (http://img204.imageshack.us/img204/5212/1mine.jpg)

Screenshot 2:
Original frame (http://img94.imageshack.us/img94/6134/2orig.jpg)
The filtered clip (Your) (http://img43.imageshack.us/img43/6799/2your.jpg)
The code above (Mine) (http://img517.imageshack.us/img517/2975/2mine.jpg)

The code:
DirectShowSource("Less-spotty.avi")
ConvertToYV12()

source = last.crop(0,0,-720,0)
filtered = last.crop(720,0,0,0)

str = 20

mt_merge(source,filtered,mt_lutxy(source,filtered,"x y - abs "+string(str)+" *",U=1,V=1).mt_expand(U=1,V=1).removegrain(19,-1,-1),luma=true,U=3,V=3)

str=20 give very good result...

No spot, but still detailled :)

After, you can use a more robust denoiser to remove the little noise while preserving details:

MCTemporalDenoise(chroma=true,flat=true,stabilize=true,sigma=2,strength=200,Tovershoot=2)

Screenshot 1 (http://img52.imageshack.us/img52/5632/1last.jpg) & Screenshot 2 (http://img43.imageshack.us/img43/1705/2last.jpg)

So, in conclusion, compare:
Before (your) (http://img204.imageshack.us/img204/8137/1your.jpg) - After (mine) (http://img52.imageshack.us/img52/5632/1last.jpg)
Before (your) (http://img43.imageshack.us/img43/6799/2your.jpg) - After (mine) (http://img43.imageshack.us/img43/1705/2last.jpg)

The choice is easy :devil:

Paazabel

8th November 2009, 15:08

Sweet results, LaTo.

Given the nature of this particular clip (someone spinning around with the camera in-hand), I'm not sure this is a "detectable" detail playing it back; no frame stands on the field of view long enough to notice. It looks good to me played as DVD.

However ...

I have a few scenes on that same tape where there is a definite "plastic" look from over-smoothing. I will try your filter on those because this one did not produce a good result in a few places.

LaTo

8th November 2009, 15:25

jmac698

3rd May 2010, 04:03

I'm having a bug in removedirt 1.0.
It works great for me on a very bad source, better than mc_spuds.
However, every 12 frames or so it's like it stops... the noise comes back, so watching the video there is a distracting noise flashing.
What could cause this? Are these frames somehow not being recognized?

Jenyok

13th February 2012, 13:22

Question.

Where could I get function TemporalRepair ?

Boulder

13th February 2012, 13:32

http://home.pages.at/kassandro/RemoveGrain/

Jenyok

14th February 2012, 07:16

This is a code of all known RemoveDirt and RemoveDirtMC functions, collected together.
MC - moution compensated.
Code is adapted to MVTools V2.

Didee
neuron2
Gavino

Any your suggestion, addition, correction and improvment of this code ?!

##
#
# File: RemoveDirt.avsi
#
# RemoveTempGrain, RemoveDirt and moution compensated RemoveDirt functions
#
# Functions:
#
# function RemoveTempGrain(clip input, int "_mode")
#
# function RemoveDirt(clip clp, int "repmode", bool "_grey")
# function RemoveDirtMC(clip clp, int "limit", bool "_grey")
#
# function RemoveDirt2(clip clp, int "limit", int "rgrain", bool "_grey")
# function RemoveDirtMC2(clip clp, int "limit", int "rgrain", bool "_grey")
#
# function RemoveDirt_HQ(clip clp, int tlimit, int rgrain, bool "_grey")
# function RemoveDirtMC_HQ(clip clp, int limit, int rgrain, bool "_grey")
#
#

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirts.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\repair.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\removegrain.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\repairt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_11_3\mvtools2.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FLUXSMOOTH_25_DLL_20040729\fluxsmooth.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\VAGUEDENOISER_25_DLL_20050926\vaguedenoiser.dll")

# =============================================================================
#
#
function RemoveTempGrain(clip input, int "_mode")
{
_mode = Default(_mode, 17)

rg = RemoveGrain(input, mode=_mode)

return TemporalRepair(rg, input)
}

# =============================================================================
#
#
function RemoveDirt(clip clp, int "repmode", bool "_grey")
{
_grey = Default(_grey, false)
repmode = Default(repmode, 16)

clmode = 17

clensed = Clense(clp, grey=_grey, cache=4)
sbegin = ForwardClense(clp, grey=_grey, cache=-1)
send = BackwardClense(clp, grey=_grey, cache=-1)
alt = Repair(SCSelect(clp, sbegin, send, clensed, debug=true), clp, mode=repmode, modeU = _grey ? -1 : repmode)
restore = Repair(clensed, clp, mode=repmode, modeU = _grey ? -1 : repmode)
corrected = RestoreMotionBlocks(clensed, restore, neighbour=clp, alternative=alt, gmthreshold=70, dist=1, \
dmode=2, debug=false, noise=10, noisy=12, grey=_grey)
return RemoveGrain(corrected, mode=clmode, modeU = _grey ? -1 : clmode)
}

# RemoveDirtMC2 by heini011 on 26 February 2006
# Suggested by johnmeyer on 09 June 2010
# Incorporated into Filmrestoration script by videoFred on 08 July 2010
# Fixed (vector direction) by nephilis on 01 August 2010
# Fixed (swapped parameters) by Emulgator to work with recent MVTools 2.5.10.1 on 20 September 2010
#
function RemoveDirtMC(clip clp, int "limit", bool "_grey")
{
_grey = Default(_grey, false)
limit = Default(limit, 6)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = RemoveDirt(clipa, repmode=limit, _grey=_grey)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function RemoveDirt2(clip clp, int "limit", int "rgrain", bool "_grey")
{
_grey = Default(_grey, false)
limit = Default(limit, 6)
rgrain = Default(rgrain, 2)

_dgr1 = 0.35 + rgrain * 0.3
_dgr2 = 0.45 + rgrain * 0.4
repmode = 1

clensed = Clense(clp, grey=_grey, cache=4)
restore = clp.FluxSmoothST(3 + 3 * rgrain, rgrain)
restore = Repair(restore, clp, mode=repmode, modeU=(_grey) ? -1 : repmode)
restore = (rgrain == 0) ? restore.RemoveGrain(1) : \
restore.VagueDenoiser(threshold=_dgr1, chromaT=_dgr1, nsteps=7, percent=75).RemoveGrain(1)
alt = clp.VagueDenoiser(threshold=_dgr2, chromaT=_dgr2, nsteps=7, percent=100).RemoveGrain(5)

return RestoreMotionBlocks(clensed, restore, neighbour=clp, alternative=alt, \
pthreshold=4 + 2 * rgrain, cthreshold=6 + 2 * rgrain, gmthreshold=40, \
dist=1, dmode=2, debug=false, noise=limit, noisy=12, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed, clp, alternative=alt, pthreshold=4, cthreshold=6, gmthreshold=40, dist=1, dmode=2, \
# debug=false, noise=limit, noisy=12, grey=_grey, show=true)
# return RestoreMotionBlocks(clensed, clp, alternative=alt, pthreshold=6, cthreshold=8, gmthreshold=40, dist=3, \
# tolerance=12, dmode=2, debug=false, noise=limit, noisy=12, grey=_ grey, show=false)

}

# RemoveDirtMC2 by heini011 on 26 February 2006
# Suggested by johnmeyer on 09 June 2010
# Incorporated into Filmrestoration script by videoFred on 08 July 2010
# Fixed (vector direction) by nephilis on 01 August 2010
# Fixed (swapped parameters) by Emulgator to work with recent MVTools 2.5.10.1 on 20 September 2010
#
function RemoveDirtMC2(clip clp, int "limit", int "rgrain", bool "_grey")
{
_grey = Default(_grey, false)
limit = Default(limit, 6)
rgrain = Default(rgrain, 2)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = RemoveDirt2(clipa, limit=limit, rgrain=rgrain, _grey=_grey)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function RemoveDirt_HQ(clip clp, int tlimit, int rgrain, bool "_grey")
{
_grey = Default(_grey, false)

_dgr = 0.4 + rgrain * 0.25

clensed = clp.RemoveTempGrain(1).FluxSmoothST(tlimit, rgrain)
restore = clp.VagueDenoiser(threshold=_dgr, nsteps=6, chromaT=_dgr, percent=100).RemoveGrain(1)
alt = restore

return RestoreMotionBlocks(clensed, restore, neighbour=clp, alternative=alt, pthreshold=3 + 2 * rgrain, \
cthreshold=3 + 2 * rgrain, gmthreshold=40, dist=1, dmode=1, debug=false, \
noise=tlimit + 1, noisy=12, grey=_grey)
}

function RemoveDirtMC_HQ(clip clp, int limit, int rgrain, bool "_grey")
{
_grey = Default(_grey, false)

super = MSuper(clp, pel=2, sharp=2)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=true)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
clipa = interleave(backw, clp, forw)
clipb = RemoveDirt_HQ(clipa, limit, rgrain, _grey=_grey)
clipc = SelectEvery(clipb, 3, 1)

return clipc
}

Jenyok

14th February 2012, 07:22

This is a code of all known RemoveNoise and various filters functions, used RemoveNoise, collected together.
MC - moution compensated.
Code is adapted to MVTools V2.

Didee
neuron2
Gavino

Any your suggestion, addition, correction and improvment of this code ?!
Also needed your help to replace function Deen() beta1 to Deen() beta2.

#
#
# File: RemoveNoise.avsi
#
# Moution compensated RemoveNoise and variuos filters functions
#
# Functions:
#
# function RemoveNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", bool "sharp", float "csharpen", bool "_grey")
# function HQ_Filter(clip clp)
# function MQ_Filter(clip clp)
# function LQ_Filter(clip clp)
#
# function RemoveNoiseMC_HQ(clip clp, int "tlimit", int "rgrain", float "csharp")
# function HQ_Filter2(clip clp)
#
# function RemoveHighNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", float "csharp", bool "_grey")
# function VLQ_Filter(clip clp)
# function VLQ_Filter_Ultimate(clip clp)
#
#

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirts.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\repairt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\removegrain.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\repair.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_11_3\mvtools2.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FLUXSMOOTH_25_DLL_20040729\fluxsmooth.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\VAGUEDENOISER_25_DLL_20050926\vaguedenoiser.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FFT3DFILTER_20070220\fft3dfilter.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\ZZZ_OLD\DEEN_25_DLL_20030813\deen.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\DEBLOCK_25_DLL_20060214\deblock.dll")

Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\SHARPFUNCTION_LIMITEDSHARPENFASTER\limitedsharpenfaster.avsi")
Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\SEESAW\seesaw.avsi")
Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.avsi")

#global idx_c = 4

# =============================================================================
#
#
function RemoveNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", bool "sharp", float "csharpen", bool "_grey")
{
_grey = Default(_grey, false)
rdlimit = Default(rdlimit, 11)
rgrain = Default(rgrain, 2)
denoise = Default(denoise, 8)
sharp = (rgrain < 1) ? Default(sharp, true) : Default(sharp, false)
csharpen = (rgrain > 2) ? Default(csharpen, 0.17) : (rgrain > 1) ? Default(csharpen, 0.15) : Default(csharpen, 0.13)

# global idx_c = idx_c + 1

dummy = clp.BlankClip(length=0)
csharpen = (sharp) ? csharpen : csharpen + 0.08
_dgr = 0.45 + rgrain * 0.4
cbs = 8
cov = (cbs > 4) ? cbs / 4 : 0
ccf = cbs * cbs / 64
cpn = (denoise > 12) ? 50 * ccf : (denoise > 8) ? 58 * ccf : (denoise > 5) ? 66 * ccf : 72 * ccf
csh = (sharp) ? 1 : 0

super = MSuper(clp, pel=2, sharp=csh)
bvec3 = MAnalyse(super, isb=true, blksize=cbs, delta=3, truemotion=true, pnew=cpn, overlap=cov)
bvec2 = MAnalyse(super, isb=true, blksize=cbs, delta=2, truemotion=true, pnew=cpn, overlap=cov)
bvec1 = MAnalyse(super, isb=true, blksize=cbs, delta=1, truemotion=true, pnew=cpn, overlap=cov)
fvec1 = MAnalyse(super, isb=false, blksize=cbs, delta=1, truemotion=true, pnew=cpn, overlap=cov)
fvec2 = MAnalyse(super, isb=false, blksize=cbs, delta=2, truemotion=true, pnew=cpn, overlap=cov)
fvec3 = MAnalyse(super, isb=false, blksize=cbs, delta=3, truemotion=true, pnew=cpn, overlap=cov)
backw1 = (rdlimit > 13) ? (rdlimit > 20) ? MFlow(clp, super, bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec1).Deblock(quant=16, aOffset=4, bOffset=4) : \
MFlow(clp, super, bvec1)
forw1 = (rdlimit > 13) ? (rdlimit > 20) ? MFlow(clp, super, fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec1).Deblock(quant=16, aOffset=4, bOffset=4) : \
MFlow(clp, super, fvec1)

# bvec2 = clip.MVAnalyse(isb=false, blksize=cbs, delta=2, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# fvec2 = clip.MVAnalyse(isb=true, blksize=cbs, delta=2, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# backw1 = (rdlimit > 13) ? (rdlimit > 20) ? clip.MVFlow(bvec1, idx=idx_c).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(bvec1, idx=idx_c).Deblock(quant=16, aOffset=4, bOffset=4) : \
# clip.MVFlow(bvec1, idx=idx_c)
# forw1 = (rdlimit > 13) ? (rdlimit > 20) ? clip.MVFlow(fvec1, idx=idx_c).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(fvec1, idx=idx_c).Deblock(quant=16, aOffset=4, bOffset=4) : \
# clip.MVFlow(fvec1, idx=idx_c)

clp = interleave(backw1, clp, forw1)
clp = clp.RemoveDirt2(rdlimit, rgrain, _grey)
# clp = clp.RemoveDirtMC2(rdlimit, rgrain, _grey)
dnc = (denoise == 0) ? clp.RemoveTempGrain(rgrain).SelectEvery(3, 1) : dummy
clp = clp.SelectEvery(3, 1)

dnc = MDegrain3(clp, super, bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, thSAD=190 + 15 * denoise, \
thSCD1=230 + 5 * denoise)

# dnc = (denoise == 0) ? dnc : clp.MVDenoise(bvec2, bvec1, fvec1, fvec2, thT=denoise, thSAD=190 + 15 * denoise, \
# thmv=40, thSCD1=230 + 5 * denoise)

vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=7, percent=75)
vid_mo = (rgrain == 1) ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")
clp = clp.SeeSaw(dnc, Sstr=csharpen, Szp=12, SdampHi=20, bias=40)

return clp
}

# for slight noisy video or spot/dirt removing only
#
function HQ_Filter(clip clp)
{
clp = clp.RemoveNoiseMC(rdlimit=6, rgrain=1, denoise=0)
clp = clp.LimitedSharpenFaster(Smode=4, strength=18, overshoot=0, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# medium noise, higher grain:
#
function MQ_Filter(clip clp)
{
clp = clp.RemoveNoiseMC(rdlimit=10, rgrain=2, denoise=8)
clp = clp.LimitedSharpenFaster(Smode=4, strength=20, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# high noise or analog capture:
#
function LQ_Filter(clip clp)
{
clp = clp.RemoveNoiseMC(rdlimit=18, rgrain=3, denoise=14)
clp = clp.VagueDenoiser(threshold=0.6, nsteps=6, chromaT=0.6, percent=75)
clp = clp.deen("a3d", rad=3, thrY=3, thrUV=5, min=0.25, tthY=2, tthUV=3, scd=6)
clp = clp.LimitedSharpenFaster(Smode=4, strength=24, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# =============================================================================
#
#
function RemoveNoiseMC_HQ(clip clp, int "tlimit", int "rgrain", float "csharp")
{
tlimit = Default(tlimit, 4)
rgrain = Default(rgrain, 1)
csharp = (rgrain > 1) ? Default(csharp, 0.14) : Default(csharp, 0.13)

# global idx_c = idx_c + 1

cbs = 4
ccf = cbs * cbs / 64
cpn = (tlimit > 6) ? 70 * ccf : 75 * ccf

super = MSuper(clp, pel=2, sharp=1)
bvec1 = MAnalyse(super, isb=true, blksize=cbs, delta=1, overlap=0, truemotion=true, pnew=cpn)
fvec1 = MAnalyse(super, isb=false, blksize=cbs, delta=1, overlap=0, truemotion=true, pnew=cpn)
backw1 = MFlow(clp, super, bvec1)
forw1 = MFlow(clp, super, fvec1)

# bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=1, overlap=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=1, overlap=0, truemotion=true, pnew=cpn, idx=idx_c)
# backw1 = clip.MVFlow(bvec1, idx=idx_c)
# forw1 = clip.MVFlow(fvec1, idx=idx_c)

dnc = interleave(backw1, clp, forw1)
dnc = dnc.RemoveDirt_HQ(tlimit, rgrain)
# dnc = dnc.RemoveDirtMC_HQ(tlimit, rgrain)
dnc = dnc.SelectEvery(3, 1)

return ((csharp == 0) ? dnc : clp.SeeSaw(dnc, Sstr=csharp, Szp=12, SdampHi=20, bias=40))
}

function HQ_Filter2(clip clp)
{
clp = clp.RemoveNoiseMC_HQ(tlimit=4, rgrain=1)
clp = clp.LimitedSharpenFaster(Smode=4, strength=15, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

#global idx_c = 25

# =============================================================================
#
#
function RemoveHighNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", float "csharp", bool "_grey")
{
_grey = Default(_grey, false)
rdlimit = Default(rdlimit, 24)
rgrain = Default(rgrain, 3)
denoise = Default(denoise, 15)
csharp = (rgrain > 2) ? Default(csharp, 0.32) : Default(csharp, 0.28)

# global idx_c = idx_c + 1

_dgr = 0.7 + rgrain * 0.5
cbs = 8
ccf = cbs * cbs / 64
cpn = (denoise > 12) ? 50 * ccf : 57 * ccf

super = MSuper(clp, pel=2, sharp=0)
bvec3 = MAnalyse(super, isb=true, blksize=cbs, delta=3, truemotion=true, pnew=cpn)
bvec2 = MAnalyse(super, isb=true, blksize=cbs, delta=2, truemotion=true, pnew=cpn)
bvec1 = MAnalyse(super, isb=true, blksize=cbs, delta=1, truemotion=true, pnew=cpn)
fvec1 = MAnalyse(super, isb=false, blksize=cbs, delta=1, truemotion=true, pnew=cpn)
fvec2 = MAnalyse(super, isb=false, blksize=cbs, delta=2, truemotion=true, pnew=cpn)
fvec3 = MAnalyse(super, isb=false, blksize=cbs, delta=3, truemotion=true, pnew=cpn)
backw1 = (rdlimit > 20) ? MFlow(clp, super, bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec1).Deblock(quant=16, aOffset=4, bOffset=4)
forw1 = (rdlimit > 20) ? MFlow(clp, super, fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec1).Deblock(quant=16, aOffset=4, bOffset=4)

# bvec3 = clip.MVAnalyse(isb=false, blksize=cbs, delta=3, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# bvec2 = clip.MVAnalyse(isb=false, blksize=cbs, delta=2, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec2 = clip.MVAnalyse(isb=true, blksize=cbs, delta=2, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec3 = clip.MVAnalyse(isb=true, blksize=cbs, delta=3, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# backw1 = (rdlimit > 20) ? clip.MVFlow(bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(bvec1).Deblock(quant=16, aOffset=4, bOffset=4)
# forw1 = (rdlimit > 20) ? clip.MVFlow(fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(fvec1).Deblock(quant=16, aOffset=4, bOffset=4)

clp = interleave(backw1, clp, forw1)
clp = clp.RemoveDirt2(limit=rdlimit, rgrain=rgrain, _grey=_grey)
# clp = clp.RemoveDirtMC2(limit=rdlimit, rgrain=rgrain, _grey=_grey)
clp = clp.SelectEvery(3, 1)

dnc = MDegrain3(clp, super, bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, thSAD=160 + 12 * denoise, \
thSCD1=255 + 4 * denoise)

# dnc = clp.MVDenoise(bvec3, bvec2, bvec1, fvec1, fvec2, fvec3, thT=denoise, thSAD=160 + 12 * denoise, \
# thmv=40, thSCD1=255 + 4 * denoise)

vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=6, percent=75)
vid_mo = (rgrain == 1) ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")

return ((csharp == 0) ? dnc : clp.SeeSaw(dnc, Sstr=csharp, Szp=12, SdampHi=20, bias=40))
}

function VLQ_Filter(clip clp)
{
clp = clp.RemoveHighNoiseMC(rdlimit=24, rgrain=3, denoise=16, csharp=0.3)
clp = clp.VagueDenoiser(threshold=1.0, nsteps=6, chromaT=1.2, percent=75)
clp = clp.deen("a3d", rad=3, thrY=4, thrUV=7, min=0.25, tthY=2, tthUV=3, scd=7)
clp = clp.LimitedSharpenFaster(Smode=4, strength=28, overshoot=1, wide=false, ss_x=1.2, ss_y=1.2)

return clp
}

function VLQ_Filter_Ultimate(clip clp)
{
clp = clp.RemoveHighNoiseMC(rdlimit=24, rgrain=3, denoise=16)
clp = clp.fft3dfilter(sigma=1.2, sharpen=0.4, plane=4)
clp = clp.deen("a3d", rad=3, thrY=4, thrUV=7, min=0.25, tthY=2, tthUV=3, scd=7)

return clp
}

Jenyok

14th February 2012, 11:41

This is a code of all known RemoveSpots and RemoveSpotsMC functions, collected together.
MC - moution compensated.
Code is adapted to MVTools V2.

Didee
neuron2
Gavino

Any your suggestion, addition, correction and improvment of this code ?!

#
#
# File: RemoveSpots.avsi
#
# RemoveSpots and moution compensated RemoveSpots functions
#
# Functions:
#
# function RemoveSpots(clip clp, int "repmode", bool "_grey")
# function RemoveSpotsMC(clip clp, int "limit", bool "_grey")
#
# function RemoveSpotsMC2(clip clp)
#
# function RemoveSpotsMC3(clip clp)
#
# function RemoveSpotsMC4(clip clp)
#
#

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirts.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\repair.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\removegrain.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\repairt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_11_3\mvtools2.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FLUXSMOOTH_25_DLL_20040729\fluxsmooth.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\VAGUEDENOISER_25_DLL_20050926\vaguedenoiser.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\DEFLICKER_25_DLL_20040816\deflicker.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\DESPOT_3_6_1\despot.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MASKTOOLS-V2_0A48\mt_masktools-25.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MASKTOOLS-V1_5_8\masktools.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FFT3DFILTER_20070220\fft3dfilter.dll")

LoadVirtualDubPlugin("C:\PROGRAM FILES\VIRTUALDUB 1.9.11\PLUGINS\spotremover.vdf", "SpotRemover", 11)

# =============================================================================
#
#
function RemoveSpots(clip clp, int "repmode", bool "_grey")
{
_grey = Default(_grey, false)
repmode = Default(repmode, 16)

clmode = 17

clensed = Clense(clp, grey=_grey, cache=4)
sbegin = ForwardClense(clp, grey=_grey, cache=-1)
send = BackwardClense(clp, grey=_grey, cache=-1)
alt = Repair(SCSelect(clp, sbegin, send, clensed, debug=true), clp, mode=repmode, modeU = _grey ? -1 : repmode )
restore = Repair(clensed, clp, mode=repmode, modeU = _grey ? -1 : repmode)
corrected = RestoreMotionBlocks(clensed, restore, neighbour=clp, alternative=alt, gmthreshold=70, dist=1, \
dmode=2, debug=false, noise=10, noisy=12, grey=_grey)

return corrected
}

function RemoveSpotsMC(clip clp, int "limit", bool "_grey")
{
_grey = Default(_grey, false)
limit = Default(limit, 16)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = RemoveSpots(clipa, repmode=limit, _grey=_grey)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function RemoveSpotsMC2(clip clp)
{
# create auxialiary smooth clip for easier motion detection
#
prefiltered = clp.DeFlicker(noise=6).Blur(1.58)

# this is for internal use by MVtools functions. MVtools' version must be 2.0 or higher.
#
super = MSuper(clp, pel=1, sharp=0)
superfilt = MSuper(prefiltered, pel=1, sharp=0)

# calculate the motion vectors
#
b_vec3 = MAnalyse(superfilt, isb=true, delta=3, blksize=8, overlap=2)
b_vec2 = MAnalyse(superfilt, isb=true, delta=2, blksize=8, overlap=2)
b_vec1 = MAnalyse(superfilt, isb=true, delta=1, blksize=8, overlap=2)
f_vec1 = MAnalyse(superfilt, isb=false, delta=1, blksize=8, overlap=2)
f_vec2 = MAnalyse(superfilt, isb=false, delta=2, blksize=8, overlap=2)
f_vec3 = MAnalyse(superfilt, isb=false, delta=3, blksize=8, overlap=2)

# create 4 additional clips: two predicted backward and two predicted forward
#
bf2 = clp.MCompensate(super, b_vec2)
bf1 = clp.MCompensate(super, b_vec1)
ff1 = clp.MCompensate(super, f_vec1)
ff2 = clp.MCompensate(super, f_vec2)

# combine the 5 frames (2+1+2), the central one undergoes the degrain filter
#
clp = interleave(bf2, bf1, clp.MDegrain3(super, b_vec1, f_vec1, b_vec2, f_vec2, b_vec3, f_vec3, plane=4, thsad=400), ff1, ff2)

# run SpotRemover over each block of 5 frames
# removing spots only, no smoothing
# !!! change to DSD:0 to turn the debug mode off
# !!! change to DSD:1 to turn the debug mode on
#
clp = clp.ConvertToRGB32()
clp = clp.SpotRemover("SSI:0 SSC:-5 SKW:3 SLI:2 SLC:2 SLD:0 DLS:24 DLC:16 DPC:16 DSF:0 DSD:0 MTX:16 MTY:12 MTL:25 MDX:1 MDY:1 MIH:0", "")
clp = clp.SpotRemover("SSI:0 SSC:-5 SKW:3 SLI:2 SLC:2 SLD:0 DLS:26 DLC:18 DPC:10 DSF:2 DSD:0 MTX:16 MTY:12 MTL:22 MDX:1 MDY:1 MIH:0 ", "")
clp = clp.SpotRemover("SSI:7 SSC:0 SKW:5 SLI:2 SLC:2 SLD:0 DLS:26 DLC:18 DPC:16 DSF:2 DSD:0 MTX:16 MTY:12 MTL:22 MDX:1 MDY:1 MIH:0", "")

# clp = clp.ConvertBackToYUY2()
clp = clp.ConvertToYV12()

# discard extra frames, keep only the central one
#
clp = clp.SelectEvery(5, 3)

# compensate for the SpotRemover lag. We will loose the last frame in the movie, but the audio will be in sync
#
# clp = clp.Trim(1, 0)

# enhance the contrast (optional)
#
clp = clp.ColorYUV(gain_y=0, off_y=10, gamma_y=0, cont_y=80, cont_u=0, cont_v=0)

return clp
}

global Width2 = 768
global Height2 = 576

# =============================================================================
#
#
function RemoveSpotsMC3(clip clp)
{
# Constants
#
block_size = 16
block_over = 2

pref = clp.deflicker(border=150, percent=70, lmin=5, lmax=240)
pref_super = pref.MSuper(hpad=block_size, vpad=block_size, pel=2, sharp=2, rfilter=2)
bvec3 = MAnalyse(pref_super,isb=true, truemotion=false, delta=3, blksize=block_size, search=3, overlap=block_over, lambda=2000, badSAD=18800)
bvec2 = MAnalyse(pref_super,isb=true, truemotion=false, delta=2, blksize=block_size, search=3, overlap=block_over, lambda=2000, badSAD=18800)
bvec1 = MAnalyse(pref_super,isb=true, truemotion=false, delta=1, blksize=block_size, search=3, overlap=block_over, lambda=2000, badSAD=18800)
fvec1 = MAnalyse(pref_super,isb=false, truemotion=false, delta=1, blksize=block_size, search=3, overlap=block_over, lambda=2000, badSAD=18800)
fvec2 = MAnalyse(pref_super,isb=false, truemotion=false, delta=2, blksize=block_size, search=3, overlap=block_over, lambda=2000, badSAD=18800)
fvec3 = MAnalyse(pref_super,isb=false, truemotion=false, delta=3, blksize=block_size, search=3, overlap=block_over, lambda=2000, badSAD=18800)
backw1 = MFlow(clp, pref_super, bvec1)
backw2 = MFlow(clp, pref_super, bvec2)
backw3 = MFlow(clp, pref_super, bvec3)
forw1 = MFlow(clp, pref_super, fvec1)
forw2 = MFlow(clp, pref_super, fvec2)
forw3 = MFlow(clp, pref_super, fvec3)

clp=interleave(backw3, backw2, backw1, clip, forw1, forw2, forw3)

clp=clp.DeSpot(p1=30, p2=10, mthres=45, pwidth=Width2, pheight=Height2, minpts=10, dilate=0, ranked=true, p1percent=5, mscene=98, \
merode=35, mwidth=20, mheight=10, sign=2, show=0, seg=0, color=true, motpn=true, fitluma=true, blur=1) \
.DeSpot(p1=25, p2=14, mthres=25, pwidth=Width2, pheight=Height2, minpts=3, dilate=3, p1percent=5, mscene=98, \
merode=35, sign=-2, show=0, seg=2, color=true, motpn=true, fitluma=true, blur=1)

# clp=clp.Despot(p1=28, p2=8, mthres=45, pwidth=850, pheight=850, minpts=10, dilate=0,ranked=true, p1percent=5, mscene=98, \
# merode=35, mwidth=20, mheight=10, sign=2, show=0, seg=0, color=true, motpn=true, fitluma=true, blur=1)
# # \ .Despot(p1=25, p2=14, mthres=25, pwidth=555, pheight=555, minpts=3, dilate=3, p1percent=5, mscene=98, \
# # merode=35, sign=-2, show=0, seg=2, color=true, motpn=true, fitluma=true, blur=1)

# clp=clp.RemoveDirtMC2(_grey=false)

clp=clp.SelectEvery(7, 3)

return clp
}

# =============================================================================
#
#
function RemoveSpotsMC4(clip clp)
{
prefiltered = FFT3Dfilter(clp, sigma=6, plane=4, bw=32, bh=32, ow=8, oh=8, bt=3)
prefilteredSuper = MSuper(prefiltered, pel=2, sharp=2)
osuper = MSuper(clp, pel=2, sharp=2)
vb = MAnalyse(prefilteredSuper, isb=true, delta=1, search=3, overlap=2, lambda=2000, badSAD=18800)
cb = MFlow(osuper, prefilteredSuper, vb, thSCD1=400)
sadb = MMask(clp, vb, ml=100, thSCD1=400, gamma=1, kind=1)
vf = MAnalyse(prefilteredSuper, isb=false, delta=1, search=3, overlap=2, lambda=2000, badSAD=18800)
cf = MFlow(osuper, prefilteredSuper, vf, thSCD1=400)
sadf = MMask(clp, vf, ml=100, thSCD1=400, gamma=1, kind=1)

msadf = MT_Binarize(sadf, 20, upper=true)
msadb = MT_Binarize(sadb, 20, upper=true)
msad = MT_Logic(msadf, msadb, "or")
msad = MT_Expand(msad)
msadi = Interleave(msad, msad, msad)
clp = Interleave(cf, clp, cb)

# big black spots
clp = clp.DeSpot(sign=1, pwidth=45, pheight=45, p1=15, p2=8, mthres=8, color=true, extmask=msadi)

# thin white lines
clp = clp.DeSpot(sign=-1, pwidth=140, pheight=100, p1=8, p2=1, mthres=5, extmask=msadi)

# thick hair lines (avoid real hair)
clp = clp.DeSpot(sign=-1, pwidth=140, pheight=100, p1=15, p2=13, mthres=13, merode=24, extmask=msadi)

# lines contacting motion, avoid motion areas
clp = clp.DeSpot(sign=-1, pwidth=140, pheight=100, p1=15, p2=13, mthres=13, merode=16, seg=1, extmask=msadi)

# small dots and dust
clp = clp.DeSpot(pwidth=6, pheight=5, p1=15, p2=8, mthres=12, tsmooth=2, extmask=msadi)

clp = clp.DeSpot(sign=2, pwidth=Width2, pheight=Height2, p1=10, p2=6, mthres=8, tsmooth=4, seg=2, blur=4, p1percent=2, extmask=msadi)

# Long horizontal scratches
clp = clp.DeSpot(sign=-1, pwidth=100, pheight=2, p1=15, p2=8, mthres=12, color=Color, extmask=msadi)

# Small compact spots
clp = clp.DeSpot(sign=-1, pwidth=10, pheight=10, p1=15, p2=8, mthres=12, color=Color, extmask=msadi)

clp = clp.SelectEvery(3, 1)

return clp
}

Jenyok

14th February 2012, 16:04

This is a code of all known RemoveDust and RemoveDustMC functions, collected together.
MC - moution compensated.
Code is adapted to MVTools V2.

Didee
neuron2
Gavino

Any your suggestion, addition, correction and improvment of this code ?!

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\removegrain.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\repair.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\repairt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\ssetools.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_11_3\mvtools2.dll")

# =============================================================================
#
#
function RemoveDustOld(clip clp, int "_mode", bool "_grey")
{
_grey = Default(_grey, false)
_mode = Default(_mode, 5)

repmode = 2
_modeU = (_grey) ? -1 : repmode

clensed = Clense(clp, grey=_grey)
rep = Repair(clensed, clp, mode=repmode, modeU = _modeU)

return (RemoveGrain(rep, mode=_mode, modeU = _modeU))
}

# =============================================================================
#
#
function RemoveDustOldMC(clip clp, int "_smooth", bool "_grey")
{
_smooth = Default(_smooth, 0)
_grey = Default(_grey, false)
repmode = 2
clmode = 4

quad = Quadruple(clp, grey=_grey) # each pixel is quadrupled, SSETools are needed
qpel = RemoveGrain(quad, mode=12, modeU=-1) # blur the luma for searching motion vectors

super = MSuper(qpel, pel=2)
# bvectors3 = MAnalyse(super, isb=true, blksize=16, delta=3, truemotion=true, lambda=2000, badSAD=18800) # for a backward search isb should be "true" not "false"
# bvectors2 = MAnalyse(super, isb=true, blksize=16, delta=2, truemotion=true, lambda=2000, badSAD=18800) # for a backward search isb should be "true" not "false"
bvectors = MAnalyse(super, isb=true, blksize=16, delta=1, truemotion=true, lambda=2000, badSAD=18800) # for a backward search isb should be "true" not "false"
fvectors = MAnalyse(super, isb=false, blksize=16, delta=1, truemotion=true, lambda=2000, badSAD=18800) # for a forward search isb should be "false" not "true"
# fvectors2 = MAnalyse(super, isb=false, blksize=16, delta=2, truemotion=true, lambda=2000, badSAD=18800) # for a forward search isb should be "false" not "true"
# fvectors3 = MAnalyse(super, isb=false, blksize=16, delta=3, truemotion=true, lambda=2000, badSAD=18800) # for a forward search isb should be "false" not "true"
backward = MCompensate(quad, super, bvectors)
forward = MCompensate(quad, super, fvectors)
# backward2 = MCompensate(quad, super, bvectors2)
# forward2 = MCompensate(quad, super, fvectors2)
# backward3 = MCompensate(quad, super, bvectors3)
# forward3 = MCompensate(quad, super, fvectors3)

clipb = interleave(backward3, backward2, backward)
clipf = interleave(forward3, forward2, forward)

# bvectors = MVAnalyse(qpel, blksize=16, lambda=200, isb=true)
# fvectors = MVAnalyse(qpel, blksize=16, lambda=200, isb=false)
# backward = MVCompensate(quad, bvectors, mode=1)
# forward = MVCompensate(quad, fvectors, mode=1)

# clensed = mcclense(quad, backward, forward, grey=_grey).shrinkby2(grey=_grey) # SSETools are needed for shrinking
clensed = Clense(quad, backward, forward, grey=_grey).shrinkby2(grey=_grey) # SSETools are needed for shrinking
# clensed = Clense(quad, clipb, clipf, grey=_grey).shrinkby2(grey=_grey) # SSETools are needed for shrinking

rep = Repair(clensed, clp, mode=repmode, modeU=(_grey) ? -1 : repmode)
rg = RemoveGrain(rep, mode=clmode, modeU=(_grey) ? -1 : clmode)

clipc = TemporalRepair(rg, rep, grey=_grey, smooth=_smooth)

# clipc = SelectEvery(3, 1)

return clipc
}

# =============================================================================
#
#
function RemoveDust(clip clp, int "_mode", bool "_grey")
{
_grey = Default(_grey, false)
_mode = Default(_mode, 5)

repmode = 2
_modeU = (_grey) ? -1 : repmode

clensed = Clense(clp, grey=_grey)
rep = Repair(clensed, clp, mode=repmode, modeU = _modeU )
rg = RemoveGrain(rep, mode=_mode, modeU = _modeU)

return (TemporalRepair(rg, rep, grey=_grey, smooth=true))
}

function RemoveDustMC(clip clp, int "mode", bool "_grey")
{
_grey = Default(_grey, false)
mode = Default(mode, 5)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = RemoveDust(clipa, _mode=mode, _grey=_frey)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function RemoveDust2(clip clp, int "repmode", int "_smooth", bool "_grey")
{
_grey = Default(_grey, false)
repmode = Default(repmode, 16)
_smooth = Default(_smooth, 0)

clmode = 4

clensed = Clense(clp, grey=_grey)
rep = Repair(clensed, clp, mode=repmode, modeU=(_grey) ? -1 : repmode)
rg = RemoveGrain(rep, mode=clmode, modeU=(_grey) ? -1 : clmode)

return (TemporalRepair(rg, rep, grey=_grey, smooth=_smooth))
}

function RemoveDustMC2(clip clp, int "mode", int "smooth", bool "_grey")
{
_grey = Default(_grey, false)
mode = Default(mode, 16)
smooth = Default(smooth, 0)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = RemoveDust2(clipa, _mode=mode, _smooth=smooth, _grey=_grey)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function RemoveDust3(clip clp, int "repmode", int "TRsmooth", int "RGmode", int "RGmodeChroma", bool "_grey")
{
_grey = Default(_grey, false)
repmode = Default(repmode, 9)
TRsmooth = Default(TRsmooth, 0)
RGmode = Default(RGmode, 2) # 1,2,5,17,18
RGmodeChroma = Default(RGmodeChroma, 1) # 1,2,5,17,18

clensed = Clense(clp, grey=_grey)
rep = Repair(clensed, clp, mode=repmode, modeU=(_grey) ? -1 : repmode)
rg = RemoveGrain(rep, mode=RGmode, modeU=(_grey) ? -1 : RGmodeChroma, modeV=(_grey) ? -1 : RGmodeChroma)

return (TemporalRepair(rg, rep, grey=_grey, smooth=TRsmooth))
}

function RemoveDustMC3(clip clp, int "mode", int "smooth", int "RGmode", int "RGmodeChroma", bool "_grey")
{
_grey = Default(_grey, false)
mode = Default(mode, 9)
smooth = Default(smooth, 0)
RGmode = Default(RGmode, 2) # 1,2,5,17,18
RGmodeChroma = Default(RGmodeChroma, 1) # 1,2,5,17,18

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = RemoveDust3(clipa, repmode=mode, TRsmooth=smooth, RGmode=RGmode, RGmodeChroma=RGmodeChroma, _grey=_grey)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function LRemoveDust_YUY2(clip clp, int "clmode", int "limit")
{
clmode = Default(clmode, 17)
limit = Default(limit, 2)

repmode = 2

clensed = Clense(clp, grey=true)
rep = Repair(clensed, clp, mode=repmode, modeU=-1)
rg = RemoveGrain(rep, mode=clmode, modeU=-1)

return (LimitChange(rg, clp, limit, limitU=255))
}

function LRemoveDust_YUY2MC(clip clp, int "mode", int "limit")
{
mode = Default(mode, 17)
limit = Default(limit, 2)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = LRemoveDust_YUY2(clipa, clmode=mode, limit=limit)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function LRemoveDust_YV12(clip clp, int "clmode", int "limit")
{
clmode = Default(clmode, 17)
limit = Default(limit, 2)

repmode = 2

clensed = Clense(clp)
rep = Repair(clensed, clp, mode=repmode)
rg = RemoveGrain(rep, mode=clmode)

return (LimitChange(rg, clp, limit))
}

function LRemoveDust_YV12MC(clip clp, int "mode", int "limit")
{
mode = Default(mode, 17)
limit = Default(limit, 2)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = LRemoveDust_YV12(clipa, clmode=mode, limit=limit)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

# =============================================================================
#
#
function LRemoveDust(clip clp, int "_mode", int "limit")
{
_mode = Default(_mode, 2)
limit = Default(limit, 4)

LL = string(limit)
STR = "x "+LL+" + y < y "+LL+" - x "+LL+" - y > y "+LL+" + x ? ?"
repmode = 2

clensed = Clense(clp)
rep = Repair(clensed, clp, mode=repmode)
rg = RemoveGrain(rep, mode=_mode)
trep = TemporalRepair(rg, rep)
trep2 = isyuy2(clp) ? trep.ConvertToYV12() : trep
input2 = isyuy2(clp) ? clp.ConvertToYV12() : clp
clp = (limit != 0) ? yv12lutxy(trep2, input2, yexpr=STR, U=2, V=2) : clp
out = isyuy2(clp) ? clp.ConvertToYUY2().MergeChroma(trep) : clp

return (out)
}

function LRemoveDustMC(clip clp, int "mode", int "limit")
{
mode = Default(mode, 2)
limit = Default(limit, 4)

super = MSuper(clp, pel=2, sharp=2)
# bvec3 = MAnalyse(super, isb=true, blksize=8, delta=3, truemotion=false)
# bvec2 = MAnalyse(super, isb=true, blksize=8, delta=2, truemotion=false)
bvec = MAnalyse(super, isb=true, blksize=8, delta=1, truemotion=false)
fvec = MAnalyse(super, isb=false, blksize=8, delta=1, truemotion=true)
# fvec2 = MAnalyse(super, isb=false, blksize=8, delta=2, truemotion=true)
# fvec3 = MAnalyse(super, isb=false, blksize=8, delta=3, truemotion=true)
backw = MFlow(clp, super, bvec)
forw = MFlow(clp, super, fvec)
# backw2 = MFlow(clp, super, bvec2)
# forw2 = MFlow(clp, super, fvec2)
# backw3 = MFlow(clp, super, bvec3)
# forw3 = MFlow(clp, super, fvec3)

clipa = interleave(backw, clp, forw)
# clipa = interleave(backw2, backw, clp, forw, forw2)
# clipa = interleave(backw3, backw2, backw, clp, forw, forw2, forw3)

clipb = LRemoveDust(clipa, _mode=mode, limit=limit)

clipc = SelectEvery(clipb, 3, 1)
# clipc = SelectEvery(clipb, 5, 1)
# clipc = SelectEvery(clipb, 7, 1)

return clipc
}

cobo

18th February 2012, 16:09

Can anyone see why I should be getting this error message from the script below? I don't see where there's a problem with the script.

Avisynth open failure:
Interleave: videos must be of the same size
(I:\2avi_RemoveSpotsMC4.avs, line 27)
(I:\2avi_RemoveSpotsMC4.avs, line 8)

LoadPlugin("C:\Program Files\AviSynth 2.58\plugins\All\mvtools2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.58\plugins\MaskTools v2.0a48\mt_masktools-25.dll")
LoadPlugin("C:\Program Files\AviSynth 2.58\plugins\All\FFT3Dfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.58\plugins\All\Despot.dll")

avisource("2.avi")

RemoveSpotsMC4()

function RemoveSpotsMC4(clip clp)
{
prefiltered = FFT3Dfilter(clp, sigma=6, plane=4, bw=32, bh=32, ow=8, oh=8, bt=3)
prefilteredSuper = MSuper(prefiltered, pel=2, sharp=2)
osuper = MSuper(clp, pel=2, sharp=2)
vb = MAnalyse(prefilteredSuper, isb=true, delta=1, search=3, overlap=2, lambda=2000, badSAD=18800)
cb = MFlow(osuper, prefilteredSuper, vb, thSCD1=400)
sadb = MMask(clp, vb, ml=100, thSCD1=400, gamma=1, kind=1)
vf = MAnalyse(prefilteredSuper, isb=false, delta=1, search=3, overlap=2, lambda=2000, badSAD=18800)
cf = MFlow(osuper, prefilteredSuper, vf, thSCD1=400)
sadf = MMask(clp, vf, ml=100, thSCD1=400, gamma=1, kind=1)

msadf = MT_Binarize(sadf, 20, upper=true)
msadb = MT_Binarize(sadb, 20, upper=true)
msad = MT_Logic(msadf, msadb, "or")
msad = MT_Expand(msad)
msadi = Interleave(msad, msad, msad)
clp = Interleave(cf, clp, cb)

# big black spots
clp = clp.DeSpot(sign=1, pwidth=45, pheight=45, p1=15, p2=8, mthres=8, color=true, extmask=msadi)

# thin white lines
clp = clp.DeSpot(sign=-1, pwidth=140, pheight=100, p1=8, p2=1, mthres=5, extmask=msadi)

# thick hair lines (avoid real hair)
clp = clp.DeSpot(sign=-1, pwidth=140, pheight=100, p1=15, p2=13, mthres=13, merode=24, extmask=msadi)

# lines contacting motion, avoid motion areas
clp = clp.DeSpot(sign=-1, pwidth=140, pheight=100, p1=15, p2=13, mthres=13, merode=16, seg=1, extmask=msadi)

# small dots and dust
clp = clp.DeSpot(pwidth=6, pheight=5, p1=15, p2=8, mthres=12, tsmooth=2, extmask=msadi)

clp = clp.DeSpot(sign=2, pwidth=Width2, pheight=Height2, p1=10, p2=6, mthres=8, tsmooth=4, seg=2, blur=4, p1percent=2, extmask=msadi)

# Long horizontal scratches
clp = clp.DeSpot(sign=-1, pwidth=100, pheight=2, p1=15, p2=8, mthres=12, color=Color, extmask=msadi)

# Small compact spots
clp = clp.DeSpot(sign=-1, pwidth=10, pheight=10, p1=15, p2=8, mthres=12, color=Color, extmask=msadi)

clp = clp.SelectEvery(3, 1)

return clp
}

Didée

18th February 2012, 16:28

Because the "MFlow" calls do have wrong 1st clip arguments:
cb = MFlow(osuper, prefilteredSuper, vb, thSCD1=400)
...
cf = MFlow(osuper, prefilteredSuper, vf, thSCD1=400)
Those should be "clp" instead of "osuper", together with "osuper" instead of "prefilteredSuper".
Or, maybe it should be "prefiltered" and "prefilteredSuper", I'm not quite sure about the intended workflow-logic.

In any case, the 1st clip argument must be a "normal" clip, and only the 2nd clip argument be a superclip.

cobo

18th February 2012, 18:56

Thanks Didée. Either way gets rid of the initial error message. Unfortunately after dealing with a couple of other ones caused by "Width2" and "COLOR" both versions of the script result in these responses:

Avisynth read error: CAVIStreamSynth: System exception - Access violation at 0x7c911895, reading

Runtime Error!
Program: I:\Virtualdub.exe
R6025
- pure virtual function call

Using QuEnc or MPCH produces the same complaints.

Jenyok

18th February 2012, 22:07

cobo

Use "clp" instead "osuper" in MFlow() function, first clip argument...
I tried, really work.

cobo

18th February 2012, 22:56

I'm still getting the same "Avisynth read error" and "Runtime Error" as in my last post.

Didée

18th February 2012, 23:26

The scipt is crashing for me, too. (After fixing width2/height2/color, by defining them.)

I cannot see why it is, but for me, it is the "MFlow" lines that cause the crash. Swap "MFlow" for "MCompensate", and it works.

cobo

18th February 2012, 23:46

My problem seems to have been caused by using an older version of Despot - 3.5.1.0. The latest version 3.6.1.0 allows me to open the script in Vdub without any error. I'm encoding with QuEnc without a problem for the first five minutes, but there's 14 hours to go yet on a two minute clip.

Didée

19th February 2012, 00:09

14 hours for those 2 minutes? Heaven help, what hardware do you have? When I start that script in Vdub, it's estimating alittle over 13 minutes for the job!

Still I'm wondering why for me it crashes when using MFlow ...

cobo

19th February 2012, 00:19

P4 2.4GHz with 1GB memory. I restarted it on my other PC which has a P4 2.66GHz CPU and 2GB memory and now it's only going to take 12 hours. I used SetMemoryMax(32), I don't know if setting it higher might speed it up.

Edit: Big difference! Deleting the SetMemoryMax line changes the encode time to 40 minutes.

Didée

19th February 2012, 00:29

OOOooohhh, sure! Wait a moment ...

With your script from above -

SetMemoryMax(32) ==> 0.4 fps

SetMemoryMax(128) ==> 3.5 fps

That's on an i7-860, your numbers surely wil be lower. However, SMM(32) definetly is much too little for such a script! As soon as MVTools is involved in any way, I usually wouldn't start lower than SMM(256).

cobo

19th February 2012, 01:52

The encode finished without crashing. I think I got about 1.58fps.

canuckerfan

6th March 2012, 07:25

^Jenyok, I seem to have encountered a typo in the code. In the function RemoveSpotsMC3, you have this piece of code:

.
.
.
forw2 = MFlow(clp, pref_super, fvec2)
forw3 = MFlow(clp, pref_super, fvec3)

clp=interleave(backw3, backw2, backw1, clip, forw1, forw2, forw3)

clp=clp.DeSpot(p1=30, p2=10, mthres=45, pwidth=Width2, pheight=Height2, minpts=10, dilate=0, ranked=true, p1percent=5, mscene=98, \
merode=35, mwidth=20, mheight=10, sign=2, show=0, seg=0, color=true, motpn=true, fitluma=true, blur=1) \
.DeSpot(p1=25, p2=14, mthres=25, pwidth=Width2, pheight=Height2, minpts=3, dilate=3, p1percent=5, mscene=98, \
merode=35, sign=-2, show=0, seg=2, color=true, motpn=true, fitluma=true, blur=1)
.
.
.

which I think should be this:

.
.
.
forw2 = MFlow(clp, pref_super, fvec2)
forw3 = MFlow(clp, pref_super, fvec3)

clp=interleave(backw3, backw2, backw1, clp, forw1, forw2, forw3)

clp=clp.DeSpot(p1=30, p2=10, mthres=45, pwidth=Width2, pheight=Height2, minpts=10, dilate=0, ranked=true, p1percent=5, mscene=98, \
merode=35, mwidth=20, mheight=10, sign=2, show=0, seg=0, color=true, motpn=true, fitluma=true, blur=1) \
.DeSpot(p1=25, p2=14, mthres=25, pwidth=Width2, pheight=Height2, minpts=3, dilate=3, p1percent=5, mscene=98, \
merode=35, sign=-2, show=0, seg=2, color=true, motpn=true, fitluma=true, blur=1)
.
.
.

By the way, I've tried all these RemoveSpot functions and what is the difference between them all?

Eluent

8th November 2012, 01:02

@Jenyok : I've added grey option to the filter functions :
#
#
# File: RemoveNoise.avsi
#
# Moution compensated RemoveNoise and variuos filters functions
#
# Functions:
#
# function RemoveNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", bool "sharp", float "csharpen", bool "_grey")
# function HQ_Filter(clip clp)
# function MQ_Filter(clip clp)
# function LQ_Filter(clip clp)
#
# function RemoveNoiseMC_HQ(clip clp, int "tlimit", int "rgrain", float "csharp")
# function HQ_Filter2(clip clp)
#
# function RemoveHighNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", float "csharp", bool "_grey")
# function VLQ_Filter(clip clp)
# function VLQ_Filter_Ultimate(clip clp)
#
#

#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirts.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\repairt.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\removegrain.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\repair.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_11_3\mvtools2.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FLUXSMOOTH_25_DLL_20040729\fluxsmooth.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\VAGUEDENOISER_25_DLL_20050926\vaguedenoiser.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FFT3DFILTER_20070220\fft3dfilter.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\ZZZ_OLD\DEEN_25_DLL_20030813\deen.dll")
#~ LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\DEBLOCK_25_DLL_20060214\deblock.dll")

#~ Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\SHARPFUNCTION_LIMITEDSHARPENFASTER\limitedsharpenfaster.avsi")
#~ Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\SEESAW\seesaw.avsi")
#~ Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.avsi")

#global idx_c = 4

# =============================================================================
#
#
function RemoveNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", bool "sharp", float "csharpen", bool "_grey")
{
_grey = Default(_grey, false)
rdlimit = Default(rdlimit, 11)
rgrain = Default(rgrain, 2)
denoise = Default(denoise, 8)
sharp = (rgrain < 1) ? Default(sharp, true) : Default(sharp, false)
csharpen = (rgrain > 2) ? Default(csharpen, 0.17) : (rgrain > 1) ? Default(csharpen, 0.15) : Default(csharpen, 0.13)

# global idx_c = idx_c + 1

dummy = clp.BlankClip(length=0)
csharpen = (sharp) ? csharpen : csharpen + 0.08
_dgr = 0.45 + rgrain * 0.4
cbs = 8
cov = (cbs > 4) ? cbs / 4 : 0
ccf = cbs * cbs / 64
cpn = (denoise > 12) ? 50 * ccf : (denoise > 8) ? 58 * ccf : (denoise > 5) ? 66 * ccf : 72 * ccf
csh = (sharp) ? 1 : 0

super = MSuper(clp, pel=2, sharp=csh)
bvec3 = MAnalyse(super, isb=true, blksize=cbs, delta=3, truemotion=true, pnew=cpn, overlap=cov)
bvec2 = MAnalyse(super, isb=true, blksize=cbs, delta=2, truemotion=true, pnew=cpn, overlap=cov)
bvec1 = MAnalyse(super, isb=true, blksize=cbs, delta=1, truemotion=true, pnew=cpn, overlap=cov)
fvec1 = MAnalyse(super, isb=false, blksize=cbs, delta=1, truemotion=true, pnew=cpn, overlap=cov)
fvec2 = MAnalyse(super, isb=false, blksize=cbs, delta=2, truemotion=true, pnew=cpn, overlap=cov)
fvec3 = MAnalyse(super, isb=false, blksize=cbs, delta=3, truemotion=true, pnew=cpn, overlap=cov)
backw1 = (rdlimit > 13) ? (rdlimit > 20) ? MFlow(clp, super, bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec1).Deblock(quant=16, aOffset=4, bOffset=4) : \
MFlow(clp, super, bvec1)
forw1 = (rdlimit > 13) ? (rdlimit > 20) ? MFlow(clp, super, fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec1).Deblock(quant=16, aOffset=4, bOffset=4) : \
MFlow(clp, super, fvec1)

# bvec2 = clip.MVAnalyse(isb=false, blksize=cbs, delta=2, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# fvec2 = clip.MVAnalyse(isb=true, blksize=cbs, delta=2, pel=2, sharp=csh, overlap=cov, truemotion=true, pnew=cpn, idx=idx_c)
# backw1 = (rdlimit > 13) ? (rdlimit > 20) ? clip.MVFlow(bvec1, idx=idx_c).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(bvec1, idx=idx_c).Deblock(quant=16, aOffset=4, bOffset=4) : \
# clip.MVFlow(bvec1, idx=idx_c)
# forw1 = (rdlimit > 13) ? (rdlimit > 20) ? clip.MVFlow(fvec1, idx=idx_c).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(fvec1, idx=idx_c).Deblock(quant=16, aOffset=4, bOffset=4) : \
# clip.MVFlow(fvec1, idx=idx_c)

clp = interleave(backw1, clp, forw1)
clp = clp.RemoveDirt2(rdlimit, rgrain, _grey)
# clp = clp.RemoveDirtMC2(rdlimit, rgrain, _grey)
dnc = (denoise == 0) ? clp.RemoveTempGrain(rgrain).SelectEvery(3, 1) : dummy
clp = clp.SelectEvery(3, 1)

dnc = MDegrain3(clp, super, bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, thSAD=190 + 15 * denoise, \
thSCD1=230 + 5 * denoise)

# dnc = (denoise == 0) ? dnc : clp.MVDenoise(bvec2, bvec1, fvec1, fvec2, thT=denoise, thSAD=190 + 15 * denoise, \
# thmv=40, thSCD1=230 + 5 * denoise)

vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=7, percent=75)
vid_mo = (rgrain == 1) ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")
clp = clp.SeeSaw(dnc, Sstr=csharpen, Szp=12, SdampHi=20, bias=40)

return clp
}

# for slight noisy video or spot/dirt removing only
#
function HQ_Filter(clip clp, bool "grey")
{
grey = Default(grey, false)
clp = clp.RemoveNoiseMC(rdlimit=6, rgrain=1, denoise=0, _grey=grey)
clp = clp.LimitedSharpenFaster(Smode=4, strength=18, overshoot=0, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# medium noise, higher grain:
#
function MQ_Filter(clip clp, bool "grey")
{
grey = Default(grey, false)
clp = clp.RemoveNoiseMC(rdlimit=10, rgrain=2, denoise=8, _grey=grey)
clp = clp.LimitedSharpenFaster(Smode=4, strength=20, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# high noise or analog capture:
#
function LQ_Filter(clip clp, bool "grey")
{
grey = Default(grey, false)
clp = clp.RemoveNoiseMC(rdlimit=18, rgrain=3, denoise=14, _grey=grey)
clp = clp.VagueDenoiser(threshold=0.6, nsteps=6, chromaT=0.6, percent=75)
clp = clp.deen("a3d", rad=3, thrY=3, thrUV=5, min=0.25, tthY=2, tthUV=3, scd=6)
clp = clp.LimitedSharpenFaster(Smode=4, strength=24, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# =============================================================================
#
#
function RemoveNoiseMC_HQ(clip clp, int "tlimit", int "rgrain", float "csharp")
{
tlimit = Default(tlimit, 4)
rgrain = Default(rgrain, 1)
csharp = (rgrain > 1) ? Default(csharp, 0.14) : Default(csharp, 0.13)

# global idx_c = idx_c + 1

cbs = 4
ccf = cbs * cbs / 64
cpn = (tlimit > 6) ? 70 * ccf : 75 * ccf

super = MSuper(clp, pel=2, sharp=1)
bvec1 = MAnalyse(super, isb=true, blksize=cbs, delta=1, overlap=0, truemotion=true, pnew=cpn)
fvec1 = MAnalyse(super, isb=false, blksize=cbs, delta=1, overlap=0, truemotion=true, pnew=cpn)
backw1 = MFlow(clp, super, bvec1)
forw1 = MFlow(clp, super, fvec1)

# bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=1, overlap=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=1, overlap=0, truemotion=true, pnew=cpn, idx=idx_c)
# backw1 = clip.MVFlow(bvec1, idx=idx_c)
# forw1 = clip.MVFlow(fvec1, idx=idx_c)

dnc = interleave(backw1, clp, forw1)
dnc = dnc.RemoveDirt_HQ(tlimit, rgrain)
# dnc = dnc.RemoveDirtMC_HQ(tlimit, rgrain)
dnc = dnc.SelectEvery(3, 1)

return ((csharp == 0) ? dnc : clp.SeeSaw(dnc, Sstr=csharp, Szp=12, SdampHi=20, bias=40))
}

function HQ_Filter2(clip clp)
{
clp = clp.RemoveNoiseMC_HQ(tlimit=4, rgrain=1)
clp = clp.LimitedSharpenFaster(Smode=4, strength=15, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

#global idx_c = 25

# =============================================================================
#
#
function RemoveHighNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", float "csharp", bool "_grey")
{
_grey = Default(_grey, false)
rdlimit = Default(rdlimit, 24)
rgrain = Default(rgrain, 3)
denoise = Default(denoise, 15)
csharp = (rgrain > 2) ? Default(csharp, 0.32) : Default(csharp, 0.28)

# global idx_c = idx_c + 1

_dgr = 0.7 + rgrain * 0.5
cbs = 8
ccf = cbs * cbs / 64
cpn = (denoise > 12) ? 50 * ccf : 57 * ccf

super = MSuper(clp, pel=2, sharp=0)
bvec3 = MAnalyse(super, isb=true, blksize=cbs, delta=3, truemotion=true, pnew=cpn)
bvec2 = MAnalyse(super, isb=true, blksize=cbs, delta=2, truemotion=true, pnew=cpn)
bvec1 = MAnalyse(super, isb=true, blksize=cbs, delta=1, truemotion=true, pnew=cpn)
fvec1 = MAnalyse(super, isb=false, blksize=cbs, delta=1, truemotion=true, pnew=cpn)
fvec2 = MAnalyse(super, isb=false, blksize=cbs, delta=2, truemotion=true, pnew=cpn)
fvec3 = MAnalyse(super, isb=false, blksize=cbs, delta=3, truemotion=true, pnew=cpn)
backw1 = (rdlimit > 20) ? MFlow(clp, super, bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec1).Deblock(quant=16, aOffset=4, bOffset=4)
forw1 = (rdlimit > 20) ? MFlow(clp, super, fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec1).Deblock(quant=16, aOffset=4, bOffset=4)

# bvec3 = clip.MVAnalyse(isb=false, blksize=cbs, delta=3, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# bvec2 = clip.MVAnalyse(isb=false, blksize=cbs, delta=2, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# bvec1 = clip.MVAnalyse(isb=false, blksize=cbs, delta=1, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec1 = clip.MVAnalyse(isb=true, blksize=cbs, delta=1, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec2 = clip.MVAnalyse(isb=true, blksize=cbs, delta=2, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# fvec3 = clip.MVAnalyse(isb=true, blksize=cbs, delta=3, pel=2, sharp=0, truemotion=true, pnew=cpn, idx=idx_c)
# backw1 = (rdlimit > 20) ? clip.MVFlow(bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(bvec1).Deblock(quant=16, aOffset=4, bOffset=4)
# forw1 = (rdlimit > 20) ? clip.MVFlow(fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
# clip.MVFlow(fvec1).Deblock(quant=16, aOffset=4, bOffset=4)

clp = interleave(backw1, clp, forw1)
clp = clp.RemoveDirt2(limit=rdlimit, rgrain=rgrain, _grey=_grey)
# clp = clp.RemoveDirtMC2(limit=rdlimit, rgrain=rgrain, _grey=_grey)
clp = clp.SelectEvery(3, 1)

dnc = MDegrain3(clp, super, bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, thSAD=160 + 12 * denoise, \
thSCD1=255 + 4 * denoise)

# dnc = clp.MVDenoise(bvec3, bvec2, bvec1, fvec1, fvec2, fvec3, thT=denoise, thSAD=160 + 12 * denoise, \
# thmv=40, thSCD1=255 + 4 * denoise)

vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=6, percent=75)
vid_mo = (rgrain == 1) ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")

return ((csharp == 0) ? dnc : clp.SeeSaw(dnc, Sstr=csharp, Szp=12, SdampHi=20, bias=40))
}

function VLQ_Filter(clip clp, bool "grey")
{
grey = Default(grey, false)
clp = clp.RemoveHighNoiseMC(rdlimit=24, rgrain=3, denoise=16, csharp=0.3, _grey=grey)
clp = clp.VagueDenoiser(threshold=1.0, nsteps=6, chromaT=1.2, percent=75)
clp = clp.deen("a3d", rad=3, thrY=4, thrUV=7, min=0.25, tthY=2, tthUV=3, scd=7)
clp = clp.LimitedSharpenFaster(Smode=4, strength=28, overshoot=1, wide=false, ss_x=1.2, ss_y=1.2)

return clp
}

function VLQ_Filter_Ultimate(clip clp, bool "grey")
{
grey = Default(grey, false)
clp = clp.RemoveHighNoiseMC(rdlimit=24, rgrain=3, denoise=16, _grey=grey)
clp = clp.fft3dfilter(sigma=1.2, sharpen=0.4, plane=4)
clp = clp.deen("a3d", rad=3, thrY=4, thrUV=7, min=0.25, tthY=2, tthUV=3, scd=7)

return clp
}

Also, I can't make removegrainT work (if i don't get unable to load RemoveGrainT.dll, then it's another dll like SSExTools, RemoveDirt(s), I've tried lots of combination, some only give me a message in mpc..).
I've searched and searched applied all the fixes I could find, nothing works. My installed dlls are the required one, I've added AvsRecursion, I already have msvcrt7(0)1.dll etc..

I don't even know if there's a point asking for help on this one. I feel like giving up. (I'm on 2.6, W7 32).

Jenyok

8th November 2012, 07:17

Some changes...
.

#
#
# File: RemoveNoise.avsi
#
# Moution compensated RemoveNoise and variuos filters functions, used RemoveNoise
#
# Functions:
#
# function NullClip(clip clp)
#
# function RemoveNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", bool "sharp", float "csharpen", bool "_grey", int "Vectors")
#
# function HQ_Filter(clip clp, int "Vectors")
# function MQ_Filter(clip clp, int "Vectors")
# function LQ_Filter(clip clp, int "Vectors")
#
# function RemoveNoise_HQMC(clip clp, int "tlimit", int "rgrain", float "csharp", int "Vectors")
# function HQ_Filter2(clip clp, int "Vectors")
#
# function RemoveHighNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", float "csharp", bool "_grey", int "Vectors")
#
# function VLQ_Filter(clip clp, int "Vectors")
# function VLQ_Filter_Ultimate(clip clp, int "Vectors")
#
#

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirts.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\removegraint.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAINT-1_0\repairt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\removegrain.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEGRAIN-1_0\repair.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_11_3\mvtools2.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FLUXSMOOTH_25_DLL_20040729\fluxsmooth.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\VAGUEDENOISER_25_DLL_20050926\vaguedenoiser.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\FFT3DFILTER_20070220\fft3dfilter.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\ZZZ_OLD\DEEN_25_DLL_20030813\deen.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\DEBLOCK_25_DLL_20060214\deblock.dll")

Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\SHARPFUNCTION_LIMITEDSHARPENFASTER\limitedsharpenfaster.avsi")
Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\SEESAW\seesaw.avsi")
Import("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\REMOVEDIRT_20050507\removedirt.avsi")

function NullClip(clip clp)
{
return BlankClip(clp, length=0)
}

# =============================================================================
#
#
function RemoveNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", bool "sharp", float "csharpen", \
bool "_grey", int "Vectors")
{
_grey = Default(_grey, false)
rdlimit = Default(rdlimit, 11)
rgrain = Default(rgrain, 2)
denoise = Default(denoise, 8)
sharp = (rgrain < 1) ? Default(sharp, true) : Default(sharp, false)
csharpen = (rgrain > 2) ? Default(csharpen, 0.17) : (rgrain > 1) ? Default(csharpen, 0.15) : Default(csharpen, 0.13)
Vectors = Default(Vectors, 1)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors
csharpen = (sharp) ? csharpen : csharpen + 0.08
_dgr = 0.45 + rgrain * 0.4
cbs = 8
cov = (cbs > 4) ? cbs / 4 : 0
ccf = cbs * cbs / 64
cpn = (denoise > 12) ? 50 * ccf : (denoise > 8) ? 58 * ccf : (denoise > 5) ? 66 * ccf : 72 * ccf
csh = (sharp) ? 1 : 0

super = MSuper(clp, pel=2, sharp=csh)
bvec3 = MAnalyse(super, isb=true, delta=3, blksize=cbs, overlap=cov, truemotion=true, pnew=cpn)
bvec2 = MAnalyse(super, isb=true, delta=2, blksize=cbs, overlap=cov, truemotion=true, pnew=cpn)
bvec1 = MAnalyse(super, isb=true, delta=1, blksize=cbs, overlap=cov, truemotion=true, pnew=cpn)
fvec1 = MAnalyse(super, isb=false, delta=1, blksize=cbs, overlap=cov, truemotion=true, pnew=cpn)
fvec2 = MAnalyse(super, isb=false, delta=2, blksize=cbs, overlap=cov, truemotion=true, pnew=cpn)
fvec3 = MAnalyse(super, isb=false, delta=3, blksize=cbs, overlap=cov, truemotion=true, pnew=cpn)

backw3 = ((Vectors == 3) && (rdlimit > 13)) ? ((Vectors == 3) && (rdlimit > 20)) ? \
MFlow(clp, super, bvec3).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec3).Deblock(quant=16, aOffset=4, bOffset=4) : \
(Vectors == 3) ? MFlow(clp, super, bvec3) : NullClip(clp)
backw2 = ((Vectors >= 2) && (rdlimit > 13)) ? ((Vectors >= 2) && (rdlimit > 20)) ? \
MFlow(clp, super, bvec2).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec2).Deblock(quant=16, aOffset=4, bOffset=4) : \
(Vectors >= 2) ? MFlow(clp, super, bvec2) : NullClip(clp)
backw1 = (rdlimit > 13) ? (rdlimit > 20) ? MFlow(clp, super, bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec1).Deblock(quant=16, aOffset=4, bOffset=4) : \
MFlow(clp, super, bvec1)
forw1 = (rdlimit > 13) ? (rdlimit > 20) ? MFlow(clp, super, fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec1).Deblock(quant=16, aOffset=4, bOffset=4) : \
MFlow(clp, super, fvec1)
forw2 = ((Vectors >= 2) && (rdlimit > 13)) ? ((Vectors >= 2) && (rdlimit > 20)) ? \
MFlow(clp, super, fvec2).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec2).Deblock(quant=16, aOffset=4, bOffset=4) : \
(Vectors >= 2) ? MFlow(clp, super, fvec2) : NullClip(clp)
forw3 = ((Vectors == 3) && (rdlimit > 13)) ? ((Vectors == 3) && (rdlimit > 20)) ? \
MFlow(clp, super, fvec3).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec3).Deblock(quant=16, aOffset=4, bOffset=4) : \
(Vectors == 3) ? MFlow(clp, super, fvec3) : NullClip(clp)

clp = (Vectors == 1) ? Interleave(backw1, clp, forw1) : \
(Vectors == 2) ? Interleave(backw2, backw1, clp, forw1, forw2) : \
Interleave(backw3, backw2, backw1, clp, forw1, forw2, forw3)

clp = RemoveDirt2(clp, limit=rdlimit, rgrain=rgrain, _grey=_grey)

dnc = RemoveTempGrain(clp, rgrain)
dnc = (Vectors == 1) ? SelectEvery(dnc, 3, 1) : \
(Vectors == 2) ? SelectEvery(dnc, 5, 2) : \
SelectEvery(dnc, 7, 3)
dnc = (denoise == 0) ? dnc : NullClip(clp)

clp = (Vectors == 1) ? SelectEvery(clp, 3, 1) : \
(Vectors == 2) ? SelectEvery(clp, 5, 2) : \
SelectEvery(clp, 7, 3)

dnc = (denoise == 0) ? dnc : MDegrain3(clp, super, bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, \
thSAD=190 + 15 * denoise, thSCD1=230 + 5 * denoise)
vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=7, percent=75)
vid_mo = (rgrain == 1) ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")
clp = clp.SeeSaw(dnc, Sstr=csharpen, Szp=12, SdampHi=20, bias=40)

return clp
}

# for slight noisy video or spot/dirt removing only
#
function HQ_Filter(clip clp, int "Vectors", bool "grey")
{
Vectors = Default(Vectors, 1)
grey = Default(grey, false)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

clp = clp.RemoveNoiseMC(rdlimit=6, rgrain=1, denoise=0, Vectors=Vectors, _grey=grey)
clp = clp.LimitedSharpenFaster(Smode=4, strength=18, overshoot=0, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# medium noise, higher grain:
#
function MQ_Filter(clip clp, int "Vectors", bool "grey")
{
Vectors = Default(Vectors, 1)
grey = Default(grey, false)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

clp = clp.RemoveNoiseMC(rdlimit=10, rgrain=2, denoise=8, Vectors=Vectors, _grey=grey)
clp = clp.LimitedSharpenFaster(Smode=4, strength=20, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# high noise or analog capture:
#
function LQ_Filter(clip clp, int "Vectors", bool "grey")
{
Vectors = Default(Vectors, 1)
grey = Default(grey, false)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

clp = clp.RemoveNoiseMC(rdlimit=18, rgrain=3, denoise=14, Vectors=Vectors, _grey=grey)
clp = clp.VagueDenoiser(threshold=0.6, nsteps=6, chromaT=0.6, percent=75)
clp = clp.deen("a3d", rad=3, thrY=3, thrUV=5, min=0.25, tthY=2, tthUV=3, scd=6)
clp = clp.LimitedSharpenFaster(Smode=4, strength=24, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

Jenyok

8th November 2012, 07:17

Some changes, continiue...
.

# =============================================================================
#
#
function RemoveNoise_HQMC(clip clp, int "tlimit", int "rgrain", float "csharp", int "Vectors")
{
tlimit = Default(tlimit, 4)
rgrain = Default(rgrain, 1)
csharp = (rgrain > 1) ? Default(csharp, 0.14) : Default(csharp, 0.13)
Vectors = Default(Vectors, 1)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

cbs = 4
ccf = cbs * cbs / 64
cpn = (tlimit > 6) ? 70 * ccf : 75 * ccf

super = MSuper(clp, pel=2, sharp=1)
bvec3 = (Vectors == 3) ? MAnalyse(super, isb=true, delta=3, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
bvec2 = (Vectors >= 2) ? MAnalyse(super, isb=true, delta=2, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
bvec1 = MAnalyse(super, isb=true, delta=1, blksize=cbs, overlap=4, truemotion=true, pnew=cpn)
fvec1 = MAnalyse(super, isb=false, delta=1, blksize=cbs, overlap=4, truemotion=true, pnew=cpn)
fvec2 = (Vectors >= 2) ? MAnalyse(super, isb=false, delta=2, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
fvec3 = (Vectors == 3) ? MAnalyse(super, isb=false, delta=3, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
backw3 = (Vectors == 3) ? MFlow(clp, super, bvec3) : NullClip(clp)
backw2 = (Vectors >= 2) ? MFlow(clp, super, bvec2) : NullClip(clp)
backw1 = MFlow(clp, super, bvec1)
forw1 = MFlow(clp, super, fvec1)
forw2 = (Vectors >= 2) ? MFlow(clp, super, fvec2) : NullClip(clp)
forw3 = (Vectors == 3) ? MFlow(clp, super, fvec3) : NullClip(clp)

dnc = (Vectors == 1) ? Interleave(backw1, clp, forw1) : \
(Vectors == 2) ? Interleave(backw2, backw1, clp, forw1, forw2) : \
Interleave(backw3, backw2, backw1, clp, forw1, forw2, forw3)

dnc = RemoveDirt_HQ(dnc, tlimit, rgrain)

dnc = (Vectors == 1) ? SelectEvery(dnc, 3, 1) : \
(Vectors == 2) ? SelectEvery(dnc, 5, 2) : \
SelectEvery(dnc, 7, 3)

clp = (csharp == 0) ? dnc : clp.SeeSaw(dnc, Sstr=csharp, Szp=12, SdampHi=20, bias=40)

return dnc
}

function HQ_Filter2(clip clp, int "Vectors")
{
Vectors = Default(Vectors, 1)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

clp = clp.RemoveNoiseMC_HQ(tlimit=4, rgrain=1, Vectors=Vectors)
clp = clp.LimitedSharpenFaster(Smode=4, strength=15, overshoot=1, wide=false, ss_x=1.3, ss_y=1.3)

return clp
}

# =============================================================================
#
#
function RemoveHighNoiseMC(clip clp, int "rdlimit", int "rgrain", int "denoise", float "csharp", \
bool "_grey", int "Vectors")
{
_grey = Default(_grey, false)
rdlimit = Default(rdlimit, 24)
rgrain = Default(rgrain, 3)
denoise = Default(denoise, 15)
csharp = (rgrain > 2) ? Default(csharp, 0.32) : Default(csharp, 0.28)
Vectors = Default(Vectors, 1)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

_dgr = 0.7 + rgrain * 0.5
cbs = 8
ccf = cbs * cbs / 64
cpn = (denoise > 12) ? 50 * ccf : 57 * ccf

super = MSuper(clp, pel=2, sharp=0)
bvec3 = (Vectors == 3) ? MAnalyse(super, isb=true, delta=3, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
bvec2 = (Vectors >= 2) ? MAnalyse(super, isb=true, delta=2, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
bvec1 = MAnalyse(super, isb=true, delta=1, blksize=cbs, overlap=4, truemotion=true, pnew=cpn)
fvec1 = MAnalyse(super, isb=false, delta=1, blksize=cbs, overlap=4, truemotion=true, pnew=cpn)
fvec2 = (Vectors >= 2) ? MAnalyse(super, isb=false, delta=2, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
fvec3 = (Vectors == 3) ? MAnalyse(super, isb=false, delta=3, blksize=cbs, overlap=4, truemotion=true, pnew=cpn) : NullClip(clp)
backw3 = ((Vectors == 3) && (rdlimit > 20)) ? MFlow(clp, super, bvec3).Deblock(quant=22, aOffset=6, bOffset=6) : \
(Vectors == 3) ? MFlow(clp, super, bvec3).Deblock(quant=16, aOffset=4, bOffset=4) : NullClip(clp)
backw2 = ((Vectors >= 2) && (rdlimit > 20)) ? MFlow(clp, super, bvec2).Deblock(quant=22, aOffset=6, bOffset=6) : \
(Vectors >= 2) ? MFlow(clp, super, bvec2).Deblock(quant=16, aOffset=4, bOffset=4) : NullClip(clp)
backw1 = (rdlimit > 20) ? MFlow(clp, super, bvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, bvec1).Deblock(quant=16, aOffset=4, bOffset=4)
forw1 = (rdlimit > 20) ? MFlow(clp, super, fvec1).Deblock(quant=22, aOffset=6, bOffset=6) : \
MFlow(clp, super, fvec1).Deblock(quant=16, aOffset=4, bOffset=4)
forw2 = ((Vectors >= 2) && (rdlimit > 20)) ? MFlow(clp, super, fvec2).Deblock(quant=22, aOffset=6, bOffset=6) : \
(Vectors >= 2) ? MFlow(clp, super, fvec2).Deblock(quant=16, aOffset=4, bOffset=4) : NullClip(clp)
forw3 = ((Vectors == 3) && (rdlimit > 20)) ? MFlow(clp, super, fvec3).Deblock(quant=22, aOffset=6, bOffset=6) : \
(Vectors == 3) ? MFlow(clp, super, fvec3).Deblock(quant=16, aOffset=4, bOffset=4) : NullClip(clp)

clp = (Vectors == 1) ? Interleave(backw1, clp, forw1) : \
(Vectors == 2) ? Interleave(backw2, backw1, clp, forw1, forw2) : \
Interleave(backw3, backw2, backw1, clp, forw1, forw2, forw3)

clp = RemoveDirt2(clp, limit=rdlimit, rgrain=rgrain, _grey=_grey)

clp = (Vectors == 1) ? SelectEvery(clp, 3, 1) : \
(Vectors == 2) ? SelectEvery(clp, 5, 2) : \
SelectEvery(clp, 7, 3)

dnc = MDegrain3(clp, super, bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, thSAD=160 + 12 * denoise, \
thSCD1=255 + 4 * denoise)

vid_mo = dnc.VagueDenoiser(threshold=_dgr, chromaT=_dgr, nsteps=6, percent=75)
vid_mo = (rgrain == 1) ? vid_mo.RemoveGrain(1) : vid_mo.RemoveGrain(5)
dnc = dnc.ConditionalFilter(dnc, vid_mo, "(YDifferenceFromPrevious()+YDifferenceToNext())/AverageLuma()", "<", "0.3")
clp = ((csharp == 0) ? dnc : clp.SeeSaw(dnc, Sstr=csharp, Szp=12, SdampHi=20, bias=40))

return clp
}

function VLQ_Filter(clip clp, int "Vectors", bool "grey")
{
Vectors = Default(Vectors, 1)
grey = Default(grey, false)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

clp = clp.RemoveHighNoiseMC(rdlimit=24, rgrain=3, denoise=16, csharp=0.3, Vectors=Vectors, _grey=grey)
clp = clp.VagueDenoiser(threshold=1.0, nsteps=6, chromaT=1.2, percent=75)
clp = clp.deen("a3d", rad=3, thrY=4, thrUV=7, min=0.25, tthY=2, tthUV=3, scd=7)
clp = clp.LimitedSharpenFaster(Smode=4, strength=28, overshoot=1, wide=false, ss_x=1.2, ss_y=1.2)

return clp
}

function VLQ_Filter_Ultimate(clip clp, int "Vectors", bool "grey")
{
Vectors = Default(Vectors, 1)
grey = Default(grey, false)

Vectors = (Vectors < 1) ? 1 : (Vectors > 3) ? 3 : Vectors

clp = clp.RemoveHighNoiseMC(rdlimit=24, rgrain=3, denoise=16, csharp=0.48, Vectors=Vectors, _grey=grey)
clp = clp.fft3dfilter(sigma=1.2, sharpen=0.4, plane=4)
clp = clp.deen("a3d", rad=3, thrY=4, thrUV=7, min=0.25, tthY=2, tthUV=3, scd=7)

return clp
}