RemoveGrain

[jarthel]

newly installed winxp setup. installed avisynth and run dvd2avi on a vob.

when I tried to load an .avs file with removegrain in it, vdubmod won't load it. if I comment out the "load removegrain" and the removegrain function call lines, I can load the .avs file into vdubmod. any ideas?

here's the script
import("C:\Program Files\NuMenu4U\new.avs\addaudio.avs")
LoadPlugin("C:\Program Files\NuMenu4U\new.dvd2avi\mpeg2dec3dg.dll")
LoadPlugin("c:\downloads\software\windows\videos\DVD authoring\avisynth\plugins25\decomb.dll")
Mpeg2Source("C:\dvd_images\originals\needs_reencoding\ORPHEN_VOLUME_FOUR\D_VTS_01_0\FRAMESERVING\001.d2v" ,idct=7,iPP=true)
FieldDeinterlace()
LoadPlugin("c:\downloads\software\windows\videos\DVD authoring\avisynth\plugins25\removegrain.dll")
LoadPlugin("c:\downloads\software\windows\videos\DVD authoring\avisynth\plugins25\removedirt.dll")
removegrain()
removedirt()
AddAudio()
ConvertToYUY2()

======
update: as I was writing this reply, I was installing norton systemworks. I tried opening up the file again and it works now!!! :|

[kassandro]

I have just uploaded the version 0.9 to the web site (http://www.removegrain.de.tf).
Various new repair modes have been added. Repair modes 3,4 have been improved (they now include the center pixel). I have added a new mode 18, which preserves thin lines and has a better chance than mode 17 to preserve the end points of thin lines as well. However, compression will probably not be comparable with mode 17, which is still my preferred mode. I have added the filters ForwardClense and BackwardClense to RemoveGrain. These filters are primarily for handling sharp scene switches in the RemoveDirt script function (a new version will be released in 1-2 weeks). The speed of the filter Clense, especially the SSE2 version, has been improved substantially.
TemporalRepair has now also the modes smooth=2,3 which are primarily for deinterlacing.
I have noticed a problem with the SSE3 version. If it is used together with avisource VirtualDubMod crashed upon exit. This is probably a problem with the Intel compiler (it seems to handle virtual destructors somewhat differently) and I can't fix it right now.

[Poutnik]

I have encountered troubles in RemoveDust/Dirt scripts with optional parameters of script. I have get "Not defined var(or so)" error in Vdub avs debugging. This disappeared only after changing default(var,value) -> var=default(var,value) Seems like "default(val,value)" assigns value to last, at least not to var.

There are test functions isplanar(), isinterleaved(). So why not use them ? I do conditional planar<->interleved conversion in my scripts.


.....
il=isinterleaved(input)
input = (il==true) ? Interleaved2Planar(input) : input
"Filtering"
return (il==true) ? Planar2Interleaved(last) : last


BTW, I like speed and approach of your filters very much, Kassandro.
What do you think about using strong DCTFilter with TemporalRepair(+Repair?). Something like "poor man" 2D variant of Fizick's FFT3DFilter.

Post Scriptum: Could you put an overview table of filter modes/their usage in your HTML docs ?. I am getting lost as there are so many ones....

[kassandro]

There are test functions isplanar(), isinterleaved(). So why not use them ? I do conditional planar<->interleved conversion in my scripts.

You can't use isplanar(), because planar YUY2 is format, which is not supported by Avisynth. For Avisynth planar YUY2 clips are identical with ordinary interleaved YUY2 clips. RemoveGrain/Repair/RemoveDirt are - so far - the only plugins which use this color space. I have done so, because for using SSE/SSE2/SSE3 one has to convert internally the interleaved format into a planar one. With planar YUY2 this conversion has only to be done once for a script function, which involves only filters from the above three plugins. This saves a lot of time.


I have encountered troubles in RemoveDust/Dirt scripts with optional parameters of script. I have get "Not defined var(or so)" error in Vdub avs debugging. This disappeared only after changing "default(var,value) -> var=default(var,value). Seems like "default(val,value)" assigns value to last, at least not to var.

Thanks, I will correct this. There will be a new version 0.9 of RemoveDirt plugin tonight or tommorrow. This will also correct two unpleasant bugs, which were not destructive enough to be discovered immediately. Also functionality is improved quite a bit especially as far as scene switches are concerned.

[Poutnik]

If supposing usage removedust().removedirt() (without cropping inbetween), are the scripts optimized for this way ?
Or is better to construct a superscript "RemoveDustAndDirt" ?

[kassandro]

Originally posted by Poutnik
If supposing usage removedust().removedirt() (without cropping inbetween), are the scripts optimized for this way ?
Or is better to construct a superscript "RemoveDustAndDirt" ?
No you shouldn't do that. The result will simply be too soft. As you can see from the discussion above, for some people RemoveDust is already too soft and Didee modified RemoveDust by simply limiting the amount of change. Because the chroma is much more sparse than the luma in YV12, RemoveDust is considerably more aggressive on the chroma. On the other hand, the chroma is especially noisy, if it originates from analog capturing. The RemoveDirt script should creat less softness, while maintaining the same or even a slightly better compression ratio, but RemoveDirt bears also some artifact risk depending of course on its configuration. I personally use RemoveDirt now, but I still in the testing phase.

[FredThompson]

When I have any of the 4 variants of the RemoveGrain package in my Avisynth/plugins directory VirtualDubMod will not load. It reports a Visual C++ error that the application requested to terminate in a non-standard manner.

Why would this happen?

[Boulder]

Which VDubMod version do you use? It works on my system with both 1.5.4.1 and 1.5.10.1 just fine.
Which Avisynth version do you have?

[midelic]

The same It happened to me ,same error, but i don't think removegrain is causing this.Any other filter I put in avisynth plugin directory gave me this error.Even Ben's plugin invert.dll gave me the same error.My avisynth plugin directory is full (has removegrain,remove dirt,almost all plugins)but wen i try to load a new one I receive this error.
Avisynth 2.55
Virtualdubmod 1.5.10.1

[FredThompson]

I'm using VirtualDubMod 1.5.10.1, both the base release and the latest update will crash. AviSynth is beta 050505.

[Boulder]

The Avisynth version is the problem I think. It's very fragile as you can read from the topic which contains the discussion around the CVS builds.

[FredThompson]

The AviSynth build doesn't have anything to do with the crashing.

Removing all the filters and testing with different combinations shows the SSE2 and SSE3 builds will cause a crash. SSE3 should be expected, it's not in the Athlon XP. SSE2 should load. This is a minor annoyance, as the 2 simplest vesions will load.

There is some kind of conflict between RemoveGrain, Repair, and SSETools when any of the following are in the plugins directory:

rawsource (200504250, TDeint 1.0 beta 3, TIVTC 0.9.8.5 and warpsharp (2003_11030).

VirtualDubMod will load properly for me with either group 1 or 2, not elements of both. SSE2 should load but it's not. Maybe the Athlon XP model 8 is an incomplete SSE2 or maybe there's something in the Intel CPUs which is actually SSE2+, so to speak.

-- edit: well, that's not entirely true. Oddly enough, it seems the order in which I copied the filters into the filter directory affects crashing. Maybe there is a sequence of filter loading which causes the crashing.

[Wilbert]

Removing all the filters and testing with different combinations shows the SSE2 and SSE3 builds will cause a crash. SSE3 should be expected, it's not in the Athlon XP. SSE2 should load.
Like i said in another forum. Athlon XP doesn't support SSE2, only SSE and iSSE.

Besides, i think we just have to live with this, ie plugins causing crashes in the plugin folder.

[Boulder]

Which Athlon XP has SSE2 instructions support? I thought that was available only for some Semprons and A64-variants.

EDIT: Wilbert was faster..

[FredThompson]

I'm sorry, you are both correct. I thought wcpuid said SSE2 support was present but I was wrong.

Turns out VirtualDub doesn't crash but VirtualDubMod does. Durn...

[midelic]

Just now i observed ,you are right,only virtualdubmod is crashing.

[vigi_lante]

So, Boulder, what is your script at this point ? I guess it's with LRemoveDust(17,2). But what LRemoveDust.avs script are you using now ?

Could you recommend a more strong setting in order to remove even more noise ? Noise is important to me, because with it I can eliminate blocks. But since I'm working with cartoons (analog TV capture), a smooth look is no problem, but if I remove to much noise, blocks will appear.

So, I'm looking for a balance between smooth and noise.

Thanks!

[Boulder]

I use either one of these two, depending on what the original colorspace is:

function LRemoveDust_YUY2(clip input, int clmode, int "limit")
{
clmode=default(clmode,17)
limit=default(limit,2)
repmode = 2
clensed = Clense(input, grey=true)
rep = Repair(clensed, input, mode=repmode, modeU=-1)
rg = RemoveGrain(rep, mode=_mode, modeU=-1)
return LimitChange(rg, input, limit, limitU=255)
}

function LRemoveDust_YV12(clip input, int clmode, int "limit")
{
limit=default(limit,2)
clmode=default(clmode,17)
repmode = 2
clensed = Clense(input)
rep = Repair(clensed, input, mode=repmode)
rg = RemoveGrain(rep, mode=clmode)
return LimitChange(rg, input, limit)
}

--
My usage for YUY2 sources is
xxxSource()
org=last
ConverttoYV12()
LRemoveDust_YUY2(17,2)
ConverttoYUY2()
MergeChroma(org)

and for YV12

xxxSource()
LRemoveDust_YV12(17,2)
--
If you need more cleaning, you can raise the value for limit, however, there's no magical setting that will clean but leave all the details. I've found out that limit=2 is a good compromise.

[vigi_lante]

In my case, I'm capturing with Huffyuv (YUY2) and my target is to compress it with XviD (YV12). So, which script should I use ?

[Boulder]

Convert to YV12 before LRemoveDust and use LRemoveDust_YV12().

[KillNoise]

First welcome back to this forum, Kassandro !
I have been worried about your disappearance after May 7th with that strange note on http://www.removegrain.de.tf/ saying you were locked out from Doom9 Forum and to use VideoProcessing.Forumer.com (which turned out as dead link for me) neither any response to e-mail at gorw@gmx.de (so was afraid you might have left your excellent work unfinished).
Thank you again for this very usefull, versatile and fast filter set - my favorite for denoising tasks.


Looks like i found two Bugs in RemoveGrain 0.9
(don't trust either my understanding or my CPU/system otherwise)
==> can anyone confirm these bugs on his machine, please ?

Use RemoveGrain latest Ver. 0.9 of 2005-05-01 with AviSynth 2.55 of 2004-09-01 on Pentium 4 CPU (Hyperthreading enabled) under Win XP SP2



1) RemoveGrain(mode=3) Bug in RemoveGrainSSE2.dll
------------------------------------------------------------

RemoveGrain(mode=3) seems to erode bright edges much stronger than it is supposed to; effect is clearly visible at bright letters on dark backgroung (e.g. movie credits).
So please try:

LoadPlugin("RemoveGrainSSE2.dll") # bug is in SSE2 version (RemoveGrain.dll seems to behave properly)
return StackHorizontal( source.RemoveGrain(3), source.RemoveGrain(4) ) # mode 3 supposed to be less agressive

Remarkable: some (about 16 ?) pixels at right and left borders seem to be correctly processed (recognize slight edge in texture ==> probably some different code for border allignment range).

With LoadPlugin("RemoveGrain.dll") everything looks as expected.
(==> guess some simple bug in neighbour pixel rank order adressing of SSE2-optimized code for mode 3; did not check SSE3 build, neither whether any other modes may be affected too)


BTW:
I prefer mode = 3 rather than 4, because mode 4 erases corner pixels and so completly removes 2x2-pixel-objekts (which typically make up important details for natural skin appearence in closeup etc.) Proper mode 3 should preserve these mostly. So it would be fine to have another more sensitive mode added, simmilar mode 17, but based on mode 3 rather than 4, so that it should preserve thin lines as well as corner pixels and 2x2-pixel-objekts. (Maybe that is what mode 18 does, but it was not any explained in RemoveGrain.htm document)





2) Very strange behaviour of RemoveGrain() after Clense()
---------------------------------------------------------------------------
There seems to be something wrong when using RemoveGrain after Clense (Maybe some problem with RemoveGrain and Clense inadmissible sharing static variables or frame cache buffers ?):
Looks somewhat like Clense messes up source frames when followed by RemoveGrain, probably requesting avoidable non-sequential seek operations on source, which would substantially slow down performance and i already experienced that at least DirectShowSource may return different frames on random seek and sequential access (seems to skip frames sometimes, returning previous frame for new number, but this might also be a special problem with my local filter setup involving ffdshow-20041012).

Please try this:

SetMemoryMax(256) # to rule out any memory limit problems

mSrc = DirectShowSource(pathSource, fRate)#.Crop(224, 160, 256, 256) # may optionally cut CPU load for testing
mSource = mSrc.ShowFrameNumber(scroll=true) # burn in frame number for analysis

mDegrain = mSource.RemoveGrain(17) # first remove spatial grain is to improve clensing results (mode 17 eats corner pixels but keeps thin lines)
mClensed = mDegrain.Clense(grey=false) # temporal min/max-clipping removes fluctuation peaks (messing up moving objects)
mRep = mClensed.Repair(mSource, mode=16) # restore clensed details from source (repair mode 16 can restore endpoints)

mTest = mSource
#mTest = mDegrain
#mTest = mClensed
#mTest = mRep

return Subtract( mTest.RemoveGrain(1), mTest.RemoveGrain(1) ).ColorYUV(analyze=true).Levels(116, 1.0, 136, 0, 255)

(ColorYUV(analyze=true) shows frame number & statiscs. Levels() is used to emphasize delta pixels)
Of course there should be no difference at all, which works as expected with mTest = mSource or mTest = mDegrain.
However using mTest = mRep or mTest = mClensed shows large differences between the two same calls of RemoveGrain - at least on my system !!!


Verifying test with UnDot() instead of RemoveGrain()
return Subtract( mTest.UnDot(), mTest.UnDot() ).ColorYUV(analyze=true).Levels(116, 1.0, 136, 0, 255)
==> shows no difference regardless of clip used for mTest, so the bug must be in the combination of Clense together with RemoveGrain


I verified for different combinations of UnDot() and RemoveGrain(), then tried to identify differences with
return StackHorizontal( mTest.RemoveGrain(1), mTest.RemoveGrain(1) )
==> turned out extremly slow with ffdshow-20041012 (much faster after uninstalling ffdshow); sometimes i saw different frame

contents with same(!) ShowFrameNumber burned in. Otherwise no visible difference, but Subtract() uncovers that the two calls of RemoveGrain do not produce same results. (Suspect Clense unnecessarilly recalculates with having differnt neighbour frame contents served on subsequent calls.)


I first stumbled on this strange effect when i looked at changes made by RemoveGrain(1) alone:
return Subtract( mRep.RemoveGrain(mode=1), mRep ).ColorYUV(analyze = true)
it is expected to show some isolated pixels clipped to threir neighbors (as UnDot does) but wondered why i saw connected clusters of pixels.

[kingmob]

NM, my problem was fixed by updating avisynth to the latest beta :o

[jarthel]

any parameter recommendation for anime sources? thanks

[J-Wo]

I'd like to know how to use RemoveGrain on hard-telecined material. I have some episodes of Veronica Mars R1 (season 1 disc 1) which are hard-telecined, i.e. encoded at 30fps but with a 3:2 mixture of progressive/interlaced frames. I'm not too familiar with the bob-weave or seperatefields-weave functions, but is that along the lines of what I should do before using this function? Thanks.

[Boulder]

Do an IVTC to get the progressive frames.

[J-Wo]

Thanks. What about for purely interlaced material, is removegrain a bad choice in this case? The readme leads me to believe so... I've been using Mode=5 with much success on film material (I do a lot of DVD-9 -> DVD-5 using DVD Rebuilder Pro)

[Boulder]

You can't use RemoveGrain directly on interlaced material. There's a lot of threads which discuss processing the issue.

[J-Wo]

Thanks boulder. Do you have any suggested filters I should look up for processing interlaced material similarly to RemoveGrain? I just want to do a little grain removal to aid with compression

[Didée]

any suggested filters I should look up for processing interlaced material similarly to RemoveGrain?
RemoveGrain is, well, sort of a "graded" median filter. There is no other filter of that kind.

The most simple way to use RemoveGrain on interlaced footage is
SeparateFields() .RemoveGrain(x) .Weave()

(Compare e.g. this here (http://forum.doom9.org/showthread.php?p=933541#post933541) - it's not exactly textbook level:o, but covers the basics.)


And regarding

(I do a lot of DVD-9 -> DVD-5 using DVD Rebuilder Pro)
[../..]
I just want to do a little grain removal to aid with compression
you might want to give a try on SPresso. After some playing with limit/bias/RGmode to find settings of your taste, you might get a better ratio of [preserved detail]:[compression gain] than with just using RemoveGrain on its own.

[J-Wo]

thanks for the suggestions Didée! Just wondering, does a filter like RemoveGrain require bobbing on interlaced footage. I'm afraid I'm pretty new to this whole processing interlaced material thing. Also in your script above, can I substitute your SPresso() filter for RemoveGrain()?

[Netuser]

the site http://www.removegrain.de.tf is not working for me, is it down forever ? where can i get the dll and degrain script please ?

Thanks alot :)

[Nightshiver]

It wokrs fine for me.

[thetoof]

Works for me too... but took a while to load.

[Netuser]

Yes finaly it worked for me too, it took hell of time to load lolzz

thanks

[papcom]

I tried to acquire "removegrain" binary via the official link on it's Website but I am landing in the nowhere with an error 404.

Does anyone has an alternative download for this plugin?

[StainlessS]

The Wiki is always a good place to look: http://avisynth.nl/index.php/Removegrain

[colours]

You might want to get RgTools (https://github.com/tp7/RgTools) instead.

Results should be identical to the original RemoveGrain for all modes other than 21 (the original has incorrect rounding) and the convolution modes (modes 11/12/19/20).

[Atlantis]

I just realized they have added TemporalRepair to RgTools v0.98.

So after I use RemoveGrain, what should I use, Repair or TemporalRepair? What's the difference in the results?

[Atlantis]

You are not going to believe this. I searched RemoveGrain and came here to say I just discovered TemporalRepair, how good is it?
Only this time I'm actually testing it! :D