MCTemporalDenoise [v1.4.20 - Update 2010/07/02]

[jmac698]

Ok,
The original Deblock_QED_MT2.avsi is at http://www.64k.it/andres/dettaglio.php?sez=avisynth. This version requires masktools <a35 to work, because of new "bias" parameter added. The newer Deblock_QED_MT2.avsi made some minor changes, and now needs masktools a45+ to work. Using a48 should be ok. The requirements are actually different if you update some of the scripts.
So just tell me, does my pack work as is?

Yes I used old version fftw3, once I tried to use new version and it didn't work for me. I should test again.

Update:
I checked the changelog from a43-a48. Some bugs were fixed and new features added, but nothing incompatible.

Alpha 48 :

added : signed and unsigned binary shifts to LUTs, ceil/floor/trunc float -> int conversions. @ is now equivalent to °

Alpha 47 :

added : signed and unsigned binary operators to LUTs

Alpha 46 :

fixed : with mt_lutf, "std" mode wasn't working. Could also affect "std" with mt_luts and mt_lutsx

Alpha 45 :

fixed : mt_clamp on sse2 platforms with resolution not multiple of 64
added : mode parameter to mt_lutspa, to clean up biased/relative mess

Alpha 44 :

added : new mode (weighted sum) for mt_luts and mt_lutsx


I will also list the differences in the new Deblock_QED_MT2.avsi:

# Changes 2008-08-18: (Didée)
# - Replaced the ugly stackXXX cascade with mt_LutSpa() (requires MaskTools v2.0a35)
# - Changed Quant and Offset defaults to 24,28,2,4,4,8

# Changes 2010-05-25:
# - Explicitly specified parameters of mt_LutSpa()
# (required due to position of new 'biased' parameter, starting from MaskTools 2.0a43)
# - Non mod 16 input is now padded with borders internally

# Changes 2010-08-18:
# - Replaced AddBorders with PointResize
# - Changed Quant and Offset defaults to 18,19,3,4,1,1 to reduce blurring

# Changes 2010-10-16:
# - Replaced 'relative' with the new 'mode' parameter in mt_LutSpa(), starting from MaskTools 2.0a45
# - Changed Quant and Offset defaults to 24,26,1,1,2,2 to increase effectiveness, but still within sensible limits.
# (see for details: http://forum.doom9.org/showthread.php?p=810932#post810932)

[totya]

Ok,
The original Deblock_QED_MT2.avsi is at http://www.64k.it/andres/dettaglio.php?sez=avisynth. This version requires masktools <a35 to work, because of new "bias" parameter added. The newer Deblock_QED_MT2.avsi made some minor changes, and now needs masktools a45+ to work. Using a48 should be ok.

Thanks for the info!

So just tell me, does my pack work as is?
Not much different for me "packs" :)


Yes I used old version fftw3, once I tried to use new version and it didn't work for me. I should test again.

All new version is work's for me (need libfftw3f-3.dll... rename it to fftw3.dll. I read this somewhere on doom9 forum). You can run benchmark "benchf.exe -opatient 64 128 256 512 1024 2048 4096" and you see speed different.

Thanks!

[jmac698]

Ok, I will update the one file and try to read the correct installation directory.
Thanks for the testing and feedback.

[Boulder]

Speaking of feedback, does anyone else experience the same issue? Or is no one else processing HD material?I've noticed a similar issue sometimes with SD material as well. Using a radius of over 2 frames is somewhat slower (or at least was with my old Q6600), and that's the big difference between medium and the lower defaults.

[asarian]

I've noticed a similar issue sometimes with SD material as well. Using a radius of over 2 frames is somewhat slower (or at least was with my old Q6600), and that's the big difference between medium and the lower defaults.

The problem for me is not so much that 'medium' takes a bit longer than 'low' (which is totally reasonable), but that sometimes 'medium' (or higher) slows down by an order of magnitude (like I said, ca. 0.08 fp/s on my i7 980X). In fact, it slows down so much that you think the process is stuck; only it isn't (it looks correctly when you examine the output) -- it just goes extremely slow; like where it would suddenly take 700 hours to complete.

Like I stated, the slowdown doesn't always happen; merely on some sources (I suspect because of the the grain being pretty intense).

[cretindesalpes]

When the framerate suddenly drops by one order of magnitude or even more, try to increase the frame cache memory, for example with SetMemoryMax(1024) at the beginning of your script.

[asarian]

When the framerate suddenly drops by one order of magnitude or even more, try to increase the frame cache memory, for example with SetMemoryMax(1024) at the beginning of your script.

Thanks! Never really understood the memory max variable. Gonna give it a try (when the current render is done); and I'll let you know if it worked. :)

[LaTo]

It would be great to find a script & sample that isolate the problem. Someone?

[chainring]

Speaking of feedback, does anyone else experience the same issue? Or is no one else processing HD material?I'm currently doing some 1080 BD to 720p downconversions and have no problems. If I try to do anything with SetMTMode, no matter the mode and number of threads, all hell breaks loose.

i7 2600K
4 GB RAM
Win7 x64 Ult

Avisynth 2.6 with SET's latest MT .dll
Latest MeGUI set for dev updates
All the correct versions of Avisynth plugin's for MCTD, except I'm using the latest Masktools.

[chainring]

When the framerate suddenly drops by one order of magnitude or even more, try to increase the frame cache memory, for example with SetMemoryMax(1024) at the beginning of your script.I noticed that with, IIRC, when playing with SetMTMode.

[asarian]

I noticed that with, IIRC, when playing with SetMTMode.

It says here (http://avisynth.org/mediawiki/Internal_functions/Control_functions) SetMemoryMax is limited to 512MB on 2.5.8. :(

[Didée]

That limit is only for the *default* value (so that e.g. on a 16GB machine, the default won't be set to a 4GB-ish value).

You can of course specify a value >512MB manually.

[asarian]

That limit is only for the *default* value (so that e.g. on a 16GB machine, the default won't be set to a 4GB-ish value).

You can of course specify a value >512MB manually.

Guess I read it wrong then. My bad. :) Thanks!

[DVDBob]

Hello.

I use DGIndexNV and this script, but i only convert with 2-3 fps, if i use medium setting.

How can i get it to convert faster???

[x264Boy]

Hello.

I use DGIndexNV and this script, but i only convert with 2-3 fps, if i use medium setting.

How can i get it to convert faster???

I got same problem.

[Reino]

I've been trying out some denoisers today, but I'm having some trouble getting MCTemporalDenoise to work.

For MCTemporalDenoise my Avisynth plugin directory contains:
addgrain.avs, AddGrainC.dll, DctFilter.dll, deblock.dll, Deblock_QED_MT2.avs, EEDI2.dll,
FFT3DFilter.dll, gradfun2db.dll, GradFun2DBmod.v1.5.avsi, LSFmod.v1.9.avsi, MCTemporalDenoise
.v1.4.20.avsi, mt_masktools-25.dll, mt_masktools-26.dll, mvtools2.dll, RemoveGrain.dll,
Repair.dll, RSharpen.dll, SangNom.dll, TTempSmooth.dll and fftw3.dll in the WINDOWS-directory

The following error-message shows up with "MCTemporalDenoise(settings="low")":
Evaluate: System exception - Illegal Instruction
(GradFun2DBmod.v1.5.avsi, line 291)
(MCTemporalDenoise.v1.4.20.avsi, line 1072)
(MCTemporalDenoise.v1.4.20.avsi, line 1090)
(New File (2), line 10)

Setting up this filter proves to be quite a challenge, so can somebody point me in the right direction?

P.s. My computer is rather old; AMD Athlon XP 3200+, so only SSE. Hope that's not the cause.

[jmac698]

It's sad that no one has noticed the installer
http://www.sendspace.com/file/1gggec

I had mentioned it here:
http://forum.doom9.org/showthread.php?t=162622
It's also here:
http://avisynth.org/mediawiki/MCTemp...quired_Filters

[Reino]

I have seen it, I have used it, but it caused Avisynth to crash completely. At least now, collecting everything by hand, I get an error message.

[jmac698]

SSE only is your problem.
I only included optimized versions I believe.

[Reino]

You're absolutely sure? Quickly looking through each filter's readme didn't reveal one to be SSE2 only.
As I mentioned earlier, Avisynth here crashed on your package with optimized versions. The fact it doesn't crash on my handpicked collection shows you it's not a SSE or SSE2 problem I guess.
Something else must be the problem...

[LaTo]

Seriously I don't understand why people have always problems to set up MCTD.
It's *very* easy, for each filter take the latest version. Put the corresponding .dll or .avsi in autoloading folder.
That's all.

(Sometime you need to choose between .dll according to your avisynth's version and to your CPU's instructions. Read the information file of each filter before.)

[jmac698]

No, not sure just a suggestion. However for me it worked, but I used avisynth 2.58 32bit. If you can find the problem and proper versions I can update the installer.

Ok 291 is

\ : Eval("dither." + custom)

so your custom string is incorrect. Since you used preset low that's strange. Try playing with the script, setting custom to empty for example...

[Reino]

Put the corresponding .dll or .avsi in autoloading folder.Keep it cool, ok? I did exactly that:For MCTemporalDenoise my Avisynth plugin directory contains:
addgrain.avs, AddGrainC.dll, DctFilter.dll, deblock.dll, Deblock_QED_MT2.avsi, EEDI2.dll,
FFT3DFilter.dll, gradfun2db.dll, GradFun2DBmod.v1.5.avsi, LSFmod.v1.9.avsi, MCTemporalDenoise
.v1.4.20.avsi, mt_masktools-25.dll, mt_masktools-26.dll, mvtools2.dll, RemoveGrain.dll,
Repair.dll, RSharpen.dll, SangNom.dll, TTempSmooth.dll and fftw3.dll in the WINDOWS-directoryAll latest versions. Also libfftw3f-3.dll resides in the WINDOWS-dir btw. Afaik the only plugin with multiple (optimized) versions available is RemoveGrain, and I'm even using the most basic one.
so your custom string is incorrectMCTemporalDenoise(settings="low") is all I initially tried. I haven't touched any other command. MCTemporalDenoise() doesn't work either.
No matter what I try, I keep getting the error message mentioned earlier.

Btw jmac698, I found out why Avisynth failed to load at all. Deblock_QED_MT2.avsi within your package is saved as UTF-8 when it should be ANSI! Have people even successfully used your package? You can't use Avisynth at all if you haven't solved this first.
AND you forgot MVTools2. Without it you'll get:Script error: there is no function named "MSuper"
(MCTemporalDenoise.v1.4.20.avsi, line 712)
And what is AvsRecursion.dll for, when it's not required for MCTemporalDenoise.v1.4.20.avsi.

This whole filter is driving me nuts atm...

Using jmac698's package, with RemoveGrain.dll, Repair.dll and RSharpen.dll instead of their SSE2 counterparts and with an ANSI version of Deblock_QED_MT2.avsi gives me the same error as in my initial post

[LaTo]

Keep it cool, ok?
Are we friends? So stop talking to me this way!
I do not see when I could be offensive to you...

I did exactly that
No you have not done what I said above.
Moreover I just saw that your mistake is obvious, simply read the documentation of each filter to find the solution.

[Reino]

Nice attitude!
Forget about it...

[the_weirdo]

..., mt_masktools-25.dll, mt_masktools-26.dll,...


Not sure this is the cause of your problem, but you only need one of those two. Choose the one that compatible with your Avisynth's version.

[Reino]

Deleting mt_masktools-26.dll (I'm using Avisynth 2.5.8) doesn't make a difference. Thanks though.

[LaTo]

The error said this:
Evaluate: System exception - Illegal Instruction
(GradFun2DBmod.v1.5.avsi, line 291)
*Illegal Instruction*: your CPU doesn't support some used instructions


First thing, we check the line 290-291 of GradFun2DBmod.v1.5.avsi:
grain = custom=="empty" ? dither.addgrainC(str,strC,0,0)
\ : Eval("dither." + custom)
diff = custom=="empty" ? blankclip(dither,color_yuv=$808080).addgrainC(str,strC,0,0)
\ : Eval("blankclip(dither,color_yuv=$808080)." + custom)
These lines only contains one filter call: AddGrainC, it should be the culprit.



Second step, we remove AddGrainC's calls to confirm the problem:
grain = custom=="empty" ? dither
\ : Eval("dither." + custom)
diff = custom=="empty" ? blankclip(dither,color_yuv=$808080)
\ : Eval("blankclip(dither,color_yuv=$808080)." + custom)
If this modification works, it's a problem with AddGrainC.



Third point, we look at the readme and the official thread.

Readme states that AddGrain only requires SSE cpu. So it should work on your CPU, strange...

But in the official thread (http://forum.doom9.org/showthread.php?t=111849), it seems that foxyshadis might forget to remove the SSE2 switch during the compilation.



If everything is correct, you should post on AddGrainC's thread to get a fix.

So you see, I hope to be enough *cool* for you :p

[jmac698]

And that's the same spot I pointed out as well.. though didn't go as far..

[Gavino]

And that's the same spot I pointed out as well.. though didn't go as far..
Ok 291 is

\ : Eval("dither." + custom)

so your custom string is incorrect.
What you failed to take into account is that for a multi-line statement, the line numbers reported might be out by one. The actual error occurred on line 290 (addgrainC).

[jmac698]

good tip, thanks!

[Reino]

Well, there you have it. AddGrainC is indeed the culprit. After having removed it's calls from GradFun2DBmod.v1.5.avsi, your filter actually works. I'll definitely stop by foxyshadis's AddGrainC thread. Thanks for your help.

You know, I've never dug deep into Avisynth filter-scripts like this. I never alter or even touch script at all. Whenever I had an Avisynth error, it was always a faulty command in the initial script or an issue with the dependencies (dlls). So all in all, there's a bit more to "It's *very* easy, for each filter take the latest version. Put the corresponding .dll or .avsi in autoloading folder." than you'd think at first, but hey, we learn everyday.

I was rather indignant at your initial post, as if I just asked the most stupidest thing. On the other hand, I know the feeling however from my own experience with people asking rather silly questions at first site. Anyway, let's forget about our altercation. Thanks again.

[LaTo]

You know, I've never dug deep into Avisynth filter-scripts like this. I never alter or even touch script at all. Whenever I had an Avisynth error, it was always a faulty command in the initial script or an issue with the dependencies (dlls). So all in all, there's a bit more to "It's *very* easy, for each filter take the latest version. Put the corresponding .dll or .avsi in autoloading folder." than you'd think at first, but hey, we learn everyday.
My first post was referring to a global view of errors caused by the lack of reading the documentation (mt_masktools-25.dll & mt_masktools-26.dll).

The purpose of my post was not to be offensive, it was just an observation about recurring errors over a lot of pages in this thread.


I was rather indignant at your initial post, as if I just asked the most stupidest thing. On the other hand, I know the feeling however from my own experience with people asking rather silly questions at first site. Anyway, let's forget about our altercation. Thanks again.
Sorry about that, but your answer was really a bit too offensive... So that is why my second post was such direct.

Anyway there is nothing personal. I'm glad the problem is partially solved, don't hesitate if you have any further questions ;)

[jmac698]

I'm glad you've resolved your issues, and it looks like there's a new version of addgrainC, so I can see if I can account for that in the installer. I think there's a cpudetect plugin for the installer. Anyhow that's low on my priority list for today...

[fbs]

Has anyone managed to run it with multithreading?

[chainring]

Has anyone managed to run it with multithreading?Yes.
I have the "SetMemoryMax()" in there, but you may or may not need it. I have 16GB RAM and bumped it up from the default max of 512 to 768. In all honesty, I haven't seen any changes by going higher or lower, so I just leave it, and that's for SD and HD sources.

From what I've read, SetMTMode has to be set to 5 before your source call and the second number is threads, which if you don't set here will effectively set you to one. I have a i7 2600K and set it to 3 since going any higher does nothing.

The later SetMTMode sets it back to a more aggressive multi-threading mode.

There ya go!

SetMemoryMax(768)
SetMTMode(5,3)
LoadPlugin("C:\Apps\Video\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("E:\VIDEO\01 IFO\00 NO ANGLES\007_WORLD_NOT_ENOUGH\VIDEO_TS\VTS_01_1.d2v", info=3)
LoadPlugin("C:\Apps\Video\MeGUI\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)

crop( 8, 62, -8, -66)

SetMTMode(2)

MCTemporalDenoise()

[Gavino]

From what I've read, SetMTMode has to be set to 5 before your source call and the second number is threads, which if you don't set here will effectively set you to one.
You're right that the second parameter is only used on the first call, but the default is the number of processors available, not one.

[fbs]

That's what I do and I always get crashes after some minutes.. maybe that's because I'm using QTGMC's dlls..? qtgmc works fine with them.. :\

[-Vit-]

It could be the opposite fbs, maybe you need more modded plugins...
The general advice I offer for multithreading in the first post of the QTGMC thread applies to all multithreaded scripts:
- Use SEt's 2.6 MT, all other MT versions have known threading bugs
- Recompile all plugins with SEt's fixes. This has been done in the QTGMC modded plugins.

MCTemporalDenoise uses plugins that are not used by QTGMC. Some of those plugins will contain known threading bugs, unless their authors have updated them recently for that reason.

[chainring]

You're right that the second parameter is only used on the first call, but the default is the number of processors available, not one.Now, for clarification sake, is that the number of physical CPU's, or the number of real cores, not including hyperthreading, etc...?

[Didée]

As much as there are diagrams in the Taskmanager. ;)

I.e. the logical cores from hyperthreading are included. A 4C8T CPU identifies itself to the OS as to have 8 cores.

[fbs]

MCTemporalDenoise uses plugins that are not used by QTGMC. Some of those plugins will contain known threading bugs, unless their authors have updated them recently for that reason.

I wonder where I could find these extra modded plugins ~~

[-Vit-]

I wonder where I could find these extra modded plugins ~~
Rear end of a compiler
:p

[chainring]

As much as there are diagrams in the Taskmanager. ;)

I.e. the logical cores from hyperthreading are included. A 4C8T CPU identifies itself to the OS as to have 8 cores.Excellent, helpful as always, Didée. Thank you!

[Corpsecreate]

My encode always crashes when I use MCTemporalDenoise on my 1080p anime source. It is an extremely noisy video source so I have to use settings = high or very high. I have downscaled it to 720p before denoising it but I still have the same problem. Is there any way I can stop this from happening? The exact line in my script is:

MT("""MCTemporalDenoise(settings="high")""", threads = 4, overlap=0)

The results of MCTD are amazing and nothing I've tried comes close to it but if it keep crashing, I may have to use something else :(

[kypec]

Is there any way I can stop this from happening? The exact line in my script is:

MT("""MCTemporalDenoise(settings="high")""", threads = 4, overlap=0)

The results of MCTD are amazing and nothing I've tried comes close to it but if it keep crashing, I may have to use something else :(
Did you try to run your script in non-MT mode? Usually it's multi-threading that causes all problems with such demanding processing...

[fbs]

It seems it only works stable if it's the LAST line of the .avs. if there's a converttorgb() or whatever at the end, it crashes after some minutes. (it's not a immediate crash)
Why would it be like that? SEt's bug? :P

setmemorymax(600)
setmtmode(3, 7)
avisource("d:\ds000.avi")
ChromaShift(C=4)
crop(16, 8, -20, -8)
setmtmode(2)
MCTemporalDenoise(settings="medium",enhance=true,stabilize=true,interlaced=true,sharp=false,chroma=true,edgeclean=true,truemotion=true,post=2,ncpu=1)

I'm afraid VirtualDub will have problems with it's output, because the source .avi is an interlaced YV12 720x480 vhs rip.. and I couldn't put a "converttorgb()" at the end... am I in trouble?

oh I'm using set's 2.6mt with -vit-'s modded plugins \o

[Corpsecreate]

Ok I managed to get it to stop crashing by removing the MT call.

I noticed that on scene changes, the denoise isnt applied well and you get grain/noise for the first 1-2 frames after a scene change. I understand that this is part of how a temporal denoiser works but is there any way I can reduce the effect? I am using the preset "high" setting.

[An IP BreAKDoWN]

So I'm using this filter for an 1080p encode on a low setting and for one it runs at 1 fps and two to fix that 1 fps I tried using MTMode. However when I use MT it encodes the first 8 frames and the rest is black. So can anyone recommend a plugin or something that will allow me to encode at a decent rate, or help me with MT? Here is my script:

##Setting Memory (HW has 16 GB)##
SetMemoryMax(1024)

##Plugins##
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\DGAVCDecode.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\Decomb521VFR1.3_P4-Athlon.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\VSFilter.dll")
Import("C:\Program Files (x86)\AviSynth 2.5\plugins\MCTemporalDenoise.v1.4.21.avsi")

##Setting Speed##
SetMTMode(5)

##Source Video##
AVCSource("G:\%PATH%\video.dga")

##Setting Cores##
SetMTMode(8)

##Decimate for timecodes##
Decimate(timecodes="G:\%PATH%\timecodes.txt",
\vfrstats="G:\%PATH%\timecodes.vfrstats")

##Crop Before Filters to Reduce Times##
crop(244, 4, -244, -4)

##Some Tweaking##
Spline36Resize(960,720)

##Them Good Ole' Watermarks##
textsub("C:\Program Files (x86)\AviSynth 2.5\plugins\BreAKDoWN.ssa")

##Filtering##
MCTemporalDenoise(settings="low")

[Taurus]

So I'm using this filter for an 1080p encode on a low setting and for one it runs at 1 fps and two to fix that 1 fps I tried using MTMode. However when I use MT it encodes the first 8 frames and the rest is black. So can anyone recommend a plugin or something that will allow me to encode at a decent rate, or help me with MT? Here is my script:

##Setting Memory (HW has 16 GB)##
SetMemoryMax(1024)

##Plugins##
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\DGAVCDecode.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\Decomb521VFR1.3_P4-Athlon.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\VSFilter.dll")
Import("C:\Program Files (x86)\AviSynth 2.5\plugins\MCTemporalDenoise.v1.4.21.avsi")

##Setting Speed##
SetMTMode(5)

##Source Video##
AVCSource("G:\%PATH%\video.dga")

##Setting Cores##
SetMTMode(8)

##Decimate for timecodes##
Decimate(timecodes="G:\%PATH%\timecodes.txt",
\vfrstats="G:\%PATH%\timecodes.vfrstats")

##Crop Before Filters to Reduce Times##
crop(244, 4, -244, -4)

##Some Tweaking##
Spline36Resize(960,720)

##Them Good Ole' Watermarks##
textsub("C:\Program Files (x86)\AviSynth 2.5\plugins\BreAKDoWN.ssa")

##Filtering##
MCTemporalDenoise(settings="low")
Should be SetMTMode(2,8)
if your eight core monster handles it.
Some encoding frontends prefer
Distributor()
at the end of the script.
But for Virtualdub and MeGui this is a NoNo.
AVCSource("G:\%PATH%\video.dga")
Is this your real path?
And maybe try another decoder...
Which Avisynth MT version?