Optimizing MCTemporalDenoise for MT

[asarian]

In Windows 7, with 32-bit AviSynth, I'm trying to optimize MCTemporalDenoise for use with my i7 980x. I tried it with SetMTMode first:


SetMTMode(5,6)
LoadPlugin("C:\Program Files (x86)\Haali\MatroskaSplitter\avss.dll")
DSS2("C:\jobs\gits.mkv")
SetMTMode(2)
Crop(32, 38, -32, -38)
LanczosResize(1920, 1068)
MCTemporalDenoise(settings="medium")
GradFun2DBmod(thr=1.8,thrC=1.0,mode=0,str=2.4,strC=0.0,temp=50,adapt=64,mask=false,show=false)
SupTitle("C:\jobs\gits_exp.sup", forcedOnly=false, swapCbCr=false, relocate=true, relocOffset="0,0,0,0")


It won't go in SetMTMode(2), for some reason (immediate crash); though SetMTMode(3) works fine. This seems to go faster, though:


FFVideoSource("C:\jobs\gits.mkv")
Crop(32, 38, -32, -38)
LanczosResize(1920, 1068)
MT("""MCTemporalDenoise(settings="medium")""", threads=5, overlap=8)
MT("""GradFun2DBmod(thr=1.8,thrC=1.0,mode=0,str=2.4,strC=0.0,temp=50,adapt=64,mask=false,show=false)""", threads=5, overlap=8)

Etc.


However, AviSynth really trips when I set threads to 6, and spills its guts all over my command box (as in: outputting a few garbled lines over the screen and crashing). At 5, it seems to run stable, though. Multi-threading still seems a hazardous game. :)

I still can't get full core saturation with the latter, though (at least not all the time). I guess 5 threads just isn't enough to occupy the 12 logical cores; but more threads make AviSynth crash badly.

I'm rather new to the whole AviSynth multi-threaded thing, so I wonder whether there's other ways I can optimize this kind of job for better multi-threading?

[board123]

Firstly, it would be slightly faster to put MCTD and Gradfun in the same MT call.
MT("""MCTemporalDenoise(settings="medium").\
GradFun2DBmod(thr=1.8,thrC=1.0,mode=0,str=2.4,strC=0.0,temp=50,adapt=64,mask=false,show=false)""", threads=5, overlap=8)

Secondly, I believe there are restrictions on the number of threads you can use for MT. I don't mean a limit, I mean the resulting clip dimensions of each MT clip (i.e. 5 threads = 5 clips). I think the resultant clips must be mod16. If your original clip is 1280x720 and you're making 5 threads, then each clip is 1280x144, which is mod16 (144 / 16 = 9). If you use 6 threads, then each clip is 1280x120, which is not mod16 (120 / 16 = 7.5).

To get around this sometimes, you can use splitvertical=true in MT to split the horizontal dimension instead of the vertical. However, 6 threads is still not doable since 1280 / 6 = 213.3.

[Gavino]

MT("""MCTemporalDenoise(settings="medium").\
GradFun2DBmod( ... )""", threads=5, overlap=8)
You can't split a string literal across lines with '\' - the '\' becomes part of the string. Either put the whole thing on one line or use the '+' (string concatenation) operator.
MT("""MCTemporalDenoise(settings="medium").""" + \
"""GradFun2DBmod( ... )""", threads=5, overlap=8)
I think the resultant clips must be mod16. If your original clip is 1280x720 and you're making 5 threads, then each clip is 1280x144, which is mod16 (144 / 16 = 9). If you use 6 threads, then each clip is 1280x120, which is not mod16 (120 / 16 = 7.5).
MT itself doesn't impose any limit (although it does ensure each clip is at least mod2), but the filters used (MCTD, etc) might well have mod restrictions. Calculating the clip size is not as simple as that, since you also have to take the overlap into account.

[board123]

You can't split a string literal across lines with '\' - the '\' becomes part of the string. Either put the whole thing on one line or use the '+' (string concatenation) operator.
Yes you can. All of my scripts split lines using '\' when using MT. If the line begins or ends with '\' then it doesn't get included inside the string literal. Try it and see.


MT itself doesn't impose any limit (although it does ensure each clip is at least mod2), but the filters used (MCTD, etc) might well have mod restrictions. Calculating the clip size is not as simple as that, since you also have to take the overlap into account.
That's a good point. The clip dimensions would be calculated that way only if you have 0 overlap. So you're saying MT only restricts to mod2? I remember reading in one thread that MT itself has dimension restrictions, but I may be mistaken.

MCTD itself must be at least mod4, but must also be mod16 when deblock=true (according to changelog). However, medium preset uses deblock=false, so that can't be the problem. Maybe MT is just unstable with 6 threads.

[Gavino]

Yes you can. All of my scripts split lines using '\' when using MT. If the line begins or ends with '\' then it doesn't get included inside the string literal.
Actually, it does get included - but you're right that it still works anyway!
Here's why - the string ends up including both the '\' and the following newline, so its contents look like this:
MCTemporalDenoise(settings="medium").\
GradFun2DBmod( ... )

When this string is passed to Eval (inside MT), the '\' is treated as a line continuation marker and all is well. Thinking about it, it would also work as a two-line string without the '.\':
MT("""MCTemporalDenoise(settings="medium")
GradFun2DBmod( ... )""", threads=5, overlap=8)
So you're saying MT only restricts to mod2?
Not restricts as such (I said 'ensures') - what it does is simply round up each subclip size to the nearest even number.

[asarian]

That's a good point. The clip dimensions would be calculated that way only if you have 0 overlap. So you're saying MT only restricts to mod2? I remember reading in one thread that MT itself has dimension restrictions, but I may be mistaken.
True or false, your mod16 trick seems to work, far as MT is concerned. :) No more crashes. I tried it with different sources and resolutions.

Thanks!

[board123]

True or false, your mod16 trick seems to work, far as MT is concerned. :) No more crashes. I tried it with different sources and resolutions.

Thanks!
No prob. I guess you can just stick with what works until something else breaks.

[asarian]

No prob. I guess you can just stick with what works until something else breaks.
The MT"" call doesn't work under every condition (which is documented, btw). On MCTemporalDenoise(settings="high"), for instance, it crashes. But SetMTMode hangs in there:

SetMTMode(5,8)
FFVideoSource("C:\jobs\highlander.mkv")
SetMTMode(3)
Crop(0, 22, 0, -22)
MCTemporalDenoise(settings="high")
GradFun2DBmod(thr=1.8,thrC=1.0,mode=0,str=2.4,strC=0.0,temp=50,adapt=64,mask=false,show=false)

Yes, I can set threads to 8, even; but CPU utilization with SetMTMode is deplorable for MCTemporalDenoise(settings="high"). I can get max < 35% core saturation out of it. :( SetMTMode(3) indeed seems (significantly?) slower, yet SetMTMode(2) crashes, so I have little choice.

[levi]

I've only had success running MCTemporalDenoise with SetMTMode(2) if MCTD is the last line in the .avs