MT 0.7(+custom avisynth)a filter to run filters multithreaded. Usefull for SMP or HT

[3ngel]

With Mt() from 50% (whitout MT()) i go to 75%.

Whit SetMTMode() i go from 50 to 55.

But there is an important point using MT().

The frames resulting are not identical to that produced whitout MT(), this because my script uses a lot of masking and spatial/sharp, so the frames with MT appear to have more details lost (due to smaller window frame passed to the masks).

So beware!

[tsp]

3ngel: yes but what about the framerate? it doesn't all ways increase linearly with the cpu utilization. About the differences in the output try increasing overlap (like MT(overlap=4) ) but again with out the actual script used it is difficult to say what is causing it.

[3ngel]

About the framerate there is an increase of +-1 fps.

[canuckerfan]

hi, I've got a script here which I'm trying to speed up...

function hq_filter(clip c)
{
  c=c.RemoveNoiseMC(rdlimit=7,rgrain=1,denoise=0,sharp=true)
  c=c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=0,wide=false,ss_x=1.3,ss_y=1.3)
  return c
}

setmemorymax(512)
mpeg2source("G:\Temporary\Scripts\VideoFile.d2v",idct=5,info=3)

Crop(0,56,0,-62)
Tweak(sat=1.07)

hq_filter()

a=last
b=a.MT("fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)")
SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)

AddBorders(0,56,0,62)
ConvertToYUY2()

i'm wondering if something could be done to speed up hq_filter, mpeg2source, and seesaw as a whole? I already got fft3d in MT since I've heard that works well...

[TheRyuu]

Originally Posted by canuckerfan

hi, I've got a script here which I'm trying to speed up...

function hq_filter(clip c)
{
  c=c.RemoveNoiseMC(rdlimit=7,rgrain=1,denoise=0,sharp=true)
  c=c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=0,wide=false,ss_x=1.3,ss_y=1.3)
  return c
}

setmemorymax(512)
mpeg2source("G:\Temporary\Scripts\VideoFile.d2v",idct=5,info=3)

Crop(0,56,0,-62)
Tweak(sat=1.07)

hq_filter()

a=last
b=a.MT("fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)")
SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)

AddBorders(0,56,0,62)
ConvertToYUY2()

i'm wondering if something could be done to speed up hq_filter, mpeg2source, and seesaw as a whole? I already got fft3d in MT since I've heard that works well...

Why not just put SetMTMode(2, 0) right in the beginning.

Also, you can MT the entire SeeSaw script like so:

MT("a=last
b=a.fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)
SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)")

[tsp]

mt and setmtmode doesn't always work to good together. You can also try this variation:

function hq_filter(clip c)
{
 return c.RemoveNoiseMC(rdlimit=7,rgrain=1,denoise=0,sharp=true)
}

function mt_filter(clip c)
{
c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=0,wide=false,ss_x=1.3,ss_y=1.3)
fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)
return SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)

}

setmemorymax(512)
mpeg2source("G:\Temporary\Scripts\VideoFile.d2v",idct=5,info=3)

Crop(0,56,0,-62)
Tweak(sat=1.07)

hq_filter()
last.mt("mt_filter()",overlap=2)

AddBorders(0,56,0,62)
ConvertToYUY2()

[0gg]

I really don't understand the syntax code for .avs ...

Can you post here a full avs script to test MT with x264 codec on my Dual Clovertown ?

I don't need crop, blur ... just :
desinterlace
720x576
the best image quality


Thank's a lot for your help

[3ngel]

I would like to a question about a thing i haven't understood well.

In normal mode (without MT) if i have

Filter1()
Filter2()

Filter2 request a frame and obtain the frame when Filter1() finishes.

But in "temporal mode", if i have the same

------
SetMTmode()
Filter1()

SetMTmode(thread=1)
Filter2()
------

in this case we have (in the minimum case) 2 parallel

Filter1(n) Filter1(n+1)

My doubt is:
If Filter1(n) finishes but Filter1(n+1) not, Filter2() request and obtain the next frame (n+1) (that at this point is not finished by Filter(n+1) an so it obtains an unprocessed frame), or waits until the second thread Filter1(n+1) finishes, and so obtain the correct (processed) n+1 frame?

Pheraps it's a basic question but i have a doubt

[foxyshadis]

When Filter2 requests frame n, Filter1 will roughly simultaneously process frames n and n+x, x in this case being the determined offset in requested frames (if you were playing that script forward, it would be 1). Filter2 will obtain n as soon as it's ready, whether n+x is ready or not. But n+x is not ready by the next access, it'll just wait around a little longer for it to finish, it never hands back an unprocessed frame.

[3ngel]

But n+x is not ready by the next access, it'll just wait around a little longer for it to finish, it never hands back an unprocessed frame.

Ok, that's what i wanted to know. That the "coherence" is mantained and the next filter waits (in the case) for the previous filter to complete and finish the requested frame before accepting it.
Thank you.

[TheRyuu]

Will the old 2.57 MT avisynth.dll work with the new 2.58a version of avisynth?

Or does that need a new dll?

I'd love to try out the new avisynth but I need the MT functions...

[Boulder]

It would require tsp to build a new MT-supported Avisynth version. That means the new release won't work.

[dchard]

Hi!

I want to multi-thread my avisynth script (because under encoding, only 55% of the CPU is used).

Finally the whole script is working except the multi threading. I read a lot, but I always got some errors.

I have a dualcire (yonah) cpu, and I always got 50% cpu useage.

This is the script I want to multi-thread:

#PLUGINS

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\warpsharp.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\Ylevels.avsi")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools.dll")



directshowsource("E:\Film\The Matrix\demuxed_tracks\matrix.grf",audio=false,fps=23.976,seekzero=false,seek=true).trim(0,195880)
converttoyv12()
crop(0,140,0,-140)


# denoise
removegrain(mode=1)

#sharpen
LimitedSharpenFaster(strength=40)

# YLevels usage:
# YLevels(0, 1.2, 255, 0, 255, false)
# YLevels(0, 1.2, 255)
YLevels(gamma=1.2)

# Tweak()
# hue=-180.0 to +180.0, default 0.0
# sat (0.0 to 10.0, default 1.0)
# bright (-255.0 to 255.0, default 0.0)
# cont (0.0 to 10.0, default 1.0)
# coring = true/false
# startHue (default 0), endHue (default 359): (both from 0 to 359; given in degrees.).
# maxSat (default 150), minSat (default 0): (both from 0 to 150 with minSat<maxSat; given in percentages).
# interp: (0 to 5, default 4)
tweak(cont=1.1)

#resizing

spline36resize(1280,544)

Please tell me what to do (what to install, what to write in the script etc).

Also, this script is for HD-DVD --> 720p reencode script, so anyone have any other idea, which I can improve the script, please tell me too. (ofcourse the values in the script are only for testing, not the final ones)

Thank you!

Dchard

[foxyshadis]

The first post should explain it all, but if you have the all-important avisynth.dll, you only need to add SetMTMode(2) to the start of the script. None of those filters appear to be incompatible.

[dchard]

Here is the multi-threaded script:

-----------------------------------

SetMTMode(2)

#PLUGINS

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\warpsharp.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\Ylevels.avsi")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Avisynth.dll")


directshowsource("E:\Film\The Matrix\demuxed_tracks\matrix.grf",audio=false,fps=23.976,seekzero=false,seek=true).trim(0,195880)
crop(0,140,0,-140)
converttoyv12()



# denoise
removegrain(mode=1)

#sharpen
LimitedSharpenFaster(strength=40)

# YLevels usage:
# YLevels(0, 1.2, 255, 0, 255, false)
# YLevels(0, 1.2, 255)
YLevels(gamma=1.2)

# Tweak()
# hue=-180.0 to +180.0, default 0.0
# sat (0.0 to 10.0, default 1.0)
# bright (-255.0 to 255.0, default 0.0)
# cont (0.0 to 10.0, default 1.0)
# coring = true/false
# startHue (default 0), endHue (default 359): (both from 0 to 359; given in degrees.).
# maxSat (default 150), minSat (default 0): (both from 0 to 150 with minSat<maxSat; given in percentages).
# interp: (0 to 5, default 4)
tweak(cont=1.1)

#resizing

spline36resize(1280,544)

-----------------------------

When I open it in megui, i got:

Script error: there is no function named "SetMTMode" (line 3)

What have I do in this case?

Dchard

[Boulder]

You'll need to extract the avisynth.dll from the MT v0.7 package into your Windows\System32 directory, overwriting the old one.

[dchard]

Originally Posted by Boulder

You'll need to extract the avisynth.dll from the MT v0.7 package into your Windows\System32 directory, overwriting the old one.

Oh, I don't know that.

Now its "working" (means I don't get any error messages), but its only uses about 50-60% of CPU time. The strange is, that both cores are used equally, but only a half. (this is a coreduo, yonah cpu with 2MB L2, and 2x512MB ddr2 in dualchannel.

Dchard

[Boulder]

In my tests, SetMTMode usually yielded less performance gain when compared to using MT(). However, in your case there's not many places where you can even use MT and they are not the bottlenecks of the script, far from it.

By the way, you probably don't want to load avisynth.dll in your script or even keep it in the plugins folder..it doesn't belong there

[dchard]

Originally Posted by Boulder

In my tests, SetMTMode usually yielded less performance gain when compared to using MT(). However, in your case there's not many places where you can even use MT and they are not the bottlenecks of the script, far from it.

By the way, you probably don't want to load avisynth.dll in your script or even keep it in the plugins folder..it doesn't belong there

Yes, I removed the avisynth.dll from the script, and form the folder :-)

This means that this is the maximum, that I can pull out from the CPU? Somewhere I read that when using avisynth multi-threading, it is normal that I can't see 100% load on cpu cores, but in reality, both cores working with full throttle.
This is true?

Dchard

[tsp]

dchard: try this version with uses mt instead of setmtmode()

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\warpsharp.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\Ylevels.avsi")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")



directshowsource("E:\Film\The Matrix\demuxed_tracks\matrix.grf",audio=false,fps=23.976,seekzero=false,seek=true).trim(0,195880)
crop(0,140,0,-140)

mt("converttoyv12().removegrain(mode=1).LimitedSharpenFaster(strength=40).YLevels(gamma=1.2).tweak(cont=1.1)",border=4)

#resizing
mt("spline36resize(1280,last.height)")
mt("spline36resize(last.width,544)",splitvertical=true)