MT 0.7(+custom avisynth)a filter to run filters multithreaded. Usefull for SMP or HT

[tsp]

I made this small filter now that dualcore processors begins to show up. I don't know how much faster the filters will be when running on a Pentium 4 HT but with my old dual celeron 400 MHz I got a 40 % speed increase. Also usefull for all the 8 way dualcore opteron computers :)
Also Included is a custom build of avisynth 2.5.7 that provides the two function SetMTMode and GetMTMode and some change to internal filters to support multithreading.

please post if there is some filters that doesn't work or what speed increase you got.

Get version 0.7 here (http://www.avisynth.org/tsp/MT_07.zip) (contains avisynth 2.57 MT version 5[src] (http://www.avisynth.org/tsp/avisynth257MT5_src.7z))
avisynth 2.57 MT version 4 (http://www.avisynth.org/tsp/avisynthMT257.4.zip) [src] (http://www.avisynth.org/tsp/avisynth257MT4_src.7z)
Get version 0.6 here (http://www.avisynth.org/tsp/MT_06.zip)
Get version 0.5 here (http://www.avisynth.org/tsp/MT_05.zip)
or version 0.41 here (http://www.tsp.person.dk/MT_041.zip)

You can also get futher help to MT at the mediaWIKI page here (http://avisynth.org/mediawiki/MT_support_page)

from the readme

MT is a filter that enables other filter to run multithreaded. This should hopeful speed up processing on hyperthreaded multicore processors or multiprocessor systems.
Technical info

Important: Allways remember to judge the result by looking at the speed improvement not the cpu utilization.

MT is a filter that split a frame up in smaller fragment that are processed in individual threads allowing full utilization of multiprocessor or hyperthread enabled computers. I tested it on my old abit bp6 with 2x celeron 400 MHz and it increased the speed by 40%. Note that if you is already getting 100% cpu utilization when processing avs scripts(ie if you're encoding to DivX/XviD) you don't need to use this filter.

The filter works like this avs function:

function PseudoMT(clip c,string filter)
{
a=eval("c.crop(0,0,src.width/2,src.height)."+filter)
b=eval("c.crop(src.width/2,0,src.width/2,src.height)."+filter)
stackhorizontal(a,b)
}


The only difference is that a and b is executed in parallel and it is possible to split the frame into more than 2 pieces. If the filter works with the above script it should work with MT if the filtercode is threadsafe. Dust does not work with the above script so if you want to use iip use another denoiser or get Steady to fix the bug.
Limitations

The filter to be run must only accept one input clip and that is last. Also the filter should not rely on the content of the whole frame(like smart deinterlacers) else there is a risk that only part of the frame will be processed. The filter should also be threadsafe. Most filters are threadsafe but some will produce a wrong result or crach.
Installation

copy mt.dll into the avisynth plugin directory and copy the included avisynth.dll into your windows\system32 directory or where avisynth.dll is located and remember to take a backup of the old avisynth.dll(rename it or something) if you don't have version 2.6 installed.

from version 0.7 two other filters are included too:

MTi() that creates two threads and let each thread process one field before combining them like this avs function
function PseudoMTi(clip c,string filter)
{
a=eval("c.AssumeFieldBased().SeparateFields.selecteven()."+filter)
b=eval("c.AssumeFieldBased().SeparateFields.selectodd()."+filter)
interleave(a,b).weave()
}
like the other pseudoscript a and b are executed in parallel. Note that only two threads are created so it will only use two (virtual) cores.

MTsource() that are used to run source filters multithreaded. It works like this:
function PseudoMTsource(string filter)
{
SetMTmode(2)
eval(filter)
SetMtmode(0)
}
So different from the two other filters it is a temporal filter that fetches frames ahead of time and store them in the cache for fast retrieval.

Syntax
MT(clip clip,string filter,int threads,int overlap,bool splitvertical)

All parameters are named. Function parameters:

clip clip = last
input clip

filter string = No default
filter to run multithreaded. Note that the filter must not change both the frame height and width (but colorspace is okay) and the only 1 input clip is allowed. It can be any build-in filter, avs defined filter or external plugin filter as long as the restrictions are observed.

threads int = 2
number of threads to run. Set this to the number of threads your computer is able to run concurrently.

overlap int = 0
- number of pixel to add at the top and bottom border or left and right border. Increase this if you see artifacts where the frame is split.

splitvertical bool = false
- if true the frame are cut vertical(and the filter is allowed to change the height) else it is cut horizontal(and the filter is allowed to change the width).

MTi
MTi(clip clip,string filter)

All parameters are named. Function parameters:

clip clip = last
input clip. Must be mod2 height for RGB and YUY2 colorspace and mod4 height for YV12 colorspace

filter string = No default
filter to run multithreaded. Note that the filter are allowed to change both width and height at the same time but only 1 input clip is allowed. It can be any build-in filter, avs defined filter or external plugin filter as long as the restrictions are observed.


MTsource
MTSource(string filter,int delta,int threads,int max_fetch)

All parameters are named. Function parameters:

filter string = No default
source filter to run multithreaded. Currently only internal and external source filters are supported (like DirectShowSource, Avisource, MPEG2Source) . You can use an avs defined filter or a non-source filter but it might crash or produce frame corruption.

delta int = 1
this is how many frames there are between each frame request so if you are only going to read every second frame set it to 2 or if you are reading the frames backwards set it to -1. More complex frame access pattern like SelectEvery(10,3,6,7) are not supported (but might work anyway as the requested frames are in the cache, there will just be some waisted memory from non requested frame in the cache)

threads int = 2
number of threads to run. Set this to the number of threads your computer is able to run concurrently.

max_fetch int = 30
This is the maximum number of frames ahead of the currently requested frame that MTsource will fetch. Setting it to low will leaving the threads idle for most of the time and setting it to high will waste to much memory.

Examples:

MT("blur(1)",2,2)

also user defined function (uses variableblur):

MT("unsharp(2,0.7)",2,2)
function unsharpen(clip c,float variance,float k)
{
blr=binomialBlur(c,vary=variance,varc=2,Y=3,U=2,V=2)
return yv12lutxy(blr,c,"y x - "+string(k)+" * y +",y=3,u=2,v=2)
}

This one will not produce the intended result but shows how to use the triple quotes:

MT(""" subtitle("Doh") """,4,0)

License

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License version 2 as published by
the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

Please consider to make some donation.
Version changes:

* 0.1 first release.
* 0.2 Should be more thread safe.
* 0.21 forgot to comment out a Sleep(0)
* 0.25 Added the splitvertical option
* 0.3 More stable(and slower)
* 0.4 Includes a custom version of avisynth 2.56 beta that should speed things up
* 0.41 Minor speed increase
* 0.5 Requires the included modified avisynth 2.5.6 or avisynth 2.6
* 0.6 Bugfix: height can be changed with splitvertical=true without crashing. Also includes modified avisynth 2.5.7
* 0.7 two new filters: MTi(), MTsource() and Avisynth MT 2.5.7.5


modified avisynth 2.5.7

It contains the two new functions SetMTMode() and GetMTMode() and is needed by MT.dll. Install it by overwriting avisynth.dll in your c:\windows\system32 (and remember to take a backup of the old file first)
Technical info

These functions enable avisynth to use more than one thread when processing filters. This is useful if you have more than one cpu/core or hyperthreading. This feature is still experimental.

Syntax:

GetMTMode(bool threads)
threads - if true GetMTMode? returns the number of threads used else the current mode is returned (see below). Default value false.

SetMTmode(int "mode",int "threads")

Place this at the first line in the avs file to enable temporal (that is more than one frame is processed at the same time) multithreading. Use it later in the script to change the mode for the filters below it.
mode - there are 6 modes 1 to 6. Default value 2.

* Mode 1 is the fastest but only works with a few filter
* Mode 2 should work with most filters but uses more memory
* Mode 3 should work with some of the filters that doesn't work with mode 2 but is slower
* Mode 4 is a combination of mode 2 and 3 and should work with even more filter but is both slower and uses more memory
* Mode 5 is slowest(Slower than not using SetMTMode) but should work with all filters that doesn't require linear frameserving (that is the frames come in order (frame 0,1,2 ... last)
* Mode 6 is a modified mode 5 that might be slightly faster (But still slower than not using SetMTMode)

threads - number of threads to use. Set to 0 to set it to the number of processors available. It is not possible to change the number of threads other than in the first SetMTMode. Default value 0.

Example:

SetMTMode(2,0) #enables multihreading using thread = to the number of available processors and mode 2
LoadPlugin("...\LoadPluginEX.dll") #needed to load avisynth 2.0 plugins
LoadPlugin("...\DustV5.dll") #Loads Pixiedust
import("limitedsharpen.avs")
src=AVIsource("test.avi")
SetMTMode(5) #change the mode to 5 for the lines below
src=src.converttoyuy2().PixieDust()#Pixiedust needs mode 5 to function.
SetMTMode(2) #change the mode back to 2
src.LimitedSharpen() #because LimitedSharpen works well with mode 2
subtitle("Number of threads used: "+string(GetMTMode(true))+" Current MT Mode: "+string(GetMTMode())) #display mode and number of threads in use

[MacAddict]

Wow:D Too bad I no longer have the AMD dualie system to test this with. Perhaps by year end most of us will be able to afford a dual core though. Nice work once again tsp.

[tsp]

thanks. The princip in this filter is really simpel, something like this:

function MT(clip c,int threads,string function)
{
clip r[threads]
for(int i=0;i<threads;i++)
{
c.crop(height/threads*i,0,height/threads,0)
r[i]=last.eval(function)
}
stackvertical(r[0],r[1],...,r[threads-1])
}


the difficult part is to make the code thread safe so two threads isn't writing to the same memory at the same time. So I'm working on making the IScriptEnvironment threadsafe.

[hartford]

Does "function" refer to only a single filter?

For example, can function refer to an imported function using several filters, whether those filters are Avisynth filters or plugins?

I'd appreciate a more thorough explanation as to what "function" means.

Thanks.

[edit]

I ran this script on a 3 minute analog capture:

loadplugin("d:\plugins\DeComb521.dll")
loadplugin("d:\plugins\mt.dll")
loadplugin("d:\plugins\FFT3dGPU9b.dll")

Avisource("d:\test-mt.avi").ConvertToYV12

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)

mt(fft3dgpu)

fft3dgpu(sigma=4.0,bw=32,bh=32,bt=3,plane=0,mode=2)


This gave an exception when opened in VirtualDub 1.5.10 and pointed to
the line

mt(fft3dgpu)

[tsp]

hartford: function is like the function to the buildt in filter scriptclip. So it can be an internal(build-in) or external(plugin) filter or a function that is defined in an AVS file. just remember the quotes(or triple quotes) because it's a string. So your example would be like this:

Avisource("d:\test-mt.avi").ConvertToYV12

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)
mt("fft3dgpu(sigma=4.0,bw=32,bh=32,bt=3,plane=0,mode=2)")

But fft3dgpu doesn't work well with MT because only one thread at a time can acces fft3dgpu (Directx really hates multithreading)

you can also use it like this:

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)
MT("fft3d()")

function fft3d(clip c)
{
c
fft3dfilter(plane=0)
fft3dfilter(plane=1)
fft3dfilter(plane=2)
}

[tsp]

new version ready. Should be more thread safe.

Also found a bug? in pixiedust. When using dynamic_cast Pixiedust throws __non_rtti_object exception. And it always complaines about "First-chance exception at 0x015327e2 in virtualdubmod.exe: 0xC000001D: Illegal Instruction" in the debugview.

[tsp]

Another update. Now the filter can change the height or width but not both.

By the way is there anyone using this filter and does it infact increase the speed? Also is there anyone who has the sourcecode for Dust and is willing to do some bugfixing because this filter really doesn't like dust.

[Manao]

Dust has always been close source. Ask Steady to modify his code, if he still maintains it.

[hartford]

mt("fft3dgpu(sigma=4.0,bw=32,bh=32,bt=3,plane=0,mode=2)")

OK, the "quote" stuff fixed it. (I used fft3dgpu only because you authored it and I thought that it would be a "torture" test)

I did a test with the above script using TBilateral and I got no errors:

mt("TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)")


I'll try MT version 0.25 soon and will report.

Thanks.



[added edit]

More testing.

Difference with respect to speed noticed between versions with
VirtualDub 1.15.10:

v 0.1about 7.22 Minutes
v 0.25 about 6:55 minutes

However, VirtualDub v.1.66 crashes when using MT version 0.25. It exits
with no error messages.

[tsp]

Hasn't it been over a year since Steady showed up last?

hartford: How much faster is mt compaired to without it and did vdub 1.6.6 crash at startup or after a couple of frames. There is a new version here (http://www.tsp.person.dk/MT_029.zip) that might be a little faster and/or stable.

[Selur]

nice filter ! :D

"VirtualDub v.1.66 crashes when using MT version 0.25. It exits with no error messages."
=> if you use a filter with mt that requires mod16 make sure your input is mod32, so that if mt splits the frame it's still mod16. (for more than 2 threads one probably has to go to mod64 and so on)

At least for me that stopped VD from closing. :)
=> correction, this prevented normal VD from closing directly though it closes after some frames :(

over here it seems like 0.1 is more stable than 0.25.
(seems to work with VD 1.66, though it crashes with VDMod1.5.10 after some frames)

0.1 + mt("undot()") works under VD1.6.6
0.25 + mt("undot()") didn't work under VD1.6.6 / VDM
0.1 + mt("mergechroma(blur(1.3))") didn't work under VD1.6.6 / VDM


Cu Selur

[hartford]

@tsp

Let me fix an error in previous report and change times to seconds (results were swapped) :

No MT = 568 sec

v0.10 = 415 sec
v0.25 = 442 sec
v0.29 = 596 sec

Yes, v0.29 is slower than not using it :(

VDub 1.66 crashed when loading the script.


@Selur
Input is 640x480.

[tsp]

That makes more sense with the timing. I figured out what was causing the filter to crash after I got the remote debugger working(clever little thing). It was crop that caused it. It just returned a pointer to the videodata so each thread was infact working in the same memory and avisynth doesn't like that. So expect a new version shortly

[tsp]

New version ready. It's not total stable but should run longer. Get version 0.29.1 here (http://www.tsp.person.dk/MT_0291.zip)

[hartford]

@tsp

Take this result with a "grain of salt."

My video processing disk is nearly full since I began testing.
I would estimate that this test might score 10% better if done
7 days ago. I'm sorry, but I have a number of videos backlogged
videos that are due for processing and I'm pressed for time. I wish
that I could give you a more consistent result.

v0.29.1 = 523 seconds

[tsp]

hartford: It is fine with an approximate time so I can get an idea of how much faster/slower my experimental releases are (I didn't expect 0.29.0 to be so much slower because it was faster on my dual 400 mhz celeron).

I discovered that the internel cach that are inserted after each filter isn't thread safe so when I figure out how to insert my own cache instead of the buildt in when using more complex filters as iip I will release a new version that should be stable (I have a version running without the cache and executing temporalsoften stable. Usually virtualdub craches after a few frames using version 0.29.1.)

[tsp]

ok version 0.30 is up. It should be more stable(I could render 13000 frames without a crash).

[hartford]

I downloaded v0.30.

I should have time to try it this Friday or Saturday.

I'll do the test by reading the Test.avi from one drive and putting the result
to another drive that has room and to which I seldom put much data. I'll test
versions 0.10, 0.29.1, and 0.30. Writing the output to another drive should make the results more accurate.

[tsp]

hartford: thanks I now had version 0.30 process about 100.000 frames without crashing compaired to about 1-20 frames with version 0.10-0.29.1 the downside is that the speed has decreased a lot. I fear it might be slower than not using mt. I will try to incorperate the necesary changes directly in avisynth. That should speed things up.

[psme]

Will it work with Decomb/TIVTC? I use Avisynth inside FFDshow for realtime playback processing.

Using DScaler 5.006 decoder playing DVD, using TIVTC's tdm in FFDShow for 3/2 pulldown recovery, on my P4 3G Northwood, CPU usage is around 60-90%.

If this filter can reduce the CPU loading by half on 2 CPUs setup then I'm thinking the new dual core system!

Thanks in advance.

regards,

Li On

Edit: sorry, just saw others are already running Decomb with MT but seems not much performance gain...

[Leak]

Edit: sorry, just saw others are already running Decomb with MT but seems not much performance gain...
Ummm... I really wouldn't use MT with plugins that make important decisions by analyzing the whole image - if you're unlucky, you get one half of the image deinterlaced and the other weaved, or one halfs is matched forward and one matched backward, or one half is considered video and the other film etc...

[mg262]

Also found a bug? in pixiedust. When using dynamic_cast Pixiedust throws __non_rtti_object exception. And it always complaines about "First-chance exception at 0x015327e2 in virtualdubmod.exe: 0xC000001D: Illegal Instruction" in the debugview.
I believe you can't call the dust filters more than once in a script, which your filter presumably implicitly does... one workaround is to load the DLL twice and use the DLLName_FilterName syntax.

[tsp]

I believe you can't call the dust filters more than once in a script, which your filter presumably implicitly does... one workaround is to load the DLL twice and use the DLLName_FilterName syntax.
yes that explains why it produces garbage but it doesn't explain why it crash when the dll is loaded the first time when dynamic_cast is used because at that time the constructor is not called yet so there is no way dust knows it is called many times. It's really a shame such a good filter is programmed so "bad" (and the source code is not available).


Ummm... I really wouldn't use MT with plugins that make important decisions by analyzing the whole image - if you're unlucky, you get one half of the image deinterlaced and the other weaved, or one halfs is matched forward and one matched backward, or one half is considered video and the other film etc...

You're right about that. If/when I succeed in making avisynth.dll threadsafe I will try to make an option to process two or more frames in parallel. So such filters can be used.

[tsp]

new version ready that includes a custom build of avisynth 2.56 that should speed this filter up.

[psme]

Thanks for the great filter. Will try it tonight.

Will it speed up Didee's LimitedSharpen? I use it for realtime DVD playback using Avisynth in FFDShow. LimitedSharpen uses up most of my P4 3G Northwood and I can only do good flag 24fps 480p NTSC DVD. 25fps 576p PAL DVD gets stutter with 100% CPU. I'll get a dual core CPU if the filter help.

Thanks in advance.

regards,

Li On

[hartford]

Sorry for the late update; had problems.

Test was done reading from one drive and writing to another.
System is a Gigabyte GA-7DPXDWP, modified xp2400+ processors,
Tekram U160 SCSI Adaptor, Maxtor 73GB and 147GB u360 drives.

Test avi is an analog capture at 640x480.


--
Script:

lloadplugin("d:\plugins\DeComb521.dll")
#loadplugin("d:\plugins\MT_0.10.dll")
#loadplugin("d:\plugins\MT_0.29.1.dll")
loadplugin("d:\plugins\MT_0.30.dll")
loadplugin("d:\plugins\TBilateral-096.dll")

Avisource("d:\test-mt.avi").ConvertToYV12

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)


#TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)

mt("TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)")

--



No MT = 522 seconds

MT 0.10 =
An out-of-bounds memory access (access violation) occurred in module 'TBilateral-096'.

MT 0.29.1 = 410 seconds

MT 0.30 = 534 seconds

[hartford]

@tsp

Can't download v0.40

[tsp]

sorry I uploaded it compresssed as a rar file instead of zip but that should be fixed now.

hartford: thanks for the test. I did fear that version 0.30 would be slower than without mt but when using version 0.40 and the custom avisynth.dll the version should be more like version 0.10 just without the crashing :D

psme: It should be able to speed Limitsharpen up if you use version 0.40 and your cpu utilization isn't 100%. I don't know how well MT will work with hyperthreading but please try and report back.

[Shadowfax3000]

Wow, 100% CPU load. This plugin is sweet! Before it I could only get CCE to use 50% of my CPU's power with the filters I'm using. I'm using a Pentium 4 3GHz Northwood (Hyperthreading). I'm using this crazy series of filters called TimeDenoise that makes it take days to encode a movie. Before this plugin I was only able to achieve a speed of 0.046 (about 72 hours to finish). Now I have 0.053 and climbing (41 hours to finish). Will shave 31 hours off the encode, can't complain about that.

[tsp]

Shadowfax3000: that is impresive. I didn't think HT would make such a big difference. Wonder what a dualcore Pentium 4 EE would do.

psme: I did a little test on my dual celeron 400 MHz and using MT speed limitedsharpen up by ~12%(version 0.40) to 35%(version 0.41) using the default setting.

[Edit]
I just released version 0.41 it might be faster than version 0.40 at least it is on my computer but please try it and report back if it is slower or faster

[psme]

Thanks tsp! I may get a DFI 865PE-TAG board $90 (the only 865 with dual core support) and a Pentium D 820 $250 in the weekend.

I once had a BP6 with 2 Celeron 500M too! :)

regards,

Li On

[Shadowfax3000]

No improvement for hyperthreading it seems like. After 20 minutes of encoding the speed reached 0.057 for both v.0.40 and v.0.41. Through testing I've noticed CCE 2.50 is faster with a Pentium 4 than any later version. I've tried 2.67 and 2.70, neither are as fast as 2.50. People have been saying that later versions are faster with the P4, I think they might be wrong. Or it's because of the filters and plugins I'm using. Anyways, I'll be on the look-out for new versions of this plugin, so consider me your hyperthreading man.

[hartford]

Script and test avi the same as before.
Huffy Compression; read one drive, write to another
Audio included
Tests run back-to-back.


No MT: 539 seconds CPU time: 54%

MT 0.10 431 seconds CPU time: 65% to 74%

MT 0.41: 540 seconds CPU time: 54%


MT 0.41-1 with special build avisynth.dll

541 seconds CPU time: 50% to 58%



MT v0.10 isn't stable, but worked this time. It will error on loading
or it will work for the test.

If I have time, I will test on a longer capture (28000-30000 frames, or about 20 min)

Slight differences probably due to read drive being a bit full (slower read on inner tracks).

[Edit:] Clarification: Differences from June 13 test.

[tsp]

hartford: did you use avisynthTS=true when you tested MT 0.41 with the custom avisynth.dll and also if you have the time could you test version 0.40 with the custom avisynth.dll and avisynthTS=true compaired to version 0.41.

Sligtly off-topic but try take a look at my dual processor test computer:
here (http://img13.echo.cx/img13/404/inside5zq.jpg) and
here (http://img13.echo.cx/img13/7057/outside2zy.jpg)

[hartford]

hartford: did you use avisynthTS=true when you tested MT 0.41 with the custom avisynth.dll

No. I wasn't aware of that, um, toggle?



and also if you have the time could you test version 0.40 with the custom avisynth.dll and avisynthTS=true compaired to version 0.41.

I'll do that if you can tell me where to put "avisynthTS=true"




Interesting machine that you have there ;)


I need to report an error that I have using the custom DLL with this script:

vid=Avisource("01-Girl2.avi").converttorgb32()
logo=AVISource("d:\masks\mask-TVpg1-2.avi").ConvertToRGB32
vid2=vid.bilinearResize(147,110)
masklogo=mask(vid2,logo)
Layer(vid,logo,"mul",128,69,40)

I get an "undefined" access error from AviSynth. The script works ok with 2.55



Ok, I found "avisynthTS" explination in version 0.41.

Will report soonest.

[tsp]

I get an "undefined" access error from AviSynth. The script works ok with 2.55

ups forgot a & in the code. Will fix that.

I'm working on integrating MT in the sourcecode so that's why there hasn't been a new version yet.

[hartford]

Same script as above except added "AvisynthTS=true".

No MT = 519 seconds
MT v0.40a = 422 seconds

[tsp]

and MT 0.41 with AvisynthTS=true?

[hartford]

No Mt 519 seconds

MT v0.41 415 seconds

[hartford]

@TSP

Is this near release?

[tsp]

I think the next version will be ready around Saturday evening/night(GMT)

[hartford]

Got my "test booties" on :)

[tsp]

I will have to postprone the next release until I have fixed the serious bugs.

[tsp]

new version ready. It just contains the avisynth.dll because it's the only place the changes are. It is based on the current avisynth 2.56 CVS plus some custom changes to allow multithreading. The way it works is that when a frame is requested the following frames are generated by the idle cpu(s) so that they are ready when needed.

usage:
SetMTmode(int mode,int threads)
place this at the first line in the avs file to enable temporal multithreading.
There are 5 modes 1 to 5
Mode 1 is the fastest but only works with a few filter.
Mode 2 should work with most filters but uses more memory
Mode 3 should work with some of the filters that doesn't work with mode 2 but is slower
Mode 4 is a combination of mode 2 and 3 and should work with even more filter but is both slower and uses more memory
Mode 5 is slowest but should work with all filters that doesn't requere linear frameserving (that is the frames come in order(frame 0,1,2 ... last)

it is posible to change mode inside the script by calling SetMTMode again. Default 2

threads.
Number of threads to use. Set to 0 to set it to the number of processors available. It is not posible to change the number of threads other than in the first SetMTMode. Default 0

example script:

SetMTMode(4)
import("limitedsharpen.avs")
function test(clip c)
{
c.blur(1)
subtitle("12")
}

src=AVIsource("e:\sim.avi")
SetMTMode(2)
src=src.assumefps(100).converttoyv12().trim(0,10000)#.separatefields().selecteven#PixieDust
SetMTMode(1)
src.test()
SetMTMode(2)
LimitedSharpen()


Download it here (http://www.tsp.person.dk/avisynth256MT.zip)

It is still very beta so please report all the bugs you find

[hartford]

new version ready. It just contains the avisynth.dll because it's the only place the changes are.



Pardon me, but, you should make it UNMISTAKABLE PLAIN that this is a NEW VERSION of AVISYNTH.

It should be SHOUTED that this is YOUR version of Avisynth.

Just my opinion.

I'll try it when I can.

[tsp]

well if you don't use SetMTMode it functions like the current CVS version but it does of course includes some changes made by me. I'm sorry if the post was a little unclear but I had to catch a train because I have a lot of out town work this summer and I wanted to release this version before I left (for the impatient people like myself).
I updated the file in the last post so that SetMTMode and GetMTMode accepts default values.

[hartford]

I wish to apologize for my previous post. I'm just a jerk. Your efforts are appreciated by me, but, again, at times, what I type in reply is just stupid.

Avisynth 2.5.7 results (same script and avi as previously):

451 seconds

[tsp]

eh did you script look like this:

SetMTMode(2)
lloadplugin("d:\plugins\DeComb521.dll")
#loadplugin("d:\plugins\MT_0.41.dll")
loadplugin("d:\plugins\TBilateral-096.dll")

Avisource("d:\test-mt.avi").ConvertToYV12

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)


TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)

#mt("TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)")

or like this

loadplugin("d:\plugins\DeComb521.dll")
loadplugin("d:\plugins\MT_0.41.dll")
loadplugin("d:\plugins\TBilateral-096.dll")

Avisource("d:\test-mt.avi").ConvertToYV12

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)


#TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)

mt("TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)")

[tsp]

I updated the above file. It now uses the latest cache code from the cvs.

[hartford]

Like this:

loadplugin("d:\plugins\DeComb521.dll")
loadplugin("d:\plugins\MT_0.41.dll")
loadplugin("d:\plugins\TBilateral-096.dll")

Avisource("d:\test-mt.avi").ConvertToYV12

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)

mt("TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)",avisynthTS=true)



[edit] I'll try new and report soonest.

[hartford]

avisynth256MT

Script as posted at 01:46, 18Jul05. :)

633 seconds.

[tsp]

you mean 19Jul05 right :)

Strange it just gets slower and slower. Is the memory usage steady climping while the script is played?
Could you try this script(with the latest avisynth.dll) and report back the speed:

SetMTMode(2)

loadplugin("d:\plugins\DeComb521.dll")
loadplugin("d:\plugins\TBilateral-096.dll")

Avisource("d:\test-mt.avi").ConvertToYV12

Telecide(order=1,Post=0,Guide=1)
Decimate(Cycle=5,Mode=0,Quality=3)

TBilateral(diameterL=3,diameterC=3,sDevL=2.0,sDevC=2.0,iDevL=6.0,iDevC=7.0,d2=true,gui=false)

[hartford]

you mean 19Jul05

Yes.

Is the memory usage steady climping while the script is played?

Don't know. Didn't occur to me to check.

Could you try this script(with the latest avisynth.dll) and report back the speed

Ok. You are refering to AviSynth.MT256, correct?

[hartford]

A bit sheepish about this:

My motherboard doesn't always boot to 2 cpus. Seems that the last test was done with only one. Sorry about that.

This test is using both cpus and your latest "code" and Avisynth-MT256: IT CRASHED.

[edit] Just tried same a second time and it completed (no reboot).

373 Seconds. AviSyth-MT256.


Why the crash the first time? Beats me.

[edit2] Did not see any any memory usage climbing.

[edit3] To be clear: previous report of some "6xx" seconds was due to the use of one CPU. This test was a reboot to 2 CPU's and then I did your "code" using VirtualDub which ended in the "crash." I then restarted VirtualDub and
loaded your "code.avs" from which I got the above "373 Seconds."

Is that clear?

[tsp]

yes I got it. So it is faster to use the SetMTmode than using MT when both cpu are recognised but the first test did crash. It's proberly because there are still some bugs left but how many frames was processed before the crash happend and was there any error message?

[hartford]

No frames were processed. VirtualDub gave the error upon loading the script (sorry, don't recall what it was).

[krieger2005]

Hi,

i tried also your "Multi-Threaded" avisynth. However... i have some strange behaivor which i want to report. I used the following Script and the dll-File from above:
SetMTMode(2)
mpeg2source("E:\VTS\Test\menue\d2v\sw4.d2v")
LeakKernelDeInt(order=1, threshold=5)
ConvertToYUY2
PixieDust(4)
VagueDenoiser(threshold=2,method=1,nsteps=6,chromaT=0.8, wiener=true, wratio=2)


First i startet the Script in Media Player Classic. My CPU-Usage was about 98% and the movie was shown fine (i could see some frames). Then i closed Media Player but my CPU-Usage was still 48%. I took a look at the processes... The 48% CPU-Usage was caused by Media Player. I got no window of the player but there was a process, which uses 48% of the CPU.
After this i started CCE-Encoder and tried to encode this little Part of the Movie. I got Speed about 1.4 :scared: . This is simply to much for this script. I stopped the thing and looked at the result.. The result was nonsense (black screen with wired red things at the bottom). Then i looked closely at the data CCE shown to me:
CCE shows me, that the clip was 244 Frames long (10 Seconds), real length was 799 frames.

I want just report this thing. Have a Prescott 3.0 GHz.

[tsp]

thanks forthe report. It might be caused by Pixiedust. That filter does not behave very well when more than one instance is used(and mode=2 creates two instances). Try without pixiedust or with SerMTMode(3) and if that works try this version:

SetMTMode(2)
mpeg2source("E:\VTS\Test\menue\d2v\sw4.d2v")
LeakKernelDeInt(order=1, threshold=5)
ConvertToYUY2
SetMTMode(3)
PixieDust(4)
SetMTMode(2)
VagueDenoiser(threshold=2,method=1,nsteps=6,chromaT=0.8, wiener=true, wratio=2)

[hartford]

A small update:

I've tried to reporduce the error that I had before at least 10 times. It must be
something weird because it hasn't happened since the first time.

I don't know what to say at this point.

Is it my particular motherboard, or cards, or operating system, or something else?

I don't know.

I'd not like you to waste your time trying to track "gremlins."

[krieger2005]

Hallo and thanks for your advice. I tried to start "PixieDust" in MTMode(4) and 3 but it does not help... However, MTMode(5) work with PixieDust.

But i see a different strange thing:
I started to convert a movie with the avisynth DLL of you from above. I wonder when i see, that VirtualDubMod processed 81000 Frames of 107500 Frames but show a process of 37%? A simple calc turns it out: 81000/(2*107500) is about 37%. But why "2*X"?
I don't know what the problem is but i see such a thing the first time. Maybe it is also not your dll. I will search for the problem. But maybe someone can reproduce this thing.

But still the same problem when starting an avs-Script with "Media Player Classic". After close of Media Player a player is still available as process (but without a window). Maybe this is something similar, why "VD-Mod" show me only 37%...?

[tsp]

I don't think the to things are connected. I will investigate it when I get back from my vaccation in 14 days. Also try this script and see if it works and if it is faster than your current script:


SetMTMode(2)
mpeg2source("E:\VTS\Test\menue\d2v\sw4.d2v")
LeakKernelDeInt(order=1, threshold=5)
c=ConvertToYUY2
SetMTMode(5)
c.Distributor()
PixieDust(4)
SetMTMode(2)
VagueDenoiser(threshold=2,method=1,nsteps=6,chromaT=0.8, wiener=true, wratio=2)

[Revgen]

This is a great plugin! :D

I just tried it out using a 40 sec 3 Stooges test clip and here's the results using X264 2-pass with MeGUI 0.2.2.2

X264 - 1 thread - No MT: 7.23FPS Average

X264 - 2 threads - No MT: 11.92FPS Average. A 65% improvement!

X264 - 2 threads - MT On: 12.68FPS. A 75% improvement over a single thread!

Here's my script in case anybody is interested:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
SetMTmode(1,2)
Trim(4482,5454)
Greyscale()
RemoveGrain(mode=7, modeU= -1).RemoveDirt(grey= true)
Greyscale()

EDIT:
I use an Athlon X2 4600+ in case anybody is curious.

EDIT2:

I just noticed that using SetMTmode(1,2) gives very occasional graphical glitches when RemoveGrain is being used. So I've now retested it using SetMTmode(2,2).

X264 - 2 Threads - MT On: 12.59 FPS. Still 74% faster than a single thread. Not much peformance loss at all.

[tsp]

looks good. Just curious. How fast is it with X264 - 1 thread and MT on?

[Revgen]

looks good. Just curious. How fast is it with X264 - 1 thread and MT on?

Well since you asked...

7.6 FPS. About a 5% increase over 1 thread W/ No MT.

It seems to me like MT works better when it's paired with a multithreading encoder.

[tsp]

I somewhat expected that the increase would be greater. But then again if avisynth uses 1 core and the X264 encoder uses the other when there are not much cpupower left for the second thread in avisynthMT. Strange that it's faster with MT compaired to no MT. If you just encode to MJPEG or HuffYUV how much is the increase then?

[Edit]
Hmm maybe the avisynth thread is using lets say only 20% of the first core because it's the encoder that's slowing things down. X264 uses 100% of the second core if running only singlethreaded and if running with 2 threads the second thread in X264 uses the rest of the first core. Although this doesn't explain why enabling multithreading in avisynth increases the performance.

[Revgen]

I've now decided to try different codecs and filters to get a better idea of what MT can do.


My Setup:

MJPEG= FFDshow MJPEG encoder. Encoded at 1-pass 1500kbps constant bitrate.

NOTE: This MJPEG codec also has an option for 2 threads, but when I tried it gave me an error message. Hopefully it gets fixed in the future.

Huffyuv=FFDshow Huffyuv encoder in YV12 colorspace, Median predictor, and NO Adaptive Huffman Tables (for more speed).

X264= 1490Kbps 2-pass using Me-GUI.

Xvid= 1490Kbps 2-pass.



Avisynth Scripts:

RemoveGrain.RemoveDirt Filter:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
RemoveGrain(mode=7,modeU= -1).RemoveDirt(grey=true)
Greyscale()

FFT3D Filter:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)

Limited Sharpen Function:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
LimitedSharpen()


Results:

MJPEG (Fast Codec) W/ RemoveGrain.RemoveDirt Filter (Fast Filter)

97.3 FPS Same Performance w/ MT On and Off.


MJPEG (Fast Codec) W/FFT3D filter (Slow filter)

23.73FPS MT OFF

24.95FPS MT ON

About an 5% increase in performance.


MJPEG (Fast Codec) W/LimitedSharpen Function (Very Slow Function)

9.45FPS MT OFF

16.49FPS MT ON

About a 74% increase in performance.


HuffYUV (Fast Codec) W/RemoveGrain.RemoveDirt Filter (Fast Filter)

About 97.3 FPS (again). Also same perf w/ MT ON and OFF.


HuffYUV (Fast Codec) W/FFT3D Filter (Slow filter)

23.73FPS w/MT OFF: Same as MJPEG

24.95FPS w/MT ON: Same as MJPEG

About a 5% peformance increase.


HuffYUV (Fast Codec) W/ LimitedSharpen Function (Very Slow function)

9.36 FPS MT OFF

16.22 FPS MT ON

About a 73 % peformance increase.



X264(Slow Codec) W/ FFT3D Filter(Slow Filter)

5.83FPS 1 Thread MT OFF

7.44FPS 1 Thread MT ON: About a 28% increase.

8.51FPS 2 Threads MT OFF: About a 46% increase.

9.64FPS 2 Threads MT ON: About a 65% increase. This is a very different result than the RemoveGrain filter. It seems that MT is less effective with multithreading ON this time around. It seems to excel in single threading more with this filter.


X264(Slow Codec) W/ LimitedSharpen Function (Very Slow Function)

4.19 FPS 1 Thread MT OFF

6.85 FPS 1 Thread MT ON: About a 63% increase over a single thread.

5.45 FPS 2 Threads MT OFF: Only a 30% increase in performance. Very Interesting. It seems that LimitedSharpen affects the encoding speed of X264 more than the other filters. It could be because sharpening makes it harder for the encoder to do it's job while denoisers make it easier.

7.96 FPS 2 Threads MT ON: About a 90% increase over a single thread. Very impressive! It looks like MT is most responsible for this result.



Xvid(Medium-Slow codec) W/RemoveGrain.RemoveDirt Filter(Fast Filter)

8.69FPS MT OFF

8.69FPS MT ON: No difference in performance.


Xvid (Medium-Slow codec) W/ FFT3D Filter (Slow Filter)

8.92FPS MT OFF:

8.46FPS MT ON: 5% slower. Very strange result. Not sure why this is occuring.


Xvid(Medium-Slow codec) W/ Limited Sharpen Function (Very Slow Function)

8.32FPS MT OFF

7.91FPS MT ON: About 5% slower again. It looks like MT and Xvid don't work well together for some reason.




I 've now decided to try Xvid without any filters using the script below:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)

Results:

Xvid recorded results of 8.76FPS both with MT ON and OFF.



Summary:

MT seems to help performance in most situations, doesn't help in some, and in the case of Xvid, can even hurt performance.

I'm not sure why it doesn't work well with Xvid.

Which has gotten me thinking...What codec was Hartford using when he reported that MT was slower than NO MT?

If it was Xvid then therein may be the problem.

I for the most part, use X264 for encoding. My Dual-Core CPU makes it worth my while compared to Xvid. But, Xvid is still a popular codec and this might be a problem for others.

Other than the Xvid issue, MT in general seems to work very effectively when combined with filters that take advantage of the extra power.

I hope this helps.

[Boulder]

VirtualDub supports multithreading natively so that might explain the XviD thing.

[Revgen]

VirtualDub supports multithreading natively so that might explain the XviD thing.

Which Version?

I used VirtualDubMod 1.5.10.

How would it adversely affect Xvid's performance?

[tsp]

all versions of virtualdub does that. 1 thread runs the avs-script the other tnread does the encoding. version 1.7 of virtualdub will support SMP more than that.
Will comment on the result later.
[edit]
well that doesn't make a lot of sense. First I would have expected that fft3dfilter combined with a less demanding codec would give more than 5% speed increase. But then again I think I know why it is so slow. It might be because fft3dfilter has an internal cache of the fast fourier transformed frames (I used that with fft3dGPU so I think Fizick does the same) and when using MT mode 2 each thread has a seperate instance of fft3dfilter that doesn't share the internal cache so that it is because of increased internal cache misses that causes the slowdown. If this is true this might be faster(if it works that is):

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
SetMTmode(3)
fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)

I don't know what happends with the xvid stuff. It might be because the 2 avisynth threads somehow slows the single xvid threads down. Try increasing the virtualdub priority. Also try increasing the stream data pipelining under options->performance options in virtualdub and see if it makes any difference. Another good way to look at how well avisynth and virtual dub interacts is to look at the cpu utilization.
One last thing. Try replacing pixiedust in limitedsharpen because that filter interacts very badly with multithreading in avisynth

[ChrisBensch]

First off, excellent work. Now that dual-cpu/dual-core are the norm for new PCs, it's nice to see more plugins/people/apps taking advantage. I've just come across this thread and am very interested. Most of my encodes are HDTV -> DivX. My sources are always clean and don't really need any processing. Here is my normal script for film source

MPEG2Source("Harry Potter and the Prisoner of Azkaban-HDTV.d2v")

Crop(2,0,-4,-6)

LanczosResize(1280,720)

My questions are: Given the script above, do you seen any possibility of speeding things up?

how do I tell MT that I'm using the special Avisynth for MT? I read the doc where it says to set the value to true, but that was using mt("filter here"). How can I tell it to work using the scripts I've seen in this thread?
For ex:

SetMTmode(2,2)
MPEG2Source("Harry Potter and the Prisoner of Azkaban-HDTV.d2v")
Crop(2,0,-4,-6)
LanczosResize(1280,720)

where woudl I put a notifier to MT that I'm using it's special avisynth.dll?

[Revgen]

The SetMTmode(3) setting causes glitches in my video, and it also makes the Xvid video encode even slower at 7.05FPS. These glitches also apear when playing back the .AVS file, so it's not Xvid's fault.

So I decided to switch back to SetMTmode(2) and keep the the Vdub settings you recommended.

The result and FPS turned out to be the same as the previous FFT3D post.


Also, according to Didee (http://forum.doom9.org/showthread.php?p=708243#post708243) Limited Sharpen only uses Masktools and the Warpsharp package. PixieDust shouldn't be the problem.


On the other hand I found a bug in the filter. It's similar to the WMP 6.4 memory leak bug that I found in the FFTGPU filter earlier. Every time I play an .AVS file with WMP when MT is used WMP 6.4 stays resident in memory and doesn't shut down automatically. I have to go into the taskbar to shut it down instead. This bug seems to only affect WMP though. Encoding apps close down properly.


Also, just a reminder, DON'T GO CRAZY! ;)

MT is still an excellent filter that works well with most codecs. You should be proud! :D

[Revgen]

My questions are: Given the script above, do you seen any possibility of speeding things up?


Try it and see what you get. Make sure to read about tsp's special version of avisynth (http://forum.doom9.org/showthread.php?p=682790#post682790) and download the file at the bottom of the post.

Replace the avisynth.dll in the C:\Windows\Sytem32\ directory with the new one. Make sure you rename the old .dll to something like "avisynth.dll.old" and keep it in case you want to use that one later.

Based on your script I believe you should be able to use the SetMTmode(2,2) script at the bottom of your page just fine.

[ChrisBensch]

I did exactly that, I get about a 1fps increase. Not bad I guess. I'm doing full 1920x1080 -> 1280x720 DivX 6 encodes. I get about 4.75fps using AVS2AVI, I'll see what VDub gets me.

[Revgen]

I've got some good news.

I've decided to try encoding my clip using AVS2AVI instead of Vdub to see if it changes anything:

Results:

HuffYUV W/FFT3D Filter:

23.73FPS MT OFF

26.29FPS MT ON

Interesting. The MT OFF result is the same as the Vdub result. Yet MT ON results in an 11% increase in performance. More than double the Vdub result of 5%.

I've go to go to sleep now, I'll test the other codecs and filters tommorow.

[tsp]

Revgen: krieger2005 also noted the problem with windows media player classic. I couldn't reproduce the error on my single processor computer but I will try it on my dual processor later. Shouldn't be that hard to fix. Proberly something like the fft3dGPU error. Also you could try using my original MT.dll with fft3dfilter and my version of avisynth.dll like this:

mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
MT("fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)",2,8,false,true)

also I mixed limitedsharpen(that doesn't use pixiedust) and iip (that uses pixiedust) up. Both not exactly the fastest scripts in avisynth. Maybe virtualdub 1.6 is faster than virtualdubmod?
Oh and thanks for the kind words

[Revgen]

@tsp

First of all lets start off with the bad news first this time. Xvid's results are still the same with all 3 apps despite whether I use the older MT filter or the newer SetMTmode setting.

Now for the good news using your suggested script:

HuffYUV W/FFT3D Filter--Encoded with AVS2AVI

33.55FPS. Thats about a 41% increase in performance over the previous 23.73FPS result W/MT OFF. Using this older filter seems to provide better benefits.

HuffYUV W/FFT3D Filter--Encoded with VirtualDubMod 1.5.10

31.39FPS. Better then with the SetMTmode setting, but still slower than AVS2AVI. This time it's slower than AVS2AVI by 7%.

HuffYUV W/FFT3D Filter--Encoded with VirtualDub 1.6.10

30.4FPS. This version of Vdub is even slower than Vdubmod(by about 3%) and about 10% slower than AVS2AVI.

MJPEG W/FFT3D Filter

All 3 encoding apps recorded the same results as HuffYUV.

I hope this helps.

[tsp]

I wonder what's causing the problem with xvid?
Good to see that the old version is working well with fft3dfilter. The old MT and the avisynth.dll version works in two different ways. MT.dll lets each thread run on a small piece of a frame and combines the result while avisynth.dll SetMTMode lets each thread work on a different frame. the MT.dll aproach is good when the filter doesn't relies on information from the whole frame so it's good when used with filters like fft3dfilter and simple sharpen and blur filters while SetMTmode is good with filters like HDRAGC and some of the deinterlacers that needs information from the entire frame.

[Revgen]

...SetMTmode is good with filters like HDRAGC and some of the deinterlacers that needs information from the entire frame.

Do you think that Masktools benefits from this mode? Maybe thats why LimitedSharpen works so well in this mode. There are many Masktools functions out there that may also reap benefits.

EDIT:

Also, is there a way to turn off SetMTmode() switch once you turn it on? I'd like to be able to switch between the old and new MT in the same script so that I can use certain filters without conflicts.

IE:

SetMTmode(2,2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
SetMTmode(OFF)
MT("fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)",2,8,false,true)
SetMTmode(2,2)
LimitedSharpen()

If not, then it's no big deal. I can wait. :)

[tsp]

Do you think that Masktools benefits from this mode? Maybe thats why LimitedSharpen works so well in this mode. There are many Masktools functions out there that may also reap benefits.

There are an extra overhead when using mt.dll compaired to SetMTMode() so I think they would benefit the most from SetMTMode because they don't have any internal cache that I'm aware off. But nearly all the functions in MaskTools onkly needs information from the pixels that surround the current pixel that are being processed so MT.dll should work fine with them (compaired to a deinterlace where only half the frame would be deinterlaced if only 1 thread thought the frame was interlaced).

Also, is there a way to turn off SetMTmode() switch once you turn it on? I'd like to be able to switch between the old and new MT in the same script so that I can use certain filters without conflicts.

SetMTmode(5) turns the multithreading off. The problem with it currently is that it also turns it of for all filters before SetMTMode(5) although you can enable it by inserting Distributor() just after SetMTMode(5) something like this:
[/code]
SetMTmode(2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
g=Greyscale()
SetMTmode(5)
g.Distributor()
MT("fft3dfilter(sigma=4,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,ratio=2)",2,8,false,true)
SetMTmode(2)
LimitedSharpen()
[/code]
It is one of the things I still need to add to the SetMTMode().

[Revgen]

The script works well.

I got 7.05FPS with MT OFF and 11.9FPS with MT ON using HuffYUV.

About a 69% increase.


Also,

I hope Shodan eventually puts your code into the upcoming version of AVS 2.6.

He told you "I'll do it ASAP" in this (http://forum.doom9.org/showthread.php?p=707678#post707678) thread, but didn't give a clear response confirming whether he did or not.

You might want to try reminding him again. ;)

[tsp]

I will but I think it will be best to fix the WMP classic and add a better working version of SetMTMode(5) also I need to add support for the ConditionalFilter/FrameEvaluate/ScriptClip/WriteFile filters.

[Revgen]

Cool. You can PM me if you need somebody to test it once it's done.

[mg262]

@tsp,

I haven't tried this filter because I only have a single-threaded single processor... but where possible I would like to be building filters in a way that is amenable to parallel execution. I can guess at the basics -- no global variables (or equivalent inside the filter), bear in mind that two frames could be requested simultaneously from the same filter, etc. ... but from the number of modes present, it seems that there are subtler issues.

If you ever have a moment, could you briefly document e.g. what a filter needs to do in order to be compatible with a particular mode?

Thanks,
@mg262

[Revgen]

@mg262

tsp gave a more technical explanation of how it works in this thread. (http://forum.doom9.org/showthread.php?t=95596)

Lots of programming blabla that I can't understand, but you might be able to. ;)

[tsp]

revgen: I can't reproduce the problem with windows mediaplayer classic on my dual celeron 400 MHz. Maybe it's to slow? It is the mplayer2.exe that are windows mediaplayer classic right?

mg262: sure. The reason there are so many modes is to allow as many as posible of the existing filters to work. But here are the requerement:

Mode 1: all access to class variables, global variables and static variables most be threadsafe by using appropriate locking(Enter/LeaveCriticalSection etc, no locking need for readonly variables) because more than 1 thread may access a class instance at a time.

Mode 2: access to class variable doesn't have to be threadsafe because there is only 1 instance of the class per thread. All global/static variable access must be threadsafe. Because each class instance only process every other frame internal caches(that is a cache inside the filter) wouldn't work well. I still need to figure out a solution to this.

Mode 3: Only 1 thread is allowed to execute code from the filter at the same time. When child->GetFrame is called another thread can enter the filter and execute code. That means that class variables/global variables/static variables shouldn't be assigned to any values before after the last child->GetFrame has been called. Instead local function variables should be used like this:

PVideoFrame __stdcall AdjustFocusV::GetFrame(int n, IScriptEnvironment* env)
{
PVideoFrame frame = child->GetFrame(n, env);//Assigned to a local variable so this will work in mode 3
env->MakeWritable(&frame);
if (!line)
line = new uc[frame->GetRowSize()+32];
uc* linea = (uc*)(((int)line+15) & -16); // Align 16
uc* buf = frame->GetWritePtr();
int pitch = frame->GetPitch();
int row_size = vi.RowSize();
int height = vi.height;
memcpy(linea, buf, row_size); // First row - map centre as upper
if ((pitch >= ((row_size+7) & -8)) && (env->GetCPUFlags() & CPUF_MMX)) {
AFV_MMX(linea, buf, height, pitch, row_size, amount);
} else {
AFV_C(linea, buf, height, pitch, row_size, amount);
}
return frame;
}

But not like this:

PVideoFrame TemporalSoften::GetFrame(int n, IScriptEnvironment* env)
{
__int64 i64_thresholds = 0x1000010000100001i64;
int radius = (kernel-1) / 2 ;
int c=0;

// Just skip if silly settings

if ((!luma_threshold) && (!chroma_threshold) || (!radius))
return child->GetFrame(n,env);


for (int p=0;p<16;p++)
planeDisabled[p]=false;


for (p=n-radius;p<=n+radius;p++) {
frames[p+radius-n] = child->GetFrame(min(vi.num_frames-1,max(p,0)), env);
//GetFrame assigned to class variable frames. This wouldn't work with Mode 3
//because the next thread that enters this getframe will overwrite the result
// from the last thread
}

//do stuff
}

but when using mode 3 there is no need for threadsafe access to class variables. And because there is only 1 instance of the class that process all frames Internal caches will work much better. The bad thing is only 1 thread can execute the filter at a time so if it's the only slow filter in the script the speed increase wouldn't be that big.

Mode 4: a combination of mode 2 and 3 so it's okay to assign class variables before the last child->getframe has been called because there is a class instance per thread but the problem with internal cache is the same as mode 2

Mode 5: No restrictions.

Mode 6: A slightly modified version of mode 5 that might be a little faster.

[mg262]

Thank you very much!

[Revgen]

I did a little more testing on the WMP bug.

It seems like the bug only appears under these circumstances:

A) It only happens when using MT with either LimitedSharpen or the RemoveGrain.RemoveDirt filter. FFT3DFilter doesn't cause these problems with MT.

B)It only occurs when using SetMTmode. Using the old MT.dll causes no issues with any of the 3 filters.

C) It only occurs when I exit WMP without pressing the "Stop" button.

As long as I press the "Stop" button then exit everything is fine.

I guess I'm just more impatient than most people :D .


This bug seems to be different than the FFT3DGPU one. But fortunately it can be avoided for now.

EDIT

Yes. Mplayer2.exe is the WMP program in the taskbar.

[Aquilonious]

Where can I find PixieDust and how would I use it in DVD-RB Pro?

I'm not at all good at script writing.

[tsp]

Aquilonious: dust (http://www.avisynth.org/warpenterprises/#dust)

All: New version ready. No need for distibutor with SetMTMode(5) and although I couldn't reproduce the WMP bug it might have been fixed with this version. Get it here (www.avisynth.org/tsp/avisynth256MT2.zip). If this version works well I will start integrating it in avisynth 2.60

[Aquilonious]

Aquilonious: dust (http://www.avisynth.org/warpenterprises/#dust)

All: New version ready. No need for distibutor with SetMTMode(5) and although I couldn't reproduce the WMP bug it might have been fixed with this version. Get it here (www.avisynth.org/tsp/avisynth256MT2.zip). If this version works well I will start integrating it in avisynth 2.60

Thanks for the links, tsp. jdobbs mentioned that Pixiedust is part of the dust collection. I'm using AviSynth 2.5 and DVD-RB Pro 1.00RC6 along with CCE SP. Since I have an Athlon XP 3200 I can't utilize multithreading.

For DVD-RB I typically use "Pixiedust(5)" in the AVS Filter Editor box. I range the value (3-8) according to the quality of the source. I first try small numbers then work my way up.

I backed up an old DVD I had, The Last House on the Left. It had a LOT of grain--more than I've ever seen in any DVD. I tried numerous filters on it but the combination that worked best for me was an initial application of Undot. Then using the output from that I applied RemoveGrain & Msharpen. There's still a bit of grain, but it's quite an improvement over the source.

I have a few questions:

1. I'm using DustV5.dll. This is the correct version for AviSynth 2.5, right?
2. FFT3D & Dust seem to do the same thing. What are the basic differences between these two filters?
3. What is your favorite filter for grain removal?

It took me 8 hours to find the right combination of filters. Yeah, I know it's a lot of time but I learned what works and what doesn't. Filtering seems to be a mass experiment. :D

[Revgen]

@Aquilonious

Also make sure to get the the LoadPluginEX.dll included in the warpsharp package. This .dll needs to be loaded before the Dust filter in order for Dust to work with the 2.5x versions of Avisynth.

New version ready. No need for distibutor with SetMTMode(5)...

It works. I used the script from the tests I did on 9-11-05 without the g=greysclae and g.Distributor lines and the performance was the same.

...and although I couldn't reproduce the WMP bug it might have been fixed with this version.

The WMP issue no longer appears with the RemoveGrain.RemoveDirt filter, but still appears with the Limited Sharpen filter.

Also, did you mention that you have a Dual CPU Celeron setup? I have an AMD X2 Dual-Core setup.

While the AVS file is playing task manager reports a 100% utilization.
Yet whenever I exit WMP and the issue arrises, task manager reports that 50% of my CPU is being utilized. This means that only one of my cores is being used while the other one isn't.

Could this be an issue with my AMD cores rather than MT?

The only other persons in these forums that I know who have dual cores and might be able to confirm my problem are Doom9 and Easy2BCheesy. Doom9 has the same 4600+ CPU as I have, but last I heard he was having issues with his MB. I'm not sure what CPU Easy2BCheesy has, he didn't mention if his was AMD or Intel.

[tsp]

Revgen: It's not the athlon X2's fault. Hartford has the same problem with a dual athlon MP (I think it was). I will create a special debug version that you can try that will create a log file.

[Revgen]

I'll PM you my email.

[Revgen]

@tsp

I just sent you the log. It should be in your inbox. Hopefully you got it, because sometimes my email doesn't work right.

[Revgen]

Sent you an email. The fix you made works.:)

[tsp]

thanks. I added the multithtreading code to the avisynth 2.60 sourcecode so it should be in the next version (hopefull). A version of the avisynth 2.60 (beware that it's very very early version so don't blame me if your house burn down after you download it) binary is available here (http://www.avisynth.org/tsp/avisynth260alfa.zip) and the latest avisynth 256MT with the above bugfix is available here (http://www.avisynth.org/tsp/avisynth256MT3.zip).
I also added a mode 6 that you can chose instead of mode 5 if the cpu utilization isn't 100%

[hartford]

@tsp

<quote>latest avisynth 256MT with the above bugfix is available here.</quote>

Thanks. (still following this thread).

[Aquilonious]

[QUOTE=Revgen]@Aquilonious

Also make sure to get the the LoadPluginEX.dll included in the warpsharp package. This .dll needs to be loaded before the Dust filter in order for Dust to work with the 2.5x versions of Avisynth.

I downloaded the LoadPluginEx.dll file and it now resides in my AviSynth plugins folder.

How do I load this pluging in DVD-RB Pro? I'm very poor at scripting. Could you give me an example in what exactly I need to enter into the AVS Filter Editor box? I'm still relatively new using filters.

[Revgen]

[QUOTE=Revgen]@Aquilonious

Also make sure to get the the LoadPluginEX.dll included in the warpsharp package. This .dll needs to be loaded before the Dust filter in order for Dust to work with the 2.5x versions of Avisynth.

I downloaded the LoadPluginEx.dll file and it now resides in my AviSynth plugins folder.

How do I load this pluging in DVD-RB Pro? I'm very poor at scripting. Could you give me an example in what exactly I need to enter into the AVS Filter Editor box? I'm still relatively new using filters.

Some people have reported that it's better to load LoadPluginEX.dll and DustV5.dll externaly. Try entering this if you want to use Dust:

LoadPlugin("MyDrive:\MyFolder\LoadPluginEX.dll")
LoadPlugin("MyDrive:\MyFolder\DustV5.dll")

mpeg2source("MyDrive:\MyFolder\YourMovie.d2v")

ConvertToYUY2() #PixieDust can only work in YUY2 and RGB colorspace.
PixieDust(5) #The default mode is mode 5. You can use higher modes for better compression, but with less preserved detail.
ConvertToYV12() #DVD's typically utilize YV12 colorspace and encode faster when it's used.

EDIT
It's best not to use SetMTmode with Dust since they both don't work well together.

[Aquilonious]

[QUOTE=Aquilonious]

Some people have reported that it's better to load LoadPluginEX.dll and DustV5.dll externaly. Try entering this if you want to use Dust:

LoadPlugin("MyDrive:\MyFolder\LoadPluginEX.dll")
LoadPlugin("MyDrive:\MyFolder\DustV5.dll")

mpeg2source("MyDrive:\MyFolder\YourMovie.d2v")

ConvertToYUY2() #PixieDust can only work in YUY2 and RGB colorspace.
PixieDust(5) #The default mode is mode 5. You can use higher modes for better compression, but with less preserved detail.
ConvertToYV12() #DVD's typically utilize YV12 colorspace and encode faster when it's used.

EDIT
It's best not to use SetMTmode with Dust since they both don't work well together.


My source is not an MPEG file but the Starship Troopers 2 DVD I own that I'm backing up. I have decrypted the DVD (using DVDFab Decrypter) and the files now reside on my drive. I'm using DVD-RB Pro 1.00 RC6, the latest version.

I have tried the script you provided and every several variations of it, but just can't get Pixiedust to run. I either get an error in Rebuilder's Preview/Edit mode (in red letters) or it crashes Rebuilder altogether.

So I looked for other alternatives. I tried STMedianFilter + MSharpen with the following script and it worked fine:

Loadplugin ("C:\Program Files\AviSynth 2.5\plugins\STMedianFilter.dll")
STMedianFilter(8,10,4,7)
Loadplugin ("C:\Program Files\AviSynth 2.5\plugins\MSharpen.dll")
MSharpen()

The results are ok but there's a bit too much blur for my taste. The settings for STMedian are lower than the default values. While MSharpen helps, using anything stronger than the default values begins to introduce artifacts into the video.

I really want to try PixieDust because I've seen the excellent results which it can produce. I want to backup my Aliens DVD, one of my fav movies, but I have to be quite careful as I don't want to alter the original detail, just degrain it a bit.

If memory serves me correctly, James Cameron mentioned that he wished he had utilized different film for some of the scenes which turned out particularly grainy, like when the marines had just entered the compound overrun by the aliens. (I think he used 16mm telcine for some of the anamorphic/wide-angle shooting because of the longer focal length of the lenses).

[Boulder]

By the way, you are pretty much off topic. You really should open a new thread :)

[Revgen]

My source is not an MPEG file but the Starship Troopers 2 DVD I own..

DVD's are MPEG2 video's.


I have decrypted the DVD (using DVDFab Decrypter) and the files now reside on my drive. I'm using DVD-RB Pro 1.00 RC6, the latest version.


Are these .vob files? If so then you need to use DGIndex and create a .d2v file so they can be read in avisynth.


I have tried the script you provided and every several variations of it, but just can't get Pixiedust to run. I either get an error in Rebuilder's Preview/Edit mode (in red letters) or it crashes Rebuilder altogether.

PixieDust can be very finicky. Try talking to JDobbs (DVD RB author) about the problem.


I really want to try PixieDust because I've seen the excellent results which it can produce. I want to backup my Aliens DVD, one of my fav movies, but I have to be quite careful as I don't want to alter the original detail, just degrain it a bit.



Another filter you can try instead of PixieDust that I prefer using is FFT3Dfilter. IMHO it's just as good if not better than PixieDust at retaining detail and preseving noise. The only disadvantage it has is blocky artifacting at high sigma values.

These values offer about the same compression as PixieDust when used with MSU Lossless Codec.

fft3dfilter(sigma=5,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)
fft3dfilter(sigma=5,beta=1,plane=1,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)
fft3dfilter(sigma=5,beta=1,plane=2,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)

If your going to use MT with this filter I recommend using the old MT.dll intstead of SetMT. It works faster. Just put the MT.dll in your Plugin folder and make sure you are using the latest version of TSP's AvisynthMT version above.

Enter the parameters like this

MT("fft3dfilter(sigma=5,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)",2,8,false,true)
MT("fft3dfilter(sigma=5,beta=1,plane=1,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)",2,8,false,true)
MT("fft3dfilter(sigma=5,beta=1,plane=2,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)",2,8,false,true)

I hope this helps.

EDIT

BTW Boulder is right. If you have any more problems, start a new thread. I or someone else can help you from there.

[Revgen]

Now that I got that out of the way... ;)

There seems to be an issue where newer versions of FFT3Dfilter (AUG 29 version in my case) crash when be used with SetMTmode. The version I used in my previous tests was the July 05 version, which worked fine.

I'm going to download the other inbetween versions from fizicks site and see if I can track which .dll starts giving the problems first.

[tsp]

Revgen: what kind of problems? I just discovered that Fizick released the sourcecode so it should be easy to find/fix the bug.

[Revgen]

VirtualDub and WMP crash immediately whenever I open an .AVS file with FFT3Dfilter loaded with SetMTmode(). I forgot to mention earlier that this happens with ALL VERSIONS of avisynthmt. Even the first one from June.

Fortunately there has been no problems using the old MT.dll though.

There are about 10 different versions of FFT3DFilter that Fizick made between the July 05 version and the Aug 29 one. I'm going to be testing them all today and report which version starts showing the problem.

In the meantime here is the avisynthMTlog that is created when VirtualDub crashes:

ThreadID:4088 called Distributor
Thread created Handle:0000013CThread created Handle: ThreadId: 000001404092 ThreadId:

I hope this helps for now.

EDIT:

Here is the AVS script I used when it crashed.

SetMTmode(2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
fft3dfilter(sigma=5,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)


EDIT2:

Okay I've now determined that the problem starts with version 1.7 (Aug 29). Version 1.6 (Aug 03) didn't crash with the above script.

According to Fizick's site (http://bag.hotmail.ru/fft3dfilter/fft3dfilter.dhtml)(bottom of the page) the 1.7 version;

"changed sharpening to Gaussian filter with new parameter scutoff;
added SSE version for sharpen mode and pattern modes bt=2,3 ;
restuctured and released code under GNU GPL v.2."

I don't know why any of these changes could cause a crash. The most significant change seems to be the SSE instructions.

Then again I'm not a programmer. :)

[Aquilonious]

DVD's are MPEG2 video's.

Are these .vob files? If so then you need to use DGIndex and create a .d2v file so they can be read in avisynth.

Yes, they are vob files. I just wanted to use a script in Rebuilder rather than converting any files to different formats. I now can do that (see below).


PixieDust can be very finicky. Try talking to JDobbs (DVD RB author) about the problem.

I asked manono and he referred me to the following D9 forum link:

http://forum.doom9.org/showthread.php?t=85384&highlight=Pixiedust

I now have PixieDust working and use the following script in Rebuilder:

LoadPlugin("C:\Other\Plugins\LoadPluginEx.dll")
LoadPlugin("C:\Other\Plugins\DustV5.dll")
Converttoyuy2()
PixieDust(2)
Converttoyv12()

Whether I'm using STMedian or Dust, it slows down processing considerably--by about 400%. Groan. I really need to upgrade! :(


Another filter you can try instead of PixieDust that I prefer using is FFT3Dfilter. IMHO it's just as good if not better than PixieDust at retaining detail and preseving noise. The only disadvantage it has is blocky artifacting at high sigma values.

These values offer about the same compression as PixieDust when used with MSU Lossless Codec.

fft3dfilter(sigma=5,beta=1,plane=0,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)
fft3dfilter(sigma=5,beta=1,plane=1,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)
fft3dfilter(sigma=5,beta=1,plane=2,bw=40,bh=40,bt=3,ow=4,oh=4,kratio=2)

Perhaps I will experiment a bit with fft3d, but I really like PixieDust's output, though it seems to wash out color a just a hair. Again, I'll have to experiment.

[tsp]

revgen: found the bug (in avisynth). It's the same problem as in this (http://forum.doom9.org/showthread.php?t=100715) thread. So if you are lucky fizick will release a new version that doesn't use MakeWritable() else you will have to wait until I make the neccesary fixes in avisynth.

[Revgen]

I'll wait. I'll just use version 1.6 for now.

Thanks.

PS

IanB mentioned in the thread (Post #2) that some of the avisynth programmers are (or at least were at one time) opposed to making the necessary changes because of conflicting philosophies.

Would this fixed version be official or unofficial?

[Revgen]

I just got your email and replied back to you.

It says that you sent it to me over 12 hours ago.

There goes my ISP again. :rolleyes:

[Revgen]

Good News.

Fizick just released version 1.8 which solves the problem.

The performance is just as fast as the Jul 05 version using the avisynth256MT3 posted in this thread.

[Revgen]

I just found out about this bug (http://support.microsoft.com/?id=896256) in Windows XP that can affect the performance Multiple Processor configurations.

It can be fixed by asking for a hotfix from M$.

I'm going to ask them for it and see if it might solve the xvid issue.

EDIT

I installed the hotfix. Unfortunately there isn't any real performance improvement.

Oh well, I tried. :rolleyes:

[lcksg]

Thanks for this great filter.

I am using VDub on a dual-Xeon system (HyperThreading enabled) and all 4x CPU loads goes 100% when 2x VDub is run concurrently (it used to be 60% max).

But after some other apps are started, the CPU loads drops to 60% or so.

[tsp]

lcksg: so if you don't start any other apps the cpu-utilization stays at 100%?

[lcksg]

lcksg: so if you don't start any other apps the cpu-utilization stays at 100%?
Yes but it goes stable around 95-98% mostly, after a few minutes. Load changes are observed in Task Manager when the 2x VDub threads switches CPUs.

If some other apps are started after 2x VDub, the Vdub loads will be reduced and stays reduced (goes back to 60-70%). Changing the thread priority (in VDub or Task Manager) produces this effect also (it doesn't matter what you change to).

To maintain close to 100% load for all CPUs, no other apps should be started after VDub starts encoding. To workaround this issue, I launched all apps I need to use before starting encoding.

Anything you need me to test? Thanks.

Regards,
lcksg

[tsp]

strange thing. Does it make any difference if you use another script?

[lcksg]

CPU loads are maxed out during the first 30 mins or so, after which one of the VDubs threads start to lock up (the encoding frames doesn't advance).

Single VDub session don't have this problem, regardless of which script is used (I have 2 .vobs, they take turns to lock up when using 2x VDub).

I use SetMTMode(2,2), are there any other options? Thanks.

[tsp]

that makes more sense. So basicly starting another application just lock one of the vdub instances up. Could you post your script? Also do you use SetMTMode(2,2) when you use only 1 instance of vdub else try SetMTmode(2,4) instead with one instance to see if a lockup happends after a couple of hours.

What version of virtualdub do you use?

[lcksg]

I can't get MT to work anymore after re-installing WinXP. How many versions of MT are available? Do I download them only from the 1st post of this thread?

Using Avisynth ver. 2.5.6 or should I use the avisynth.dll bundled with MT? Thanks.

[tsp]

go to my homepage www.avisynth.org/tsp and download MT (MultiThreading in avisynth) version 0.41 and multithreaded version of avisynth 2.56 use it with MT or alone if you only uses SetMTMode() and not MT() you only need the multithreaded version of avisynth 2.56. Note that only avisynth.dll is included in that download so first install avisynth 2.56 and then overwrite avisynth.dll.

[lcksg]

Using Avisynth 2.5.6, Xvid 1.0.3

tsp's MT version 0.41 & multithreaded version of avisynth 2.56 (avisynth.dll)

Some prelim observations using 1x Vdub :

SetMTMode(2,0) uses 99% CPU (max)
SetMTMode(2,2) uses 72% CPU (max)
SetMTMode(2,4) uses 98% CPU (max)

SetMTMode(4,0) uses 55% CPU (max)
SetMTMode(4,2) uses 60% CPU (max)
SetMTMode(4,4) uses 45% CPU (max)

CPU load observed in Task Manager, thread priority in VDub & Task Manager is set to Normal (default).

[tsp]

lcksg: Did you experince any lockups with only 1 vdub running? Also could you post the script you used. It is a little strange that there are any difference between SetMTmode(x,0) and SetMTmode(x,4) because when using 0 it default to the number of cores that in your case is 4. Also somewhat more informative would be the time it takes to encode the clip so we can see how well the multithreading codes scales.

[lcksg]

LoadPlugin("C:\Program Files\PLUGINS\AVISYNTH\DGDecode.dll")
LoadPlugin("C:\Program Files\PLUGINS\AVISYNTH\UnDot.dll")

SetMTmode(2,0)
MPEG2Source("D:\VOBS\THE_COUNT_OF_MONTE_CRISTO\VTS_01_PGC_01_1.d2v", idct=4, cpu=0)
Crop(0,6,-0,-6)
LanczosResize(704,396)
UnDot()
http://img428.imageshack.us/img428/7870/setmtmode205xm.th.jpg (http://img428.imageshack.us/my.php?image=setmtmode205xm.jpg)

There's no lockups at all with 1x Vdub, CPU loads in Taskmgr hovers between 80% - 92% (above 85% mostly).

Task Manager report CPU loads in between threads so it's difficult to be accurate but I don't think there's much difference between SetMTmode(x,0) and SetMTmode(x,4).

I'll try to rip a few small chapters later to test encoding times.

SetMTMode(2,x) is much faster & uses more CPU loads than SetMTMode(4,x).

[lcksg]

Script same as above but something is not right with the encoding times & CPU usage

MT mode | time mm:ss | CPU usage in Taskmgr

HT-on
MT-off | 1:35 | 45% CPU
MT-20 | 2:16 | 92% CPU
MT-22 | 1:46 | 67-70% CPU
MT-40 | 3:58 | 36%-41% CPU
MT-42 | 2:12 | 48%-54% CPU

HT-off
MT-off | 1:29 | 87-90% CPU
MT-20 | 2:00 | 88%-100% CPU
MT-40 | 2:26 | 82%-89% CPU

[tsp]

Very strange indeed. I don't know whats going on there. Will try to reproduce it on my dual celeron 400 MHz

[danpos]

@TSP

Dear colleague:

Is there any chance on you building a MT filter version for 'grid computing' ? (By 'grid' I mean to spread the work on a Local Network, with thread started at the individual CPUs for each 'node'). I think that would be very usuful for several avisynth users that have access to a LAN.

Cheers,

[tsp]

I think TCPdeliver (http://www.avisynth.org/TCPDeliver) would be a better solution for that.

[danpos]

I think TCPdeliver (http://www.avisynth.org/TCPDeliver) would be a better solution for that.

Thanks for your answer. I will try this out. :)

Cya!

[Devinator]

I am trying to figure out if I would benefit from this? I use DVD Rebuilder + CCE(which uses avisynth and the dgdecode plugin). I have an X2 system crrent using avisynth 2.5.

[Revgen]

Does DVD Rebuilder allow you to customize your AviSynth settings?

If so than use SetMTmode(2) before your mpeg2source() line and see what happens.

[Socio]

Hey!

I have been testing this with my dual P4's and Limitedsharpen with realtime DVD playback it seems to work with the script just fine and improves performance quite a bit.

Here is how I used it:

MT("LimitedSharpen(ss_x=1.0,ss_y=1.0,Smode=3,strength=40,overshoot=7)")


Awesome plugin TSP!

[Revgen]

Limited Sharpen works even faster in SetMTmode(2) in my experience.

[Socio]

I have tried that before but the SetMTmode parameter does not work using LimitedSharpen with ffdshow for real time DVD playback, I get an error " no function called SetMTmode" when I try it.

Using just the call the way I have it does work and work well however.

[Koroshiya]

My system is a Dual Xeon 2.4Ghz 533FSB in HT mode

=======================================================
With standard avisynth 2.56+vdubmod using the following settings for a 24 minute Divx video to Xvid:

Undot()
VagueDenoiser(threshold=2,method=1,nsteps=6,chroma= true)
BlindDeHalo3(1.5,1.5,100).LimitedSharpen2(ss_x=2.0,ss_y=2.0,Smode=3,strength=100)

Final Time of Encoding for single pass: 1h55min
Average FPS: 5
Average CPU Load: 35%

=======================================================

Using Avisynth2.56MT3 with the following settings for the same 24 min video:

SetMTmode(2)
Undot()
VagueDenoiser(threshold=2,method=1,nsteps=6,chroma= true)
BlindDeHalo3(1.5,1.5,100).LimitedSharpen2(ss_x=2.0,ss_y=2.0,Smode=3,strength=100)

Final Time of Encoding for single pass: 52min
Average FPS: 12
Average CPU Load: 100%

=======================================================

This is one godly plugin for multi-processor users. Thanks tsp!

Only downside(not really) is my system is extremly laggy in mode2, but that is to be expected since it's using 100% load ^_^.

[tsp]

Socio: you need the newest special multithreaded version of avisynth 2.56 that you can get at my homepage (link in my signature.).

[Socio]

Thanks TSP,

I now run it with this call and it works great:


Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpen.avs")

SetMTmode(2)
MT("LimitedSharpen(ss_x=1.0,ss_y=1.0,Smode=3,strength=40,overshoot=7)")

[tsp]

Socio: try using

Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpen.avs")
MT("LimitedSharpen(ss_x=1.0,ss_y=1.0,Smode=3,strength=40,overshoot=7)",avisynthTS=true)

or

SetMTMode(2)
Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpen.avs")
LimitedSharpen(ss_x=1.0,ss_y=1.0,Smode=3,strength=40,overshoot=7))

and see what is the fastest

[Socio]

This is the faster version no question, thanks! :thanks:

Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpen.avs")
MT("LimitedSharpen(ss_x=1.0,ss_y=1.0,Smode=3,strength=40,overshoot=7)",avisynthTS=true)

[Socio]

I tested it some more this afternoon, because I am using it for processing DVD playback in realtime it is not possible to get a concrete benchmark.

What I did however was scale the DVD to 1080p and set ffdshow to show the decoder fps, I ran three tests all with the same processing settings.

The first test was with no MT and stock Avisynth.dll and I got 6-12 Fps would peak at 12 fps but mostly stayed in the 6-8 range.

Then with the first call you suggested I got 12-14 fps. never dropped below 12 and never higher than 14.

Lastly I tried the last call you suggested and got 16 -21 fps, never dropped below 16 and never got higher than 21 which is near double what I got without MT and tells me your plug in not only works it freaking rocks!

[Revgen]

...which is near double what I got without MT and tells me your plug in not only works it freaking rocks!

Amen Brother! :)

[Devinator]

I am trying to figure out if I would benefit from this? I use DVD Rebuilder + CCE(which uses avisynth and the dgdecode plugin). I have an X2 system crrent using avisynth 2.5.


Does DVD Rebuilder allow you to customize your AviSynth settings?

If so than use SetMTmode(2) before your mpeg2source() line and see what happens.


I manged to anwser my own question...

Firstly DVD Rebuilder does allow you to customize your AVS scripts. Not very well however. You can add lines but it places them after the mpeg2source() line and before the filters it applies.

There is a tool called RP-Opt (http://forum.doom9.org/showthread.php?t=75202) which allows you to edit the avs files without having to manually open each one. You edit them for each title set. I was able to add the loadplugin and SetMTmode lines in the correct position. I saw a CCE speed gain that varies from .5 to 1.5 in the various segments and cpu utilization of CCE increased from 60-75% to nearly 100%.

Also be aware that DVD Rebuilder must use an older version of DGDecode.dll, which is incompatible with the current DGIndex so set up the rebuilder friendly DLL with a different name or rename it when using each.

I also saw a very large, nearly 100% performance gain in DVD->x264 encoding via this plugin. Most impressive!

[Badness]

Any further luck using MT on Dual Xeon's encoding XviD?

[tsp]

Badness: It seems to depend on the script you plane to use. Some (Koroshiya) did experience faster encoder while others (lcksg) had slower encoding.

[mg262]

tsp,

Can I trouble you with another abstract question? I'm planning a filter that needs intermediary working space (i.e. a big array or VideoFrame) in each GetFrame call. If two GetFrames are called simultaneously, they each need their own working space, *not* shared. So far so good, except that I don't want to allocate/deallocate every GetFrame call, since there are no guarantees on the allocation speed.

Apart from the intermediate space, everything is fully parallel-friendly, so I would like to find a similarly parallel-friendly way of providing intermediate space. The best way I can think of to deal with this is to have a pool of appropriately sized arrays -- i.e., effectively to build a custom allocator that is called in every GetFrame... but then the allocator might be called twice simultaneously. Any advice on a clean way to go? Should I try and put some kind of lock on the allocator,* or should I just leave it for MT(mode = 2) to deal with or is there a simpler solution?

*which will presumably affect portability.

It's going to be a while before I try building something like this, so there really is no hurry to answer the question. (It's not terribly important anyway... but I would much rather build things to be parallel-friendly where possible.)

Thanks,
M.

[AssassiNBG]

OMG I'm so tired, I can't think straight. Someone tell me why this syntax doesn't work please ?

MT("ffdshow("default")")

It says it expected a "," or a ")". Hmm... don't I have them ?

[tsp]

mg262:

with mode=2 there will never be two GetFrame calls to the same filter instance as there are as many filter instances as there are threads. So it should be safe to allocate the working space in the constructor and release it in the destructor.
If you wanted to share some variables between the different instances of your filter you would need to use the PClipLocalStorage (http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/core/Attic/cliplocalstorage.h?view=auto&rev=1.1.2.3&sortby=date&only_with_tag=avisynth_2_6) in avisynth 2.6.

OMG I'm so tired, I can't think straight. Someone tell me why this syntax doesn't work please ?

MT("ffdshow("default")")

It says it expected a "," or a ")". Hmm... don't I have them ?

try this

MT(""" ffdshow("default") """)

[AssassiNBG]

Yay it worked! Thankz!

PS Although it didn't do any changes. Speed is exactly the same. :( Second core doesn't seem to work according to EVEREST (benchmark app). I've got a P4 HT @ 3000 GHz.

[Revgen]

hmm...interesting.

I used to use this in order to get it to work.


d="default"

MT("ffdshow(d)")

[AssassiNBG]

Still not doing anything. :(

[tsp]

try seing how many cores windows task manager shows (http://www.tomshardware.com/cpu/20051128/how_to_build_a_triple_core_pc-02.html) if the total cpu utilization is already 100% before running MT() when you will not gain much by running it.

[AssassiNBG]

It is not. Both are going not more than 50%. And a benchmark app shows only the second core is actualing processing, the first one is idling.

[tsp]

it might be because ffdshow avisynth filter doesn't support multithreading. I will try to figure out.

[mg262]

If you wanted to share some variables between the different instances of your filter you would need to use the PClipLocalStorage (http://cvs.sourceforge.net/viewcvs.py/avisynth2/avisynth/src/core/Attic/cliplocalstorage.h?view=auto&rev=1.1.2.3&sortby=date&only_with_tag=avisynth_2_6) in avisynth 2.6.Thank you!

[tsp]

Here is an exsample on how the PClipLocalStorage can be used to share a cache between multiple instances(that are created with mode=2,4):

class Cache
{
public:
//These function should be threadsafe. The most simple way is to use a
//critical section like this
PVideoFrame GetCachedFrame(int framenumber)
{
EnterCriticalSection(&cs);
//Code
//...


LeaveCriticalSection(&cs);
return retval;
}
SetCachedFrame(PVideoFrame frame);
private:
CRITICAL_SECTION cs;
}



class Sample : public GenericVideoFilter{
public:
Sample(PClip _child, IScriptEnvironment* env);
~Sample();
PVideoFrame __stdcall GetFrame(int n, IScriptEnvironment* env);
protected:
PClipLocalStorage cls;
Cache* FrameCache;
}

Sample::Sample(PClip _child, IScriptEnvironment* env) :
GenericVideoFilter(_child),cls(env)
{
//if the cache has not been created yet GetValue will return 0
if(cls->GetValue()==0) {
//create the cache and save the address in the PClipLocalStorage
FrameCache = new Cache();
cls->SetValue(static_cast<void*>(FrameCache));
}
// The cache has been created so assign the address to FrameCache
else {
FrameCache=static_cast<cache*>(cls->GetValue());
}
}

Sample::~Sample()
{
//only delete FrameCache if it is not delete yet.
if(cls->GetValue()!=0) {
delete FrameCache;
cls->SetValue(0);//Signal that the cache is deleted
}
}

[shpitz]

Script same as above but something is not right with the encoding times & CPU usage

MT mode | time mm:ss | CPU usage in Taskmgr

HT-on
MT-off | 1:35 | 45% CPU
MT-20 | 2:16 | 92% CPU
MT-22 | 1:46 | 67-70% CPU
MT-40 | 3:58 | 36%-41% CPU
MT-42 | 2:12 | 48%-54% CPU

HT-off
MT-off | 1:29 | 87-90% CPU
MT-20 | 2:00 | 88%-100% CPU
MT-40 | 2:26 | 82%-89% CPU


the weirdest thing...

i have the same problem:

if i don't set mtmode in the script, it encodes at a certain speed.

if i use any of the mtmodes, cpu will go to 100% but encoding speed is reduced to half in some of the mt modes !

so from the testing i've done it appears that using mt mode actually worsens instead of speed up which really doesn't make any sense.

i've placed the avisynth with mt-included dll from TSP's website in system32.

my spec is dual xeon 3.2ghz with 1gb ram.

my script is as follows:

# PLUGINS
LoadPlugin("D:\TBS\Filters\Decomb521.dll")
LoadPlugin("D:\TBS\Filters\DGDecode.dll")
LoadPlugin("D:\TBS\Filters\UnDot.dll")
LoadPlugin("D:\TBS\Filters\MSharpen.dll")
LoadPlugin("D:\TBS\Filters\RemoveGrainSSE3.dll")

SetMTMode(2,2)

mpeg2source("F:\hd_capture.d2v",idct=3)
SelectEven()

trim(5017, 6815)

Crop(162,4,-164,-4)

undot()

BicubicResize(512,384,0,0.75)


am i missing something?

[tsp]

shpitz: No not really. Might try SetMTMode(2) instead but I don't think it will improve it. You could try to but some of the more cpu demanding filters in like limitedSharpen or vaguedenoise because it seems to be very dependent on what script is used. Rather strange with these dual Xeon. Could anyone with an Athlon X2 test if they got the same problem with the above script?

[Socio]

the weirdest thing...

i

my script is as follows:

# PLUGINS
LoadPlugin("D:\TBS\Filters\Decomb521.dll")
LoadPlugin("D:\TBS\Filters\DGDecode.dll")
LoadPlugin("D:\TBS\Filters\UnDot.dll")
LoadPlugin("D:\TBS\Filters\MSharpen.dll")
LoadPlugin("D:\TBS\Filters\RemoveGrainSSE3.dll")

SetMTMode(2,2)

mpeg2source("F:\hd_capture.d2v",idct=3)
SelectEven()

trim(5017, 6815)

Crop(162,4,-164,-4)

undot()

BicubicResize(512,384,0,0.75)


am i missing something?

Don't you have to load the MT.dll as well?

[shpitz]

thanks for the reply TSP.

the really weird thing about all this thing is the fact that the cpu usage goes up while the encoding speed goes down. it's like it is going up-hill, more power is needed yet the climbing rate is getting slower and slower, it is really bizarre.

what really gets me is the fact that during an encode, cpu usage is around 40%, no more, no less. and this gives me the feeling the pc is not used to its whole potential...

i could run 2 instances of vdub and encode 2 clips at the same time, but still the encoding speed of 1 will affect the other, so 2 instances doesn't really mean double the speed.

so all i want is to maximize the speed of 1 single encoding session.

[tsp]

not if he only uses SetMTMode because that is included in the modified avisynth.dll

[shpitz]

not if he only uses SetMTMode because that is included in the modified avisynth.dll

that's kinda confusing TSP... what does setmtmode do if i don't use mt.dll on filters in the script? what does each one of them do? how does it work?

thanks

[tsp]

well all the information about the modified avisynth.dll is scattered around in this thread. Some day I really should sit down and write some proper documentation for this.
Basicly SetmtMode change the script parser and the internal cache so that more than one frame can be processed at a time. To do this I had to modify avisynth so that is the reason why it's not included in mt.dll.
Mt on the other hand split the frame up in smaller pieces and run each piece on a different thread/processor. This can be done with a plugin allthough it works best with the modfied dll.
For more information try reading the whole thread (or just my post :) ).
The strange thing as you said is that it runs slower while using 100% cpu time. There are of cource an overhead by running avisynth multithreaded but not that high. Maybe running all these instances pollutes the cache? Does it makes any difference to disable HT?

[shpitz]

usually disabling HT gives a little boost in performance in some apps since it reduces the overhead like you said, since HT is not really 2 physical cpus...

so what you're saying is that if i just use the modified avisynth and set mtmode in the script it should be enough and therefore using mt() is redundant then?

[tsp]

yes but some filters works better with mt() like fft3dfilter while others that depends on information about the whole frame(like motion compensation or smart deinterlacers) work better with setmtmode. It's possible to use both in the same script.

[mg262]

A thought for the long run: it would be nice if filters had some way to indicate to the environment what MT modes they were compatible with.

[tsp]

mg262: the filters can call env->SetMTMode(int mode,int threads,bool temporary)
where mode is the requested mode, threads can be ignored(only used by SetMTMode), temporary indicate if the mode change only should be applied to this filter.
If multithreading is not enabled this command will do nothing.
It also possible to get the current mode by calling env->GetMTMode(bool return_nthreads)
set return_nthreads to return the number of threads created if false then the current mode is returned.

[puffpio]

I get a Script error: there is no function named "SetMTMode"

I put MT.dll in my avisynth plugin directory, and I put the avisynth.dll in my windows\system32 directory which overwrote the stock avisynth.dll

here is my script

LoadPlugin("C:\Program Files\dgindex\DGDecode.dll")

SetMTMode(2)

video = mpeg2source("fm2.d2v")
video = video.DeDot()
video = video.AssumeTFF()
interp = video.separatefields().selecteven().EEDI2(field = 1)
video = video.tdeint(edeint=interp)
video = video.MT("FFT3DFilter(sigma=3, plane=4, bt=4, sharpen=0.7)", avisynthTS=true)
video = video.crop(4, 0, -4, -4)
video = video.LanczosResize(512, 384)

return video

[puffpio]

btw i tried rebooting in case the old avisynth was still in memory..I also searched my hard drive and that is the only location that avisynth.dll exists

[puffpio]

woot got it to work..

tsp: the avisynth.dll in your MT_041.zip is NOT the multithreaded version... :P

[tsp]

woot got it to work..

tsp: the avisynth.dll in your MT_041.zip is NOT the multithreaded version... :P
it just the version before I implemented SetMTMode :)
also try this version

LoadPlugin("C:\Program Files\dgindex\DGDecode.dll")

SetMTMode(2)

video = mpeg2source("fm2.d2v")
video = video.DeDot()
video = video.AssumeTFF()
interp = video.separatefields().selecteven().EEDI2(field = 1)
video = video.tdeint(edeint=interp)
SetMTmode(6)
video = video.MT("FFT3DFilter(sigma=3, plane=4, bt=4, sharpen=0.7)", avisynthTS=true)
SetMTMode(2)
video = video.crop(4, 0, -4, -4)
video = video.LanczosResize(512, 384)

return video

because MT does it's own multithreading (it creates it's own thread) so it wouldn't take advantage of SetMTMode(2).

[Badness]

Dual XeoN 3.6
2 Gigs DDR2
Sata 250 hard drive
avisynth256MT3 Encoding XviD
HD Source 1920X1080

SetMTMode(2)
LoadPlugin("C:\Program Files (x86)\AutoGK\DGMPGDec\dgdecode.dll")
mpeg2source("D:\Work\test.d2v")
crop(4,4,-4,-8)
LanczosResize(1280,720)

MT Enabled FPS - 15.938
MT Disabled FPS - 19.262

So what am I doing wrong here? It runs at 100% CPU with the MT enabled but still slower than with the MT disabled. Disabled only uses 40% CPU. I really wish I could get full use out of this sucker.

[shpitz]

Dual XeoN 3.6
2 Gigs DDR2
Sata 250 hard drive
avisynth256MT3 Encoding XviD
HD Source 1920X1080

SetMTMode(2)
LoadPlugin("C:\Program Files (x86)\AutoGK\DGMPGDec\dgdecode.dll")
mpeg2source("D:\Work\test.d2v")
crop(4,4,-4,-8)
LanczosResize(1280,720)

MT Enabled FPS - 15.938
MT Disabled FPS - 19.262

So what am I doing wrong here? It runs at 100% CPU with the MT enabled but still slower than with the MT disabled. Disabled only uses 40% CPU. I really wish I could get full use out of this sucker.

exactly what i'm reporting above.

which os are you running? xp x64 or 2k3 x64?

also, why you are not deinterlacing? 1080i is interlaced...

[tsp]

shpitz, Badness: If you encode to Huffyuv or MJPEG is the framerate still slower with MT enabled?

[puffpio]

I tried using SetMTMode(6) before the FFT3DFilter but it crashed the encode after 200-300 frames.

I'm on a P4 w/ HT so I just decided to go back to disabling HT and running everything w/o MT to be safe.

[tsp]

puffpio: thanks for trying. I will try and see if I can reproduce it. Oh and no need to disable HT as long as you don't use SetMTMode()/MT()

[puffpio]

if i dont disable HT, then I will not be able to achieve 100% CPU utilization correct? Or is that just not represented correctly in the task manager?

[Revgen]

Perhaps this issue is related to this issue (http://news.zdnet.co.uk/0,39020330,39237341,00.htm) and not MT.

[Badness]

exactly what i'm reporting above.

which os are you running? xp x64 or 2k3 x64?

also, why you are not deinterlacing? 1080i is interlaced...


OS is XP X64
Shotime Directv HD even though it is 1080 is 99% Straight Film.
Same with HBO.

Why, I dont know, but always has been. And my test clip was off shotime.

[tsp]

if i dont disable HT, then I will not be able to achieve 100% CPU utilization correct? Or is that just not represented correctly in the task manager?
remember that the utilization represent two cores when using HT so that 50% utilization with HT = 100% utilization without HT (that is of course not entirely true because the second core is virtual) so allways look at the speed first and ulitization second (like these dual xeon users that gets higher cpu utilization but slower speed).
Else try to just use MT with fft3dfilter and don't use SetMTMode. That should work.

Perhaps this issue is related to this issue and not MT.

might be some of the explanation. But this seems to happend with the more "simple" avisynth scripts. Revgen could you try and see if you get the same result as Badness with his script on your X2?

[Revgen]

might be some of the explanation. But this seems to happend with the more "simple" avisynth scripts. Revgen could you try and see if you get the same result as Badness with his script on your X2?

You know what, I just looked at Badass' post and realized that he was encoding his video to Xvid. Seems like this Xvid issue doesn't discriminate against Intel procs either. The fact that he's trying to encode at 1280x720 probably amplifies the problem. I'll try a test with the 1080 material I have and see if it is the same problem for me.

I was too tired last night to notice. :rolleyes:

@shpitz and puffpio

Are you guys trying to encode to Xvid? The Xvid issues are documented earlier in the thread for you to digest. If not than I'll try your scripts and see if I have any success.

@all

Anybody having issues with MT should try encoding to X264 instead of Xvid and see if your performance decreases continue.

[puffpio]

revgen: I was encding to x264 w/ sharktooth's insane profile and setting the # of threads to 2

[tsp]

puffpio: You don't get 100% cpu utilization with that setting? Anyway I tested the script on my dual celeron. tdeint and Dedot didn't work because my celerons don't have integer sse. Altso it looks like fft3dfilter 1.8.5 and MT together is causing some problems (it works without mt and with SetMTmode(2)). I will try an older version and see if that helps.

Revgen: I totally forgot about the XviD issues. Koroshiya didn't have this problem when encoding to XviD from DivX.

[puffpio]

no, I got 100% cpu quite easily :)

[Revgen]

@Badass

I used your script on some 1080p material and it seems that your Crop and LanczosResizing commands slow down SetMT.

I usually never use Crop or Lanczos Resize very much since I like to keep my DVD backups at their original resolution.

The speed difference was about 16% improvement when they were removed.

I did these tests with huffyuv.

This may not be an xvid issue afterall.

@TSP

hmm... that Divx to Xvid post is interesting. I've only used .d2v files to encode to Xvid so far. I'm going to check this out.

EDIT

Tried encoding xvid from a huffyuv file. Unfortunately there was no performance difference.

[krieger2005]

I also noticed a slowdown. Bu since my pc seems to do problems with SMP i don't post it here. I noticed that at the begining it start very fast the calculation but slowdown then.

I can't say which script i used or which modified avisynth-version. Sorry... since i noticed this slowdown i stopped to use mt-modified-avisynth. BTW: The cpu-usage was about 90%-100%. Allways.

[shpitz]

i've been encoding xvid only with xvid 1.1.0 beta 2 ICL7.0 .

since 99% of my encodes are from tv i must crop and resize, so i got no other choice but to use them.

i just installed xvid dec 09 cvs head and will try it using setmtmode.

[shpitz]

shpitz, Badness: If you encode to Huffyuv or MJPEG is the framerate still slower with MT enabled?

i tried to encode a 1min clip from a 720p capture.

when setmtmode is not used, i get:

50 seconds for xvid 1st-pass
48 seconds for huffy 2.2.0
fps is around 36.0
cpu usage is around 40%

when setmtmode is set to (2,2), i get:
55 seconds for xvid 1st-pass
43 seconds for huffy 2.2.0
fps is around 32.0
cpu usage is around 70%

if i set the mode to (2,4) or higher, cpu usage does up to 100% and encoding speed is reduced to a crawl...

so it looks like it's encoder-independant...

i'll keep on testing.

thanks

[tsp]

shpitz: Thanks for the test. So XviD really doesn't like SetMTMode while h.264 and/or more complex scripts benefits more from it.

[shpitz]

i haven't really gotten into h264 so i can't share my own experience...

i will try some SD caps as well as using mt() and report back.

[morsa]

I've tried MT with EEDI2 and it is impossible to use this filter with the multithreaded option.I'm I right?

[Revgen]

I've also had problems using this filter in SetMTmode. It crashes every time I open my AVS file.

I tried using this script.

SetMTmode(2)
mpeg2source("mydrive:\myfolder\my.d2v")
DeDot()
interp = separatefields().selecteven().EEDI2(field=1)
deinted = tdeint(edeint=interp,order=1,field=1)
tfm(order=1,mode=1,clip2=deinted)
tdecimate(mode=1)

and it crashed instantly.

I tried changing the SetMTmode to 3 then 4 then 1, but that didn't work either. Getting rid of the interp, deinted, and the clip2=deinted parameters or turning off SetMTmode was the only way to solve the crashes.

[shpitz]

revgen, comment each time 1 line and see if you can find the offending line...

[Revgen]

As I've mentioned in my last post, the offending lines are the

interp = separatefields().selecteven().EEDI2(field=1)
deinted = tdeint(edeint=interp,order=1,field=1)
tfm(order=1,mode=1,clip2=deinted)

filter chain.


Substituting this filter chain with

tfm(order=1,mode=1)

solves the problem.

[tsp]

I can't test it on my celeron because tdeint (or EEDI2) requeres integer SSE. I looked at the source code and it didn't seem to contain any offending code (but hey something is wrong.)

[tritical]

I can't test it on my celeron because tdeint (or EEDI2) requeres integer SSE. If that's true then there is a bug somewhere, tdeint/eedi2 should work fine without an isse capable cpu or even an mmx capable cpu. Though I think avisynth requires an mmx capable cpu since it issues _asm emms instructions in some spots without checking for mmx support. TDeint doesn't even have any assembly code in it, so it must be somewhere in eedi2. I'll look into this.

[Revgen]

How would the integerSSE/no-integerSSE issue affect SetMTmode?

[tritical]

I don't know if it would effect setmtmode at all, I was mainly interested just because tdeint/eedi2 shouldn't require an isse cpu at all.

Revgen, could you test the following in the filter chain and see if it works or not:

deinted = tdeint(order=1,field=1)
tfm(order=1,mode=1,clip2=deinted)

just tdeint and no eedi2. Might help narrow down the problem to eedi2.

[Revgen]

@Tritical

I tried your lines that you gave me and SetMTmode works fine now. It looks like the problem is EEDI2.

[Revgen]

@TSP

I tried an old test version of your avisynth.dll you gave me awhile back and decided to try it on my script. Here is the log message after the crash. I don't know if this helps or not.

ThreadID:3892 called Distributor
Thread created Handle:00000220 ThreadId: 884
Thread created Handle:00000224 ThreadId: 1748

[tritical]

Could you try this test version of EEDI2 and see if it fixes things. EEDI2_test (http://bengal.missouri.edu/~kes25c/EEDI2_test.dll)

[Revgen]

Tried it. It still crashes.

[psme]

Does it work with Resize?

Source is NTSC 720x480 DVD. I tried:

MT("BicubicResize(720,480)",avisynthTS=true)

It complains:
MT: Function changed the height! Try splitvertical=true

MT("BicubicResize(720,480)",splitvertical=true,avisynthTS=true)

It complains:
MT: Function changed the width! Try splitvertical=false

MT("BicubicResize(720,480)",splitvertical=false,avisynthTS=true)

It complains:
MT: Function changed the height! Try splitvertical=true

Tried BicubicResize, LanczosResize, Lanczos4Resize with same result.

Thanks in advance.

regards,

Li On

[Revgen]

Use SetMTmode when you resize. Not the older MT filter.

[Revgen]

When I heard that Xvid 1.1 had been officially released as a stable build I decided to try to see if it would work at all with MT. I was surpised to see that it actually worked this time. The results are way better than with previous builds. Which is interesting because previous 1.1 betas had issues with the MT filters (at least for me). I guess the Xvid devs fixed it.

I used these filters:

FFT3DFilter v. 1.85
DGDecode v. 1.46 beta 2

Script for normal processing:

mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
fft3dfilter(sigma=5,beta=1,plane=4,bw=48,bh=48,bt=4,ow=4,oh=4,kratio=2,degrid=1.0)


Script for MT.dll :

mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
MT("fft3dfilter(sigma=5,beta=1,plane=4,bw=48,bh=48,bt=4,ow=4,oh=4,kratio=2,degrid=1.0)",2,0,false,true)


Script for SetMTmode:

SetMTmode(2)
mpeg2source("E:\3s-Goofs and Saddles\goofs.d2v",idct=6)
Trim(4482,5454)
Greyscale()
fft3dfilter(sigma=5,beta=1,plane=4,bw=48,bh=48,bt=4,ow=4,oh=4,kratio=2,degrid=1.0)



Xvid 1.1 Stable Build Results:

MT OFF - W/FFT3DFilter - 8.18 FPS
MT.dll ON - W/FFT3DFilter - 9.54 FPS - About a 16.6% increase.
SetMTmode ON - W/FFT3DFilter - 11.72 FPS - About a 25.2% increase.

EDIT:

I also tried Koepi's experimental build with Syskin's MultiThread patch, but it didn't improve performance at all because FFT3DFilter put a barrier on it's potential. However I did do a no filter test using this source in this thread. (http://forum.doom9.org/showthread.php?p=765046#post765046)

[psme]

From the posts here, it seems the function SetMTMode is a internal function of Avisynth 2.56. So I downloaded the 2.5.6a from http://sourceforge.net/project/showfiles.php?group_id=57023

But it still complain "no such function SetMTMode" with this script:

SetMTMode(2)
BicubicResize(720,480)

Thanks in advance.

regards,

Li On

[tsp]

well it's an internal function of my modified avisynth 2.5.6 that you can get from my homepage(link below). It should be included in the ordinary avisynth version 2.6 when it is released.

[Revgen]

well it's an internal function of my modified avisynth 2.5.6 that you can get from my homepage(link below). It should be included in the ordinary avisynth version 2.6 when it is released.

Perhaps you could edit your first post in the thread and add the link. That way people will know.

[tritical]

@Revgen
I'd really like to solve the two problems with EEDI2 (requiring isse processor and the problem with setmtmode(2)); however, I don't have access to a non-isse capable comp and I don't have access to a dual-processor or dual-core system so it is tough to reproduce the problems. I ran the script you posted with tsp's version of avisynth and setmtmode(2,2) on my laptop with no problems. I am pretty sure the isse problem has to be in my planarframe class since it has the only assembly stuff in EEDI2, but I can't find anything wrong in it. As for the mt problems I'm also at a loss. Could you try the following script and see if it crashes:

SetMTmode(2)
mpeg2source("mydrive:\myfolder\my.d2v")
EEDI2(field=1)

I'll make a debug build with some output strings tonight which should help pinpoint where the problem is.

[Revgen]

The script you gave above works fine for some material, and not for others. It's seems to be the same old story.


I've been testing for a few days now and it seems that it sometimes crashes and sometimes doesn't crash and I can't figure out why.

For example with some sources (.d2v, .avi, etc) using:

interp = separatefields().selecteven().EEDI2(field=1)
deinted = tdeint(edeint=interp,order=1,field=1)
tfm(order=1,mode=1,clip2=deinted)

will crash when it's used.


With some others it won't, but right after I add:

Tdecimate(mode=1)

so I can turn it into a 23.976 file, then it crashes.

Some scripts don't crash at all.

Sometimes changing SetMTmode to 4 or 3 will work and sometimes it won't.

I just can't figure it out.:confused: It doesn't seem to matter whether I use the test version or not.


I hope the debug build can help point to the problem. Thanks.:)



PS- It's TSP that has the iSSE issue.

[tsp]

tritical: I will try and test it on my dual processor non-iSSE machine(I think I will just compile the source code and see where it breaks).

[tsp]

good news.
first both tivtc,tdeint and eedi2 works fine on my non-isse celeron processor. Must have been undot that confused me(sorry for that),
next I discovered one of the causes that lead to the crach with SetMTMode.
In the function PVideoFrame __stdcall TFMPP::GetFrame(int n, IScriptEnvironment *env)
the destination frame is sometimes just assigned from child2->GetFrame. This causes an acces error when calling dst->GetWritePointer in PutHint because env->MakeWritable is not called before. And because TDecimate has its own cache(that is usually cleared on a single processor machine before TFMPP::GetFrame is called) that contains the destination frame the crach occours. A simple solution is to add
this to end of the function TFMPP::GetFrame

if (display) writeDisplay(dst, np, n, fieldSrc);
env->MakeWritable(&dst);
putHint(dst, fieldSrc, hint);
return dst;
}


tritical how much of a problem is it that each instance of TDecimate only process every other frame with SetMTMode(2)?

TIVTC v1.0 Beta 7 for Avisynth 2.5.x with the above change(only thing different) is available here (http://www.avisynth.org/tsp/TIVTC.zip)

[Revgen]

This new TIVTC.dll build solves the Tdecimate problem for me. Thanks.:)

Now the other problems I have are with a Huffyuv capture clip. Whenever I use SetMTmode(2) with it and try the:

AssumeTFF()
EEDI2(field=1)

It crashes. Changing SetMTmode to 3, or 4 fixes the problem. But this script doesn't pose problems for other sources at SetMTmode(2).

I'll eventually find time to cut it and post it to Tritical's FTP site.

[J-Wo]

Hey guys, just stumbled on this thread and it sounds very interesting! Has anyone tried DeGrainMedian with this filter? I started using it with my DVD-RB encodes with the parameters DeGrainMedian(limitY=5,limitUV=5,mode=3). According to the docs this is for "Subtle filtering but useful for hot pixels removal". I find it helps to grain compression without much blurring. The first page mentioned MT 0.41 wouldn't work with dust, so I wasn't sure if this denoiser would work either. Another thread here mentioned using fft3dfilter instead of pixiedust... How do you guys find these compare? Any recommended settings/parameters for use with MT? Thanks in advance!

[tritical]

Thanks for finding that bug tsp. I'll add that fix in the next release.

Processing every other frame would only cause problems for modes 3 and 7 of tdecimate. All other modes should be fine and produce the same output. However, I'm not sure it would make things much faster for modes 0 and 1 since both instances would be doing calculations for all frames. Modes 0/1 work by calculating everything for the entire cycle the first time a frame from a new cycle is requested. During that initial processing it is decided which frame(s) will be dropped and which will be kept and from then on the only processing done for requests for frames in that cycle is just a LUT lookup.

[Revgen]

@Tritical

I posted the clip on your FTP server. It's EEDI2_TestClip.avi.

[J-Wo]

okay guys, been giving these filters a try and had some comments/questions.

To start off, I found it a bit confusing from the first post that there are in fact two different versions of this filter, the older mt.dll and the newer setmtmode. I was also confused that the updated avisynth.dll was in the third attachement and was not the same as the one included with mt.dll. Perhaps you could split your first post off and date the two sections so they appear a bit more seperate?

My confusion actually led me to the discovery that the combination of mt("filter",2,0) + SetMTMode() lead to a faster encode time for my test than either two alone. I'm not sure if I was doing things right, but this is the script I tried:

LoadPlugin("D:\Program Files\DVD-RB PRO\DGDecode.dll")
mpeg2source("F:\BABYLON5_SEASON2_DISC1\D2VAVS\V01.D2V",idct=6)
trim(5,7384)
Crop(16,16,-16,-16)
SetMTMode(1)
MT("DeGrainMedian(limitY=5,limitUV=5,mode=3)",2,2)
AddBorders(16,16,16,16)
ConvertToYUY2(interlaced=true)

This is a 5 min clip used for DVD-RB Pro. My system is an Opteron 165 @ 2.6GHz and I'm using CCE 2.70.02. Strange thing is my CPU utilization never goes beyond 79%. Amy I doing something wrong? Any suggestions for the script I used? Thanks!

[Revgen]

The Task Manager is never accurate when reporting CPU usage. Always measure performance when using different filters.

Using SetMTmode and MT.dll can sometimes result in faster performance sometimes not. Some filters work better exclusively with MT.dll or SetMTmode.

You just have to experiement.

SetMTmode(1) is the fastest that you can use. But it's not that much faster than SetMTmode(2). Often it doesn't work for most scripts I use. For most scripts and filters, SetMTmode(2) tends to work best.

[J-Wo]

Thanks Revegen. I did some more testing after that post and found a large speed increase by putting SetMTMode(2) and the top of my script before mpeg2source, without using the old MT() filter. Bit of an annoyance with DVD-RB as a previous poster mentioned, as I'll have to use a 3rd party program Rb-Opt to manually insert the line. For some reason SetMTMode(1) crashes when I put it at the top of my script... I'll have to play around with this a bit more

[Revgen]

Thanks Revegen. I did some more testing after that post and found a large speed increase by putting SetMTMode(2) and the top of my script before mpeg2source, without using the old MT() filter. Bit of an annoyance with DVD-RB as a previous poster mentioned, as I'll have to use a 3rd party program Rb-Opt to manually insert the line. For some reason SetMTMode(1) crashes when I put it at the top of my script... I'll have to play around with this a bit more

I usually put SetMTmode at the top of my script, since the performance is more steady. When it's not at the top, sometimes the performance can fluctuate throughout the encode.

SetMTmode in general is faster and more compatible with most scripts. However the old MT.dll filter works faster with some filters like FFT3Dfilter.

[J-Wo]

Hmmm I just put SetMTMode(2) at the top of my DVD-RB scripts, but some reason when CCE was encoding the third segment (RB splits the movie up into cells for encoding) the encoder froze. Under Task Manager the CCE process was stuck at 50% CPU utilization but it wasn't doing any encoding, normally it's in the 80's. But all my AVS scripts are viewable under Vdub-Mpeg2 and media player classic. I switched the line to SetMTMode(5) and so far so good. I don't know what's causing the problem!

[tsp]

the first SetMTMode line must appear before the first filter that returns a clip(video) for SetMTMode to be enabled also use SetMTMode(5) with MT because MT does its own multithreading. I know the documentation sucks and I think I will release a new version of MT.dll that includes the newest version of my modified avisynth.dll(and in fact will requere it). That might reduce the confusion.
Could you try this version of your script:

LoadPlugin("D:\Program Files\DVD-RB PRO\DGDecode.dll")
SetMTMode(2)#or 5 if you still get lockups
mpeg2source("F:\BABYLON5_SEASON2_DISC1\D2VAVS\V01.D2V",idct=6)
trim(5,7384)
Crop(16,16,-16,-16)
SetMTMode(5)
MT("DeGrainMedian(limitY=5,limitUV=5,mode=3)",2,2)
SetMTMode(2)
AddBorders(16,16,16,16)
ConvertToYUY2(interlaced=true)

[J-Wo]

does it make a difference if I have my setmtmode line as the very first line in the script, or does it have to be directly preceeding the mpeg2source line?

I'm getting some even more lockups using this script now. mode2 is still causing cce to hang, but so is mode1 and now mode4. One time while testing ALL the modes caused vdub or cce to lock up, and I had to reboot. Really really strange...

[J-Wo]

okay, after some further testing it seems setmtmode(3) was the fastest for me, other modes either locked up cce or were much slower. Before adding any of these new filters, encoding speed was at 97fps. Placing setmtmode(3) at the top brought that up to 108fps using the following script:
LoadPlugin("D:\Program Files\DVD-RB PRO\DGDecode.dll")
SetMTMode(3)
mpeg2source("F:\BABYLON5_SEASON2_DISC1\D2VAVS\V01.D2V",idct=6)
trim(5,7384)
Crop(16,16,-16,-16)
DeGrainMedian(limitY=5,limitUV=5,mode=3)
AddBorders(16,16,16,16)
ConvertToYUY2(interlaced=true)
But as a test I tried going back to just mt.dll and found it was even faster, at 112fps, using this script:
LoadPlugin("D:\Program Files\DVD-RB PRO\DGDecode.dll")
mpeg2source("F:\BABYLON5_SEASON2_DISC1\D2VAVS\V01.D2V",idct=6)
trim(5,7384)
Crop(16,16,-16,-16)
MT("DeGrainMedian(limitY=5,limitUV=5,mode=3)",2,2,false,true)
AddBorders(16,16,16,16)
ConvertToYUY2(interlaced=true)
But if I try to add a combination of setmtmode and mt as tsp suggested the encoding speed becomes worse than without these filters. Interesting...

[mrcleeo]

will this work with just a straight mpeg2 in dvd-rb encode with no filters?

LoadPlugin("C:\Program Files\DVD-RB PRO\DGDecode.dll")
mpeg2source("F:\HP WORKPATH\D2VAVS\V01.D2V")
trim(0,465)
ConvertToYUY2(interlaced=true)
AudioDub(BlankClip())


LoadPlugin("C:\Program Files\DVD-RB PRO\DGDecode.dll")
SetMTMode(2,2) <----?
mpeg2source("F:\HP WORKPATH\D2VAVS\V01.D2V")
trim(0,465)
ConvertToYUY2(interlaced=true)
AudioDub(BlankClip())


i have an amd x2 4400 and i was trying to get it to utilize the whole cpu.

ive tried to get it to work a few times and when it encodes it makes the video with red lettering of errors

[tsp]

It should work if you use my custom avisynth.dll from the first post. What error do you get?

[mrcleeo]

when i go to look at the video it encoded, the video says

"Script Error: There is no function named SetMTMode"

("F:\HP WORKPATH\D2VAVS\v01000000001001.avs line 6")

[tsp]

ok download this file:
http://www.avisynth.org/tsp/avisynth256MT3.zip
It contains a file called avisynth.dll. Make a backup of your existing C:\windows\system32\avisynth.dll and exctract this file to c:\windows\system32

[mrcleeo]

ok that worked

but im not seing a speed increase :/

thanks for your help

----

actually its a bit slower

[tsp]

well it is a rather simple script you are using so that might be why.

[mrcleeo]

well i dont use filters on retail dvds

what else could i add to try to make it speed up?

i get a speed of 4 in cce without MT and with MT i get 3.2


--------

this is the full avs from dvd rebuilder

#------------------
# AVS File Created by DVD Rebuilder
# VOBID:01, CELLID:01
#------------------
LoadPlugin("C:\Program Files\DVD-RB PRO\DGDecode.dll")
SetMTMode(2,2)
mpeg2source("F:\HP WORKPATH\D2VAVS\V01.D2V")
trim(0,465)
ConvertToYUY2(interlaced=true)
AudioDub(BlankClip())

[Revgen]

I believe that CCE takes advantage of multi-threading, right?

If so, then SetMT may be taking processing power away from CCE, while being unable to speed up what is a relatively simple script.

Both Mt.dll and SetMTmode are better used to speed up resource hungry avisynth plugins. If you want more speed you should contact the CCE devs and ask what they are doing to speed up multi-threading routines.

I typically use SetMTmode or MT.dll to speed up slow filters like FFT3DFilter or resource hungry deinterlacers like the EEDI2/TDeint combo. This can turn an encode that may take 24 Hrs. and cut it down to 12 hrs without sacrificing quality.

[Boulder]

CCE doesn't utilize multithreading like for example VDub does. I've mostly seen CPU usage range between 50 and 65 percent. On all my tests, using MT has provided the biggest performance gain.

[tsp]

new release of MT and a new avisynth.dll that are based on the final 2.5.6 version. Also includes some fixes to sharpen and blur so that they work correct with mt and SetMTMode. So please update both. Get them from the first post

[Boulder]

The new version doesn't seem to work correctly with Didée's MCNR_simple2.

http://img379.imageshack.us/img379/8075/men16sr.th.jpg (http://img379.imageshack.us/my.php?image=men16sr.jpg)
http://img484.imageshack.us/img484/3167/men29kc.th.jpg (http://img484.imageshack.us/my.php?image=men29kc.jpg)

The top one is with the new package, the bottom one with v0.41 and a vanilla AVS v2.5.6.

The function is here for example : http://forum.doom9.org/showthread.php?p=766898#post766898

And the script:

DirectShowSource("i:\men.avi",pixel_type="YUY2",audio=false)
Trim(23798,42353).FadeIn(5) ++ Trim(46787,63550).FadeOut(250)
AssumeTFF()
TMCBob()
ConverttoYV12()
Crop(12,4,-4,-4,true)
BilinearResize(656,544)
MT("MCNR_simple2(frames=2,removdirt=true,lprad=2.0)",2)
AssumeTFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()

[tsp]

try increasing the overlap like this:

DirectShowSource("i:\men.avi",pixel_type="YUY2",audio=false)
Trim(23798,42353).FadeIn(5) ++ Trim(46787,63550).FadeOut(250)
AssumeTFF()
TMCBob()
ConverttoYV12()
Crop(12,4,-4,-4,true)
BilinearResize(656,544)
MT("MCNR_simple2(frames=2,removdirt=true,lprad=2.0)",2,overlap=4)
AssumeTFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()

[Boulder]

I had to use overlap=16 due to MVTools, but it doesn't help. The similar problem exists although not at the same frame.

[tsp]

what removedirt script do you use and is it only some of the frames where this occurs(and is it random)?

[Boulder]

what removedirt script do you use and is it only some of the frames where this occurs(and is it random)?
The function apparently uses RemoveDirt at its default settings. I'll try disabling it when my current encode ends and see if it helps.

(EDIT: It's the old removedirt.dll that's used, i.e. the one which has the RemoveDirt() function only)

The problem cannot be seen at every frame but it is the same frames for the same clip every time.

[Boulder]

Nope, the problem still remains even without RemoveDirt.

[tsp]

could you upload a 10 frame clip with the problem frame in the middle to ftp://tempclips%40avisynth%2Eorg:QfJY(86m@avisynth.org/ so that I can try to reproduce the error.

[Boulder]

OK, the sample's there. I had to upload a larger clip (~13MB) as I've got another encode going on which will last several hours. Let me know if you absolutely need a small clip. It's encoded as ffdshow mjpeg.

[tsp]

Boulder: I couldn't reproduce the error but that might be because I had to disable half the filters to get it to work on my non-SSE celeron. Could you try to find out part of the MCNR_simple2 script that causes the error so I can take a closer look at the sourcecode for that filter.

[Boulder]

Ok, will do as soon as I've got the time. I'll try MT on each part of the script and see if anything blows up.

[Boulder]

Hmm, how can I use MT with this:

mt_lutxy(orig,NR,yexpr="x y - 128 +",uexpr="y",vexpr="y",Y=3,U=3,V=3)

I always get the "invalid arguments to MT" error. I tried MT("""mt_lutxy(orig,NR,yexpr="x y - 128 +",uexpr="y",vexpr="y",Y=3,U=3,V=3)""",2)

Also this gives an error regardless of what I've tried:
_repair ? repair(comp,clp,1,3) : comp
I've tried repair with single and triple quotes.

I tried the MaskTools v1.5-branch and MaskTools v2-branch versions of the script. They both failed so it might be that MVTools v1.0.3 is actually the culprit. I cannot try Manao's last MVTools version (v0.9.9.1) as it doesn't work with the script.

[tsp]

you can't. MT only accepts 1 input clip. I will need to write a custom scriptparser to handle more than one input also you can't use a named input like this:

a=last.sharpen(0.5)
MT("blur(a,1)")

so mayby comment out the lines that you want to test?

[Kador]

Hello
Why do I get a "MT does not have a named argument"avisynthTS", which I did not get before ?

[Boulder]

so mayby comment out the lines that you want to test?
OK, a really quick-and-dirty test showed that MVFlow might be the one that doesn't like MT.

MVTools v1.0.3 can be found at least here : http://home.arcor.de/dhanselmann/_stuff/

[tsp]

Kador: Because version 0.5 requires the included avisynth.dll(or 2.6 when it is released) so I removed that argument.

Boulder: I suspected that MVTools might cause it. The sourcecode is rather complicated so it might take a while until then you could try to use cloudeds motion filter instead (http://forum.doom9.org/showthread.php?t=101859)

[Kador]

Thanx I removed it.

After an update, I'm using the latest filters in ZP+ffdshow :
- masktools2
- MT + avisynth dll (from you link)
- the second LSF from the wiki page
- hqdn3d
I get a crash every time after several seconds of read, of course no crash without those filters ...

if I use the LSF call without MT no crash

any idea ?

[tsp]

how does your script look like? Try commenting/removing the different filters and see if that helps(and post what filter that causes the crash).

[Kador]

My script is very simple

MT("hqdn3d(2)LimitedSharpenFaster(ss_x=1.5,ss_y=1.5,Smode=3,strength=50,overshoot=1)")

[Socio]

My script is very simple

MT("hqdn3d(2)LimitedSharpenFaster(ss_x=1.5,ss_y=1.5,Smode=3,strength=50,overshoot=1)")

Try it like this:


MT("hqdn3d(2).LimitedSharpenFaster(ss_x=1.5,ss_y=1.5,Smode=3,strength=50,overshoot=1)")

or this:

MT("hqdn3d(2)
LimitedSharpenFaster(ss_x=1.5,ss_y=1.5,Smode=3,strength=50,overshoot=1)")

[tsp]

Kador: I made it crash with that script now I only needs to figure out why it does it :)

[Mr.Bitey]

tsp,

There seems to be an incompatability between the version of avisynth included with MT05 and the version of avisynth required for post 18 versions of masktools (see the limited sharpen thread) - a similar discussion is occuring there also..

Will AviSynth 2.5.6a (28th oct 2005) or AVS 2.5.6 RC2 (7th Oct 2005) both on sourcefourge work with MT05 or only the included version?

Cheers,
Bitey

[Kador]

Kador: I made it crash with that script now I only needs to figure out why it does it :)

Great (that you reproduced, I'm not alone ;-) )

[swaaye]

I'm running a dual core Opteron and am more than a little fascinated by this MT filter. I've read through a bunch of this thread and I'm more than a little unsure just what the limitations or ramifications of using this are.... I'm using a pretty simple script to work on some episodes of a TV show:


MPEG2Source("R:\DVD\TNGS6D6\VTS_02_PGC4\TNGS6D6.d2v", idct=7)
tfm(d2v="R:\DVD\TNGS6D6\VTS_02_PGC4\TNGS6D6.d2v")
tdecimate(hybrid=1)
autocrop(mode=0,wmultof=4,hmultof=4,samples=10, aspect=-1,threshold=34,samplestartframe=100,leftadd=0,rightadd=0,topadd=0,bottomadd=0)
BicubicResize(640,480,0,0.75)


This, along with multithreaded Divx 6.1.1, puts me around 75%+ CPU, sometimes nearly 100%. But there is room in there for more CPU use, so I thought MT would be worth a try.

Does what I'm using look convertable for use with MT?

[tsp]

Mr.Bitey: So you say that it crash when using my modified avisynth.dll and the latest masktool when you don't use mt or setmtmode? I know that it crach when using mt and masktools together but exactly what goes wrong is not clear yet (other than it looks like a problem with the STL allocator in masktool. I did use a couple of hours yesterday with manao to try debug it).
mt only works with the included avisynth or a fresh build from the latest avisynth 2.6 CVS.

swaaye:
if you want to try I will sugest this script:

Setmtmode(2)
MPEG2Source("R:\DVD\TNGS6D6\VTS_02_PGC4\TNGS6D6.d2v", idct=7)
tfm(d2v="R:\DVD\TNGS6D6\VTS_02_PGC4\TNGS6D6.d2v")
tdecimate(hybrid=1)
autocrop(mode=0,wmultof=4,hmultof=4,samples=10, aspect=-1,threshold=34,samplestartframe=100,leftadd=0,rightadd=0,topadd=0,bottomadd=0)
BicubicResize(640,480,0,0.75)

with mt 0.5 and the included avisynth.dll.

[Mr.Bitey]

tsp,

It crashes with a21 of masktools (and I believe any version after a18 (which is ok)) when using setmtmode and MT. I havent tried it without setmtmode or MT..

Glad to hear you and manao are on the case - im sure a workaround or fix wont be long! :-)

Cheers,
Bitey

[Manao]

I'm not that optimistic. I don't own a smp cpu, and i can barely manage to make MT + masktools crash on my computer ( I've succeeded only thrice, and without getting any useful information on what may be going on ).

Tsp is able to systematically reproduce the bug on his computer, so I can give him some custom builds to try, but in any case it'll be tedious, and long, for both of us.

Tsp : I've reviewed this thread entirely, and two nice threads it linked to, which concerned the changes you made on avisynth, and the ugly MakeWritable issue.

Masktools > v2.0a18 might actually suffer from that same issue ( I stumbled on it the before, found a quick and dirty hack without knowing what was really the problem, but now with the nice explanation of IanB and you, I get what's happening, why my hack worked ).

Here is how I get the frames my filters will be needing :// First, the current frame, processed differently since most filters work in place
PVideoFrame dst = T::is_in_place() ? childs[0]->GetFrame(n, env) : env->NewVideoFrame(vi);

if ( T::is_in_place() )
env->MakeWritable( &dst );

// request all plane pointers, through GetWritePtr

// then, the other source frames, all read only
std::vector<PVideoFrame> frames;
for ( int i = 0; i < k; i++ )
frames.push_back(childs[f(i)]->GetFrame(g(i), env));

// request all plane pointers, through GetReadPtr

// do the processing

frames.clear();

return dst;All masktools filters work alike. f(i) and g(i) say that the ith frame get the g(i)th frame from the f(i)th child.

I don't see anything obviously wrong with the code above.

So, what I'll do is make a "full debug log" build of the masktools, one that in particular checks each plane pointers that are given to the filters. I'll put some DebugPrintf() around the parts that strongly arise suspicion ( especially the T::filter_signature() one ). Whenever you ready, I can send it to you. I expect a huge debug ouput, but hopefully, only the few last hundred lines should be useful.

So feel free to bug me whenever you have the time to do so. The debug version is available here : http://manao4.free.fr/mt_masktools.dll

At all others with smp computers, and willing to help. You can also try that version. Just download the debug version, and DebugView.exe ( first hit on google ), launch it, then virtual dub, then load the faulty script and play it. When it crashes, the DebugView windows will have some information on it. You can send it to me ( manao <at> melix <dot> net, you don't necessarily need to send all, but if you don't send all, select in priority the end of the log ).

[vanessam]

So feel free to bug me whenever you have the time to do so. The debug version is available here : http://manao4.free.fr/mt_masktools.dll.

Zoom player crashed with this dll. it closed alone.

[swaaye]

swaaye:
if you want to try I will sugest this script:

Setmtmode(2)
MPEG2Source("R:\DVD\TNGS6D6\VTS_02_PGC4\TNGS6D6.d2v", idct=7)
tfm(d2v="R:\DVD\TNGS6D6\VTS_02_PGC4\TNGS6D6.d2v")
tdecimate(hybrid=1)
autocrop(mode=0,wmultof=4,hmultof=4,samples=10, aspect=-1,threshold=34,samplestartframe=100,leftadd=0,rightadd=0,topadd=0,bottomadd=0)
BicubicResize(640,480,0,0.75)

with mt 0.5 and the included avisynth.dll.

I tried it out and it did hold the CPU at nearly 100% the whole time. Only gave maybe a few % gain in speed though. I'd say it was pratically equally as fast as without it. Very interesting plugin though and certainly useful in the right situation.

[Manao]

OK, thanks a lot Boulder for the debug log.

Here is another dll : http://manao4.free.fr/mt_masktools2.dll

There no guarantee at all that it shouldn't crash, but at least the debug log he sent me pinpoint without a doubt where the crash was occuring. So if that build doesn't work, I still might be able to find the bug.

[vanessam]

OK, thanks a lot Boulder for the debug log.

Here is another dll : http://manao4.free.fr/mt_masktools2.dll

There no guarantee at all that it shouldn't crash, but at least the debug log he sent me pinpoint without a doubt where the crash was occuring. So if that build doesn't work, I still might be able to find the bug.

Yeah !!
No crash !
Thank's
I tested it tonight

[Boulder]

Playing the avs script doesn't crash but when I close the file in VDubMod, the program vanishes after a while. I don't know if it's actually MT that causes it though. Maybe tsp could also provide a debug build so that could be double-checked.

[Manao]

Ok. Well, after all, it was - as always - a silly mistake. I was compiling the masktools with the single thread windows library. What amazes me is that pre a18 were also compiled in that fashion, but didn't crash.

Anyway, I'll clean the mess I've done with the debug log, fix the YUY2 crash, add another of tsp's optimization for the median computation, and release a new, proper version.

Thanks a lot Boulder and vanessam for the help.

[Boulder]

A new item that doesn't like SetMTMode(2) is Cedocida, the open source DV codec. See the sticky thread in the DV forum for its own thread if you're interested. Using SetMTMode(2) produces weird artifacts which disappear when you use for example ffdshow for decoding DV content.

[tsp]

Boulder:
Does it help to use something like this:

SetMTMode(5)
Avisource("c:\foo.avi")
SetMTMode(2)
filter()
filterMore()

[Boulder]

I couldn't reproduce the problem with SetMTMode(2) again but I now started encoding a long DV clip with SetMTMode(5).AVISource().SetMTMode(2) and see if there are any artifacts. When they appeared, they looked like the ones you get with a broken stream although they were not green as usually.

[Boulder]

OK, I didn't see any artifacts in the one encoded with SetMTMode(5) in the beginning so that's what I'm going to use from now on - in all of my scripts just to be safe ;)

[J-Wo]

Just wondering but has there been any further development on this wonderful filter just to iron out some of the bugs? Thanks!

[tsp]

Due to lack of spare time I did not worked very much on MT but in the next couple of days I will get a dualcore opteron 165 so that should speed up development. Most of the "bugs" is incompatiple filters though and it is difficult for me to fix that.

[aberforthsgoat]

I'm almost positive that I've already seen some posts about this, but I can't find them:

When I run this filter I get a thin hoizontal line across the middle of my screen:

SetMTMode(2)
MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,Smode=3,strength=60,overshoot=7)")

When I run it without MT, as follows, I don't get a line:

LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,Smode=3,strength=60,overshoot=7)

Can someone help me figure out where it's coming from? It's pretty subtle and isn't a big bother - but I'd still like to get rid of it.

Thanks!

Mike

P.S. Did I mention how awsome this thing is? I have a 2.8 Northwood, and I can't run LMSF + a denoiser in realtime without major league stuttering - until I put MT in the mix. Wow.

[foxyshadis]

You shouldn't mix MT() with mt-enabled avisynth. Instead, change SetMTMode until you get to a mode that works for the filter; each step higher is more compatible.

MT is more of a workaround if you can't install mt-avisynth.

(Since I run 2.5.7-pre, I should try MT, that might've sped up my last 18-hour filtering...)

Edit: TSP, have you considered using Tritical's avisynth releases to base your filter on? They include some nice patches that aren't in cvs (such as lifting autoload restrictions, \n in subtitle, and a bunch of others I forgot about). That would be most excellent. =D

[shpitz]

You shouldn't mix MT() with mt-enabled avisynth. Instead, change SetMTMode until you get to a mode that works for the filter; each step higher is more compatible.

MT is more of a workaround if you can't install mt-avisynth.

(Since I run 2.5.7-pre, I should try MT, that might've sped up my last 18-hour filtering...)


so 2.5.7 has no MT capability?

[aberforthsgoat]

You shouldn't mix MT() with mt-enabled avisynth. Instead, change SetMTMode until you get to a mode that works for the filter; each step higher is more compatible.


Hmm. No. I must be doing something wrong here.

When I try to drop the MT() part and just use the SetMTMode command, I loose my multithreading. CPU two maxes out while CPU one stays stuck very low.

However, I have realized that I can run MT() without the SetMTMode line. Multithreading comes back to normal. However, the horizontal line also remains in the screen.

Could it be an installaton error? I reinstalled avisynth 2.5.6, copied the modified file into my system32 directory and copied in the MT plugin - all to no avail. Same as before.

Bother.

Mike

P.S. Could it be that I installed the wrong version of avisynth 2.5.6? There seems to be a 2.5.6a and a 2.5.6 RC2 on the avisynth site.

[foxyshadis]

so 2.5.7 has no MT capability?
I doubt it will unless it works for all filters. 2.5.7 is supposed to mostly be a bugfix and cleanup update, 2.6 is where the main features go. But if it's comptible enough, who knows.

When I try to drop the MT() part and just use the SetMTMode command, I loose my multithreading.
Odd. Hope tsp can help, but LSF is a pretty complex function. Not surprising that MT fails on it, since it doesn't have good border handling, so try overlap=4 or so. That should prevent the artifacts.

[aberforthsgoat]

Odd. Hope tsp can help, but LSF is a pretty complex function. Not surprising that MT fails on it, since it doesn't have good border handling, so try overlap=4 or so. That should prevent the artifacts.

Hmm. Well, I removed every lst plugin from my plugin folder and downloaded all new stuff, and the line is gone. So I must have a had a funky version of mt_tools or removegrain in there.

However, the SetMTMode command still won't do it for me. MT() is working nicely though.

Thanks for the tips!

Mike

[tsp]

You shouldn't mix MT() with mt-enabled avisynth. Instead, change SetMTMode until you get to a mode that works for the filter; each step higher is more compatible.

the latest version of mt 0.5 requieres my custom avisynth.dll that also includes setmtmode.


MT is more of a workaround if you can't install mt-avisynth.

I would more say that mt works better with some filters like fft3dfilter while setmtmode is better suiter for others like deinterlacers. Also using mt without my custom avisynth.dll would slow down the filter so it is nearly the same speed as running without mt (that was the reason I requires the custum avisynth.dll)


(Since I run 2.5.7-pre, I should try MT, that might've sped up my last 18-hour filtering...)

Edit: TSP, have you considered using Tritical's avisynth releases to base your filter on? They include some nice patches that aren't in cvs (such as lifting autoload restrictions, \n in subtitle, and a bunch of others I forgot about). That would be most excellent. =D
maybe

[foxyshadis]

Oh! That makes sense, sorry about that.

[Mr.Bitey]

aberforthsgoat,

Ive found that line to be a problem between versions of MT and masktools - I cant tell you which combination causes it - only that ive fixed it previously on my system by the same method (re-installing versions of each) :)

Cheers,
Bitey

[aberforthsgoat]

Ive found that line to be a problem between versions of MT and masktools - I cant tell you which combination causes it - only that ive fixed it previously on my system by the same method (re-installing versions of each) :)

Odd. Very odd. BTW, are you able to run LimitedSharpenFaster with the SetMTMode thing or do you MT() it?

Mike

[Mr.Bitey]

I do both...

SetMTMode at the beginning of the avisynth script, then MT with hqdn3d and limitedsharpen faster inside the call to MT..

Cheers,
Bitey

[tsp]

aberforthsgoat : are you using version 0.5 of mt and the included avisynth.dll (and copied to the c:\windows\system32 directory)? And please when using both setmtmode and mt inside the same script try testing of it is actually faster(that is fps not cpu utilization) than not using both because there are some overhead by using both filters together.

[Mr.Bitey]

TSP,

I didnt realise you wernt supposed to use setmtmode and mt() - there is a large thread on avsforums about using Limitedsharpenfaster and its become 'assumed' (incorrectly by the sound of it) that people should use both.

So people should use either MT(lsf) or SetMTmode() and not both?

Cheers,
Bitey

[aberforthsgoat]

aberforthsgoat : are you using version 0.5 of mt and the included avisynth.dll (and copied to the c:\windows\system32 directory)? And please when using both setmtmode and mt inside the same script try testing of it is actually faster(that is fps not cpu utilization) than not using both because there are some overhead by using both filters together.

Thanks for the response!

I *do* have the whole 0.5 package installed - I'm dead sure of that. But as far as I can tell, I mt() is making a big difference, but setmtmode is making no difference at all in my scripts.

BTW, it's hard to get much of a read on the fps because the OSD flickers constantly between two numbers - I think I'm seeing 21 and 32. With mt() I get a steady flicker between the two. With setmtmode(2) by itself I seems to stick on the 21 more - and occasionally dip all the way down to 9 or 16 or the like.

Peace,

Mike

[foxyshadis]

According to the first page, there's no SetMTMode(0), so something like this to reduce the overhead isn't possible, am I correct?

SetMTMode(3)
Filter()
SetMTMode(0)
MT("Filter2()")
SetMTMode(2)
...

aber, try avstimer to get accurate readings on the speed on scripts (although I haven't read through this thread to see if it's threadsafe), or just time a complete null render with and without.

[Boulder]

It used to be possible to disable multithreading where needed by calling SetMTMode(0), don't know if it's been removed.

[tsp]

Mr.Bitey: I don't say you shouldn't use setmtmode with mt. I just say that sometimes it's slower to use both together and that you should always check if the framerate is faster with both setmtmode and mt compaired to using mt alone. Another thing the SetMTmode should be the first line in the script before import("") and this script disables SetMTMode completely because only mode 5 is used:

#note setmode before import
SetMTMode(5)
Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpenFaster.avs")
MT("HQDN3D(1)")
LanczosResize(1440x960)
MT("LimitedSharpenFaster ss_x=1.0,ss_y=1.0,Smode=3,strength=100,overshoot=7)")


aberforthsgoat: try using virtualdub and take time on how long it takes to process the script.

foxyshadis : There are no mode 0. Use mode 5 or 6 instead. mt 0.5 internaly calls SetMTMode(5) and restores the old mode afterwards so you don't have to add SetMTmode(5) before mt and restore the old mode after.

[Mr.Bitey]

TSP: Thanks for the clarification. I think i'll go revisit my scripts ;-) I do recall SetMTMode(5) seemed to run quicker when used with MT() in my Limitedsharpenfaster script.. Its going to be a long night I think :) You also mention SetMTMode() should be the first line - so if im using a .avsi named version of limitedsharpen - I should rename it to .avs and manually include?

Cheers,
Bitey

[tsp]

Mr.Bitey
one way to test if it is necesary to rename the avsi is to add this after the first SetMTMode:

subtitle(string(GetMTMode)))

it should show the current mode. If it is larger than zero the SetMTMode works correct.

[Kador]

do you have plans to integrate your avisynth.dll in the mail avisynth development (dir the 2.5.7 final for example) ?

[tsp]

kador it will be in the 2.6 release. The changes are too big to be included in the 2.5 version.

[Serbianboss]

Today i tested with version 0.5.

Firstly i copy MT.dll in avisynth plugin directory and copy avisynth.dll in system32.

I am using A64 X2 3800+, capturing to DV avi and than with avisynth and CCE i encode to mpeg2 (720x576)

And always use Convolution 3d filter (avisynth filter).

I tested on 15 minute material and took 36 minute in 2-pass CBR in CCE without crash.


This is code:

SetMTMode(2)
LoadPlugin("Convolution3d.dll")
avisource("D:\CAPTURE\bmw.avi")
ConvertToYuY2(interlaced=true)
SeparateFields()
odd=SelectOdd.Convolution3D (1, 32, 128, 16, 64, 10, 0)
evn=SelectEven.Convolution3D (1, 32, 128, 16, 64, 10, 0)
Interleave(evn,odd)
Weave()
crop(8,4,-8,-12)
AddBorders(8,8,8,8)



With SetMTMode(2) i get 50% speed up in CCE:D

So i am interesting is OK just to put SetMTMode(2) at begin?

p.s Congratulations to your multithreaded filter.

[Serbianboss]

Now i have one error.

When encoding begin, few minutes after it sais:

Mux video buffer overflow.

http://img412.imageshack.us/img412/774/untitled6mb.jpg (http://imageshack.us)


When i use Mainconcept encoder i get this error

http://img327.imageshack.us/img327/1125/untitled9mh.th.jpg (http://img327.imageshack.us/my.php?image=untitled9mh.jpg)

When i try without scrip everything is OK. What can be this?

best regard

[Serbianboss]

I figure what was problem.

Problem was in CCE. When i use option CBR-system then show me "Mux video buffer overflow". In other case everything as OK.

New results: I encoded 2 hours movie in 2-pass without crach using MT 0.5. It was 40% faster.

TSP, excellent filter.

[tsp]

Serbianboss: It's okay to use that script with setmtmode(2) in the first line. Try watching a minute or so for artifacts that might appear if the filters used is not thread safe.
Good you figured out the error with CCE.

[Serbianboss]

What kind of artifacts you mean?

I encoded two hours and i thing that there no have artifacts.

[Boulder]

Compare the output in VirtualDub, one script with SetMTMode and one without it.

I've noticed that sometimes MVTools-related functions require SetMTMode(5), with SetMTMode(2) there are occasional artifacts - they appear as white/grey horizontal stripes.

[Serbianboss]

I didnt see any artifacts in virtual dub (tested on minute source).
But when in virtual dub go left or right with slider, virtual dub has cloused.

One more thing: With SetMTMode (5) i dont have any increase in speed.

[tsp]

that is because setmtmode(5) disables multithreading for the filters below it.
The artifacts is different from filter to filter. Like using blur and sharpen in the same filter using an older version of avisynth.dll caused excessive blur or sharpen in some of the frames.

[Boulder]

Is the documentation incorrect then? It says that modes 1-6 are available and that mode 5 is a rather safe one.

[tsp]

it is rather safe because only one thread is used so it makes no sense only to use mode 5 alone but it should be used for only part of the script that doesn't work with lower modes like mvtools.

[Serbianboss]

Using SetMTMode (2) with Convolution 3D filter dosnt have any artifact.

I always work with DV avi files.

I will test with more material.

[egandt]

I've been playing with Limitedsharpenfaster within ffdshow for a few days now and the results are very good, now I have a dual Core system (AMD 4600+), and I want to spread around the load, since I also am using the hqdn3d filter.
Since its dual core I wanted to spread the load, using the MT plugin, Ive tried both
SetMTmode(2) and MT("string") modes and I've found that the CPU load on th first core (the one used by everything), oes up by about 12% when I turn on MT, on the second core the CPU goes from 12% to 33%, so it is distrubiting the load, but why am I seeing no improvement on the first core (core 0 CPU normally is about 72% and goes to 85% when MT is on).

Is this normal?
ERIC

[Serbianboss]

I have little problem. I encoded 2 hour(vhs capture) in CCE, 2 pass VBR, and first pass was ok and second pass at 97% cpu usage fall at 50% and encoding was stop.

I am using this script:

SetMTMode(2)
LoadPlugin("Convolution3d.dll")
avisource("C:\Documents and Settings\Nenad\Desktop\Video 1.avi")
ConvertToYuY2(interlaced=true)
SeparateFields()
odd=SelectOdd.Convolution3D (1, 32, 128, 16, 64, 10, 0)
evn=SelectEven.Convolution3D (1, 32, 128, 16, 64, 10, 0)
Interleave(evn,odd)
Weave()
crop(8,4,-8,-12)
AddBorders(8,8,8,8)

What can be problem? Everything was OK and at the end, CPU ussage fall to 50% and CCE stopped.

best regard

[Boulder]

Would it be possible to have an Avisynth build based on tritical's build? His build includes some memory management tweaks which seem to reduce memory consumption quite nicely. Or are you planning on releasing a build based on the latest CVS code?

[tsp]

You mean the 2.5.7 branch or 2.6? Is triticals code included in that? I could try make a build based on his code.

[Boulder]

His build is based on the 2.5.7-branch, it's here http://www.missouri.edu/~kes25c/#c3. The modified sources are there too.

By the way, when using your build, the plugin autoloading doesn't work with some (not sure if with all) plugins. Last time I had to use LoadPlugin to get DGDecode (the latest beta) and ColorMatrix loaded even though they were in the plugins folder.

[J-Wo]

hey tsp, any update on the status of this filter? Do you know if it will be included in a future brance of Avisynth? Have to say I love this filter for the speed benefits (Opteron 165 here), but often find it causes CCE to crash...

[tsp]

the modification to avisynth will be included in 2.6 but mt.dll(that contains the filter mt) will be included as a seperate filter.
reason why CCE might crash with mt
* the filter used with mt isn't threadsafe
* the extra work cause your overclocked opteron to overheat after a while (you DO overclock it right? I know I do)
* A bug in avisynth or mt

Currently I'm working on fft3dgpu and creating a helper dll to AvsP and I'm missing a TODO list for mt (might be creating a better doc)

[Jeremy Duncan]

tsp,

Are you making a new Avisynth.dll to use with MT, and leaving MT unchanged ? Or are you updating the MT.dll too ?

Will it be compatible with Avisynth 2.5.7 ?

[tsp]

when avisynth 2.5.7 goes final I will try to merge the multithreading stuff into it and release a new avisynth.dll (the official 2.5.7 will not be compatible with mt.dll).
If I make some changes to the mt() filter I will release a new version of mt.dll

[tomos]

thank you for this. !

just messing around with my new core2duo pc and when using EEDI2, my CPU usage never went above 60%. with this, went straight to 99%!

thanks to OP :D

[Teebeeke]

I updated thefilmmachine (TFM), and now CCE crashes everytime i add "SetMTMode(2)" in teh avisynth script. CCE doesnt crash when i don't put it in there.
I then copied MT.DLL into avisynth2.5 dir, and avisynth.dll nto xp/system32 dir after i updated TFM, but it still crashes.

Which avisynth should i install for now ?

[tsp]

you could post the avisynth script so I can see what filters are used. Probable the new vesion of thefilmmachone uses new filter and/or new version of existing filters.

[Teebeeke]

# 16:9 encoding
SetMTMode(2)
AviSource("c:\movie.avi", false)
ConvertToYUY2()
FadeIn(50)
Lanczos4Resize(720,456,0.0,0.6)
AddBorders(0,12,0,12)

[tsp]

hmm what version of cce do you use. How fast does CCE crash (instant or after a while). Does playing the aviscript in windows media player cause a crash

[Backwoods]

Any version of CCE freezes the encode for me at different times. Usually after some time, ex: 40min to 1hr 10min. Then there will be times a 3hr encode will work fine. All with the same script, odd behavior. The script can be as simple as the one posted above too.

AMD Athlon 64 X2 Dual 4400+

[Teebeeke]

hmm what version of cce do you use. How fast does CCE crash (instant or after a while). Does playing the aviscript in windows media player cause a crash

1) I use CCE SP 2.67
2) It crashes random, sometimes after 10% , sometimes after 90%
3) WMP, Bsplayer can open and play the .avs correctly.

When i remove SetMTMode(2) it converts the movies/series everytime. It used to work fine until i upgraded TFM.

E6600 | 2 Gb DDR2

[Alizar]

Could someone post a mirror for the MT version of avisynth since avisynth.org is down? I'd really like to try this.

[tsp]

http://www.tsp.person.dk/MT_05.zip

[tomos]

thank you :)

[Alizar]

Woot! Thank You!

[boombastic]

Hi!I've got a E6600 dual core CPU,i'm using latest tritical version of avisynth and mt.dll is placed into the autoload plugin folder but id i open this script into virtualdubmod it says there's no SetMtMode function:

SetMTmode(5)
MPEG2Source("H:\NY1\VIDEO_TS\VTS_01_1.d2v")
import("f:\Programmi\AviSynth 2.5\plugins\SeeSaw.avs")
#SeparateFields()
#a=SelectEven().FFT3DGPU().SeeSaw(NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)
#b=SelectOdd().FFT3DGPU().SeeSaw(NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)
#weave()
TomsMoComp(1,3,1)
ColorMatrix(d2v="H:\NY1\VIDEO_TS\VTS_01_1.d2v")
Crop(8,80,-32,-80)
LumaYV12(0,0.9)

i'm also using the mt_masktools2 dll linked some post above.
Where am i wrong?

[Boulder]

You need to use tsp's Avisynth build included in the package.

[boombastic]

Infact now it works!I tought that the latest versin of avisynth could handle this function natively.

[barbapapa5800]

when i am trying to let diko convert a movie from avi to mpeg the load get's spread, but there is no increase of load.
my specs:
amd x2 3800+ @ 4600+ (2.4 ghz)
2gig ram

the script i am using is:

SetMTmode(2)
converttoyv12()
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
Deen("c2d",2,4,6,4,6,0.5,9,"")
asharp(1,4)
UnDot()
DivXResize(WIDTH, HEIGHT, 0, "BicubicResize",WIDESCREEN)
Blockbuster(method="noise",detail_min=1,detail_max=10,variance=0.3,seed=2)
DivXBorders(HEIGHT,OVERSCAN)
AddAudio()
Subtitle("@ SKVCD.NL - CREW @", 130, 450, 75, 150, font="verdana", size=18, text_color=$ffFFFF)



the avs file that diko makes:

#########################
# DIKO Generated Script #
#########################
#
# Loading plugins and functions... #
####################################
#
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\asharp.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\atc.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\AutoCrop.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\avsmon25a.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\blockbuster.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\Convolution3DYV12.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\DctFilter.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\deen.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\MPEG2Dec3.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\MT.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\Sampler.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\STMedianFilter.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\UnDot.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\UnFilter.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\VSFilter.dll")
Import("C:\Program Files\DIKO\avisynth plugins\addaudio.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\fmin.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\jdl-interlace.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\jdl-util.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\resize.avsi")
#
# Script begins here... #
#########################
#
Avisource("C:\Program Files\diko\Temp\solaris.dvdrip.xvid-deity.avi",false)
AssumeFPS(25)
WIDESCREEN=0
Overscan=1
source_widescreen=1
WIDTH=720
HEIGHT=576
INTERLACED=false
SetMTmode(2)
converttoyv12()
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
Deen("c2d",2,4,6,4,6,0.5,9,"")
asharp(1,4)
UnDot()
DivXResize(WIDTH, HEIGHT, 0, "BicubicResize",WIDESCREEN)
Blockbuster(method="noise",detail_min=1,detail_max=10,variance=0.3,seed=2)
DivXBorders(HEIGHT,OVERSCAN)
AddAudio()
Subtitle("@ SKVCD.NL - CREW @", 130, 450, 75, 150, font="verdana", size=18, text_color=$ffFFFF)letterbox(8,8,8,8)
TextSub("C:\Program Files\diko\Temp\movie0.srt")
MonitorFilter


i placed the new avisynth.dll in windows/system32 and the MT.dll in the diko/avisynth scripts.

am i forgetting something?

[tsp]

you need to place SetMTMode before the first filter in this case it should appear before import(or at least before avisource). To be safe try placing at the first line in the script

[Bh4i]

Problem: i cant get this work in DVD-Rebuilder Pro :(
When i remove the SetMTMode line, the speed is even faster...

When using this script in MeGUI it gives me 100% CPU + ~30-35 fps speed.


LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools v2.0.a30\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrain\RemoveGrainSSE3.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avs")
SetMTMode(2,0)
DGDecode_mpeg2source("D:\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)
TDeint()
crop( 0, 54, -16, -58)
removegrain(mode=2)
LanczosResize(640,272)
LimitedSharpenFaster(Smode=4,strength=70,wide=true,lmode=3,ss_x=1.25,ss_y=1.25)

And i added these lines to DVD-Rebuilder:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools v2.0.a30\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrain\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avs")
SetMTMode(2,0)
removegrain(mode=2)
LimitedSharpenFaster(Smode=4,strength=70,wide=true,lmode=3,ss_x=1.25,ss_y=1.25)

DVD-Rebuilder gives me speed of ~0.80 and uses only ~55-60% CPU :(

I have "multiple encoder processes" enabled..

My PC: Core2Duo E6600 @2.4 Ghz, 2 GB RAM

[canuckerfan]

I've got this script here and I'm wondering which functions would it be safe to use MT mode 2? It would be nice to speed things up.

LoadPlugin("g:\Kartick's Stuff\Tools\dvd editors\dgmpgdec149b3\DGDecode.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\DeHalo_alpha_mt.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpenFaster.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\RemoveDirt.avs")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\ColorMatrix.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\EEDI2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\TDeint.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\DeSpot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MaskTools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\SSE3Tools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mvtools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\fft3dfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RepairSSE3.dll")

mpeg2source("g:\Hindi Movies\Dil.Chahta.Hai.2001.DVD9-bj1988\encode_stuff\raw files\VideoFile.d2v",idct=5,info=3)

Setmemorymax(512)

AssumeTFF()
Interp = SeparateFields().SelectEven().EEDI2(field=1)
Deinted=TDeint(order=1,field=1,edeint=Interp)
TFM(mode=6,order=1,PP=7,slow=2,mChroma=true,Clip2=Deinted)
TDecimate(mode=1)

ColorMatrix(mode="Rec.601->Rec.709",hints=true)

Tweak(sat=1.15,cont=1.12,bright=8)

DeSpot(pwidth=65,pheight=65,p1=35,p2=14,mthres=25)
RemoveDirt()

FFT3DFilter(sigma=2.2,ow=48/2,oh=48/2)
Dehalo_alpha()
LimitedSharpenFaster(Smode=4,Lmode=2,overshoot=5,strength=225)

Crop(0,48,0,-48)
AddBorders(0,48,0,48)

ConvertToYUY2()

[foxyshadis]

Problem: i cant get this work in DVD-Rebuilder Pro :(
When i remove the SetMTMode line, the speed is even faster...

When using this script in MeGUI it gives me 100% CPU + ~30-35 fps speed.
....
And i added these lines to DVD-Rebuilder:
....
DVD-Rebuilder gives me speed of ~0.80 and uses only ~55-60% CPU :(

I have "multiple encoder processes" enabled..

My PC: Core2Duo E6600 @2.4 Ghz, 2 GB RAM

a. If you're already running multiple processes, you won't see any gain by further multithreading the operation. Two processes encoding different chunks of video is about as parallel as you can possibly get. Using far more processing threads than cores guarantees contention (=slowdown).

b. As tsp just told barbapapa, the setmtmode has to come before the source filter, in yours it doesn't. If DVDRB just tacks the source filter to the top of the script, you'll have to find some way to work around that. (Multiple processes is the better solution, when available and well-implemented, than multithreading though.)

[tsp]

I've got this script here and I'm wondering which functions would it be safe to use MT mode 2? It would be nice to speed things up.



You could try mt instead. It works better with fft3dfilter

something like this
LoadPlugin("g:\Kartick's Stuff\Tools\dvd editors\dgmpgdec149b3\DGDecode.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\DeHalo_alpha_mt.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpenFaster.avs")
Import("C:\Program Files\AviSynth 2.5\plugins\RemoveDirt.avs")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\ColorMatrix.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\EEDI2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\TDeint.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\TIVTC.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\DeSpot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MaskTools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\SSE3Tools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mvtools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\fft3dfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RepairSSE3.dll")

function f(clip c)
{
c
ColorMatrix(mode="Rec.601->Rec.709",hints=true)

Tweak(sat=1.15,cont=1.12,bright=8)

DeSpot(pwidth=65,pheight=65,p1=35,p2=14,mthres=25)
RemoveDirt()

FFT3DFilter(sigma=2.2,ow=48/2,oh=48/2)
Dehalo_alpha()
LimitedSharpenFaster(Smode=4,Lmode=2,overshoot=5,strength=225)
}

mpeg2source("g:\Hindi Movies\Dil.Chahta.Hai.2001.DVD9-bj1988\encode_stuff\raw files\VideoFile.d2v",idct=5,info=3)

Setmemorymax(512)

AssumeTFF()
Interp = SeparateFields().SelectEven().EEDI2(field=1)
Deinted=TDeint(order=1,field=1,edeint=Interp)
TFM(mode=6,order=1,PP=7,slow=2,mChroma=true,Clip2=Deinted)
TDecimate(mode=1)
mt("f()")

Crop(0,48,0,-48)
AddBorders(0,48,0,48)

ConvertToYUY2()

Bh4i:
try using mt around limitedsharpen instead of setmtmode and see if it is faster

[canuckerfan]

thanks for the suggestion, tsp.:)

I am currently using 2.5.7 of avisynth. Are there any modded files for MT of that version yet? Or will I have to revert to 2.5.6 to make use of mt?

EDIT: nevermind, I read the previous page. So I'll have to revert back, right?

[barbapapa5800]

you need to place SetMTMode before the first filter in this case it should appear before import(or at least before avisource). To be safe try placing at the first line in the script

the problem is that diko autogenerates the avs file, so i can't make changes into the avs.
al least i don't think so...

[tsp]

canuckerfan: Yes but take a backup so you can revert back.

barbapapa5800: You can try to cheat diko by adding SetMTmode to the first line of the file C:\Program Files\DIKO\avisynth plugins\addaudio.avsi as this is the first file imported in your script file

[barbapapa5800]

ok, i did what you say'd.

the conversion starts with the cpu at full speed on both cores, but after a second or 10 CCE crashes.
i use cce 2.70.02.06

here is the movie0.avs file that diko generated.
the strange thing is that i can't find any traces of setmtmode in it.




#########################
# DIKO Generated Script #
#########################
#
# Loading plugins and functions... #
####################################
#
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\asharp.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\atc.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\AutoCrop.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\avsmon25a.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\blockbuster.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\Convolution3DYV12.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\DctFilter.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\deen.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\MPEG2Dec3.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\MT.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\Sampler.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\STMedianFilter.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\UnDot.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\UnFilter.dll")
LoadPlugin("C:\Program Files\DIKO\avisynth plugins\VSFilter.dll")
Import("C:\Program Files\DIKO\avisynth plugins\addaudio.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\fmin.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\jdl-interlace.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\jdl-util.avsi")
Import("C:\Program Files\DIKO\avisynth plugins\resize.avsi")
#
# Script begins here... #
#########################
#
Avisource("C:\Program Files\DIKO\Temp\You Me and Dupree.avi",false)
AssumeFPS(23.976)
WIDESCREEN=0
Overscan=1
source_widescreen=1
WIDTH=480
HEIGHT=480
INTERLACED=false
converttoyv12()
Blockbuster(method="noise",detail_min=1,detail_max=3,variance=0.1,seed=1)
Deen("c2d",2,4,6,4,6,0.5,9,"")
asharp(1,4)
UnDot()
DivXResize(WIDTH, HEIGHT, 0, "BicubicResize",WIDESCREEN)
Blockbuster(method="noise",detail_min=1,detail_max=10,variance=0.3,seed=2)
DivXBorders(HEIGHT,OVERSCAN)
AddAudio()
Subtitle("@ SKVCD.NL - CREW @", 130, 450, 75, 150, font="verdana", size=18, text_color=$ffFFFF)letterbox(8,8,8,8)
TextSub("C:\Program Files\diko\Temp\movie0.srt")
MonitorFilter
Converttoyv12()

[tsp]

did you add setmtmode to addaudio.avsi? If that is the case the setmtmode is read at the line Import("C:\Program Files\DIKO\avisynth plugins\addaudio.avsi")
So one of the filters is not compatible with setmtmode(2). Try inserting SetMTMode(5) after Avisource. If this works move the setmtmode(5) line below the next line(assumefps) and if this works continue moving setmtmode(5) down until it crashes and report back what filter caused it.

[barbapapa5800]

when i put the setmtmode(2) at the last line of the resize.avsi the rip goes great.
only thing is that i don't have the subs in it, but that wil my a mistake of mine i hope.
i will do some more testing and hopefully there are subs in it :)

ow yeah, to find out where to put the setmtmode(2) i used avsp.
more info here (http://forum.doom9.org/showthread.php?p=871134#post871134)

when i put the setmtmode(2) line to high in the script avsp crashed, this is how i knew verry fast where to put it :sly:

[tsp]

barbapapa5800: just curious is it faster placing setmtmode(2) in resize.avsi than not using setmtmode at all?

[barbapapa5800]

a normal rip takes about 2 hours, with setmtmode(2) it takes only 1u20min.
so it is faster.
the only thing that's a problem is that there arn't any subs.
without setmtmode(2) this isn't a problem.
is this normal?

[tsp]

umm not the subtitles should appear. Try placing the line SetMTMode(5) just before textsub in the script. This disables multithreading for textsub

[Mug Funky]

hmm. i was just thinking it would be cool to have a multithreading mode that allows you to assign a processor per clip.

for example:

genericsource("blah.foo")

global clip1 = last.onefunction().useproc(1)
global clip2 = last.anotherfunction().useproc(2)

scriptclip(last," somecondition==true? clip1 : clip2 ")

this may not make much sense immediately... basically i'd like to be able to filter clip1 using processor 1, and clip2 using processor 2, and have the conditional filter just output whichever one is needed.

my specific case is choosing between FILM, 30p and 60i and wanting to perform a different PAL conversion to each case using a different thread for each. is this possible? it might speed things up a bit...

[barbapapa5800]

umm not the subtitles should appear. Try placing the line SetMTMode(5) just before textsub in the script. This disables multithreading for textsub

doing that makes an mpg of about 300mb and the rip is finfished after 12 min.
when i try to playback the mpg it say's that there is an error in line 52.
when i remove SetMTMode(5) the rip goes at normal speed with setmtmode(2), so faster then without setmtmode.

the difference is that setmtmode is load with an avsi file and that setmtmode is entered in the script.

[foxyshadis]

TextSub is the very last line of your script, and SetMTMode(5) was just before that, right? That's the way it's normally used, if you have something else at the end you can put SetMTMode(2) after TextSub as well. All mode 2 will always be faster than bits of mode 5 here and there, but if it needs mode 5 to actually work, well...

But I don't understand this line:
when i try to playback the mpg it say's that there is an error in line 52.
You mean opening up the mpg in a player or vd-mpeg2 gives you a script error? That makes no sense.

[barbapapa5800]

there is only sound when i play the mpeg in powerdvd.
in the picture is a red line that say's that there is en error in script at line 52.
i am giving it a try right now again.

i am sorry, but i think i am giving up on it for now.
maybe i'll try later...

thnx

[shpitz]

hmm. i was just thinking it would be cool to have a multithreading mode that allows you to assign a processor per clip.


that's a great idea, kinda like processor affinity in the task manager.

i know there is a util by MS called imagecfg that modifies an exe file to work only on a particular cpu, so in theory you can have 2 vdubs each forced to use only 1 cpu. i'm not sure if that will work...

[tsp]

hmm. i was just thinking it would be cool to have a multithreading mode that allows you to assign a processor per clip.

for example:

genericsource("blah.foo")

global clip1 = last.onefunction().useproc(1)
global clip2 = last.anotherfunction().useproc(2)

scriptclip(last," somecondition==true? clip1 : clip2 ")

this may not make much sense immediately... basically i'd like to be able to filter clip1 using processor 1, and clip2 using processor 2, and have the conditional filter just output whichever one is needed.

my specific case is choosing between FILM, 30p and 60i and wanting to perform a different PAL conversion to each case using a different thread for each. is this possible? it might speed things up a bit...
in this case you wouldn't get any benefit by using a thread per clip as scriptclip only needs to get a frame from one of the clips depending on "someconditon" so the thread that process the other clip are just waisting time as the frame is not needed. I could see some use for it if used with masktools or stackhorizontal/vertical where frames from both clip are needed.

[Jeremy Duncan]

Will the New Avisynth.dll be ready soon ?
Which Avisynth version is it being made for ?
Will it make Multi threading Faster ?

[Terranigma]

Will the New Avisynth.dll be ready soon ?
Which Avisynth version is it being made for ?
Will it make Multi threading Faster ?

I'd like an answer on this as well. I'm always getting a "Use Avisynth Version 2.6 or greater, or 2.5.7 rc1" to run. I could've sworn the version i'm using is 2.5.7 RC1. I mean I downloaded the latest 3 files posted by (tsp?): TCPdeliver.dll, avisynth.dll, & DirectShowSource.dll, and got the same thing. Maybe someone who actually has version 2.6 or 3.0 could hook me up with the files needed to run a later version? :p

[Boulder]

If you need a recent v2.5.7 build (that's just about as far as you can go stable at the moment), see http://bengal.missouri.edu/~kes25c/

[foxyshadis]

I'd like an answer on this as well. I'm always getting a "Use Avisynth Version 2.6 or greater, or 2.5.7 rc1" to run. I could've sworn the version i'm using is 2.5.7 RC1. I mean I downloaded the latest 3 files posted by (tsp?): TCPdeliver.dll, avisynth.dll, & DirectShowSource.dll, and got the same thing. Maybe someone who actually has version 2.6 or 3.0 could hook me up with the files needed to run a later version? :p

You really don't want 2.6 or 3.0. None of your plugins will even run! Maybe I could put a few up for 2.6.

[Romario]

I am sorry, but what going on with MT AviSynth development? What can I expect from MT?

[tsp]

I'm waiting on the final version of avisynth 2.5.7 before I add the MT code to it.There are still a bug somewhere that causes CCE to crash after a couple of hours also I want to create a interlaced mode for mt() so that there are a thread for each field.

foxyshadis: The current plugins for avisynth 2.5 should work with avisynth 2.6. They should be recompiled to support the new colorspaces of course.
For avisynth 3.0 the current plugins will not work.

[foxyshadis]

Plugins will work without recompiling, or plugins require recompiling first? I thought it was the latter, I can't tell if you're implying the former. If so, that's cool, all the more reason to give it a test install then. (I assume that most users of the usage forum can't recompile, generally.)

[tsp]

I mean the former. Avisynth 2.5 plugins should work without recompile in avisynth 2.6

[IanB]

It is a design goal of 2.6 that 2.5 plugins work without recompile. The user however is responsible for making sure new colour spaces are not pushed thru naughty old filters that incompletely test the colour space.

i.e. If a 2.5 filter only tests for isRGB() and isYV12() and assumes what is left over is YUY2 it will be in for a rude surprise. Such a filter would of course be a bit naughty. The rule (unwritten?) is to exhaustivly test for the colour space your are prepared to accept, this is so we can transparently add new colour spaces (as is happening in 2.6)

[krieger2005]

I have seen in a commercial video-restoration software a method of multithreading. I don't know if you have an equal attempt here so i will describe it because it is so general, that it can be used with every filter.

1. For Processing the Program split the Movie in parts, i guess every 100 Frames long or so
2. Every Part can be processed with the script in an other thread
3. In the end it process some frames on the connections of two parts (because of temporal filtering issue)

Cons are:
1. This attempt is difficult, because it split the movie (so the internal cache-work should be very good or optimized for this type of processing).
2. Further it process some Frames (on connections of two parts) twice.

Do you think this could be an option of MT?

[tsp]

krieger2005: not really. It should be doon in the program that opens the aviscript and not by avisynth as avisynth has no idea which frame the program will request (most of the time it is linear access but it can't be garantied). Tobias already created such a program called ELDER (http://www.funknmary.de/bergdichter/projekte/index.php?page=ELDER) you might want to try this.

[krieger2005]

ELDER seems not to be exactly the same but have the same idea... But after thinking longer about i think you are right. An external program should do this...

But if an external program would do this is MT in Avisynth needed?

PS: For this method one must open the same script twice. So a complex script, which need much RAM would need n-times more memory... Hm... Bad sighns for such a program

[tsp]

krieger2005: no not really.

[Serbianboss]

Just two question.

I tried MT on MVtools script and seems to work good. I test on short clip.

I just put settmmode(2) at the beginning of script. CPU usage is 100%


SetMTMode(2)
source=AVISource("C:\Documents and Settings\Nenad\Desktop\vulkani.avi").convertToYuy2(interlaced=true).Trim(11368,12867)


fields=source.AssumebFF().SeparateFields()

backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = fields.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = fields.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
fields.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=800,idx=1)

fft3dfilter(sigma=1.5)

Weave()

In this script i am using fft3d filter so my question is: is it safe to use setmtmode(2) on fft3d filter. I read that maybe for fft3d is better to work with MT instead with setmtmode.


Second, on this reinterlacing script i have problem. Every time is crach when put setmtmode(2) at beginning. This script i am using with different filters and always with limitedsharpenfaster. So i read that setmtmode(2) is good for LSF but this script is always craching at starting of encoding.

So any suggestion what to change. Here is the script:


setmtmode(2)
LoadPlugin("RemoveGrainSSE3.dll")
LoadPlugin("PeachSmoother")
LoadPlugin("LeakKernelDeint.dll")
LoadPlugin("mt_masktools.dll")
import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avs")


avisource("C:\Documents and Settings\Nenad\Desktop\vulkani.avi")
Trim(11368,12867)
crop(8,4,-8,-12)
ConvertToYuY2(interlaced=true)
AssumeBFF()

LeakKernelBob(Order=0,threshold=1)
PeachSmoother(NoiseReduction = 80, Stability = 25, Spatial = 200)
limitedsharpenfaster(strength=100)

AssumeBFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
AddBorders(8,8,8,8)

[tsp]

Serbianboss:
You can safely use setmtmode(2) with fft3dfilter but it is faster to use mt() with

you can try to see if this script is faster than your:

SetMTMode(2)
source=AVISource("C:\Documents and Settings\Nenad\Desktop\vulkani.avi").convertToYuy2(interlaced=true).Trim(11368,12867)


fields=source.AssumebFF().SeparateFields()

backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = fields.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = fields.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
fields.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=800,idx=1)
Setmtmode(5)
mt("fft3dfilter(sigma=1.5)")

Weave()

mvtools might produce wrong result with setmtmode so be sure to watch the final result for signs of wrong motion compensation (and please report back if you do)

about your second script, it doesn't crash on my computer so if you could post what version of the filters you use. The first line in my limitedsharpenfaster.avs contains this:
LimitedSharpen() ( a modded version, 29 Oct 2005 )

[Boulder]

MVTools is still not multithreading friendly. The idx's get all mixed up which will cause problems sooner or later. I think Fizick is considering turning MVTools into multithread-friendly way but I don't know if he's come up with any ideas yet.

[Serbianboss]

I try your script for MVtools and cpu usage is 53%. Its slower, i dont know why.


mvtools might produce wrong result with setmtmode


Does some users complain about this? I only tested on few minutes of source. It dosnt crash and didnt see anything strange. I will test on 2 hour video to see does have some artifacts or no(if succeed to encode to the end).

For second script- also used limitedsharpen faster (modded version, 29 Oct 2005), 2.5.6 avisynth, new mt_masktools, MT 0.5

That script sometimes work good sometimes work bad. For example now at moment its encode ok one minute test video clip. And when test again that script fall. So its not reliable.

Can you suggest parameters for second script.

@Boulder

What if we dosnt use idx parameter, maybe be better.

[Boulder]

It might or might not work. You could use a metrics comparison (SSIM, PSNR) to check whether the output is exactly the same as without SetMTMode and using idx. Still, if you don't use idx, MVTools is much slower so you might not benefit from multithreading at all.

[Serbianboss]

you could use a metrics comparison (SSIM, PSNR) to check whether the output is exactly the same as without SetMTMode and using idx


How to do this to be sure?

I now test MVtools script with different parameters. I insert lambda,delta,overlap and script post above have delta pel overlap sharp.


setmtmode(2)
source = AVISource("C:\Documents and Settings\Nenad\Desktop\vulkani.avi").Trim(11368,12867)

fields=source.AssumeBFF().SeparateFields()

backward_vec2 = fields.MVAnalyse(isb = true, lambda = 1000, delta = 2,overlap=4,idx=1)
backward_vec1 = fields.MVAnalyse(isb = true, lambda = 1000, delta = 1,overlap=4,idx=1)
forward_vec1 = fields.MVAnalyse(isb = false, lambda = 1000, delta = 1,overlap=4,idx=1)
forward_vec2 = fields.MVAnalyse(isb = false, lambda = 1000, delta = 2,overlap=4,idx=1)
fields.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=800,idx=1)
Weave()


I test this script with and without idx=1 and result are same. Speed was 0.12 in CCE(cpu usage 98-100%) with setmtmode(2) with and without idx=1.

[Boulder]

Find the necessary plugins and read the docs.

[tsp]

serbianboss: Ok got your script to crash. It seems that it is leakkernelbob that are causing it so try this version:

setmtmode(2)
LoadPlugin("RemoveGrainSSE3.dll")
LoadPlugin("PeachSmoother")
LoadPlugin("LeakKernelDeint.dll")
LoadPlugin("mt_masktools.dll")
import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avs")


avisource("C:\Documents and Settings\Nenad\Desktop\vulkani.avi")
Trim(11368,12867)
crop(8,4,-8,-12)
ConvertToYuY2(interlaced=true)
AssumeBFF()
setmtmode(6)
LeakKernelBob(Order=0,threshold=1)
setmtmode(2)
PeachSmoother(NoiseReduction = 80, Stability = 25, Spatial = 200)
limitedsharpenfaster(strength=100)

AssumeBFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
AddBorders(8,8,8,8)


The above script still crashes so you will have to use another bober or setting threshold=0 as it looks like the code that produces the motion mask in leakkerneldeint causes the crashes.

The other script that I posted might be slower due to the overhead of changing mt mode from 2 to 5 in the script

[Serbianboss]

That script also crash evan if set thresold=0. Main reason is leakkernelbob.

I now try this script. I am using Tdeint(mode=1).


setmtmode(2)
LoadPlugin("RemoveGrainSSE3.dll")
LoadPlugin("PeachSmoother")
LoadPlugin("mt_masktools.dll")
import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avs")

avisource("C:\Documents and Settings\Nenad\Desktop\vulkani.avi")
Trim(11368,12867)
crop(8,4,-8,-12)
#ConvertToYuY2(interlaced=true)
AssumeBFF()

TDeint(mode=1)
DeGrainMedian (limitY=5,limitUV=7,mode=1)
VagueDenoiser(threshold=4, method=1, nsteps=6)

limitedsharpenfaster(smode=4,strength=100)

AssumeBFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
AddBorders(8,8,8,8)


I tried this script and didnt crached(test 1 minute video). Must to try more material to be shure that is threadafe. It seems that tdeint dosnt have threshold parameter.

Its slower(because tdeint is slow) but need to test more.

Script with MVtools is ok just to stay Setmtmode(2) at beginning of script?

@boulder

I find SSIM plugin on wanterprise site but didnt find any documentation. And didnt find PSNR plugin.

So if you know where can find PSNR and documentation for SSIM it would be helpful.

[Boulder]

It seems that tdeint dosnt have threshold parameter.You should really learn how to read the docs..
I find SSIM plugin on wanterprise site but didnt find any documentation.A simple search for "ssim" would have shown you this thread : http://forum.doom9.org/showthread.php?t=61128&highlight=ssim

[Serbianboss]

A simple search for "ssim" would have shown you this thread


searched but didnt showed on list.

@tsp

Script with tdeint crashed. Similar like likekernelbob, sometimes crash sometimes work.

[tsp]

Serbianboss: I'm trying to track down what is happening but as usual it is rather hard to figure out this bugs.

[tsp]

hmm I think I found the bug. I created a new version multithreaded version of avisynth (based on 2.5.7 RC3) that contains the fix. It can be downloaded here (http://www.avisynth.org/tsp/avisynth.zip). It might also fix the rare CCE crash although I haven't tested that.

[foxyshadis]

Ah, so all this time all we needed to get an updated MT was to find a bug! ;) (I know, RC3 is probably going to be the final.)

[Serbianboss]

I am still using 2.56. Can this be used with that version or need to install new version

[tsp]

I think you can use it without installing 2.57 but as it just went final I would recommend you to install it first

[Serbianboss]

I will install new avisynth and test.
Did you test with new avisynth, did crash or no?

[tsp]

I played about 45000 frames with no crash with your script using the old version of avisynthMT it crashed or locked up after about 45-4000 frames.

[Serbianboss]

I install avisynth 2.57 (31 dec version) and then copy your avisynth dll(7 januar version).

Test on 1500 frames in CCE, two pass and script didnt crash.

Using this script, just with setmtmode(2) CPU was all time 100%


setmtmode(2)

LoadPlugin("RemoveGrainSSE3.dll")
LoadPlugin("LeakKernelDeint.dll")
LoadPlugin("mt_masktools.dll")
import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avs")

avisource("C:\Documents and Settings\Nenad\Desktop\vulkani.avi")
Trim(11368,12867)
crop(8,4,-8,-12)
#ConvertToYuY2(interlaced=true)
AssumeBFF()

LeakKernelBob(Order=0,threshold=2)
DeGrainMedian (limitY=5,limitUV=7,mode=1)
VagueDenoiser(threshold=4, method=1, nsteps=6)


limitedsharpenfaster(strength=100)

AssumeBFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
AddBorders(8,8,8,8)



I played about 45000 frames with no crash with your script using the old version of avisynth


You mean avisynth 2.56. MT is still 0.5 version

[pyrates]

Now that AviSynth 2.5.7 has been released, is there going to be a new release of the MT plugin made for it? Like there was one made for 2.5.6a.

[tsp]

pyrates: Yes you can get in in this (http://forum.doom9.org/showthread.php?p=927761#post927761) post (RC3=final). I will make more official announcement later.

[Zep]

pyrates: Yes you can get in in this (http://forum.doom9.org/showthread.php?p=927761#post927761) post (RC3=final). I will make more official announcement later.

hi tsp,

AMD 64 X2 (dual core)

just tried it and something is wrong. A super simple script from HDTV TS source encoding to Xvid. The first I get 68 FPS and total CPU at 71% the second i get a wild roller coaster constantly the whole way from 4 (feels like it is stalling) to 80 fps with an average of just 33 FPS and Total CPU only hitting 50% ( each core at 50% give or take so total CPU = 50%) The old version 2.5.6 from your site i got 99% total CPU and 87FPS as each core would hit 99%. The older version really helps the Lanczos4Resize call big time.

#SetMTMode(6)
mpeg2source("D:\test.d2v",cpu=0)
Crop(8,8,-8,-12)
#SetMTMode(2)
Lanczos4Resize(624,352)

SetMTMode(6)
mpeg2source("D:\test.d2v",cpu=0)
Crop(8,8,-8,-12)
SetMTMode(2)
Lanczos4Resize(624,352)


doing this makes it even slower and a more jerky FPS

SetMTMode(2)
mpeg2source("D:\test.d2v",cpu=0)
Crop(8,8,-8,-12)
Lanczos4Resize(624,352)


Thoughts?

Thanks

[tsp]

Zep: I tried your last script:

SetMTMode(2)
mpeg2source("D:\test.d2v",cpu=0)
Crop(8,8,-8,-12)
Lanczos4Resize(624,352)

and it ran with 90-98% cpu utilization. What version of mpeg2source do you use and does it happen with every source?

[Zep]

Zep: I tried your last script:

SetMTMode(2)
mpeg2source("D:\test.d2v",cpu=0)
Crop(8,8,-8,-12)
Lanczos4Resize(624,352)

and it ran with 90-98% cpu utilization. What version of mpeg2source do you use and does it happen with every source?

i use 1.4.9.beta 7 currently of the DGDecode.dll

yes it happens when I use avisource and directshowsource
on all video formats as well as well as all source indexed sources.

i went back to 2.5.6 and all is fine. Also 2.6 works fine.
(the alpha that IIRC wilbert compiled a few months ago)

[tsp]

Zep: Could you try this (http://www.avisynth.org/tsp/avisynthMT257.2.zip) version.

[foxyshadis]

That seems to work much better in SetMTMode mode. The first version seemed to have lock contention and offer minimal gain, but this was posted before I could complain. ;)

[tsp]

foxyshadis: yes that sound right as I removed the Enter/LeaveCriticalSection that I added in the first 2.5.7MT version in the cache code. It was added in the first place because I overlooked an unprotected relinking of a double-linked list (that moves a frame that are being generated to the front of the list of cached frames when the frame is ready). Turns out it was not really needed(as it is done by the thread that generated the frame) so I removed the relinking in last posted version.

[Zep]

Zep: Could you try this (http://www.avisynth.org/tsp/avisynthMT257.2.zip) version.

better but still not as good as older versions and still getting wild frame rate roller coaster. in fact if i play a preview ( do not encode ) and use assumefps(300) so that it previews at max speed from an avisource xvid using either of the following I get total stalls and stuttering and half the frame rate (if that) compared to turning off MT.

SetMTMode(2)
avisource("d:\test.avi")
Lanczos4Resize(320,240)
AssumeFPS(300)

SetMTMode(6)
avisource("d:\test.avi")
SetMTMode(2)
Lanczos4Resize(320,240)
AssumeFPS(300)



So basically without MT it plays smooth and at 230 FPS with MT on I get wild rates from 0 (stalls out) to 130 and the preview stutters like crazy. granted that is an extreme example. Anyway, without MT I'm only using 70% CPU and with MT it now uses about 90% but FPS tanks and stalls and stutters. On slower FPS and encoding it was so so. much better than version 1 but not as good as 2.5.6 and the 2.6 alpha.


maybe you could try that extreme example above and the problem will show up for you :)

thanks

[tsp]

Zep: Now I can reproduce it with a xvid source(using xvid 1.1 final but not with ffdshows libavcodec) but not with mjpeg encoded files. Seems to be some pretty bad interactions between xvid 1.1 and my latest version.

[Serbianboss]

I now tested on 133738 frames (1 hour 30 minute) and script didnt crash. I am using avisynth 2.57 and first release avisynthMT 2.57(7 januar version)
Just use setmtmode(2) at beginning of script and cpu usage 100%

Does we need to install this avisynthmt.dll
http://forum.doom9.org/showthread.php?p=930674#post930674

[tsp]

Serbianboss: You might want to try it as it should be a little faster.

[Zep]

Zep: Now I can reproduce it with a xvid source(using xvid 1.1 final but not with ffdshows libavcodec) but not with mjpeg encoded files. Seems to be some pretty bad interactions between xvid 1.1 and my latest version.

hmmm ok I'm using the Xvid 1.2 SMP build here so the decoding must be about the same as 1.1 since you saw it there as well.

So you are seeing some sort of decoding caching issue or something? I'm just curious to the details of what is going on :D


thanks

[Zep]

Just use setmtmode(2) at beginning of script and cpu usage 100%



i hate to rain on your parade but you should never go by CPU usage. You should go by FPS/TIME and compare that to a run without MT. I'm just saying increased CPU load does not always equal higher FPS and faster encode times etc...

[Serbianboss]

You might want to try it as it should be a little faster.


I try that version(9 januar) on 5 minutes video and speed was the same like previous avisynthMT.

i hate to rain on your parade but you should never go by CPU usage. You should go by FPS/TIME and compare that to a run without MT. I'm just saying increased CPU load does not always equal higher FPS and faster encode times etc...

Its importan cpu usage, because when i always have 100% or 80% cpu usage i have almost double speed in CCE.
For example, with reinterlance script instead 8 hour to encode, with 100% cpu usage i have double speed and need 4 hour to encode complete video.

[MacAddict]

Its importan cpu usage, because when i always have 100% or 80% cpu usage i have almost double speed in CCE.
For example, with reinterlance script instead 8 hour to encode, with 100% cpu usage i have double speed and need 4 hour to encode complete video.

Think what Zep was saying is that you shouldn't trust the CPU usage entirely when judging the MT plugin builds. You might see both of your CPU graphs using 95-100% at all times yet your encode is 50% slower because of a possible bug somewhere. Time tests seems to be the only accurate and sure way to judge the effectiveness.

[IanB]

Firetrucks are red.
Your car is red.
Therefore your car is a firetruck!

100% CPU means no idle time. It does not mean that the work is 100% useful. Dumb code running around in a hard loop waiting for the other thread to release a resource will be 100% busy but do no usefull work!

Idle CPU time simply means the CPU is not the bottleneck! Something else is at 100% i.e. disk, PCI bus, Memory Bus, Cache or etc

[pelle412]

Hi, I just started play with this recently to see if I could get a RemoveDustMC script to run a bit faster. I am using the latest version of MT.dll and the version of avisynth.dll that was updated from RC3 of 2.5.7.

The script runs fine and quite a bit faster but after maybe half an hour it just stops. I tried both VirtualDub and avs2avi and they just stop progressing after a while. I haven't really experimented much with different modes. Would that be a reason it hangs after a while?


SetWorkingDir("E:\AviSynth\")
LoadPlugin("MT.dll")
SetMTMode(2,0)
LoadPlugin("DGDecode.dll")
LoadPlugin("Decomb.dll")
LoadPlugin("RemoveDirt.dll")
LoadPlugin("RemoveGrain.dll")
LoadPlugin("ColorMatrix.dll")
LoadPlugin("Repair.dll")
LoadPlugin("SSETools.dll")
LoadPlugin("mvtools.dll")
LoadPlugin("MaskTools.dll")
LoadPlugin("mt_masktools.dll")
LoadPlugin("VagueDenoiser.dll")
Import("E:\AviSynth\RemoveNoiseMC.avs")
mpeg2source("E:\Movie\a\x.d2v",info=3).ColorMatrix(hints=true,interlaced=true)
AssumeTFF()
Telecide(guide=1,post=4,hints=true).Decimate(mode=3,quality=3)
MT("RemoveNoiseMC()",2)

[Boulder]

You can't use any MVTools-related stuff with MT or SetMTMode. Besides, you shouldn't use both at the same time.

[henryho_hk]

Is there a MT version of avisynth.dll for AVISynth 2.5.7 release?

Edit: Oops ~~~ Thanks kurt & Serbianboss.

[kurt]

http://forum.doom9.org/showthread.php?p=930674#post930674

[Serbianboss]

Read posts!

http://forum.doom9.org/showthread.php?p=930674#post930674

[J-Wo]

Hey, decided to give the new MT avisynth.dll for 2.5.7 a try. I'm doing a 2-pass VBR encode with CCE and while my CPU usage started around 95-98%, after 30 minutes it's dropped down to 77-83%. Encoding speed has likewise dropped down from 3.60x to 3.33x. Not sure if that is meaningful at all, just thought I'd post here tho. Here's my script:setmtmode(2)
loadplugin("D:\Program Files\AviSynth 2.5\plugins\DGDecode.dll")

mpeg2source("E:\MOVIE\VTS_01_1.d2v",idct=6)
Lanczos4Resize(720, 464, 0, 12, 720, 552)
AddBorders(0, 8, 0, 8)
AssumeFPS(23.976, true)
LetterBox(0, 0, 8, 8)
ConvertToYUY2()

[Revgen]

Hey, decided to give the new MT avisynth.dll for 2.5.7 a try. I'm doing a 2-pass VBR encode with CCE and while my CPU usage started around 95-98%, after 30 minutes it's dropped down to 77-83%. Encoding speed has likewise dropped down from 3.60x to 3.33x. Not sure if that is meaningful at all, just thought I'd post here tho. Here's my script:setmtmode(2)
loadplugin("D:\Program Files\AviSynth 2.5\plugins\DGDecode.dll")

mpeg2source("E:\MOVIE\VTS_01_1.d2v",idct=6)
Lanczos4Resize(720, 464, 0, 12, 720, 552)
AddBorders(0, 8, 0, 8)
AssumeFPS(23.976, true)
LetterBox(0, 0, 8, 8)
ConvertToYUY2()

CCE is a 1-thread encoder right?

The first pass may not be hindering SetMT from using the cores. Once the 2nd pass starts going, CCE may be bottlenecking Avisynth.

[tsp]

J-Wo: It could be because one of the avisynth threads stalled. TRy downloading process explorer (http://www.microsoft.com/technet/sysinternals/ProcessesAndThreads/ProcessExplorer.mspx) Run it while CCE is running (before and after you experience the slowdown) and double-click on the cce process. Select the thread tab. The cpu utilization between the two avisynth threads should be evenly distributed if one of them is hovering about 0-10% it might be a bug in avisynth.

[J-Wo]

The avisynth threads never get down that low, they seem to be steady around 25% +/- 5%. Again after ~30 mins of encoding, CPU utilization has dropped to 76-80%. If you'd like I could run a comparison with an older version of MT. BTW, is there an easy way to have CCE log how long the encoding process took? I'm not often at my computer during encoding so I often miss the window before it closes...

[tsp]

it would be fine if you could compare it with the 2.5.6MT version and the non-MT 2.5.7 version. The option to enable the log file hides in the options->outputs menu.

[henryho_hk]

I have problem using Writefile() with SetMTMode(). Below is a simplified scenario:

Case 1: The values are nicely placed in c:\1.txt delimited by a space.

Blankclip(pixel_type="yuy2",color=$000000,length=71,width=720,height=240,fps=30000,fps_denominator=1001).killaudio()
trim(0,-1)
space=" "
writefile("c:\1.txt", "Width", "space", "Height", "space", "Framerate", "space", "FramerateNumerator", "space", "FramerateDenominator", "space", "HasAudio", append=false)


Case 2: The space characters become something like "I don't know what "space" is".

SetMTMode(2)
Blankclip(pixel_type="yuy2",color=$000000,length=71,width=720,height=240,fps=30000,fps_denominator=1001).killaudio()
trim(0,-1)
space=" "
writefile("c:\2.txt", "Width", "space", "Height", "space", "Framerate", "space", "FramerateNumerator", "space", "FramerateDenominator", "space", "HasAudio", append=false)


I am using Avisynth.dll in http://www.avisynth.org/tsp/avisynthMT257.2.zip

[foxyshadis]

Try
global space = " "
I guess it's just a scoping issue.

[tsp]

global space =" " would work but it is a bug that happens because each thread has each own var table but they didn't have access to the local var table that are created when the filter chain is created (that contains the variable space in this case) I have fixed this in this (http://www.tsp.person.dk/avisynthMT257.3.zip) version (avisynthMT 2.5.7.3)

[henryho_hk]

tsp, thank you very much.

[halsboss]

A dummy's query, does a socket-939 AMD 64 3500+ Venice Core have hyperthreading ? AMD's website doesn't help me. Another way of putting it, is MT useful with that processor ?

[Boulder]

MT's not useful as your CPU isn't dual-core. AMD doesn't have anything like hyperthreading IIRC.

[Revgen]

global space =" " would work but it is a bug that happens because each thread has each own var table but they didn't have access to the local var table that are created when the filter chain is created (that contains the variable space in this case) I have fixed this in this (http://www.tsp.person.dk/avisynthMT257.3.zip) version (avisynthMT 2.5.7.3)

I was having problems with the 2.56 version where the program would quickly exit whenever it would encode a certain frame. For some reason, this version doesn't have any issues. I'll definitely keep it for now.

[Adub]

I believe that I have found a new bug. I was recently trying to encode a movie, Step Up, to x264, using the following script:

SetMTmode(2)
DGdecode_Source("C:/Step Up/decomb ivtc.avs", info=3)
Colormatrix(hints=true)
AssumeTFF.Telecide(guide=1).Decimate()
Autocrop(0)
Spline36resize(1280,702)


Except every time that I go to encode it, eventually x264, versions 620 and 621, spit out an in windows, saying that it needs to close. When I click for more information, the dll in question switches between "decomb.dll" and "Avisynth.dll". Also, I got the error to occur using xvid encraw, so I know that it is not encoder specific. However, I could only get xvid to error out on the Step Up script, not the blankclip script, as I could with x264.

My avisynth.dll is your 2.5.7.2 I suppose. I have yet to try your latest one, but I will check right now.

Edit: okay, 2.5.7.3 errors out as well.
You can find the error using megui if you use the HQ-Slow profile and the following script:
setmtmode(2)

blankclip(pixel_type="YV12",length=100000).AddGrain(100,0,0)
Colormatrix()
AssumeTFF().Telecide(guide=1).Decimate()
#crop
#resize
#denoise

Spline36resize(1280,704)

[Adub]

TSP, can you confirm the bug?

Can anyone confirm the bug?

[tsp]

Merlin7777: 80 % done with second pass x264 HQ-Slow profile and no crash yet with the blankclip script.

[Adub]

What version of decomb.dll are you using?

[webzeb]

Hello,

Is there any solution to use MT/SetMT for resizing (Spline36Resize) under FFDShow ?

SetMTMode seems not working (GetMTMode return '0')...

Thank you,

Fred

[bikes302]

I must say that I am quite thrilled that I finally found this multithread mod to avisynth. I bought a new dual core machine hoping to encode faster, but when I tried it was just as slow. It was taking me 12 hrs per pass, 24 hrs total on my 2.66 Ghz machine. I noticed it was only using one core and I was quite upset. I found out that avisynth was bottlenecking my encode and found this. I added in the MT dll and replaced my avisynth.dll file and my encode now uses 100% both cores and went from 12 hrs per pass to abt 5 hrs per pass and 10 total. I must say I am more than thrilled with this.

Just thought I should mention though that there are a few issues. I am not sure what exactly, but I cant do both passes continuously. Once the one pass finishes in VDub, I have to close and reopen to do the second pass or else I get one of those VDub needs to close, sorry for the inconvience. It also happens occassionally as soon as it starts to encode. If thats the case I just close and reopen and keep trying until it works, eventually it starts to work and once it starts it has no issues. Which puzzles me because sometimes it works and sometimes it doesnt with the same settings, but I dont care since my encode time was cut in half.

[tsp]

Merlin7777: decomb 5.2.2

[Adub]

okay, i am testing on a different computer, same filter versions, only single core. Also, TSP, do you have your threads set to "auto" in your HQ-Slow profile?

[tsp]

merlin7777: yes threads are set to auto

[Adub]

Okay, well a short little preliminary test of the blank clip script on the single core led to nothing. I aborted it after 12.1%. Mostly because it crashed @ 12% on my dual core.

I will complete a full test on the single core, it will just take a long time. I have access to another dual core that I will test eventually, and see if I can isolate it further.

@TSP
Is there anything I can do to help you diagnose this better? Like send you the error report created by windows when it crashes?

I don't know if I mentioned this or not, but it crashes on Xvid as well. The weired thing about it is that the blank clip script doesn't crash (atleast, not that I remember), but it does crash on the Step Up script, so I am sure that it is not just x264's problem.

[squid_80]

Don't you have an E6600 overclocked to 3ghz? Maybe it's not as stable as you think it is.

[Adub]

Nope, its not the overclock for two reasons.

1) I just updated by bios and my clocks are now back to normal, so I tested to be sure, and it still crashed.

2) I just tested on a new core 2 duo build of mine, actually I built it for my mom, but it also crashes using the same exact filters/MeGUI setup/Avisynth setup.

Edit: I think that I found the problem!!!!!!!!

TSP, try running the blankclip encode using the new colormatrix 2.1. If I removed Colormatrix() from the script, it worked! So, I think that the new multithreaded version of Colormatrix is having issues with you multithreaded Avisynth.

[tsp]

merlin7777: Yes colormatrix 2.1 does crash very fast on my machine. setting threads to 1 in colormatrix seems to be more stable

[Adub]

So is this an official bug? Should I report it to tritical in the Colormatrix 2.1 thread? or should I just go along using "1" threads, even though according to the manual, that should be the default.

[webzeb]

Hello,

Is there any solution to use MT/SetMT for resizing (Spline36Resize) under FFDShow ?

SetMTMode seems not working (GetMTMode return '0')...

Thank you,

Fred
:confused:

[Adub]

What kind of processor are you using?
Besides MT doesn't like resizing, and that is the one that you will want for realtime watching.
SetMTmode is better for encoding, unfortunately for you, it supports resizing but it wont usually speed up realtime watching significantly, at least it doesn't for me. That is why I use MT for realtime viewing.

[tsp]

webzeb: The problem is that ffdshow insert a source filter before the avisynth script so SetMtmode is never the first filter in the script. You can see it by removing the selection in "add ffdshow video source" checkbox in the avisynth option in ffdshow.

Merlin7777: Yes the bug is official. I don't know if it is triticals code or mine that are wrong (probably mine :D ). Also if threads is not specified the default value is the number of cores available according to the manual.

[webzeb]

What kind of processor are you using?
Besides MT doesn't like resizing, and that is the one that you will want for realtime watching.
SetMTmode is better for encoding, unfortunately for you, it supports resizing but it wont usually speed up realtime watching significantly, at least it doesn't for me. That is why I use MT for realtime viewing.
I use a Core 2 Duo E6300, overclocked @3.326Mhz.

I play HD (720p/1080p) stuff, resized to 1024*768px.
(I watch it on a 1024*768px plasma TV. ;) )

Thank you for your answer,

Fred

[tsp]

It is possible to use a resizer with mt() (at least after I fixed a bug in mt.dll see first post) by first doing the vertical resize and afterwards the horizontal resize. Something like this:

src=avisource("c:\test.avi")
s.mt("spline36resize(1280,last.height())",splitvertical=false)
mt("spline36resize(last.width(),704)",splitvertical=true)


It produces the exact same result as spline36resize(1280,704) because avisynth internally split the resizer up into a seperate horizontal and vertical resize

[Terranigma]

Thanks for the new version tsp (0.6) :D

[BigDid]

Hi,

@TSP, many thanks from a young dualcore owner (still in test). Continue the good work.

Regarding speed some test results here http://forum.doom9.org/showthread.php?p=948439#post948439
or here http://forum.doom9.org/showthread.php?p=949322#post949322

Did

[webzeb]

It is possible to use a resizer with mt() (at least after I fixed a bug in mt.dll see first post) by first doing the vertical resize and afterwards the horizontal resize. Something like this:

src=avisource("c:\test.avi")
s.mt("spline36resize(1280,last.height())",splitvertical=false)
mt("spline36resize(last.width(),704)",splitvertical=true)


It produces the exact same result as spline36resize(1280,704) because avisynth internally split the resizer up into a seperate horizontal and vertical resize
Many thanks ! :)

[Adub]

Actually TSP you are wrong, the default is one thread at least it is supposed to be:
threads:

Sets the number of threads Colormatrix will use for processing. Can be any value greater than 0 and, for YUY2, less than the frame height, for YV12, less than the frame height divided by 2. If set to 0, ColorMatrix will automatically detect the number of available processors and set threads equal to that value.

default - 1 (int)


Yeah, and you are right about setting the threads to 1, is last just a little bit longer, but still crashes.

[tsp]

Merlin7777: yes threads=1 are the default must have seen the default value for thrdmthd. Can you reproduce the crash with version 2.0?

[Adub]

if you mean colormatrix 2.0, then no. On my preliminary tests with colormatrix 2.0 and the blankclip script, I couldn't get it to crash. It appears that the new multithreading in colormatrix is very skittish when pared with MT Avisynth.

[Jeremy Duncan]

tsp,

Is mt.dll version 0.5 different from mt.dll 0.6 ? Or is only the avisynth.dll different ?

[tsp]

Jeremy Duncan: MT 0.6 contains a new mt.dll (that fixes the bug with vertical resizing) and avisynth.dll(same the one from post #407)

[Jeremy Duncan]

Thank you.

[BlueCup]

Just throwing this out there, but is it possible this filter can put too much constant strain on the CPU/mobo and burn something out or cause damage somewhere? Let's say with a 12hr+ encode.

[foxyshadis]

Only if your hardware is 10 years old and improperly cooled, or your power supply is about to die.

Computers are designed to be able to run at full utilization for a full year straight if necessary, pushing it hard for 12 hours can't hurt it, and can't even crash it if it's properly cooled.

[tritical]

@tsp
Is the source code for your version of avisynth available? or just diffs from the cvs code? I got a chance to test Merlin7777's script using debug builds of Decomb, Colormatrix, and AddGrainC, along with your 2.5.7.3 avisynth.dll and 4 out of 5 times the script stopped on an access violation several calls deep in avisynth.dll. 1 time it stopped in Decimate::FindDupicate() because the data pointer of one of the pvideoframe object's videoframe pointer's vfb pointer was null causing an access violation when decimate tried to use the pointer it was returned by ->GetReadPtr().

[tsp]

tritical: the sourcecode is available here (http://www.avisynth.org/tsp/avisynth257MT3_src.7z). It includes a vs 2003 solution.
I also encountered the bug with decimate calling env->bltbit with a null pointer as the dst pointer.

[Zep]

Tsp, is there a way to TOTALLY shut off SetMTMode in parts of script? I was testing this following basic script using Tritical's TWriteAvi and i keep getting a file exists error which appears to coming from 2 threads of TWriteAvi each trying to create/open/write to "C:\test.avi" (work fine when i do not use MT)

Very basic failing example:

SetMTMode(5)
mpeg2source("D:\AutoEncode\part1\part1.d2v",cpu=0) #source input is 1080i
SetMTMode(2)
Lanczos4Resize(624,352)
SetMTMode(5)
TWriteAvi(fname="C:\test.avi",overwrite=true,showAll=true)


tsp i thought you once said there was a SetMTMode(0) for that but i can't find the thread. (Maybe i dreamt it haha)


Thanks

[jeffy]

tsp i thought you once said there was a SetMTMode(0) for that but i can't find the thread. (Maybe i dreamt it haha)
Thanks

More reading here:
http://forum.doom9.org/showthread.php?t=94996&page=15
http://forum.doom9.org/showthread.php?p=812583#post812583

[tsp]

Zep: mode=0 is set when setmtmode is not used. It can not be set. Mode 5 should only create one instance if the filter(TWriteAvi).
If you only wants to use multithreading to resize the video use mt() instead:

mpeg2source("D:\AutoEncode\part1\part1.d2v",cpu=0)
mt("Lanczos4Resize(624,last.height())",splitvertical=false)
mt("Lanczos4Resize(last.width(),352)",splitvertical=true)
TWriteAvi(fname="C:\test.avi",overwrite=true,showAll=true)

[tritical]

Thanks for the source. I haven't been able to figure anything out except that (on this comp at least) it is definitely a problem with filters having pvideoframes with references to vfbs with null data pointers and datasize=0... which seems to indicate that those vfbs had their destructor code run by i->~LinkedVideoFrameBuffer() in ScriptEnvironment::GetFrameBuffer2().

[tsp]

tritical: It suggest that the refcounting somewhere is messed up. If the refcount only temporarily goes down to 0 in a filter and increase to 1 before the filter return it would only appears as an error when running with more than 1 thread.

[Spuds]

I have been using 0.5 with good success. I upgraded the avisynth and mt dll's to the new 0.6 version and began to experience some problems.

Using just a simple script of setting mt mode=2 and opening an avi file (vhs capture, about 1hr long) with no processing. I then opened that in virtualdub and the playback has cracking in the audio and some single frames are fully pixelated with random colored blocks. I backed out to 0.5 and opened the same script with no issues. Anyone else experience problems?

[tritical]

When I was testing I did stick an _ASSERTE(vfb->refcount>0) inside VideoFrame::AddRef(), and also changed

if (InterlockedCompareExchange((long*)&i->refcount,1,0)==0)

to

if (InterlockedCompareExchange((long*)&i->refcount,0,0)==0)

in GetBuffer2() so that the refcount would stay 0 on those. The assert triggered once, but I wasn't able to figure out the sequence of events that lead to it happening. The call stack pointed back through PVideoFrame::Init() and PVideoFrame::PVideoFrame(VideoFrame* x) to this line in cacheMT.cpp:

PVideoFrame retval=BuildVideoFrame(i, n); // Success!

Unfortunately, I don't have access to that comp right now so can't do anymore testing until my comp is free.

[Zep]

More reading here:
http://forum.doom9.org/showthread.php?t=94996&page=15
http://forum.doom9.org/showthread.php?p=812583#post812583

yeah my memory is shot forgot about some of that. thx

[Zep]

Zep: mode=0 is set when setmtmode is not used. It can not be set. Mode 5 should only create one instance if the filter(TWriteAvi).
If you only wants to use multithreading to resize the video use mt() instead:

mpeg2source("D:\AutoEncode\part1\part1.d2v",cpu=0)
mt("Lanczos4Resize(624,last.height())",splitvertical=false)
mt("Lanczos4Resize(last.width(),352)",splitvertical=true)
TWriteAvi(fname="C:\test.avi",overwrite=true,showAll=true)



ok will do. I sorta thought i would have to use MT() but sometimes my triple and quad quoting doesn't work haha and why i like to stick to set SetMTMode() whenever i can :)

will give it a go

thx

[Zep]

Zep: mode=0 is set when setmtmode is not used. It can not be set. Mode 5 should only create one instance if the filter(TWriteAvi).
If you only wants to use multithreading to resize the video use mt() instead:

mpeg2source("D:\AutoEncode\part1\part1.d2v",cpu=0)
mt("Lanczos4Resize(624,last.height())",splitvertical=false)mt("Lanczos4Resize(last.width(),352)",splitvertical=true)
TWriteAvi(fname="C:\test.avi",overwrite=true,showAll=true)

Having fun testing out all this stuff :)

I ran into a few problems though. Most of the following comes from me wanting to only MT certain functions and not MT the whole script with SetMTmode() since some of the filters i use are slower with setMTmode(5) or (6) than no MT at all and if i try to go 2 they crash and if 3 or 4 are really slow. Anyway...


1) I get an access violation error every time i try to do
mt("Decimate(cycle=5,quality=0)")
Any idea if the bug is in MT or Decimate?

I wonder because
SetMTMode(2)
Decimate(cycle=5,quality=0)

seems to works fine


2) for the life of me i can't figure out the syntax for
mt("mpeg2source("D:\whatever.d2v",cpu=0)")

I have tried triple quotes with no luck
mt(""" mpeg2source("D:\whatever.d2v",cpu=0) """)

any ideas on getting that to work?


3) 2.6 alpha with built in setMTmode() is faster for me by a rather large amount (about 25%) than the 2.5.7 MT 0.6 package. Any idea as to why that could be?


4) when using setMTmode() is there a way to set splitvertical to either true or false? Does it internally always use false?


5) Some filters work with setMTmode() but not with MT() or vise versa. Are there some basic differences we should know about between the two methods?

6) using as example Lanczos4Resize() with MT() is MUCH faster then using it in setMTmode() run. Not only that but CPU usage goes through the roof as well without any speed gain over the MT() way even when i do something like this

SetMTMode(2)
Lanczos4Resize(960,last.height())
Lanczos4Resize(last.width(),528)


oh i think i will stop for now :)

Thanks for such a wonderful tool!

[Boulder]

2) for the life of me i can't figure out the syntax for
mt("mpeg2source("D:\whatever.d2v",cpu=0)")

I have tried triple quotes with no luck
mt(""" mpeg2source("D:\whatever.d2v",cpu=0) """)
Have you tried MT("""MPEG2Source("path\clip.d2v")""") , that is, without any extra spaces?

In SetMTMode a thread requests the whole frame so no need to split in vertical or horizontal direction. MT requests one frame and splits it in two halves.

[foxyshadis]

2) Since mt() splits the input in pieces, that can't work, it's not smart enough to figure out when a source shows up; if it did, it'd probably give you two copies of the source stacked. All the multithreading comes from being able to split the input.

5) The basic difference is that mt() splits and sends both pieces of the frame to two copies of each filter, while setmtmode() says "give me 3, oh and run 4 while you're at it" on the assumption that the next frame is probably going to be needed, without splitting anything. I have no idea how smart the frame decision part is though. Some mt modes will try to use the same copy of the filter, some will create new ones. (That should explain why #4 isn't needed.)

The rest of the answers are buried in the code somewhere, I only know the basics. ;_;

[tsp]

Having fun testing out all this stuff :)

1) I get an access violation error every time i try to do
mt("Decimate(cycle=5,quality=0)")
Any idea if the bug is in MT or Decimate?

I wonder because
SetMTMode(2)
Decimate(cycle=5,quality=0)

seems to works fine

2) for the life of me i can't figure out the syntax for
mt("mpeg2source("D:\whatever.d2v",cpu=0)")

I have tried triple quotes with no luck
mt(""" mpeg2source("D:\whatever.d2v",cpu=0) """)

any ideas on getting that to work?

mt only works with filters taking 1 clip as a input (luckily most filters does that. But mpeg2source, avisource, DirectShowSource doesn't take a input clip so they wouldn't work. Also see foxyshadis explanation.

3) 2.6 alpha with built in setMTmode() is faster for me by a rather large amount (about 25%) than the 2.5.7 MT 0.6 package. Any idea as to why that could be?

I added some more thread synchronization to the 2.5.7 version to fix some crashes. This will case more waiting for each thread and thereby reducing performance. What script exactly do you experience this with?

4) when using setMTmode() is there a way to set splitvertical to either true or false? Does it internally always use false?


5) Some filters work with setMTmode() but not with MT() or vise versa. Are there some basic differences we should know about between the two methods?

See foxyshadis explanation.

6) using as example Lanczos4Resize() with MT() is MUCH faster then using it in setMTmode() run. Not only that but CPU usage goes through the roof as well without any speed gain over the MT() way even when i do something like this

SetMTMode(2)
Lanczos4Resize(960,last.height())
Lanczos4Resize(last.width(),528)


Well MT uses a more simple method so for filters that doesn't need information about the complete frame it should be used instead of SetMTMode (unfortunately Setmtmode and MT() doesn't mix very well in the same script). I will try to improve the performance with setmtmode but unfortunately I don't have unlimited time to do that :)

oh i think i will stop for now

Thanks for such a wonderful tool![/QUOTE]

I have no idea how smart the frame decision part is though.
A special filter is inserted at the end of the script and before each Setmtmode(5) and 6 that calculates the difference between the current and the last frame and add it to the current frame to see what frame to request next. No very intelligent but works fine if mode=5 or =6 is not used as most application that opens the avisynth script request the frame in linear order.

[Zep]

2) Since mt() splits the input in pieces, that can't work, it's not smart enough to figure out when a source shows up; if it did, it'd probably give you two copies of the source stacked. All the multithreading comes from being able to split the input.

interesting.


5) The basic difference is that mt() splits and sends both pieces of the frame to two copies of each filter, while setmtmode() says "give me 3, oh and run 4 while you're at it" on the assumption that the next frame is probably going to be needed, without splitting anything. I have no idea how smart the frame decision part is though. Some mt modes will try to use the same copy of the filter, some will create new ones. (That should explain why #4 isn't needed.)

After reading your post I can see why MT() is faster on my resize example.

[Zep]

Have you tried MT("""MPEG2Source("path\clip.d2v")""") , that is, without any extra spaces?

yes i did and it does not work but tsp explained why above :)


In SetMTMode a thread requests the whole frame so no need to split in vertical or horizontal direction. MT requests one frame and splits it in two halves.



ahhh and now i see why resize works fine in setmtmode without having to worry about changing both width and height at the same time unlike MT()

[Zep]

mt only works with filters taking 1 clip as a input (luckily most filters does that. But mpeg2source, avisource, DirectShowSource doesn't take a input clip so they wouldn't work. Also see foxyshadis explanation.

ahh right I should have realized that on my own.


I added some more thread synchronization to the 2.5.7 version to fix some crashes. This will case more waiting for each thread and thereby reducing performance. What script exactly do you experience this with?

all scripts the 2.6 alpha that wilbert compiled is faster
than all other version i have tried by a good amount.



Well MT uses a more simple method so for filters that doesn't need information about the complete frame it should be used instead of SetMTMode (unfortunately Setmtmode and MT() doesn't mix very well in the same script). I will try to improve the performance with setmtmode but unfortunately I don't have unlimited time to do that :)

well in the 2.6alpha i use the latest MT plug-in from the 0.6 package with it and using the methods together so far has worked very well.

example being

setmtmode(3)
mpeg2source*
mt("Lanczos4Resize(960,last.height())",splitvertical=false)
mt("Lanczos4Resize(last.width(),528)",splitvertical=true)

note mode 3 is faster than mode 2 for mpeg2source which i find interesting. mode 2 puts that filter into a roller coaster ride but mode 3 speeds it up some and keeps it at a steady rate on my box anyway :)

thx!

[BigDid]

...all scripts the 2.6 alpha that wilbert compiled is faster
than all other version i have tried by a good amount.

Hi,

Could you give a link to the Wilbert 2.6 alpha cause I can see only the 2.6 pre-alpha like here http://forum.doom9.org/showthread.php?p=862298#post862298
or here
http://forum.doom9.org/showthread.php?p=841359#post841359

Thanks

Did

[Wilbert]

There is no 2.6 alpha. I think that besides a new mode for Histogram nothing is changes wrt to the pre-alpha version i posted.

[BigDid]

There is no 2.6 alpha. I think that besides a new mode for Histogram nothing is changes wrt to the pre-alpha version i posted.
Hi Wilbert,

Indeed, I have searched elsewhere without success :(
Thanks for confirming this.

Did

[foxyshadis]

The big upside to less thread synchronization is more speed, as you notice. The big downside is that the less you have, the more often things crash, corrupt, or otherwise don't work right. Getting lucky most of the time isn't worth it if it crashes every 5 minutes, even if it's only on specific cpu/bus configurations.

Still, Win32 synchronization is enormously slower than pthreads, because it's designed to be cross-process, whereas in avisynth's case there's no interprocess communication.

[Zep]

Hi Wilbert,

Indeed, I have searched elsewhere without success :(
Thanks for confirming this.

Did

sorry about that. i have had it installed for so long I didn't remember he called it PRE alpha.

[Zep]

The big upside to less thread synchronization is more speed, as you notice. The big downside is that the less you have, the more often things crash, corrupt, or otherwise don't work right. Getting lucky most of the time isn't worth it if it crashes every 5 minutes, even if it's only on specific cpu/bus configurations.

i agree but so far everything has worked fine with the pre alpha 2.6 and 0.6 mt plugin on over 30+ encodes since the 0.6 release.

I did have problems with the 2.5.7 ( 0.6 package) though strangely enough.

[tsp]

Still, Win32 synchronization is enormously slower than pthreads, because it's designed to be cross-process, whereas in avisynth's case there's no interprocess communication.

It is only true if mutex'es is used. CriticalSections is not designed to be cross-process so that is what is used in avisynth.

[BigDid]

i agree but so far everything has worked fine with the pre alpha 2.6 and 0.6 mt plugin on over 30+ encodes since the 0.6 release.

I did have problems with the 2.5.7 ( 0.6 package) though strangely enough.
Hi,

Just installed the 2.6 pre-alpha dll with 0.6MT. It seems speedy and MT seems ok. I may have problems with setmtmode I was not having with 2.57/0.5. I'll switch back to 2.57/0.6 and test.

@TSP I send a PM a few days ago. Unless not readed I suppose the answer is no. No big deal, please confirm.

Did

[foxyshadis]

It is only true if mutex'es is used. CriticalSections is not designed to be cross-process so that is what is used in avisynth.

Okay, if they're all CS then I understand, that makes sense. The speed difference must come down to some general threading weirdness somewhere, then.

[BigDid]

@ TSP

No news from my last PM.

The draft for the Wiki is finished (discussion tab), please check and let me know when Ok to update the main page :cool:

Did

[tsp]

to all: Bigdid, Pookie and I created a WIKI support page at the avisynth mediawiki for MT. It contains usefull citation from this 470 post long thread and other usefull information. You are welcome to see and contribute to it here (http://avisynth.org/mediawiki/MT_support_page)

[Pookie]

Thanks for the kind words, but the depth of my contribution was suggesting two lines in the code examples be modified. Otherwise, BigDid is the author.

[IanB]

There seems to be a problem with the pictures (I think Wilbert might be investigating)

Perhaps you could improve the enumeration of the the SetMT modes by explaining actually what each mode does, rather than the vague "this is faster but most filters don't work". I have worked with the code and even I am still not sure exactly what the intent of each mode truely is.

[BigDid]

There seems to be a problem with the pictures (I think Wilbert might be investigating)
Hi,

Now fixed, thanks to Richard Berg: http://forum.doom9.org/showthread.php?p=960059#post960059

Perhaps you could improve the enumeration of the the SetMT modes by explaining actually what each mode does, rather than the vague "this is faster but most filters don't work". I have worked with the code and even I am still not sure exactly what the intent of each mode truely is.
Added in the Wiki/Requests paragraph.

@Pookie, moral "pulling the leg" is important :)

Did

[tsp]

I finally found the bug that was causing Merlin7777 script to crash in MeGUI. It affects general stability but only causes a crash in rare circumstances. You can get the new avisynth.dll here:
avisynth 2.57 MT version 4 (http://www.avisynth.org/tsp/avisynthMT257.4.zip) [src] (http://www.avisynth.org/tsp/avisynth257MT4_src.7z)

I also created a mediaWIKI page to try explain how the different modes is supposed to work. You can see it here:
http://avisynth.org/mediawiki/MT_modes_explained
modes 3-6 is still missing text. I will add it later. You're welcome to modify the content to something more readable.

[Adub]

Yeah!!!!!!!!!!!!!!!
Great job TSP!!!!!!!!!
Thank you so much! Your programing and hardwork is highly appreciated, so keep that in mind. Also know that you will always have a fan.
I will download and use as soon as is humanly possible.
Thanks again man!

[BigDid]

...It affects general stability but only causes a crash in rare circumstances. You can get the new avisynth.dll here:
avisynth 2.57 MT version 4 (http://www.avisynth.org/tsp/avisynthMT257.4.zip) [src] (http://www.avisynth.org/tsp/avisynth257MT4_src.7z)

Hi all,

I was having problem with degrainmedian() with avisynth 2.5.7.3 or 2.6 pre-alpha and Mt 0.6, either with setmtmode(2) or (3) or (4). Only MT("") was working.
It now works with the new 2.5.7.4 and setmtmode(2)
Could somebody confirm so the wiki can be updated?

:thanks:

Did

[tsp]

new version of MT.dll is up. It includes two new filters: MTi() for interlaced processing and MTsource for speeding up source filters (or slow them down, at least mpeg2dec seems to be slower with it but try and see for yourself)
It also contains a new version of avisynth 2.57 MT version 5. Only new thing in this version is that it is possible to invoke cacheMT2 using InternalCacheMT with more than 1 input clip. This is used by MTsource.

[n3w813]

TSP,
I'm getting an error when using the new MTi() function, getting an error saying the function does not exist. I extracted the MT.dll to avisynth/plugins folder and Avisynth.dll to windows/system32 folder, replacing both existing files.

Here is my MT call
MTi("SeeSaw(NRlimit=0, NRlimit2=5, Sstr=1.20, Szp=1, Slimit=50, Sdamplo=6, Spower=0, bias=0, sootheT=0, SdampHi=52)",3)

FYI, MT() works fine, MTi() does not.

[tsp]

n3w813: Could you post a frame from this script (use www.imageshack.ws to host it) :

mtpluginpath="c:\mt.dll" #path to mt.dll
s=loadplugin(mtpluginpath)
version.subtitle(s)

[Adub]

Uh, TSP, sorry man, but I still have a problem. I am still using that blank clip script, but x264 rev 628 just kind of hangs there about 25% or so. I doesn't even report an error, it just sits there. The progress window says 25%, yet with the FPS rate slowly going down.

I am encoding it with MeGUI, using HQ-Slow profile with threads set to "0", or "auto".

Atleast it doesn't crash, huh?

[tsp]

Merlin7777: how often does it happen? Is it still Colormatrix 2.1 that triggers it?

[Adub]

It happened both (two) the times I tested it, and yes I am still using Colormatrix 2.1. You can't reproduce it?

[tsp]

Sorry no I can't reproduce it. Could you download and use this (http://www.avisynth.org/tsp/a.zip) version of avisynth.dll instead and when the deadlock appears run Process Explorer (http://www.microsoft.com/technet/sysinternals/utilities/ProcessExplorer.mspx ) double click x264.exe select the "threads" tab and display and copy the stack of each thread and send the content to me.

[Adub]

Okay, I am on the wrong computer right now and I will be off it for several days. However, when I get back on, Monday night, I will do as you ask.

[Adub]

Huh, this is weired, I can't reproduce the bug again. Even after watching it happen twice, I still can't get it to happen.

I have made it successfully through the first pass not problem (where it was crashing before) and now I am on to the second pass, which I will finish.

If I don't report back today, then everything completed fine and there is no reason to worry.

Sorry for the trouble.

[Adub]

Okay, it happened again.
Here is what I found. It is only for the first thread, but I will get the rest of the threads later.
ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1c5e
ntkrnlpa.exe!ZwYieldExecution+0x2540
ntkrnlpa.exe!NtWaitForSingleObject+0x381
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForMultipleObjects+0x18
ntkrnlpa.exe+0x29b9e
ntkrnlpa.exe+0xd826d

[Adub]

Okay I have all of the threads. Here you go:
ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x381
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForMultipleObjects+0x18
ntkrnlpa.exe+0x29b9e
ntkrnlpa.exe+0xd826d




ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1c5e
ntkrnlpa.exe!ZwYieldExecution+0x2540
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12




ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x381
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForMultipleObjects+0x18
ntkrnlpa.exe+0x2a1b8
ntkrnlpa.exe+0x2ac70
ntkrnlpa.exe+0x8fcf5



Here is the order in which I got them:
http://i34.photobucket.com/albums/d125/Merlin7777/x264stackpic.jpg

[tsp]

Merlin7777: thanks unfortunately I can't see where the deadlock happens in avisynth. I will see if there are some way I can figure that out. I have run the blank clip script 4 times with megui without problems :(

[Adub]

All right. I am going to try it on another dual core rig of mine and see if the same thing happens. It could be something is wrong with my windows install, it has been acting rather funny lately.

[Adub]

Yep, it still happens on the other rig. I will post picture and thread details in a few minutes.

The odd thing is that Windows doesn't throw me a Dr. Watson. In fact, I get no error dialog whatsoever. Maybe this is a bug in x264 itself?

Also TSP, what version of x264 are you using? I am using 628.
I am using:
Colormatrix v2.1
Decomb 5.22
Avisynth= the version you had me download special
MT 0.7
AddGrain 0.1.0.0


Edit: Okay, here is the stack:
http://i34.photobucket.com/albums/d125/Merlin7777/x264stack.jpg

And here is the thread report:
ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1c5e
ntkrnlpa.exe!ZwYieldExecution+0x2540
ntkrnlpa.exe!NtWaitForSingleObject+0x381
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForMultipleObjects+0x18



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1c5e
ntkrnlpa.exe!ZwYieldExecution+0x2540
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1c5e
ntkrnlpa.exe!ZwYieldExecution+0x2540
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1c5e
ntkrnlpa.exe!ZwYieldExecution+0x2540
ntkrnlpa.exe!NtWaitForSingleObject+0x9a
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForSingleObject+0x12



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x381
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForMultipleObjects+0x18



ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!NtWaitForSingleObject+0x381
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb74
ntdll.dll!KiFastSystemCallRet
!WaitForMultipleObjects+0x18

[foxyshadis]

None of those do much good, because they don't include avisynth.dll (or anything related), as it is it's just a bunch of system calls related to paused threads. You can use shift-click to select them all before you copy though.

If kernel32.dll!WaitForSingleObject+0x12 or kernel32.dll!WaitForMultipleObjects+0x18 really are at the bottom of all the stacks, then they've been trashed by whatever crashed the program, or the stacks just can't be walked. =\ Unfortunately, that happens a LOT whenever frame pointers are disabled.

[Adub]

What would you have me do?

Oh, and I can't shift click the threads themselves before I view the stack, only the stack contents.

[tsp]

Merlin7777: You can't do much about it. I have created yet another special version of avisynth.dll that prints out the debug info in the program dbgview (http://www.microsoft.com/technet/sysinternals/utilities/debugview.mspx). If you could try running this version with dbgview running in the background. If the program stalls please PM the last 100 lines or so from dbgview.

The reason windows doesn't throw any error is that no fatal error has happened. You can read about the problem at the wikipedia here (http://en.wikipedia.org/wiki/Deadlock)

I'm using the same version of x268, decomb, addgrain and colormatrix as you are.

[edit]
ups here is the link to the new special version:
http://www.avisynth.org/tsp/a2.zip

[Boulder]

How does overlapping in MT work, that is, does it simply use the result of the other half of the frame or does it do some averaging? I'm asking this because RemoveGrain doesn't work on the very edges of the frame and I've thought about using a small, 2-4 frame overlap to compensate this when using MT.

[tsp]

it uses the result of the other half of the frame. So it should work with RemoveGrain

[TheRyuu]

This works wonders in AutoMKV.

Even wrote a wiki on it. (http://automkv.wiki-site.com/index.php/Advanced_Multi-threading)

In particular, it really makes the xvid first pass really fast. It use to only use like 50% CPU (on a dual core machine). MT makes it run like 100% on both cores for almost double the speed.

[canuckerfan]

quick question... would degrainmedian and TComb work under this syntax without causing artifacts/acting weird?

function f(clip c)
{
c
TComb()
degrainmedian()
}

mt("f()")

[tsp]

mt might confuse the screen change detection in TComb() else it should work

[3ngel]

I've tried right now the filter + the modified avisynth, and i report my experience

Using a fixed script, with the "temporal" mode with SetMTMode() i get small effect. The cpu stays around the 55%.

With the "spatial" mode MT(), the results is around 75%.

Is that normal?

But in the end very good work! :)

[tsp]

3ngel: it depends on the script and what program you are using the avs script with but usually the spatial mode is more efficient producing higher cpu utilization but the big question is if the script is faster with MT() than without it (more important than cpu utilization).

[3ngel]

With Mt() from 50% (whitout MT()) i go to 75%.

Whit SetMTMode() i go from 50 to 55.

But there is an important point using MT().

The frames resulting are not identical to that produced whitout MT(), this because my script uses a lot of masking and spatial/sharp, so the frames with MT appear to have more details lost (due to smaller window frame passed to the masks).

So beware!

[tsp]

3ngel: yes but what about the framerate? it doesn't all ways increase linearly with the cpu utilization. About the differences in the output try increasing overlap (like MT(overlap=4) ) but again with out the actual script used it is difficult to say what is causing it.

[3ngel]

About the framerate there is an increase of +-1 fps.

[canuckerfan]

hi, I've got a script here which I'm trying to speed up...

function hq_filter(clip c)
{
c=c.RemoveNoiseMC(rdlimit=7,rgrain=1,denoise=0,sharp=true)
c=c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=0,wide=false,ss_x=1.3,ss_y=1.3)
return c
}

setmemorymax(512)
mpeg2source("G:\Temporary\Scripts\VideoFile.d2v",idct=5,info=3)

Crop(0,56,0,-62)
Tweak(sat=1.07)

hq_filter()

a=last
b=a.MT("fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)")
SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)

AddBorders(0,56,0,62)
ConvertToYUY2()

i'm wondering if something could be done to speed up hq_filter, mpeg2source, and seesaw as a whole? I already got fft3d in MT since I've heard that works well...

[TheRyuu]

hi, I've got a script here which I'm trying to speed up...

function hq_filter(clip c)
{
c=c.RemoveNoiseMC(rdlimit=7,rgrain=1,denoise=0,sharp=true)
c=c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=0,wide=false,ss_x=1.3,ss_y=1.3)
return c
}

setmemorymax(512)
mpeg2source("G:\Temporary\Scripts\VideoFile.d2v",idct=5,info=3)

Crop(0,56,0,-62)
Tweak(sat=1.07)

hq_filter()

a=last
b=a.MT("fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)")
SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)

AddBorders(0,56,0,62)
ConvertToYUY2()

i'm wondering if something could be done to speed up hq_filter, mpeg2source, and seesaw as a whole? I already got fft3d in MT since I've heard that works well...

Why not just put SetMTMode(2, 0) right in the beginning.

Also, you can MT the entire SeeSaw script like so:
MT("a=last
b=a.fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)
SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)")

[tsp]

mt and setmtmode doesn't always work to good together. You can also try this variation:

function hq_filter(clip c)
{
return c.RemoveNoiseMC(rdlimit=7,rgrain=1,denoise=0,sharp=true)
}

function mt_filter(clip c)
{
c.LimitedSharpenFaster(Smode=4,strength=18,overshoot=0,wide=false,ss_x=1.3,ss_y=1.3)
fft3dfilter(ow=32/2,oh=32/2,sigma=1.5)
return SeeSaw(a,b, NRlimit=3, NRlimit2=4, Sstr=1.5, Slimit=5, Spower=5, Sdamplo=6, Szp=16)

}

setmemorymax(512)
mpeg2source("G:\Temporary\Scripts\VideoFile.d2v",idct=5,info=3)

Crop(0,56,0,-62)
Tweak(sat=1.07)

hq_filter()
last.mt("mt_filter()",overlap=2)

AddBorders(0,56,0,62)
ConvertToYUY2()

[0gg]

I really don't understand the syntax code for .avs ...

Can you post here a full avs script to test MT with x264 codec on my Dual Clovertown ?

I don't need crop, blur ... just :
desinterlace
720x576
the best image quality


Thank's a lot for your help

[3ngel]

I would like to a question about a thing i haven't understood well.

In normal mode (without MT) if i have

Filter1()
Filter2()

Filter2 request a frame and obtain the frame when Filter1() finishes.

But in "temporal mode", if i have the same

------
SetMTmode()
Filter1()

SetMTmode(thread=1)
Filter2()
------

in this case we have (in the minimum case) 2 parallel

Filter1(n) Filter1(n+1)

My doubt is:
If Filter1(n) finishes but Filter1(n+1) not, Filter2() request and obtain the next frame (n+1) (that at this point is not finished by Filter(n+1) an so it obtains an unprocessed frame), or waits until the second thread Filter1(n+1) finishes, and so obtain the correct (processed) n+1 frame?

Pheraps it's a basic question but i have a doubt :)

[foxyshadis]

When Filter2 requests frame n, Filter1 will roughly simultaneously process frames n and n+x, x in this case being the determined offset in requested frames (if you were playing that script forward, it would be 1). Filter2 will obtain n as soon as it's ready, whether n+x is ready or not. But n+x is not ready by the next access, it'll just wait around a little longer for it to finish, it never hands back an unprocessed frame.

[3ngel]

But n+x is not ready by the next access, it'll just wait around a little longer for it to finish, it never hands back an unprocessed frame.
Ok, that's what i wanted to know. That the "coherence" is mantained and the next filter waits (in the case) for the previous filter to complete and finish the requested frame before accepting it.
Thank you.

[TheRyuu]

Will the old 2.57 MT avisynth.dll work with the new 2.58a version of avisynth?

Or does that need a new dll?

I'd love to try out the new avisynth but I need the MT functions... :)

[Boulder]

It would require tsp to build a new MT-supported Avisynth version. That means the new release won't work.

[dchard]

Hi!

I want to multi-thread my avisynth script (because under encoding, only 55% of the CPU is used).

Finally the whole script is working except the multi threading. I read a lot, but I always got some errors.

I have a dualcire (yonah) cpu, and I always got 50% cpu useage.

This is the script I want to multi-thread:

#PLUGINS

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\warpsharp.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\Ylevels.avsi")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools.dll")



directshowsource("E:\Film\The Matrix\demuxed_tracks\matrix.grf",audio=false,fps=23.976,seekzero=false,seek=true).trim(0,195880)
converttoyv12()
crop(0,140,0,-140)


# denoise
removegrain(mode=1)

#sharpen
LimitedSharpenFaster(strength=40)

# YLevels usage:
# YLevels(0, 1.2, 255, 0, 255, false)
# YLevels(0, 1.2, 255)
YLevels(gamma=1.2)

# Tweak()
# hue=-180.0 to +180.0, default 0.0
# sat (0.0 to 10.0, default 1.0)
# bright (-255.0 to 255.0, default 0.0)
# cont (0.0 to 10.0, default 1.0)
# coring = true/false
# startHue (default 0), endHue (default 359): (both from 0 to 359; given in degrees.).
# maxSat (default 150), minSat (default 0): (both from 0 to 150 with minSat<maxSat; given in percentages).
# interp: (0 to 5, default 4)
tweak(cont=1.1)

#resizing

spline36resize(1280,544)

Please tell me what to do (what to install, what to write in the script etc).

Also, this script is for HD-DVD --> 720p reencode script, so anyone have any other idea, which I can improve the script, please tell me too. (ofcourse the values in the script are only for testing, not the final ones)

Thank you!

Dchard

[foxyshadis]

The first post should explain it all, but if you have the all-important avisynth.dll, you only need to add SetMTMode(2) to the start of the script. None of those filters appear to be incompatible.

[dchard]

Here is the multi-threaded script:

-----------------------------------

SetMTMode(2)

#PLUGINS

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\warpsharp.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\Ylevels.avsi")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Avisynth.dll")


directshowsource("E:\Film\The Matrix\demuxed_tracks\matrix.grf",audio=false,fps=23.976,seekzero=false,seek=true).trim(0,195880)
crop(0,140,0,-140)
converttoyv12()



# denoise
removegrain(mode=1)

#sharpen
LimitedSharpenFaster(strength=40)

# YLevels usage:
# YLevels(0, 1.2, 255, 0, 255, false)
# YLevels(0, 1.2, 255)
YLevels(gamma=1.2)

# Tweak()
# hue=-180.0 to +180.0, default 0.0
# sat (0.0 to 10.0, default 1.0)
# bright (-255.0 to 255.0, default 0.0)
# cont (0.0 to 10.0, default 1.0)
# coring = true/false
# startHue (default 0), endHue (default 359): (both from 0 to 359; given in degrees.).
# maxSat (default 150), minSat (default 0): (both from 0 to 150 with minSat<maxSat; given in percentages).
# interp: (0 to 5, default 4)
tweak(cont=1.1)

#resizing

spline36resize(1280,544)

-----------------------------

When I open it in megui, i got:

Script error: there is no function named "SetMTMode" (line 3)

What have I do in this case?

Dchard

[Boulder]

You'll need to extract the avisynth.dll from the MT v0.7 package into your Windows\System32 directory, overwriting the old one.

[dchard]

You'll need to extract the avisynth.dll from the MT v0.7 package into your Windows\System32 directory, overwriting the old one.

Oh, I don't know that.

Now its "working" (means I don't get any error messages), but its only uses about 50-60% of CPU time. The strange is, that both cores are used equally, but only a half. (this is a coreduo, yonah cpu with 2MB L2, and 2x512MB ddr2 in dualchannel.

Dchard

[Boulder]

In my tests, SetMTMode usually yielded less performance gain when compared to using MT(). However, in your case there's not many places where you can even use MT and they are not the bottlenecks of the script, far from it.

By the way, you probably don't want to load avisynth.dll in your script or even keep it in the plugins folder..it doesn't belong there;)

[dchard]

In my tests, SetMTMode usually yielded less performance gain when compared to using MT(). However, in your case there's not many places where you can even use MT and they are not the bottlenecks of the script, far from it.

By the way, you probably don't want to load avisynth.dll in your script or even keep it in the plugins folder..it doesn't belong there;)

Yes, I removed the avisynth.dll from the script, and form the folder :-)

This means that this is the maximum, that I can pull out from the CPU? Somewhere I read that when using avisynth multi-threading, it is normal that I can't see 100% load on cpu cores, but in reality, both cores working with full throttle.
This is true?

Dchard

[tsp]

dchard: try this version with uses mt instead of setmtmode()


LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\warpsharp.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\limitedsharpenfaster.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\Ylevels.avsi")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\Masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MT.dll")



directshowsource("E:\Film\The Matrix\demuxed_tracks\matrix.grf",audio=false,fps=23.976,seekzero=false,seek=true).trim(0,195880)
crop(0,140,0,-140)

mt("converttoyv12().removegrain(mode=1).LimitedSharpenFaster(strength=40).YLevels(gamma=1.2).tweak(cont=1.1)",border=4)

#resizing
mt("spline36resize(1280,last.height)")
mt("spline36resize(last.width,544)",splitvertical=true)

[Boulder]

You probably cannot wrap LimitedSharpenFaster inside MT..by default it does supersampling and you don't want to disable that;)

[tsp]

Boulder: it still does that (it's why i set the overlap to 4 to prevent artifacts at the border). Limitedsharpen works on each of the two parts the frame is split into and oversample each of them. The filters are allowed to change both height and width of the frame temporarily as long as the final frame only change one or none of them.

[Boulder]

Aha, so you don't need to split the resizing process internally in two MT instances in the LSF function..thanks, I didn't know that :)

[foxyshadis]

Try to measure the decode separately, and then the decode+full script, so you can rule out the encoding process as the actual bottleneck. If your source is some 30mbps HD movie, which it appears to be, the decoding might be single-threaded and taking up most of the remaining time, while the script is fully threaded.

[Blue_MiSfit]

Hmm good point. I wonder what decoder he's using in his filter graph.

Switch to a multithreaded decoder?

~MiSfit

[IanB]

directshowsource("E:\...\matrix.grf",...,fps=23.976...What format are you actually rendering?My ASF renders start fast and finish slow
Microsoft in their infinite wisdom chose to implement ASF stream timing in the ASF demuxer. As a result it is not possible to strip ASF format files any faster than realtime. This is most apparent when you first start to process the streams, usually after opening the Avisynth script it takes you a while to configure your video editor, all this time the muxer is accumulating credit time. When you then start to process your stream it races away at maximum speed until you catch up to realtime at which point it slows down to the realtime rate of the source material....

[Lele-brz]

Hello,
I'd like to use MT plugin in order to speed up performance on a Intel Xeon 8 core.
I use a Baseline profile in x264 where the input is this avs file:

DirectShowSource("C:\video\0601tTechBuzzGoogle_wx.wmv" , fps=30.0)
BicubicResize(448, 336, 0, 0.5)
ConvertToYV12()

sometimes I also use

SmoothDeinterlace()

What I noticed is that without the MT plugin the encoding time is very high compare to cheaper dual core machine, and the cpu usage is very low (I know I have to compare the time and not only CPU)

So I'd like to know how would you use MT with this script.

Thanks for any help

Bye

[pyrates]

Hello,
I'd like to use MT plugin in order to speed up performance on a Intel Xeon 8 core.
I use a Baseline profile in x264 where the input is this avs file:

DirectShowSource("C:\video\0601tTechBuzzGoogle_wx.wmv" , fps=30.0)
BicubicResize(448, 336, 0, 0.5)
ConvertToYV12()

sometimes I also use

SmoothDeinterlace()

What I noticed is that without the MT plugin the encoding time is very high compare to cheaper dual core machine, and the cpu usage is very low (I know I have to compare the time and not only CPU)

So I'd like to know how would you use MT with this script.

Thanks for any help

Bye

Your avisynth script isn't particularly cpu intensive. MT was designed to be used with filters where it is cpu intensive. You don't need it in this case. It is already passing the frames fast enough that the slowdown would be in the cpu cores instead. So far I've only seen x264 being tested on a 4 core system. I'm not sure how it would perform on an 8 core system, if it even can since that could slow it down.

What is the resolution of the .wmv file though that you are referencing? If it can be played back in realtime with only one core, then that isn't the bottle neck here.

[Zep]

Hello,
I'd like to use MT plugin in order to speed up performance on a Intel Xeon 8 core.
I use a Baseline profile in x264 where the input is this avs file:

DirectShowSource("C:\video\0601tTechBuzzGoogle_wx.wmv" , fps=30.0)
BicubicResize(448, 336, 0, 0.5)
ConvertToYV12()

sometimes I also use

SmoothDeinterlace()


like pyrates said you are not doing much at all avs side. your bottleneck is elsewhere. With 8 cores I would use high profile and max settings and make sure you use 12+ threads in x264 (8 will not max out that system)

[foxyshadis]

The wmv decoder is single-threaded, isn't it? So that'll be the bottleneck, which you can't do much about right now. >.>

[IanB]

@foxyshadis, No, the WMV decoder (splitter is to blame actually) is worse than single-threaded, it is rate constrained. For some dumb reason (DRM???) M$ implemented the stream timing in the splitter for ASF/WMV. End result is you cannot rip any faster than realtime.

[pyrates]

like pyrates said you are not doing much at all avs side. your bottleneck is elsewhere. With 8 cores I would use high profile and max settings and make sure you use 12+ threads in x264 (8 will not max out that system)

Actually just set the threads to auto in the x264 command line you are using. Then it'll automatically use the right number of threads every time regardless of how many cores you have.

[Zep]

Actually just set the threads to auto in the x264 command line you are using. Then it'll automatically use the right number of threads every time regardless of how many cores you have.

No auto is crap. it only will use 8 threads on an 8 core box and that is NOT good. You want 12+ threads which will then max your FPS/CPU usage.

[foxyshadis]

Zep, that was the old behavior, it was changed at the same time the threading was changed:
encoder/encoder.c

if( h->param.i_threads == 0 )
h->param.i_threads = x264_cpu_num_processors() * 3/2;


Ian, ouch. That's horrible.

[Lele-brz]

Hi guys,
thanks for all your answers, I don't want to go off topic since I'm realizing the bottle neck is elsewere.
Anyway I use the script to read any input format (mp4, wmv, mpg2....) supported by DirectShowSource so using ffdshow codecs.

What I realized is that on the 8 core the performance of encoding in x264 (I use a base profile) are very bad compare to a dual core (with -threads 2)

I tried with 8 threads and up to 20 but I couldn't see any improvements.

So probably is something related to x264 even though in
http://www.videohelp.com/forum/archive/megui-cpu-time-test-compare-different-cpus-encoding-the-same-file-t322687.html
there are some 8 core with good preformance.

thanks again

[chros]

@tsp: can you look at this post and a bit further ...
http://forum.doom9.org/showthread.php?p=1012965#post1012965

It seems there's something wrong with the modded avisynth.dll ...

[Zep]

Zep, that was the old behavior, it was changed at the same time the threading was changed:
encoder/encoder.c

if( h->param.i_threads == 0 )
h->param.i_threads = x264_cpu_num_processors() * 3/2;





still only uses 8 threads here with auto so something is borked.

I have to tell x264 via CLI or in MeGui to use 12 threads else it only uses 8. I'm using r659 but it has been this way always for me :)

[Zep]

What I realized is that on the 8 core the performance of encoding in x264 (I use a base profile) are very bad compare to a dual core (with -threads 2)

I tried with 8 threads and up to 20 but I couldn't see any improvements.

So probably is something related to x264 even though in
http://www.videohelp.com/forum/archive/megui-cpu-time-test-compare-different-cpus-encoding-the-same-file-t322687.html
there are some 8 core with good preformance.

thanks again


but did you use high profile and HQ slow settings? the correct thread count in your case is sorta moot since it will only help if there is lots to crunch and you are not doing much in x264 with low quality setting and such low rez video.

and of course if there is no bottleneck elsewhere like what IanB said about the splitter (BTW use a NON m$ splitter to get around that one. On my Mac I use flip splitter and it does not have that problem so there should be one on windows that is better than the m$ one also)

[Zep]

@foxyshadis, No, the WMV decoder (splitter is to blame actually) is worse than single-threaded, it is rate constrained. For some dumb reason (DRM???) M$ implemented the stream timing in the splitter for ASF/WMV. End result is you cannot rip any faster than realtime.

ok for the heck of it I just tested this and the m$ decoder/splitter does not rate constrain for me. I was able to encode using

DirectShowSource("D:\test.wmv")
BicubicResize(448, 336, 0, 0.5)
ConvertToYV12()

at 271 FPS to an xvid avi. the test.wmv is a wmv9 format. I tried a few more wmv files both older and newer and all encoded just as fast.

NOTE: graphedit shows WMvideo Decoder DMO. Now I have no idea if they are DRM protected or not and I do not feel like trying to find out but these encode fast :D

[tsp]

chros: I can confirm that media player classic crashes when avisynth MT is enabled in recent builds of ffdshow. I didn't happen with build 10xx. When I get visual c++ reinstalled I will try to see what is happening.

[jeffy]

chros: I can confirm that media player classic crashes when avisynth MT is enabled in recent builds of ffdshow. I didn't happen with build 10xx. When I get visual c++ reinstalled I will try to see what is happening.

Thank you for your reply and confirmation.

[IanB]

@Zep,

As I said it's the ASF splitter not the decoder. You are lucky if your default ASF splitter works correctly (desirably :D), many peoples do not.

I am just pointing out that there is (was?) a known problem and they may need to look at which splitter they have. DirectShow is such a bag of worms. :(

[pyrates]

No auto is crap. it only will use 8 threads on an 8 core box and that is NOT good. You want 12+ threads which will then max your FPS/CPU usage.

Perhaps you don't read the change log then, I shall illustrate on just how wrong you are:

r607 | pengvado | 2006-12-16 00:03:36 +0100 (Sat, 16 Dec 2006) | 13 lines

New threading method:
Encode multiple frames in prallel instead of dividing each frame into slices.
Improves speed, and reduces the bitrate penalty of threading.

Side effects:
It is no longer possible to re-encode a frame, so threaded scenecut detection
must run in the pre-me pass, which is faster but less precise.
It is now useful to use more threads than you have cpus. --threads=auto has
been updated to use cpus*1.5.
Minor changes to ratecontrol.

New options: --pre-scenecut, --mvrange-thread, --non-deterministic

[foxyshadis]

It's megui's fault, actually. The command line will say --threads auto, but the command it actually uses will be --threads x, where x is the # of detected cpus. That's a pretty nasty bug.

Edit: Ah, you can work around it by removing "automatically set number of threads" in options, until I get a patch in.

[pyrates]

It's megui's fault, actually. The command line will say --threads auto, but the command it actually uses will be --threads x, where x is the # of detected cpus. That's a pretty nasty bug.

good thing then I only run it directly from the command line. By the way, don't use --non-deterministic. It is buggy and crashy.

[Zep]

Perhaps you don't read the change log then, I shall illustrate on just how wrong you are:

r607 | pengvado | 2006-12-16 00:03:36 +0100 (Sat, 16 Dec 2006) | 13 lines

New threading method:
Encode multiple frames in prallel instead of dividing each frame into slices.
Improves speed, and reduces the bitrate penalty of threading.

Side effects:
It is no longer possible to re-encode a frame, so threaded scenecut detection
must run in the pre-me pass, which is faster but less precise.
It is now useful to use more threads than you have cpus. --threads=auto has
been updated to use cpus*1.5.
Minor changes to ratecontrol.

New options: --pre-scenecut, --mvrange-thread, --non-deterministic

First off READ WHAT I SAID. I said to fox that auto does not work and that I know internally what it does. geez mate where do you think I got 12 threads from? I used 1.5 x 8 then inputted that number manually. Get it now? I spelled it out and still you missed the main point.

second as foxs pointed out There is a bug JUST LIKE I SAID here let me repeat what I said "something is borked" and now we know what it is.

so just admit I was right and move on :D

[Zep]

@Zep,

As I said it's the ASF splitter not the decoder. You are lucky if your default ASF splitter works correctly (desirably :D), many peoples do not.

I am just pointing out that there is (was?) a known problem and they may need to look at which splitter they have. DirectShow is such a bag of worms. :(

haha well I have not tried .asf I doubt I even have any so not sure something is different in the splitting-->decoding them but all my wmv work at full speed.

graphedit didn't show the splitter per se'. it just showed RAW AUDIO 1 and RAW VIDEO 2 being spilt from and unnamed splitter and no info on the splitter in the main graph but I looked just now to see what splitter has highest merit and it is indeed the m$ one. I disable everything else and ran it again to make sure the m$ splitter was being used and it ran full speed. Now I'm pretty sure when you install WMP 11 you get the latest splitter and decoders so maybe those fixed that very bad cough feature cough :D

[pyrates]

First off READ WHAT I SAID. I said to fox that auto does not work and that I know internally what it does. geez mate where do you think I got 12 threads from? I used 1.5 x 8 then inputted that number manually. Get it now? I spelled it out and still you missed the main point.

second as foxs pointed out There is a bug JUST LIKE I SAID here let me repeat what I said "something is borked" and now we know what it is.

so just admit I was right and move on :D

Read my last post, I did that. But since I mainly encode directly from the command line, --threads auto has always meant number of cpu's * 1.5.

[Zep]

Read my last post, I did that. But since I mainly encode directly from the command line, --threads auto has always meant number of cpu's * 1.5.

yes I saw your reply to fox. Not exactly the same thing as replying to me :) Anyway, because you use CLI doesn't mean we all do so your post about going auto actually triggered the bug for Lele-brz if he is using MeGUI. yuck! lol

[TheRyuu]

Trying to MT a script:
LoadPlugin(...
Import(...
SetMemoryMax(768)

movie = mpeg2source("C:\VTS_04_1.d2v",info=3)
function getOrder(clip c) {
order = GetParity(c) ? 1 : 0
Return order }

movie = tfm(movie,d2v="C:\movie.d2v").tdecimate()
last = movie
crop(4,0,-4,-0)

#Resize
Spline36Resize(704,396)

#Filters
Combination of:
Removegrain, hqdn3d, LimitedSharpenFaster, aSharp, aWarpsharp, LineDarkenMOD


If I throw SetMTMode(2, 2) in the beginning it just makes it crash sometimes, although the crashing is random. If I put it right before Spline36Resize, it does work, AFAIK (although I'm not sure if that'll make it crash too), although I'm not sure if it's the best place to put it. Or, should I use MT()? Instead of SetMTMode?


Any suggestions how to do it?

[chros]

chros: I can confirm that media player classic crashes when avisynth MT is enabled in recent builds of ffdshow. I didn't happen with build 10xx. When I get visual c++ reinstalled I will try to see what is happening.
Thank you.
I hardly get crashing, but I always get the effect which was described the above links:
if one file is playing, then I try to play another, mpc doesn't play it: Failed to query the needed interface.

[TheRyuu]

BTW, on the script I posted above that I'm trying to get working with MT, I know that whenever you have "source" you have to use SetMTMode() right?

But why would adding it right in the beginning be causing xvidencraw to crash on me?
Or is that just because this "MT Mode" isn't stable?

[Leak]

Thank you.
I hardly get crashing, but I always get the effect which was described the above links:
if one file is playing, then I try to play another, mpc doesn't play it: Failed to query the needed interface.
Well, at least MPC just displays an error message.

Graphedit crashes right out as soon as you hit "File > New" after rendering a media file that uses ffdshow and AviSynth... :(

[tsp]

Leak:
Do you get any usefull debug/error output when running a debug build of ffdshow with my version of avisynth MT??
I have tried a debug version of avisynth MT but that doesn't produce any obvious indication of where the error is.

[Leak]

Leak:
Do you get any usefull debug/error output when running a debug build of ffdshow with my version of avisynth MT??
I have tried a debug version of avisynth MT but that doesn't produce any obvious indication of where the error is.
I'll give it a try - could you perhaps hook me up with your latest AviSynth source in case I have to debug into it? Your first post only has a link to the source for version 4...

Oh, and is there perhaps a chance of your MT code being included into AviSynth 2.5.8? (Just being curious there...)

[Deinorius]

Couldn't read the whole thread (too much).

I wonder, if there exists a list, which filters can be used with this or another MT-Mode.

[tsp]

Deinorius: try the mediawiki page:


You can also get futher help to MT at the mediaWIKI page here (http://avisynth.org/mediawiki/MT_support_page)


Leak: I have uploaded the source for version 5 here (http://www.avisynth.org/tsp/avisynth257MT5_src.7z).
The MT code will first be included in avisynth 2.6 to preserve stability in the current 2.5 release ;)

[Deinorius]

Oh thanks, didn't see that you wrote it there.

[chros]

@Leak and tsp: I'm using this modded version quite a long time (nowdays the v0.7), and as I recall the v0.5 has reproduced this error in some time, but the v0.7 does it always ! (maybe this helps)

[pyrates]

Anyone loading deen.dll, make sure SetMTMode(5,0) or (6,0) is used. If you use SetMTMode(2,0) it will crash the encoding when using x264.

[damngod]

Hi everyone,

I have a very simple question for you. I'm currently using two very small scripts for XviD and CCE encodings and i'm wondering if there would be a chance to improve speeds using mt.dll. So my CCE script is the following :

LoadPlugin("DGDecode.dll")
Mpeg2Source("video.d2v")
ConvertToYUY2()

For XviD encodings, i use GKnot. After cleaning the script, it looks something like :

LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\UnDot.dll")
mpeg2source("G:\video.d2v")
crop(2,62,714,356)
Undot()

From these, can i use mt.dll to improve my encoding speeds and if so, what lines do i need to add (MT(), SetModeMT()...) ?

Thanks in advance, best wishes.

DamnGod

[IanB]

... So my CCE script is the following :

LoadPlugin("DGDecode.dll")
Mpeg2Source("video.d2v") # A Source filter
ConvertToYUY2()LoadPlugin("DGDecode.dll")
Mpeg2Source("video.d2v", upconv=True)The most you could do here is get Mpeg2Source to directly output YUY2 to remove the output format conversion, this method also make use of the PROGRESSIVE flag to get better results. There is now nothing to thread.
For XviD encodings, i use GKnot. After cleaning the script, it looks something like :

LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\UnDot.dll")
mpeg2source("G:\video.d2v")
crop(2,62,714,356)
Undot()LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\UnDot.dll")
mpeg2source("G:\video.d2v") # A source filter
crop(2,62,714,356) # A zero cost filter
MT("Undot()", threads=?, overlap=1 ) # Unlikely improvementAs Undot is a very fast filter (almost as fast as a blit) any gain in threading it will likely be lost to the time the Blit that reassembles the parts takes. SetMTMode might give a minor improvment as long as there is no out of order accesses to the source filter.

Both case are to trivial to expect any real improvement.

[Boulder]

Replacing Undot() with RemoveGrain(mode=1) could give a very tiny boost, but it does fix one small Undot bug. The SSE2 or SSE3 version of RemoveGrain might be somewhat faster compared to Undot though.

[damngod]

Ok thanks guys. Never heard of the YUY2 "trick". I'll give it a try.

[sh0dan]

@TSP: Would it be possible to you to add a simple pre-fetcher?

I'm thinking along the lines of PreFetch(frames=5, priority=-1). The prefetcher is just a single thread that prefetches the frames from the filters above (in one separate thread) and delivers them to the requesting filters.

It would be very useful for non-threaded apps, like HC, avs2avi and similar, which all use linear access anyway. Other uses could be input filters that are struggling with IO.

If it proves to be useful it could even be added as a default filter on output on multicore machines. Since there is only one filter running at a given time, it should be 99% safe (since nothing is 100% safe).

[tsp]

sh0dan: Yes that should be possible. So the frames parameter should control how many frames ahead of the current requested frame that should be in the cache?

[IanB]

I must still be misunderstanding the code, because I thought this was how the code already currently worked. i.e. pre-rendering the next {thread count} frames all thru the graph.

Or is this a request to explicitly control the level of prefetch.

[tsp]

IanB: yes for the special case where thread count=1. I think the idea is that avisynth prefetch frames in a separate thread while the singlethreaded application does it work. As there are only 1 thread working most of the multithreading hassle is avoided while the singlethreaded application doesn't have to wait for the next frame.

[sh0dan]

The reason I would like it to be a single thread requesting mulitple frames are:

* It's 99% safe, and you don't get a performance penalty, except on non-linear access.

* The reason for multiple frames is, that most modern codecs don't request frames with the same intervals, since stuff as B-frames and similar often makes them request 3 or more frames without any processing inbetween.

Are you saying I can test this with MT right now?

I can see that MTSource does something similar, but I would like to be able to put this into the script at arbitrary places.

[tsp]

sh0dan: no it is not implemented in MT right. It will need a minor change of MTsource.

[Delerue]

I'm trying to use some MVTools scripts with MT, but no sucess. Do anyone know if it's my mistake, or MT simple can't work with MVTools? I'm using this script (inside the AviSynth tab inside the FFDShow) to increase the FPS :


SetMTMode(2)
loadplugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")

source=ffdshow_source()

backward_vec = source.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = source.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
return source.MVFlowFps(backward_vec, forward_vec, num=48, den=1, ml=100, idx=1)

Do I have to load the MT.dll? I mean this:


SetMTMode(2)
loadplugin("C:\arquivos de programas\avisynth\plugins\mt.dll")
loadplugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")

source=ffdshow_source()

backward_vec = source.MVAnalyse(isb = true, truemotion=true, pel=2, idx=1)
forward_vec = source.MVAnalyse(isb = false, truemotion=true, pel=2, idx=1)
return source.MVFlowFps(backward_vec, forward_vec, num=48, den=1, ml=100, idx=1)

The MT version of avisynth.dll is already inside the system32 folder, and I don't get any avisynth script error; just no dual-thread.

Sorry for any mistake, but I really tried to understand before posting.

Thanks

[Boulder]

MVTools is not multithreading compatible.

[Delerue]

MVTools is not multithreading compatible.

Very sad. This script is very useful but very heavy too. And my second core isn't doing anything... :(

Thanks anyway.

[Boulder]

What is your destination format? If you don't need to worry about filesizes (i.e. you use constant quality/quant), you could split the video in two halves and encode them simultaneously. I usually do this if I have a heavy script and have the diskspace to use a lossless intermediate file.

[Delerue]

What is your destination format? If you don't need to worry about filesizes (i.e. you use constant quality/quant), you could split the video in two halves and encode them simultaneously. I usually do this if I have a heavy script and have the diskspace to use a lossless intermediate file.

Hmmm... In my case I'm just watching (decoding). So it must be in real-time, of course. That's why I need more CPU power or dual-thread support.

[ChiDragon]

First time trying MT, I get the error: "AVI Import Filter error: (Unknown) (80040154)" when I open a script even as simple as "Version()" in VirtualDub. MPC gives a "cannot render the file" error. When I replace the modified avisynth.dll with normal 2.5.7 everything opens fine. I tried versions 4 and 5.

[tsp]

ChiDragon: Sounds like a missing dll file. Do you have msvcr71.dll in the windows\system32 path?

[ChiDragon]

I do actually, I also tried reinstalling the VC2005 SP1 redist with no luck (I formatted recently).

[ChiDragon]

Alright, I had msvcr71.dll but needed msvcp71.dll! I copied it from my Battlefield 2142 directory to system32 and it's working now.

Oddity to report though, using the newest MT & newest avisynth MT along with DGMPGDec 1.4.9 & VirtualDub 1.6.14. Script is just MTSource("""MPEG2Source("F:\test.d2v")"""), with test.mpg being a 1080i HD stream. CPU usage from Task Manager and FPS from VDub's video analysis pass rendering rate.

No MTSource: ~72% CPU, 38-42 fps
MTSource, open VDub and run analysis: ~92% CPU, 2-30 fps mostly in the low 10s
MTSource, open VDub, CTRL+G, frame 1000, run analysis: ~74% CPU, 40-48 fps until just before it gets to frame 1000, then back to ~92% CPU, 2-30 fps

I've repeated each of these a few times and get the same results always. When looking at the graphs I can see that the DGDecode analysis alone uses ~50% of core 0 and ~95% of core 1 (prior to running, utilization is at ~5% so nothing else major is running). Does this mean that it is actually multithreaded already, or is VDub using half of a core just to deal with the null AVI data?

EDIT: Forgot to mention, this is with A64 X2 4600+.

[Leak]

Exactly what's the idea in MT'ing a source filter?

Are you aware that would read the file twice, decode the whole frame twice (since you can't tell a source filter "only decode lines n to m") and then build a final frame by combining the top of one frame and the bottom of the other? (I do think this doesn't even happen since there's no input clip, but meh...)

In other words - don't do that. The only sensible solution here is to multithread the decoder itself, if at all possible. MT only makes sense when using existing clips that can be split up into parts, not for creating new clips out of thin air - or data from disk, for that matter...

np: Funckarma - Fedwick (Bion Glent)

[ChiDragon]

Leak, I'm not quite that dumb. :p From the MT doc:

"from version 0.7 two other filters are included too:"
<snip>
" * MTsource() that are used to run source filters multithreaded. It works like this:

function PseudoMTsource(string filter)
{
SetMTmode(2)
eval(filter)
SetMtmode(0)
}

So different from the two other filters it is a temporal filter that fetches frames ahead of time and store them in the cache for fast retrieval."

[foxyshadis]

Threading a source filter can never gain you anything if you have no weight on the output side, whether it's avisynth filters or a slow encoder. All you have are synchronization slowdowns, and I wager there are a lot of deadlocks showing up.

In virtualdub, threading source filters without more filtering is further useless because vdub already has avisynth on its own thread, pulling new frames while old ones are encoded. (x264 and some others do the same thing.) So the only good it'll do you is if you have enough filters in the script to make a speed impact.

Determining the cause of the deadlocks is still worthwhile though.

[ChiDragon]

Sorry, most of that went right over my head... I thought multithreading the source would help when it's the bottleneck as opposed to the processing filters? Perhaps a diagram? :confused:

[Leak]

Sorry, most of that went right over my head... I thought multithreading the source would help when it's the bottleneck as opposed to the processing filters? Perhaps a diagram? :confused:
"Multithreading the source" in this case is running the decoder in it's own thread. That means there's frames being decoded while the rest of the script runs in parallel.

If the rest of the script is doing nothing then all you do is still only decoding frames. In a single thread. On the other core, though...

MTSource can't magically make a singlethreaded decoder multithreaded. Instead, it runs decoding and filtering in parallel to give 2 cores something to do, which of course can make things faster - just not in your testcase here.

np: Apparat - Fractales Pt.I (Walls)

[ChiDragon]

"Threads" defaults to 2, so I figured it's decoding frames using both cores. What does it mean then?

[Leak]

"Threads" defaults to 2, so I figured it's decoding frames using both cores. What does it mean then?
Well, I suppose it could open the same file several times in different threads and fetch several frames at once, but I'm honestly not sure if that's going to work too well, at least when you have a codec where frames depend on each other.

It could work if you start a thread at every keyframe (although I'm not sure how to get that information from a source filter) or with keyframe-only formats like HuffYUV or MJPEG, but probably not too well with MPEG4 ASP or H.264 with several hundred frames of max keyframe distance...

np: Retina.IT - Zucchine Alla Scapece (Semeion)

[Tanma]

I'm posting just to say thanks for the MT filter, it works nice with most of plugins =)

[Gackt]

Hi,
I've got a small problem with MT. Here is my avs script:


Import("limitedSharpen.avs")
LoadPlugin("deen.dll")
LoadPlugin("masktools.dll")
LoadPlugin("textsub.vdf")
directshowsource("sample_movie.avi", fps=23.976, pixel_type="YV12")
SetMTMode(1)
deen("a3d",3,6,1,6)
LimitedSharpen(Lmode=2)
textsub("sub.ass")
SetMTMode(3) #other modes does not work :(
splash_video=ImageSource("splash.png").LanczosResize(704,396).ConvertToYV12()
splash=AudioDub(splash_video,BlankClip()).convertfps(last.framerate).trim(0,150)
SetMTMode(6) #other modes does not work :(
final=splash+last
subtitle(string(GetMTMode()))
return final

Thanks to MT, the firsts filters (deen, limitedshapen and even textsub) are working very fine with SetMTMode(1). But the problem is that MT seems to hate 'merging' movies :mad:. Why ?
So, i load my script with virtualdubMod, i play the video with it, and... it works only until the end of "splash" movie; i mean: the video plays fine, but it crashed 2-3 frames after the start of "last" (the main "sample_movie.avi" movie), so at ~153 frames. If i replace the line "final=splash+last" by "return last" (to return the entire "sample_movie.avi" file), it works very well in mode 5-6 or 2.

My specs are:
c2d e6600
2Gb of ram...etc

Anybody had a fix for this ? :thanks:


And sorry for my very bad english :rolleyes:

[Revgen]

SetMT should come before directshowsource.

[chros]

Thank you.
I hardly get crashing, but I always get the effect which was described the above links:
if one file is playing, then I try to play another, mpc doesn't play it: Failed to query the needed interface.
Any progress with this?

Thanks

[Serbianboss]

I didnt watch long time this thread about MT, so i have few questions. Two version changed since i last visited this thread.

This version is for avisynth 2.57(i have build: Dec 31 2006)


Does with latest version(MT 0.7) i can use SetMtMode for interlaced material? (DV avi files) Because i see new MTi() mode or is better to use MTi() mode for interlaced material?

[mroz]

MT's working fine in my script /except/ when I use it as input to mencoder. In that case while there are no errors, cpu usage never goes above 25% (on a Q6600) & I get 3fps.

I'm running Megui (for x264 encodes) & doing a pre-rendering step with a slow avisynth script, hence reliance on MEncoder to manage the huffyuv encode.

If I play my script in the likes of Zoomplayer or VDub, I get 100% cpu & 11fps. I can even use VDub to perform the lossless huffy encode at 11fps. It's just MEncoder that's giving me the problem.

I've asked in the Megui troubleshooting thread (http://forum.doom9.org/showthread.php?t=105920&page=72) with no response yet & also on the MEncoder Users mailing list (where I get told to forget about bothering with any tool other than MEncoder & also that I should obviously be asking the MT dev/s what the problem is).

Any ideas?

I have this issue if I run MEncoder from the command line, even though it shows as having affinity with all four cores.

Shouldn't I get over 25% cpu even if MEncoder does the encoding to huffy in a single thread, as ought not the avisynth script still use multiple cores? In any case, this happens even if I modify the mencoder command line to perform the encode using multiple threads.

[Adub]

No, I am pretty sure that Mencoder is doing what it is supposed to do. I ran a few test a while back doing pretty much the same thing, and I never did get it to go above 50% (dual core). I think it has something to do with the way Mencoder calls avisynth.

Your best bet, IMHO, is to split the script up into 4 different sections, encode to huffyuv, then join and feed to your encoder using a new avs script.

[mroz]

I read your post/s discussing this. There's a thread on the mencoder users mailing list now. No solution or even conclusive explanation, but one of the devs Reimar Döffinger has posted a couple of times which is mildy hopeful if I can keep his attention.

His first suggestion was as you said that it was due to SetMTMode no longer being the first instruction in the avs, as MEncoder handles the avs by creating a script environment via the avisynth.dll api, passing it an Import(<script path>) call & then if necessary a ConvertToYV12(), effectively wrapping the script in another.

However this is easy to test by doing precisely the above & giving it to VDub. It didn't impact performance. So that's almost certainly not it.

My suspicion is that it's down to in some sense inheritance of the single threaded nature of MPlayer, but I have no tech knowlege in respect of multithreaded coding in Windows. NB to avoid any confusion, MEncoder will run the encode step multithreaded if told to do so & the encoder supports it (eg in the cases of x264 or huffy), but this isn't relevant here; what matters is that the decoder of the input, which is MPlayer code, isn't multithreaded.

It's just the extent to which it matters that I don't know, nor how easy it would be to fix, purely in respect of invoking external multithreaded decoders.

That Reimar hasn't immediately said it won't work because of x, y & z gives me a little hope.

Aside: if this limitation can't be solved within Megui, isn't inclusion of MEncoder & the pre render step redundant for all dual/quad core users?

In this case, shouldn't Megui be looking at fixing MEncoder or replacing it with an alternative? Any options/objections in this respect?

I like Megui, so very much want to see this issue fixed.

As to splitting the script, given the hassle & that it's not automated, I might as well use VDub for the pre-render step, which is my current solution.

[IanB]

At a guess it sounds like you are not getting the Distributor module added to your graph.

Try adding the following to the end of your script....
Distributor()
InternalCacheMT()

[tsp]

IanB: Sounds like it is what is happening. The distibutor is only inserted if the avs file is opened as an AVIFile.

[mroz]

IanB, tsp: I assume such a script might no longer open in other contexts, so wasn't too surprised when trying to open it in Zoomplayer & Megui gave me an error:
Evaluate - System exception - Access Violation reported via Avisynth.

Unfortunately, trying to offer it to MEncoder as input also resulted in an error:

E:\Work\test>mencoder.exe "E:\Work\test\fm-test.avs" -o "E:\Work\test\hfyu_fm-test.avi" -of avi -forceidx -ovc lavc -lavcopts vcodec=ffvhuff:vstrict=-2:pred=2:context=1
MEncoder Sherpya-SVN-r24537-4.2.1 (C) 2000-2007 MPlayer Team
CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (Family: 6, Model: 15, Stepping: 11)
CPUflags: Type: 6 MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1
Compiled for x86 CPU with extensions: MMX MMX2 SSE SSE2

success: format: 0 data: 0x0 - 0x743
============ Sorry, this file format is not recognized/supported =============
=== If this file is an AVI, ASF or MPEG stream, please contact the author! ===
Cannot open demuxer.

Exiting...


Glad you seem to have ideas as to what the problem might be though. Anything else you'd like me to try?

Edit: probably a nonsense even to try but... with just the Distributor call at the end of the script, there's no error but also no change in performance (*); with the InternalCacheMT the error always results.

Edit2: Re (*) above - Oops, yes there is. See my next post below.

[tsp]

mroz: could you try adding subtitle(string(GetMTmode())) to the end of the script just before distributor() and report back what number it displays to see if the mode is different from 0

[mroz]

Yesterday I tried as an afterthought appending just the Distributor call & reported it made no performance difference (which I didn't look into further as I assumed the InternalCacheMT call was indeed a requirement & as I said, that was crashing MEncoder).

I've just done as you requested which led to a few other checks.

:stupid:

Yesterday either something very odd was happening or more likely, I did something stupid such as not resaving the script after uncommenting the Distributor call, since...

The mode reported by your line is indeed 2 /and/ with the Distributor call as the last line in the script, I do indeed get 100% cpu out of MEncoder, both when invoked via the commandline & via Megui.

:cool: :cool: :cool: :thanks:

Thanks for the workaround :)

Is there any problem with having that line inserted at the end of the script if it is then invoked in any other context?

What would be a better solution? Can AVISynth/MT fix this or would it need a patch to MEncoder to notice MT is in use & call Distributor?

[tsp]

Is there any problem with having that line inserted at the end of the script if it is then invoked in any other context?

yes you would get double the number of threads leading to lower performance if the avs file was opened as an avi file

What would be a better solution? Can AVISynth/MT fix this or would it need a patch to MEncoder to notice MT is in use & call Distributor?
MEncoder would need to check what MT mode in use and if it was between 1 and 4 it should insert Distributor() at the end of the script. It is difficult for avisynth to know when the script is fully loaded as there are no "end of script" command.

[TheRyuu]

Why do I get horizontal lines going across when I use the following:
MT("awarpsharp(depth=8, cm=1)",3)

This only persists with awarpsharp.
Is it just a compatibility problem with awarpsharp were nothing can be done? (does any mt mode work with it?)
Or is awarpsharp just a filter were what it does prevents it from being multi-threaded.

I'm just curious why it does what it does. awarpsharp is plenty fast enough without MT avisynth.

Thanks.

[foxyshadis]

Because awarpsharp does strange things on frame borders. You need to add a couple of pixels of overlap, which is the argument after # of threads. Watch out for green frames when using old awarpsharp in chroma mode, as well.

[mroz]

yes you would get double the number of threads leading to lower performance if the avs file was opened as an avi file
Understood, thanks.
MEncoder would need to check what MT mode in use and if it was between 1 and 4 it should insert Distributor() at the end of the script. It is difficult for avisynth to know when the script is fully loaded as there are no "end of script" command.
Difficult, practically impossible or impossible?

What about when the first frame is requested? Does that indicate the script is complete? If so, is it too late then to insert Distributor()?

As the issue is specific to MT usage, I imagine you'd agree the best place for a fix would be in MT/Avisynth, as that would benefit any other code which accesses avs files in this way.

Second best would be a patch to MEncoder & any other similar code. I don't suppose it's a sufficiently simple task to propose a patch to MEncoder, bearing in mind MT might not be in use at all, that you have the time to make a suggestion here & now? I'll append the relevant code from MEncoder to this post, just in case.

If neither of these are options for any of the devs concerned, from the point of view of Megui I imagine either they need to patch their own MEncoder build, or simply copy & modify the avs script before invoking MEncoder.

Any comments would be appreciated. I'll feed them back to the MEncoder Users list & the Doom9 Megui troubleshooting thread (unless the relevant people are reading this).

Thanks again for all the help.

Here's the MEncoder avs demuxer code:

MPlayer-1.0rc1 > libmpdemux/demux_avs.c


static demuxer_t* demux_open_avs(demuxer_t* demuxer)
{
int found = 0;
AVS_T *AVS = (AVS_T *) demuxer->priv;
AVS->frameno = 0;

mp_msg(MSGT_DEMUX, MSGL_V, "AVS: demux_open_avs()\n");
demuxer->seekable = 1;

AVS->clip = AVS->avs_take_clip(AVS->handler, AVS->avs_env);
if(!AVS->clip)
{
mp_msg(MSGT_DEMUX, MSGL_V, "AVS: avs_take_clip() failed\n");
return NULL;
}

AVS->video_info = AVS->avs_get_video_info(AVS->clip);
if (!AVS->video_info)
{
mp_msg(MSGT_DEMUX, MSGL_V, "AVS: avs_get_video_info() call failed\n");
return NULL;
}

if (!avs_is_yv12(AVS->video_info))
{
AVS->handler = AVS->avs_invoke(AVS->avs_env, "ConvertToYV12", avs_new_value_array(&AVS->handler, 1), 0);
if (avs_is_error(AVS->handler))
{
mp_msg(MSGT_DEMUX, MSGL_V, "AVS: Cannot convert input video to YV12: %s\n", avs_as_string(AVS->handler));
return NULL;
}

AVS->clip = AVS->avs_take_clip(AVS->handler, AVS->avs_env);

if(!AVS->clip)
{
mp_msg(MSGT_DEMUX, MSGL_V, "AVS: avs_take_clip() failed\n");
return NULL;
}

AVS->video_info = AVS->avs_get_video_info(AVS->clip);
if (!AVS->video_info)
{
mp_msg(MSGT_DEMUX, MSGL_V, "AVS: avs_get_video_info() call failed\n");
return NULL;
}
}

// TODO check field-based ??

/* Video */
if (avs_has_video(AVS->video_info))
{
sh_video_t *sh_video = new_sh_video(demuxer, 0);
found = 1;

demuxer->video->sh = sh_video;
sh_video->ds = demuxer->video;

sh_video->disp_w = AVS->video_info->width;
sh_video->disp_h = AVS->video_info->height;

//sh_video->format = get_mmioFOURCC(AVS->video_info);
sh_video->format = mmioFOURCC('Y', 'V', '1', '2');
sh_video->fps = (float) ((float) AVS->video_info->fps_numerator / (float) AVS->video_info->fps_denominator);
sh_video->frametime = 1.0 / sh_video->fps;

sh_video->bih = (BITMAPINFOHEADER*) malloc(sizeof(BITMAPINFOHEADER) + (256 * 4));
sh_video->bih->biCompression = sh_video->format;
sh_video->bih->biBitCount = avs_bits_per_pixel(AVS->video_info);
//sh_video->bih->biPlanes = 2;

sh_video->bih->biWidth = AVS->video_info->width;
sh_video->bih->biHeight = AVS->video_info->height;
sh_video->num_frames = 0;
sh_video->num_frames_decoded = 0;
}

#ifdef ENABLE_AUDIO
/* Audio */
if (avs_has_audio(AVS->video_info))
{
sh_audio_t *sh_audio = new_sh_audio(demuxer, 0);
found = 1;
mp_msg(MSGT_DEMUX, MSGL_V, "AVS: Clip has audio -> Channels = %d - Freq = %d\n", AVS->video_info->nchannels, AVS->video_info->audio_samples_per_second);

demuxer->audio->sh = sh_audio;
sh_audio->ds = demuxer->audio;

sh_audio->wf = (WAVEFORMATEX*) malloc(sizeof(WAVEFORMATEX));
sh_audio->wf->wFormatTag = sh_audio->format = 0x1;
sh_audio->wf->nChannels = sh_audio->channels = AVS->video_info->nchannels;
sh_audio->wf->nSamplesPerSec = sh_audio->samplerate = AVS->video_info->audio_samples_per_second;
sh_audio->wf->nAvgBytesPerSec = AVS->video_info->audio_samples_per_second * 4;
sh_audio->wf->nBlockAlign = 4;
sh_audio->wf->wBitsPerSample = sh_audio->samplesize = 16; // AVS->video_info->sample_type ??
sh_audio->wf->cbSize = 0;
sh_audio->i_bps = sh_audio->wf->nAvgBytesPerSec;
sh_audio->o_bps = sh_audio->wf->nAvgBytesPerSec;
}
#endif

// I imagine this is the right place to make the mode check & call Distributor if necessary? <<<<<<<-----------------------------

AVS->init = 1;
if (found)
return demuxer;
else
return NULL;
}

[mroz]

Something like this?

--- demux_avs.c 2006-10-22 23:32:31.000000000 +0100
+++ demux_avs_modded.c 2007-10-03 01:45:45.359375000 +0100
@@ -236,6 +236,7 @@
int found = 0;
AVS_T *AVS = (AVS_T *) demuxer->priv;
AVS->frameno = 0;
+ AVS_Value threads = avs_new_value_bool(0); // we want to read mode, not threads, from GetMTMode

mp_msg(MSGT_DEMUX, MSGL_V, "AVS: demux_open_avs()\n");
demuxer->seekable = 1;
@@ -333,6 +334,21 @@
}
#endif

+ /* Ensure compatability with MT filter */
+ // nb I think defs for avs_is_int & avs_as_int need to be added to demux_avs.h
+ AVS->handler = AVS->avs_invoke(AVS->avs_env, "GetMTMode", avs_new_value_array(&threads, 1), 0);
+ if (avs_is_int(AVS->handler)) // assume error implies MT not in use (it shouldn't error otherwise & there's not much we can do about it - opinions?)
+ {
+ int mode = avs_as_int(AVS->handler);
+ if (mode>0 && mode<5) {
+ // unsure abt following line for several reasons:
+ // 1. Does Distributor have any args or return value - eg & importantly, the video clip (I can't find docs on this) ?
+ // 2. How are zero args for Distributor passed to invoke? As below or can the avs_new_value_array be replaced by 0 ?
+ AVS->handler = AVS->avs_invoke(AVS->avs_env, "Distributor", avs_new_value_array(0, 0), 0);
+ // Should we handle an error? What could we do with it in any case?
+ }
+ }
+
AVS->init = 1;
if (found)
return demuxer;


Comments very welcome as this evening is the first time I've looked at any avisynth related code & I've not written any C for, um, about a decade.

Afterthought: Is it possible to check if Distributor has already been called? If this isn't checked, MEncoder is patched as has been suggested & /then/ the issue is worked around in MT/Avisynth so that the patch isn't needed, the patch would result in doubled thread count. Something safer would be desirable if possible.

Another query, for scripts not using MT, what will be the overhead of the error returning GetMTMode call? Negligible I'd assume, but figured I ought to check.

[IanB]

You should be catching IScripEnvironment::NotFound exceptions from Invoke. "GetMTMode" and "Distributor" are not available in non-MT avisynth.dll's

[mroz]

You should be catching IScripEnvironment::NotFound exceptions from Invoke. "GetMTMode" and "Distributor" are not available in non-MT avisynth.dll's
Ah, so that won't just result in the function call returning an error then? I wasn't sure.

Exceptions in C? I've no idea how that's handled in code using Kevin Atkinson's AviSynth C Interface. I'll have to stop meddling & hope someone with a clue steps in.

Edit: this (http://kevin.atkinson.dhs.org/avisynth_c/api.html) & the MEncoder avs demuxer code is all I'm holding, so I guess I have to fold :(

[tsp]

AviSynth C Interface will return AVS->handler of type 'e' (Error) with the value "Function Not Found" if GetMTMode is not available. Distributor takes 1 argument, the input clip(AVS->clip) (if it is not specified invoke will use 'last') and returns an AVS_Value of type clip (that should replace AVS->clip).

[IanB]

Yes in the "C" interface you just need to test the return value for the error state.(if it is not specified invoke will use 'last')No it will not!

ScriptEnvironment::Invoke (See avisynth.cpp@1482) optionally applies argument naming and then calls the ApplyFunc, i.e. the Creator function, for the named function.

To get standard script lexography with respect to Last you need to Invoke("Eval", args) where args is a script string expressing your intent.

[mroz]

Could you possibly comment as to whether the following looks sound?


--- demux_avs.c 2006-10-22 23:32:31.000000000 +0100
+++ demux_avs_modded.c 2007-10-04 01:42:59.875000000 +0100
@@ -20,6 +20,7 @@

#include <stdio.h>
#include <stdlib.h>
+#include <string.h>
#include <unistd.h>

#include "config.h"
@@ -236,6 +237,8 @@
int found = 0;
AVS_T *AVS = (AVS_T *) demuxer->priv;
AVS->frameno = 0;
+ AVS_Value mt_threads = avs_new_value_bool(0); // we want to read mode, not threads, from GetMTMode
+ AVS_Value mt_clip;

mp_msg(MSGT_DEMUX, MSGL_V, "AVS: demux_open_avs()\n");
demuxer->seekable = 1;
@@ -333,6 +336,45 @@
}
#endif

+ /* Ensure compatability with MT filter */
+ // nb I think defs for avs_is_int, avs_as_int, avs_new_value_bool & avs_new_value_clip need to be added to demux_avs.h
+ AVS->handler = AVS->avs_invoke(AVS->avs_env, "GetMTMode", avs_new_value_array(&mt_threads, 1), 0);
+ if (avs_is_error(AVS->handler))
+ {
+ if (strcasecmp(avs_as_string(AVS->handler), "Function Not Found")) // if no match, we have a problem error...
+ {
+ mp_msg(MSGT_DEMUX, MSGL_V, "AVS: GetMTMode failed: %s\n", avs_as_string(AVS->handler));
+ return NULL;
+ } // ...else can just conclude MT is not in use & skip remainder of patch
+ }
+ else if (avs_is_int(AVS->handler))
+ {
+ int mode = avs_as_int(AVS->handler);
+ if (mode>0 && mode<5)
+ {
+ mt_clip = avs_new_value_clip(AVS->clip);
+ AVS->handler = AVS->avs_invoke(AVS->avs_env, "Distributor", avs_new_value_array(&mt_clip, 1), 0);
+ // Since GetMTMode is supported, an error now isn't expected to be 'Function Not Found' & so indicates a problem
+ if (avs_is_error(AVS->handler))
+ {
+ mp_msg(MSGT_DEMUX, MSGL_V, "AVS: Cannot invoke MT Distributor: %s\n", avs_as_string(AVS->handler));
+ return NULL;
+ }
+ AVS->clip = AVS->avs_take_clip(AVS->handler, AVS->avs_env);
+ if(!AVS->clip)
+ {
+ mp_msg(MSGT_DEMUX, MSGL_V, "AVS: avs_take_clip() failed\n");
+ return NULL;
+ }
+ }
+ }
+ else
+ {
+ mp_msg(MSGT_DEMUX, MSGL_V, "AVS: Unexpected return type from GetMTMode\n");
+ return NULL;
+ }
+ /* End of 'compatability with MT filter' patch */
+
AVS->init = 1;
if (found)
return demuxer;


I can't test it as I don't have a build environment set up, but if it looks okay I can offer it as a patch when I post further about this issue on the MEncoder list. Hopefully that will increase the chances of a fixed build being forthcoming.

[IanB]

Seems on the right track. I would have assumed if there is any problem with either "GetMTMode" or "Distributor" just assume a non-MT .dll and ignore the MT patch. i.e. fail as soft as possible.

Remember that users are very devious and some smarta... might put in a .avsi which overloads either "GetMTMode" and/or "Distributor". You should aim to behave as closely as possible to the unpatched code in aberant conditions. Also some other smarta... (me!) might improve the text of the error messages in a future revision. ;)

[mroz]

Seems on the right track. I would have assumed if there is any problem with either "GetMTMode" or "Distributor" just assume a non-MT .dll and ignore the MT patch. i.e. fail as soft as possible.

Remember that users are very devious and some smarta... might put in a .avsi which overloads either "GetMTMode" and/or "Distributor". You should aim to behave as closely as possible to the unpatched code in aberant conditions. Also some other smarta... (me!) might improve the text of the error messages in a future revision. ;)

Understood. I did hesitate over that decision & my concern that an unreported error might lead to problem behaviour that was then harder to track down pushed me this way. However I agree with your concerns & bow to your experience, so I'll simplify it to ignore those errors.

In any case I've just noticed avs_new_value_clip calls avs_set_to_clip which needs to be imported & currently isn't, so that needs a few additions too.

I'll post a revised version later & then mail the MEncoder Users list with the suggestion.

Thanks again.

[mroz]

I thought I'd be done by now, but there's one more thing I'm unsure of. Sorry. Would you mind taking another quick look?

I made the mistake of looking at the relevant avisynth source & giving myself a headache ;)

The bit I'm concerned about is does the avs_set_to_clip require an avs_release_value after the avs_invoke on Distributor?


--- demux_avs.c 2006-10-22 23:32:31.000000000 +0100
+++ demux_avs_modded.c 2007-10-04 20:17:41.640625000 +0100
@@ -50,6 +50,7 @@
typedef WINAPI AVS_Value (*imp_avs_invoke)(AVS_ScriptEnvironment *, const char * name, AVS_Value args, const char** arg_names);
typedef WINAPI const AVS_VideoInfo *(*imp_avs_get_video_info)(AVS_Clip *);
typedef WINAPI AVS_Clip* (*imp_avs_take_clip)(AVS_Value, AVS_ScriptEnvironment *);
+typedef WINAPI void (*imp_avs_set_to_clip)(AVS_Value *, AVS_Clip *);
typedef WINAPI void (*imp_avs_release_clip)(AVS_Clip *);
typedef WINAPI AVS_VideoFrame* (*imp_avs_get_frame)(AVS_Clip *, int n);
typedef WINAPI void (*imp_avs_release_video_frame)(AVS_VideoFrame *);
@@ -80,6 +81,7 @@
imp_avs_invoke avs_invoke;
imp_avs_get_video_info avs_get_video_info;
imp_avs_take_clip avs_take_clip;
+ imp_avs_set_to_clip avs_set_to_clip;
imp_avs_release_clip avs_release_clip;
imp_avs_get_frame avs_get_frame;
imp_avs_release_video_frame avs_release_video_frame;
@@ -112,6 +114,7 @@
IMPORT_FUNC(avs_invoke);
IMPORT_FUNC(avs_get_video_info);
IMPORT_FUNC(avs_take_clip);
+ IMPORT_FUNC(avs_set_to_clip);
IMPORT_FUNC(avs_release_clip);
IMPORT_FUNC(avs_get_frame);
IMPORT_FUNC(avs_release_video_frame);
@@ -236,6 +239,8 @@
int found = 0;
AVS_T *AVS = (AVS_T *) demuxer->priv;
AVS->frameno = 0;
+ AVS_Value mt_threads = avs_new_value_bool(0); // we want to read mode, not threads, from GetMTMode
+ AVS_Clip *mt_clip;

mp_msg(MSGT_DEMUX, MSGL_V, "AVS: demux_open_avs()\n");
demuxer->seekable = 1;
@@ -333,6 +338,28 @@
}
#endif

+ /* Ensure compatability with MT filter */
+ // nb defs for avs_is_int, avs_as_int, avs_new_value_bool need to be added to demux_avs.h
+ AVS->handler = AVS->avs_invoke(AVS->avs_env, "GetMTMode", avs_new_value_array(&mt_threads, 1), 0);
+ if (avs_is_int(AVS->handler)) // On other (unexpected) return types & errors (an error probably just means MT not in use), fallback to non_MT behaviour
+ {
+ int mode = avs_as_int(AVS->handler);
+ if (mode>0 && mode<5)
+ {
+ AVS->avs_set_to_clip(&AVS->handler, AVS->clip); // do I need an avs_release_value() when I'm done with this to avoid screwing up
+ // internal IClip ref count (which would mean not overwriting it below) ?
+ AVS->handler = AVS->avs_invoke(AVS->avs_env, "Distributor", avs_new_value_array(&AVS->handler, 1), 0);
+ // Fail silently on any error, falling back to non-MT behaviour
+ if (!avs_is_error(AVS->handler))
+ {
+ mt_clip = AVS->avs_take_clip(AVS->handler, AVS->avs_env);
+ if (mt_clip)
+ AVS->clip = mt_clip; // only update AVS->clip when we know this has worked, else leave it in pre patch state
+ } // nb I'm assuming AVS->handler doesn't need similar preservation - it seems to be used only in this fn & to pass clip into this fn from initAVS
+ }
+ }
+ /* End of 'compatability with MT filter' patch */
+
AVS->init = 1;
if (found)
return demuxer;

[foxyshadis]

I just realized that I knew how to make mvtools work multithreaded and I just never put two together. To wit:

mt("""
idx = rand()
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
""")

Works in setmtmode(2) as well, and with the other mvtools tools. I r teh ultimate nub. (Of course the chance of a collision increases as threadcount goes up. Still pretty remote.)

So I request from tsp: A global variable with a unique value for each thread. The system thread-id might work! There's no hope for Dust, since you can't pass an idx in, but I can modify all of momonster's and other conditional functions to take advantage of it too.

[Boulder]

I guess you need to use some overlapping in MT there, MVAnalyse doesn't seem to detect motion well when searching near the edges of the frame?

EDIT: it would also be nice if tsp could update his build to use up-to-date code :)

[mroz]

I just realized that I knew how to make mvtools work multithreaded and I just never put two together. To wit:

mt("""
idx = rand()
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
""")

Works in setmtmode(2) as well, and with the other mvtools tools. I r teh ultimate nub. (Of course the chance of a collision increases as threadcount goes up. Still pretty remote.)

So I request from tsp: A global variable with a unique value for each thread. The system thread-id might work! There's no hope for Dust, since you can't pass an idx in, but I can modify all of momonster's and other conditional functions to take advantage of it too.

Now I'm confused (again). So currently mvtools shouldn't normally work with SetMTMode(2)? What error does one see?

Only I've been using the likes of
SetMTMode(2,0)
# Set DAR in encoder to 37 : 20. The following line is for automatic signalling & comes from initial script creation in Megui>Tools>AVISynthScriptCreator
global MeGUI_darx = 37
global MeGUI_dary = 20
DGDecode_mpeg2source("E:\Work\test\FM.d2v", info=3)
ColorMatrix(hints=true,interlaced=true)
#Not doing anything because the source is progressive
#crop( 0, 0, 0, 0)
#LanczosResize(720,576) # Lanczos (Sharp)
#denoise

# from: http://forum.doom9.org/showthread.php?t=119486&page=2
# Now we do the clever & slow deblocking/denoise, followed by debanding
# nb most of the debanding dither will be lost in a subsequent x264 encode, but still need this step as it helps reduce blocking in any case
source = last
backward_vec2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
mask = mvmask(kind=1, vectors=forward_vec1).UtoY().spline36resize(source.width(), source.height())
smooth = source.degrainmedian(mode=3).fft3dfilter(bw=16, bh=16, bt=3, sigma=4, plane=0)
source2 = mt_merge(source, smooth, mask)
source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3.gradfun2db(1.5)

without problems.

[Fizick]

foxyshadis,
interesting workaround!
But seems, idx is assigned at parse and constructor stage.
For some reason MT() uses single thread at this stage.
Many threads are used at Getframe.
I will try make some internal chages in MVTools
(like adding CurrentThreadId to idx internally).
MT() should works.
I am not sure about SetMtmode.

[mroz]

Can someone please briefly summarise (or link to info on) the problems in trying to use MT with MVTools?

I assume it must involves some artifacts rather than a crash or error, since as I said above, I've been using the two together in the quoted script & didn't realise there was a problem.

Thanks.

[Spuds]

@mroz

I'll take a shot at answering you question and let the folks who really understand this correct me where I've gone into the weeds.

The problem is the idx parameter which is an index pointer to previously computed frame data. What this does is significantly speed up the MV processing by saving computed data so that it can be reused by the next instance of the function.

In your particular script you are looking at a frame X and deciding what the motion vectors are based on the previous two and next two frames. When doing that you save data with idx so each successive call to mvanalyze does not have to start at ground zero but can use data from the previous call.

Now when you run MT you now have two frames in play, X and Y. Y is kind of X+1 or X/2 depending on how you called MT. Anyway the important thing is that you now have two different frames trying to use the same index and therefore data but the data is only correct for one of the frames not both. So all the computations you are doing can and are being applied to the wrong base frame/data. How that will manifest itself I'm not sure, could be artifacts or completely missing data.

Running without the idx will be slower with MT then running with idx without MT. Running with idx and MT is something I hope Foxyshadis and Fizick can figure out !

[ChrisW77]

How that will manifest itself I'm not sure, could be artifacts or completely missing data.

A couple of weeks ago I tried this out on some Fraps-captured video of a side scrolling videogame I was playing, and while at the time it seemed like MT had worked, when it came to playing back the video you could clearly see a ghost of the previous frame overlaid onto the current and next frames.
This was obviously more evident because it was videogame material, and you could quite easily see portions of the graphics kind of floating around.
Take out MT, and it went away completely.

[mroz]

@spuds

Thanks, that makes sense. Thus artifacts are far more likely than a crash. Oh joy.

Thing is, while I didn't closely view the entire output, I did /very/ closely view sections of it (as in using an avs script to stack the source next to it & step through several minutes of footage both in realtime & frame by frame) :confused:

@ChrisW77

Ah yes, use some footage with simple motion, great idea...

/Goes & checks credits in some material previously processed...

I still can't find obvious unwanted artifacts.

Chris, were you using SetMTMode(2) or MT()? Given one processes frames in parallel & the other parts of the one frame in parallel, I imagine artifacts would occur in both cases but be quite different.

@Anyone

This is asking a lot, but if anyone has some cpu cycles & time to kill, any chance you could run the script I quoted over a short sample in order to confirm it does screw up & in that case what kind of artifacts one sees? Even better would be if the test was with a clip I can also access, so I can check I can repeat the results myself.

Of course one of my problems is that I don't understand my own script well enough to anticipate how artifacts might emerge when the wrong data is used :stupid:

[foxyshadis]

I've been testing, because I couldn't tell when you first asked. I can't really tell, since the artifacts could be subtle - one wrong frame with high metrics can be covered up by the other good frames. But I haven't really come across any really glaring errors with SetMTMode, so it may be that it really does work okay.

MT() makes it painfully obvious, but they do use very different strategies.

[Fizick]

I do not understand, how Setmtmode(2) should work? odd frmes by one thead, even thead by second thread?
If so, you will not get any artifactes, but simply some more different (bad) vectors and denoising.

Somebody must encode some short clip to two uncompressed Avi (with and without Setmtmode). And then compare Avis by subtract().

second question. Is any speed gain with setmtmode?

[mroz]

@Foxyshadis

Thanks for the feedback. I suppose depending on the script it could conceivably simply not do it's job yet not introduce glaring artifacts, as a best/worst case (depending on ones point of view - personally I'd rather have realised/noticed sooner).

@Fizick

I'm using a quad core, so there'll be four threads, however I think your description is essentially correct, although I'm guessing there's no guarantee the allocation between threads would necessarily be precisely 1,2,3,4,1,2,3,4,1,2,3,4,... if for some reason one takes significantly longer than another.

Of course it never occurred to me to try subtract. Will run a short test in a few minutes.

Speedwise, the script I quoted on a dvd resolution huffy pre-render within Megui, gives me nearly 3fps without SetMTMode & just over 11fps with SetMTMode(2,0), thus the speed gain is very nearly linear in the number of threads & cores.

NB Due to a problem in the way Mencoder handles avs input, one must append Distributor() as the list line to the script before giving it to Mencoder. This doesn't apply if using VDub for an encode, say.

[ChrisW77]

I used both, but not in the same script.

SetMTMode(2,0)

and

MT("
",2,2)

The problem I had with MT, is where to put the ",2,2) ?
I used it on Interlace material, and wasn't sure whether to put it before or after Weave()

second question. Is any speed gain with setmtmode?

I had quite a huge gain using it, sometimes almost double, going from say 8fps MT off to 16fps on.

[Fizick]

to be clear, I say about test of some simplest script with mvtools, without any other filters. for example

backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)

[mroz]

Just tested a 9s clip from a nasty dark & foggy noisy scene. Didn't need to use subtract - the files were the same sizes so I checked md5 hashes; they're identical.

I'd question the test, but I just sat here & watched one encode at 10.4fps & the other at 2.9fps.

Comments?

@Fizick: Can you suggest a simple MVTools using script which should create problems with SetMTMode if there are any to be seen?

Edit: Ah, I see you just have. Will use that, thanks. Be back in a few minutes.

[mroz]

Here's my script:

SetMTMode(2,0)

DGDecode_mpeg2source("E:\Work\test\FM.d2v", info=3)
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=1)

__film = last
__t0 = __film.trim(120776, 121000)
__t0

Distributor()


For the comparison I comment out the first & last lines.

I've ran the above & then repeated with a different clip from the same dvd.

The first gives me a speedup of factor 3.6395 & the second, 3.6398 (around 24.5fps vs 6.5fps).

In each case, the pair of output files are identical.

[foxyshadis]

I ran a longer test with a very fast-moving scene and a very high threshold, still no difference except in speed with setmtmode.

[IanB]

global idx = 123
mt("""
global idx = idx + 1
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
""", overlap=8)As MT "Eval"'s the command with a different input clip for each thread this should work as the Rand() version does without the indeterminacy. Also this probably wants a good amount of overlap so any motion vectors near the join won't get truncated.

[Spuds]

I spent some time this AM with an VHS copy that had a horse running back and forth, good motion and start stop action for MV to chew on. Cut it down to 2500 frames (standard 720x480i) and encoded to an uncompressed format.

I ran the following

backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
""")
last

in several variations, including idx set to 1, idx set to rand(). non mt mode, with setmtmode(2,2) and inside an MT wrapper. I then took each output and subtracted from the original non MT run of the above code.

In terms of speed:

non MT -- 9:19
setmtmode(2,2) -- 5:02
setmtmode(2,4) -- 4:45 (seen this before where this is faster on my core 2 duo
MT() -- 5:18
MT() idx = rand() -- 5:24
setmtmode(2,2) idx=rand() -- 5:01


Here are some examples of delta frames from the subtract operation.
SETMTMODE(2,2) idx=1
http://img518.imageshack.us/img518/4784/mt22artifactcopywm8.jpg
an occasional frame was bad

SETMTMODE(2,2) idx=rand()
http://img518.imageshack.us/img518/6686/mt22randartcopylf6.jpg
same as above, maybe a few less frames but sporadic bad ones

mt() idx=1
http://img98.imageshack.us/img98/3497/mtartifactcopytz8.jpg
This was a mess, 80% of the frames (bottom half) had deltas

mt() idx=rand()
http://img98.imageshack.us/img98/4400/mtrandartcopyks1.jpg
This one was really strange, it was like watching a faint ghost image the entire video, this is just one frame I enhanced in photoshop so you could see all deltas. watching the subtract clip was like seeing an faint embossed image.

setmtmode(2,4) idx=1
http://img98.imageshack.us/img98/2629/mt24artifactcopybm1.jpg
lots of these to be found

[mroz]

How big is the source 2500 frame clip? Any chance you can make it available on some file sharing service?

What was the lossless format you encoded to?

[Spuds]

I used the original lossless format called uncompressed RGB/YCbCr for my output :) The file is pretty large as is >1G

I did some more testing with the same video but this time ran it through yadif to get a progressive source to see it that made a difference in the above.

Doing the same experiment as above with a progressive source same number of input frames but double framerate. I could not find a delta with setmtmode and idx=1. MT with idx=1 was just a mess, MT with idx=rand was better but pretty much the same as with the interlaced source, lots of noise removal deltas between the frames.

I'll probably go back and do a longer progressive comparison with setmt and see if I can spot a problem.

[Fizick]

Spuds, what is your full scipts, including source filter?
Is any differences with SetMTmode before source or after it?

[Boulder]

SetMTMode should always be before the source is loaded, otherwise it won't work properly.

[Spuds]

Here is the script I was running .... obviously I commented and uncommented as required depending on what run I was doing. For kicks I ran it again and setmode(2,4), mt idx=1, mt idx=rand() all repeated as above, setmtmode(2,2) did not :devil:. Strange, the first time with setmtmode(2,2) there were 2 or 3 frames out of 2500 that were different, granted thats less than 1/10 of 1% but still.


setmtmode(2,2)
SetMemoryMax(1024)
Plugin = "C:\Program Files\AviSynth 2.5\plugins\"
Scripts = "C:\Program Files\AviSynth 2.5\plugins\scripts\"
Videos = "C:\Documents and Settings\All Users\Documents\My Videos\"
#load_stdcall_plugin(plugin + "yadif.dll")

setmtmode(5)
avisource(videos + "colt test.avi")
#assumebff
#yadif(mode=1,order=-1)
setmtmode(2)

#mt("""
#idx=rand()
idx=1
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
#""")


I just ran the below script figuring two forward and backward vectors might be more interesting:

#setmtmode(2,2)
SetMemoryMax(1024)
Plugin = "C:\Program Files\AviSynth 2.5\plugins\"
Scripts = "C:\Program Files\AviSynth 2.5\plugins\scripts\"
Videos = "C:\Documents and Settings\All Users\Documents\My Videos\"
#load_stdcall_plugin(plugin + "yadif.dll")

#setmtmode(5)
avisource(videos + "colt test.avi")
#assumebff
#yadif(mode=1,order=-1)
#setmtmode(2)

#mt("""
#idx=rand()
idx=1
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
backward_vec2 = MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec2 = MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx)
#""")

and found a couple of these in setmtmode(2,2) vs no mt mode
http://img87.imageshack.us/img87/6705/mvgrain2ls0.jpg

[mroz]

Any chance you can compress your source & repeat the tests with that material? Then if they're still the same, which one would expect, upload the source somewhere?

I've only experimented with SetMTMode(2,0) but have now tried interlaced sources; still can't find a difference between SetMT & no SetMT.

/grasping at straws

Are you using the latest versions of the filters?

MT 0.7 1/3/7 and mvtools 1.8.4 22/8/7 here.

[Fizick]

what about setmtmode(3,2) ?

[MfA]

I've been thinking ... what's the point in multithreading for something like video processing filters? There is no finegrained parallelism (there might be inside the filter, but that's another story) and the amount of code which has to be replicated for everything to run in it's own process isn't that big a deal either.

Code wise starting extra processes and passing objects through shared memory is a little more work ... but you are freed from filters having to be thread safe.

[foxyshadis]

A complex script (such as anything with mv or mcbob) can easily consume massive amounts of memory, which limits what you can do with multiple processes. It's easy to run out with two script - sometimes just one, heh. The easy way of multiprocessing can already be done, but requires huge temporary lossless files, which makes a lot of people think twice; it's harder to make it work with stats files, bitrate distributions, and so on.

Squid_80 is making a multiprocess version of avisynth to handle 64/32 bit, which could likely be extended to multiple 32-bit processes more performant than tcpserver.

[MfA]

If all the actual framebuffers are shared through shared memory (memory mapped pagefile in windows) the only extra memory you take compared to multithreading is basically an extra instance of the avisynth core for every process ... which in the grand scheme of things doesn't matter much.

[IanB]

Gees guys, take a minute and think about this.

If you are going to use the IDX option in Mvtools with multithreading then you need a scheme to ensure that every thread has a unique IDX value.

It is easy to test if your current idea is working as expected by putting a SubTitle(String(IDX)) in appropriate places in your script. None of the SetMtMode's are going to allow this.

[foxyshadis]

Ian, Spuds is just testing how well it works without doing that, since mroz and I both got good results with it - your solution is probably the most elegant, now that it's been proven to fail.

MFA, shared memory sections = race conditions, which is no better than we have now. Everything has to be made threadsafe at either the plugin or the scripthost level anyway.

Fizick, as long as you're here :p have you ever thought of creating the idx clip by cloning vi with x2 or x4 height/width and requesting blank frames from this new clip? This way it will be cached and can be limited by setmemorymax - the downside being that if it ever gets limited, it'll drop performance, but not nearly as much as swapping to disk. Since it's a global, it may have lifetime issues that using it in a single class doesn't, I don't know nearly enough about avisynth's smart pointers to know.

[MfA]

MFA, shared memory sections = race conditionsEverything is shared memory with multithreading ...
Everything has to be made threadsafe at either the plugin or the scripthost level anyway.
You could get away with adding a single global lock to make the scripthost thread safe (since there could only be internal threading inside a single instance of a filter for a given scripthost you could fundamentally not cause deadlock that way AFAICS).

Since the plugins would never run as threads themselves they would obviously not need to be thread safe.

[foxyshadis]

If you have a mutex/semaphore (global lock), all that's needed to deadlock is one process not giving up the lock, instead of one thread. Kill the process and it goes away, of course, same as killing a thread for a critical section.

x264farm deals with a similar issue by also running a monitoring process that will kill and restart hung processes where they left off.

More complexity, especially since you don't actually gain anything: The vast majority of avisynth plugins are threadsafe as-is, so is the core now, and there's a workaround for the biggest exception. You're just rewriting MT avisynth in a different way, with a more time-consuming synchronization primitive.

However, I could see it being useful for a 64-bit operating system with a dozen gigs of memory going to waste in the single 32-bit process you need to run your plugins in. Then you want all the processes you can get, and don't even need synchronization.

[Fizick]

foxyshadis,
of course I considered (and is considering) to make idx clip with MVPrepare. It is in to do list (to kill idx).

IanB,
No, every thread should not to have a unique IDX value,
with proper locking.

[MfA]

If you have a mutex/semaphore (global lock), all that's needed to deadlock is one process not giving up the lock
The common meaning in CS of deadlock means a situation where to give up a lock a process needs to acquire the lock in some roundabout way ... you need some type of loop in the call graph. A lock never being given up because of a livelock (ie. infinite loop or something) is different from a deadlock.

I freely admit with already so much effort being put in enabling multithreading multiprocessing doesn't make too much sense anymore ... but still in the end it's a poor design. Why share a memory space when it has no performance benefits and only enables a multitude of very subtle bugs?

I know it's a bit of an appeal to authority ... but still worth mentioning, Avery Lee considers making filters thread safe enough of a pain he fundamentally wants to never let any run in parallel with itself.

[IanB]

@Fizick,

Well we don't have any Script level locking syntax. And if you put locking in your filter, then you will probably end up single tasking it. The real problem is the whole IDX concept, it is unAvisynth. All comunication between modules in MV should be thru the IClip interface with the Avisynth CACHE providing all the buffering needed, backdoor hacks like private buffers and IDX numbers make the whole design fragile.

[Fizick]

IanB, i say about SetMtMode(), not about MT().
if idx is unique for every thread, we will have duplicate caclulation
Two ways:
1. If we preserve indx syntax, I (we or somebody) can implement internal three-state indicator of "interpolated idx frame":
- not-exist (may be calculated and written right now by current thread),
- ready to read (may be read by current thread),
- currently updated by other thread (current thread must wait to read).

2. Yes, it is better to remove idx. It is my dream too. But is need in more work (and some tricks again).
I need to send some parameters about vectors. Avisynth still does not have normal way to trasfer additional parameters (metadata). When it will be implemented? I do not see it even in Avisynth 2.6. Currently I consider to use audio parameters as metadata. I will try analyse and formulate suggestions in MVTools thread.

Final notes.
I (and you) do not have dual-core anyway, but probably you can add some part of AvisynthMT to version 2.5.8? I say about minimal changes to support MT (and MTi ?) filter only (it is rather stable) as a first step.
How big chages are need?
SetMTmode may be added later, in v2.6.0.
May be tsp can comment this.

[mroz]

Another post from the ignorant here: Can anyone (Fizick?) explain why with the current code, SetMTMode(2) & MVTools isn't produing frequent obvious corruption? Only Spuds has seen that & even then with SetMTMode(2,0) found very little corruption at most. Surely if the processing is using cached data from completely the wrong frames around 1/2 to 3/4 of the time (roughly & depending on number of cores/threads) we should see obvious artifacting?

Can I also recap - the SetMTMode(2,0) with global idx = idx+1 workaround...

** Ought this to work around any problem resulting in artifacting, but at the expense of forcing MVTools to perform duplicate calculations? Have I got that right? **
Edit: Sorry, **'s rubbish isn't it - other than when including idx = idx+1 within the string passed to MT, this will be of no benefit whatsoever. Hence no help in the SetMTMode only case, yes?

I welcome a proper fix, of course, but if we're currently talking not until AVISynth 2.6, a short term hack is needed, if it's possible at all.

[Fizick]

mroz,
Fizick officially does not support MT with MVTools.
It is under development, as well as MT itself.
Probably most time two thead spend similar time for processing, i.e. work almost in synchro.

[tsp]

IanB, i say about SetMtMode(), not about MT().
if idx is unique for every thread, we will have duplicate caclulation
Two ways:
1. If we preserve indx syntax, I (we or somebody) can implement internal three-state indicator of "interpolated idx frame":
- not-exist (may be calculated and written right now by current thread),
- ready to read (may be read by current thread),
- currently updated by other thread (current thread must wait to read).

I could give it a try.


Final notes.
I (and you) do not have dual-core anyway, but probably you can add some part of AvisynthMT to version 2.5.8? I say about minimal changes to support MT (and MTi ?) filter only (it is rather stable) as a first step.
How big chages are need?
SetMTmode may be added later, in v2.6.0.
May be tsp can comment this.
If SetMTmode/GetMTmode shouldn't be supported it would only be necessary to implement most of the changes in avisynth.cpp/.h and cache.cpp.

[Fizick]

I created separate thread (multi-threaded discussion! :)) about developing MVTools without idx
http://forum.doom9.org/showthread.php?t=131033

[mroz]

mroz,
Fizick officially does not support MT with MVTools.
It is under development, as well as MT itself.
Probably most time two thead spend similar time for processing, i.e. work almost in synchro.

That's a shame, though good to know development is on going.

I'm not sure what you're saying by the last sentence. I just know on a quad core I get nearly four times the performance from MVTools intensive scripts when I use SetMTMode. Perhaps more importantly, that speed up makes scripts practical that otherwise I couldn't afford to make frequent use of (I know it's personal, but for me, 11 fps is acceptable for a prerender, while 3 fps isn't generally).

I'll subscribe to your new thread & will jump in if/when there are any releases needing testing, though I imagine this won't be for some time.

Edit: still very puzzled why MVTools/SetMTMode(2) seems to work for some, including myself. I'd welcome any comments as to what might be going on internally & am happy to conduct any testing if helpful.

[IanB]

@mroz,

Simply put it's a crap shot, i.e. roll the dice and pray.

MVTools stores internal state so when accessed in a normal sequential manner it can reuse previous calculations to save time. Using the IDX parameter allows different MVAnalyse() instances to share the same calcs if available.

If you don't set IDX then you cannot share the calc so there can be no speed up but then they cannot be stuffed up.

If the accesses are not sequential then the state is again useless and a full recalc happens. No stuff ups here.

Things go wrong when the shared calculations are not appropriate. Even without an IDX hitting the same MVAnalyse instance twice at just the right time can confuse things.

[mroz]

@mroz,

Simply put it's a crap shot, i.e. roll the dice and pray.

MVTools stores internal state so when accessed in a normal sequential manner it can reuse previous calculations to save time. Using the IDX parameter allows different MVAnalyse() instances to share the same calcs if available.

Understood - btw your post in the 'MTools without idx' thread was most informative as to some of the internal workings of both avisynth & mvtools; I'll keep reading & maybe some will sink in.

If you don't set IDX then you cannot share the calc so there can be no speed up but then they cannot be stuffed up.

Do you mean if it wasn't used in that way or literally that calling MV<whatever> without an idx parameter will avoid many of the problems at the expense of speed? Only I get no significant speed difference here, though have only run one quick test.

If the accesses are not sequential then the state is again useless and a full recalc happens. No stuff ups here.

Understood.

Things go wrong when the shared calculations are not appropriate. Even without an IDX hitting the same MVAnalyse instance twice at just the right time can confuse things.

Indeed. That makes perfect sense & is highly plausible. I imagine my failure to understand why artifacts aren't more common is simply as I don't have much understanding of a typical execution profile for this process. Naively it still feels to me that corruptions should be a common occurence, not rare.

Thanks for the feedback.

[Spuds]

@morz,

I can't say it much better than that. All I can say is from my testing is that I've had several 'clean' runs, of about 1.5min of footage that is LOL. As I sat there watching the mind numbing pure grey screen of the subtract output of those runs, every now and then I'd catch an occasional hideous frame.

thats the way he likes it ... Well thats the way he gets it ... I don't like it anymore than you men ....

[IanB]

The area of code that is vunerable is quite short, the code that does all the hard work is literally millions of time longer. So shooting snake eyes is more likely. However low the chances are not zero so you will get a dud frame eventually.

[Fizick]

What if multithreded SetMTMode(2,2) is on?
I do not know exactly, probably it use one thread for even output frames, and second thread for odd frames.
In example above, the "idx clip" some frame may be calculated by else even thread or odd thread.
Buffer size is limited, so sometimes it's memory is overwritten.
IMO it is the MOST DANGEROUS OPERATION (around line 493 of MVFrame.cpp).
If one thread is reading frame from buffer (it is not momentary process), and at same time other thread is overwriting this frame buffer (by other frame), we will get mess. Probability is not high (time to read is small relatively to other calculations).
If this unknown value is about 1/10000, then every 1 from 10000 frames is bad, i.e. 10 bad frames per 1 hour.
IMO it is dependent on mumber MV functions in script, number of CPU (thread) and on buffer array size.
Probably it may depend on some other processes in system.

Partial workaround: increase constant MV_BUFFER_FRAMES array size. Now it is 10.
Here is test versions with different array sizes (4,5,6,15):
http://avisynth.org.ru/tmp/mvtools4.dll
http://avisynth.org.ru/tmp/mvtools5.dll
http://avisynth.org.ru/tmp/mvtools6.dll
http://avisynth.org.ru/tmp/mvtools15.dll

Please test.

Better workaround: use counter of threads (or functions) accessed the frame, like smart pointers. Increment it while access, decrement it when leave.

May be try use variable array size and increment it if all current buffers are filled (locked).

[tsp]

What if multithreded SetMTMode(2,2) is on?
I do not know exactly, probably it use one thread for even output frames, and second thread for odd frames.
it is more or less random which thread that will process the next frame (depending on both thread are ready to process a new frame or only one is ready)

[mroz]

@Fizick Will run your tests, but was thinking, as one needs to process a large number of frames & look for differences in the output it makes sense to write a script to count the deviant frames.

/goes & reads about conditional filters et al

Think I know enough now.

Just started my baseline non-SetMTMode huffy encode of your simple script applied to a film of 144,700 frames, which should show something up. I'll follow this later today with SetMTMode(2,0) & the same script via first the current MVTools then the above versions.

Great to see this being worked on now btw - I imagined the problems would be put to one side for weeks if not months.

Edit: This comparison script look ok?

leaf1="noSetMT"
leaf2="SetMT-v2"
global c1=AVISource("E:\Work\test\hfyu_MVMT-test-"+leaf1+".avi")
global c2=AVISource("E:\Work\test\hfyu_MVMT-test-"+leaf2+".avi")
global total=0
file="E:\Work\test\compare-"+leaf1+"-"+leaf2+".log"
Subtract(c1,c2)
WriteFileStart(file, """ "vim:tabstop=6"+chr(10) """, """ Time("%#c")+chr(10) """, """ "Frame"+chr(9)+"Total"+chr(9)+"Variation" """, append=true)
#nb compare against 0.0001 instead of 0 to allow for rounding errors - 2 identical files will report a variation of 0.000022
WriteFileIf(file, "variation>0.0001", "current_frame", "chr(9)", "total", "chr(9)", "variation")
WriteFileEnd(file, """ "Test complete."+chr(10) """)
ScriptClip( "global variation = LumaDifference(c1,c2)+ChromaUDifference(c1,c2)+ChromaVDifference(c1,c2)"+chr(13) \
+"global total = total + ((variation>0.0001) ? 1 : 0)"+chr(13) \
+"Subtitle(String(variation))" \
)

[mroz]

First result for the original MV_BUFFER_FRAMES=10

vim:tabstop=6
Tuesday, October 23, 2007 12:58:52
Frame Total Variation
15858 1 1.816673
15859 2 3.181405
15861 3 0.210538
Test complete.

So at least I can now see an error - three close frames out of nearly 150,000.

@Fizick:

Before I test with the modified MVTools, can I just check with you, will they all produce bitwise identical output when run with no SetMTMode? If so, I only need run the SetMTMode variants & compare to my baseline avi, which will cut the time for each run from nearly 8 hours to only 1.5 hours.

BTW The links you posted to the dlls are broken - you missed the s off the end of each mvtool.

[Fizick]

mroz,
Thanks for fast testing. (I corrected a links).
Yes, all MVTools without SetMTMode must produce identical results (only speed may vary a little).

Modifications is not big, it is not a solution of the problem, but some indicative test.

[tsp]

Fizick: a possible (temporary) solution could be to use a smart pointer to MVGroupOfFrames that correctly decremented the refcount of MVGroupOfFrames when it goes out of scope and only allow recycling of MVGroupOfFrames that has a refcount of 0. Currently only IncRefcount is implemented for MVGroupOfFrames.

[Fizick]

tsp,
yes, and buffer size must be changed from constant (it may be not enough) to variable.

[tsp]

Fizick: I will try implementing the MVGroupOfFrames smartpointer and a variable buffersize

[tsp]

Ok a new version that incorporates the above suggestion is ready. Please try it and see if it improves anything when used with setmtmode. You can get it here:
http://www.avisynth.org/tsp/mvtoolsMTcomp.zip

[mroz]

mroz,
Thanks for fast testing. (I corrected a links).
Yes, all MVTools without SetMTMode must produce identical results (only speed may vary a little).

Great, that'll save lots of time :)

Modifications is not big, it is not a solution of the problem, but some indicative test.

Understood but I appreciate that work is being done.

Here are my corresponding results for your MV_BUFFER_FRAMES=6 dll

vim:tabstop=6
Tuesday, October 23, 2007 17:53:11
Frame Total Variation
11035 1 0.015687
11037 2 0.018902
11038 3 0.248984
11086 4 0.349582
11090 5 0.972039
15095 6 1.939990
15098 7 0.280023
16865 8 0.721465
16866 9 0.812497
16889 10 0.670940
16890 11 1.173558
17302 12 0.038992
17305 13 0.505871
17306 14 0.719297
18118 15 0.089515
20351 16 0.502222
20354 17 0.210579
22854 18 0.376654
24835 19 1.557945
24838 20 0.165073
28903 21 0.197361
28906 22 0.091904
29180 23 0.232695
29182 24 0.322055
29183 25 0.373993
30282 26 0.510847
30424 27 0.019494
30426 28 0.110264
30798 29 0.373128
30802 30 0.260479
31510 31 0.228801
33562 32 0.211502
33565 33 0.082752
33657 34 0.125897
33670 35 1.098033
33673 36 0.058188
35916 37 0.302044
37413 38 0.024555
37416 39 0.248225
40088 40 0.365724
41457 41 0.024742
41460 42 0.077433
42949 43 0.379954
42950 44 0.346376
43619 45 0.728546
43622 46 0.182506
44745 47 0.629474
44746 48 0.399838
46131 49 0.021676
46134 50 1.285427
48111 51 0.603243
48114 52 0.082384
48194 53 0.166780
51767 54 0.038720
51770 55 0.071487
52195 56 0.209815
52198 57 0.053900
52971 58 0.046540
52974 59 4.742179
55760 60 0.520754
55761 61 0.170728
55762 62 0.237919
57478 63 0.295764
61243 64 0.337114
61246 65 0.081116
61503 66 0.092666
61506 67 0.048821
61607 68 0.289923
61610 69 0.082652
61687 70 0.152583
61690 71 0.175322
62591 72 0.920901
62594 73 0.357663
62823 74 0.711127
63833 75 2.121323
63834 76 3.187475
64275 77 0.001378
64961 78 0.162714
64962 79 0.345326
67111 80 0.002054
67114 81 0.187752
69703 82 0.040604
69706 83 0.476924
71515 84 0.708087
71518 85 0.101423
71834 86 0.353010
73215 87 0.048512
73218 88 0.375491
74395 89 0.028145
74398 90 0.184067
74921 91 0.010113
74922 92 0.176437
75031 93 1.834347
75034 94 0.122357
77595 95 0.355491
77598 96 0.125475
78220 97 0.178333
78222 98 0.295954
78223 99 0.385514
78555 100 0.036707
78558 101 0.038212
79946 102 0.019544
79950 103 0.314157
80103 104 0.072031
80106 105 0.443512
80794 106 0.231408
80798 107 0.429387
82511 108 3.504525
82514 109 0.131558
83095 110 0.241904
83098 111 0.229109
83167 112 0.042083
83170 113 0.502277
83946 114 1.424533
84443 115 0.632131
85035 116 1.133995
85038 117 0.234205
86291 118 2.460492
86294 119 0.241094
89296 120 0.267791
89298 121 0.630590
89299 122 0.276484
89646 123 0.175880
91951 124 0.657542
91952 125 0.011080
91953 126 0.386147
92610 127 0.351812
92613 128 0.278424
93207 129 0.117969
93208 130 0.293646
94782 131 0.339254
94785 132 0.127108
95146 133 0.055489
95149 134 0.245969
95480 135 0.032614
95481 136 0.241876
96537 137 0.145288
96720 138 0.001228
96724 139 0.246610
96725 140 0.059872
96781 141 0.075450
96866 142 1.591221
96869 143 0.085255
99546 144 0.090596
99549 145 0.111317
99646 146 0.028165
99649 147 0.356581
99974 148 4.203999
99977 149 0.189322
100734 150 0.052224
100737 151 0.317698
101410 152 0.380507
101413 153 0.268456
101446 154 0.851510
101449 155 0.201051
101566 156 0.627114
101569 157 0.112604
102066 158 0.909575
102069 159 0.114531
103164 160 0.172163
103165 161 0.194189
104458 162 2.803576
104459 163 0.095143
104461 164 0.217942
104798 165 0.232049
104801 166 0.390398
106811 167 0.479852
106813 168 1.080945
106814 169 0.366071
107395 170 0.290446
107397 171 0.756489
107398 172 0.094236
108499 173 0.035672
108501 174 0.772666
109116 175 0.013842
109117 176 0.693476
109263 177 0.419134
109265 178 0.433827
109887 179 0.037398
109889 180 0.456709
110363 181 0.638587
110365 182 0.273036
111294 183 0.519425
112855 184 0.436112
112856 185 0.461187
112858 186 0.197535
113339 187 0.465803
113342 188 0.735912
118453 189 0.144465
118456 190 0.255532
118482 191 0.030365
118483 192 0.052658
118484 193 0.430398
119627 194 0.005023
119628 195 0.373068
119993 196 0.174600
119996 197 0.494724
120457 198 0.356260
120460 199 0.047622
124524 200 0.323934
125926 201 0.825262
125927 202 0.336890
125928 203 0.762235
125929 204 0.391387
125930 205 0.597881
125931 206 0.380590
130107 207 0.216572
130109 208 0.507754
130153 209 0.247313
133415 210 0.019257
133416 211 0.025065
133418 212 0.018010
133851 213 0.830890
133853 214 0.449552
134842 215 2.091352
134843 216 2.824228
134845 217 0.755368
134846 218 2.100201
134902 219 0.515646
134905 220 0.719432
134906 221 3.407452
135722 222 0.713153
137604 223 0.508993
137605 224 1.093383
137607 225 0.392873
137608 226 0.923373
138706 227 1.288582
138709 228 0.682757
143484 229 3.782573
143693 230 0.248797
143695 231 1.875079
143696 232 0.889129


They seem to be consistent with expectations. Do you have any guess as to how number of errors will scale with buffer size?

Now I know I don't need to repeat the non MT runs for the new dlls I should be able to post all other results by tomorrow.

Edit: Will of course also test tsp's mod.

[Fizick]

thanks tsp!
I independenty made patched version too :)
(it support MVDegran1 and MVAnalyse only)
http://avisynth.org.ru/tmp/mvtools.dll

probably your is more smart.

(More comments in evening.)

mroz,
thanks for test! probably we really get weak chain.

[Vesi]

since i have athlon 3200+, when i use this script i get about 4-5fps.
how can i use mt with this script to see how it goes?
should i use mt with lsf or frfun7?
FRFun7(Lambda=1.1,T=6.0,Tuv=0)
dull = last
sharp = dull.LimitedSharpenfaster( ss_x=1.0, ss_y=1.0,smode=3, strength=60, overshoot=1,special=true )
Soothe( sharp, dull, 20 )
Tweak(sat=1.1,bright=-8,cont=1.1)
@ where to get the latest MT?
Edit: i have found some info that mt is meant for cce encoding, not for divx/xvid, so can we use it with x264?

[tsp]

Vesi: Unless you have a hyperthreading or multicore or multiprocessor computer you wouldn't see any improve using the mt plugin.

Fizick: I more or less copied the PVideoClip class and modified it to
use MVGroupOfFrames pointer instead of VideoClip pointer and changed the return type of MVFrames::GetNewFrame and MVFrames::GetFrame to the new PMVGroupOfFrames class and fixed all the compiler error what came afterwards (about conversion from PMVGroupOfFrames to MVGroupOfFrames * not possible). So it should work with all the functions that uses idx

[mroz]

thanks tsp!
I independenty made patched version too :)
(it support MVDegran1 and MVAnalyse only)
http://avisynth.org.ru/tmp/mvtools.dll

probably your is more smart.

(More comments in evening.)

mroz,
thanks for test! probably we really get weak chain.

Just finishing running all the huffy encodes on my test material. Will queue up the comparison analysis then & that should take about another 1 to 2 hours.

Your dll above resulted in a silent crash about 1% of the way into the encode. There's no useful info in the Megui log from Mencoder. It just seems to have terminated prematurely. I've not had any other problems on this machine since building it a couple of months ago, but I suppose it could still be anything.

I'm re running that encode now & currently it's about 24% of the way through, so if the problem was down to mvtools, it isn't deterministic, but then if it's a threading issue one wouldn't expect it to be.

Update: it crashed at 68% of the way through. Again, no useful log info; it just terminated unexpectedly.

Regarding anticipated results, I can say before performing the comparisons that I've noticed the more errors there are, the more compressible the output, thus file size is some indication of error rate. For the buffer size variations it looks as though there's a very rapid increase in error numbers as the buffer size drops.

The table below shows this. Under version, b<x> indicates the Fizick builds with buffer size <x>.


Version Error # Undersizing (as a fraction of clean output)
b4 ? 0.0092
b5 ? 0.0024
b6 232 0.0000039
b10 3 0.000000062
b15 ? 0.00000000032
tsp ? 0 (though not bitwise identical)


Edit: re the output from the crashes, I've corrected the unwritten header info (AVIRepair) & rebuilt the index (DivFix), just out of curiosity. DivFix failed on the 2nd crash file (maybe it doesn't like large files - it gave an i/o error). However the first small one, about 1min 53s long (321MB) is playable. Nothing unusual is visible just before the encode terminated, however about 1min 14s into it there's a weird artifact lasting 35 frames in the bottom 1/10 of the display. It's equivalent to a black rectangle smoothly rising up from the bottom, obscuring the intended content, reaching a maximum height & then dropping back down. IOW for the duration of the effect, the bottom n lines of the display are blanked, with n rising from 0 to about 50 then falling back to 0.

[mroz]

Results:

Version Error # Undersizing (as a fraction of clean output)
b4 128886 0.0092
b5 75865 0.0024
b6 232 0.0000039
b10 3 0.000000062
b15 3 0.00000000032
tsp 2 0 (though not bitwise identical)


Interpreting the last two rows above requires a recap & clarification of a few aspects of this test. Note that rounding error leads to a 'Variation' (see script below) of 0.000022 between identical frames, so I arbitrarily chose to log any variation greater than 0.0001 as a dubious frame.

Now for b<x>, x<=10, logged errors are typically 0.1 to 1 in magnitude & almost never below 0.01. The resulting artifacts shown by subtract are visible to the eye without any enhancement & resemble ghosting around moving edges, corresponding to incorrect buffers being accessed I assume that actually correspond to temporally nearby frames.

Recall the log from b10:

Tuesday, October 23, 2007 12:58:52
Frame Total Variation
15858 1 1.816673
15859 2 3.181405
15861 3 0.210538
Test complete.


Cases b15 & tsp1 are significantly different as the logs below show:

b15:

Thursday, October 25, 2007 01:44:06
Frame Total Variation
76888 1 0.000386
128332 2 0.000904
128331 3 0.001176
Test complete.


tsp1:

Thursday, October 25, 2007 02:07:57
Frame Total Variation
7037 1 0.001146
63052 2 0.003961
Test complete.


The few errors that have been noticed are unusually (though not exceptionally) small. Visually, I can't see them in a simple subtract. Subtract(c1,c2).Levels(124, 1, 132, 0, 255) renders them visible & shows them to be somewhat different to the usual. These five are all similar in character; tsp1 frame 63052 is a reasonable example:

http://img86.imageshack.us/img86/2687/difftsp163052ah2.png (http://imageshack.us)
By digitalrat (http://profile.imageshack.us/user/digitalrat)

They all occur on the right side & resemble a formless small patch of noise almost. Typically within a low contrast area of the video & subject to high motion.

Do you think these are indicative of the multithreading problems? I know Fizick said the b<x> variations should produce pixel wise identical output to each other when run without MT, which implies the errors in b15 must be the result of interaction with MT, but does that also apply to the tsp modified MVTools?

I'll run a non MT encode overnight using the tsp variant just to check.

Can anyone draw any conclusions, or likely conclusions, from the above?

Any comments? Anything more I can do?

For completeness, here's the script I used to look for differences:
SetMTMode(2,0)
leaf1="noSetMT"
leaf2="SetMT-t1-b10"
global c1=AVISource("E:\Work\test\hfyu_MVMT-longtest-"+leaf1+".avi")
global c2=AVISource("E:\Work\test\hfyu_MVMT-longtest-"+leaf2+".avi")
global total=0
global file="E:\Work\test\compare-"+leaf1+"-"+leaf2+".log"
BlankClip(c1, width=16, height=16) # maybe 15% faster than using c1 as our 'dummy' clip (& 100% faster than using subtract(c1,c2) with variation subtitled)
WriteFileStart(file, """ "vim:tabstop=6"+chr(10) """, """ Time("%#c")+chr(10) """, """ "Frame"+chr(9)+"Total"+chr(9)+"Variation" """, append=true)
WriteFileEnd(file, """ "Test complete."+chr(10) """)
ScriptClip( """
variation = LumaDifference(c1,c2)+ChromaUDifference(c1,c2)+ChromaVDifference(c1,c2)
# nb1 keeping writefile inside here allows us to store variation as local, reducing problems with running in MTMode 2
# nb2 aside: writefile doesn't want to when called inside FrameEvaluate; anyone know why?
# nb3 sometimes the running total is reported incorrectly in MTMode 2 as another thread updates the total before code writes out data to log;
# this isn't terribly important & can be corrected as is implicit in data set
# nb4 compare against 0.0001 instead of 0 to allow for rounding errors - 2 identical files will report a variation of 0.000022
WriteFileIf(file, "variation>0.0001", "current_frame", "chr(9)", "thistotal", "chr(9)", "variation")
global total = total + ((variation>0.0001) ? 1 : 0)
thistotal = total # massively reduce chance of MTMode 2 screwing up logging of total
return last
""" )

[Fizick]

tsp,
thanks, I see the source code now.
some question about memory optimizing.
I see if all buffer spaces are filled, you increase its size by doubling.
Why double the buffer size instead of incrementing it?


mroz,
thanks for massive test info.
there is no fast conclusion.
wait.
But it is better fogrget about my broken mvtools.dll.
tsp mod shoold be better (more safe).

[mroz]

Wrt the non-MT encode using tsp's build, you probably don't need me to tell you, but it did indeed result in a bitwise identical output to the other builds' non-MT output. So, whatever is causing the small variance in output, it is MT related.

[tsp]

Fizick: mainly to avoid resizing the array to many times. But incrementing it might be better as it doesn't take much time to copy the pointers.
The corruption in the lower right corner could be because both thread are running MVPlane::Refine/RefineExt at the same time. I have created a new version that uses a critical section to avoid this. It is available from here (http://www.avisynth.org/tsp/mvtoolsMTcomp2.zip)

mroz: Thanks for the testing. Could you try the new version above?


ups wrong version. I have updated the file so please download again. This one should work.

[Boulder]

The resulting artifacts shown by subtract are visible to the eye without any enhancement & resemble ghosting around moving edges, corresponding to incorrect buffers being accessed I assume that actually correspond to temporally nearby frames.I have seen this behaviour recently even without any MT - however the application I used to encode uses multithreading. I'm trying to reproduce it to provide a better report, I first need to figure out if it's the encoder or MVTools causing the issue.

[Fizick]

Tsp, thanks for update (I will mirror your version at my site after test).

Can two threads simultaniuosly refine different planes now?
If not, is any performance penalty?

Can somebody (mroz) make a speed test?
(and artefact test too, of course)

[tsp]

Fizick: Yes they can refine different planes at the same time as there are one lock per plane. In a very small speedtest with a 720x576 source and this script

SetMTmode(2)
function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
idx=default(idx,1) # use various idx for different sources in same script
fields=source.SeparateFields() # separate by fields
backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
backward_vec4 = fields.MVAnalyse(isb = true, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec4 = fields.MVAnalyse(isb = false, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct)
fields.MVDegrain2(backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=400,idx=idx)
Weave()
}

source=AVISource("F:\20070513-193504.avi").trim(0,101)
mvdegrain2i(source,4,1,1)

the new and old version of mine has the same speed (1.54 fps, 66 sec). I didn't test for artifacts in this run. The weird thing is that without SetMTmode and the above script the speed is 0.74 fps (138 sec) or 2.1(110% speed increase) times faster with 2 cores. I don't know how to explain this. Very strange.

[Fizick]

I do not understand why you use same critical section both for frame and for plane class?

[tsp]

it's not the same critical section. It is both defined in the MVFrames and MVPlane class. I use the same variable name for the CRITICAL_SECTION sorry if that is confusing. So it is one CRITICAL_SECTION per class instance.

[Fizick]

I am novice in C++ and MT. :)
But I see only one "CRITICAL_SECTION cs;" declaration in MVinterface.h (line 721, MVFrames class).

Sorry, but may be uploded code is not from latest version?

[tsp]

sorry my mistake. You are right the MVinterface.h was from my first version. I have updated the source in the file.

[Spuds]

I've run several passes with tsp's latest modified mvtools using my test script. So far I have not found any defects either through subtract and watching or by using Mroz's script to look for changes.

[mroz]

Fizick: mainly to avoid resizing the array to many times. But incrementing it might be better as it doesn't take much time to copy the pointers.
The corruption in the lower right corner could be because both thread are running MVPlane::Refine/RefineExt at the same time. I have created a new version that uses a critical section to avoid this. It is available from here (http://www.avisynth.org/tsp/mvtoolsMTcomp2.zip)

mroz: Thanks for the testing. Could you try the new version above?


ups wrong version. I have updated the file so please download again. This one should work.


Sorry for my lack of response. Firstly my main machine was disassembled for several days, then when I came back I obviously misunderstood how the mail notifications of thread updates work, as I had thought I'd have at least one notification per thread to which a post had been made, but perhaps one only gets a single notification for the first updated thread & then nothing until revisiting the forum, at which point notifications resume, but are only sent for new posts, not last unread. Anyhow, sorry.

I see Spuds has probably done the testing you need, but I'll add my two penneth when my current encodes are done - probably in 6 to 12 hours time - as you hope it will address the specific artifact I was seeing (thankfully I still have the source & non-MT encode of it).

[mroz]

Encode & comparison using http://www.avisynth.org/tsp/mvtoolsMTcomp2.zip now complete. Not only are there no measurable differences between frames in the MTcomp2 encode compared to a non MT encode, the resulting files are actually bitwise identical. Nice :)

Speed was roughly as expected & as before, but I didn't keep speed stats from earlier tests so don't have precise values; also this box was doing a few other tasks at the same time, albeit of low cpu intensity.

If you want I can rerun some shorter tests just to measure performance.

[IanB]

Some question about memory optimizing.
I see if all buffer spaces are filled, you increase its size by doubling.
Why double the buffer size instead of incrementing it?Yes I have wondered about the amount of buffers allocated. For a pure single sided linear script with no IDX surely only 1 buffer is needed. Given the size of the buffers and with TSP's mods maybe you should just start with 1 and add 1 at a time.

To get back performance with multiple access patterns using an IDX you could track the last few (8 or 16) frame numbers in to the Analyse GetFrame for that IDX number. If the same number occurs again, count how many different frames numbers occur in the list since last time then number occured and make make the buffer count at least that many + 1. i.e. access pattern :-
2, 3, 4, 5, 3, 4, 5, 6, 4, 5, 6, 7, 4, 5, 6, 7

For the first 4 accesses there are no repeats, on the 5 access for frame 3 there is a repeat in slot 2 and you do not still have the buffer for frame 3 (i.e a cache miss), so you make sure there are at least (5-2)+1=4 buffers.

On the 13th access for frame 4 there is a repeat in slot 9, but you do still have the buffer for frame 4 (i.e a cache hit), so you do nothing. If it was a miss you would increase to (13-9)+1=5

[tsp]

mroz: It would be nice if you could make 4 short test with no SetMTmode and setMtmode(2,2), setMtmode(2,3) and setMtmode(2,4) to see how well it scales.

IanB: Your method would be an improvement. Currently the buffer size only grows if all the buffers has an refcount greater than 0 (assigned to a smartpointer somewhere) while there are no test for/adoption to cache misses. With your suggestion the max buffer count would be limited to the size of the track list but dynamically increasing the size of this list could be a solution to this.

[mroz]

A few test results to show how performance scales with number of threads.

I've run the tests using two different short clips (to help establish scaling behaviour is independent of video content for 'typical' video {ie not pathological cases}). I've also run initial tests twice to see what magnitude variation is due to (low) background activity & hence how many figures in the results are significant. Lastly I've run tests with both the last official release of mvtools & the latest tsp mvtools-comp2, for comparison.

clip1 mvtools-tsp2
run1
Name Input FPS
job1 noSetMT 6.00 1.0000
job4 SetMT1 5.96 0.9933
job7 SetMT2 11.80 1.9667
job10 SetMT3 17.53 2.9217
job13 SetMT4 22.98 3.8300
job16 SetMT0 22.91 3.8183

run2
Name Input FPS
job1 noSetMT 6.00 1.0000
job4 SetMT1 5.98 0.9967
job7 SetMT2 11.82 1.9700
job10 SetMT3 17.49 2.9150
job13 SetMT4 22.98 3.8300
job16 SetMT0 23.19 3.8650



clip1 mvtools-orig
run1
Name Input FPS
job1 noSetMT 6.01 1.0000
job4 SetMT1 5.98 0.9950
job7 SetMT2 11.78 1.9601
job10 SetMT3 17.53 2.9168
job13 SetMT4 23.09 3.8419
job16 SetMT0 22.83 3.7987

run2
Name Input FPS
job1 noSetMT 6.00 1.0000
job4 SetMT1 5.98 0.9967
job7 SetMT2 11.80 1.9667
job10 SetMT3 17.32 2.8867
job13 SetMT4 23.21 3.8683
job16 SetMT0 22.95 3.8250



clip2 mvtools-tsp2
run1
Name Input FPS
job1 noSetMT 6.07 1.0000
job4 SetMT1 6.04 0.9951
job7 SetMT2 11.92 1.9638
job10 SetMT3 17.71 2.9176
job13 SetMT4 23.44 3.8616
job16 SetMT0 23.21 3.8237



clip2 mvtools-orig
run1
Name Input FPS
job1 noSetMT 6.07 1.0000
job4 SetMT1 6.04 0.9951
job7 SetMT2 11.92 1.9638
job10 SetMT3 17.71 2.9176
job13 SetMT4 23.47 3.8666
job16 SetMT0 23.20 3.8221

[Fizick]

IanB,
for linear access the optimal buffer sise is dependent on number of MVanalyse calls in script (with same idx) .
as number+1.
So, 5 is usually fine for max delta=2, and 7 is for delta=3.
Probably it can simply be reset internally in every MVanalyse to (2*delta)+1 as a start value.

[foxyshadis]

Now I know that I have the same MT avisynth that I've always had (downloaded the latest just to make sure), but on a very simple script it's been wedged at one thread in avsp and virtualdub (1.7.1) for some reason:

setmtmode(2)
MPEG2Source("V:\video\work\psb\VTS_01_1 - 0xE0 - Video - MPEG-2 - 720x576 (PAL) - 4~3.d2v", cpu=4)
assumetff
tdeint(mode=1,mthreshl=3)
fft3dfilter(sigma=1,plane=4)
Spline36resize(720,480)
BlendFPS(60000/1001.,aperture=.4)
separatefields.selectevery(4,0,3).weave.AssumeTFF()

And even removing all but the source is the same.

GetMTMode(false) reports 0 although true reports 2. Fortunately I remembered the mencoder issue and adding Distributor fixed it. Any idea why it might be happening? I know it was working fine when I was testing mvtools last week, so I don't really understand what could have changed. Weird.

Wait, nm, Distributor fixes the speed but the video is totally mangled now. I guess that wasn't it.

[tsp]

new testversion of mvtools with ian's and fizicks suggestion for memory optimization that is tracking the last buffersize+3 frames accessed to see if the buffersize should increase and initial buffersize based on delta value for mvanalyse. You can get it here:
http://www.avisynth.org/tsp/mvtoolsMTcomp3.zip

foxyshadis: Sounds like it is MPEG2Source? What version are you using?
Does
SetMTmode(2)
blankclip()
work?
Using distributor when the MTmode is 0 will produce garbage because there are only one instance of each filter (like MTmode=1) and the ordinary non-thread safe version of the internal cache is used (the filtergraph is not setup for multithreading). With memcoder the MTmode=2 so the filtergraph is correctly created but distributor is never inserted at the end because avisynth doesn't know when memcoder is done creating the filtergraph.
At least fft3dfilter and spline36resize will work with MT() but I will see if I can reproduce the problem.

[foxyshadis]

Yes, it does work. DGDecode is 1.5.0, the current beta, though it also happens with 1.4.9 and other sources (avi, ffmpeg). I'm going to reboot and see if that helps.

Actually I just emptied the plugin folder and hey, it works. Now to hunt the culprit down.

Ah, mc_spuds.avsi. A cursory look doesn't really tell me why, unless all the globals are causing trouble. (Unnecessary now that mvtools is mt-friendly anyway, I guess.)

[mroz]

new testversion of mvtools with ian's and fizicks suggestion for memory optimization that is tracking the last buffersize+3 frames accessed to see if the buffersize should increase and initial buffersize based on delta value for mvanalyse. You can get it here:
http://www.avisynth.org/tsp/mvtoolsMTcomp3.zip

Cheers. Same performance & reliability here :) If you want any memory usage tests just supply a suitable script.

[Boulder]

tsp,

are you planning to update your build using the latest Avisynth 2.5.8 sources or should we wait for v2.6?

Thanks for the MVTools fix :)

[tsp]

Boulder: I will update it when avisynth 2.5.8 final is ready.

mroz: Sounds good. The memory usage will be higher for some of the more complex scripts or when running with more threads but should result in better performance (less cache misses). Oh and BTW thanks for the speed test. It is very good to see that one of the slowest avisynth filter scales so well with 4 cores. Now who wants to test with 8 cores?

foxyshadis: I think it could be the
global nullclp = blankclip(width=16,height=16)
in MC-Spuds that causes all the problem as it is evaluated before SetMTmode when saved as an .avsi file using import after setmtmode instead of autoloading should fix it.

[foxyshadis]

Thanks a lot.

I just want to throw out another vote for IanB to include MT officially in 2.5.8 now that it's basically stable.

[badshah]

I have core2duo, 1GB RAM

I installed avisynth 2.5.7

then from MT_07 package, I have put MT.dll in plugins directory & avisynth.dll from the same pack to windows/system32 folder

I am getting this error, while loading script into Vdub

AVI Import Filter error: (Unknown) (80040154)

If I put original avisynth.dll into system32 folder ... it works fine but then, I am not able to use MT

pls help

regards

[Leak]

I am getting this error, while loading script into Vdub

AVI Import Filter error: (Unknown) (80040154)
Which script are you talking about? :script:

Also, does that happen with a script simply consisting of just Version() as well?

[badshah]

Which script are you talking about? :script:

Also, does that happen with a script simply consisting of just Version() as well?

Version() doesnt work as well !!

script :
SetMTmode(2)
DGDecode_mpeg2source("G:\My DVD\VTS_01_1.d2v",cpu=2,info=3)

crop( 12, 64, -12, -68)

Spline36Resize(672,368) # Spline36 (Neutral)
#denoise

[Boulder]

What are the contents of your Avisynth 2.5 plugins folder?

[badshah]

contents of plugins folder .............

http://i17.tinypic.com/729fdpu.jpg

[badshah]

I don't know what happened.... its working fine now ... but speed is very slow with setmtmode(2)

without that line i am getting 20fps ... while with that line speed is 2 to 4 fps :confused:

[Boulder]

I've tried using HC with SetMTMode. It seems that there are no extra threads being used at all! The number of threads is the same even if SetMTMode(2) is called before the source is loaded. When I do the same in VDubMod, the number of threads is higher with SetMTMode.

[mroz]

@badshah: given SetMTMode is meant to be the first command in a script, is it safe to have LimitedSharpenFaster.avsi autoloading from plugins? I don't know if this makes any difference; just mentioning in case.

@Boulder: Try adding as the last line to your script
Distributor()
before giving the script to HC. This might be the same issue as with avs input to MEncoder & to Megui's Analysis pass.

[Boulder]

@Boulder: Try adding as the last line to your script
Distributor()
before giving the script to HC. This might be the same issue as with avs input to MEncoder & to Megui's Analysis pass.Yep, this works. Too bad that using SetMTMode seems to kill performance, going from 6fps down to 3fps :(

[mroz]

Yep, this works. Too bad that using SetMTMode seems to kill performance, going from 6fps down to 3fps :(

Just with HC or with VDub as well? What's the script?

[Boulder]

I only tried HC.

The script is:

SetMTMode(2)
MPEG2Source("path\clip.d2v",cpu=4)
Crop(12,78,-12,-78,true)
den=DegrainMedian(mode=2)
DegrainFFTC(last,den,sad=200,ol=4)
AddBorders(16,78,8,78)
Distributor()
And the function DegrainFFTC is in degrain.avsi which contains a few functions similar to each other. The function itself is:

global idx_counter = 10
global idx_counter_2 = 50

function DegrainFFTC( clip c, clip cleaned, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_counter = idx_counter + 1
global idx_counter_2 = idx_counter_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(cleaned,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(cleaned,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(cleaned,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(cleaned,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_counter_2,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_counter_2,plane=pl),c,limy,limuv) : nolimit
}The CPU usage goes to 90-99% but speed drops down seriously.

[IanB]

HC requests frames slightly out of order, if this makes it thru to MPEG2Source worst case you can end up re-decoding each source GOP every time instead of just decoding the next delta frame.

Add a SetMemoryMax(big??) to increase the available cache buffers.

Also try abusing ChangeFPS(Last, Last, True). Add it after your Distributor call, it will act as a 10 frame EnsureVBRMP3Sync for video.

[badshah]

@badshah: given SetMTMode is meant to be the first command in a script, is it safe to have LimitedSharpenFaster.avsi autoloading from plugins? I don't know if this makes any difference; just mentioning in case.


I did import of LSF in begining lines & its running better now. but still encoding speed is 60% of total CPU capacity .... how to increase the speed further:o

[Boulder]

HC requests frames slightly out of order, if this makes it thru to MPEG2Source worst case you can end up re-decoding each source GOP every time instead of just decoding the next delta frame.

Add a SetMemoryMax(big??) to increase the available cache buffers.

Also try abusing ChangeFPS(Last, Last, True). Add it after your Distributor call, it will act as a 10 frame EnsureVBRMP3Sync for video.I have SetMemoryMax(512) as an avsi file so that should suffice I think. I'll try the ChangeFPS trick later and report back.

[mroz]

I did import of LSF in begining lines & its running better now. but still encoding speed is 60% of total CPU capacity .... how to increase the speed further:o
The script you quoted isn't very complex so maybe the bottleneck is the encoder. What is it? x264 will get to about 80% on a first pass & around 100% on a second, however ime xvid only manages about 60% - increasing the number of threads it uses will up cpu usage, but actually drops frame rate; it doesn't seem optimised for multithreading (*) (this is on a Q6600). Of course I might simple have it misconfigured (though I was using default profiles in Megui).

(*) By which I mean while it will run multithreaded with improved performance, there does seem to be this limitation preventing full cpu exploitation.

[badshah]

^^ its Xvid ... and exactly, it doesnt go above 60 to 70 %. anyway, its fine.

[mroz]

You can always up efficiency manually on a per job basis, by splitting the encode in two & running both at the same time (in separate workers, if you're using Megui), then join the outputs when done.

It's not something I've bothered with due to the hassle & the facts I do little xvid encoding & can usually find another use for those cpu cycles.

[Boulder]

I have SetMemoryMax(512) as an avsi file so that should suffice I think. I'll try the ChangeFPS trick later and report back.The ChangeFPS trick seemed to fix things. The performance is better with SetMTMode than without it when encoding with HC.

[IanB]

I have plans for a "LinearAccess" filter for 2.6, I'll bump it up the todo list a bit.

[vcmohan]

In some plugins(of mine ) process parameters are derived from analysis of a particular field (or frame) of the clip and applied all through. In case of MT in the mode where alternate fields are processed in different threads whether the params will be computed in both threads or is there a control on the place from which forking takes place?

If this querry does not belong to this thread pl bump it to a new thread.

[mroz]

@IanB, tsp

So far we've seen MEncoder, MeGUI & HC being unable to exploit MT for the same reason that Distributor is never invoked.

tsp, I think, said it would be very hard/impossible to modify MT/Avisynth so that this call is made automaticaly at the end of a script, other than in the AVIFile case as at present.

Since the problem is affecting many tools even in my limited experience, I'm brought back to considering this.

Is there really no way to automate a solution internal to AviSynth? Is it not possible to insert the relevant code immediately prior to processing of the first frame request?

If this can't be done, how about simplifying the modifications that need to be made to the third party programs affected? Rather than require they check to see if MT is in use & then invoke Distributor conditional on the current MT mode, why not hide all of that in a single function which is safe to call regardless of MT usage & mode? Then the only third party mod is to ensure they invoke this function at the end of script building.

Obviously this is only a full solution once MT becomes an official part of AviSynth, since prior to that the above function invokation will error, but that's probably safe to handle by documenting the function with the instruction to ignore any returned error/catch & ignore any exception.

I think simplifying the mods needed will make them more likely to be accepted. For instance, when I described the changes needed in respect of MeGUI's analysis pass option, the response I got failed to comment on the involved mode check but concentrated on confirming that switching to AVIfile access would eliminate the problem at the loss of certain abilities/controls; this strongly suggested to me that a quick simply change was being sought that wouldn't require any MT specific research prior to implementation, which makes sense for a busy dev.

Comments?

[squid_80]

IMO any apps accessing avisynth directly (bypassing avifile) should know exactly what they're doing and make the call themselves. Otherwise use AVIFile.

[mroz]

The problem with that is that atm they can know exactly what they're doing in respect of AviSynth & won't know about this, as MT isn't integrated into AviSynth.

And even well after it is, surely it still makes sense, at the very least, to require no more than the script ends with a simple native function invokation which internally deals with checking MT mode & calling distributor if needed.

If in the future it becomes necessary for AviSynth to know when a script is complete for other reasons, this would also facilitate that.

Furthermore, it would allow the particular call to be included in scripts opened via AVIfile without any penalties - at the moment a call to distributor will screw up efficiency if included directly in a script which is also opened via AVIfile. I can't see the harm in at least avoiding that problem.

[IanB]

Well the thing is you were never meant to directly call "Distributor" in scripts, I mearly suggested it as a test for a certain issue as I became aware of it. And yes it certainly makes sense to make MT as transparent as possible.

[tsp]

vcmohan: The params will be computed in both threads as the class will be instantiated for each thread.

mroz: making a simple native function EndScript() could be easy. But it isn't the optimal way to make it transparent.

[mroz]

mroz: making a simple native function EndScript() could be easy. But it isn't the optimal way to make it transparent.

Indeed. So what would the optimal way be? :)

[IanB]

So what would the optimal way be?Doing nothing!

[vcmohan]

vcmohan: The params will be computed in both threads as the class will be instantiated for each thread.



Does this mean that if params are to be computed from field x, then in both threads they are from same field and not from x in one and x+1 in the other? I have this doubt as in the two threads alternate fields will be processed and the plugin will have no control on that.

[mroz]

Doing nothing!

Um, but I've already been told that isn't possible, hence can't be regarded as an optimal solution since it isn't a solution at all. Unless I'm missing something.

[Fizick]

vcmohan,
Are you say about MTi() function ?

[IanB]

@vcmohan,

The PVideoFrames from all the GetFrame calls will be shared thru the cache. If the filter does internal calculations from additional GetFrame calls then each thread will independantly calculate the same (or similar) result. So only the internal work of the filters own GetFrame processing is unique. If the filters maintain static data, Class level or Global, that is not interlocked then a more restrictive SetMTMode may be required.

[Boulder]

With Distributor() in my script for HC with tsp's latest MVTools build, I get these weird artifacts in the first couple of frames of the encode (see the bottom right corner of the actual video area of the screenshot).

With Distributor()
http://img141.imageshack.us/img141/370/rwandamtuj1.th.png (http://img141.imageshack.us/my.php?image=rwandamtuj1.png)
Without Distributor() (no multithreading)
http://img401.imageshack.us/img401/7632/rwandanomtgg4.th.png (http://img401.imageshack.us/my.php?image=rwandanomtgg4.png)

[tsp]

IanB: Good description of SetMTmode(1). With SetMTmode(2), MT() or MTi() the class variables is not shared between threads.

Boulder: Could you upload a short clip (10 frames or so) and the script you use?

[Boulder]

Boulder: Could you upload a short clip (10 frames or so) and the script you use?I'll upload the sample clip for you tomorrow.

[Boulder]

Here is the sample m2v file from the original DVB capture: http://www.savefile.com/files/1196029. See the artifacts appear when you encode in HC022 using the script below. When Distributor() is removed, there are no artifacts but no multithreading Avisynth processing either.

My script:SetMTMode(2)
MPEG2Source("rwanda.d2v",cpu=4)
Crop(4,72,-4,-70,true)
DegrainC(sad=200)
AddGrain(3,0,0)
AddBorders(8,72,0,70)
Distributor()
ChangeFPS(last,last,true)

DegrainC is in degrain.avsi in the Avisynth plugins directory. The contents of the file are:global idx_1 = 100
global idx_2 = 200

function Degrain( clip c, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(c,isb=true,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(c,isb=false,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(c,isb=true,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(c,isb=false,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl),c,limy,limuv) : nolimit
}

function DegrainC( clip c, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(c,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(c,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(c,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(c,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl),c,limy,limuv) : nolimit
}

function DegrainFFTC( clip c, clip cleaned, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(cleaned,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(cleaned,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(cleaned,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(cleaned,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl),c,limy,limuv) : nolimit
}

function DegrainFFT( clip c, clip cleaned, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(cleaned,isb=true,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(cleaned,isb=false,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(cleaned,isb=true,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(cleaned,isb=false,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl),c,limy,limuv) : nolimit
}

[tsp]

Boulder: It is not MVtools that ist he problem. This simple script produces the same result:

SetMTMode(2)
MPEG2Source("rwanda.d2v",cpu=4)
Distributor()

both with HC and virtualdub. DGIndex complained about a Opening GOP is not closed. Maybe this is only causing problem with multithreading.

[Boulder]

In case of that non-closed opening GOP, DGDecode should take the first good frame and repeat it for those undecodable frames. The distributor is causing the issue, but I don't know if there is a way to fix it without changes in the way HC handles video input. Since it is a Fortran application, it might be that there is no way to avoid using Distributor.

[tsp]

Boulder: It is not HC that is causing it. It is MPEG2Source. Try this version instead:

SetMTMode(5)
MPEG2Source("rwanda.d2v",cpu=4)
SetMtmode(2)
Crop(4,72,-4,-70,true)
DegrainC(sad=200)
AddGrain(3,0,0)
AddBorders(8,72,0,70)
Distributor()
ChangeFPS(last,last,true)

[IanB]

@tsp,

I have often wondered about the wisdom of doing the followingSetMTMode(2)
MPEG2Source("rwanda.d2v",cpu=4)I know it is "safe", but doesn't it lead to potential non contiguous frame order requests with the associate performance penalty of redecoding GOP's.

Certainly with AviSource if you get the request order wrong the penalty is harsh because frames skipped during short forward seeks are not cached and lead to a full redecode back from the last key frame.

[tsp]

IanB: I did some testing with a 10.000 frame long h264 encoded using ffdshow rev 1220 720x576 pixel 1804 kbps 98 keyframes size: 86 MB.
reported time it took to complete "run video analysis pass" in virtualdub 1.6.17 with my opteron 165(dual core) @2400MHz

AVISource("D:\simH264.avi")

97 sec (103 fps) 86 MB read

SetMTmode(2)
AVISource("D:\simH264.avi")

182 sec (55 fps) 277 MB read

SetMTmode(2)
AVISource("D:\simH264.avi")
fft3dfilter()

534 sec (18,7 fps)

SetMTmode(5)
AVISource("D:\simH264.avi")
setmtmode(2)
fft3dfilter()

531 sec (18,8 fps)

AVISource("D:\simH264.avi")
fft3dfilter()

654 sec (15,3 fps)

AVISource("D:\simH264.avi")
fft3dfilter(ncpu=2)

603 sec(16,8 fps)

function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
idx=default(idx,1) # use various idx for different sources in same script
fields=source.SeparateFields() # separate by fields
backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
backward_vec4 = fields.MVAnalyse(isb = true, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec4 = fields.MVAnalyse(isb = false, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct)
fields.MVDegrain2(backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=400,idx=idx)
Weave()
}

setmtmode(2)
AVISource("D:\simH264.avi").trim(0,-500)
mvdegrain2i(4,1,1)

315 sec (1,59 fps)

function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
... same as above
}

setmtmode(5)
AVISource("D:\simH264.avi").trim(0,-500)
setmtmode(2)
mvdegrain2i(4,1,1)

312 sec (1,60 fps) 5 MB read

function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
... same as above
}

AVISource("D:\simH264.avi").trim(0,-500)
mvdegrain2i(4,1,1)

1143 sec (0,44 fps) 777 MB read (I did repeat this 3 times with the same result!)

So the conclusion is that for fast script it is a very good idea to use mode=5 for avisource(and mpeg2dec) while for slower scripts it doesn't matter as much or if a codec that only saves keyframes is used (MJPEG or HUFFYUV). The explanation, I think, is that the threads are desynced in the slower scripts so most of the time only one thread are executing code inside Avisource. Something weird are happening in the last example. I will try repeating it with the official 2.5.7 as it looks like a severe cache miss/bug in my version?


1130 sec for the official avisynth 2.5.7 in the last script.



this script

function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
... same as above
}

setmtmode(5)
AVISource("D:\simH264.avi").trim(0,-500)
mvdegrain2i(4,1,1)

completes in 617 sec (0,81 fps) 5 MB read
so it looks like it is the non multi-threaded cache that are messing up. (as only 1 thread is used in the above code).

[Fizick]

it looks like a severe cache miss/bug in my version? :scared:

can you test more old version (may be even 1.6.4-1.7.0 ?)

BTW, what is "777 MB read" ?

[tsp]

it looks like a severe cache miss/bug in my version? :scared:

can you test more old version (may be even 1.6.4-1.7.0 ?)

more test results, same script (with no setmtmode)
avisynth 2.5.8 alpha 2+mvtools 1.8.4.4: 1023 sec
avisynth 2.5.8 alpha 2+mvtools 1.6.4: 910 sec
avisynth 2.5.7MT + mvtools 1.6.4: 1007 sec

so I still believe it is more of a cache issue than a MVTools issue.

BTW, what is "777 MB read" ?
that the virtualdub process has read 777 MB from the harddisk. The number of page fault is the same with and without SetMTmode so I believe that the source file is read multiple times.

[IanB]

@tsp,

Interesting results. At a raw 103fps your machine averages 9.7msecs to decode a frame, and at 15.3fps your machine takes 65.4msecs to decode+fft3dfilter a frame.

So given 65.4-9.7 = 55.7msecs for fft3dfilter, 2 cores but only sequentially accessing frames, the best time expected could be 9.7+55.7/2 = 37.5msecs or 26.6fps. Which we don't seem to come close to. :confused:

I would be interested to see if forcing sequentialness, i.e.SetMTmode(5)
AVISource("D:\simH264.avi")
ChangeFPS(last,last,true)
setmtmode(2)
fft3dfilter()andAVISource("D:\simH264.avi")
ChangeFPS(last,last,true)
fft3dfilter(ncpu=2)makes any significant difference.

Also if you have time you might test some medium weight filters like Blur() or ...Resize()

[Fizick]

fft3dfilter has internal fft cache, I never consider how (in)effective it will be in multitreading env.

[tsp]

some more test, same source as before:

setmtmode(2)
AVISource("D:\simH264.avi")
changefps(last,last,true)

94 sec (106 fps)
(so just as good as with no multithreading)

AVISource("D:\simH264.avi")
MT("fft3dfilter()",2,8)

521 sec (19,2 fps) slightly better than using setmtmode(2) but still fft3dfilter doesn't scales to well.

AVISource("D:\simH264.avi")
changefps(last,last,true)
MT("fft3dfilter()",2,8)

532 sec (18,8 fps) slightly slower. No surprise as MT() split the frame in two and process each part in a separate thread so it is sequential access even without changefps (same story for fft3dfilter(ncpu=2))

setmtmode(5)
AVISource("D:\simH264.avi")
changefps(last,last,true)
setmtmode(2)
fft3dfilter()

548 sec (18,2 fps) without changefps() 536 sec (18,7 fps). Again slower with changefps() and still slower than mt() (probably due to fft3dfilters intertal fft cache).

and now for a medium weight filter spline16resize

AVISource("D:\simH264.avi")
Spline16Resize(640,480)

154 sec (64,9 fps)

setmtmode(2)
AVISource("D:\simH264.avi")
Spline16Resize(640,480)

172 sec (58,1 fps) slower than without multithreading due to non sequential access.

setmtmode(5)
AVISource("D:\simH264.avi")
setmtmode(2)
Spline16Resize(640,480)

357 sec! (28 fps) ouch even worse, so always using mode=5 for avisource is not a good idea.

setmtmode(5)
AVISource("D:\simH264.avi")
changefps(last,last,true)
setmtmode(2)
Spline16Resize(640,480)

117 sec (85,5 fps) large improvement suggesting that non-sequential access is the culprit.

setmtmode(2)
AVISource("D:\simH264.avi")
changefps(last,last,true)
Spline16Resize(640,480)

105 sec (95,2 fps) fastest time.

so changefps(last,last,true) is most efficient when used with fast scripts while it makes the script slower in slow scripts.
Also I noticed that changefps made the largest difference in script where the fps was very unstable.

[IanB]

Hmm, there is something else going on here, ChangeFPS should be a zero cost filter, all it does is enforce sequential access within a 10 frame range forwards. It should never make things worse. Maybe the Cache behind it is not strong enough and is faulting, negating the sequential access protection. Hmmmmm :confused:

[tsp]

repeated the test with setmemorymax(1500). The difference between AVISource("D:\simH264.avi").changefps(last,last,true).MT("fft3dfilter()",2,2) and AVISource("D:\simH264.avi").MT("fft3dfilter()",2,2) is down to 2-3 sec. So more or less insignificant.

[IanB]

@tsp,

Are 2 and 5 the best modes to use here with changefps(last,last,true) and are we putting them in the right place? We just want to ensure the AviSource filter gets hit strictly in order. We do not want any incidental interlocking on the cache between changefps and avisource.

If the request order into changefps is 2, 1, 0, 5, 4, 3, ... the expected request order into the next cache should be 0, 1, 2, 1*, 0*, 1*, 2*, 3, 4, 5, 4*, 3*, ... where the entries marked with * should be cache hits. thus the request order into avisource should be 0, 1, 2, 3, 4, 5, ...

[Delerue]

tsp and Fizick, I'm trying to use MVFlowFPS script (from MVTools 1.8.5.1) with this code inside the FFDShow (v. 1620) Avisynth tab:


LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
SetMTMode(2)
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
backward_vec = source.MVAnalyse(blksize=16, isb = true, pel=2, search=2, idx=1)
forward_vec = source.MVAnalyse(blksize=16, isb = false, pel=2, search=2, idx=1)
source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), \
den=FramerateDenominator(source), mask=1, idx=1)


Although it works, I mean, it doubles the FPS, the second CPU/thread still inactive (I'm using an Allendale E4300 running at 3 GHz).

I also tried this (from Fizick website, just to test), but I got a script error (invalid arguments to function 'MT', last line):


LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
global idx1 = 10 # global hint by IanB
MT("""
idx1 = idx1 + 1
# different threads for top and bottom half of frame must have different idx (trick by Foxishadis)
backward_vec2 = MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec2 = MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
last.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx1)
""",2) # two threads


I copied the 'avisynth.dll' to the 'system32' folder, and the 'MT.dll' to the 'AviSynth\plugins' folder, inside the 'program files' folder. Any idea? :)

Thanks!

P.S.: Fizick, as you can see, your latest MVTools can finally works using the 'num=2*FramerateNumerator(source)' command with FFDShow. ;)

[tsp]

Delerue: Try this version:

SetMtmode(5)
LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
SetMTMode(2)
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
backward_vec = source.MVAnalyse(blksize=16, isb = true, pel=2, search=2, idx=1)
forward_vec = source.MVAnalyse(blksize=16, isb = false, pel=2, search=2, idx=1)
source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), \
den=FramerateDenominator(source), mask=1, idx=1)
distributor()

and this:

function MVD(clip c)
{
idx1 = idx1 + 1
# different threads for top and bottom half of frame must have different idx (trick by Foxishadis)
backward_vec2 = c.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
backward_vec1 = c.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec1 = c.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec2 = c.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
return c.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx1)
}

LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
global idx1 = 10 # global hint by IanB
MT("""MVD(source)""",2) # two threads

put please note that using MVTools with MT() is not always good as MT() split the frame in two parts and process each part in a separate thread so MVTools can't motion compensate if the motion occurs from one part of the frame to the other part.
It is better to use SetMTmode(2)

IanB: mode=5 should be the best but my speed test indicate otherwise. I will repeat them to see if it is just random variation.

[Delerue]

Delerue: Try this version:

[code1]

put please note that using MVTools with MT() is not always good as MT() split the frame in two parts and process each part in a separate thread so MVTools can't motion compensate if the motion occurs from one part of the frame to the other part. It is better to use SetMTmode(2)

Thanks, man! It works, but there's something strange. Although the second thread is activated now, the total CPU usage rarely goes beyond 50% and never beyond 60%, so forbidding the script to actually double the FPS in some heavy videos. It seems that the 'job division' isn't good enough, or something like this. I tried to play with other modes, but didn't help. Do you have a faster dual-core CPU to test this script?

Also, I want to congratulate you and Fizick once more. You're making an amazing job, and the 24 FPS movies problem are going soon even with HD videos. :)

[Fizick]

...
I also tried this (from Fizick website, just to test), but I got a script error (invalid arguments to function 'MT', last line):

...
source=...
...


Please be correct. It is NOT exact script from my website (original works well).
"source" is not used by you.


tsp, sorry for many mvtools staff in your thread :)

[Delerue]

Please be correct. It is NOT exact script from my website (original works well).
"source" is not used by you.

Well, I used source in this line 'source=ffdshow_source()'. I only adapted to use with FFDShow. I can't see what you're talking. Anyway, the second tsp script doesn't work either.

tsp, sorry for many mvtools staff in your thread :)

Well, we're talking about MT plugin as well. :)

[tsp]

Thanks, man! It works, but there's something strange. Although the second thread is activated now, the total CPU usage rarely goes beyond 50% and never beyond 60%, so forbidding the script to actually double the FPS in some heavy videos. It seems that the 'job division' isn't good enough, or something like this. I tried to play with other modes, but didn't help. Do you have a faster dual-core CPU to test this script?

Yes with two threads it looks like it hits a limit about 60 % cpu utilization. Just double the number of threads will fix that (at least it did for me from 60% to 90 %).
try this version:

SetMtmode(2,4)
source=ffdshow_source()
source=changefps(source,source,true)
backward_vec = source.MVAnalyse(blksize=16, isb = true, pel=2, search=2, idx=1)
forward_vec = source.MVAnalyse(blksize=16, isb = false, pel=2, search=2, idx=1)
source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), \
den=FramerateDenominator(source), mask=1, idx=1)
distributor()

What is the problem with the second script? Try SetMtmode(2) instead of MT()

[Delerue]

Yes with two threads it looks like it hits a limit about 60 % cpu utilization. Just double the number of threads will fix that (at least it did for me from 60% to 90 %).
try this version:
[code]

Wow! That's for real! I tried with your script and the CPU usage indeed goes beyond 90%. But there is some problems with four threads here when the CPU usage is high (but still not 100%). I noticed some stutter, glitches and other weird things, specially in scenes with too much camera movements. So I tried 5 threads, and it worked perfectly.

About the other script, I'll try later.

Thanks once more. ;)

[Fizick]

Livesms from russian forum post report about artifactes with script:

SetMTmode(2)
AviSource("TV.avi")
AssumeTFF()
MVDegrain2i(4,0,1)
TomsMoComp(-1,15,1)
Crop(8,6,-8,-6).Lanczos4Resize(512,384)
FadeIO(25)
####################################################################################
function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
overlap=default(overlap,0) # величина перекрытия value (от 0 до 4 для blksize=8)
dct=default(dct,0) # используйте dct=1 для клиров с некоторыми мерцаниями
idx=default(idx,1) # используйте различные idx для разных источников в том же скрипте
fields=source.SeparateFields() # разделим на поля
backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
backward_vec4 = fields.MVAnalyse(isb = true, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec4 = fields.MVAnalyse(isb = false, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct)
fields.MVDegrain2(backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=400,idx=idx)
Weave()
}


around 96 frame at:
http://dump.ru/files/n/n53176759/
press "Скачать" button near center under this link :)

[tsp]

Fizick: Is the sample the result of the above script or the source for the script?

IanB: After some more test the most optimal mode appears to be:

setmtmode(5)
AVISource("D:\simH264.avi")
setmtmode(2)
changefps(last,last,true)

even with a MJPEG encoded file.

[IanB]

@tsp,

So this will results in multiple instances of ChangeFPS all individually asserting linear access apon a gate5 cache into an interlocked AviSource. The gate5 cache will probably be thrashed very hard but it's all pointer shuffling without much bulk data copying, so its okay.

[Fizick]

tsp,
it is result.
does TomsMoComp(-1,15,1) work with setmtmode?

[Livesms]

tsp
I'll try to upload original video and encoded.
This night I left my Core2Duo E6600 encoding two more video.
I'll check it in a hour...

[Livesms]

Original video is too huge for my GPRS internet connection.

I've done some test and found SetMTMode(2) to work incorrectly with MVDegrain2 script for MVTools documentation.
Simple one thread MVDegrain2 (without MT), MVDegrain2 with SetMTMode(4) and MVDegrain2 with SetMTMode(5) produce output video without any artifacts, while SetMTMode(2) + MVDegrain2 and SetMTMode(5) + AviSource + SetMTMode(2) + MVDegrain2 do have some problems.

So I tried to look closely to SetMTMode(4) and MTi(). Here is the results:
Source:
* FourCC: YV12
* Frames: 1003
* Resolution: 512x384
* Frame rate: 25.000 FPS

SetMTMode(2)
Destination:
* Pass 1/1: Finished in 00:02:44.819 (6.09 FPS)
* Frames: 1003 (1003 keyframes)
* Memory: 121 788 MB

SetMTMode(4)
Destination:
* Pass 1/1: Finished in 00:02:48.024 (5.97 FPS)
* Frames: 1003 (1003 keyframes)
* Memory: 144 252 MB

MTi()
Destination:
* Pass 1/1: Finished in 00:02:57.276 (5.66 FPS)
* Frames: 1003 (1003 keyframes)
* Memory: 160 340 MB

So MTi() uses more memory than any of script using SetMTMode() and gives no speed increase comparing to SetMTMode(2) or SetMTMode(4)
SetMTMode(4) is 2% slower than SetMTMode(2) in my test. 1000 frames clips denoise time increased with 795ms. SetMTMode(4) uses more memory than SetMTMode(2) – 144252MB comparing to 121788MB.

Allowing 2% speed decrease and 15% more memory usage (comparing to SetMTMode(2)), it is necessary to check SetMTMode(4) + MVDegrain2().

In the moment my PC is encoding two more video (2pass xvid) with SetMTMode(2) and SetMTMode(4) in search of errors...

[IanB]

@Livesms,

Do you have base timings for the above script i.e. without MT?

It is important to know how much boost you are getting. i.e. 3fps to 6fps would be fantastic, 5.9fps to 6.0fps indicates some development work is still needed.

[tsp]

Livesms: Is it the same script Fizick posted?

Fizick: From ToMoComp's source code it looks like it should be compatible with SetMtMode(2) (No non-const global/static variables. Not like the evil Dust filter)

IanB: Even if the cache instance in front of avisource is hit by two thread it is still faster than one thread stalling in front of changefps (that would happend if SetMtMode(5) was placed after changeFPS). I wonder if it would be faster to implement the linear access requirement in the cache instead as it has better track on the last accessed frame (as the two instances of ChangeFPS doesn't share the lastframe variable).

[IanB]

@tsp,

Yes ChangeFPS is a hack proof of concept. And it seems to be somewhat successful when abused appropriately. :D

And yes a proper implementation would probably need to involve the cache or actually be a special case of cache.

Better still AviSource, et el, probably should be internally defending against out of order access. Some time ago I changed code to just reuse the same VFB, the original grabbed a stack of new VFB's as it seeked forwards from a Keyframe, it didn't remember them so it was a wasted effort that just generally trashed the cache. In hindsight a better fix would have been to add code to remember the VFB's. Hmmm!

I don't think always enforcing linear access would be a good idea, however as an optional weapon in the arsenal it's gonna be worthwhile.

Think SelectEven into a huffyuv file versus an Xvid file. With linear access you force the huffyuv to read all the disk blocks instead of just half. With the Xvid file you have to read all the disk blocks regardless and the out of order penalty can be extreme.

[Chainmax]

I just recently started to get into this subject and 38 pages seems like an awful lot to read. Therefore, I'd like to ask you if the following filterchain:

MPEG2Source("C:\simp\Simp.d2v",info=3)
ColorMatrix(hints=true,interlaced=true)
TComb()
AssumeTFF()
TFM(d2v="C:\simp\Simp.d2v",order=1,mode=6,PP=7,slow=2,mChroma=false,micmatching=3)
TDecimate(mode=1)
DeBlock(quant=35)
DeGrainMedian()
Crop(16,8,700,466,align=true)
Spline36Resize(320,240)
AddBorders(16,16,16,16)
aWarpSharp(depth=16,cm=1)
Crop(16,16,320,240,align=true)
gradfun2db(thr=2.4)
Dup(threshold=2,blend=true,blksize=8)

would work well with MT and which would be the correct way to implement it. Any guidance on the matter will be greatly appreciated.

[Livesms]

Livesms: Is it the same script Fizick posted?
Yes. I'm trying to speed up Fizick's MVDegrain2

@Livesms,
Do you have base timings for the above script i.e. without MT?

It is important to know how much boost you are getting. i.e. 3fps to 6fps would be fantastic, 5.9fps to 6.0fps indicates some development work is still needed.

Yes - SetMTMode() speed up to 2 times Fizick's MVDegrain2 script.
I have already post some test in russian forum. Here is my first test result

Source:
* FourCC: YV12
* Frames: 385
* Resolution: 768x576
* Frame rate: 25.000 FPS

No MT (one thread)
Destination:
* Pass 1/1: Finished in 00:02:28.758 (2.59 FPS)
* Frames: 385 (385 keyframes)

SetMTMode(2)
Destination:
* Pass 1/1: Finished in 00:01:12.566 (5.31 FPS)
* Frames: 385 (385 keyframes)

[tsp]

chainmax: Give this a try:

setmtmode(5)
MPEG2Source("C:\simp\Simp.d2v",info=3)
setmtmode(2)
changefps(last,last,true)
ColorMatrix(hints=true,interlaced=true)
TComb()
AssumeTFF()
TFM(d2v="C:\simp\Simp.d2v",order=1,mode=6,PP=7,slow=2,mChroma=false,micmatching=3)
TDecimate(mode=1)
DeBlock(quant=35)
DeGrainMedian()
Crop(16,8,700,466,align=true)
Spline36Resize(320,240)
AddBorders(16,16,16,16)
aWarpSharp(depth=16,cm=1)
Crop(16,16,320,240,align=true)
gradfun2db(thr=2.4)
Dup(threshold=2,blend=true,blksize=8)


Livesms: with MTi() you should use MVDegrain2 instead of MVDegrain2i. I will try reproducing the faulty frames latter

[Fizick]

Livesms found the problem with TomsMoComp and SetMTMode(2)

[Chainmax]

Thanks for the advice tsp, it's much appreciated :).

I'm also going to try this:

SetMTMode(5)
DirectShowSource("C:\Para la PSP\Videos\Sources\street_fighter_the_later_years_6.flv",fps=23.976,convertfps=true,audio=false)

SetMTMode(2)
ConvertToYV12()

Deblock_QED_MT2(quant1=35,aOff1=16,quant2=45,aOff2=6)

RemoveGrain(mode=5)

DeGrainMedian()

FFT3DFilter(sigma=3,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16)

GaussResize(800,448)

LimitedSharpenFaster(SMode=4)

nnediresize2x(true,true,true)

source=last
denoised=DegrainMedian().FFT3DFilter(sigma=3,plane=3,bw=32,bh=32,bt=3,ow=16,oh=16)
backward_vec2 = MVAnalyse(denoised,isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(denoised,isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(denoised,isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(denoised,isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
MVDegrain2(source,backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)

Spline36Resize(480,272)

RemoveGrain(mode=5)

LimitedSharpenFaster(SMode=4,Strength=200)

Tweak(sat=1.1)

gradfun2db(thr=2.4)

AddGrainC(5,2)


In case the SetMTMode choices aren't correct, what kind of errors/artifacting should I be on the lookout for?


[edit]Instead of putting MT.dll in the autoloading folder I can just use a loadplugin call to it right after SetMemoryMax, right?

[Spuds]

FWIW ... I've had some problems with ConvertToYV12() outside of setmtmode(5) with patchwork quilt color frames from time to time. I tend to leave it with avisource shielded with the setmtmode(5) and go setmtmode(2) after.

[Fizick]

Chainmax,
1. encode some clip,
2. remove all MT..., encode same clip to other file.
Compare per frame (with subtract, see post by mroz above) and you will see artifacts type if any.

MT... is experimental.

Probably in AviSynth 2.6x it would be useful to have some method for (new 2.6) plugin to say what are supported modes for this plugin to Avisynth (with exception or at least debug message)?.
Or may be some GetCurrentMTMode() info available to new plugins ?

Or it have no sense, and supported modes are dependent on combination of plugins?

[Chainmax]

That's a good idea Fizick, I'll try in on my following encodes since the current one has finished.


tsp: I tried the last script. There doesn't seem to be anything wrong with the file, and the encode was somewhere between 1.8x and 2.25x as fast :eek:. Thanks so much for this, it's truly amazing. Keep up the good work, man :) http://smilies.vidahost.com/otn/wink/thumb.gif.


One thing though: supposedly MVTools couldn't be multithreaded and yet there seem to bea few tests of MVDegrain2 going on. Does that mean that it works? If so, could the latest McBob be expected to work somewhere down the line?

[foxyshadis]

The latest, 1.8.5.1, is actually compatible. Give it a shot. =D

[mroz]

:o

Sorry, ignore/delete this post.

I messed the testing up. The issue doesn't seem to depend on the version of AviSynth, so is probably a long standing issue with Nic's plugin or a borken ac3 stream.

[tsp]

Probably in AviSynth 2.6x it would be useful to have some method for (new 2.6) plugin to say what are supported modes for this plugin to Avisynth (with exception or at least debug message)?.
Or may be some GetCurrentMTMode() info available to new plugins ?

Or it have no sense, and supported modes are dependent on combination of plugins?
the IScriptEnvironment has been extended with
virtual int __stdcall GetMTMode(bool return_nthreads)
that returns the current mode or number of threads and the function
virtual void __stdcall SetMTMode(int mode,int threads,bool temporary)
that the plugin can use to change the current mt mode for the following filters(including self) or only temporary for it's own instance. So the plugin will choose the right mode.

Chainmax: sounds good just be aware that sometimes the artifacts are only visible in 1 out of 10 000-100 000 frames (but in this case you could argue how much it would matter if it is so rarely)

[Razorholt]

Can I use SetMTMode in Avisynth 2.58 ?

Thanks,
- Dan

[Chainmax]

How do you reckon would a filter like Dup behave in mode 5 and in mode 2?

[Tanma]

Try to set mode 3. It works in almost everything. The only thing I encode in mode 5 or 6 are subtitles (textsub).

And the only thing I couldn't encode well at all in any mode was the vmtoon script, the result was a faulty video.

[Zelos]

Hi all,

i have problem with megui and my quadcore when using mipsmooth filter.
Speed decrease a lot so i would like to know if it was possible to use this script with mipsmooth ?
here is the my script:

Directshowsource (d:\test.grf,video=true,audio=false,framecount=10000,fps=23.976)
Converttoyv12 ()
crop(0,138,0,-138,align=true)
mipsmooth(preset="moviehq2")


Thanks for the help.

[tsp]

Razorholt: No
Chainmax: I think both mode=2 and 5 should work with dup judging from the source code.
Zelos:
Try this:

Directshowsource (d:\test.grf,video=true,audio=false,framecount=10000,fps=23.976)
Converttoyv12 ()
crop(0,138,0,-138,align=true)
mt(""""mipsmooth(preset="moviehq2")"""",4,4)

[Boulder]

I'm getting crashes quite frequently with SetMTMode and HC. The crashes occur during the loading of the script. Is there any way to see what actually causes the crash? The crash doesn't occur every time, it seems to be quite random. I'm using MVTools v1.8.5.1 which should be multithreading-compliant.

This is one example of a crashing script:
SetMTMode(5)
LoadPlugin("C:\Program Files\DVD-RB PRO\DGDecode.dll")
mpeg2source("D:\TEMP\DVD-RIP\REBUILDER\D2VAVS\V01.D2V",idct=3)
SetMTMode(2)
Import("c:\progressive.avs")
trim(4596,4739)
ConvertToYV12()
The progressive.avs is
den=FFT3DFilter(sigma=7,bt=3,plane=4)
Crop(0,78,-0,-78,true)
DegrainFFTC(last,den,sad=200,limy=1,ol=8)
AddBorders(0,78,0,78)
Distributor()
ChangeFPS(last,last,true)
DegrainFFTC is in degrain.avsi which is
global idx_1 = 100
global idx_2 = 200

function Degrain( clip c, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(c,isb=true,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(c,isb=false,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(c,isb=true,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(c,isb=false,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl),c,limy,limuv) : nolimit
}

function DegrainC( clip c, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(c,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(c,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(c,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(c,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl),c,limy,limuv) : nolimit
}

function DegrainFFTC( clip c, clip cleaned, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(cleaned,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(cleaned,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(cleaned,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(cleaned,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl),c,limy,limuv) : nolimit
}

function DegrainFFT( clip c, clip cleaned, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(cleaned,isb=true,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(cleaned,isb=false,truemotion=true,delta=1,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(cleaned,isb=true,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(cleaned,isb=false,truemotion=true,delta=2,pel=2,chroma=false,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl),c,limy,limuv) : nolimit
}

function Stabilize( clip o, bool "compare" )
{
compare = default( compare, false )
f = o.MinBlur(1,2).MinBlur(2,2).RemoveGrain(11,-1)
f.FluxSmoothT(7).mt_AddDiff(mt_MakeDiff(o,f,U=1,V=1),U=4,V=4)
mt_LutXY(last,o,"x 2 + y < x 2 + x 2 - y > x 2 - x ? ?",U=2,V=2)
compare ? Interleave(o,last) : last
}

I've also noticed that having Distributor() in the script causes a lot of problems when loading the script in VDubMod. Most of the time, the application just vanishes when I try to load the script.

EDIT: I have even more crashes when using Didée's function found in this post : http://forum.doom9.org/showthread.php?p=1076491#post1076491

[Zelos]

works fine with megui and automen :)
on automkv , i had no gain , but the quadcores were at 100% before the mip trick, so it's normal.
In megui and automen, the quadcores were at 50%-70% max before the mip , and reach almost 100% after mip trick.
Great works @tsp !
thanks.

[EasyStart]

If my video source is vcd 352x288 resolution, can I run LimitedSharpenFaster( ss_x=2.5, ss_y=2.25, strength=120, smode=4, dest_x=720, dest_y=576 ) inside mt() ?

Easystart

[foxyshadis]

MT gets very angry if you change the height (or, with splitvertical=true, the width), so unless you want to rewrite the guts of the function you'll need to use SetMTMode.

[squid_80]

I've also noticed that having Distributor() in the script causes a lot of problems when loading the script in VDubMod. Most of the time, the application just vanishes when I try to load the script.Distributor() should only be used when loading the script into applications which talk to avisynth directly e.g. HC and MeGUI. If the program opens the .avs file as an avi file (like virtualdub/mod, x264, xvid_encraw etc.) you don't need (and shouldn't have) the distributor call at the end.

[tsp]

Boulder: The usual way is to figure out which filters that are causing the trouble by commenting them out one at a time but as the crash only occurs sometimes it is less useful. If the crash dialog appears you can see in which dll the crash occurred under the more info link.
Your script crashed every time for me both with and without setmtmode(). I will see if I can figure out why.

[Boulder]

I also emailed hank315 and described the situation, maybe he has some ideas. I tried batch encoding the same script (using 250 frames) 100 times with HC, the first crash occurred around the 80th encode.

I think I should try encoding the script without any imported stuff and having the functions in the script itself and see what that causes.

[Boulder]

Distributor() should only be used when loading the script into applications which talk to avisynth directly e.g. HC and MeGUI. If the program opens the .avs file as an avi file (like virtualdub/mod, x264, xvid_encraw etc.) you don't need (and shouldn't have) the distributor call at the end.What about MeGUI calling x264, doesn't it get all confused when Distributor() is being used in the script?

I'm going to do some testing as my time allows, starting with a simple script and adding complex functions along the way. I'm particularly interested in seeing if MaskTools v2 causes any instability as Didée's cleaning function includes that one compared to my simple MVDegrain2-function.

What about global idx values, are they needed with the MT-friendly MVTools?

[vcmohan]

My search for Distributor did not yield satisfactory results. I also found no mention in Avisynth main page, external plugins. Is this a plugin or a script function? Can a reference to it be posted. If a plugin then can it be placed in new plugins, functios sticky thread?

[squid_80]

The discussion regarding Distributor started in this thead, at this post (http://forum.doom9.org/showthread.php?p=1050236#post1050236).

[IanB]

@vcmohan,

Distributor is an internal part of TSP's Avisynth_MT. It is not intended for normal use, but due to some mis-interaction with other applications it has come to light.

[tsp]

Boulder: It is unnecessary to add distibutor at the end when megui is calling x264. It shouldn't crash if added to the end of the script if the mtmode is between 1 and 4.

The global idx values are still needed to avoid problem if you call the functions more than once in a script

[Morte66]

Hi guys.

I'd like to experiment with a strong denoiser for pre-filtered clips that won't end up in the final video, e.g. to use for motion analysis with mvanalyse but not final denoising with mvdegrain2. I want something MT-friendly that won't hurt overall speed too much.

I've gotten in the habit of fft3dgpu as my fast denoiser, but I don't think it plays ball with MT; and I'm not up to speed on other denoisers. I've tried fft3dfilter, but it slows the overall script too much (and doesn't multithread that well either). Do you guys have any suggestions for a strong denoiser that's an order of magnitude faster than fft3dfilter (bt=5) and works nicely with MT?

My first thought was Convolution3D. Apparently it's 3 frame spatio-temporal, does anybody know if it works with MTMode 2?

[Boulder]

Why not try DegrainMedian, it gets a huge boost at least when wrapped inside an MT call. It's also quite powerful.

[Morte66]

Why not try DegrainMedian, it gets a huge boost at least when wrapped inside an MT call. It's also quite powerful.

I remember trying DegrainMedian about a year ago, and finding that it sort of switched off around moving edges. It's one of the things that got me into motion compensated denoising in the first place. So it might not be ideal for pre-filtering a motion analysis clip. But I'll give it a go.

I figure I ought to try a purely spatial denoiser for the pre-filtering.

[foxyshadis]

Personally, I like VagueDenoiser and some of vcmohan's filters for that. They have a large radius and are really good at highlighting edges. Alternately minblur, or a concoction with mt_edge.

[Morte66]

I tried prefiltering with degrainmedian(mode=0,limitY=8,limitUV=15).HQdn3D(4,3,6,6) and found that:
- It works fine with MT, the speed and CPU utilisation are good.
- IMO the effect was positive but not huge, arguably worth the time.
- It seems to make more difference with smaller blksize in mvanalyse. [The mvtools docs do say that mvanalyse rejects noise better for large blocks.]

I will have a go at the filters foxy suggested next.

[Zep]

ok here is one for the gurus :)

The below code is from a script Didee wrote and it works fine when not using setmtmode but as soon as I use it even with just a
SetMTMode(5,5) vdub/avisynth crash with an exception error. it works fine with no setmtmode.

it crashes on all the .mt_merge(alt,SAD_***,U=3,V=3)

if I comment them out no crash but of course it then does not do what it is supposed to do :D

I get 3x speed up when I use SetMTMode(2,5) so I would love to get this to work.

thoughts?



thanks



function Scripted_MVDegrain3(clip c, clip "mvbw", clip "mvfw", clip "mvbw2", clip "mvfw2", clip "mvbw3", clip "mvfw3",
\ int "thSAD", int "plane", int "limit", clip "pelclip", int "idx")
{
thSAD = default(thSAD, 400)
plane = default(plane, 4)
limit = default(limit, 255)
_idx = default(idx, -11)

thSAD = thSAD / 8

alt = c.FFT3DFilter(sigma=10,sigma2=6,sigma3=4,sigma4=2,bw=16,bh=16,ow=8,oh=8,bt=1,degrid=1,ncpu=1)

SAD_fw3 = c.MVMask(mvfw3, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_fw2 = c.MVMask(mvfw2, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_fw1 = c.MVMask(mvfw, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_bw1 = c.MVMask(mvbw, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_bw2 = c.MVMask(mvbw2, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_bw3 = c.MVMask(mvbw3, kind=1, ml=thSAD, gamma=0.999, Ysc=255)

comp_fw3 = c.MVCompensate(mvfw3, idx=_idx).mt_merge(alt,SAD_fw3,U=3,V=3)
comp_fw2 = c.MVCompensate(mvfw2, idx=_idx).mt_merge(alt,SAD_fw2,U=3,V=3)
comp_fw1 = c.MVCompensate(mvfw, idx=_idx).mt_merge(alt,SAD_fw1,U=3,V=3)
comp_bw1 = c.MVCompensate(mvbw, idx=_idx).mt_merge(alt,SAD_bw1,U=3,V=3)
comp_bw2 = c.MVCompensate(mvbw2, idx=_idx).mt_merge(alt,SAD_bw2,U=3,V=3)
comp_bw3 = c.MVCompensate(mvbw3, idx=_idx).mt_merge(alt,SAD_bw3,U=3,V=3)

black = blankclip(c,color_yuv=$008080)
long = interleave( comp_fw3,comp_fw2,comp_fw1, c, comp_bw1,comp_bw2,comp_bw3 )

long.temporalsoften(3,255,255,24,2)
SelectEvery(7,3)
}

[Boulder]

I've also had more instability when having Masktools v2 stuff in the function I've used with SetMTMode(2).

[Zep]

I've also had more instability when having Masktools v2 stuff in the function I've used with SetMTMode(2).

trouble is for me anyway it crashes hard every time before even showing the first frame. even with a setmtmode(5) and that is something I never saw before. this is the first time that mode 5 is not safe for a call I have run into so I can not even mode 5 that part out and keep the rest of script at mode 2.

[-=KaMaL=-]

Hello everybody i have big problem with MT when i add in my script anyone script in little script also SetMTMode(2,8), in 1st pass after few mins xvid_encraw.exe is close automatically (xvid_encraw.exe is the processus of the window where i can see the status of my rip like in pourcent ) but in megui in queue for the status of the job i can see error
please help me thanks in advance also i have in processor intel duo quad q6600

[Razorholt]

can we use SetMTMode with Avisynth 2.5.8 now?

Thanks,
- Dan

[Sagekilla]

I don't believe the 2.5.8 build has SetMTmode inside of it right now. Also, the latest version in this thread seems to be 2.5.7. I could be wrong, though.

[Dreassica]

I have core2duo, 1GB RAM

I installed avisynth 2.5.7

then from MT_07 package, I have put MT.dll in plugins directory & avisynth.dll from the same pack to windows/system32 folder

I am getting this error, while loading script into Vdub

AVI Import Filter error: (Unknown) (80040154)

If I put original avisynth.dll into system32 folder ... it works fine but then, I am not able to use MT

pls help

regards

I have same problem now as he does, but he didn't mention what he did to fix it. I can't for the life of it fidn the reason no script will load anymore when I use the MT modded avisnth dll.
Version() does same btw.

[alph@]

i would like to know, if there is a difference in this way of using 'MT'

1.ex.
SetMemoryMax(700)
SetMTMode(2,0) dull = last
sharp = dull.LimitedSharpenFaster(ss_x=1.0, ss_y=1.0, Smode=3, strength=180, overshoot=7)
Soothe( sharp, dull,30 )


2.ex.
SetMemoryMax(700)
MT("dull = last
sharp = dull.LimitedSharpenFaster(ss_x=1.0, ss_y=1.0, Smode=3, strength=180, overshoot=7)
Soothe( sharp, dull,30 )",2)


If there is a difference, which one ?
thanks.

[tsp]

alph@: this might explain it: http://avisynth.org/mediawiki/MT_support_page#Differences_between_MT.28.29_and_SetMTMode.28.29

Dreassica: A missing dll might be the problem: msvcr71.dll msvcp71.dll should be in the windowssytem32 directory

[alph@]

Thank you Tsp,
Is there a mode which is more adviced than another for the used of the limitedsharpen, and for the speed, are they the same ?

[tsp]

most of the times MT() is faster than Setmtmode but for filters that requires information about the complete frame MT() gives wrong result due to the way it works (like a smart deinterlacer that only deinterlace half of the frame). LimitedSharpen doesn't need information about the complete frame so MT() works slightly better

[Dreassica]

THanks, dlls in system32 made it work, but setting setmtmode(2,4) made fps go up like 0.1fps max, so basically nothing.

[yup]

Hi folk!
Please advice could be work MT on Intel Pentium 4 CPU 531 3.00GHz, which have only hyperthreading.
I try use MVDegrain2 with SetMTMode(2) and do not see speed up, before I read doc and install customized avisynth.dll to system32 folder and mt.dll to plugin folder.
yup.

[tsp]

yup: It's not always that hyperthreading gives any speedup. It depends on the script

Dreassica: Again the speed increase is very dependent on the script used and how slow the encoder is.

[yup]

tsp!
Thank you for reply.
I use script for MVDegrain in MT thread, I do not use encoding. My source AVI caapture and I save this file in VirtualDub using huffyuv as codec.
yup.

[Undead Sega]

so ifi wanted to runa very slow process like MCBob, do i just download, and install this like an ordinary avisynth and keep the same scripting as well?

[Adub]

Make sure you enable multithreading in your script with something like "SetMTMode(2)" or "MT(filter, 3)" or something.

Edit:Oh, and also remember to update your version of Masktools. The older versions aren't compatible with MT.

Just read the documentation and it will make sense.

[valnar]

Make sure you enable multithreading in your script with something like "SetMTMode(2)" or "MT(filter, 3)" or something.

Edit:Oh, and also remember to update your version of Masktools. The older versions aren't compatible with MT.

Just read the documentation and it will make sense.

I found this thread from a post on the AutoMKV thread. Forgive me for not reading all 41 pages - I just had a simple question.

Is there any reason not to include SMP support (or detection) in the base AVISynth application now that multicore CPU's are becoming more common?

-Robert

[Adub]

I meant the Documentation contained in the MT zip file.

[Mug Funky]

sorry if this has already been covered, but i was wondering if it was possible to set processor affinity for different filters using MT.

for example, i'm doing some very heavy mvtooling, and it'd be good if i could split the mvanalyse/mvmask/masktooling up between processors (i'm making 6 motion compensated frames and averaging them with the current in fancy ways).

maybe something like

setaffinity(<cpu_1>)
...
filters()
...
setaffinity(<cpu_2>)
...
filters()
...
setaffinity(<cpu_3>)
...
filters()
...

etc

basically i have a dual quad and a single quad and want to make them earn their keep.

the other option is splitting frames into chunks but i rather like the idea of operating on complete frames for mocomp stuff.

denoising 2k stuff is quite slow using 1 core :)

[Mr VacBob]

The modified avisynth MT dll gave a "Script load error" with no details whenever I opened anything, even Version(). It turned out to be caused by me saving Didee's scripts as avsi files using UTF-8, because I didn't want to lose the accent in the name...

[Ranguvar]

@Mug Funky: Yeah, just got my paws on a Q6600, and a SetAffinity function would rule.

[IanB]

Okay, I'll bite. How will locking processor affinity help in any way in avisynth processing?

The multi-threading code allows the frame rendering task to be distributed between the available processor cores.

Currently there are 2 models :-
1. Split frames into strips and give a strip to each processor, i.e MT()
2. Pre-render frames in anticipation, i.e. SetMTMode()

With either or both models the amount of improvement relies on being able to split the work enough ways to keep all the processor cores active.

Are you proposing a 3rd model for spliting the work? if so please elaborate.

[MfA]

Getting rid of one or two buffer evictions from cache won't make a huge difference for performance ... and you run a big risk of letting processors stand idle needlessly.

[Atak_Snajpera]

How to use MT() with Yadif deinterlacer?

video=MT("yadif(video,mode=0,order=1)",2) = invalid arument ?!?!?!

[Ranguvar]

Okay, I'll bite. How will locking processor affinity help in any way in avisynth processing?

The multi-threading code allows the frame rendering task to be distributed between the available processor cores.

Currently there are 2 models :-
1. Split frames into strips and give a strip to each processor, i.e MT()
2. Pre-render frames in anticipation, i.e. SetMTMode()

With either or both models the amount of improvement relies on being able to split the work enough ways to keep all the processor cores active.

Are you proposing a 3rd model for spliting the work? if so please elaborate.
Setting affinity would definitely not be as efficient as using MT() or SetMTMode(), but it would still help on functions that don't like MT.

[IanB]

@Ranguvar,

I still don't see how it helps in any way.

Avisynth renders frames by calling the GetFrame method of the filter object at the end of the graph built from a script. Each filter object in turn calls the GetFrame method of the object before it in the graph. Before it can do any work a filter must wait for that GetFrame call to return a video frame.

So simply enforcing affinity achieves nothing, you still need to divide up the work somehow.

[Ranguvar]

Ach. You're right. So, affinity would need to be applied at the VERY start, which is pointless anyways since MT does that better (distribution instead of simple 1 core per task)... etc.

Thanks for the enlightenment :) Learning is good.

[Jeremy Duncan]

I think what people are talking about with affinity.
Is that MT be given a method to see what cpu is idle or at 25% and take a thread going to a maxed out core and put it to the idle core.

How it could be done, is the mt can see if a core is at a certain percent, and if it is then it can put a thread on a different core.
It would do this for each core, seeing if it needs to offload a thread, until all the cores are maxed out.

I think it could be done if MT can see if a certain number of thread is maxing out one core. And if it is, distribute the threads differently. :helpful:

[IanB]

If a thread is ready to run, i.e. not waiting for a lock, etc AND there is a spare processor core THEN it will run.

[Mug Funky]

So simply enforcing affinity achieves nothing, you still need to divide up the work somehow.

that's what i intend to do :)

basically, i've got a mocomped denoise script quite similar to mvdegrain3 (but i like control over tweakage, right or wrong).

say i have 8 cores to play with. my script needs to compensate 3 frames either side of the current frame. that means running 6 mvanalyse calls, as well as mvmask and various other masktools calls.

if i could give each mvanalyse to a different core, and handle the masky stuff on another core, then have another one just for the ride, it could be quite useful.

basically i'd like to use mvtools more efficiently, and this seems like a cool way to do it.

[MfA]

It would be useful if it were faster ... if Avisynth were to simply run each independent branch of your filter in it's own thread then most of the time the data of such a branch would stay local to a processor anyway without trying to force affinity. Forcing affinity only enforces idle processors. Moving data is a better option than idleness, it's not like with such slow plugins the time it takes to fill the cache is a huge deal.

[Jeremy Duncan]

If a thread is ready to run, i.e. not waiting for a lock, etc AND there is a spare processor core THEN it will run.

But the thread will run even though the core is at a certain used percentage, so it will go slower this way than if once the core reached a certain used percentage, it send threads to a different core. In ffdshow, if the core passes 70% then it's going to affect performance.
So if you could set mt to send a thread to a different core once one core has reached 70% it would boost overall performance, in ffdshow at least.

[IanB]

... if Avisynth were to simply run each independent branch of your filter in it's own thread ...And that is the whole point nothing in avisynth is independant, everywhere something is waiting for a GetFrame call to complete. Cunning filter authors can multi thread there filters GetFrame calls and internal processing, but then that is a feature of that one filter not avisynth. We need to find ways to simulate independance.

@Jeremy Duncan,

Threads running has nothing to do with percentage used of any given core. If there is a thread in any process ready to run and there is a spare processor core then that thread will run! And yes most of the time all the processor cores are doing nothing because all of the threads in all of the processes are waiting for something to complete, i.e. a disk i/o to finish or a network packet to arrive or an extreeeeemly slow human to push a button or move a mouse.

Think about the scale of things here. :-

Modern cpu's run a GigaHertz clock i.e. instructions run in the nanosecond range.

Really fast ESata 10000+rpm disks are still around the milliSecond range.

You beaut Gigabit networks can do 10's of microSeconds under ideal conditions, but thats for a 1 off transaction. Real network protocols need 1000's of transactions to do real work.

Humans, we live upwards of the 1/4 second mark.

[Jeremy Duncan]

Could there be a trigger for mt to move a thread, besides the one in place that simply waits till the one core in use can't run the thread?

I mean, if mt can sense to move the thread to a free core in the first place.
Couldn't a rule be made to do this at a certain core usage instead of the way it's being done now, where mt waits till the core in use can't be used and a new core Has to be used for the thread?

It would run faster if mt looked for a new core at 70% core usage.

[MfA]

And that is the whole point nothing in avisynth is independant, everywhere something is waiting for a GetFrame call to complete.
Even without the MT methods that isn't necessarily true. With branches in the graph multiple plugins can in theory be waiting for the same GetFrame call to complete, after which those branches become independent.

PS. maybe avisynth should have a way for plugins to say they are safe for split frame rendering, together with what overlap they need to do it without artifacts. SetMTMode is the more elegant way of creating parallelism, and should be the default, but when buffers stop fitting in L2/3 and spill to main memory split frame rendering becomes more efficient (which with very large resolutions and/or floating point is quite possible).

[IanB]

@Jeremy Duncan,

Go and study up on how multitasking works!

Then re-read my last 2 posts.

It's really simple, if a thread is not ready to run, then no amount of shuffling will make it run. It will only run when the resource it is waiting for becomes available.

@MfA,

Yes it is totally true, "nothing in avisynth is independant", the graph is always purely linearly dependant.

Branches in a graph are the result a filter having 2 PClip sources. Only that instance of that filter knows this.

[MfA]

Oops, yes of course ... wasn't thinking.

[Nikos]

I had read all the thread and i am confuse about the correct idx's values in mvtoos with SetMTMode(). In mvtools manual there are informations only for the MT().
Which of the following scripts are corrects?
If none, please someone experienced user let's write the correct one.

1.

SetMTMode(2,4)
.....

source=last
prefilter=source.fft3dfilter()

b2_vec=prefilter.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=1)
b1_vec=prefilter.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f1_vec=prefilter.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f2_vec=prefilter.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=1)

source.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=2)


2.

SetMTMode(2,4)
.....

source=last
prefilter=source.fft3dfilter()

b2_vec=prefilter.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=1)
b1_vec=prefilter.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=2)
f1_vec=prefilter.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=3)
f2_vec=prefilter.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=4)

source.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=5)


3.

SetMTMode(2,4)
.....

global idx_1 = 100
global idx_1 = idx_1 + 1

source=last
prefilter=source.fft3dfilter()

b2_vec=prefilter.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=idx_1)
b1_vec=prefilter.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=idx_1)
f1_vec=prefilter.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=idx_1)
f2_vec=prefilter.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=idx_1)

source.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=idx_1)


4.

SetMTMode(2,4)
.....

global idx_1 = 100
global idx_2 = 200

global idx_1 = idx_1 + 1
global idx_2 = idx_2 + 1

source=last
prefilter=source.fft3dfilter()

b2_vec=prefilter.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=idx_1)
b1_vec=prefilter.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=idx_1)
f1_vec=prefilter.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=idx_1)
f2_vec=prefilter.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=idx_1)

source.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=idx_2)


:thanks:

[foxyshadis]

#1 is fully correct. #4 is nearly correct but the idx initializations have to be above the setmtmode call, though it won't matter with current mvtools. #2 & #3 are just plain wrong use of idx.

[Nikos]

Thank you foxyshadis for the quick answer.
Now with prefilter and external denoiser the below script is correct?


SetMTMode(2,4)
.....

source=last
prefilter=source.fft3dfilter()

b2_vec=prefilter.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=1)
b1_vec=prefilter.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f1_vec=prefilter.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f2_vec=prefilter.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=1)

b2_comp=source.MVCompensate(b2_vec, idx=2)
b1_comp=source.MVCompensate(b1_vec, idx=2)
f1_comp=source.MVCompensate(f1_vec, idx=2)
f2_comp=source.MVCompensate(f2_vec, idx=2)

inter=interleave(f2_comp, f1_comp, source, b1_comp, b2_comp)

denoise=inter.mydenoise()
denoise.selectevery(5,2)

I notice that, with SetMTMode() the idx's, if i use numbers (1, 2, 3, ...) it's the same like original avisynth and only with MT() there is difference in idx's.
I am correct or not?

Edit:
I corrected the script (Blue letters).

[foxyshadis]

has to be =inter.mydenoise(), but yes. Your way will run very fast, though ;) I don't use MT() at all anymore so I don't know.

[Nikos]

Thanks again foxyshadis.
Now the last question about the nasty idx's, because i want to be sure :)
The below scripts are correct?

1. MVDegrain2 without prefilter .

SetMTMode(2,4)
.....

source=last

b2_vec=source.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=1)
b1_vec=source.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f1_vec=source.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f2_vec=source.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=1)

source.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=1)


2. MVTools without prefilter but with external denoiser.

SetMTMode(2,4)
.....

source=last

b2_vec=source.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=1)
b1_vec=source.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f1_vec=source.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f2_vec=source.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=1)

b2_comp=source.MVCompensate(b2_vec, idx=1)
b1_comp=source.MVCompensate(b1_vec, idx=1)
f1_comp=source.MVCompensate(f1_vec, idx=1)
f2_comp=source.MVCompensate(f2_vec, idx=1)

inter=interleave(f2_comp, f1_comp, source, b1_comp, b2_comp)

denoise=inter.mydenoise()
denoise.selectevery(5,2)



3. If i combine two MVDegrain2 with prefilter the idx's are correct?
Especially the idx with red color must be 3 or 2?


SetMTMode(2,4)
.....

source=last
prefilter=source.fft3dfilter()

b2_vec=prefilter.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=1)
b1_vec=prefilter.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f1_vec=prefilter.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f2_vec=prefilter.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=1)

den1=source.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=2)
den1.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=3)


4. If i combine two MVDegrain2 without prefilter the idx's are correct?
Especially the idx with red color must be 2 or 1?


SetMTMode(2,4)
.....

source=last

b2_vec=source.MVAnalyse(isb=true, delta=2, blksize=8, overlap=4, pel=2, idx=1)
b1_vec=source.MVAnalyse(isb=true, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f1_vec=source.MVAnalyse(isb=false, delta=1, blksize=8, overlap=4, pel=2, idx=1)
f2_vec=source.MVAnalyse(isb=false, delta=2, blksize=8, overlap=4, pel=2, idx=1)

den1=source.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=1)
den1.MVDegrain2(b1_vec, f1_vec, b2_vec, f2_vec, idx=2)



5. May i combine MVDegrain2 with Motion Compensate external denoiser?
If yes, post the script please.

[foxyshadis]

All look correct; although that's a somewhat suboptimal way to double-up mvdegrain, it's definitely faster.

If you want to use mvdegrain + something like fft3d or dfttest, just combine 1 & 2. Do all the analyse and degrain, then another set of analyse, compensate, and filter, with new idx.

[Nikos]

Thank you very much foxyshadis for the confirmation, now i feel sure for my long time encodings :)

It's a good idea, Fizick to be include some examples in the new mvtools manual.

[Livesms]

I found a problem last time
SetMTMode(2) speed up MVDegrain2 from 4.65FPS to 9.31FPS (in my Core2Duo E6600)

SetMTmode(2)
#------------------------------------------------------------------------------------------------------#
function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
overlap=default(overlap,0)
dct=default(dct,0)
idx=default(idx,1)
fields=source.SeparateFields()
backward_vec1 = fields.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec1 = fields.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=overlap, idx = idx,dct=dct)
backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
fields.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx)
Weave()
}
#------------------------------------------------------------------------------------------------------#
MPEG2Source("0.d2v", info=3)
ColorMatrix(hints=true,interlaced=true)
Interleave(Crop(0, Height/2, 0, 0), Crop(0, 0, 0, Height/2)).AssumeFieldBased().AssumeBFF().Weave().AssumeTFF()
MVDegrain2i(4,0,1)
#AssumeTFF().TDeint(order=-1).Crop(4,40,-4,-48).SimpleResize(552,356)
#DeGrainMedian(mode=3,limitY=5,limitUV=5,interlaced=false)
FadeIO(25)

But when I try to decoment (add to script)
TDeint(order=-1)
DeGrainMedian(mode=3,limitY=5,limitUV=5,interlaced=false)
I have ~4.5FPS backю CPU load near 50%

Why script slow down with SetMTMode(2) and TDeint or/and DeGrainMedian?

SetMTMode(3), SetMTMode(4) gives me error for
ColorMatrix(hints=true,interlaced=true) and then TDeint()
http://keep4u.ru/imgs/s/080306/22/2293406173b57e692b.jpg (http://keep4u.ru/full/080306/2293406173b57e692b/jpg)

[Ranguvar]

SetMTMode() doesn't like me :p

This script works in AvsP preview, but crashes all media players without error message and locks VDub:

SetMTMode(1,4)
AVISource("Source.avi", audio=false)
#FFT3DGPU(sigma=2.5, bt=4)
Deblock_QED(quant1=60, quant2=80)
Spline36Resize(640, 480)

This one works:
AVISource("Source.avi", audio=false)
SetMTMode(1,4)
FFT3DGPU(sigma=2.5, bt=4)
Deblock_QED(quant1=60, quant2=80)
Spline36Resize(640, 480)

(And yes, if I disable FFT3DGPU in the second it still works)

No other mode works either.

[Boulder]

The latter one works because it doesn't multithread at all. The first SetMTMode call must be before loading the source.

You shouldn't use multithreading for loading the source, try
SetMTMode(5,4)
AVISource("Source.avi", audio=false)
SetMTMode(1,4) # or SetMTMode(2,4)
#FFT3DGPU(sigma=2.5, bt=4)
Deblock_QED(quant1=60, quant2=80)
Spline36Resize(640, 480)FFT3DGPU doesn't work properly with multithreading.

[Livesms]

Is there any way to use DegrainMedian with SetMTMode?
And what about Deinterlace?
I need Deinterlace with multithread to.

[Boulder]

DegrainMedian should work just fine, and deinterlacers too. I think DegrainMedian benefits much more from using MT than using SetMTMode.

Using multiple threads for processing doesn't mean that the CPU usage will always be well past 50%. Multithreading is not a simple thing, there are many factors that affect the performance.

[Livesms]

DegrainMedian should work just fine, and deinterlacers too. I think DegrainMedian benefits much more from using MT than using SetMTMode.

Using multiple threads for processing doesn't mean that the CPU usage will always be well past 50%. Multithreading is not a simple thing, there are many factors that affect the performance.

Ok, why script
SetMTmode(2)
#------------------------------------------------------------------------------------------------------#
function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
overlap=default(overlap,0)
dct=default(dct,0)
idx=default(idx,1)
fields=source.SeparateFields()
backward_vec1 = fields.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec1 = fields.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=overlap, idx = idx,dct=dct)
backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
fields.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx)
Weave()
}
#------------------------------------------------------------------------------------------------------#
MPEG2Source("0.d2v", info=3)
ColorMatrix(hints=true,interlaced=true)
MVDegrain2i(4,0,1)
FadeIO(25)
Gives 9.3FPS and 100% load when code
SetMTmode(2)
#------------------------------------------------------------------------------------------------------#
function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx")
{
overlap=default(overlap,0)
dct=default(dct,0)
idx=default(idx,1)
fields=source.SeparateFields()
backward_vec1 = fields.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec1 = fields.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=overlap, idx = idx,dct=dct)
backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct)
fields.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx)
Weave()
}
#------------------------------------------------------------------------------------------------------#
MPEG2Source("0.d2v", info=3)
ColorMatrix(hints=true,interlaced=true)
MVDegrain2i(4,0,1)
AssumeTFF().TDeint(order=-1).Crop(4,40,-4,-48).SimpleResize(552,356)
DeGrainMedian(mode=3,limitY=5,limitUV=5,interlaced=false)
FadeIO(25)
Gives 4.5fps with 50% CPU Load

[Ranguvar]

The latter one works because it doesn't multithread at all. The first SetMTMode call must be before loading the source.

You shouldn't use multithreading for loading the source, try
SetMTMode(5,4)
AVISource("Source.avi", audio=false)
SetMTMode(1,4) # or SetMTMode(2,4)
#FFT3DGPU(sigma=2.5, bt=4)
Deblock_QED(quant1=60, quant2=80)
Spline36Resize(640, 480)FFT3DGPU doesn't work properly with multithreading.
Nope, doesn't work. I did say, other modes didn't work either.

I got a lovely purple-ish screen with a grid of normal-colored lines (not perfect lines) for a split second before the crash.

[Boulder]

@Livesms:You are adding complexity to the second script. The different functions may not scale so well as they are not optimized for multithreading. There are Avisynth cache issues etc. that affect performance.

(a side note: Do not use SetMTMode(2) for loading the source, use SetMTMode(5) and then SetMTMode(2) after the source is loaded)

[Boulder]

Nope, doesn't work. I did say, other modes didn't work either.

I got a lovely purple-ish screen with a grid of normal-colored lines (not perfect lines) for a split second before the crash.Using SetMTMode(5) before FFT3DGPU should work because it disables multithreading. If it doesn't, maybe tsp has some ideas since he's the author of both MT and FFT3DGPU.

[Livesms]

@Livesms:You are adding complexity to the second script. The different functions may not scale so well as they are not optimized for multithreading. There are Avisynth cache issues etc. that affect performance.

(a side note: Do not use SetMTMode(2) for loading the source, use SetMTMode(5) and then SetMTMode(2) after the source is loaded)

I tested SecureBob from MvBob.
Rather good result (quality) but without multithread support - setmtmode gives no gain

Is there any Deinterlace with speed increase and 100% MultiCore load while using SetMTMode

[Ranguvar]

Well, my PC's going crazy anyways... random crashing in every app. I'm going to reformat+reinstall, we'll see if that fixes it.

UPDATE: Well, I've got some form of hardware problem :P So please disregard the errors I reported.

[Deano123]

When "SetMTMode(4)" is used xvid_encraw crashes (MeGUI) heres the log...

Log:

MeGUI Version: 0.2.6.1045
OS used: Microsoft Windows XP Professional Service Pack 2
Framework used: 2.0 SP1
------------------------------------------------------

Looking for job processor for job...

Processor found!



------------------------------------------------------


Starting job job1 at 15:07:08

Starting preprocessing of job...

Preprocessing finished!

successfully started encoding

Processing ended at 15:12:11

------------------------------------------------------

Log for job job1

Job commandline: "C:\Documents and Settings\****\My Documents\DVDRip Programs\MeGUI\tools\xvid_encraw\xvid_encraw.exe" -i "C:\Documents and Settings\****\My Documents\DVDRip Process\DV2\Humraaz Script.avs" -pass1 "C:\Documents and Settings\****\My Documents\DVDRip Process\DV2\Humraaz Script.stats" -bitrate 1019 -kboost 100 -overhead 0 -turbo -nopacked -vhqmode 4 -qtype 1 -closed_gop -bvhq -par 1:1 -threads 0
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


------------------------------------------------------

End of log for job1
------------------------------------------------------

[Ranguvar]

Post your script please...

[dansus]

Hi, ive been using mt.dll happily for several months on SetMTMode(2)

All of a sudden my fps on the first pass has dropped from 125fps to 70fps and 100% to 70% cpu.

I use the same script as ever with no changes to anything im aware of.

I tried removing SetMTMode(2) to see if i lost any speed but it behaved exactly as described, so im guessing its not picking mt.dll any more, but i have no idea why.

Any help appreciated. (urgent)

Thanks.

ps: ive renewed the mt.dll package in plugins and system32.

[Ranguvar]

I don't think mt.dll needs to be in System32.

Try renewing the avisynth.dll that came with MT in system32.

[weisskreuz]

it went error on a d2v(contact 6 vob) process through megui
if i use setmtmode in the avs, process will stop at next vob file
if use mt or not using any mt function, status will be error, and won't start process
but, if use mplayer for preview, there was no problem

only i change avisynth.dll back to 2.58 070518, process runs

[Ranguvar]

Question. When using SetMTMode(), since it gives each core a different frame, can it ever hurt the quality of a video being processed with advanced temporal functions that still works with SetMTMode? Or will it only be less efficient, or not work?

I already know MT(), since it divides spatially, can hurt some things like MVTools.

Thanks.

[foxyshadis]

Unless there's a lot of global bookkeeping (as in the old MVtools), the output of any script should always be 100% identical with or without SetMTMode. Threads do get different frames, but they're never processed without every dependent frame prior in the script already prepared.

Sometimes Directshowsource can screw things up if the script has big jumps, though.

[TheRyuu]

Sometimes Directshowsource can screw things up if the script has big jumps, though.

Which is why we have ffmpegsource :)

[saint-francis]

OK then, I have a question. When working with HD sources and Haali media splitter is used with directshowsource and setMTmode (2,0) the results are atrocious. The final product jumps about if the encoding finishes at all. I have band-aid fixed this issue in a fairly unsophisticated fashion by using MT later in the script before anything real intensive. Is there a more effective way of dealing with this then?

[Ranguvar]

OK then, I have a question. When working with HD sources and Haali media splitter is used with directshowsource and setMTmode (2,0) the results are atrocious. The final product jumps about if the encoding finishes at all. I have band-aid fixed this issue in a fairly unsophisticated fashion by using MT later in the script before anything real intensive. Is there a more effective way of dealing with this then?

SetMTMode will not work later in the script, unless you mean you're using MT() later in the script.

What's your script?

How about SetMTMode(3,0)?

[saint-francis]

SetMTMode will not work later in the script, unless you mean you're using MT() later in the script.

What's your script?

How about SetMTMode(3,0)?

Yes I mean MT() later in the script.
Never tried setMTmode (3,0). I'll give it a try. THX

Example:

DirectShowSource("D:\whatever movie.mkv",fps=23.9759856527702,audio=false)
crop()
Spline36Resize()
MT ("TTempSmooth (maxr=6, lthresh=4, cthresh=5, lmdiff=2, cmdiff=3, strength=2, scthresh=12.0, fp=true, vis_blur=0, debug=false, interlaced=false).FFT3DFILTER(sigma=1.9, ncpu=2, bw=14, bh=14, ow=7, oh=7, bt=2, hr=1)",3)

[jordisound]

what about new MVtools and MVdegrain3?
It works with MT? can anybody show me an script that works fine?

[KML]

Guys is it possible to use MT 0.7 with "PentiumD 3.00GHz"?

[Ranguvar]

@both above: Try'n'see.

[scharfis_brain]

I suggest not to use mvtools with mt.
the problem is the splitting of the frame.
a camera pan always will show lesser denoising and more artifacting at the borders where the frame has gotten split up for mt.

I hope some time we have a native mt-support within mvtools.dll

[KML]

I suggest not to use mvtools with mt.
the problem is the splitting of the frame.
a camera pan always will show lesser denoising and more artifacting at the borders where the frame has gotten split up for mt.

I hope some time we have a native mt-support within mvtools.dll

Thanks..

I should find a way to make fast mv-tools

[scharfis_brain]

Is it possible to assign each filter to a selectable thread eg.:

avisource("blah.avi")
thread("blur(1)",1) # assign blur(1) to cpu 1
thread("temporalsoften(3,12,13)",2) to cpu 2

this should make it possible to use multithreading with filters that cannot be used with split up frames or temporal order of the frames (eg. motion compensated filters)

I altered the frame of the documentation a bit:
http://home.arcor.de/scharfis_brain/samples/mt-request.png

[thetoof]

This would be AWESOME! Great idea :)

[cweb]

I'm really liking this. Please let us know if there is any update, or the mainstream avisynth becomes multithreaded enough to replace this.

[bestsoft666]

Perhaps by year end most of us will be able to afford a dual core though. Nice work once again tsp.

[Konrad Klar]

Sorry if this question was answered earlier.

LoadPlugin("J:\Program Files\AviSynth 2.5\plugins\exinpaint.dll")
Logo=ImageSource("K:\afd\WRHH1logoTL.bmp"). ConvertToYV12(matrix="pc.601")

AviSource("test.avi")

SetMTMode(2,2)
ExInpaint(last,Logo,color=$FF8080, Radius=100)

SetMTMode(2,2) and (1,2), (3,2), (4,2) does not give any speed advantage with ExInpaint(). CPU usage is 48-51% on my C2D (in MPC, VirtualDub, and HC22.1).
Is there any method to make ExInpaint working with (and benefiting from) SetMTMode?

[Boulder]

Move SetMTMode so that it is the first line of your script.

[Konrad Klar]

Move SetMTMode so that it is the first line of your script.

Many thanks, Boulder! Now it is working as intended.

Have you any suggestion, how improve following script:
SetMTMode(2,2)

LoadPlugin("J:\Program Files\AviSynth 2.5\plugins\exinpaint.dll")
Logo1=ImageSource("K:\afd\WRHH1logoTR.bmp"). ConvertToYV12(matrix="pc.601")
Logo2=ImageSource("K:\afd\WRHH1logoTL.bmp"). ConvertToYV12(matrix="pc.601")

Video=AviSource("example.avi", audio=false)

A=Trim(Video,0,115)
Exinpaint(A,Logo1,color=$FF8080, Radius=100)
LanczosResize(720,540)
A=last

B=Trim(Video,116,0)
Exinpaint(B,Logo2,color=$FF8080, Radius=100)
LanczosResize(720,540)
B=last

AlignedSplice(A,B)


SetMTMode(5,2)
FFT3DGPU()

SetMTMode(2,2)
AddBorders(0,18,0,18)
?

Source video has two types of logo at begining and in middle, hence AlignedSplice(A,B).

This script is rendered by VirtualDub at 99% CPU usage, however HCEnc 22.1 only utilize 49-52% while encoding it.

[Zep]

I just purchased 4 gigs of ram just so I could increase my thread count but I have run into what appears to be a memory wall. I am encoding 1080p HDTV to x.264 and everything works ok with SetMTMode(5,5) and it get an increase in FPS and CPU from 40% to 60% compared to not using it but I have a quad and 5 threads is not enough but when I try to use more than 5 threads I get a

"Microsoft visual C++ library: Run time error. this application asked runtime to terminate in an unusual way"

I have 3.2 gigs really since XP 32 bit even so I should have more than enough but and there is just over 1.1 gig of real ram (of the 3.2) still unused when I run my avs with 5 threads. If I try 6 more like I said above I get that error. It appears MT or avisynth or VDub will not use more than 2 gigs of real ram which surprised the hell out me :)



Now if I add a SetMemoryMax(64) to my avs I can do a SetMTMode(5,10) even since the total ram used stays under 2 gigs but it is super slow as now each thread does not have enough ram to do its thing. So I know it is not a thread COUNT problem Per Se' but appears to be a ram usage issue.


Has anyone run into this problem before? is MT limited to using 2 gigs? Is avisynth limited? Any thoughts on a work around?

thanks

[Konrad Klar]

It is rather off topic, but anyway:
Normally single process (not thread) can allocate up to 2GB. This is Windows 32bit limitation.
If Windows is launched with switch /3G some programs can allocate up to 3GB.
Some programs that uses AWE (Address Windowing Extensions) can allocate more memory that it has virtual address space.
http://en.wikipedia.org/wiki/Address_Windowing_Extensions

[foxyshadis]

This script is rendered by VirtualDub at 99% CPU usage, however HCEnc 22.1 only utilize 49-52% while encoding it.

HC uses custom avisynth routines, you have to add Distributor() to the end of the script. It's come up a couple of times in the thread already.

[thetoof]

@ Zep
Win32 has a 2GB process limit ... :D

[Zep]

It is rather off topic, but anyway:
Normally single process (not thread) can allocate up to 2GB. This is Windows 32bit limitation.
If Windows is launched with switch /3G some programs can allocate up to 3GB.
Some programs that uses AWE (Address Windowing Extensions) can allocate more memory that it has virtual address space.
http://en.wikipedia.org/wiki/Address_Windowing_Extensions

ouch :) 32 bit pointer is 4 gig so talk about a limiting apps grrrrrr guess I should have read up on XP before I trusted it lol

BTW - I feel it is on topic since not only is MT() the trigger but not using MT() would never come close to hitting that limit and besides knowing this limit is very important when using MT() now that HDTV and Bluray encodes with lots of cores are becoming common and we want more than than 40% CPU :)

thanks!

[Zep]

@ Zep

like I said OUCH!!! :D Yes I wish I read this whole thread lol


talk about a total bummer I go get 4 gigs just for this and find out XP (32) can only use 3.2 and now I find out apps can't even use that but are limited to 2 gigs. So now I guess I start searching about XP 64 and if avisynth can use it :)


thanks

[Zep]

ok found a work around and it appears to work fine so far in my 1 test run with setMTmode(5,8).

*************************************************
The /3GB switch allocates 3 GB of virtual address space to an application that uses IMAGE_FILE_LARGE_ADDRESS_AWARE in the process header. This switch allows applications to address 1 GB of additional virtual address space above 2 GB.
************************************************

just put that switch in your boot.ini like this

/FASTDETECT /3GB



There is another switch you can add also I see on the Microsoft site but I have not tried that yet.

read it all here

http://www.microsoft.com/whdc/system/platform/server/PAE/PAEdrv.mspx

and here

https://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx


thanks

[TSchniede]

I just tried for several days to get the best performance (while maintaining correct output) out of MVTools. I even tried to MT MVAnalyse with OpenMP. (It only got slower - I suppose the memory overhead and crating and closing Thread several times for each frame is way too high)

I can tell, that the MT method can never produce the same output , even for very large overlaps. That's because MVAnalyse gets its best predictors (the location where the search is started from) from the block surrounding it AND which have been fully computed on this hierarchy plane. So splitting will always miss data.

SetMTmode(2,0) as suggested both in this forum thread as in the documentation doesn't work.
-MT 0.7
-VirtualDub 1.7.8
-MVTools 1.9.2/3 (I'm testing MVDegrain3)
(And no, I am NOT overclocking any component of my C2Q9300, although it would run flawlessly at 3GHz - tested with prime95 for more than 12hours at default fan control, now I set it to better cooling)

I am still testing for safe settings or old versions which work to find the cause.

I can tell right now that real 100% CPU usage and many threads clearly increases errors (two clips encoded at the same time with SetMTmode(2,0) caused faulty frames (chroma green/purple frame from somewhere else with minor errors or "rainbow" with mostly garbage in the luma plane) about once every 100-2000 frames (strange - only the middle frame has visible changes but minor pixel errors appear around it, the faulty fames seem to be wrong before the sad is determined).
only one instance of Virtualdub which results in approximatively 50% CPU usage, produces errors up to 40000 frames apart.
The codec for the resulting file seems to be irrelevant (tried Virtualdubs uncompressed and Lagarith).

I am still verifying errors i got, so this is only a first analysis of the whole issue.

[Zep]

I just tried for several days to get the best performance (while maintaining correct output) out of MVTools. I even tried to MT MVAnalyse with OpenMP. (It only got slower - I suppose the memory overhead and crating and closing Thread several times for each frame is way too high)

I can tell, that the MT method can never produce the same output , even for very large overlaps. That's because MVAnalyse gets its best predictors (the location where the search is started from) from the block surrounding it AND which have been fully computed on this hierarchy plane. So splitting will always miss data.


that is why you need to overlap. trouble is if you over lap enough to catch all motion you slow it down so much you might as well not use MT()

[Konrad Klar]

SetMTMode(2,0)

LoadPlugin("J:\Program Files\AviSynth 2.5\plugins\exinpaint.dll")
Logo1=ImageSource("K:\afd\WRHH1logoTR.bmp"). ConvertToYV12(matrix="pc.601")
Logo2=ImageSource("K:\afd\WRHH1logoTL.bmp"). ConvertToYV12(matrix="pc.601")

Video=AviSource("example.avi", audio=false)

A=Trim(Video,0,115)
Exinpaint(A,Logo1,color=$FF8080, Radius=100)
LanczosResize(720,540)
A=last

B=Trim(Video,116,0)
Exinpaint(B,Logo2,color=$FF8080, Radius=100)
LanczosResize(720,540)
B=last

AlignedSplice(A,B)


SetMTMode(5,0)
FFT3DGPU()

SetMTMode(2,0)
AddBorders(0,18,0,18)

Distributor()

I have noticed that HCEnc 0.23 processing this script uses by most time only 50% of CPU (Core2Quad). Currently I have replaced:
SetMTMode(5,0)
FFT3DGPU()

SetMTMode(2,0)
by
FFT3DFilter()
and it gives significant speedup on my current config (and CPU usage is at 69-75%), but in future I will replace my 8600GT with faster gfx.

My question is about FFT3DGPU. FFT3DGPU does not work in pararrel threads for obvious reasons. Which mode would be adequate for this filter? Is SetMTMode(5,0) good or maybe it should be replaced by SetMTMode(5,1)?
Any suggestions?

[TSchniede]

Ok, now I have tried every possible variation I could come up.
The input clip is a full screen PAL capture in YUY2, which works flawlessly in singletreaded and up to #cores processes.

Using this simple Script:

setMTmode(5,0)

LoadPlugin("d:\avs game\filter\mvtools192.dll")

AVISource("testclipYUY2.avi")
#FFmpegSource("testclipYUY2.avi")
#DirectShowSource("testclipYUY2.avi", pixel_type="YUY2").converttoyuy2()
#MTsource("""AVISource("testclipYUY2.avi")""",delta=1,threads=1)

setMTmode(2)

idxref=30
ol=4
ts=320


backward_vec2 = source.MVAnalyse(isb = true, delta = 2, overlap=ol,idx=idxref)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, overlap=ol,idx=idxref)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, overlap=ol,idx=idxref)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, overlap=ol,idx=idxref)

source.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=ts,idx=idxref)

Avisource alone crashed always after a couple of frames with an out of bounds memory access,. no matter the used (huffyuv-2.1.1, huffyuv(decoded by ffdshow), Lagarith), YUY2 uncompressed is a bit more stable, same is avisource called by MTsource.
AssumeFPS and requestlinear(TIVTC) can reduce the number of bad frames.
Directshowsource didn't produce obvious errors, but it always converted to RGB.
FFmpegSource worked without any errors (output binary identical to single thread).
Generated data (blankclip or loops) based on the avisource worked without noticeable errors.
MT 0.6/0.7 and the included avisynth.dlls produced virtually identical results. the .4 crashed a bit less than the other two.
The hardware should not be the cause, I tried on my Q9300 and on my Athlon X2 3800+, fresh WinXP/SP2(3) 32-bit. Both worked without any errors single threaded (and multi process - 2 or 4 clips parallel). Even MT() in an otherwise single threaded script didn't produce the corrupted frames/crashes (I didn't check on minor errors yet). With SetMTMode !=0,5 or 6 on the whole script crashes and corrupted frames occurred. The number of processors and the cpu load seemed to increase failures. A strange fact puzzled me for some time - longer more complex scripts, where the errors first occurred were more stable in every aspect. So making the script simpler usually increased the errors. Simple scripts like Avisource("clip1.avi").reduceby2() produce errors too, MVdegrain2 only produces nearly 100% cpu load and works at least with some datasources without errors, MVDegrain2 from mvtoolsMTcomp2.zip or Fizicks builds have similar errors. Getting additional load to the cpu with prime95 torture tests provide nearly identical results on less cpu-intensive scripts.

So am I the only one with unusable AVISource in scripts with SetMTMode 1-4 somewhere in the script?
I guess a critical section in the getFrame method of AVISource would prevent this, as the corrupted frames, if they don't crash the whole script, seem to be shifted(with garbage on top or bottom)/garbage luma with "missing" chroma and the luma seems to belong to some frame several frames away. so either they refer to frames(memory) already freed, or not yet filled with the decompressed input.

[Graigddu]

Can i ask whether the MT filter could be used to speed up a noise filter like fluxsmooth in a
tools GUI such as AVS2DVD or FAVC while encoding with HCEnc and how i would go about setting
that up in a script.
Sorry but i'm a noobie when it comes to avisynth and haven't hade much luck with it

All help gratefully received

[TSchniede]

(Spatio-)temporal filter usually profit more from MT() than from SetMTMode. Which one is better or which one works (if any at all) has to be tested. First make sure your system can profit from multi threading (at least HT or dualcore). Then that you can use MT() or SetMTMode() (mode 2 or 3 for most filters) for example masktools-v2.0 work with MT() and SetMTMode(2).

IF both your system and the filter works then a script like the following would be what you seek:
#loadplugin("mt.dll") # or use the autoload feature (see installation on on first page)
Avisource("clip.avi")
MT("FluxSmoothT(7)")
#or MT("FluxSmoothST(7,7)",overlap=2) or use MTi on interlaced sources


SetMTMode(5)
#load your input here - for example FFMpegsource("clip.wmv")
SetMTMode(4) # 2 or 3 faster
FluxSmoothT(7)


Sorry, I don't know if fluxsmooth() works and my encoding rig is busy today. :)

[Graigddu]

Thanks for the scripts TShniede yes my computer is a dualcore so i'll have to give your script a try
and see which is the best at working with fluxsmooth if any

would this script still work if other avisynth scripts were run without MT first such as resize,crop

[TSchniede]

Of course if the simple script works (that does includes outright crashes, but most of the time only corrupted frames or "minor" pixel errors), other things can be added between the sourcefilter and fluxsmooth. In the best case the output is identical to single treaded operation.

crop or resize could be used freely multithreaded with SetMTMode(2), but not MT().

You could even use SetMTMode for everything except FluxSmooth.

if you use SetMTMode anywhere, then the first line has to be SetMTMode(5) or SetMTMode(5,2). Be careful with Avisource and SetMTMode!

This script should work:

SetMTMode(5)
FFMpegSource("input.clip")
SetMTMode(2)
crop(8,8,-8,-8).BilinearResize(1024,1024)
MT("FluxSmoothST()") #implicit change to mode 5 and back to 2
crop(16,16,-16,-16)

I don't know if you can really benefit from multithreading here, all involved filters are quite fast. A fast test over 10000 frames didn't produce errors, but at a cpu load of below 50% that doesn't say much.

[Graigddu]

think i follow you TSchniede

the script i had was this

# 4:3 encoding
AviSource("C:\Test Movies\test\test.avi", false)
ConvertToYUY2()
FadeIn(50)
Crop(0,16,720,544)
Lanczos4Resize(720,576,0.0,0.6)
loadplugin("mt.dll")
MT("FluxSmoothST(7,7)",overlap=2)

would this actually work or would SetMTMode for everything
be a better option.

really gratefull for your help as i'm totally lost on the whole when it comes to avisynth scripts.

[TSchniede]

If the script without MT() does what you intend to, that should be correct.
It should even be faster, if you use this:

SetMTMode(5)
# 4:3 encoding
loadplugin("mt.dll") # should be done outside of parallel region
AviSource("C:\Test Movies\test\test.avi", false)
SetMTMode(2)
ConvertToYUY2()
FadeIn(50)
Crop(0,16,720,544)
Lanczos4Resize(720,576,0.0,0.6)
#SetMTMode(5) #implicit done anyway on current version
MT("FluxSmoothST(7,7)",overlap=2)


One last hint, I can't use AVISource with SetMTMode (see post http://forum.doom9.org/showthread.php?p=1145725#post1145725)
the script without SetMTMode should work though.
Is the input clip RGB? otherwise ConvertToYUY2() makes no sense, keeping it YV12 would be faster.

[Graigddu]

Thanks for the help TSchniede,much appreciated :)

when i checked the gui avisynth script the source was actually from DirectShowSource so i ran this script and it worked

SetMTMode(5)
# 16:9 encoding
loadplugin("C:\Program Files\Avisynth 2.5\Plugins\MT.dll") # should be done outside of parallel region
DirectShowSource("C:\Test Movies\test\test.avi", false)
SetMTMode(2)
ConvertToYV12()
FadeIn(50)
Crop(0,16,720,544)
Lanczos4Resize(720,576,0.0,0.6)
#SetMTMode(5) #implicit done anyway on current version
MT("FluxSmoothST(12,12)",overlap=2)

it seems to work fine and i noticed that a 12 setting to temporal and spatial in fluxsmooth as file was blocky in parts produced a 1st pass encode of around 58 mins the normal without filtering being about 30 mins sadly unable to see what the 2nd pass would have been as had to leave for work.

[buletti]

Hi there,
can we expect MT support for AviSynth 2.5.8 RC1? As far as my experience goes AVS alphas and RCs live quite for some time. So even a pre-release update might be worth the effort...

[leeperry]

Hi there,
can we expect MT support for AviSynth 2.5.8 RC1? As far as my experience goes AVS alphas and RCs live quite for some time. So even a pre-release update might be worth the effort...

+1

this RC1 is mod2 for YV12 resizing instead of mod4.

very useful for SD content with LSF.

any chance seeing a MT patch please ?
it's too slow for real time use in ffdshow otherwise :(

[Warpman]

first: thank you for your great work tsp
filters that were supposed to be deadslow are now fast as hell :D

however i encountered one strange behavior:
if i drag & drop a script (even version() will do) into Megui it works 2-3 times and then stop working... if i do the same with unmodified avisynth it works always.

Maybe you have a clue whats going on...

:thanks:

[superuser]

+1

this RC1 is mod2 for YV12 resizing instead of mod4.

very useful for SD content with LSF.

any chance seeing a MT patch please ?
it's too slow for real time use in ffdshow otherwise :(

+2.

without MT, many of the filters with 2.5.8 are pretty slow. anxiously looking forward to release support latest avisynth release.

thnxs

[Dreassica]

Does Setmtmode work properly with allot of freezeframes, duplicate and deleteframes as wel as trims?
I ask this because I'm encoding with setmtmode, but i get memory access errors a few seconds into encode, with d2v as source.

[Zep]

Does Setmtmode work properly with allot of freezeframes, duplicate and deleteframes as wel as trims?
I ask this because I'm encoding with setmtmode, but i get memory access errors a few seconds into encode, with d2v as source.

some filters can not be used at all. Some need setmtmode=5.
Using mt eats up memory fast and can cause errors. Trims work fine but make sure you trim right after the source call in mode 5.

[halsboss]

Just purchased an Intel Q9450 Duo Quad4 and am looking forward to thrashing the living daylights out of it :)

New to MT and SetMTmode ... I gather

MT - splits frames vertically (or horizontally if specified) for each thread to process
SetMTmode - each thread takes and alternate frame to process (can't locate what the different modes do) edit yes I can, http://avisynth.org/mediawiki/MT_modes_explained
Its good to use OVERLAP=2 to 8 to ensure motion detection works in say MVtools and spatio-temporal filters like Convolution3D and fft3dfilter ?

and from TSP http://forum.doom9.org/showpost.php?p=1008355&postcount=522 that LimitedSharpenFaster works in MT..

However I am not sure if "resizing to a final size" all in one step works, like

MT("LimitedSharpenFaster(smode=4, dest_x=704, dest_y=576)",overlap=8,threads=4)
since I recall seeing somewhere that this was the way to resize in MT (substitute your resizer) -
MT("spline36resize(704, last.height())",splitvertical=false,overlap=8,threads=4)
MT("spline36resize(last.width(), 576)",splitvertical=true,overlap=8,threads=4)

Any suggestions ?

[halsboss]

Did a quick search but couldn't spot how MT's "threads=N" divides up the source for input into each thread and deals with Width/Height issues such as Width and Height both not evenly divided by N.

Does anyone know, eg does MT "smartly" add borders before and remove afterward to avoid the issue, or do I have to do those calcs and Addborders myself ?

Also, if N>2 what are the interpretations of MT's "splitvertical" parameter in those cases ? I guess it means something like "striped slices" into N vertical or horizontal segements.

For MT's N>2, and thinking about optimal motion detection - given generally horizontal panning for my sources - does "splitvertical=false" ensure nice wide horizontal striped slices going into each thread (I know it's the default) ?

On a quad core, is threads=3 optimal ? ie 3 for MT and one for HC; a 3800 frame test with Conv3D/LimitedSharpenFaster/spline36resize indicates a small elapsed time saving in N=4 over N=3 (only a few secs) whereas N=1 and N=2 and N=3 seem to have larger savings of 13 or 14 secs gaps.

[halsboss]

I'm not sure why the following dramatic changes occur with HC023 encode times when using a function to encapsulate most of the filters - any suggestions ?

MT_Nthreads=2 38.2 fps
MT_Nthreads=3 46.2 fps
MT_Nthreads=4 49.8 fps
from

SetMemoryMax(256)
WIDTH=704
HEIGHT=576
MT_Nthreads=2
MT_overlap=4
AviSource("G:\test\test-24Mb.avi", audio=false)
AssumeFPS(25)
global resizeWidth = 0
global resizeHeight = 0
global resizeBorderHalfHeight = 0
bh = wCalcResize(LAST, WIDTH, HEIGHT)
ConvertToYUY2(interlaced=FALSE)
MT("Convolution3D(0, 32, 128, 32, 128, 10, 0)",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=false)
MT("LimitedSharpenFaster(smode=4)",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=false)
MT("spline36resize(resizeWidth,last.height())",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=false)
MT("spline36resize(last.width(),resizeHeight)",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=true)
Addborders(0, resizeBorderHalfHeight, 0, resizeBorderHalfHeight)
Converttoyv12()
SetPlanarLegacyAlignment(True)


MT_Nthreads=2 48.5 fps
MT_Nthreads=3 56.8 fps
MT_Nthreads=4 59.7 fps
from

SetMemoryMax(256)
WIDTH=704
HEIGHT=576
MT_Nthreads=2
MT_overlap=4
AviSource("G:\test\test-24Mb.avi", audio=false)
AssumeFPS(25)
global resizeWidth = 0
global resizeHeight = 0
global resizeBorderHalfHeight = 0
bh = wCalcResize(LAST, WIDTH, HEIGHT)
ConvertToYUY2(interlaced=FALSE)
Function wMulti(clip "inpclp") {
zclp=inpclp.Convolution3D(0, 32, 128, 32, 128, 10, 0)
zclp=zclp.LimitedSharpenFaster(smode=4)
zclp=zclp.spline36resize(resizeWidth,zclp.height())
return zclp
}
MT("wMulti(LAST)",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=false)
MT("spline36resize(last.width(),resizeHeight)",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=true)
# unfortunately the height resize must be done separately since "splitvertical=true" is not the same as the rest
Addborders(0, resizeBorderHalfHeight, 0, resizeBorderHalfHeight)
Converttoyv12()
SetPlanarLegacyAlignment(True)

PS single-thread equivalent = 32.7 fps

[foxyshadis]

MT("wMulti(LAST)",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=false)
is equivalent to
MT("Convolution3D(0, 32, 128, 32, 128, 10, 0)
LimitedSharpenFaster(smode=4)
spline36resize(resizeWidth,last.height())",threads=MT_Nthreads,overlap=MT_overlap,splitvertical=false)
not three separate MTs. The perf drop is from all the back and forth copies being made when you split up the filters.

Did a quick search but couldn't spot how MT's "threads=N" divides up the source for input into each thread and deals with Width/Height issues such as Width and Height both not evenly divided by N.
It rounds to the nearest mod 2. One being a couple of lines shorter than another isn't really a noticeable perf difference.
Does anyone know, eg does MT "smartly" add borders before and remove afterward to avoid the issue, or do I have to do those calcs and Addborders myself ?
This is more like what overlap=X does. You're right about it being only needed for spatial filters and motion-compensated filters, small amounts are enough for most filters but large overlap is needed for mvtools.
On a quad core, is threads=3 optimal ?
Optimal is pretty much whatever works best in your workflow - for HC with your combination of filters, 3 is the way to go; with much heavier filtering or another encoder 4 or 2 can be much better. With multiple instances of HC (ie, HCEnc^N), less may be optimal.

[Revgen]

I always use SetMTMode when doing deinterlacing. MT is good for almost everything else as long as it's not deinterlacing or motion compensation.

[halsboss]

Yes thanks. took me a while to guess with various old filters I use.

Just for interest (pick holes, if you will), here a couple of scripts I use and comment/uncomment the filter combination on a per-case basis.
1. make DVD from HDTV for timeshifted viewing

# Resize HDTV 1080i to 576i - run DGIndex over it 1st to yield the DV file
SetMTmode(mode=5,threads=3) # start with mode=5 forAVIsource http://forum.doom9.org/showthread.php?p=1067216#post1067216
SetMemoryMax(768) # why not, 4Gb is installed
# be lazy and just load the filters used by a range of scripts
LoadPlugin("C:\SOFTWARE\DGindex\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\TDeint.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\EEDI2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\Yadifmod.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\NNEDI.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\DePan.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\AGC.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\Cnr2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\dctfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\fft3dfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\degrainmedian.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\Convolution3d.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\despot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\WarpSharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\aWarpSharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\mvtools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\Unfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\AddgrainC.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\hqdn3d.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\RemoveGrainSSE2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\RepairSSE2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\RemoveDirtSSE2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins2.0\LoadPluginEx.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins2.0\DustV5.dll")
Import("C:\Program Files\AviSynth 2.5\LimitedSharpenFaster.avsi")
Import("C:\Program Files\AviSynth 2.5\dumbresize.avsi")
Import("C:\Program Files\AviSynth 2.5\Deblock_QED_MT2.avs")

MPEG2Source("G:\HDTV\1\test1080iHD.d2v",cpu=6) # cpu=6: DEBLOCK_Y_H, DEBLOCK_Y_V, DEBLOCK_C_H, DEBLOCK_C_V, DERING_Y, DERING_C
AssumeFPS(25)
AssumeTFF()

FRAMERATE=25
WIDTH=704
HEIGHT=576
LastW=LAST.width()
LastH=LAST.height()

# This link shows how to DEBLOCK AN INTERLACED SOURCE
#http://forum.doom9.org/showthread.php?p=1121574#post1121574
# Deblock before resize or it doesn't deblock
#SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
#SeparateFields().PointResize(LastW,LastH).Addborders(0,0,0,8)
#Deblock_QED_MT2().Cropbottom(8).AssumeFrameBased() #default quant1=20
#Deblock_QED_MT2(quant1=25).Cropbottom(8).AssumeFrameBased()
#Deblock_QED_MT2(quant1=30).Cropbottom(8).AssumeFrameBased()
#Deblock_QED_MT2(quant1=35).Cropbottom(8).AssumeFrameBased()
#Deblock_QED_MT2(quant1=40).Cropbottom(8).AssumeFrameBased()
#SeparateFields().SelectEvery(4,0,3).Weave()

# This link shows how to resize interlaced
#http://forum.doom9.org/showthread.php?p=1115201#post1115201
SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
bicubicresize(704,LAST.height())
tdeint(mode=1,order=1) # mode=0=same rate output mode=1=double rate output (bobbing) order=0=BFF order=1=TFF
bicubicresize(last.width(),576)
separatefields()
selectevery(4,0,3)
weave()
## OR but not both
##SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
##bicubicresize(704,LAST.height())
## see http://forum.doom9.org/showthread.php?p=1114448#post1114448
## tcritical plugins http://bengal.missouri.edu/~kes25c/
## for yadifmod tff Order=1 ; for nnedi field=-2 means assumeTFF is important
##yadifmod(mode=1, order=1, edeint=nnedi(field=-2)) #bob deinterlace it without stairstepping to 50
##bicubicresize(last.width(),576)
##separatefields()
##selectevery(4,0,3)
##weave()

Converttoyv12(interlaced=true)
SetPlanarLegacyAlignment(True)
# Distributor() should only be used when loading the script into applications which talk to avisynth directly
# e.g. HC and MeGUI. If the program opens the .avs file as an avi file (like virtualdub/mod, x264, xvid_encraw etc.)
# then you don't need (and shouldn't have) the distributor call at the end.
# http://forum.doom9.org/showthread.php?p=1136518#post1136518
##### "HC uses custom avisynth routines, you have to add Distributor() to the end of the script."
Distributor() # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1063622#post1063622
# http://forum.doom9.org/showthread.php?p=1067589#post1067589
# ChangeFPS(Last, Last, True) makes fast scripts faster and slow scripts slower !!
#ChangeFPS(Last, Last, True) # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1064118#post1064118

2. for the simpler 16:9 clips

SetMTmode(mode=5,threads=3) # start with mode=5 forAVIsource http://forum.doom9.org/showthread.php?p=1067216#post1067216
SetMemoryMax(768)
LoadPlugin("C:\SOFTWARE\DGindex\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\DePan.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\AGC.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\Cnr2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\dctfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\fft3dfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\degrainmedian.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\Convolution3d.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\despot.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\WarpSharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\aWarpSharp.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\mvtools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\Unfilter.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\AddgrainC.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\hqdn3d.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\mt_masktools.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\RemoveGrainSSE2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\RepairSSE2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-noautoload\RemoveDirtSSE2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins2.0\LoadPluginEx.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins2.0\DustV5.dll")
Import("C:\Program Files\AviSynth 2.5\LimitedSharpenFaster.avsi")
Import("C:\Program Files\AviSynth 2.5\dumbresize.avsi")
Import("C:\Program Files\AviSynth 2.5\Deblock_QED_MT2.avs")
Import("C:\Program Files\AviSynth 2.5\DeHalo_alpha.avsi")

FRAMERATE=25
WIDTH=704
HEIGHT=576
#Assume 16:9 input and output, no adding borders necessary at the end
AviSource("G:\HDTV\1\testProgressiveclip.avi", audio=false)
AssumeFPS(25)

global resizeWidth = 704
global resizeHeight = 576

# If it's got Blocks, try to remove them !!
#SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
#ConvertToYV12(interlaced=FALSE) # for Deblock_QED_MT2 and DeHalo_alpha
#Deblock_QED_MT2() #default quant1=20
#Deblock_QED_MT2(quant1=25)
#Deblock_QED_MT2(quant1=30)
#Deblock_QED_MT2(quant1=35)
#Deblock_QED_MT2(quant1=40)

# If it's got ghosts, try to remove them !!
#SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
#ConvertToYV12(interlaced=FALSE) # for Deblock_QED_MT2 and DeHalo_alpha
#DeHalo_alpha(rx=5,ry=5)

#SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
#ConvertToYUY2(interlaced=FALSE) # Ensure YUY2 for the rest including Convolution3D

#SetMTmode(mode=5,threads=3) # # mode=5 for safety including using MT
# PixieDust doesn't like MT ? Use SetMTmode(5)
#Pixiedust(output="YUY2")
#Pixiedust(limit=5,output="YUY2")
# PixieDust doesn't like MT ? Use SetMTmode(5)

#SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
# HDRAGC *can't* be used inside MT or AGC will be calculated differently in each slice
#HDRAGC(coef_gain=0.1, min_gain=0.1, max_gain=0.5, coef_sat=1.0, corrector=0.8, reducer=2.0, black_clip=1.0)
#HDRAGC()
#
SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
ConvertToYUY2(interlaced=FALSE) # Ensure YUY2 for the rest including Convolution3D

# Hmm, MVanalyses etc should go in mode=2 rather than in MT ... lots of work was done on that by TAP, Fizick
#eg per http://forum.doom9.org/showthread.php?p=1080770#post1080770
#SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
#function DegrainC( clip c, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" ) {
#global idx_1 = idx_1 + 1
#global idx_2 = idx_2 + 1
#blk = default( blk, 16 )
#ol = default( ol, 0 )
#sh = default( sh, 2 )
#sad = default( sad, 200 )
#pl = default( pl, 4 )
#div = default( div, 0 )
#limuv = default( limuv, 255 )
#vbw1=MVAnalyse(c,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#vfw1=MVAnalyse(c,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#vbw2=MVAnalyse(c,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#vfw2=MVAnalyse(c,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl)
#defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_1,plane=pl),c,limy,limuv) : nolimit
#}
#function DegrainFFTC( clip c, clip cleaned, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
#{
#global idx_1 = idx_1 + 1
#global idx_2 = idx_2 + 1
#blk = default( blk, 16 )
#ol = default( ol, 0 )
#sh = default( sh, 2 )
#sad = default( sad, 200 )
#pl = default( pl, 4 )
#div = default( div, 0 )
#limuv = default( limuv, 255 )
#vbw1=MVAnalyse(cleaned,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#vfw1=MVAnalyse(cleaned,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#vbw2=MVAnalyse(cleaned,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#vfw2=MVAnalyse(cleaned,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_1,sharp=sh,overlap=ol,divide=div)
#nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl)
#defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_2,plane=pl),c,limy,limuv) : nolimit
#}

SetMTmode(mode=5,threads=3) # # mode=5 for safety including using MT
Function Do_Stuff_In_MT(clip "inpclp") {
zMTclp=inpclp
# If black and white movie, use these
#zMTclp=zMTclp.Limiter()
#zMTclp=zMTclp.Greyscale()
# If GAMMA needs adjusting, do so here
#zMTclp=zMTclp.Levels(0, 1.15, 255, 0, 255, coring=false)
########
#zMTclp=zMTclp.Convolution3D(0, 3, 4, 3, 4, 2.8, 0)
zMTclp=zMTclp.Convolution3D(0, 6, 10, 6, 8, 2.8, 0)
#zMTclp=zMTclp.Convolution3D(0, 32, 128, 16, 64, 10, 0)
#zMTclp=zMTclp.Convolution3D(0, 32, 128, 32, 128, 10, 0)
#zMTclp=zMTclp.Convolution3D(1, 32, 128, 32, 128, 10, 0) #blur more for very very bad
#zMTclp=zMTclp.FFT3DFilter(sigma=3, sharpen=1.0, degrid=1.0, interlaced=false)
#zMTclp=zMTclp.FFT3DFilter(sigma=4, plane=0, sharpen=1.0, degrid=1.0, interlaced=FALSE) # luma first
#zMTclp=zMTclp.FFT3DFilter(sigma=8, plane=3, sharpen=1.0, degrid=1.0, interlaced=FALSE) # chromas next with more filtering
# http://forum.doom9.org/showthread.php?p=1084125#post1084125
# DegrainMedian, it gets a huge boost at least when wrapped inside an MT call
# http://forum.doom9.org/showthread.php?p=1109515#post1109515
# DegrainMedian benefits much more from using MT than using SetMTMode
#zMTclp=zMTclp.DeGrainMedian(limitY=5,limitUV=7,mode=0,interlaced=FALSE)
# mode=1 Lesser but strong grain removal, on top
#zMTclp=zMTclp.DeGrainMedian(limitY=2,limitUV=3,mode=1,interlaced=false)
#zMTclp=zMTclp.DeGrainMedian(limitY=2,limitUV=3,mode=1,interlaced=false)
#zMTclp=zMTclp.DeSpot(interlaced=false)
#zMTclp=zMTclp.DeSpot(median=false, interlaced=false, seg=2, show=0) # SAFE
#zMTclp=zMTclp.DeSpot(median=false, interlaced=false, seg=0, show=0) # seg=0 means STRONG
#zMTclp=zMTclp.DeSpot(p1=35, p2=14, mthres=25, interlaced=false) # UP the limits before it's considered noise
zMTclp=zMTclp.LimitedSharpenFaster(smode=4) # ,strength=110
# 1ST HALF OF THE RESIZE IN THE mt FUNCTION
zMTclp=zMTclp.spline36resize(resizeWidth,zMTclp.height()) # 1st half of resize (2nd half outside this MT function !!
RETURN zMTclp
} # end of function Do_Stuff_In_MT
MT("Do_Stuff_In_MT(LAST)",threads=3,overlap=4,splitvertical=false)
# iF DID THE 1ST HALF OF THE RESIZE IN THE mt FUNCTION, DON'T FORGET TO DO THE LAST HALF RIGHT NOW
MT("spline36resize(last.width(),resizeHeight)",threads=3,overlap=4,splitvertical=true)

SetMTmode(mode=2,threads=3) # mode=2 for temporal multi-threading (interleaved frames)
Converttoyv12()
SetPlanarLegacyAlignment(True)
# Distributor() should only be used when loading the script into applications which talk to avisynth directly
# e.g. HC and MeGUI. If the program opens the .avs file as an avi file (like virtualdub/mod, x264, xvid_encraw etc.)
# then you don't need (and shouldn't have) the distributor call at the end.
# http://forum.doom9.org/showthread.php?p=1136518#post1136518
##### "HC uses custom avisynth routines, you have to add Distributor() to the end of the script."
Distributor() # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1063622#post1063622
# http://forum.doom9.org/showthread.php?p=1067589#post1067589
# ChangeFPS(Last, Last, True) makes fast scripts faster and slow scripts slower !!
#ChangeFPS(Last, Last, True) # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1064118#post1064118

[halsboss]

On a quad-core do you think it's safe to do MTi(MT("...",threads=2,overlap=4)) for an interlaced clip with simple function containing FFT3d or Convolution3d or maybe even an MVdegrain ? Assuming there'd be a performance benefit from MT within the MTi. Don't suppose HDRAGC would be a good idea though :)

[Ranguvar]

DGAVCDecode seems borked with SetMTMode under every setting... can anyone confirm this?

[pitch.fr]

is it me or every try to run ConvertToRGB32 multithreaded fails ?

it doesn't change anything if I use MT("",4), and it actually crashes on some files.

I also tried SetMode(1,4) but that doesn't change the CPU load either...

I've also tried to do ConvertToYUY2, then ConvertToRGB32 but no luck either.

this ConvertToRGB32 is so unoptimized, it's pretty depressing :(

[IanB]

... this ConvertToRGB32 is so unoptimized, ...You seem to be having some difficulty here.

On my 3GHz P4 with 8K L1 & 512K L2 cache (a fairly low spec machine these days) I get :-Frame 720x576 :-
1000fps -- YV12.Crop(2,0,0,0,True) i.e A full frame Blit
650fps -- YV12.ConvertToYUY2()
340fps -- YUY2.ConvertToRGB32()
230fps -- YV12.ConvertToRGB32()

Frame 1920x1080 :-
281fps -- YV12.Crop(2,0,0,0,True)
140fps -- YV12.ConvertToYUY2()
72fps -- YUY2.ConvertToRGB32()
47fps -- YV12.ConvertToRGB32()None of these are what I regard as unexpected, unoptimised or slow. The HD size frame is 4 times my L2 cache size and is still a credible speed.

For multi threading to work there has to be things to do in parallel. Given MT() adds a synchronous frame blit to the end of the process, there will be very little gain in using it only on ConvertToRGB alone. You need to process a reasonable amount per stripe to amortise the final MT() blit cost, this has been previously discussed here, i.e. T=1+(T/N), T>=N. Also the source filters have a penalty for out of order decoding, some quite high, this has also been previously discussed here, i.e. abuse ChangeFPS().

I may have missed it, but I do not remember you ever having posted your complete script for this tale of woe, all the focus seems to have been on the output ConvertToRGB(). Perhaps it is about time you stopped fumbling around in the dark.

[pitch.fr]

lol OK sorry IanB :D

well I do a lot of post-processing in ffdshow with HD h264 files, and the built-in RGB32HQ ffdshow routines are very fast.

but as I soon as I go the ConvertToRGB32 route with the AVS filter, it sucks up all my precious CPU cycles on my 3.8Ghz E7200 and I end up with bad jitter in Haali's Renderer :(

here's my ffdshow script :

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",4)
MT("""ULevels(preset="tv2pc")""",4)
SetMTMode(1,4)
ConvertToYUY2()
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=4,opt=1)


maybe this would be more efficient to run ConvertToRGB32 in MT ?
http://forum.doom9.org/showthread.php?t=139629

thanks for your help,

[IanB]

For a start, don't MT() every filter, as I have previously said, the reassemble blit is synchronous, in your script you take 3 steps forward and 4 back. Just MT the whole block (assuming all the filters are thread safe) and only have a single reassemble blit overhead.MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
""", 4)
Also ULevels is not a plugin it is quite a complicated script function. It can call many other functions. You should investigate what is actually being run.

Not 100% sure but I think the built-in RGB32HQ ffdshow routines have been taken from Avisynth which took them from HuffYUV.

And ffdshow has some unique constraints in how it integrates Avisynth scripts, for one there is no ...Source() statement, ffdshow provides an implicit source. Second the AVIFile interface is not used. The Distributor() functionality, used by SetMTMode(), is built into the AVIFIle interface. To use SetMTMode() you will need to manually implement the Distributor() functionality.

AVS_Pipeline may be an alternative, you should test it.

You cannot just randomly throw threading commands into a script and expect magic. Whatever you do it needs to be considered.

[pitch.fr]

ok makes sense, thank you for the explanations Ian

well it's actually worse on 1.78 720p files if we simply MT4 the whole stuff, instead of doing it for each of them.

I've got no idea how to run the AVS_Pipeline with the full script..

I'm gonna grab a Q6600, o/c it to 3.8GHz and hopefully with 4 cores that will run smoothly :D

[IanB]

You cannot just randomly throw threading commands into a script and expect magic. Whatever you do it needs to be considered.I suggest you start by taking this out of the ffdshow environment and run the scripts up under VirtualDub so you can test and compare various solutions.

Start with this script :-SetMemoryMax(512) # Adjust value as required

ColorBars(1280, 720, pixel_type="YV12") # Size and pixel type as appropriate

# Insert Code under test here
Crop(2, 0, 0, 0, True) # Force a simple Blit


# Throw away most of the image we are not testing VD's i/o
Crop(0, 0, 16, 16)Okay load it into VirtualDub and Run a video analysis pass to get the FPS, it should be about 1000fps or more.

Replace the Crop(2, ... with each statement your are wanting to use and get the FPS for each, change the ColorBars pixel_type to suit the statement under test, i.e. RGB32 for ddcc.

Time the statements in combination, add some MT to parts of the script, ...

Tabulate your findings so we can help you formulate a plan of attack.

Then test it, modify it, test it again.

[halsboss]

I'm gonna grab a Q6600, o/c it to 3.8GHz and hopefully with 4 cores that will run smoothly :D
Ive just bought a Q6600 and they run at 2.4GHZ. Standard air cooler, old poor-ventilation case. (Also a Q9450 @ 2.66GHz). ... pls let me know what you do and how you go :) as I want to use MT at it's fullest with processor speed too.

[halsboss]

Just a thought, I sometimes run 2 or 3 or 4 threads depending on what I'm up to at the time, and I run "standard scripts" which take a couple parameters including #threads.

I gather MT chops up the frame size depending on number of threads, which is nice, however a couple of my scripts seem to throw up with filters that require vertical & horizontal sizes as multiples of 4 or 8 or 16 depending...

Do you think MT and maybe MTi could include an extra parameter such as "MULTIPLE=", like "multiple=16" which (when specified) then means treat the "overlap=" as a minimum value and increases "overlap" until the cut frame size (both vert and horiz if possible) is a multiple of "MULTIPLE" ... and then backs-down appropriately when putting the frame back together. That'd then make MT and MTi more independent of the number of threads in terms of filters that break.

If I'm off with the pixies, or there's a better way to do it, please let me know.

[saint-francis]

DGAVCDecode seems borked with SetMTMode under every setting... can anyone confirm this?

SetMTmode 4 seems to work OK for me but anything before that doesn't.

[Boulder]

IIRC, you are not supposed to use anything but SetMTMode(5) before loading the source anyway.

[pitch.fr]

I suggest you start by taking this out of the ffdshow environment and run the scripts up under VirtualDub so you can test and compare various solutions.
oh great! this is far more accurate than timecodec....which is mostly measuring the i/o as you said.

SetMTMode() doesn't seem to work on my system....not sure why ?

doing ConvertToRGB32 directly is faster than ConvertYUY2/ConvertToRGB32 for some reason...also jitter-wise in Haali's Renderer

and you were right, it's faster to MT the whole stuff, but the jitter seemed worse in HR, maybe it's flooding the CPU ?

do you know how I could use the pipeline technique on LSF+Ulevels ?
I'm in contact with Haali, he said he would offer gamut conversion within his renderer so soon or later I won't need to bother with DDCC and ConvertToRGB32 :)

ok here we go with the results on a 3.8Ghz E7200(dual core), your original script was 2900fps

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 8)
=100.4

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 6)
=99.48

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 4)
=95

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",4)
MT("""ULevels(preset="tv2pc")""",4)
=97


MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 4)
=94

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",2)
MT("""ULevels(preset="tv2pc")""",2)
=94

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",7)
MT("""ULevels(preset="tv2pc")""",7)
=96

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 2)
=89

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",6)
MT("""ULevels(preset="tv2pc")""",6)
=98.55

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",8)
MT("""ULevels(preset="tv2pc")""",8)
=98 solid

SetMTMode(1,2)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
=77

SetMTMode(1,4)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
=77

LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
=77

SetMTMode(2,4)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
=77

SetMTMode(3,4)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
=77


--------


MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",6)
MT("""ULevels(preset="tv2pc")""",6)
MT("ConvertToYUY2()",6)
MT("""ConvertToRGB32(matrix="PC.709")""",6)
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=6,opt=1)
=44.33

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",6)
MT("""ULevels(preset="tv2pc")""",6)
MT("""ConvertToRGB32(matrix="PC.709")""",6)
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=6,opt=1)
=47.20

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",6)
MT("""ULevels(preset="tv2pc")""",6)
MT("""ConvertToRGB32(matrix="PC.709")""",4)
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=6,opt=1)
=46.50

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",6)
MT("""ULevels(preset="tv2pc")""",6)
MT("""ConvertToRGB32(matrix="PC.709")""",4)
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=4,opt=1)
=46.50

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",6)
MT("""ULevels(preset="tv2pc")""",6)
MT("""ConvertToRGB32(matrix="PC.709")""",8)
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=8,opt=1)
=46.80

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",8)
MT("""ULevels(preset="tv2pc")""",8)
MT("""ConvertToRGB32(matrix="PC.709")""",8)
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=8,opt=1)
=47.10

----

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
""", 2)
=46.20

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
""", 4)
=48.50

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
""", 6)
=50solid

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
""", 8)
=49.70

[Leak]

SetMTMode() doesn't seem to work on my system....not sure why ?
If this is in ffdshow - you probably need to put Distributor() at the very end of the script, as that is what's done automatically when you open an AVS file, but not if you use AviSynth directly.

np: Mindless Drug Hoover - The Reefer Song (Grass Garden Of Child's Mix) (Auntie Aubrey's Excursions Beyond The Call Of Duty Part 2 (Disc 2))

[pitch.fr]

well I already noticed that SetMTMode() alone didn't work in ffdshow, thanks to timecodec

but right now I'm still benchmarking in VirtualDub 1.83, and when I input :

SetMTMode(2,4)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
Distributor()

VDub crashes instantly, I've tried all the MT modes....

I'm not too lucky with this SetMTMode() stuff :(

[IanB]

@pitch.fr,

Interesting random spread of results. Fundamentally the YV12 part of the process looks to be about 77-100fps, and adding the RGB32 part slows this to 45-50fps. But we are still shooting in the dark.Replace the Crop(2, ... with each statement your are wanting to use and get the FPS for each, change the ColorBars pixel_type to suit the statement under test, i.e. RGB32 for ddcc.But we still do need the raw single core FPS values for each individual line of the script, not just the aggregate of the MT of all the filters. Please make sure you test each of YV12->RGB32, YV12->YUY2 and YUY2->RGB32 individually. And testing the 1280x720 single blit [Crop(2, ... True) ]times with YV12, YUY2 and RGB32 data types could be instructive.
SetMTMode() doesn't seem to work on my system....not sure why ?As Leak said it is a Distributor() issue and, as said in the documentation, you also need a SetMTMode(...) command at the start of your script. This is why I suggested we move to VDub for the analysis phase. But we get ahead of ourselves.
doing ConvertToRGB32 directly is faster than ConvertYUY2/ConvertToRGB32 for some reason...also jitter-wise in Haali's RendererWell using MT you do get an extra blit frame copy overhead and you may also loose some L2 cache locality. Splitting them has to be an informed decision. Wait for the analysis results.
and you were right, it's faster to MT the whole stuff, but the jitter seemed worse in HR, maybe it's flooding the CPU ?Yes there are many issues with multi threading, Haali needs some CPU cycles as well. Wait for the analysis results.
do you know how I could use the pipeline technique on LSF+Ulevels ?For best performance the pipeline needs to be between every object in the filter graph, this makes it's use harder with pre-canned complicated scripts like ULevels, i.e. you need to manually pull the script apart. There may also be issues to do with being able to prefetch frames under the ffdshow environment. But we get ahead of ourselves again.

Crawl before you Walk before you Run!

:Edit: You must only have 1 Distributor instance in a script, with VDub you get 1 automatically, with FFDShow you need to add it.

[pitch.fr]

You must only have 1 Distributor instance in a script, with VDub you get 1 automatically, with FFDShow you need to add it.
well how come it didn't make a diff in VDub then ?

whatever arguments, it was still 77".....which is the same as without any MT...


OK here we go again :cool:

YV12->YV12
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
=224

YUY2->YUY2
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
=128

YV12->YV12
ULevels(preset="tv2pc")
=112

YV12->YV12
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
=77

--

YV12->YUY2
ConvertToYUY2()
=980

YV12->RGB32
ConvertToYUY2()
ConvertToRGB32(matrix="PC.709")
=280

YV12->RGB32
ConvertToRGB32(matrix="PC.709")
=286

RGB32->RGB32
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
=72

RGB32->RGB32
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=2,opt=1)
=138

RGB32->RGB32
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=4,opt=1)
=138

RGB32->RGB32
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=6,opt=1)
=140

RGB32->RGB32
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=8,opt=1)
=139

opt - Sets what cpu optimizations to use. Possible settings:
0 - C routines
1 - SSE3 routines

RGB32->RGB32
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=0)
=24.75

RGB32->RGB32
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=6,opt=0)
=49

YV12->RGB32
ConvertToYUY2()
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
=55

YV12->RGB32
ConvertToYUY2()
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=2,opt=1)
=93

YV12->RGB32
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
=56solid

YUY12->RGB32
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
=59

[IanB]

@pitch.fr,

Okay summarising and converting to elapsed milliseconds :-Filter FPS msec
====== === ====
Blit 2900 0.34

LSF yv12 224 4.46
LSF yuy2 128 7.81

ULevels 112 8.93

YV12->YUY2 980 1.02
YV12->RGB32 283 3.53
YUY2->RGB32 400 2.51 (estimated)

DDCC,t=1 72 13.89
DDCC,t=2+ 139 7.19

DDCC,C++,t=1 25 40.40
DDCC,C++,t=6 49 20.41

LSF+Ulevels 77 13.00 8.93+4.46=13.39ms -> 75fps

YV12->DDCC,t=1 56 18.20 3.53+13.89=17.42->57
YV12->DDCC,t=2 93 10.75 3.53+7.19=10.72->93We now have estimates of how much time each component of the script contributes. And we can compare estimates with reality, and see the results are close.

In practice chaining filters can give a slightly faster or slower result due to good or bad L2 cache utilisation between the 2 filters, we see about +3% with LSF+ULevels.

So Assuming 1 source blit, 1 output blit and ignoring L2 cache speed ups, the total time with a single core should be about 0.34+4.46+8.93+3.53+13.89+0.34=31.49ms -> 32FPS. I guess you should test this case as a base line ;)

The slow points seem to be DDCC at 14ms and ULevels at 9ms, a total of 23ms of a whole 32ms per frame.

I would suggest this script as a starting pointMT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
""", 6)
ddcc(chr_i=3, gam_i=5, ofile="C:\display.txt", threads=6, opt=1)WIth 2 cpu cores fully utilized and using the internal DDCC mt, the best we could expect 0.34+((4.46+8.93+3.53)/2+0.34)+7.19+0.34=16.67ms -> 60fps In reality there are synchronisation overheads so the final result can be anywhere from 32fps to 60fps, you currently seem to be at about the 45 to 50fps mark.

I would recomend having a good look at the ULevels.avsi script to see where all the time is going in it.

I am a little surprised at how slow DDCC is, given it is SSE3, but I haven't looked at the code, and it does go from 24 with C++ to 55 with SSE3. so I guess it is working very hard. Perhaps you could plead with Tritical to implement a direct YV12 version, still doing gamma thingy on the U and V planes might be a bit trickey.

And finally you might try experimenting starting with this :-SetMTMode(5, 3)
ChangeFPS(Last, Last, True) # Force linear access
SetMTMode(2)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
Distributor() # cheat and start MT here
SetMTMode(5)
ddcc(chr_i=3, gam_i=5, ofile="C:\display.txt", threads=6, opt=1)

[pitch.fr]

Yes, I'm also surprised how slow DDCC is.

so any idea how to get SetMTMode() working in Vdub ?
or should I give up on this mode altogether ?

ok so in YV12>RGB32 :
MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
""", 6)
ddcc(chr_i=3, gam_i=5, ofile="C:\display.txt", threads=6, opt=1)
=48.30

the second script, VDub crashes instantly...so let me use timecodec with ffdshow on a 2.35 1080p h264 file :)

my 50fps script :
MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=1,opt=1)
""", 6)
= crashes instantly in ffdshow

with my old script :
MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",4)
MT("""ULevels(preset="tv2pc")""",4)
MT("""ConvertToRGB32(matrix="PC.709")""",4)
ddcc(chr_i=3,gam_i=5,ofile="C:\display.txt",threads=4,opt=1)
User: 2s, kernel: 0s, total: 2s, real: 7s, fps: 104.4, dfps: 38.7
User: 2s, kernel: 0s, total: 2s, real: 7s, fps: 111.5, dfps: 39.1
User: 2s, kernel: 0s, total: 2s, real: 7s, fps: 131.0, dfps: 38.9
User: 2s, kernel: 0s, total: 2s, real: 7s, fps: 112.8, dfps: 39.4
User: 2s, kernel: 0s, total: 2s, real: 7s, fps: 116.9, dfps: 39.0

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
""", 6)
ddcc(chr_i=3, gam_i=5, ofile="C:\display.txt", threads=6, opt=1)
= crashes instantly in ffdshow

SetMTMode(5, 3)
ChangeFPS(Last, Last, True) # Force linear access
SetMTMode(2)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
ConvertToRGB32(matrix="PC.709")
Distributor() # cheat and start MT here
SetMTMode(5)
ddcc(chr_i=3, gam_i=5, ofile="C:\display.txt", threads=6, opt=1)
= crashes instantly in ffdshow

anyhow Haali will add the gamut conversion through a PS script in his renderer, so I'd like to get LSF+Ulevels as optimized as possible :)

MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 8)
User: 1s, kernel: 0s, total: 1s, real: 5s, fps: 154.9, dfps: 51.8
User: 2s, kernel: 0s, total: 2s, real: 5s, fps: 130.1, dfps: 52.3
User: 2s, kernel: 0s, total: 2s, real: 5s, fps: 130.1, dfps: 52.2
User: 1s, kernel: 0s, total: 2s, real: 5s, fps: 151.3, dfps: 52.1
User: 2s, kernel: 0s, total: 2s, real: 5s, fps: 146.8, dfps: 51.9

SetMTMode(1, 8)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
Distributor()
= crashes instantly in ffdshow

SetMTMode(5, 3)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
Distributor()
= crashes instantly in ffdshow

I don't think I'll ever see that SetMTMode() working in either VDub or ffdshow...

[IanB]

Both LimitedSharpenFaster and ULevels are very complicated script functions. Normally this abstraction is a good thing, it encapsulate a lot of thought and functionality into a simple to use function call. However there is a downside and you are experiencing it here, i.e. the surrendering of control of your environment to the script authors, Didée and LaTo.

You are going to have to bite the bullet and understand these scripts and carefully rewrap their component parts in a thread safe and tolerant manner. Obviously some components are not thread safe, you need to identify these and take remedial steps to protect these components so that you can maximise the threaded performance of the remaining thread safe components.

Failing that level of dedication you are going to be stuck with this simplistic solutionMT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 6)Your results of nearly 50% faster are actually pretty good. Yes it is very difficult to utilise all the power of multiple cpu cores, at nearly 1000 posts in this thread, it says something.

Ask yourself these questions :- Is ULevels(preset="tv2pc") worth the 9ms per frame processing time and the obvious angst it is causing? What crucial benefit does it give over a more pedestrian Levels(16, 1.0, 235, 0, 255, coring=false) or other similar filter or not doing it at all?

[pitch.fr]

so are you saying that SetMTMode() isn't working in either VDub or ffdshow because of how the scripts were coded in the first place ?

kinda strange that it works fine with MT() then ?!

but yeah, they use completely different ways of achieving MT, so I guess that would make sense.

anyhow I'm getting a Q6600 G0 CPU, that should o/c nicely to 3.6GHz or more.

a friend of mine who's also using AVS scripts in ffdshow said he can run LSF with ss1.3 on 1.78 1080p high bitrate h264 files(on 12 threads w/ a Q9450) :eek:

I can only use it on 720p and 2.35 1080p with SS1.0 on my 3.8Ghz E7200

well these 2 filters are pretty vital to me.

I use Haali's Renderer in 24Hz with Reclock set to resample 23.976@24fps, so I get judder-free HD movies at their original speed.

LSF doesn't make any EE and sharpens the motion blur, this thing is too awesome for words(using the spline36 version) in combination with Haali's Renderer :)

then I've set some slight unsharp masking in ffdshow(I couldn't find the equivalent in AVS script ?! it seems to be based on neuron's VDUB plugin, but it's got a lot more settings.....) to get some very slight EE(it increases the 3D depth), and it's rendered in RGB32HQ in ffdshow(very optimized conversion).

the Ulevels() is really nice because it increases the gamma curve in the shadows, which increases the contrast......and considering I'm doing SMPTE-C gamut conversion, and that this gamut is known to have "orangey" reds, Ulevels() slightly darkens the reds.

Ulevels()/ffdshow (gamut conversion applied in both cases) :

http://thumbnails9.imagebam.com/1015/c5507110145747.gif (http://www.imagebam.com/image/c5507110145747)http://thumbnails9.imagebam.com/1015/c4554710145749.gif (http://www.imagebam.com/image/c4554710145749)http://thumbnails9.imagebam.com/1015/1965d810145751.gif (http://www.imagebam.com/image/1965d810145751)
http://thumbnails8.imagebam.com/991/b61b119905830.gif (http://www.imagebam.com/image/b61b119905830)http://thumbnails8.imagebam.com/991/9b2ef49905832.gif (http://www.imagebam.com/image/9b2ef49905832)http://thumbnails8.imagebam.com/991/3b5c889905834.gif (http://www.imagebam.com/image/3b5c889905834)

http://thumbnails9.imagebam.com/1015/0954ae10145746.gif (http://www.imagebam.com/image/0954ae10145746)http://thumbnails9.imagebam.com/1015/c31f9c10145748.gif (http://www.imagebam.com/image/c31f9c10145748)http://thumbnails9.imagebam.com/1015/68644910145750.gif (http://www.imagebam.com/image/68644910145750)
http://thumbnails8.imagebam.com/991/f1ffe19905831.gif (http://www.imagebam.com/image/f1ffe19905831)http://thumbnails8.imagebam.com/991/0b3b449905833.gif (http://www.imagebam.com/image/0b3b449905833)http://thumbnails8.imagebam.com/991/fa99c29905835.gif (http://www.imagebam.com/image/fa99c29905835)

and my gamut conversion, from SMPTE-C on my HC3100 pj :

http://pix.nofrag.com/4/1/0/6146664a9e7e23f8f006fc5e3875dtt.jpg (http://pix.nofrag.com/4/1/0/6146664a9e7e23f8f006fc5e3875d.html)

the black triangle is SMPTE-C, the white one is the corrected gamut and the third one is the original one.

[Zep]

anyhow I'm getting a Q6600 G0 CPU, that should o/c nicely to 3.6GHz or more.



Only if you get one with a good vid. Most batches from Intel for months now have had high vid and thus lousy OC. Expect 3.4Ghz max with a vid of 1.325 (unless you water cool) Your best bet is to hop on e-bay and the like and buy one that has a vid of 1.255 or less. (the lower the better)

[pitch.fr]

I'm getting it from a friend, it's got a 1.25V VID :)

[Zep]

I'm getting it from a friend, it's got a 1.25V VID :)

that will get you 3.5GHz running pretty hot. If 3.6GHz then running VERY HOT. voltage increase to get 100Mhz ratio is not linear and at 3.4GHz it takes about .05 volts to get 100Mhz. The rated max is 1.5

vid 1.325 = 3.4GHz at 1.505 volts (give or take) VERY HOT
vid 1.255 = 3.4GHz at 1.435 volts (give or take) HOT
vid 1.155 = 3.4GHz at 1.335 volts (give or take)

etc...

[pitch.fr]

well this thread seems to imply that the VID is not the only thing that matters when it comes to o/c :
http://www.tomshardware.co.uk/forum/243746-11-q6600-owners

anyhow 4*3.5Ghz cores will run my AVS scripts smoother than 2*3.8Ghz I think(my E7200 has a 1.05V VID and could reach 4Ghz if I had better RAM) :D

so are you saying that the idle voltage should never exceed 1.5V in CPU-Z ?

[halsboss]

here's the Q6600 specs http://processorfinder.intel.com/details.aspx?sSpec=SLACR

[pitch.fr]

yeah OK :)

but for 45nm, the intel white papers state 1.45V as the top limit that should not be exceeded :
http://www.xtremesystems.org/Forums/showthread.php?t=173977

apparently the top limit for 65nm would be 1.55V idle in CPU-Z

you have to remain at -20C below the TJ anyway

well you see, VID doesn't explain everything :
http://www.xtremesystems.org/forums/showthread.php?t=194094

http://www.xtremesystems.org/forums/showpost.php?p=3178455&postcount=14

[steptoe]

If you intend to run a Go stepping Q6600 at 3.6ghz you will need some serious cooling, without starting a thread on cooling, I think water would be the best option with that sort of overclock

I have just built a Q6600 GO stepping on an Asus P5K premium wifi and its at least twice as fast as my previous AMD dual core 3.0ghz 6000+ running at stock speeds

It was running stable at 3.2ghz and 1.4v core using a ThermalRight Ultra 120 cooler with 120mm fan and its quiet but not as quiet as water cooling would be. It was fine until I used video encoding as that was pushing all 4 cores at about 98% so would lock up frequently. Running at default CPU speeds has cured that for the moment

People have managed 4Ghz on this processor but you need a good cooling set-up and a good motherboard that will let you change the settings to squeeze the best out of everything. The Asus P5k has plenty of settings and adjusts things in very small amounts, I nearly bought an Nvidia chipset but the reviews say these run very hot even at default speeds, so thought I'd try the P35 chipset as its an Intel CPU. Great setup

I assume the memory couldn't take the overclock as its only DDR2 667 Crucial Value Select memory which is great for every day use but doesn't like being overclocked or the CAS setting being altered too much, so I'm looking at replacing it with 4GB of Corsair Dominator DD2 PC-8500 1066Mhz which should make things a lot more stable

I need to do more work on getting it stable with the faster memory as it looks like the memory is now the bottleneck


Running DVD-RB with HC Encoder running at virtually 100% on all 4 cores I'm getting an average of around 35-45fps per instance of HC as DVD-RB fully supports running HC encoder on all 4 cores

The temperatures were running around 59-61c per core which is hot, but stable and drops back to around an average 34-35c when doing no video encoding. So 3.4Ghz will need some serious cooling

[pitch.fr]

even this doesn't work in ffdshow :

SetMTMode(1,4)
VD_UnsharpMask(diameter=5,strength=70,threshold=50)
Distributor()

as usual, ffdshow crashes instantly...I'm dubious about getting SetMTMode to work in ffdshow/vdub :D

and actually there's no point to MT ConvertToRGB32 :

YV12>RGB32
ConvertToRGB32(matrix="PC.709")
=295

YV12>RGB32
ConvertToYUY2()
ConvertToRGB32(matrix="PC.709")
=285

YV12>RGB32
MT("""ConvertToRGB32(matrix="PC.709")""",4)
=250

YV12>RGB32
MT("""ConvertToRGB32(matrix="PC.709")""",8)
=256

and BTW, when using this kinda script :

SetMemoryMax(1024)
MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 8)
ConvertToRGB32(matrix="PC.709")
MT("""
VD_UnsharpMask(diameter=5,strength=75,threshold=100)
ddcc(chr_i=3,gam_i=5,ofile="C:\COLOR.txt",threads=1,opt=1)
""", 8)

=29.2

SetMemoryMax(1024)
MT("""
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
ULevels(preset="tv2pc")
""", 8)
ConvertToRGB32(matrix="PC.709")
MT("VD_UnsharpMask(diameter=5,strength=75,threshold=100)",8)
ddcc(chr_i=3,gam_i=5,ofile="C:\COLOR.txt",threads=6,opt=1)
=30

does it make any difference if I set the SetMemoryMax() to 512 or 1024 ?

does it lower the jitter or sumthing if I go 1024 ?

lol this script is using 92% of my 3.8Ghz E7200.......good thing I'll have a Q6600 in 2 days :D

EDIT : humm Vdub doesn't like that I use VD_UnsharpMask() in my test.avs
it crashes almost each time....so I can't benchmark the whole script :(

..but it looks pretty good :D

http://pix.nofrag.com/e/5/e/57a2aab322bcaa91d7a95add65d2ftt.jpg (http://pix.nofrag.com/e/5/e/57a2aab322bcaa91d7a95add65d2f.html)

http://pix.nofrag.com/2/8/9/3363fffe1a03a6ba45508b4c8779ett.jpg (http://pix.nofrag.com/2/8/9/3363fffe1a03a6ba45508b4c8779e.html)

[ikarad]

1) Does MT filter work with telecine filter in avisynth (Telecide + decomb) and accelerate inverse telecine with multithread support?

2) Does Mt filter work with ffdshow filters (like gradfun ou resize filters for example) to have multihtread support in these filters?

[Leak]

1) Does MT filter work with telecine filter in avisynth (Telecide + decomb) and accelerate inverse telecine with multithread support?
Don't even think about doing that - MT works by cutting the frames into pieces and handling each of those pieces separately then pasting the results together.

Telecide and Decomb need to have access to the whole image to make their decisions, though - otherwise you might get the top half of the image decimated and the bottom half blended, or different matches for the top and bottom half, not to mention that the frame to be decimated can also be totally different for both parts MT creates.

In short: Don't do it.

2) Does Mt filter work with ffdshow filters (like gradfun ou resize filters for example) to have multihtread support in these filters?
Since ffdshow only uses AviSynth in it's AviSynth filter - no.

[Bexley]

Sorry if this is a dumb question, but I've just gotten a Q6600 and I'm working with MT for the first time. I haven't read this entire thread carefully, but I have skimmed it and searched for some key terms and haven't found what I'm looking for.

I'm working with some very difficult and poor quality VHS material, and the only script I've found that I'm happy with is iip (I know it's old, but it's the only thing I've found that gives a decent result). Everything I've found says to use SetMTMode() instead of MT() with a resizing function, but SetMTMode(2,0) and SetMTMode(2,4) both appear to only be using 1 core (CPU usage stays at 25% in VDub) and I'm getting 1-2 fps. Surely I should be getting better than that with a quad-core.

I know Dust doesn't work with MT, so I've replaced it with LRemoveDust as the iip denoiser, so that's not it. Here's the script I'm using:

SetMemoryMax(1024)

AVISource("C:\Untitled Clip 01.avi").ConverttoYV12()
a=Trim(2316,8063).Fadein(15).Fadeout(15)
b=Trim(11760,27505).Fadein(15).Fadeout(15)
c=Trim(31500,43149).Fadein(15).Fadeout(15)
d=Trim(46838,54026).Fadein(15).Fadeout(15)
AlignedSplice(a,b,c,d)

AssumeBFF()
Telecide(order=0).Decimate()

SetMTMode(2,0)

iip(720,480,duststr=16,detailcontr1=256,detailcontr2=0,contr_radius=3,ss1_x=1.4,ss1_y=3.0,ss2_x=3.0,ss2_y=5.0)
Tweak(sat=1.1)
Crop(8,0,-8,-6)
Addborders(8,3,8,3)

Any ideas on this, or is 1-2 fps really the best a Q6600 can do?

[martino]

Since ffdshow only uses AviSynth in it's AviSynth filter - no.
Well, there is an ffdshow() filter provided with the ffdshow package that allows you to use its filters. Just search for "ffavisynth", but I've never tried myself whether it could be used with MT.

Or did I misinterpret that sentence?

[Jeremy Duncan]

will the current mt avisynth.dll change the latest version of avisynth if i copy over the new version in the system32 folder with the mt version?
or will it just add mt to it?
or worded differently, which version of avisynth is mt compatable with?

[Adub]

2.57. I believe that tsp is waiting until 2.58 becomes final before creating an MT version.

[Leak]

Or did I misinterpret that sentence?
I was assuming he wanted to know whether MT was multithreading ffdshow's "regular" filters when using ffdshow for playback, which of course it doesn't.

Using the filters in AviSynth in ffdshow could work, but that's just plain evil so I didn't even consider it...

[cweb]

I'm working with some very difficult and poor quality VHS material, and the only script I've found that I'm happy with is iip (I know it's old, but it's the only thing I've found that gives a decent result). Everything I've found says to use SetMTMode() instead of MT() with a resizing function, but SetMTMode(2,0) and SetMTMode(2,4) both appear to only be using 1 core (CPU usage stays at 25% in VDub) and I'm getting 1-2 fps. Surely I should be getting better than that with a quad-core.

I'm no MT expert but I usually use SetMTMode(2,2) on my athlon64....

[Adub]

@Bexley,
What codec are you encoding with?

[Bexley]

What codec are you encoding with?

MPEG-2. This is for DVD.

I'm not encoding with VDub, just using it to test the script.

[Adub]

Hmm...Virtualdub can be a little awkward when showing processor usage.

Actually, showing processor usage of Avisynth scripts in general is a little weird. I find the best way to know processor usage is by dragging and dropping the script into Media Player Classic, and look at the cpu usage there. If it still says 25%, then that is an issue and we can work to solve that.

However, it it racks up and starts using ~80% or more, and you are only getting 1-2fps, then there is nothing you can do about it. (except overclock or get a better processor.)

[Bexley]

No, it's really 25%. I've checked it in XP with Performance Monitor and in Wine with Gnome System Monitor. Performance is also the same whether I have the SetMTMode() command or not. It looks like it's really only using 1 core.

2FPS is only marginally better than I was getting with my old single-core Athlon64 3500+.

[thetoof]

iirc, setmtmode must be the very first line of your script... and if the filters are not compatible with SetMTMode(2,0), make several calls of SetMTMode by changing the mode.
Also, I recommend testing increasing the # of thread since you can use more threads than the # of CPU you have to make it run faster... it all depends on your filterchain.

[Bexley]

That got it. I moved SetMTMode to the first line and not much happened. But then I increased the number of threads to 8 and CPU usage shot up to 75%. At 12 threads it hit 100% and I'm getting 6-8fps. That'll do nicely.

Thanks for the assist. :thanks:

HOWEVER,

I understood from my searching not to use SetMTMode before Telecide().Decimate() because it splits the frame up and telecide needs the whole frame intact to do its combing detection. That's why I had SetMTMode after telecide in my script. Is this not right, or have I misunderstood something?

[squid_80]

SetMTMode does not split frames for processing. MT() does.

[lych_necross]

I was wondering, is there any plan to update MT to support Avisynth 2.58 when it goes final?

[Fizick]

Yes, such plan was existed. http://forum.doom9.org/showthread.php?p=1063222#post1063222
but tsp is absent since jan2008.

may be you will do it? ;)

[lych_necross]

I would love to do it; however, I don't know how. Maybe one of the Avisynth devs could undertake the update or maybe merge MT with 2.58 when it goes final.

[Fizick]

The official support of MT is planned by Avisynth dev in Avisynth 2.6.0

[halsboss]

tsp's last post was 4 April 2008 http://forum.doom9.org/showthread.php?p=1121340#post1121340

I hope like Crikey tsp's OK and stays around, or I'll be with the older MT version until 2.6.0 ... I'd suggest lots of people now depend on MT for significantly increased throughput since multi-cores have become ubiquitous :)

[cyberbeing]

tsp's last post was 4 April 2008 http://forum.doom9.org/showthread.php?p=1121340#post1121340

I hope like Crikey tsp's OK and stays around, or I'll be with the older MT version until 2.6.0 ... I'd suggest lots of people now depend on MT for significantly increased throughput since multi-cores have become ubiquitous :)
TSP is still around, he probably is just waiting for 2.5.8 final to be released before making a new MT version.


Last Activity: 11th September 2008 13:21

[foxyshadis]

The official support of MT is planned by Avisynth dev in Avisynth 2.6.0

Or more precisely: Whenever IanB obtains a dual-core processor.

[Leak]

Or more precisely: Whenever IanB obtains a dual-core processor.
Good god - the cheapest dual-core celerys go for 38 EUR around here, with cheap mainboards at around the same price... :eek:

Anybody for setting up an "get IanB a dual-core machine" fund? ;)

[halsboss]

If we could paypal it, I'm in for au$10.

[Zep]

If we could paypal it, I'm in for au$10.

me too


I would really love to get MT at the IanB level lol as well as a file read/decode to MEGA cache that can read WAY ahead and use as much ram as I throw at it. (ram disk is just not as good since it would have to be 20+ gigs to fit my .TS file on it lol)

SetDecodeCacheSize(2048)

etc...

I wonder if that could be made as a separate process so those using 32 bit OS could use more than 2 gigs?

[Quark.Fusion]

MTsource
MTSource(string filter,int delta,int threads,int max_fetch)

All parameters are named. Function parameters:

filter string = No default
source filter to run multithreaded. Currently only internal and external source filters are supported (like DirectShowSource, Avisource, MPEG2Source) . You can use an avs defined filter or a non-source filter but it might crash or produce frame corruption.

delta int = 1
this is how many frames there are between each frame request so if you are only going to read every second frame set it to 2 or if you are reading the frames backwards set it to -1. More complex frame access pattern like SelectEvery(10,3,6,7) are not supported (but might work anyway as the requested frames are in the cache, there will just be some waisted memory from non requested frame in the cache)

threads int = 2
number of threads to run. Set this to the number of threads your computer is able to run concurrently.

max_fetch int = 30
This is the maximum number of frames ahead of the currently requested frame that MTsource will fetch. Setting it to low will leaving the threads idle for most of the time and setting it to high will waste to much memory.


Just use last parameter, that not in size, but you usually don't know how much memory avisynth will use before running test pass. Use Width()*Height()*1.5*frames for cache size for YV12.

[Quark.Fusion]

For separate process you can use TCPserver/TCPsource, but they will eat some CPU time. And you still need 64-bit OS to use >3.5GB of memory.

[saint-francis]

Or more precisely: Whenever IanB obtains a dual-core processor.

That's nuts. We'll buy him one. I'm in for some cash too.

[lych_necross]

I don't think that its the lack of a dual-core processor thats stopping IanB (or anyone else). They're pretty cheap now a days. I think its a lack of interest thats preventing further development. Or maybe severe procrastination.

[squid_80]

I think its a lack of interest thats preventing further development. Or maybe severe procrastination.If you're going to speculate, at least pick some decent reasons:
- All work on avisynth is done on a volunteer basis
- Most programmers have full-time jobs
- Some programmers even have families (so I've heard)

There's three reasons off the top of my head that I find a lot more believable (and less offensive) than your thinly veiled accusation of laziness.

[Zep]

For separate process you can use TCPserver/TCPsource, but they will eat some CPU time. And you still need 64-bit OS to use >3.5GB of memory.

yes I know but 2 gigs VS 3 (.5 for XP) is a nice increase. and why I brought it up. I hit the 2 gig wall on most of my encodes.

[Zep]

Just use last parameter, that not in size, but you usually don't know how much memory avisynth will use before running test pass. Use Width()*Height()*1.5*frames for cache size for YV12.

I assume that was directed towards me. Anyway, I tried that long ago and it is much slower and prone to crashing.

[pitch.fr]

hi there,

I've never managed to get SetMTMode() working in ffdshow, but apparently that's because I need to use ffdshow_source() after using it....as it needs to be the first line of the script or it won't work.

anyone knows why it says "invalid arguments to function LSF" please ?

SetMtmode(1,4)
source=ffdshow_source()
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
Distributor()

I've tried with all the MTmodes available..

this works perfectly fine :

MT("LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)",4)

thanks,

[Gavino]

anyone knows why it says "invalid arguments to function LSF" please ?
SetMtmode(1,4)
source=ffdshow_source()
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
Use simply:
ffdshow_source()

[pitch.fr]

great, thanks for the fast reply!

and what should I replace it by to do benchmarks in VirtualDub ?

it crashes if I remove ffdshow_source(), and If I also remove Distributor() it's single threaded.

I'd like to see whether it's faster than MT()

here's my test script for benchmarks in VDUB, that IanB gave me :

SetMemoryMax(512) # Adjust value as required

ColorBars(1280, 720, pixel_type="YV12") # Size and pixel type as appropriate

# Insert Code under test here
SetMtmode(2,4)
LimitedSharpenFaster(ss_x=1.0,ss_y=1.0,strength=40)
Distributor()

# Throw away most of the image we are not testing VD's i/o
Crop(0, 0, 16, 16)

[MADAJ]

hi there :)

I have an Intel(R) Core(TM)2 Quad CPU

and I am trying to apply this filter to my AVS script
loadplugin("F:\dgmpgdec150rc5\DGDecode.dll")
mpeg2source("ep3.d2v")
MT(4,2)

Crop(2,0,-2,-2)
tfm.tdecimate
LanczosResize(740,480)

but I get an error which is ..
Script error: Invalid arguments to function MT

I placed avisynth.dll to system32
and MT.dll to my AVS plugin

have I done something wrong??


thanks in advance

[Quark.Fusion]

RTFM :) (syntax in first post in THIS topic)

[MADAJ]

RTFM :) (syntax in first post in THIS topic)

sorry if I misunderstood the 1st post, but after seeing some people's scripts, I come up with this one.




SetMTmode(4,4)
loadplugin("F:\dgmpgdec150rc5\DGDecode.dll")
mpeg2source("ep3.d2v")
MT(4,4)
Crop(2,0,-2,-2)
tfm.tdecimate
LanczosResize(740,480)

I says "there is no function "SetMTmode"



Please tell me what I am missing. :(

[halsboss]

after seeing some people's scripts, I come up with this one

Look a bit closer at them :) You're thinking of SetMTmode, not MT ... RTFM means read the forgoodnessake manual. Don't forget to "install" the modified avisynth. Take a look at the parameters for MT - you're missing the critical one.

See Wiki info at http://avisynth.org/mediawiki/MT_support_page and http://avisynth.org/mediawiki/MT_modes_explained

[MADAJ]

thanks halsboss...

I don't an error with this one


SetMTMode(2,4)
loadplugin("F:\dgmpgdec150rc5\DGDecode.dll")
mpeg2source("ep3.d2v")
MT("blur(0.2)",4,2)
Crop(2,0,-2,-2)
tfm.tdecimate
LanczosResize(740,480)


http://img88.imageshack.us/img88/1813/19247821eg4.jpg
but it gives me the same speed whether I use it or I don't.

sorry for that guys.

[halsboss]

You're using a combination of MT mode 2 and MT ... if your source interlaced ? I suggest you have a look at the examples again and the text that goes with them. You should get a lot better than "the same".

[mgh]

If I load my plugins and then
setmtmode(2,0)
and then load my video and filters (other than builtin, the only ones i use are dfttest and shockwave, both of which are sloooow). There is no problem, cpu loading on my four cores is 95%+ and speed is 90% frames encoded in real time as long as i am not exceeding D1 resolution.
If i go to double D1 resolution using EEDI2 or tdeint, the encode crashes at completion or does not load. I need to change it to setmode(5,0) to avoid that. Only about 40 to 45% of the cores are used and encoding is about 1/6th frames in real time.
On a hunch, tried to use setmtmode (2,0) to start and put setmode(5,0) just before resizing(which i do at the end). success! no crash, cpu loading 95%+ and encoding speed was half the frames in real time.

[halsboss]

IIRC setMTmode needs to be 5 before the file open mpeg2source, and then changed after or bad things can happen. I'm sure there's posts on it somewhere. I've used TDEINT no worries although I use the "other" resizing method now.

[mgh]

did more trials
setmtmode(2,0) before loading the video file works where final encoded resolution is at the most 720x480 (NTSC DVD), also works with vcds blown up to double the resolution.
setmtmode(5,0) before loading the video file and setmtmode(2) afterwards works for higher encoded resolution (including PAL DVD 720x576!)
with my quad 2400 Mhz 2GB RAM XP SP2

[superuser]

tsp can be shed more light about Script Clip and in regards to MT?

Offtopic: which other known avisynth plugins use scriptclip functionality?

...
...

Reason I am asking about this coz I was running into errors with SRestore when using with MT:

Before you start to test with MT i would ask the developer if MT support ScriptClip-Enviroments yet. I don't think so. MT is still in development...

Thanks in advance

[rkalwaitis]

TSP or anyone else in the know :)

I tried using the following settings and I do not think Ive done it correctly. Does the below look right?

I have an AMD Athlon(tm) Dual Core Processor 4200+ 2.20GHz

Im peaking at about 4.73fps, I know that fftd3filter is not fast by nature.

SetMTMode(2,0)
DGDecode_mpeg2source("C:\VTS_08_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 8, 64, -8, -64)
MT("YToUV(fft3dfilter(sigma=3, plane=1).UToY,\
fft3dfilter(sigma=3, plane=2).VToY,\
fft3dfilter(sigma=2, plane=0))
FFT3DFilter(bt=-1, sharpen=0.8)
fastlinedarkenmod(thinning=0, strength=25)")

not using the MT filter I peak at about 5.15fps

like this..

DGDecode_mpeg2source("C:\VTS_08_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 8, 64, -8, -64)
YToUV(fft3dfilter(sigma=3, plane=1).UToY,\
fft3dfilter(sigma=3, plane=2).VToY,\
fft3dfilter(sigma=2, plane=0))
FFT3DFilter(bt=-1, sharpen=0.8)
fastlinedarkenmod(thinning=0, strength=25)

not sure what im doing wrong or if Im even using the fft3filter optimally.

thanks guys, your knowledge is valued.

[rkalwaitis]

SetMTMode(2,0)
DGDecode_mpeg2source("C:\Users\Baba-Nator\Desktop\Movies\Zoolander\VTS_08_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 8, 64, -8, -64)

MT("fft3dfilter(sigma=3, sigma2=5, sigma3=10, sigma4=20, plane=0, bt=3, bw=16, bh=16, ow=8, oh=8, sharpen=0.3, smin=20, smax=1000, wintype=2, kratio=1.0, measure=true, interlaced=false, degrid=1)")

This way gets 5.65fps


DGDecode_mpeg2source("C:\Users\Baba-Nator\Desktop\Movies\Zoolander\VTS_08_1.d2v",info=3)
ColorMatrix(hints=true)
crop( 8, 64, -8, -64)

fft3dfilter(sigma=3, sigma2=5, sigma3=10, sigma4=20, plane=0, bt=3, bw=16, bh=16, ow=8, oh=8, sharpen=0.3, smin=20, smax=1000, wintype=2, kratio=1.0, measure=true, interlaced=false, degrid=1)

This way gets 6.06 fps

[Adub]

Pick one or the other here. You are using both SetMTmode and MT. My guess is that you only want to thread fft3dfilter here, especially since the other filters are already fast enough. So just us MT and see if that fixes it. Remove your "SetMTMode(2,0)" line.

[rkalwaitis]

Thanks Ill give it a shot. I knew I was dorking something up :)

[Adub]

Yeah, be sure to report back with your results.

[kemuri-_9]

why not just use fft3dfilter's ncpu parameter?
it has built in multithreading capabilities...

[Fizick]

kemuri-_9, it is (currently) implemented not very effectively.

[kemuri-_9]

ah i see, well is there plans to increase the efficiency of the multithreading capability?

[Fizick]

i can ask you same question :)

[rkalwaitis]

Merlin, as you can see this is with the MT filter. I averaged 5.9 fps to 5.92fps. It moved around in between those numbers.

DGDecode_mpeg2source("C:\Users\Baba-Nator\Desktop\Movies\Kung Fu Hustle\VTS_01_1.d2v",cpu=4,info=3)
ColorMatrix(hints=true)
#deinterlace
crop( 0, 72, 0, -72)

#resize
#denoise
undot()
MT("fft3dfilter(sigma=3, sigma2=5, sigma3=10, sigma4=20, plane=0, bt=3, bw=16, bh=16, ow=8, oh=8, sharpen=0.7, smin=20, smax=1000, wintype=2, kratio=1.0, measure=true, interlaced=false, degrid=1)")

without the filter

DGDecode_mpeg2source("C:\Users\Baba-Nator\Desktop\Movies\Kung Fu Hustle\VTS_01_1.d2v",cpu=4,info=3)
ColorMatrix(hints=true)
#deinterlace
crop( 0, 72, 0, -72)

#resize
#denoise
undot()
fft3dfilter(sigma=3, sigma2=5, sigma3=10, sigma4=20, plane=0, bt=3, bw=16, bh=16, ow=8, oh=8, sharpen=0.7, smin=20, smax=1000, wintype=2, kratio=1.0, measure=true, interlaced=false, degrid=1)

I averaged about 5.61fps to 5.63fps

[rkalwaitis]

Fizick is there a need to undot() with the fft3dfilter?

[Fizick]

rkalwaitis, probably no, but who know :)

But somewhat is wrong with MT for you. try to add explicit paramer for number of threads to end of MT command.
and look cpu usage in Task menager
.

[rkalwaitis]

Fizick, how do I do that?

MT("fft3dfilter(sigma=3, sigma2=5, sigma3=10, sigma4=20, plane=0, bt=3, bw=16, bh=16, ow=8, oh=8, sharpen=0.7, smin=20, smax=1000, wintype=2, kratio=1.0, measure=true, interlaced=false, degrid=1)",2)


Like this?

[rkalwaitis]

I wanted to use the FFT3dGPU filter and even though my video card is listed as a good for use, I can never get it to work :(

The fft3dfilter is faster for me.

[lansing]

Haven't seen somewhat big gain in speed with MT to my script here.

x264 encoding with Megui, E4300 cpu

Before: 2.0 fps, 60-65% cpu usage
after: 2.14 fps, 61-70% cpu usage


SetMTMode(2,0)

AVISource("source.avi")

########## Stage 1 - Denoise clip ##########
limitedsharpenfaster(strength=80)
toon(0.7)
tweak(bright=10,sat=0.9)

source = last
pred = source # to get stronger denoising, put denoisers here, they will change how motion vectors are predicted

backward_vec2 = pred.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
backward_vec1 = pred.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec1 = pred.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)
forward_vec2 = pred.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=2, idx = 1, truemotion=true)

maskp1 = mvmask(kind=1, vectors=forward_vec1, ysc=255).UtoY()
maskp2 = mvmask(kind=1, vectors=forward_vec2).UtoY()
maskp3 = mvmask(kind=1, vectors=backward_vec1, ysc=255).UtoY()
maskp4 = mvmask(kind=1, vectors=backward_vec2).UtoY()
maskf = average(maskp1, 0.25, maskp2, 0.25, maskp3, 0.25, maskp4, 0.25).spline36resize(source.width, source.height)

SetMTMode(5)
smooth = pred.fft3dgpu(bw=16, bh=16, ow=8, oh=8, bt=1, sigma=4, plane=0)
SetMTMode(2)
source2 = maskedmerge(source, smooth, maskf)

source3 = source2.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)
source3
ttempsmooth(maxr=7)
gradfun2db(1.51)

########## Stage 2 - DeSpot denoised clip ##########
s2 = last # pass denoised clip to Stage 2

backward_vectors = s2.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1) # we use explicit idx for more fast processing
forward_vectors = s2.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1)
forward_compensation = s2.MVFlow(forward_vectors, idx=1, thSCD1=500) # or use MVCompensate
backward_compensation = s2.MVFlow(backward_vectors, idx=1, thSCD1=500) # or use MVCompensate
# create interleaved 3 frames sequences
interleave(forward_compensation, s2, backward_compensation)

DeSpot(p1=12,p2=1,p1percent=10,pwidth=40,pheight=40,mthres=5,mheight=5,mwidth=7,merode=30,maxpts=0,minpts=5,\
sign=0,show=0,seg=2,color=true,motpn=true)

filtered = selectevery(3,1) # pass filtered clip to Stage 3

########## Stage 3 - Apply scene-change-frame replacement ##########
prev = filtered.selectevery(1,-1)
next = filtered.selectevery(1,1)
filtered.SCSelect(next,prev,filtered,dfactor=2)

########## Stage 4 - protect good frames manually from wrong frame replacement from Stage 3 ##########
s3 = last

section1=s3.trim(0,3235)
section2=filtered.trim(3236,3726) # for isolation of good frames
section3=s3.trim(3727,11791)
section4=filtered.trim(11792,12167) # good frame
section5=s3.trim(12168,13116)
section6=filtered.trim(13117,13928) # good frames
section7=s3.trim(13929,0)

section1+section2+section3+section4+section5+section6+section7

[Sagekilla]

If I had to give a guess, I'd say it was because your clip has very complex temporal dependencies, which isn't something that's easily multithreaded.

[Fizick]

this script has also wrong using of idx...

[Sagekilla]

IIRC, you need to set up a global idx and modify that inside of your MT function or something to that nature.

[lansing]

like this?

idx1=1
idx2=2

backward_vec2 = pred.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=2, idx = idx1, truemotion=true)

...

backward_vectors = s2.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = idx2)

[Sagekilla]

Sorry, ignore my last comment about global idx's. I was thinking of if you were using MT("""Function()"""). I don't think MVTools can multithread properly with SetMTMode(). I remember trying to use it but it didn't really help even on a simple script that looked like:


SetMTMode(2,0)
MPEG2Source("source.d2v")
source = last

bvec = source.MVAnalyse(isb=true, delta=1, idx=1)
fvec = source.MVAnalyse(isb=false, delta=1, idx=1)

source.MVDegrain1(bvec, fvec, idx=1)

[Adub]

I use motion compensated scripts with setmtmode(2) all of the time. I am running a script as we speak with MC_Spuds with setmtmode(2). There shouldn't be any issues.

[thetoof]

Or.... get MVTools2 to to prevent any idx mess up.

[halsboss]

Looks like there might be a bug in the combination of MT with HCenc, to do with scene changes ? After a scene change, the 1st to 2nd frame "go backward" and then motion starts forward again. Very unusual and quite offputting to view.

Took me quite a while to get around to checking it out as I thought it was just my eyes... then I got jack of it and did 2 HCEnc encodes of exactly the same clip, one with MT and the other without ... and sure enough the problem is there when frame-by-frame forwarding one and then not in the other when frame-by-frame forwarding the other.

Any suggestions ?

MT code (has been chopped up a bit from a larger .avs from which I comment/uncomment options):

SetMTmode(mode=5,threads=4) # start with mode=5 forAVIsource http://forum.doom9.org/showthread.php?p=1067216#post1067216
SetMemoryMax(256)
# snip
AviSource("G:\DVD\src\src.avi", audio=false)
AssumeFPS(25)
SetMTmode(mode=2,threads=4) # mode=2 for temporal multi-threading (interleaved frames)

ConvertToYV12(interlaced=FALSE) # for Deblock_QED_MT2 and DeHalo_alpha
Deblock_QED_MT2(quant1=30)

SetMTmode(mode=2,threads=4) # mode=2 for temporal multi-threading (interleaved frames)
ConvertToYUY2(interlaced=FALSE) # Ensure YUY2 for the rest including Convolution3D

SetMTmode(mode=5,threads=4) # # mode=5 for safety including using MT
Function Do_Stuff_In_MT(clip "inpclp") {
zMTclp=inpclp
zMTclp=zMTclp.Convolution3D(0, 6, 10, 6, 8, 2.8, 0)
zMTclp=zMTclp.spline36resize(720,zMTclp.height()) # 1st half of resize (2nd half outside this MT function !!
RETURN zMTclp
} # end of function Do_Stuff_In_MT
MT("Do_Stuff_In_MT(LAST)",threads=4,overlap=4,splitvertical=false)
MT("spline36resize(last.width(),576).LimitedSharpenFaster(smode=4,strength=100)",threads=4,overlap=4,splitvertical=true)
SetMTmode(mode=2,threads=4) # mode=2 for temporal multi-threading (interleaved frames)
Converttoyv12()
SetPlanarLegacyAlignment(True)
Distributor() # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1063622#post1063622


non-MT code (has been chopped up a bit from a larger .avs from which I comment/uncomment options):

AviSource("G:\DVD\src\src.avi", audio=false)
AssumeFPS(25)

ConvertToYV12(interlaced=FALSE)
Deblock_QED_MT2(quant1=30)

ConvertToYUY2(interlaced=FALSE)
Convolution3D(0, 6, 10, 6, 8, 2.8, 0)
LimitedSharpenFaster(smode=4, dest_x=720, dest_y=576)

Converttoyv12()
SetPlanarLegacyAlignment(True)

[Adub]

One reason could be that you are multithreading the crap out of that script. It is usually a good idea to use either MT OR SetMTMode, not both. Also, you don't need to keep calling setmtmode(2,4).

Example:

SetMTmode(mode=5,threads=4) # start with mode=5 forAVIsource http://forum.doom9.org/showthread.php?p=1067216#post1067216
SetMemoryMax(256)
# snip
AviSource("G:\DVD\src\src.avi", audio=false)
AssumeFPS(25)
SetMTmode(mode=2,threads=4) # mode=2 for temporal multi-threading (interleaved frames)

ConvertToYV12(interlaced=FALSE) # for Deblock_QED_MT2 and DeHalo_alpha
Deblock_QED_MT2(quant1=30)
ConvertToYUY2(interlaced=FALSE) # Ensure YUY2 for the rest including Convolution3D

#SetMTmode(mode=5,threads=4) # # mode=5 for safety including using MT ####This should only be used if you are seeing issues with it commented, otherwise enable it.
Convolution3D(0, 6, 10, 6, 8, 2.8, 0)
Converttoyv12()
spline36resize(720,576).LimitedSharpenFaster(smode=4,strength=100)
SetPlanarLegacyAlignment(True)
Distributor() # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1063622#post1063622

BIG FAT DISCLAIMER: I have not tested this script at all. I do not know if it will function. I posted this edited version to give an example of my point. This will need to be fully tested and probably tweaked for full compatibility, but right now it is just provided as an example.