MT 0.7(+custom avisynth)a filter to run filters multithreaded. Usefull for SMP or HT

[badshah]

contents of plugins folder .............

[badshah]

I don't know what happened.... its working fine now ... but speed is very slow with setmtmode(2)

without that line i am getting 20fps ... while with that line speed is 2 to 4 fps

[Boulder]

I've tried using HC with SetMTMode. It seems that there are no extra threads being used at all! The number of threads is the same even if SetMTMode(2) is called before the source is loaded. When I do the same in VDubMod, the number of threads is higher with SetMTMode.

[mroz]

@badshah: given SetMTMode is meant to be the first command in a script, is it safe to have LimitedSharpenFaster.avsi autoloading from plugins? I don't know if this makes any difference; just mentioning in case.

@Boulder: Try adding as the last line to your script
Distributor()
before giving the script to HC. This might be the same issue as with avs input to MEncoder & to Megui's Analysis pass.

[Boulder]

Originally Posted by mroz

@Boulder: Try adding as the last line to your script
Distributor()
before giving the script to HC. This might be the same issue as with avs input to MEncoder & to Megui's Analysis pass.

Yep, this works. Too bad that using SetMTMode seems to kill performance, going from 6fps down to 3fps

[mroz]

Originally Posted by Boulder

Yep, this works. Too bad that using SetMTMode seems to kill performance, going from 6fps down to 3fps

Just with HC or with VDub as well? What's the script?

[Boulder]

I only tried HC.

The script is:

SetMTMode(2)
MPEG2Source("path\clip.d2v",cpu=4)
Crop(12,78,-12,-78,true)
den=DegrainMedian(mode=2)
DegrainFFTC(last,den,sad=200,ol=4)
AddBorders(16,78,8,78)
Distributor()

And the function DegrainFFTC is in degrain.avsi which contains a few functions similar to each other. The function itself is:

global idx_counter = 10
global idx_counter_2 = 50

function DegrainFFTC( clip c, clip cleaned, int "blk", int "ol", int "sh", int "sad", int "pl", int "div", int "limy", int "limuv" )
{
global idx_counter = idx_counter + 1
global idx_counter_2 = idx_counter_2 + 1
blk = default( blk, 16 )
ol = default( ol, 0 )
sh = default( sh, 2 )
sad = default( sad, 200 )
pl = default( pl, 4 )
div = default( div, 0 )
limuv = default( limuv, 255 )
vbw1=MVAnalyse(cleaned,isb=true,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
vfw1=MVAnalyse(cleaned,isb=false,truemotion=true,delta=1,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
vbw2=MVAnalyse(cleaned,isb=true,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
vfw2=MVAnalyse(cleaned,isb=false,truemotion=true,delta=2,pel=2,chroma=true,blksize=blk,idx=idx_counter,sharp=sh,overlap=ol,divide=div)
nolimit = MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_counter_2,plane=pl)
defined(limy) ? LimitChange(MVDegrain2(c,vbw1,vfw1,vbw2,vfw2,thSAD=sad,idx=idx_counter_2,plane=pl),c,limy,limuv) : nolimit
}

The CPU usage goes to 90-99% but speed drops down seriously.

[IanB]

HC requests frames slightly out of order, if this makes it thru to MPEG2Source worst case you can end up re-decoding each source GOP every time instead of just decoding the next delta frame.

Add a SetMemoryMax(big??) to increase the available cache buffers.

Also try abusing ChangeFPS(Last, Last, True). Add it after your Distributor call, it will act as a 10 frame EnsureVBRMP3Sync for video.

[badshah]

Originally Posted by mroz

@badshah: given SetMTMode is meant to be the first command in a script, is it safe to have LimitedSharpenFaster.avsi autoloading from plugins? I don't know if this makes any difference; just mentioning in case.

I did import of LSF in begining lines & its running better now. but still encoding speed is 60% of total CPU capacity .... how to increase the speed further

[Boulder]

Originally Posted by IanB

HC requests frames slightly out of order, if this makes it thru to MPEG2Source worst case you can end up re-decoding each source GOP every time instead of just decoding the next delta frame.

Add a SetMemoryMax(big??) to increase the available cache buffers.

Also try abusing ChangeFPS(Last, Last, True). Add it after your Distributor call, it will act as a 10 frame EnsureVBRMP3Sync for video.

I have SetMemoryMax(512) as an avsi file so that should suffice I think. I'll try the ChangeFPS trick later and report back.

[mroz]

Originally Posted by badshah

I did import of LSF in begining lines & its running better now. but still encoding speed is 60% of total CPU capacity .... how to increase the speed further

The script you quoted isn't very complex so maybe the bottleneck is the encoder. What is it? x264 will get to about 80% on a first pass & around 100% on a second, however ime xvid only manages about 60% - increasing the number of threads it uses will up cpu usage, but actually drops frame rate; it doesn't seem optimised for multithreading (*) (this is on a Q6600). Of course I might simple have it misconfigured (though I was using default profiles in Megui).

(*) By which I mean while it will run multithreaded with improved performance, there does seem to be this limitation preventing full cpu exploitation.

[badshah]

^^ its Xvid ... and exactly, it doesnt go above 60 to 70 %. anyway, its fine.

[mroz]

You can always up efficiency manually on a per job basis, by splitting the encode in two & running both at the same time (in separate workers, if you're using Megui), then join the outputs when done.

It's not something I've bothered with due to the hassle & the facts I do little xvid encoding & can usually find another use for those cpu cycles.

[Boulder]

Originally Posted by Boulder

I have SetMemoryMax(512) as an avsi file so that should suffice I think. I'll try the ChangeFPS trick later and report back.

The ChangeFPS trick seemed to fix things. The performance is better with SetMTMode than without it when encoding with HC.

[IanB]

I have plans for a "LinearAccess" filter for 2.6, I'll bump it up the todo list a bit.

[vcmohan]

In some plugins(of mine ) process parameters are derived from analysis of a particular field (or frame) of the clip and applied all through. In case of MT in the mode where alternate fields are processed in different threads whether the params will be computed in both threads or is there a control on the place from which forking takes place?

If this querry does not belong to this thread pl bump it to a new thread.

[mroz]

@IanB, tsp

So far we've seen MEncoder, MeGUI & HC being unable to exploit MT for the same reason that Distributor is never invoked.

tsp, I think, said it would be very hard/impossible to modify MT/Avisynth so that this call is made automaticaly at the end of a script, other than in the AVIFile case as at present.

Since the problem is affecting many tools even in my limited experience, I'm brought back to considering this.

Is there really no way to automate a solution internal to AviSynth? Is it not possible to insert the relevant code immediately prior to processing of the first frame request?

If this can't be done, how about simplifying the modifications that need to be made to the third party programs affected? Rather than require they check to see if MT is in use & then invoke Distributor conditional on the current MT mode, why not hide all of that in a single function which is safe to call regardless of MT usage & mode? Then the only third party mod is to ensure they invoke this function at the end of script building.

Obviously this is only a full solution once MT becomes an official part of AviSynth, since prior to that the above function invokation will error, but that's probably safe to handle by documenting the function with the instruction to ignore any returned error/catch & ignore any exception.

I think simplifying the mods needed will make them more likely to be accepted. For instance, when I described the changes needed in respect of MeGUI's analysis pass option, the response I got failed to comment on the involved mode check but concentrated on confirming that switching to AVIfile access would eliminate the problem at the loss of certain abilities/controls; this strongly suggested to me that a quick simply change was being sought that wouldn't require any MT specific research prior to implementation, which makes sense for a busy dev.

Comments?

[squid_80]

IMO any apps accessing avisynth directly (bypassing avifile) should know exactly what they're doing and make the call themselves. Otherwise use AVIFile.

[mroz]

The problem with that is that atm they can know exactly what they're doing in respect of AviSynth & won't know about this, as MT isn't integrated into AviSynth.

And even well after it is, surely it still makes sense, at the very least, to require no more than the script ends with a simple native function invokation which internally deals with checking MT mode & calling distributor if needed.

If in the future it becomes necessary for AviSynth to know when a script is complete for other reasons, this would also facilitate that.

Furthermore, it would allow the particular call to be included in scripts opened via AVIfile without any penalties - at the moment a call to distributor will screw up efficiency if included directly in a script which is also opened via AVIfile. I can't see the harm in at least avoiding that problem.

[IanB]

Well the thing is you were never meant to directly call "Distributor" in scripts, I mearly suggested it as a test for a certain issue as I became aware of it. And yes it certainly makes sense to make MT as transparent as possible.