Avisynth 2.6 MT

[DarkT]

Simple, you can't use 64 bit plugins with 32 bit avisynth. Stick to the 32 bit plugins if you're using 2.6.

Wait... Wait... so this 2.6 MT version of set... it's 32bits?! Damn... I though it's 64bits >.< *sighs* that would indeed explain everything... *sighs* Ah well, this is even better *Grins* no need to torment myself now with "nice that you got MT working... BUT... if it were MT AND 64bit, it would prolly be even FASTER!!!" *chuckles*

Edit:

I am encoding as we speak... wow... Jsut... freaking... WOW... I did NOT expect that...

The script:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\nnedi.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\addgrainc.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\TDEINT.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\TMM.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\TIVTC.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\mt_masktools.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\removegrainsse2.dll")
Import("C:\Program Files (x86)\AviSynth 2.5\EXTRA\LimitedSharpenFaster.avs")
Import("C:\Program Files (x86)\AviSynth 2.5\EXTRA\MCTemporalDenoise.v1.4.20.avsi")
Import("C:\Program Files (x86)\AviSynth 2.5\EXTRA\LSFmod.v1.9.avsi")
Import("C:\Program Files (x86)\AviSynth 2.5\EXTRA\GradFun2DBmod.v1.5.avsi")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\fft3dfilter.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\gradfun2db.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\RepairSSE2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\EXTRA\Yadifmod.dll")

SetMemoryMax(512)
SetMTMode(3)
AVISource("Lulz.avi").ConvertToYv12(interlaced=true)
SetMTMode(2)
FFT3DFilter(sigma=1.0, sharpen=0.3,interlaced=true)
Yadifmod(order=0, field=-1, mode=0, edeint=nnedi(field=-1))
LimitedSharpenFaster(strength=37)
MCTemporalDenoise()
Lanczos4Resize(872,480)
Crop(12,0,-12,-0)
Trim(1,66562)

Thus far - when I have tried it without MT it was encoding at 4fps... Now? 12 fps... It is THRICE as fast!!! 300% gain, WTF?! AMAZING... SET, MAKE ME A KID!!! *grins*

It did crash when I set memory to 768. Crashed each time I tried it.

It hadnt finished encoding yet, but it will in an hour or so, I'll report how it went.

[ryrynz]

Can't say I've had any major stability problems with 2.58MT, I run about six filters in realtime, I tested 2.6MT but found it to be considerably slower with my script even with using SetMTMode(1). I guess everybody's mileage will vary.
Will have to try limiting threads and see what happens, how does 2.6MT run compared to 2.58MT for you guys performance wise generally speaking?

[Mosfetov]

Some syntax works some do not.

Is it still possible to use mt() with current 2.6.0MT?
Every syntax resuilt in error "wrong avisynth version...".

[Zep]

Is it still possible to use mt() with current 2.6.0MT?
Every syntax resuilt in error "wrong avisynth version...".

if i recall correctly Set said no more support for mt(). just use the SetMTmode() calls

[SEt]

I'm not maintaining MT(), so it quite possible broken. MTMode is way to go.

[DarkT]

Got a problem with fft3dfilter and fftw.dll on MT on HD material.

Ok, if you look a few posts up, you can see I used fft3dfilter with MT mode 2, just fine, but it was on SD material

Now, I am trying to use it on HD material, and it's a no go, When I set it to MT mode 1 it does work, and it does help - without it it's at 2fps, with it it's at ~5fps.

I just wonder what's up.

I tried the same fft3dfilter line I used on teh SD material on the HD material... It's a no go... how strange... Any ideas? I'll try to re-run it on SD material, perhaps something in the environment changed, and it'd no longer be possible on SD as well, will edit later on.

edit:

"An exception occurred in module 'KERNELBASE'."

Crashed :( Gonna try and go with fft3d only.

Edit2:

IT finished the encoding, but the video is messed up, 1 second you see it, tthe next you see half the pic, etc etc... how weird..

[SEt]

Looks like crash due to memory usage. Read the first post.

Do not use mode 1 unless you know filter is designed for that mode. Otherwise it won't necessary crash but likely produce very wrong result like described.

[DarkT]

What's weird though is that fft3d on SD material works fine with mode 2, but on HD it insta-crashes... So weird... Meh, I'm running lagarith lossless 1st pass only for fft3d purposes... 25hours, and 100gb for it... *sighs* ah well...

[TheRyuu]

What's weird though is that fft3d on SD material works fine with mode 2, but on HD it insta-crashes...

A larger size frame taking up more memory? Who would've thunk it.

SetMemoryMax is your friend.
The thing with SetMTMode is that it seems mostly dependent on the amount of RAM that you can give it, which generally means (at least) SetMemoryMax(2048) for HD material for even remotely complicated scripts, which means you better have a 64-bit operating system which can give processes 4GB of address space, which means your program better have the LARGEADDRESSAWARE flag set.

Then after a bit of praying and a script which actually threads well, it just might work.

That being said I don't really like SetMTMode. What it all boils down to is that SetMTMode is a hack and shouldn't really be relied upon or expected to work in any capacity. If it works congrats, you got lucky, if it doesn't oh well and just move on. 64-bit avisynth is the same way, except that's way more hacky and a giant piece of shit, MTMode is at least a respectable attempt.

[SEt]

64-bit Avisynth is not 'hacky'. It has only one problem - no one maintains it.

SetMemoryMax is not as trivial as 'set to 2GB and you are done' - sometimes smaller values give better result than larger ones.

[TheRyuu]

64-bit Avisynth is not 'hacky'. It has only one problem - no one maintains it.

Several internal functions don't work and for a time it didn't even properly abide by the Windows x64 ABI (and I'm pretty sure the version people recommend to use doesn't properly abide by it). Sure does seem a little hacky to me.

Have you even looked at the msvc project files for x64? They have a huge amount of useless (optimization) settings turned on which give the impression whoever did it had little to no idea what they were doing.

SetMemoryMax is not as trivial as 'set to 2GB and you are done' - sometimes smaller values give better result than larger ones.

Well it was more of a guesstimate than anything else really, just based on playing around with it a little. 2GB does seem like a good starting point though at least for HD material. As soon as you start throwing temporal filters into the mix the need for a larger cache size seems to go up real quick.

I don't know how exactly avisynth (and MT related company) are doing their frame caching but if my SetMemoryMax value is too small it seems as though avisynth just chews up cpu cycles for nothing and bogs down to something extremely slow (further hurting avisynth-mt's credibility in the process). I don't really see how having too large of a frame cache could hurt performance. On a side note I guess the function name "SetMemoryMax" isn't exactly the best name for it since all it really equates to is "UseThisAmountOfMemory()", and doesn't actually define a maximum amount of memory which avisynth can chose to use up to as the name implies.

[-Vit-]

If you increase the memory for the frame cache then you reduce the memory available for the filters themselves. Some filters use little memory, but others allocate quite a lot. Setting a huge frame cache increases the chance of the filters running out of memory and crashing. SetMemoryMax is a misleading name - it really sets a memory balance between cache and free memory for filters. There is a sweet spot for any given script for SetMemoryMax, where you have reserved just enough for its caching needs and no more. I.e. you should SetMemoryMax to the smallest value that doesn't slow down processing.

Since MT processes several frames simultaneously its memory and caching demands are multiplied. MT is much more likely to run out of memory than non-MT given the 32-bit constraint of avisynth. Setting thread counts / SetMemoryMax values that just won't fit in 32-bit memory is a frequent cause of MT problems.

[TheRyuu]

Setting thread counts / SetMemoryMax values that just won't fit in 32-bit memory is a frequent cause of MT problems.

Both of which can be reduced by uses processes which have the LARGEADDRESSAWARE flag set and be running them on a 64-bit OS (yay 4GB per process!).

[SEt]

I wasn't talking about exact x64 Avisynth build but rather about x64 version of Avisynth in general - it's stalled project no one works on. Haven't looked at that x64 build so can't comment about it. Windows x64 ABI is plain retarded in part about saving xmm registers. As for msvc project settings - I see that all the time, got used to almost always recreating them myself, sigh.

As for memory consumption - I think it's too early to blame 32-bit address space. Current MT Avisynth cache wastes way more memory than required.

[-Vit-]

I think it's too early to blame 32-bit address space. Current MT Avisynth cache wastes way more memory than required.
Yes, I know what you mean. Running out of memory is the problem, but it shouldn't be. A 32-bit address space should be plenty for extreme HD processing. But avisynth caches the output of every single filter in your script (and the final output frame - unnecessary for encoding). But really only filter outputs that are accessed temporally (+ some subtle dependencies) need caching beyond the current frame, very few in the typical script. I analyzed QTGMC and it has maybe 10 intermediate frames that really need to be cached (per output frame), but 100s are actually considered for caching. I tried to use SetCacheHints to help reduce this, but at that time (2.58, haven't looked at the code recently) SetCacheHints seemed to have been rather hacked about and I had limited success.

[DarkT]

Hmm, I see I see... So what you guys are basically saying is that I should go FFT3DGPU? *grins*

[cattolicboy]

Hi SEt, do you think to implement the possibility to load LoadVirtualDubPlugin in the future?
At the moment seems not working, maybe someone knows some trick to load it...

[SEt]

I've never used such feature and can't comment about it. It looks like a request for official Avisynth: http://forum.doom9.org/showthread.php?t=149113

[debpk77]

Hi SEt, do you think to implement the possibility to load LoadVirtualDubPlugin in the future?
At the moment seems not working, maybe someone knows some trick to load it...

I've never used such feature and can't comment about it. It looks like a request for official Avisynth: http://forum.doom9.org/showthread.php?t=149113

SEt,
Is not a request for official avisynth...
I've tried it too and avisynth exits with the following code:
Traceback (most recent call last):
File "AvsP.pyo", line 6291, in OnSliderReleased
File "AvsP.pyo", line 8822, in ShowVideoFrame
File "AvsP.pyo", line 9256, in UpdateScriptAVI
File "pyavs.pyo", line 167, in __init__
File "avisynth.pyo", line 117, in Invoke
WindowsError: exception: access violation reading 0xA6446895

I'm using x64 version of windows on an intel 64 bit processor.
Its not a memory issue. I've tried various values for setmemorymax() - all the way up to 4096 and it gives the same error.

That means we cant access the vdub filters (*.vdf) from within avisynth. which is a pity..

grateful if someone can help...

[Youka]

AvsP has it's own problems, so it crashes if a plugin throws an error during filtering process and not just at initialization. Try it simply with MPC and look for an error message.

[zerowalker]

What should i do if i want to Encode with x264 10bit, and use a script that use Ditherpost() in the end?

I mean, i don´t want it to go from 16bit, down to 8bit, then to 10bit, can´t i ditherpost to 10bit?

Thanks:)

[cretindesalpes]

Instead of DitherPost(), use Dither_convey_yuv4xxp16_on_yvxx(). Check this section of the Dither manual for the encoding command line. If you want straight dithering to 10 bit from DitherPost, see this post (http://forum.doom9.org/showthread.php?p=1519194#post1519194).

[sugarkang]

I am confirming MT stable running QTGMC. As others have noted, the key to getting this to work right was setting SetMemoryMax. Once I set it to 512, no more rainbow glitch screens. I have Win 7 x64 and 16GB RAM installed, but parameters 1024 and up resulted in errors; I have not tried anything between 512-1024 yet.

Set to 512, the speed increase is roughly double that of regular Avisynth. Used to get 10-13FPS, but now getting about 22-26FPS on DV source 720x480.

Running Win7 x64 Ultimate. 2.58 x86 Avisynth was the base install; 2.6 x86 MT avisynth.dll copied to SysWOW64 and system32 (don't know which is redundant; conflicting install instructions). Core i5-2500k, OC'ed to 4.4Ghz, everything is running swimmingly.

Hope this data point helps. My exact script:

SetMemoryMax(512)
SetMTMode(5, 4)

AviSource( "M:\_ process\2_ cedocida\DV3.avi" )

ConvertToYV12(interlaced=true)
QTGMC( Preset="Slow" )

[Lighto]

I am confirming MT stable running QTGMC. As others have noted, the key to getting this to work right was setting SetMemoryMax. Once I set it to 512, no more rainbow glitch screens. I have Win 7 x64 and 16GB RAM installed, but parameters 1024 and up resulted in errors; I have not tried anything between 512-1024 yet.

Set to 512, the speed increase is roughly double that of regular Avisynth. Used to get 10-13FPS, but now getting about 22-26FPS on DV source 720x480.

Running Win7 x64 Ultimate. 2.58 x86 Avisynth was the base install; 2.6 x86 MT avisynth.dll copied to SysWOW64 and system32 (don't know which is redundant; conflicting install instructions). Core i5-2500k, OC'ed to 4.4Ghz, everything is running swimmingly.

Hope this data point helps. My exact script:

You are only required to replace the existing standard avisynth.dll in the SysWOW64 folder with the MT one in your case.
You are missing SetMTMode(2) before QTGMC() for your script, might want to double check your script with the first post in the QTGMC thread.

[Gavino]

You are missing SetMTMode(2) before QTGMC() for your script
Exactly - with script as written, QTGMC is run with MT mode 5, ie single threaded.

[sugarkang]

You are only required to replace the existing standard avisynth.dll in the SysWOW64 folder with the MT one in your case.
You are missing SetMTMode(2) before QTGMC() for your script, might want to double check your script with the first post in the QTGMC thread.

Damn. I thought I solved a problem. I'll retry, but every other combination resulted in random rainbow block frames.

[tormento]

SEt, is it compiled with the LARGEADDRESSAWARE flag? If not could you please provide one for testing?

[Chikuzen]

SEt, is it compiled with the LARGEADDRESSAWARE flag? If not could you please provide one for testing?

http://blogs.msdn.com/b/oldnewthing/archive/2010/09/22/10065933.aspx

[tormento]

Wow. Learning is a neverending job.

[jinkazuya]

I have a question. I am just wondering why the encoding process never fully uses all the cores of the CPU and the threads.

I set my MT like this

SetMTMode(2,8). I thought the encoding is gonna make use of the four cores along with the hyperthreading, but when I looked at the CPU usage, it was only 18% - 20%

[TheRyuu]

I have a question. I am just wondering why the encoding process never fully uses all the cores of the CPU and the threads.

I set my MT like this

SetMTMode(2,8). I thought the encoding is gonna make use of the four cores along with the hyperthreading, but when I looked at the CPU usage, it was only 18% - 20%

Depends what your script looks like. Certain things don't thread well and most stuff non-trivial (save QTGMC) presents various other problems (http://mod16.org/hurfdurf/?p=234) for MT avisynth.

[ryrynz]

I would try calling some different MT modes in your script and or removing some filters to find out where the problem lies.

[lansing]

am i missing something here, the mt.dll for avisynth 2.6 is not in the package file?

update: I was trying to multithread fft3dfilter, but the only usage i've found is through MT("....."), how do i do this?

[pbristow]

lansing: mt.dll (and the filters it provides - MT(), MTi(), etc. ) is not supported with Avisynth 2.6. The MT *modes* are supported, but as they are built into AVisynth, they don't require mt.dll .

[Atak_Snajpera]

AviSynth 2.6.0 MT installer -> http://www.mediafire.com/?6vq1i35kz9cqjl1

Installer source files -> http://www.mediafire.com/?g7ex29zb0c1k4xt

[lansing]

lansing: mt.dll (and the filters it provides - MT(), MTi(), etc. ) is not supported with Avisynth 2.6. The MT *modes* are supported, but as they are built into AVisynth, they don't require mt.dll .

thanks for the explanation. But how do I use mt with fft3dfilter? I use it with SetMTMode(2) on my AMD athlon 4 core, but still only 25% cpu usage

[Atak_Snajpera]

you can also try hqdn3d istead of ff3dfilter.

[SEt]

Looks like no one reads explanations in first post, ever...

[GMJCZP]

With this version of Avisynth yesterday I tried to load a plugin for VirtualDub with LoadVirtualDubPlugin the way I've always done and then VirtualDubMod crashes, what is that?

In 2.5.8 MT I could perfectly.
I'll be doing something wrong or need to install anything?

[GMJCZP]

Please, I need an answer!
I had to go back to version 2.5.8 MT. I can use there are the VirtualDub fliters.

[SubJunk]

Thanks a lot for these updates, SET. I find your builds are the best ones!
It seems that env->GetMTMode() is missing, is there a reason for this? I have a plugin that relies on that function

[GMJCZP]

Please, I need an answer!
I had to go back to version 2.5.8 MT. I can use there are the VirtualDub fliters.

Any comments or suggestions?

[pbristow]

With this version of Avisynth yesterday I tried to load a plugin for VirtualDub with LoadVirtualDubPlugin the way I've always done and then VirtualDubMod crashes, what is that?

In 2.5.8 MT I could perfectly.
I'll be doing something wrong or need to install anything?

Hi GMJCZP,

To get any answers, you'll need to supply a bit more information, such as what specific plugin you're trying to load and exatcly how (the simplest and best way to do this is to post the exact script you're using), what exact symptoms you get (e.g. any error messages that occur before the crash), what other plugins have you tried and did they behave the same way, etc. Otherwise, the only suggestion anyone can reasonably make is "investigate the problem".

Also, you need to post the question in the Usage forum, not here in the Development forum.

Good luck! :)

[GMJCZP]

Hi GMJCZP,

To get any answers, you'll need to supply a bit more information, such as what specific plugin you're trying to load and exatcly how (the simplest and best way to do this is to post the exact script you're using), what exact symptoms you get (e.g. any error messages that occur before the crash), what other plugins have you tried and did they behave the same way, etc. Otherwise, the only suggestion anyone can reasonably make is "investigate the problem".

Also, you need to post the question in the Usage forum, not here in the Development forum.

Good luck! :)

Sorry if use this thread.

The problem is straightforward: All plugin Virtualdub I try to carry this latest version of Avisynth, to boot, for example, VirtualDubMod, I get the error message that this program will close.

That's all.

[SEt]

Update:

1) Updated to CVS, so everyone can try new interfaces: http://forum.doom9.org/showthread.php?p=1567792#post1567792
2) Some bugfixes/improvements, mostly exceptions-related.
3) VirtualDub plugins now work again.
4) Changed compiler to VS2010 - something could break due to this.


SubJunk, it's not missing - just moved in tables due to 2.6 stuff. Rebuild your plugin with new headers and it should work fine.

[Boulder]

Thanks, I'll download and start using the new build right away.

[SubJunk]

Thanks a lot for the new release!
If I rebuild with the new headers is there a risk of 2.5 incompatibility? Do you think 2.6 will eventually require most 2.5.8 plugins to recompile?

[SEt]

SubJunk, since it uses MT functions it's already incompatible with regular Avisynth, it will be incompatible with 2.5.8 MT due to new placement of MT functions, but there is nothing bad in it - 2.5.8 shouldn't be used anyway. If MT functions are called conditionally - you can still build it and maintain compatibility with 2.5.
2.6 will likely support 2.5 plugins without changes, but I hope they'll be recompiled if just to remove the ancient bad code that is compiled into all plugins and the main reason of "instability" of MT Avisynth mode.

[SubJunk]

OK, thanks for the explanation :)

[GMJCZP]

Thanks a lot.