Avisynth 2.6 MT

[TheFluff]

Seems like what I wrote wasn't clear enought(perhaps not only for you) - I will restate: Least headache, while using as much of the CPU power as possible.

I do want to use multi threading. But I wonder whether 2.5.8 MT is better than 2.6. Or is it perhaps better to just run 2 or 4 simultaneous encoding proccesses? I do not know. I was hoping somebody would help me with making that decision, instead of gruelly trying it out myself.

Like Groucho2004 says, the best choice is still to use single-threaded 32-bit 2.5.8. If you want more threads, invoke more script environments.

All options other than the vanilla 2.5.8 (including all the 64-bit variants, all the MT variants and all the Avisynth 2.6 variants) are either:
a) way too unreliable,
b) way too limited, or
c) all of the above
to be useful in a production environment.

[DarkT]

64bit is a no go as well?! Blah. It's not a "production environment" either... mmm, I mean, you make it sound "serious" or something *chuckles*

I just want to encode fast... Shit... That's all... Maybe I should run 2-4 64 bits avisynth?

[TheFluff]

64bit is a no go as well?! Blah. It's not a "production environment" either... mmm, I mean, you make it sound "serious" or something *chuckles*

I just want to encode fast... Shit... That's all... Maybe I should run 2-4 64 bits avisynth?

If there existed a completely single-threaded 64-bit version, that'd at least have the potential to be stable enough for actually getting shit done (that's what I mean by "production environment"). Unfortunately, there is no such version, and using 64-bit also severely limits the amount of available plugins.

Just stick with Ye Olde 32-bit 2.5.8. If it was good enough for your grandfather, it should be good enough for you, and if you have to use so many filters that you run out of memory, the source was probably so bad that it wasn't worth encoding anyway.

Remember this, kids: trying to add multithreading to a huge software project that was written without a single thought of thread-safety in mind is madness and a recipe for an eternity of crashing. If said software project also loads and uses many third-party plugins that ALSO were written without a single thought of thread-safety in mind, you might as well kill yourself and end up in Hell already, because that's probably going to be more pleasant than what you're going to end up with.

[DarkT]

If there existed a completely single-threaded 64-bit version that'd at least have the potential to be stable enough for actually getting shit done (that's what I mean by "production environment"). Unfortunately, there is no such version, and using 64-bit also severely limits the amount of available plugins.

Dunno - I saw plenty people posting nice gains in speeds using MT ver and no stability problems. *sighs* I suppose I'll just have to try it out for myself if nobody else has anything helpful to say PRO one of teh choices... *sighs* ah well... Thanks for hte input thus far guys ;).

[TheFluff]

Dunno - I saw plenty people posting nice gains in speeds using MT ver and no stability problems.

And there are also plenty of people complaining about stuff crashing all the time. It's a multithreaded version of a non-threadsafe application, which in plain English means it may work just fine ten times in a row or crash ten times in a row, depending on factors such as the phase of the moon, your hair color, cosmic radiation levels and whether you sacrificed a goat before using the application or not.

edit: it might work if you only use it in single-thread mode, but honestly, avs-mt is just so buggy that you're not likely to notice much of a difference in stability anyway...

[SubJunk]

For me the most stable versions are SET's 2.6 builds. Every version is stable for me without multithreading, but using multithreading I have found 2.6 to crash less than 2.5.8.
It seems to be different for everyone but that's how it is for me :)

[Groucho2004]

Dunno - I saw plenty people posting nice gains in speeds using MT ver and no stability problems.

I could never get the MT versions to work properly. Some people have no problems at all.
There are a shitload of variables involved - complexity of the script, which filters are used (even the version of the filter can be crucial), in which order the filters are used, number of threads - should I go on?

All you can do is try for yourself.

[SubJunk]

The trick for MT stability for me was SetMemoryMax. Setting that to between 600-800 fixed crashes for me.

[DarkT]

Wait, so if you aren't using 2.6 for MT, what are the benefits then of 2.6 over 2.5.8?

[TheFluff]

The trick for MT stability for me was SetMemoryMax. Setting that to between 600-800 fixed crashes for me.

The operative phrase here being "for me". Since the entire thing is basically thread-unsafe, poking at anything that might change the way your OS schedules threads may also change the application's behavior and crashiness/non-crashiness. This is of course entirely impossible to predict and depends on so many factors (including things such as your OS version and your hardware configuration) that it's essentially meaningless to try to replicate.

[TheFluff]

Wait, so if you aren't using 2.6 for MT, what are the benefits then of 2.6 over 2.5.8?

It has a few new colorspaces. That's basically the only reason to use it right now.

[Groucho2004]

I have found 2.6 to crash less than 2.5.8.
I have seen statements like this about MT Avisynth several times but it still seems bizarre to me.
It's like hearing something like "I have a car that only breaks down once a week instead of every other day."

[DarkT]

I don't get it. What's good about moare colorspaces? You mean that yv12/yuy/rgb/whatever thingies? If you encode to h264, ain't you bounf to yv12 at the end anywya?

[TheFluff]

I don't get it. What's good about moare colorspaces? You mean that yv12/yuy/rgb/whatever thingies? If you encode to h264, ain't you bounf to yv12 at the end anywya?

2.6 supports YUV 4:4:4 which is like YV12 but without the chroma subsampling. x264 does support encoding that these days, too. Unfortunately, almost nobody actually uses 4:4:4, and the thing people actually do use (high bitdepth YUV, such as 10-bit YV12) is not supported.

[SubJunk]

Wait, so if you aren't using 2.6 for MT, what are the benefits then of 2.6 over 2.5.8?Official changelog between 2.6 and 2.5.8 (http://sourceforge.net/projects/avisynth2/files/AviSynth_Alpha_Releases/AVS%202.6.0%20Alpha%203%20%5B110525%5D/)

[DarkT]

2.6 supports YUV 4:4:4 which is like YV12 but without the chroma subsampling. x264 does support encoding that these days, too.

What's it good for? Does it take care of the color banding or something? Does it require more bitrate? More encoding time?

[SubJunk]

I have seen statements like this about MT Avisynth several times but it still seems bizarre to me.
It's like hearing something like "I have a car that only breaks down once a week instead of every other day."No, I run encoding every day and it never, ever, ever crashes.
The only exception to that statement was 2011.07.19 caused crashing but that was fixed with 2011.09.13

[TheFluff]

What's it good for? Does it take care of the color banding or something? Does it require more bitrate? More encoding time?

It has full chroma resolution. It does not really solve problems with banding the way high bitdepth does, but it does provide "sharper" colors, I guess. It requires a lot more bitrate, takes a lot longer to encode and has quite bad decoder support still.

[Groucho2004]

It has a few new colorspaces. That's basically the only reason to use it right now.

Also, the resizers are much faster (at least on more recent CPUs).

[TheFluff]

Also, the resizers are much faster (at least on more recent CPUs).

That's cool, I suppose, but I've never really felt I've been bottlenecked on the speed of a resizer...

[junh1024]

I use MT all the time. I found 2.6 may be faster, but more unstable than 2.58.

With 2.58, I don't need to use setmemorymax. With 2.6, I baisically need to use setmemorymax(768) all the time.

Also, I only enclose the bottlenecks in my script, like fft3d, hqdering, or mdegrain 2 in MT(""" """)

[DarkT]

I use MT all the time. I found 2.6 may be faster, but more unstable than 2.58.

With 2.58, I don't need to use setmemorymax. With 2.6, I baisically need to use setmemorymax(768) all the time.

Also, I only enclose the bottlenecks in my script, like fft3d, hqdering, or mdegrain 2 in MT(""" """)

So... you are pro using 2.5.8 mt- in 32bit mode I am guessing? or 64?

[Groucho2004]

and if you have to use so many filters that you run out of memory, the source was probably so bad that it wasn't worth encoding anyway.
Good point.

[junh1024]

I usin 2.6 on Windows 7 Professional 32bit.

[DarkT]

Good point.

I wanted to use 64 bits coz I saw that on set's page, where he links to many filters, like, masktools works 25% faster or something, I figured, that's extra speed gains...

Infact, I'd rather use 32 bits, because there's more filters... Some of my favourite filters, no 64bit version :(

[TheFluff]

I use MT all the time. I found 2.6 may be faster, but more unstable than 2.58.

With 2.58, I don't need to use setmemorymax. With 2.6, I baisically need to use setmemorymax(768) all the time.

Also, I only enclose the bottlenecks in my script, like fft3d, hqdering, or mdegrain 2 in MT(""" """)

Do I really have to quote myself from the bottom of last page? Let me try one last time:

It is basically meaningless to make any statements about the relative stability of different multithreaded Avisynth variants (this includes the MT() filter).
A script and environment that is rock solid for one user may crash every time for another, due to the inherent nondeterministic behavior of using a non-threadsafe application in multithreaded mode. There are simply way too many variables that can affect how the application behaves in different scenarios to make any reliable predictions about what might crash and what might not.

[DarkT]

If many people get it to work consistantly... or semi consistantly... I'd say that puts your idea of why it sux outta business tbh...

[SubJunk]

TheFluff, we're just talking about our experiences. If it's so upsetting for you to "let" us talk to eachother then just go away, I assure you no one will care either way.

[TheFluff]

If many people get it to work consistantly... or semi consistantly... I'd say that puts your idea of why it sux outta business tbh...

But you're not asking many people. You're asking two people, and their findings are different.

[DarkT]

i am asking many people. As many as read this particular page. Out of those who do, and chose to reply, I'd say the percentage is quite high :). Nattttttttturally, this ain't no poll - which I think is actualyl quite a good idea... To make a thread where people would contribute their experience with MT... Hmmm...

Edit:

P.s. Greetings to all our friends at DarkHold ;).

[-Vit-]

I know of many people who can basically use MT 24/7 without problem. The recent fixes by SEt have made a marked difference (thanks!). Given that SEt is actively working on this, MT users should start with his latest 2.6 MT from this thread as well as plugins recompiled with his fixes. People overlook this last point: some of the buggy avisynth code is embedded in every plugin, so they need to be recompiled with fixes. A couple of plugin authors have done this, but the only other source for these plugins I know is my own collection of modded plugins for QTGMC (http://www.mediafire.com/?mfs7bp2rprbhp22).

Worth noting that some "threading" crashes are actually just out-of-memory conditions, what with all the threads sharing a 32-bit 2Gb address space. Can't just throw 8 threads at any script and expect it to work, then blame "threading" if it runs out of memory. Use less threads to fix, or maybe SetMemoryMax if you understand what it does. Unfortunately, 64-bit avisynth is truly problematic as there is no active development - no fixes coming there....

Splitting the encode into chunks and running each in a different process works well, but doesn't quite have the same performance and uses up much memory per thread. But interestingly the very best performance, especially for complex scripts+HD is to use this kind of splitting and MT scripts at the same time (e.g. split encode into 4 processes, each running a 4-thread MT). Best use of both CPU and memory (>2Gb). I'm finding this is the best way to use QTGMC + 16-bit processing on HD (and it's not true to say that complex scripts imply you are working with a poor source, the most complex part of QTGMC is source match, designed to precisely capture quality sources. 16-bit processing similar).

[SEt]

-Vit- pretty much summed it all. Except memory limit for 32-bit application is more: 4Gb on x64 OS and can be 3GB on 32-bit OS.

TheFluff, please stop spreading false information.

First of all MT avisynth is not about running non-threadsafe application in multithread mode - it's about making avisynth into threadsafe application. And it's definitely possible. The worst parts here are actually not threading itself, but cache inefficiency and initial design flaw of compiling part of avisynth into every plugin, so plugins that were built with old headers have to be recompiled.

Second: MT avisynth in non-MT mode is not any less stable than non-MT avisynth (the only difference I can remember is MT avisynth requires you to add brackets on function calls while non-MT forgives that error). If you have example where official avisynth 2.6.0 Alpha3 works but my build is not - do post it.


About relation to 2.5.8 MT 32 and 64 bit versions. They are definitely less stable than 2.6. The reason is simple: they are based on old sources. There were many MT problems found and resolved (in 2.6) since then, but no one backported the fixes to 2.5.8.

[asarian]

I could never get the MT versions to work properly. Some people have no problems at all.

For me, the sweet-spot is at 2.5.8, 32-bit, with a script as follows (as example):

SetMTMode(5,4)

FFVideoSource("f:\jobs\test.mkv")

SetMTMode(3)

Crop(0, 140, 0, -140)

MCTemporalDenoise(settings="medium", stabilize=true)
GradFun2DBmod(thr=1.4,thrC=1.0,mode=0,str=1.2,strC=0.0,temp=50,adapt=64,mask=false,show=false)

Not the absolute fastest way, but rock stable! The MT"" call is a no go for me: 7/10 times the process crashes somewhere (as luck would have it, usually when the job is 98% done).

The 64-bit AviSynth MT builds are a nightmare for me: extreme memory leaks, and almost instant crashes (if it hangs in there for more than 5 mins, it's much). Kudos if you got it to work for you, but I gave up on it.

I haven't tried 2.6 yet (is it even downwards compatible?).

[TheFluff]

Second: MT avisynth in non-MT mode is not any less stable than non-MT avisynth
About relation to 2.5.8 MT 32 and 64 bit versions. They are definitely less stable than 2.6.

these two statements seem to be contradictory

Furthermore, the problem with MT is not limited to the non-threadsafeness of the entire thing; it's also a matter of introducing a whole lot of new and interesting bugs while in the process of poking at basically everything in the entire application.

[asarian]

these two statements seem to be contradictory

How so?

Originally Posted by SEt
Second: MT avisynth in non-MT mode is not any less stable than non-MT avisynth

All this says is, that if you use MT AviSynth in a single-threaded manner, it will be no more unstable than its non-MT counterpart (which is true).

Originally Posted by SEt
About relation to 2.5.8 MT 32 and 64 bit versions. They are definitely less stable than 2.6.

And this just says both 32- and 64-bit versions of 2.5.8 MT are less stable than their 2.6 counterparts.

I don't see the contradictory information.

[DarkT]

*shrugs* If we take the testimonies of some of the posters, regarding consistently working without crashes with MT, then it is certainly a worthy try to see if person X can be one of such lucky ones. You could argue about what's better, or what works more consistently - but for me? The fact that SOME people can get it to work Consistently, is quite enought as a green light to go ahead and see if I can be one of the chosen lucky ones *grins*.

Anyway, thanks for the help guys, I think I am pretty much closed on the direction I'm going to try and go in ;).

I think that there needs to be a "come and test avisynth 2.6a3 by Set" thread, where people could give more feedback about what they try and what works and what not... BAsically, right now, some might try it coz it's there, but I think perhaps more should be encouraged to try it... Perhapes it could speed up the discovery of whatever bugs there are, etc... I dunno.

Anyyyyway - thanks for the info guys ;). I'll open a thread with my own experience once I get on it to let you all know how it went.

[SEt]

Furthermore, the problem with MT is not limited to the non-threadsafeness of the entire thing; it's also a matter of introducing a whole lot of new and interesting bugs while in the process of poking at basically everything in the entire application.
How about any proof for any of your claims?

DarkT, this is actually the thread you are speaking about. But I agree that there is not enough information for regular people to understand what is going on. MT avisynth now is not about "are you lucky enough to run it?" but rather "do you understand enough to use it correctly" (though luck does help ^_^). I'll try to add some information to first post.

[DarkT]

DarkT, this is actually the thread you are speaking about. But I agree that there is not enough information for regular people to understand what is going on. MT avisynth now is not about "are you lucky enough to run it?" but rather "do you understand enough to use it correctly" (though luck does help ^_^). I'll try to add some information to first post.

Thing is, this thread is in Avisynth Development... I see "development" I run for the hills *Grins* I don't want nuclear testing done on MY computer... *grins* ya know?

Basically, If you could JUST change name from alpha to beta maybe? *grins* or RC? *grins*

Some more info would be GREATLY appritiated... Like what the numbers mean in the SetMTMode or whatever the command meant... I tried to find that info, couldn't though, but I did get some help from one of the posters in this thread, he had sent me a cool beginner script which I will be trying out soon.

Anyway, some examples would also help, like, filter X needs this, and filter Y needs that... You know? Because... To get the average "noob"(me) involved, you do need to make things... Ummm simpler/easier ;). And I think there's a great value in getting more people involved, in terms of feedback, ya know? You basically need more guinea pigs :).

[SEt]

Edited first post. Something else should be mentioned there?

Removed frightening words "alpha" etc. ^_^ Not sure if it should be moved to "Avisynth usage" forum - I'd like to have here more constructive replays than "It doesn't work for me!!" (being here doesn't completely help, though).

[-Vit-]

It would be good to have a master list of plugins that are known to use your fixed header file (given that not many people can/will check the source code).
Then encourage people to recompile and post any other plugins with the new avisynth.h header onto this thread.

[SEt]

I'm not going to maintain such list. But I can link to other thread if someone is willing to compile and update the list. ;)

[Gavino]

Second: MT avisynth in non-MT mode is not any less stable than non-MT avisynth (the only difference I can remember is MT avisynth requires you to add brackets on function calls while non-MT forgives that error).
Omitting the brackets in a function call is not an error, eg from http://avisynth.org/mediawiki/Grammar:
If no arguments are being passed to the function, you can also make the function call without parentheses, e.g. FilterName. The primary reason for this is to retain compatibility with old scripts. However, it's sometimes convenient to leave off the parentheses when there's no possibility of confusion.

MT Avisynth will likely crash if you do this (see this post), and even when running single-threaded it will fail to add a cache to the function call, hitting performance. Could this problem not be fixed (or at least produce a compile-time error)?

As a separate point, from the first post of this thread:If you are using it with ffdshow for realtime processing - at the very end add
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last

Won't GetMTMode(false) always return 1 following SetMTMode(1) ?
Or is the idea to detect the case where you are running single-threaded (no earlier SetMTMode in the script)? But in that case you could just leave out the extra code. :confused:

[SEt]

I'm a C programmer and when calling a function brackets are required in my book (this also improves readability as you don't need to guess each time is it a variable or function call). Compatibility with some abstract old scripts is not a good reason - correcting scripts is trivial. As for compilation error - maybe, not sure I want to touch the parsing code.

It's very clever code actually. If you rem first SetMTMode in script the rest are automatically invalidated (being after ffdshow_source() call). So last line allows switching between MT and non-MT modes with changing just one line in the beginning of script.

[Gavino]

I'm a C programmer and when calling a function brackets are required in my book (this also improves readability as you don't need to guess each time is it a variable or function call).
True, but the Avisynth language isn't C, and since the language spec permits omitting brackets, this must be regarded as a deficiency in MT Avisynth, especially when it causes a crash whose origin is far from obvious to the user.

[SEt]

I agree that it should not result in undefined behavior, but I rather forbid it than allow for the stated readability improvement.

[Gser]

All
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last
does for me in FFDShow is crash it. It works fine without it though.

[cweb]

The trick for MT stability for me was SetMemoryMax. Setting that to between 600-800 fixed crashes for me.

I never had problems and I've had that line in my scripts for quite a while.. hmmm, could be why my MT was always stable..

[levi]

I disagree with the unstable categorization of MT. Once you get a script working, it's not going to stop working or give different results. That would be unstable. Getting everything set up correctly can be challenging, but once you do it will work fine and continue to work fine.

[DarkT]

Been trying for an hour or two now to get the thing installed. It won't go.
I have windows 7 64 bits on.
I had 2.5.8 installed, and on top of that the Set version which was MT installed. It worked. I put the avisynth.dll in syswow64, and then I got in virtualDub 64bits an error: AviImport Filter error: (unknown) (80040154).
I uninstalled avisynth, and used the uninstaller provided by the Set version. I rebooted.
I installed avisynth 2.5.8 official. Checked that it works, it did(using AVSPmod - 32bit program, in the script, at this point in time I was using 32 bit directshowsource to load the video - that's all there was). The I have downloaded the 2.6 MT version. I've placed it in syswow64 dir. I have again checked with AVSPMod - using 32bit directshowsource that all was working, it did. I did version() and it said 2.6 or something like that. It was no longer the 2.5.8 ver. Then I have modified the script to load DirectShowSource.dll which came with the modified Set version of avisynth - the DSS which is 64bits. VDub64 throws that error. Then I tried the AVSS64.dll, and DSS2() as a loading command. nothing, no matter what I write in the script, VDub64 dies. AND - now AVSPMod won't load also... perhapes coz I deleted the DirectShowSource.dll(32bit) from the plugins folder.

Anyway, I also checked that MSVCRT71 and the MSCVRPT or whatever are all in place, they weren't but I got them and put them where they were needed.

It still won't work ;(. Ideas?

[um3k]

Simple, you can't use 64 bit plugins with 32 bit avisynth. Stick to the 32 bit plugins if you're using 2.6.