Avisynth 2.6 MT

[SEt]

Current version 2015.02.20: https://www.dropbox.com/s/dckxoowjlzwku1s/avisynth_20150220.7z

Previous version 2013.09.28: https://www.dropbox.com/s/6f56nvqvxrde22q/avisynth_20130928.7z


Note: Starting from version 2012.04.03 plugin's folder is searched first for its dll dependences. You can now move some things from system32 folder to plugins folder, but check that there are no unexpected old dll versions in plugins folder in case you see strange errors.

Here is my builds of Avisynth 2.6 with several (mostly MT) fixes. MT mode stability is way better than Avisynth 2.5.8, general stability - the same as official Avisynth 2.6 (which is again better than 2.5.8 due to many bugfixes since then).

How to use it correctly

1. Installation
Install any recent official version of Avisynth.
Replace avisynth.dll in your system32 (on 32-bit OS) or syswow64 (on 64-bit OS) folder with the one from this thread.

2. Usage in scripts
Understanding different MT modes would help a lot, but most things are going to work ok with script like this one:

SetMemoryMax(512)
SetMTMode(3)
source_filter()
SetMTMode(2)
processing_filters()

If you are using it with ffdshow for realtime processing - at the very end add
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last

and ensure you have "Buffer back/ahead" enabled with enough number for "ahead" ("back" usually can be 0 with no problems). Use 1-2 more than it sets with "Use Current" button while playback is running.
Note: current ffdshow has bug in interaction with avisynth that results in often crashes on stop of MT scripts. It does not affect playback itself.

3. In case something crashes or works wrong
Avisynth is desinged in such a way that part of it is compiled into each plugin. That means if you experience crashes first of all you should check that all used plugins are built with recent headers as old ones are not compatible with multithreading.

The other thing is memory usage. Being 32-bit Avisynth has certain memory limits.
On 32-bit OS: you are likely limited to 2GB per process. With /3GB windows boot option it can be increased to 3GB but it's not problem free.
On 64-bit OS: it's 4GB per 32-bit application, so there are benefits of using 64-bit OS even if 32-bit applications are all you need.
Note: exe file of application must be correctly compiled to use mentioned above 3GB or 4GB. Not an avisynth or its plugins problem, but worth noting.

If you see (in Task Manager) that application crashes with memory usage near 2/3/4 GB (depending on your environment) - you are facing cache inefficiency of MT Avisynth. Try increasing or decreasing SetMemoryMax, fewer number of threads (second parameter in first SetMTMode). No guaranteed results here.

Plugins/scripts that depend on sequential frame requests are not going to work correctly with current MT mode. Not really MT problem, as they also won't work correctly in non-MT mode when frame requests from next filter are not sequential.

If you are sure mentioned above 3 cases don't apply to your crash/wrong result - post here a way to reproduce it.

4. Short guide to SetMTMode
SetMTMode is the command that switches Avisynth into MT mode. Without it MT Avisynth works like the usual one.
Useful MT modes:
1 - one filter instance per call in script, no guarding from Avisynth, only for filters that are designed for this mode, like distributor().
3 - one filter instance per call in script, Avisynth guards requests for output. The mode to use with source filters. (Mode 5 will do for them too, but is overkill and should be avoided.)
2 - each call in script produces N (number of threads) instances of filter. Use for the rest.

When using MT mode with filters that have ability to spawn several internal processing threads - be sure to set such filters to use only 1 thread. Otherwise for example with 8-thread system MT mode will spawn 8 threads with once instance of such filter per thread (in MT mode 2), each filter instance will spawn 8 threads. End result: 64 threads (ouch).


Older versions:
2013.03.09: https://www.dropbox.com/s/xhqggxamegia420/avisynth_20130309.7z
2013.02.20: https://www.dropbox.com/s/x58c1ofdvc31dxi/avisynth_20130220.7z
2012.08.28: http://www.mediafire.com/file/4dm34kc7tug7rrk/avisynth.7z
2012.05.16: http://www.mediafire.com/file/20gn1c2s45la14w/avisynth.7z
2012.04.03: http://www.mediafire.com/file/rlzn2kkrcad3j71/avisynth.7z
2012.03.31: http://www.mediafire.com/file/k9g67qbe6tyo8xa/avisynth.7z
2011.09.13: http://www.mediafire.com/file/3gd3851mb9o9wq7/avisynth.7z
2011.07.19: http://www.mediafire.com/file/9jk0cb9r56myduz/avisynth.7z
2011.07.14: http://www.mediafire.com/file/358wvsb4bngqjgi/avisynth.7z
2009.09.19: http://www.mediafire.com/file/2dz4y0wztzz/avisynth.7z
2009.08.15: http://www.mediafire.com/file/zjaelwemjii/avisynth_26.7z
2009.08.08: http://www.mediafire.com/file/hdvgzm5xdmn/avisynth_26.7z

[Keiyakusha]

Thanks. I've used Kemuri's build for some time and it works stable as last stable 2.5.8. I will try your build now.

Edit: Next time we'll see working avisynth 3.0 build from SEt, right? ;)

[Fizick]

SEt, it is interesting, but please be more respective to GPL

[SEt]

I hope you don't mean including gpl.txt everywhere. As for sources - i'm respective and you can get them if you ask. (Well, actually i asked Wilbert and waiting for cvs access now.)

[TheFluff]

but you have to make a written offer to provide the sources! and it must be valid for at least 3 years! you're gonna get sued!!111!

late irony edit: http://forum.doom9.org/showthread.php?t=32863

[10L23r]

Is mediafire dead or something?? I've been stuck at "Processing download request..." for about five minutes :(

[Fizick]

Here is download link to source code I have got from SEt


http://www.mediafire.com/file/mmlzutjueuv/AviSynth26.7z

Changes:
- MT related fixes.
- General code cleanup (for scope, const).
- Project splitted and settings cleaned up. Converting from VS6 is a great way to get strange settings.
- Converted .asm to yasm. If we ever hope to be portable - it'll most likely be yasm for assembler.


First external look: changes to current CVS are rather massive! IMHO they should not be committed at once. but by portions for more easy revision (tracking) of changes.

More comments later.

[IanB]

Yes, "changes to current CVS are rather massive!", there are a lot of "white" changes like adding "const" and "extern" declaration to things that probably should have had them from the beginning. Arbitrary changes like using "long long" for "__int64" and fiddling with the for loops are not transparent as VC6 (yes we still support that) do not support them. Getting the "interlocked*" logic in many places is relevant, I had already found a lot of these but it help to have independent review. The code in cacheMT.cpp is very out of date (circa 2.5.5) and does not conform to 2.5.8 expectations for a Cache:: module, medium term this module will be deleted and the code subsumed back into the main cache code, it was only ever a place holder while TSP worked on the code.

[Dark Shikari]

but you have to make a written offer to provide the sources! and it must be valid for at least 3 years! you're gonna get sued!!111!Or just include the source code with the binary... :rolleyes:

[lych_necross]

Dark Shikari is right. Just include the sources with the binary.

[SEt]

I think there is absolutely no reason for supporting VC6 compiler and flooding the code with
stupid compiler forgets to save ebx!!
Damn compiler does not correctly cast to intermediate results to float
and hacks to avoid it. Code should be moved towards standard C/C++ and maybe GCC compatibility. Of course supporting the VC6 build environment is perfectly fine.

Cache is the real problem - i'm getting bad memory leaks with heavy threading and trying to fix it now. Changes submitted are intermediate so it won't crash and disturb testing other things.

[TheFluff]

using "long long" for "__int64"

int64_t

e: isn't a major version bump a very good occasion to drop support for an ancient and buggy compiler anyway, I mean if you don't drop it now you'll still be sitting here supporting it in 2020 or so judging by how often avisynth gets major version bumps

[SEt]

Fixed minor threading bug and major memory leak:
--- avisynth.cpp 2009-08-08 22:20:34.550375000 +0400
+++ avisynth.cpp 2009-08-13 15:12:07.305375000 +0400
@@ -144,8 +144,9 @@
return &i->vf;
LinkedVideoFrame* result = (LinkedVideoFrame*)::operator new(sizeof(LinkedVideoFrame));
result->vf.refcount=1;
- result->next = g_Bin->g_VideoFrame_recycle_bin;
- result->next = (LinkedVideoFrame*)InterlockedExchangePointer((long*)&g_Bin->g_VideoFrame_recycle_bin,result);//g_VideoFrame_recycle_bin is assigned twice to result->next just in case another thread reads result->next before the last assignment
+ do
+ result->next = g_Bin->g_VideoFrame_recycle_bin;
+ while (InterlockedCompareExchangePointer((volatile PVOID*)&g_Bin->g_VideoFrame_recycle_bin, result, result->next) != result->next);
return &result->vf;
}

@@ -1639,7 +1640,7 @@
InterlockedIncrement(&i->sequence_number); // Signal to the cache that the vfb has been stolen
return i;
}
- if (i->GetDataSize() > size) {
+ if (c > Cache::PC_Nil && i->GetDataSize() > size) {
// Remember the smallest VFB that is bigger than our size
if ((j == 0) || (i->GetDataSize() < j->GetDataSize()))
{
@@ -1659,7 +1660,7 @@
if (!CacheHead) break; // cache state will not change in next loop iterations

// Plan C: Steal the oldest, smallest free buffer that is greater in size
- if (j && c > Cache::PC_Nil) {
+ if (j) {
++g_Mem_stats.PlanC;
InterlockedIncrement(&j->sequence_number); // Signal to the cache that the vfb has been stolen
return j;

[IanB]

@SEt,

Thanks, you must be psychic I noticed the 2nd issue while reading your other big change bundle. It is still not quite right. In the MT case CacheHead could be NUL because CacheMT doesn't implement the new poke cache stuff yet, so you exit the loop without actually using "the smallest VFB that is bigger than our size". that's the trouble with quickly retro fitting code from the main branch. ;)

As for the 1st bit with g_Bin yeah it's stuffed. I am not happy with that whole concept, so ..... :cool:

[SEt]

No, the behavior is intended - i don't like plan C and would rather disable it. In case of memory starvation this "memory leak" can be quite unpleasant and more dangerous than memory fragmentation that it tries to prevent.

Let's see what can happen after we give bigger buffer than required (and usually it'll be 2..4+x bigger):
1) Big buffer won't be required in future - we are wasting memory, bad.
2) Big buffer will be required before allocated smaller frame is freed - we'll likely have to allocate another big buffer, wasting memory again, bad.
3) Big buffer will be required after smaller frame is freed, but before another smaller frame uses it - good, but i'd say unlikely.

[Keiyakusha]

Sorry if this thread not suits for my question...

So here (http://www.petaimg.com/u95/819test.png) is my image.
Here is my script:
Imagesource("test.png")
Converttoyv24().SwapUV()
And here is what I see when I open it in VirtualDub:
http://www.petaimg.com/u143/96123.png

So the question is, why chroma is swapped so I need to add SwapUV()? Is this some bug in VDub? Because if I add Converttorgb() to the end of the script - chroma returns to its place...
But more importantly why I'm getting this "extra" chroma or whatever is that? Am I doing something wrong?

EDIT: I'm using SEt's build from 1st post.

[SEt]

Color format conversions are currently broken (at least their dynamically compiled part). Also note that i haven't rebuilt the dll with posted bugfixes yet.

[BigDid]

Hi SET,

Here is the 1st official release of Avisynth 2.6. The main pupose of this release is to confirm compatibility with version 2.5.8. For scripts using only 2.5.8 features the results should be identical.
...
The next release will contain Sh0dans latest FastWire updates and his SSE3+ code enhancements.
...
and
...
Time is precious so I am skipping the frills for expedience sake. I am also restricting my participation in this forum, don't feel offended if I appear to be ignoring you, I am applying self discipline and not responding to anybody ;) ( I am skimming the posts every few days for any active items). This way I actually have time to work on the code.
...

If I read IanB comments well, he will first focus on compatibility... so I believe MT in 2.6 is under the stack.

If true, it will be up to you (if you have time to do so) to update your 2.6MT build. Can it be expected?

I do not want to put pressure on any developper (IanB or you) just have a roadmap on what to expect or not.

:thanks:

Did

[a451guy451]

I wanted to just note briefly, that I was testing the above build for a while and it was unstable during transcodes on both quad and 8 core systems. I detailed it briefly in the above "AviSynth 2.6.0 Alpha1 [August 20th]" thread. Tested it exhaustively on a dual core though, and it was great; no issues. thanks for the continued work on it!

[SEt]

Of course build 2009.08.08 is unstable (at least due to described in patch bugs). As this thread still interest someone, i've added newer build with posted patch applied (nothing else changed) - it's way more stable, but be sure to set enough SetMemoryMax to avoid late crash due to address space fragmentation.

BigDid, i have several MT-related ideas to write, but i don't want to maintain my own version of avisynth 2.6 unrelated to official. I've asked for official trunk/stable svn repositories but there was no reaction yet.

[BigDid]

Hi,

Thanks for the update. I have tried it, but the colors problems are forbidding a regular usage.
That could be something to put on top of the stack :)

I have followed (but note always understood) the "main/single-threaded" 2.6 thread and have seen these exchanges.
I will not put pressure on either side but if I can be of any help for the MT side ;) let me know.

Did

[levi]

Hey guys, it seems my problem is related to my quad core CPU as I have exactly the same symptoms as mentioned here: http://forum.doom9.org/showthread.php?p=1319483#post1319483

Switched to several different SetMTmodes, but it eventually stalls (CPU use & fps in vdub drop to zero) It starts out fast though.

So I was going to try the other modes...


MTSource(""" MPEG2Source("D:\my.d2v") """)


but i get:
Avisynth open failure:
Script error: Invalid arguments to function "MTSource"

I checked the syntax a few times but I can't find the error

[Gavino]

I checked the syntax a few times but I can't find the error
The syntax looks right. Perhaps it can't find MPEG2Source.
Do you have DGDecode.dll in your plugins folder?

[levi]

Yes, dgdecode is in plugins - it works fine with
MPEG2Source("D:\my.d2v")

MTSource(""" AviSource("pass2.avi") """)
gives the same error, but works fine when used normally

MT 0.7 (includes modified avisynth 2.5.7.5) works fine

:<

[SEt]

There actually was one more threading fix i haven't bothered to post due to expected rewrite of cache/MTMode in 2.6 that should solve the problem with hanging. See the first post for the rebuilt dll.

--- cacheMT.cpp 2009-08-08 22:03:34.034750000 +0400
+++ cacheMT.cpp 2009-10-05 15:15:03.734375000 +0400
@@ -430,7 +430,9 @@
InterlockedDecrement(&e_generated_refcount[i->e_generated_index]);
++g_Cache_stats.vfb_found;
LockVFB(i, env); // BuildVideoFrame expect the VFB to be locked
+ EnterCriticalSection(&cs_cache);
PVideoFrame retval=BuildVideoFrame(i, n);
+ LeaveCriticalSection(&cs_cache);
InterlockedDecrement((long*)&retval->refcount);
return retval;
}

[a451guy451]

Thanks for posting the new build. I'm looking forward to playing with it on a 4 and 8 core machine today...

[levi]

I'm curious to hear your results. I did not see any improvements in the hangs / crashes.

[a451guy451]

I'm still running my first test on my 8-core. It's a really complicated script that usually gives me 3-4fps, and it's running 12-14fps with this new DLL. It's been rock solid for the last 2h 30min, and I have another hour to go. BUT, when I was using the previous build it would hang itself in less than 5 minutes. I expect this will complete just fine. After I finish the 8-core test, I'll hop onto my quad core box and see if I can duplicate the results (I see no reason why it would give me any problems).

Sorry you're not seeing any improvement though. I'm only using Setmtmode(2,0), so perhaps you should give that a try if you haven't already.

VERY promising though. Thanks again SEt!

[a451guy451]

8-core test complete, with no issues (yay). I'll try to post the results of my test on a quad core tomorrow.

[7ekno]

8-core test complete, with no issues (yay). I'll try to post the results of my test on a quad core tomorrow.

Was that with the link called "Version 2009.09.19" ?

If that's the case, I will retry it later too ...

7ek

[a451guy451]

Yes it was.

[a451guy451]

Quad core works as well. Really, almost as fast as the 8-core, but I think the explanation for that is that I'm hitting a bottleneck elsewhere in my system. Cool deal either way.

[levi]

set, does it matter which version of devil.dll is used?

[BigDid]

Color format conversions are currently broken (at least their dynamically compiled part). Also note that i haven't rebuilt the dll with posted bugfixes yet.
Hi,

The last 2.6-MT from SET is really stable for me, but I am still getting some horizontal colors bands; don't know if related to the above quote though?

I could try to pinpoint what script/filter does trigger these bands if interested...

Did

[SEt]

I can't say anything about devil.dll. And of course if you want something fixed you need to post a way how to reproduce the bug.

[BigDid]

... And of course if you want something fixed you need to post a way how to reproduce the bug.
Hi,

Here are the infos I can give you:

Having a mpegsouce or a directshowsource,

Source being this test movie:
http://www.mediafire.com/file/r2rnvnnn2y1/bride1.m2v
from this thread:
http://forum.doom9.org/showthread.php?p=1328915#post1328915

autocrop() gives some colors problems (patchs of blue and red plus a thin horizontal green line at the bottom) with a resulting resolution of 700*384

Crop(movie,0,46,0,-50) is OK

mt("DeGrainMedian(mode=5)") is Ok

lanczosresize(720,432) is ok (as is autocrop +lanczos)

BlindPP(cpu2="xxxxoo", quant=16) -deblocking- or
BlindPP(cpu2="ooooxx", quant=5) -deringing- gives a green horizontal thick line at the bottom quite similar to the one in the 2.6 thread:
http://forum.doom9.org/showthread.php?p=1337438#post1337438
Blindpp() being a separate use of the cpu function from dgdecode, this could be expected.

Complex scripts sharpening and/or denoising with or whitout MT are Ok:
lsfmod()or
*mt("""lsfmod(strength=150, soothe=true,keep=10,edgemode=1)""")
MCtemporaldenoise() or
*mt("""MCTemporalDenoise(settings="medium",sigma=20,sharp=true,strength=200,protect=true,flat=true)""")
Softsharpen() or
*mt("""softsharpen(preset="film", net=14,debrT=1,debr=0, flou=16,derin=0)""",2,4,splitvertical=true)

* are examples only; parameters may not be appropriate or only if associated with other filters.

So I hope that the solution(s) from the main 2.6 thread can apply here. If not, someone (not me) will have to ask an update of dgdecode().

[IanB]

@BigDid,

It ain'i rocket science put a SetPlanarLegacyAlignment (True) after the BlindPP() line and see if it hides the problem.

(Hint: Having looked at the code, just halving the pitch, as is done here, is going to cause this problem.)

[levi]

BigDid, how are you doing MT("...") syntax with 2.6? where did you get 2.6 MT.DLL? I only saw avisynth.dll in those 7zips set posted.

[BigDid]

@BigDid,

It ain'i rocket science put a SetPlanarLegacyAlignment (True) after the BlindPP() line and see if it hides the problem...
Hi IanB,

Works fine. As I use the deblock part of blind PP before denoising/sharpening and the dering part after, I have tested that 1 statement of "SetPlanarLegacyAlignment (True)" at the end of the script will work fine also.
Thanks alot, good to have some constructive answers in these difficult times.

BigDid, how are you doing MT("...") syntax with 2.6?
Same as you are. Some syntax works some do not.
Basics are on the wiki page:
http://avisynth.org/mediawiki/MT_support_page
summarizing the basic MT 0.7 thread:
http://forum.doom9.org/showthread.php?t=94996

You have to juggle with triple quotes, overlap and/or splitvertical. I have updated the complex scripts in the above post as examples.


where did you get 2.6 MT.DLL? I only saw avisynth.dll in those 7zips set posted.
Same as yours:
http://forum.doom9.org/showthread.php?p=1331510#post1331510
http://www.avisynth.org/tsp/MT_07.zip

[IanB]

...
Works fine. As I use the deblock part of blind PP before denoising/sharpening and the dering part after, I have tested that 1 statement of "SetPlanarLegacyAlignment (True)" at the end of the script will work fine also.
It is not really fine to put SetPlanarLegacyAlignment(True) at the end of the script. Doing so may force any SSE2 code to use the slower unaligned memory access code path for YV12 chroma planes or poorly written filters may even crash. It also only works for YV12.

For filters that make illegal assumptions about pitch you should put SetPlanarLegacyAlignment(False) right before the problem filters statement and SetPlanarLegacyAlignment(True) right after it. i.e.Mpeg2Source("fred.d2v, cpu=1) # Bad planar code!
SetPlanarLegacyAlignment(True) # Do slow legacy alignment
...
SetPlanarLegacyAlignment(False) # Restore fast alignment
BlindPP(cpu2="xxxxoo", quant=16) # Bad planar code!
SetPlanarLegacyAlignment(True) # Do slow legacy alignment
...

[Delerue]

Just to say that the last version worked flawlessly for me. I'm using MVTools with a MFlowFPS script and with SetMTMode enabled. Performance is great and no artifacts at all.

Thanks, SEt. ;)

[levi]

As you can see in the example link (you probably have to download the image in order to see it), there are some artifacts not present in the source that turn up in the xvid encode. Please let me know if you have any idea what's wrong. I have run into this a couple of times now with different sources...


SetMemoryMax(512)
SetMTmode(3,0)
mpeg2source("AE.d2v")
SetMTmode(2)
TomsMoComp(1,5,1)
crop(268,8,1404,1072)
LanczosResize(576,432)

Using:
http://members.optusnet.com.au/squid_80/xvid_encraw.zip

xvid_encraw.exe -max_key_interval 250 -vhqmode 4 -bvhq -progress 250 -imax 4 -pmax 7 -bmax 10 -qtype 0 -threads 3

Example:
http://www.mediafire.com/i/?zmmmzttzlmz

Source:
http://www.mediafire.com/?zllfwwinm4m

Encode:
http://www.mediafire.com/?zllfwwinm4m

[hxhxd]

@levilevi,
avisynth 2.6 has no mt feature. you should ask in SEt's thread.

[levi]

what & what? :p

This is set's thread & he has posted links to multi-threaded 2.6 in the first post.

[Lynx Pardinus]

In the past I have experienced the same artifacts when using TomsMoComp in MT Avisynth 2.57.

Try using a different deinterlacer or try moving the SetMTMode after the TomsMoComp.

[a451guy451]

This is a small bug, but Histogram() seems to be broken in this build (I'm using histogram("level") specifically). It still shows up, but the waveform is nearly black (you can actually see it a little, but just barely and it doesn't appear to be accurate). It's not a huge deal, because I can always use one of the Virtualdub filters. I just wanted to bring it up because I've used this build extensively now and it's the only issue I've come across.

[hxhxd]

what & what? :p

This is set's thread & he has posted links to multi-threaded 2.6 in the first post.

Oh, sorry, I've make a big mistake. :rolleyes:

[ChaosKing]

I think i found a bug in Blackmanresize().
The bug only occurs if I set the taps parameter in the resize function:


subtitle("blackmanresize(640,480,50)")
blackmanresize(640,480,50)
crop(0,0,-0,-400)

http://chaosking.de/zeugs/blackman.png
http://chaosking.de/zeugs/blackman10.png
http://chaosking.de/zeugs/blackman50.png

[Gavino]

I think i found a bug in Blackmanresize().
The bug only occurs if I set the taps parameter in the resize function:
...
blackmanresize(640,480,50)
The third parameter is src_left, not taps, so you are shifting the image.
Use blackmanresize(640,480,taps=50).

[ChaosKing]

hmm ok, but Mediawiki says:

Basic usage:
BlackmanResize(clip clip, int target_width, int target_height, int taps)

so shouldn't be the third parameter = the taps parameter in my case??