Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 8th August 2009, 20:37   #1  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Avisynth 2.6 MT

Current version 2015.02.20: https://www.dropbox.com/s/dckxoowjlz...th_20150220.7z

Previous version 2013.09.28: https://www.dropbox.com/s/6f56nvqvxr...th_20130928.7z


Note: Starting from version 2012.04.03 plugin's folder is searched first for its dll dependences. You can now move some things from system32 folder to plugins folder, but check that there are no unexpected old dll versions in plugins folder in case you see strange errors.

Here is my builds of Avisynth 2.6 with several (mostly MT) fixes. MT mode stability is way better than Avisynth 2.5.8, general stability - the same as official Avisynth 2.6 (which is again better than 2.5.8 due to many bugfixes since then).

How to use it correctly

1. Installation
Install any recent official version of Avisynth.
Replace avisynth.dll in your system32 (on 32-bit OS) or syswow64 (on 64-bit OS) folder with the one from this thread.

2. Usage in scripts
Understanding different MT modes would help a lot, but most things are going to work ok with script like this one:
Code:
SetMemoryMax(512)
SetMTMode(3)
source_filter()
SetMTMode(2)
processing_filters()
If you are using it with ffdshow for realtime processing - at the very end add
Code:
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last
and ensure you have "Buffer back/ahead" enabled with enough number for "ahead" ("back" usually can be 0 with no problems). Use 1-2 more than it sets with "Use Current" button while playback is running.
Note: current ffdshow has bug in interaction with avisynth that results in often crashes on stop of MT scripts. It does not affect playback itself.

3. In case something crashes or works wrong
Avisynth is desinged in such a way that part of it is compiled into each plugin. That means if you experience crashes first of all you should check that all used plugins are built with recent headers as old ones are not compatible with multithreading.

The other thing is memory usage. Being 32-bit Avisynth has certain memory limits.
On 32-bit OS: you are likely limited to 2GB per process. With /3GB windows boot option it can be increased to 3GB but it's not problem free.
On 64-bit OS: it's 4GB per 32-bit application, so there are benefits of using 64-bit OS even if 32-bit applications are all you need.
Note: exe file of application must be correctly compiled to use mentioned above 3GB or 4GB. Not an avisynth or its plugins problem, but worth noting.

If you see (in Task Manager) that application crashes with memory usage near 2/3/4 GB (depending on your environment) - you are facing cache inefficiency of MT Avisynth. Try increasing or decreasing SetMemoryMax, fewer number of threads (second parameter in first SetMTMode). No guaranteed results here.

Plugins/scripts that depend on sequential frame requests are not going to work correctly with current MT mode. Not really MT problem, as they also won't work correctly in non-MT mode when frame requests from next filter are not sequential.

If you are sure mentioned above 3 cases don't apply to your crash/wrong result - post here a way to reproduce it.

4. Short guide to SetMTMode
SetMTMode is the command that switches Avisynth into MT mode. Without it MT Avisynth works like the usual one.
Useful MT modes:
1 - one filter instance per call in script, no guarding from Avisynth, only for filters that are designed for this mode, like distributor().
3 - one filter instance per call in script, Avisynth guards requests for output. The mode to use with source filters. (Mode 5 will do for them too, but is overkill and should be avoided.)
2 - each call in script produces N (number of threads) instances of filter. Use for the rest.

When using MT mode with filters that have ability to spawn several internal processing threads - be sure to set such filters to use only 1 thread. Otherwise for example with 8-thread system MT mode will spawn 8 threads with once instance of such filter per thread (in MT mode 2), each filter instance will spawn 8 threads. End result: 64 threads (ouch).


Older versions:
2013.03.09: https://www.dropbox.com/s/xhqggxameg...th_20130309.7z
2013.02.20: https://www.dropbox.com/s/x58c1ofdvc...th_20130220.7z
2012.08.28: http://www.mediafire.com/file/4dm34k...rk/avisynth.7z
2012.05.16: http://www.mediafire.com/file/20gn1c...4w/avisynth.7z
2012.04.03: http://www.mediafire.com/file/rlzn2k...71/avisynth.7z
2012.03.31: http://www.mediafire.com/file/k9g67q...xa/avisynth.7z
2011.09.13: http://www.mediafire.com/file/3gd385...q7/avisynth.7z
2011.07.19: http://www.mediafire.com/file/9jk0cb...uz/avisynth.7z
2011.07.14: http://www.mediafire.com/file/358wvs...gi/avisynth.7z
2009.09.19: http://www.mediafire.com/file/2dz4y0wztzz/avisynth.7z
2009.08.15: http://www.mediafire.com/file/zjaelw...avisynth_26.7z
2009.08.08: http://www.mediafire.com/file/hdvgzm...avisynth_26.7z
Attached Files
File Type: 7z diff.7z (29.8 KB, 3209 views)

Last edited by SEt; 20th February 2015 at 01:21.
SEt is offline   Reply With Quote
Old 9th August 2009, 00:15   #2  |  Link
Keiyakusha
契約者
 
Keiyakusha's Avatar
 
Join Date: Jun 2008
Posts: 1,576
Thanks. I've used Kemuri's build for some time and it works stable as last stable 2.5.8. I will try your build now.

Edit: Next time we'll see working avisynth 3.0 build from SEt, right?

Last edited by Keiyakusha; 9th August 2009 at 00:18.
Keiyakusha is offline   Reply With Quote
Old 9th August 2009, 06:02   #3  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
SEt, it is interesting, but please be more respective to GPL
Fizick is offline   Reply With Quote
Old 9th August 2009, 07:55   #4  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
I hope you don't mean including gpl.txt everywhere. As for sources - i'm respective and you can get them if you ask. (Well, actually i asked Wilbert and waiting for cvs access now.)
SEt is offline   Reply With Quote
Old 9th August 2009, 16:41   #5  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
but you have to make a written offer to provide the sources! and it must be valid for at least 3 years! you're gonna get sued!!111!

late irony edit: http://forum.doom9.org/showthread.php?t=32863

Last edited by TheFluff; 10th August 2009 at 02:08.
TheFluff is offline   Reply With Quote
Old 10th August 2009, 05:37   #6  |  Link
10L23r
Registered User
 
Join Date: Apr 2009
Posts: 122
Is mediafire dead or something?? I've been stuck at "Processing download request..." for about five minutes
10L23r is offline   Reply With Quote
Old 10th August 2009, 18:07   #7  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Here is download link to source code I have got from SEt

Quote:
Originally Posted by SEt
http://www.mediafire.com/file/mmlzutjueuv/AviSynth26.7z

Changes:
- MT related fixes.
- General code cleanup (for scope, const).
- Project splitted and settings cleaned up. Converting from VS6 is a great way to get strange settings.
- Converted .asm to yasm. If we ever hope to be portable - it'll most likely be yasm for assembler.
First external look: changes to current CVS are rather massive! IMHO they should not be committed at once. but by portions for more easy revision (tracking) of changes.

More comments later.
Fizick is offline   Reply With Quote
Old 11th August 2009, 01:02   #8  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
Yes, "changes to current CVS are rather massive!", there are a lot of "white" changes like adding "const" and "extern" declaration to things that probably should have had them from the beginning. Arbitrary changes like using "long long" for "__int64" and fiddling with the for loops are not transparent as VC6 (yes we still support that) do not support them. Getting the "interlocked*" logic in many places is relevant, I had already found a lot of these but it help to have independent review. The code in cacheMT.cpp is very out of date (circa 2.5.5) and does not conform to 2.5.8 expectations for a Cache:: module, medium term this module will be deleted and the code subsumed back into the main cache code, it was only ever a place holder while TSP worked on the code.
IanB is offline   Reply With Quote
Old 11th August 2009, 01:06   #9  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,666
Quote:
Originally Posted by TheFluff View Post
but you have to make a written offer to provide the sources! and it must be valid for at least 3 years! you're gonna get sued!!111!
Or just include the source code with the binary...
Dark Shikari is offline   Reply With Quote
Old 11th August 2009, 07:31   #10  |  Link
lych_necross
ZZZzzzz...
 
lych_necross's Avatar
 
Join Date: Jan 2007
Location: USA
Posts: 303
Dark Shikari is right. Just include the sources with the binary.
lych_necross is offline   Reply With Quote
Old 11th August 2009, 07:51   #11  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
I think there is absolutely no reason for supporting VC6 compiler and flooding the code with
Quote:
stupid compiler forgets to save ebx!!
Quote:
Damn compiler does not correctly cast to intermediate results to float
and hacks to avoid it. Code should be moved towards standard C/C++ and maybe GCC compatibility. Of course supporting the VC6 build environment is perfectly fine.

Cache is the real problem - i'm getting bad memory leaks with heavy threading and trying to fix it now. Changes submitted are intermediate so it won't crash and disturb testing other things.
SEt is offline   Reply With Quote
Old 11th August 2009, 23:31   #12  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,100
Quote:
Originally Posted by IanB View Post
using "long long" for "__int64"
int64_t

e: isn't a major version bump a very good occasion to drop support for an ancient and buggy compiler anyway, I mean if you don't drop it now you'll still be sitting here supporting it in 2020 or so judging by how often avisynth gets major version bumps

Last edited by TheFluff; 11th August 2009 at 23:35.
TheFluff is offline   Reply With Quote
Old 13th August 2009, 12:22   #13  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Fixed minor threading bug and major memory leak:
Code:
--- avisynth.cpp	2009-08-08 22:20:34.550375000 +0400
+++ avisynth.cpp	2009-08-13 15:12:07.305375000 +0400
@@ -144,8 +144,9 @@
       return &i->vf;
   LinkedVideoFrame* result = (LinkedVideoFrame*)::operator new(sizeof(LinkedVideoFrame));
   result->vf.refcount=1;
-  result->next = g_Bin->g_VideoFrame_recycle_bin;
-  result->next = (LinkedVideoFrame*)InterlockedExchangePointer((long*)&g_Bin->g_VideoFrame_recycle_bin,result);//g_VideoFrame_recycle_bin is assigned twice to result->next just in case another thread reads result->next before the last assignment 
+  do
+		result->next = g_Bin->g_VideoFrame_recycle_bin;
+  while (InterlockedCompareExchangePointer((volatile PVOID*)&g_Bin->g_VideoFrame_recycle_bin, result, result->next) != result->next);
   return &result->vf;
 }
 
@@ -1639,7 +1640,7 @@
           InterlockedIncrement(&i->sequence_number);  // Signal to the cache that the vfb has been stolen
           return i;
         }
-        if (i->GetDataSize() > size) {
+        if (c > Cache::PC_Nil && i->GetDataSize() > size) {
           // Remember the smallest VFB that is bigger than our size
           if ((j == 0) || (i->GetDataSize() < j->GetDataSize()))
           {
@@ -1659,7 +1660,7 @@
 	if (!CacheHead) break; // cache state will not change in next loop iterations
 
     // Plan C: Steal the oldest, smallest free buffer that is greater in size
-    if (j && c > Cache::PC_Nil) {
+    if (j) {
       ++g_Mem_stats.PlanC;
       InterlockedIncrement(&j->sequence_number);  // Signal to the cache that the vfb has been stolen
       return j;
SEt is offline   Reply With Quote
Old 13th August 2009, 23:49   #14  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
@SEt,

Thanks, you must be psychic I noticed the 2nd issue while reading your other big change bundle. It is still not quite right. In the MT case CacheHead could be NUL because CacheMT doesn't implement the new poke cache stuff yet, so you exit the loop without actually using "the smallest VFB that is bigger than our size". that's the trouble with quickly retro fitting code from the main branch.

As for the 1st bit with g_Bin yeah it's stuffed. I am not happy with that whole concept, so .....
IanB is offline   Reply With Quote
Old 14th August 2009, 02:14   #15  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
No, the behavior is intended - i don't like plan C and would rather disable it. In case of memory starvation this "memory leak" can be quite unpleasant and more dangerous than memory fragmentation that it tries to prevent.

Let's see what can happen after we give bigger buffer than required (and usually it'll be 2..4+x bigger):
1) Big buffer won't be required in future - we are wasting memory, bad.
2) Big buffer will be required before allocated smaller frame is freed - we'll likely have to allocate another big buffer, wasting memory again, bad.
3) Big buffer will be required after smaller frame is freed, but before another smaller frame uses it - good, but i'd say unlikely.

Last edited by SEt; 14th August 2009 at 02:40.
SEt is offline   Reply With Quote
Old 15th August 2009, 22:37   #16  |  Link
Keiyakusha
契約者
 
Keiyakusha's Avatar
 
Join Date: Jun 2008
Posts: 1,576
Sorry if this thread not suits for my question...

So here is my image.
Here is my script:
Quote:
Imagesource("test.png")
Converttoyv24().SwapUV()
And here is what I see when I open it in VirtualDub:


So the question is, why chroma is swapped so I need to add SwapUV()? Is this some bug in VDub? Because if I add Converttorgb() to the end of the script - chroma returns to its place...
But more importantly why I'm getting this "extra" chroma or whatever is that? Am I doing something wrong?

EDIT: I'm using SEt's build from 1st post.

Last edited by Keiyakusha; 15th August 2009 at 22:43.
Keiyakusha is offline   Reply With Quote
Old 15th August 2009, 22:58   #17  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Color format conversions are currently broken (at least their dynamically compiled part). Also note that i haven't rebuilt the dll with posted bugfixes yet.
SEt is offline   Reply With Quote
Old 2nd September 2009, 20:43   #18  |  Link
BigDid
Actually in reserve
 
Join Date: Oct 2004
Posts: 1,605
Hi SET,

Quote:
Originally Posted by IanB View Post
Here is the 1st official release of Avisynth 2.6. The main pupose of this release is to confirm compatibility with version 2.5.8. For scripts using only 2.5.8 features the results should be identical.
...
The next release will contain Sh0dans latest FastWire updates and his SSE3+ code enhancements.
...
and
Quote:
Originally Posted by IanB View Post
...
Time is precious so I am skipping the frills for expedience sake. I am also restricting my participation in this forum, don't feel offended if I appear to be ignoring you, I am applying self discipline and not responding to anybody ( I am skimming the posts every few days for any active items). This way I actually have time to work on the code.
...
If I read IanB comments well, he will first focus on compatibility... so I believe MT in 2.6 is under the stack.

If true, it will be up to you (if you have time to do so) to update your 2.6MT build. Can it be expected?

I do not want to put pressure on any developper (IanB or you) just have a roadmap on what to expect or not.



Did
__________________
Having a problem with AutoGK? Read & use the FAQ & MORE FAQ first
Want to exchange on AutoGK? try doom10.org
In reserve (inactive) for an undefined period of time.
BigDid is offline   Reply With Quote
Old 3rd September 2009, 00:21   #19  |  Link
a451guy451
Xbox Live: o 4lif o
 
a451guy451's Avatar
 
Join Date: Jun 2009
Location: Monrovia, CA
Posts: 64
I wanted to just note briefly, that I was testing the above build for a while and it was unstable during transcodes on both quad and 8 core systems. I detailed it briefly in the above "AviSynth 2.6.0 Alpha1 [August 20th]" thread. Tested it exhaustively on a dual core though, and it was great; no issues. thanks for the continued work on it!
a451guy451 is offline   Reply With Quote
Old 4th September 2009, 23:57   #20  |  Link
SEt
Registered User
 
Join Date: Aug 2007
Posts: 374
Of course build 2009.08.08 is unstable (at least due to described in patch bugs). As this thread still interest someone, i've added newer build with posted patch applied (nothing else changed) - it's way more stable, but be sure to set enough SetMemoryMax to avoid late crash due to address space fragmentation.

BigDid, i have several MT-related ideas to write, but i don't want to maintain my own version of avisynth 2.6 unrelated to official. I've asked for official trunk/stable svn repositories but there was no reaction yet.
SEt is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 11:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.