Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th October 2007, 01:23   #641  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
Gees guys, take a minute and think about this.

If you are going to use the IDX option in Mvtools with multithreading then you need a scheme to ensure that every thread has a unique IDX value.

It is easy to test if your current idea is working as expected by putting a SubTitle(String(IDX)) in appropriate places in your script. None of the SetMtMode's are going to allow this.
IanB is offline   Reply With Quote
Old 20th October 2007, 01:50   #642  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Ian, Spuds is just testing how well it works without doing that, since mroz and I both got good results with it - your solution is probably the most elegant, now that it's been proven to fail.

MFA, shared memory sections = race conditions, which is no better than we have now. Everything has to be made threadsafe at either the plugin or the scripthost level anyway.

Fizick, as long as you're here have you ever thought of creating the idx clip by cloning vi with x2 or x4 height/width and requesting blank frames from this new clip? This way it will be cached and can be limited by setmemorymax - the downside being that if it ever gets limited, it'll drop performance, but not nearly as much as swapping to disk. Since it's a global, it may have lifetime issues that using it in a single class doesn't, I don't know nearly enough about avisynth's smart pointers to know.
foxyshadis is offline   Reply With Quote
Old 20th October 2007, 02:39   #643  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
Quote:
Originally Posted by foxyshadis View Post
MFA, shared memory sections = race conditions
Everything is shared memory with multithreading ...
Quote:
Everything has to be made threadsafe at either the plugin or the scripthost level anyway.
You could get away with adding a single global lock to make the scripthost thread safe (since there could only be internal threading inside a single instance of a filter for a given scripthost you could fundamentally not cause deadlock that way AFAICS).

Since the plugins would never run as threads themselves they would obviously not need to be thread safe.
MfA is offline   Reply With Quote
Old 20th October 2007, 03:45   #644  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
If you have a mutex/semaphore (global lock), all that's needed to deadlock is one process not giving up the lock, instead of one thread. Kill the process and it goes away, of course, same as killing a thread for a critical section.

x264farm deals with a similar issue by also running a monitoring process that will kill and restart hung processes where they left off.

More complexity, especially since you don't actually gain anything: The vast majority of avisynth plugins are threadsafe as-is, so is the core now, and there's a workaround for the biggest exception. You're just rewriting MT avisynth in a different way, with a more time-consuming synchronization primitive.

However, I could see it being useful for a 64-bit operating system with a dozen gigs of memory going to waste in the single 32-bit process you need to run your plugins in. Then you want all the processes you can get, and don't even need synchronization.
foxyshadis is offline   Reply With Quote
Old 20th October 2007, 07:03   #645  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
foxyshadis,
of course I considered (and is considering) to make idx clip with MVPrepare. It is in to do list (to kill idx).

IanB,
No, every thread should not to have a unique IDX value,
with proper locking.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 20th October 2007, 15:44   #646  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
Quote:
Originally Posted by foxyshadis View Post
If you have a mutex/semaphore (global lock), all that's needed to deadlock is one process not giving up the lock
The common meaning in CS of deadlock means a situation where to give up a lock a process needs to acquire the lock in some roundabout way ... you need some type of loop in the call graph. A lock never being given up because of a livelock (ie. infinite loop or something) is different from a deadlock.

I freely admit with already so much effort being put in enabling multithreading multiprocessing doesn't make too much sense anymore ... but still in the end it's a poor design. Why share a memory space when it has no performance benefits and only enables a multitude of very subtle bugs?

I know it's a bit of an appeal to authority ... but still worth mentioning, Avery Lee considers making filters thread safe enough of a pain he fundamentally wants to never let any run in parallel with itself.

Last edited by MfA; 20th October 2007 at 15:48.
MfA is offline   Reply With Quote
Old 21st October 2007, 03:44   #647  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
@Fizick,

Well we don't have any Script level locking syntax. And if you put locking in your filter, then you will probably end up single tasking it. The real problem is the whole IDX concept, it is unAvisynth. All comunication between modules in MV should be thru the IClip interface with the Avisynth CACHE providing all the buffering needed, backdoor hacks like private buffers and IDX numbers make the whole design fragile.
IanB is offline   Reply With Quote
Old 21st October 2007, 11:28   #648  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
IanB, i say about SetMtMode(), not about MT().
if idx is unique for every thread, we will have duplicate caclulation
Two ways:
1. If we preserve indx syntax, I (we or somebody) can implement internal three-state indicator of "interpolated idx frame":
- not-exist (may be calculated and written right now by current thread),
- ready to read (may be read by current thread),
- currently updated by other thread (current thread must wait to read).

2. Yes, it is better to remove idx. It is my dream too. But is need in more work (and some tricks again).
I need to send some parameters about vectors. Avisynth still does not have normal way to trasfer additional parameters (metadata). When it will be implemented? I do not see it even in Avisynth 2.6. Currently I consider to use audio parameters as metadata. I will try analyse and formulate suggestions in MVTools thread.

Final notes.
I (and you) do not have dual-core anyway, but probably you can add some part of AvisynthMT to version 2.5.8? I say about minimal changes to support MT (and MTi ?) filter only (it is rather stable) as a first step.
How big chages are need?
SetMTmode may be added later, in v2.6.0.
May be tsp can comment this.
Fizick is offline   Reply With Quote
Old 21st October 2007, 15:16   #649  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Another post from the ignorant here: Can anyone (Fizick?) explain why with the current code, SetMTMode(2) & MVTools isn't produing frequent obvious corruption? Only Spuds has seen that & even then with SetMTMode(2,0) found very little corruption at most. Surely if the processing is using cached data from completely the wrong frames around 1/2 to 3/4 of the time (roughly & depending on number of cores/threads) we should see obvious artifacting?

Can I also recap - the SetMTMode(2,0) with global idx = idx+1 workaround...

** Ought this to work around any problem resulting in artifacting, but at the expense of forcing MVTools to perform duplicate calculations? Have I got that right? **
Edit: Sorry, **'s rubbish isn't it - other than when including idx = idx+1 within the string passed to MT, this will be of no benefit whatsoever. Hence no help in the SetMTMode only case, yes?

I welcome a proper fix, of course, but if we're currently talking not until AVISynth 2.6, a short term hack is needed, if it's possible at all.

Last edited by mroz; 21st October 2007 at 15:32.
mroz is offline   Reply With Quote
Old 21st October 2007, 18:18   #650  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
mroz,
Fizick officially does not support MT with MVTools.
It is under development, as well as MT itself.
Probably most time two thead spend similar time for processing, i.e. work almost in synchro.
Fizick is offline   Reply With Quote
Old 21st October 2007, 21:03   #651  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Quote:
Originally Posted by Fizick View Post
IanB, i say about SetMtMode(), not about MT().
if idx is unique for every thread, we will have duplicate caclulation
Two ways:
1. If we preserve indx syntax, I (we or somebody) can implement internal three-state indicator of "interpolated idx frame":
- not-exist (may be calculated and written right now by current thread),
- ready to read (may be read by current thread),
- currently updated by other thread (current thread must wait to read).
I could give it a try.

Quote:
Final notes.
I (and you) do not have dual-core anyway, but probably you can add some part of AvisynthMT to version 2.5.8? I say about minimal changes to support MT (and MTi ?) filter only (it is rather stable) as a first step.
How big chages are need?
SetMTmode may be added later, in v2.6.0.
May be tsp can comment this.
If SetMTmode/GetMTmode shouldn't be supported it would only be necessary to implement most of the changes in avisynth.cpp/.h and cache.cpp.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 21st October 2007, 21:15   #652  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
I created separate thread (multi-threaded discussion! ) about developing MVTools without idx
http://forum.doom9.org/showthread.php?t=131033

Last edited by Fizick; 10th October 2008 at 16:09.
Fizick is offline   Reply With Quote
Old 21st October 2007, 23:18   #653  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Quote:
Originally Posted by Fizick View Post
mroz,
Fizick officially does not support MT with MVTools.
It is under development, as well as MT itself.
Probably most time two thead spend similar time for processing, i.e. work almost in synchro.
That's a shame, though good to know development is on going.

I'm not sure what you're saying by the last sentence. I just know on a quad core I get nearly four times the performance from MVTools intensive scripts when I use SetMTMode. Perhaps more importantly, that speed up makes scripts practical that otherwise I couldn't afford to make frequent use of (I know it's personal, but for me, 11 fps is acceptable for a prerender, while 3 fps isn't generally).

I'll subscribe to your new thread & will jump in if/when there are any releases needing testing, though I imagine this won't be for some time.

Edit: still very puzzled why MVTools/SetMTMode(2) seems to work for some, including myself. I'd welcome any comments as to what might be going on internally & am happy to conduct any testing if helpful.

Last edited by mroz; 21st October 2007 at 23:20.
mroz is offline   Reply With Quote
Old 22nd October 2007, 03:20   #654  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
@mroz,

Simply put it's a crap shot, i.e. roll the dice and pray.

MVTools stores internal state so when accessed in a normal sequential manner it can reuse previous calculations to save time. Using the IDX parameter allows different MVAnalyse() instances to share the same calcs if available.

If you don't set IDX then you cannot share the calc so there can be no speed up but then they cannot be stuffed up.

If the accesses are not sequential then the state is again useless and a full recalc happens. No stuff ups here.

Things go wrong when the shared calculations are not appropriate. Even without an IDX hitting the same MVAnalyse instance twice at just the right time can confuse things.
IanB is offline   Reply With Quote
Old 22nd October 2007, 03:54   #655  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Quote:
Originally Posted by IanB View Post
@mroz,

Simply put it's a crap shot, i.e. roll the dice and pray.

MVTools stores internal state so when accessed in a normal sequential manner it can reuse previous calculations to save time. Using the IDX parameter allows different MVAnalyse() instances to share the same calcs if available.
Understood - btw your post in the 'MTools without idx' thread was most informative as to some of the internal workings of both avisynth & mvtools; I'll keep reading & maybe some will sink in.

Quote:
If you don't set IDX then you cannot share the calc so there can be no speed up but then they cannot be stuffed up.
Do you mean if it wasn't used in that way or literally that calling MV<whatever> without an idx parameter will avoid many of the problems at the expense of speed? Only I get no significant speed difference here, though have only run one quick test.

Quote:
If the accesses are not sequential then the state is again useless and a full recalc happens. No stuff ups here.
Understood.

Quote:
Things go wrong when the shared calculations are not appropriate. Even without an IDX hitting the same MVAnalyse instance twice at just the right time can confuse things.
Indeed. That makes perfect sense & is highly plausible. I imagine my failure to understand why artifacts aren't more common is simply as I don't have much understanding of a typical execution profile for this process. Naively it still feels to me that corruptions should be a common occurence, not rare.

Thanks for the feedback.
mroz is offline   Reply With Quote
Old 22nd October 2007, 04:08   #656  |  Link
Spuds
dumber every day
 
Spuds's Avatar
 
Join Date: Dec 2006
Location: Planet Earth
Posts: 154
@morz,

I can't say it much better than that. All I can say is from my testing is that I've had several 'clean' runs, of about 1.5min of footage that is LOL. As I sat there watching the mind numbing pure grey screen of the subtract output of those runs, every now and then I'd catch an occasional hideous frame.

thats the way he likes it ... Well thats the way he gets it ... I don't like it anymore than you men ....
Spuds is offline   Reply With Quote
Old 22nd October 2007, 08:18   #657  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
The area of code that is vunerable is quite short, the code that does all the hard work is literally millions of time longer. So shooting snake eyes is more likely. However low the chances are not zero so you will get a dud frame eventually.
IanB is offline   Reply With Quote
Old 22nd October 2007, 18:56   #658  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
What if multithreded SetMTMode(2,2) is on?
I do not know exactly, probably it use one thread for even output frames, and second thread for odd frames.
In example above, the "idx clip" some frame may be calculated by else even thread or odd thread.
Buffer size is limited, so sometimes it's memory is overwritten.
IMO it is the MOST DANGEROUS OPERATION (around line 493 of MVFrame.cpp).
If one thread is reading frame from buffer (it is not momentary process), and at same time other thread is overwriting this frame buffer (by other frame), we will get mess. Probability is not high (time to read is small relatively to other calculations).
If this unknown value is about 1/10000, then every 1 from 10000 frames is bad, i.e. 10 bad frames per 1 hour.
IMO it is dependent on mumber MV functions in script, number of CPU (thread) and on buffer array size.
Probably it may depend on some other processes in system.

Partial workaround: increase constant MV_BUFFER_FRAMES array size. Now it is 10.
Here is test versions with different array sizes (4,5,6,15):
http://avisynth.org.ru/tmp/mvtools4.dll
http://avisynth.org.ru/tmp/mvtools5.dll
http://avisynth.org.ru/tmp/mvtools6.dll
http://avisynth.org.ru/tmp/mvtools15.dll

Please test.

Better workaround: use counter of threads (or functions) accessed the frame, like smart pointers. Increment it while access, decrement it when leave.

May be try use variable array size and increment it if all current buffers are filled (locked).
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.

Last edited by Fizick; 23rd October 2007 at 16:42. Reason: links
Fizick is offline   Reply With Quote
Old 22nd October 2007, 20:00   #659  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Quote:
Originally Posted by Fizick View Post
What if multithreded SetMTMode(2,2) is on?
I do not know exactly, probably it use one thread for even output frames, and second thread for odd frames.
it is more or less random which thread that will process the next frame (depending on both thread are ready to process a new frame or only one is ready)
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/

Last edited by tsp; 22nd October 2007 at 20:44.
tsp is offline   Reply With Quote
Old 23rd October 2007, 01:17   #660  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
@Fizick Will run your tests, but was thinking, as one needs to process a large number of frames & look for differences in the output it makes sense to write a script to count the deviant frames.

/goes & reads about conditional filters et al

Think I know enough now.

Just started my baseline non-SetMTMode huffy encode of your simple script applied to a film of 144,700 frames, which should show something up. I'll follow this later today with SetMTMode(2,0) & the same script via first the current MVTools then the above versions.

Great to see this being worked on now btw - I imagined the problems would be put to one side for weeks if not months.

Edit: This comparison script look ok?

Code:
leaf1="noSetMT"
leaf2="SetMT-v2"
global c1=AVISource("E:\Work\test\hfyu_MVMT-test-"+leaf1+".avi")
global c2=AVISource("E:\Work\test\hfyu_MVMT-test-"+leaf2+".avi")
global total=0
file="E:\Work\test\compare-"+leaf1+"-"+leaf2+".log"
Subtract(c1,c2)
WriteFileStart(file, """ "vim:tabstop=6"+chr(10) """, """ Time("%#c")+chr(10) """, """ "Frame"+chr(9)+"Total"+chr(9)+"Variation" """, append=true)
#nb compare against 0.0001 instead of 0 to allow for rounding errors - 2 identical files will report a variation of 0.000022 
WriteFileIf(file, "variation>0.0001", "current_frame", "chr(9)", "total", "chr(9)", "variation")
WriteFileEnd(file, """ "Test complete."+chr(10) """)
ScriptClip( "global variation = LumaDifference(c1,c2)+ChromaUDifference(c1,c2)+ChromaVDifference(c1,c2)"+chr(13) \
           +"global total = total + ((variation>0.0001) ? 1 : 0)"+chr(13) \
           +"Subtitle(String(variation))" \
)

Last edited by mroz; 23rd October 2007 at 05:46.
mroz is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 11:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.