Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 26th October 2007, 16:22   #681  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
I am novice in C++ and MT.
But I see only one "CRITICAL_SECTION cs;" declaration in MVinterface.h (line 721, MVFrames class).

Sorry, but may be uploded code is not from latest version?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 26th October 2007, 17:31   #682  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
sorry my mistake. You are right the MVinterface.h was from my first version. I have updated the source in the file.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 30th October 2007, 02:40   #683  |  Link
Spuds
dumber every day
 
Spuds's Avatar
 
Join Date: Dec 2006
Location: Planet Earth
Posts: 154
I've run several passes with tsp's latest modified mvtools using my test script. So far I have not found any defects either through subtract and watching or by using Mroz's script to look for changes.
Spuds is offline   Reply With Quote
Old 30th October 2007, 14:06   #684  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Quote:
Originally Posted by tsp View Post
Fizick: mainly to avoid resizing the array to many times. But incrementing it might be better as it doesn't take much time to copy the pointers.
The corruption in the lower right corner could be because both thread are running MVPlane::Refine/RefineExt at the same time. I have created a new version that uses a critical section to avoid this. It is available from here

mroz: Thanks for the testing. Could you try the new version above?

[edit]
ups wrong version. I have updated the file so please download again. This one should work.
[/edit]
Sorry for my lack of response. Firstly my main machine was disassembled for several days, then when I came back I obviously misunderstood how the mail notifications of thread updates work, as I had thought I'd have at least one notification per thread to which a post had been made, but perhaps one only gets a single notification for the first updated thread & then nothing until revisiting the forum, at which point notifications resume, but are only sent for new posts, not last unread. Anyhow, sorry.

I see Spuds has probably done the testing you need, but I'll add my two penneth when my current encodes are done - probably in 6 to 12 hours time - as you hope it will address the specific artifact I was seeing (thankfully I still have the source & non-MT encode of it).
mroz is offline   Reply With Quote
Old 30th October 2007, 20:17   #685  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Encode & comparison using http://www.avisynth.org/tsp/mvtoolsMTcomp2.zip now complete. Not only are there no measurable differences between frames in the MTcomp2 encode compared to a non MT encode, the resulting files are actually bitwise identical. Nice

Speed was roughly as expected & as before, but I didn't keep speed stats from earlier tests so don't have precise values; also this box was doing a few other tasks at the same time, albeit of low cpu intensity.

If you want I can rerun some shorter tests just to measure performance.

Last edited by mroz; 30th October 2007 at 20:20.
mroz is offline   Reply With Quote
Old 31st October 2007, 07:39   #686  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,167
Quote:
Originally Posted by Fizick View Post
Some question about memory optimizing.
I see if all buffer spaces are filled, you increase its size by doubling.
Why double the buffer size instead of incrementing it?
Yes I have wondered about the amount of buffers allocated. For a pure single sided linear script with no IDX surely only 1 buffer is needed. Given the size of the buffers and with TSP's mods maybe you should just start with 1 and add 1 at a time.

To get back performance with multiple access patterns using an IDX you could track the last few (8 or 16) frame numbers in to the Analyse GetFrame for that IDX number. If the same number occurs again, count how many different frames numbers occur in the list since last time then number occured and make make the buffer count at least that many + 1. i.e. access pattern :-
2, 3, 4, 5, 3, 4, 5, 6, 4, 5, 6, 7, 4, 5, 6, 7

For the first 4 accesses there are no repeats, on the 5 access for frame 3 there is a repeat in slot 2 and you do not still have the buffer for frame 3 (i.e a cache miss), so you make sure there are at least (5-2)+1=4 buffers.

On the 13th access for frame 4 there is a repeat in slot 9, but you do still have the buffer for frame 4 (i.e a cache hit), so you do nothing. If it was a miss you would increase to (13-9)+1=5
IanB is offline   Reply With Quote
Old 31st October 2007, 21:14   #687  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
mroz: It would be nice if you could make 4 short test with no SetMTmode and setMtmode(2,2), setMtmode(2,3) and setMtmode(2,4) to see how well it scales.

IanB: Your method would be an improvement. Currently the buffer size only grows if all the buffers has an refcount greater than 0 (assigned to a smartpointer somewhere) while there are no test for/adoption to cache misses. With your suggestion the max buffer count would be limited to the size of the track list but dynamically increasing the size of this list could be a solution to this.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 1st November 2007, 01:35   #688  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
A few test results to show how performance scales with number of threads.

I've run the tests using two different short clips (to help establish scaling behaviour is independent of video content for 'typical' video {ie not pathological cases}). I've also run initial tests twice to see what magnitude variation is due to (low) background activity & hence how many figures in the results are significant. Lastly I've run tests with both the last official release of mvtools & the latest tsp mvtools-comp2, for comparison.

Code:
clip1 mvtools-tsp2                      
        run1            
Name    Input   FPS     
job1    noSetMT 6.00    1.0000
job4    SetMT1  5.96    0.9933
job7    SetMT2  11.80   1.9667
job10   SetMT3  17.53   2.9217
job13   SetMT4  22.98   3.8300
job16   SetMT0  22.91   3.8183

        run2            
Name    Input   FPS     
job1    noSetMT 6.00    1.0000
job4    SetMT1  5.98    0.9967
job7    SetMT2  11.82   1.9700
job10   SetMT3  17.49   2.9150
job13   SetMT4  22.98   3.8300
job16   SetMT0  23.19   3.8650



clip1 mvtools-orig                      
        run1            
Name    Input   FPS     
job1    noSetMT 6.01    1.0000
job4    SetMT1  5.98    0.9950
job7    SetMT2  11.78   1.9601
job10   SetMT3  17.53   2.9168
job13   SetMT4  23.09   3.8419
job16   SetMT0  22.83   3.7987

        run2            
Name    Input   FPS     
job1    noSetMT 6.00    1.0000
job4    SetMT1  5.98    0.9967
job7    SetMT2  11.80   1.9667
job10   SetMT3  17.32   2.8867
job13   SetMT4  23.21   3.8683
job16   SetMT0  22.95   3.8250



clip2 mvtools-tsp2                      
        run1            
Name    Input   FPS     
job1    noSetMT 6.07    1.0000
job4    SetMT1  6.04    0.9951
job7    SetMT2  11.92   1.9638
job10   SetMT3  17.71   2.9176
job13   SetMT4  23.44   3.8616
job16   SetMT0  23.21   3.8237



clip2 mvtools-orig                      
        run1            
Name    Input   FPS     
job1    noSetMT 6.07    1.0000
job4    SetMT1  6.04    0.9951
job7    SetMT2  11.92   1.9638
job10   SetMT3  17.71   2.9176
job13   SetMT4  23.47   3.8666
job16   SetMT0  23.20   3.8221
mroz is offline   Reply With Quote
Old 1st November 2007, 06:00   #689  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
IanB,
for linear access the optimal buffer sise is dependent on number of MVanalyse calls in script (with same idx) .
as number+1.
So, 5 is usually fine for max delta=2, and 7 is for delta=3.
Probably it can simply be reset internally in every MVanalyse to (2*delta)+1 as a start value.
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.

Last edited by Fizick; 1st November 2007 at 06:02.
Fizick is offline   Reply With Quote
Old 6th November 2007, 01:55   #690  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Now I know that I have the same MT avisynth that I've always had (downloaded the latest just to make sure), but on a very simple script it's been wedged at one thread in avsp and virtualdub (1.7.1) for some reason:
Code:
setmtmode(2)
MPEG2Source("V:\video\work\psb\VTS_01_1 - 0xE0 - Video - MPEG-2 - 720x576 (PAL) - 4~3.d2v", cpu=4)
assumetff
tdeint(mode=1,mthreshl=3)
fft3dfilter(sigma=1,plane=4)
Spline36resize(720,480)
BlendFPS(60000/1001.,aperture=.4)
separatefields.selectevery(4,0,3).weave.AssumeTFF()
And even removing all but the source is the same.

GetMTMode(false) reports 0 although true reports 2. Fortunately I remembered the mencoder issue and adding Distributor fixed it. Any idea why it might be happening? I know it was working fine when I was testing mvtools last week, so I don't really understand what could have changed. Weird.

Wait, nm, Distributor fixes the speed but the video is totally mangled now. I guess that wasn't it.
foxyshadis is offline   Reply With Quote
Old 6th November 2007, 22:59   #691  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
new testversion of mvtools with ian's and fizicks suggestion for memory optimization that is tracking the last buffersize+3 frames accessed to see if the buffersize should increase and initial buffersize based on delta value for mvanalyse. You can get it here:
http://www.avisynth.org/tsp/mvtoolsMTcomp3.zip

foxyshadis: Sounds like it is MPEG2Source? What version are you using?
Does
SetMTmode(2)
blankclip()
work?
Using distributor when the MTmode is 0 will produce garbage because there are only one instance of each filter (like MTmode=1) and the ordinary non-thread safe version of the internal cache is used (the filtergraph is not setup for multithreading). With memcoder the MTmode=2 so the filtergraph is correctly created but distributor is never inserted at the end because avisynth doesn't know when memcoder is done creating the filtergraph.
At least fft3dfilter and spline36resize will work with MT() but I will see if I can reproduce the problem.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 7th November 2007, 00:23   #692  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Yes, it does work. DGDecode is 1.5.0, the current beta, though it also happens with 1.4.9 and other sources (avi, ffmpeg). I'm going to reboot and see if that helps.

Actually I just emptied the plugin folder and hey, it works. Now to hunt the culprit down.

Ah, mc_spuds.avsi. A cursory look doesn't really tell me why, unless all the globals are causing trouble. (Unnecessary now that mvtools is mt-friendly anyway, I guess.)
foxyshadis is offline   Reply With Quote
Old 7th November 2007, 02:50   #693  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Quote:
Originally Posted by tsp View Post
new testversion of mvtools with ian's and fizicks suggestion for memory optimization that is tracking the last buffersize+3 frames accessed to see if the buffersize should increase and initial buffersize based on delta value for mvanalyse. You can get it here:
http://www.avisynth.org/tsp/mvtoolsMTcomp3.zip
Cheers. Same performance & reliability here If you want any memory usage tests just supply a suitable script.
mroz is offline   Reply With Quote
Old 7th November 2007, 12:58   #694  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
tsp,

are you planning to update your build using the latest Avisynth 2.5.8 sources or should we wait for v2.6?

Thanks for the MVTools fix
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 7th November 2007, 22:05   #695  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Boulder: I will update it when avisynth 2.5.8 final is ready.

mroz: Sounds good. The memory usage will be higher for some of the more complex scripts or when running with more threads but should result in better performance (less cache misses). Oh and BTW thanks for the speed test. It is very good to see that one of the slowest avisynth filter scales so well with 4 cores. Now who wants to test with 8 cores?

foxyshadis: I think it could be the
global nullclp = blankclip(width=16,height=16)
in MC-Spuds that causes all the problem as it is evaluated before SetMTmode when saved as an .avsi file using import after setmtmode instead of autoloading should fix it.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/

Last edited by tsp; 7th November 2007 at 22:09.
tsp is offline   Reply With Quote
Old 8th November 2007, 07:50   #696  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,558
Thanks a lot.

I just want to throw out another vote for IanB to include MT officially in 2.5.8 now that it's basically stable.
foxyshadis is offline   Reply With Quote
Old 8th November 2007, 09:12   #697  |  Link
badshah
Registered User
 
badshah's Avatar
 
Join Date: Jan 2007
Posts: 45
I have core2duo, 1GB RAM

I installed avisynth 2.5.7

then from MT_07 package, I have put MT.dll in plugins directory & avisynth.dll from the same pack to windows/system32 folder

I am getting this error, while loading script into Vdub

AVI Import Filter error: (Unknown) (80040154)

If I put original avisynth.dll into system32 folder ... it works fine but then, I am not able to use MT

pls help

regards
badshah is offline   Reply With Quote
Old 8th November 2007, 09:36   #698  |  Link
Leak
ffdshow/AviSynth wrangler
 
Leak's Avatar
 
Join Date: Feb 2003
Location: Austria
Posts: 2,441
Quote:
Originally Posted by badshah View Post
I am getting this error, while loading script into Vdub

AVI Import Filter error: (Unknown) (80040154)
Which script are you talking about?

Also, does that happen with a script simply consisting of just
Code:
Version()
as well?
__________________
now playing: [artist] - [track] ([album])
Leak is offline   Reply With Quote
Old 8th November 2007, 10:14   #699  |  Link
badshah
Registered User
 
badshah's Avatar
 
Join Date: Jan 2007
Posts: 45
Quote:
Originally Posted by Leak View Post
Which script are you talking about?

Also, does that happen with a script simply consisting of just
Code:
Version()
as well?
Version() doesnt work as well !!

script :
Code:
SetMTmode(2)
DGDecode_mpeg2source("G:\My DVD\VTS_01_1.d2v",cpu=2,info=3)

crop( 12, 64, -12, -68)

Spline36Resize(672,368) # Spline36 (Neutral)
#denoise

Last edited by badshah; 8th November 2007 at 10:16.
badshah is offline   Reply With Quote
Old 8th November 2007, 12:16   #700  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
What are the contents of your Avisynth 2.5 plugins folder?
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.