Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 13th February 2014, 11:15   #601  |  Link
ryrynz
Registered User
 
ryrynz's Avatar
 
Join Date: Mar 2009
Posts: 3,646
Running Avsmeter with prefetch enabled causes the benchmark to slow down considerably.

Code:
[General info]
Log file created with:   AVSMeter 1.7.6 (AVS 2.6, x86)
Avisynth version:        AviSynth+ 0.1 (r1689, x86)


[Clip info]
Number of frames:                 1726
Length (hhh:mm:ss.ms):   000:01:11.989
Frame width:                       704
Frame height:                      480
Framerate:                      23.976 (2500000/104271)
Interlaced:                         No
Colorspace:                       YV12
Audio channels:                      2
Audio bits/sample:                  16
Audio samples/sec:               48000
Audio samples:                 3454656


[Runtime info]
Frames processed:               8 (0 - 7)
FPS (min | max | average):      0.017 | 237169 | 0.044
CPU usage (average):            6%
Thread count:                   27
Physical Memory usage (peak):   90 MB
Virtual Memory usage (peak):    173 MB
Time (elapsed):                 000:03:00.148


[Script]
SetFilterMTMode("", 2)
SetFilterMTMode("Directshowsource", 3)
Directshowsource("c:\1.mkv")
Deblock(20)
VagueDenoiser(threshold=1.0, nsteps=8, chromaT=0)
Hysteria(strength=0.2, highthresh=200)
Prefetch(4)


[Performance data]
       Frame    Frames/sec   Time/frame(ms)   CPU(%)   Threads   PhysMEM(MB)   VirtMEM(MB)
           0         0.017     60057.016993        0        32            82           168
           1         0.017     60014.952892        0        28            88           171
           2       171.206         5.840931        0        28            88           171
           3         8.837       113.165406       12        28            90           173
           4     33203.609         0.030117       12        28            90           173
           5    237168.577         0.004216       12        28            90           173
           6        83.888        11.920692       12        28            90           173
           7         0.017     59944.863827        0        27            90           173
Even just having deblock as the only filter with the prefetch call causes it to lag too, so it appears directly related to the prefetch call.
This script using Avisynth 2.6MT finishes in 33 seconds, avs+ without the prefetch call finishes it in 40 seconds.
ryrynz is offline  
Old 13th February 2014, 11:39   #602  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Looks like it breaks because of the SetFilterMTMode("", 2) line. SetFilterMTMode("audiodub",3) should fix it. Apparently AudioDub (called by dss internally) has issues with MT mode 2 which you try to apply to it by default.
And I still believe this way of setting the default MT mode should be removed. Brings nothings but problems.


Quote:
Originally Posted by jpsdr View Post
Should it be better to detect if MT mode is enabled ?
If someone use Avisynth+ without using multi-threading, i think internal multi-threading should be kept.
In theory - yes. But you have to use IScriptEnviroment2 to get number of threads and it's strongly discouraged right now.

The most optimal way from performance point of view would be providing a wrapper function around the plugin which would set number of threads to 1 and pass all other parameters unchanged right to the plugin. Something like:
Code:
function awesome_filter(clip c, int "thresh") {
    awesome_filter(c, thresh, threads=1)
}
But this is again far from optimal because it's not currently possible to get number of threads in the script (it's not known at "compile" time) so you can't check if you're actually running MT in this function. But at least the user can comment this wrapper out if he wants to re-enable internal threading.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth

Last edited by TurboPascal7; 13th February 2014 at 12:03. Reason: rewriting the history
TurboPascal7 is offline  
Old 13th February 2014, 12:10   #603  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by TurboPascal7 View Post
Apparently AudioDub (called by dss internally) has issues with MT mode 2
That's strange, since AudioDub() doesn't do any real work of its own - it just redirects video and audio requests (GetFrame() and GetAudio()) to the appropriate input clip.
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline  
Old 13th February 2014, 15:58   #604  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 478
I'm getting a few instances whereby my script just stops suddenly. It doesn't crash, the encoding process just stops. Perhaps signs of a deadlock?

Here's my script:

Code:
SetFilterMTMode("QTGMC",2)
SetFilterMTMode("FFVideoSource",2)
LoadPlugin("C:\Program Files (x86)\MeGUI\tools\ffms\ffms2.dll")
FFVideoSource("I:\Temp\Test.mkv", fpsnum=24000, fpsden=1001, threads=1)
Dither_convert_8_to_16()
Dither_y_gamma_to_linear()
Dither_resize16(1280,720)
Dither_y_linear_to_gamma()
DitherPost()
QTGMC(Preset="Slow",InputType=1)
InterFrame(Cores=1,GPU=True)
PreFetch(3)
I'm also seeing the same thing when QTGMC'ing interlaced DVD content. I'm using Vit's modded Masktools and RGTools in place of RemoveGrain. Is there some logging mode I can invoke to help diagnose this problem?

Last edited by aegisofrime; 13th February 2014 at 16:01.
aegisofrime is offline  
Old 13th February 2014, 16:19   #605  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by aegisofrime View Post
Code:
[B]SetFilterMTMode("QTGMC",2)
SetFilterMTMode("FFVideoSource",2)
You really should read ultim's notes on how to use MT.
FFVideoSource should have Mode 3. I also doubt that you can set a mode for an entire script that uses a whole bunch of filters (QTGMC) which might each need to be treated differently.
Groucho2004 is offline  
Old 13th February 2014, 16:40   #606  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 478
Quote:
Originally Posted by Groucho2004 View Post
You really should read ultim's notes on how to use MT.
FFVideoSource should have Mode 3. I also doubt that you can set a mode for an entire script that uses a whole bunch of filters (QTGMC) which might each need to be treated differently.
Thanks, I guess I missed the part about FFVideoSource. I guess I will also need to do some trial and error for the individual filters in QTGMC?
aegisofrime is offline  
Old 13th February 2014, 17:40   #607  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
can someone put the mt version on another storage site? I can't download it from mega, when I try to download it as a zip, it always stuck at 17%.
lansing is offline  
Old 13th February 2014, 17:55   #608  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by aegisofrime View Post
I guess I will also need to do some trial and error for the individual filters in QTGMC?
Yes, although here is already some info.
Groucho2004 is offline  
Old 13th February 2014, 21:18   #609  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
You cannot set the MT mode on script function calls, only on binary filters. So to make QTGMC work, all filters that it uses internally need their MT mode set. I did a basic test though when the the current test build was published, and QTGMC worked fine with a single SetFilterMTMode("", 2) call at the beginning and default arguments, so that is a start. But keep in mind that non-default arguments can easily require more complex MT-mode settings.

Anyway, thanks for all the testing. A few more days and I will begin hacking around it at large again. I am also thinking it might be very useful to have a logging framework in place (and maybe start with that), it could provide useful information about MT mode settings or other warnings to script writers. There is definetely a place to improve on the performance of the current version, and be assured that I will look at that in detail, as well as the issues that have been posted.
__________________
AviSynth+
ultim is offline  
Old 14th February 2014, 03:39   #610  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Quote:
Originally Posted by lansing View Post
can someone put the mt version on another storage site? I can't download it from mega, when I try to download it as a zip, it always stuck at 17%.
This should do.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Old 14th February 2014, 05:11   #611  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
I did a simple test with mpeg2source, the prefetch() switch doesn't seem to do anything. Whether i put it there, prefetch(1) for single thread or prefetch(x), mt is always on.
lansing is offline  
Old 14th February 2014, 05:18   #612  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Quote:
Originally Posted by lansing View Post
I did a simple test with mpeg2source, the prefetch() switch doesn't seem to do anything. Whether i put it there, prefetch(1) for single thread or prefetch(x), mt is always on.
Please do post your full script. Also the best way of running a script singlethreaded is not including a prefetch call at all.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Old 14th February 2014, 05:52   #613  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
Code:
SetFilterMTMode("MPEG2Source",3)
MPEG2Source("sample.d2v", cpu=0)
Even when I took out the setfiltermtmode line, it still ran on mt. Here's the log from avsmeter:

Code:
AVSMeter 1.7.6 (AVS 2.6, x86) by Groucho2004
AviSynth+ 0.1 (r1689, x86)

Number of frames:                36853
Length (hhh:mm:ss.ms):   000:20:29.662
Frame width:                       720
Frame height:                      480
Framerate:                      29.970 (30000/1001)
Interlaced:                         No
Colorspace:                       YV12

Frames processed:               6180 (0 - 6179)
FPS (min | max | average):      248.7 | 340.6 | 295.5
CPU usage (average):            12%
Thread count:                   13
Physical Memory usage (peak):   35 MB
Virtual Memory usage (peak):    69 MB
Time (elapsed):                 000:00:20.915

Press any key to exit...
lansing is offline  
Old 14th February 2014, 05:54   #614  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
The fact that you see more threads doesn't mean it's running MT. It's a minor implementation detail which might change in the future or it might not.
You can see that your CPU load is still locked at 12% which means one running thread on a 8-threads system.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Old 14th February 2014, 06:29   #615  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
No, it's running at 12% cpu because that's the max speed mpeg2source can run in my machine on multithread. If I ran the same vob file with ffvideosource, it would clearly show that mt is on because it would be a lot faster.

Code:
ffvideosource("sample.vob", fpsnum=30000, fpsden=1001)
Code:
AVSMeter 1.7.6 (AVS 2.6, x86) by Groucho2004
AviSynth+ 0.1 (r1689, x86)

Number of frames:                36849
Length (hhh:mm:ss.ms):   000:20:29.528
Frame width:                       720
Frame height:                      480
Framerate:                      29.970 (30000/1001)
Interlaced:                         No
Colorspace:                       YV12

Frames processed:               36849 (0 - 36848)
FPS (min | max | average):      1242 | 1818 | 1660
CPU usage (average):            65%
Thread count:                   30
Physical Memory usage (peak):   52 MB
Virtual Memory usage (peak):    68 MB
Time (elapsed):                 000:00:22.202
lansing is offline  
Old 14th February 2014, 06:32   #616  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Yeah because ffvideosource has internal mulithreading. It even has threads parameter which you can set to one.
There is no spoon.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Old 14th February 2014, 07:31   #617  |  Link
lansing
Registered User
 
Join Date: Sep 2006
Posts: 1,657
Quote:
Originally Posted by TurboPascal7 View Post
Yeah because ffvideosource has internal mulithreading. It even has threads parameter which you can set to one.
There is no spoon.
You are right. I tried ran it with tnlmeans and took out the prefetch call, then it always ran at 12% cpu. When I change it to prefetch(8), it ran at 99%.
lansing is offline  
Old 14th February 2014, 07:40   #618  |  Link
ryrynz
Registered User
 
ryrynz's Avatar
 
Join Date: Mar 2009
Posts: 3,646
Any ideas on what caused my slowdown with avsmeter? Can it be reproduced?
ryrynz is offline  
Old 14th February 2014, 07:47   #619  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Quote:
Originally Posted by ryrynz View Post
Any ideas on what caused my slowdown with avsmeter? Can it be reproduced?
Did you try what I posted? First two lines are about your directshowsource problem.
__________________
Me on GitHub | AviSynth+ - the (dead) future of AviSynth
TurboPascal7 is offline  
Old 14th February 2014, 07:54   #620  |  Link
ryrynz
Registered User
 
ryrynz's Avatar
 
Join Date: Mar 2009
Posts: 3,646
That's it. Wow. Avs+ did in 3 seconds what 2.6MT took 30 seconds to do..
Can audiodub be made mtmode 2 compatible?

Last edited by ryrynz; 14th February 2014 at 07:57.
ryrynz is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.