Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th November 2007, 21:38   #741  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
some more test, same source as before:
Code:
setmtmode(2)
AVISource("D:\simH264.avi")
changefps(last,last,true)
94 sec (106 fps)
(so just as good as with no multithreading)
Code:
AVISource("D:\simH264.avi")
MT("fft3dfilter()",2,8)
521 sec (19,2 fps) slightly better than using setmtmode(2) but still fft3dfilter doesn't scales to well.
Code:
AVISource("D:\simH264.avi")
changefps(last,last,true)
MT("fft3dfilter()",2,8)
532 sec (18,8 fps) slightly slower. No surprise as MT() split the frame in two and process each part in a separate thread so it is sequential access even without changefps (same story for fft3dfilter(ncpu=2))
Code:
setmtmode(5)
AVISource("D:\simH264.avi")
changefps(last,last,true)
setmtmode(2)
fft3dfilter()
548 sec (18,2 fps) without changefps() 536 sec (18,7 fps). Again slower with changefps() and still slower than mt() (probably due to fft3dfilters intertal fft cache).

and now for a medium weight filter spline16resize
Code:
AVISource("D:\simH264.avi")
Spline16Resize(640,480)
154 sec (64,9 fps)
Code:
setmtmode(2)
AVISource("D:\simH264.avi")
Spline16Resize(640,480)
172 sec (58,1 fps) slower than without multithreading due to non sequential access.
Code:
setmtmode(5)
AVISource("D:\simH264.avi")
setmtmode(2)
Spline16Resize(640,480)
357 sec! (28 fps) ouch even worse, so always using mode=5 for avisource is not a good idea.
Code:
setmtmode(5)
AVISource("D:\simH264.avi")
changefps(last,last,true)
setmtmode(2)
Spline16Resize(640,480)
117 sec (85,5 fps) large improvement suggesting that non-sequential access is the culprit.
Code:
setmtmode(2)
AVISource("D:\simH264.avi")
changefps(last,last,true)
Spline16Resize(640,480)
105 sec (95,2 fps) fastest time.

so changefps(last,last,true) is most efficient when used with fast scripts while it makes the script slower in slow scripts.
Also I noticed that changefps made the largest difference in script where the fps was very unstable.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 20th November 2007, 23:12   #742  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
Hmm, there is something else going on here, ChangeFPS should be a zero cost filter, all it does is enforce sequential access within a 10 frame range forwards. It should never make things worse. Maybe the Cache behind it is not strong enough and is faulting, negating the sequential access protection. Hmmmmm
IanB is offline   Reply With Quote
Old 21st November 2007, 18:30   #743  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
repeated the test with setmemorymax(1500). The difference between AVISource("D:\simH264.avi").changefps(last,last,true).MT("fft3dfilter()",2,2) and AVISource("D:\simH264.avi").MT("fft3dfilter()",2,2) is down to 2-3 sec. So more or less insignificant.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 22nd November 2007, 04:42   #744  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
@tsp,

Are 2 and 5 the best modes to use here with changefps(last,last,true) and are we putting them in the right place? We just want to ensure the AviSource filter gets hit strictly in order. We do not want any incidental interlocking on the cache between changefps and avisource.

If the request order into changefps is 2, 1, 0, 5, 4, 3, ... the expected request order into the next cache should be 0, 1, 2, 1*, 0*, 1*, 2*, 3, 4, 5, 4*, 3*, ... where the entries marked with * should be cache hits. thus the request order into avisource should be 0, 1, 2, 3, 4, 5, ...
IanB is offline   Reply With Quote
Old 23rd November 2007, 08:58   #745  |  Link
Delerue
Registered User
 
Join Date: Jun 2005
Posts: 365
tsp and Fizick, I'm trying to use MVFlowFPS script (from MVTools 1.8.5.1) with this code inside the FFDShow (v. 1620) Avisynth tab:

Code:
LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
SetMTMode(2)
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
backward_vec = source.MVAnalyse(blksize=16, isb = true,  pel=2, search=2, idx=1)
forward_vec = source.MVAnalyse(blksize=16, isb = false, pel=2, search=2, idx=1)
source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), \
   den=FramerateDenominator(source), mask=1, idx=1)
Although it works, I mean, it doubles the FPS, the second CPU/thread still inactive (I'm using an Allendale E4300 running at 3 GHz).

I also tried this (from Fizick website, just to test), but I got a script error (invalid arguments to function 'MT', last line):

Code:
LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
global idx1 = 10  # global hint by IanB
MT("""
idx1 = idx1 + 1 
# different threads for top and bottom half of frame must have different idx (trick by Foxishadis)
backward_vec2 = MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec2 = MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
last.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx1)
""",2) # two threads
I copied the 'avisynth.dll' to the 'system32' folder, and the 'MT.dll' to the 'AviSynth\plugins' folder, inside the 'program files' folder. Any idea?

Thanks!

P.S.: Fizick, as you can see, your latest MVTools can finally works using the 'num=2*FramerateNumerator(source)' command with FFDShow.

Last edited by Delerue; 23rd November 2007 at 09:25.
Delerue is offline   Reply With Quote
Old 23rd November 2007, 18:38   #746  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Delerue: Try this version:
Code:
SetMtmode(5)
LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
SetMTMode(2)
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
backward_vec = source.MVAnalyse(blksize=16, isb = true,  pel=2, search=2, idx=1)
forward_vec = source.MVAnalyse(blksize=16, isb = false, pel=2, search=2, idx=1)
source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), \
   den=FramerateDenominator(source), mask=1, idx=1)
distributor()
and this:
Code:
function MVD(clip c)
{
idx1 = idx1 + 1 
# different threads for top and bottom half of frame must have different idx (trick by Foxishadis)
backward_vec2 = c.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
backward_vec1 = c.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec1 = c.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx1)
forward_vec2 = c.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx1)
return c.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx1)
}

LoadPlugin("C:\Arquivos de programas\AviSynth\plugins\MT.dll")
source=ffdshow_source()
LoadPlugin("C:\arquivos de programas\avisynth\plugins\mvtools.dll")
global idx1 = 10  # global hint by IanB
MT("""MVD(source)""",2) # two threads
put please note that using MVTools with MT() is not always good as MT() split the frame in two parts and process each part in a separate thread so MVTools can't motion compensate if the motion occurs from one part of the frame to the other part.
It is better to use SetMTmode(2)

IanB: mode=5 should be the best but my speed test indicate otherwise. I will repeat them to see if it is just random variation.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 23rd November 2007, 19:53   #747  |  Link
Delerue
Registered User
 
Join Date: Jun 2005
Posts: 365
Quote:
Originally Posted by tsp View Post
Delerue: Try this version:

[code1]

put please note that using MVTools with MT() is not always good as MT() split the frame in two parts and process each part in a separate thread so MVTools can't motion compensate if the motion occurs from one part of the frame to the other part. It is better to use SetMTmode(2)
Thanks, man! It works, but there's something strange. Although the second thread is activated now, the total CPU usage rarely goes beyond 50% and never beyond 60%, so forbidding the script to actually double the FPS in some heavy videos. It seems that the 'job division' isn't good enough, or something like this. I tried to play with other modes, but didn't help. Do you have a faster dual-core CPU to test this script?

Also, I want to congratulate you and Fizick once more. You're making an amazing job, and the 24 FPS movies problem are going soon even with HD videos.
Delerue is offline   Reply With Quote
Old 23rd November 2007, 20:19   #748  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Quote:
Originally Posted by Delerue View Post
...
I also tried this (from Fizick website, just to test), but I got a script error (invalid arguments to function 'MT', last line):

...
source=...
...
Please be correct. It is NOT exact script from my website (original works well).
"source" is not used by you.


tsp, sorry for many mvtools staff in your thread
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.

Last edited by Fizick; 23rd November 2007 at 20:21.
Fizick is offline   Reply With Quote
Old 23rd November 2007, 21:02   #749  |  Link
Delerue
Registered User
 
Join Date: Jun 2005
Posts: 365
Quote:
Originally Posted by Fizick View Post
Please be correct. It is NOT exact script from my website (original works well).
"source" is not used by you.
Well, I used source in this line 'source=ffdshow_source()'. I only adapted to use with FFDShow. I can't see what you're talking. Anyway, the second tsp script doesn't work either.

Quote:
Originally Posted by Fizick View Post
tsp, sorry for many mvtools staff in your thread
Well, we're talking about MT plugin as well.

Last edited by Delerue; 26th November 2007 at 21:37.
Delerue is offline   Reply With Quote
Old 24th November 2007, 21:47   #750  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Quote:
Originally Posted by Delerue View Post
Thanks, man! It works, but there's something strange. Although the second thread is activated now, the total CPU usage rarely goes beyond 50% and never beyond 60%, so forbidding the script to actually double the FPS in some heavy videos. It seems that the 'job division' isn't good enough, or something like this. I tried to play with other modes, but didn't help. Do you have a faster dual-core CPU to test this script?
Yes with two threads it looks like it hits a limit about 60 % cpu utilization. Just double the number of threads will fix that (at least it did for me from 60% to 90 %).
try this version:
Code:
SetMtmode(2,4)
source=ffdshow_source()
source=changefps(source,source,true)
backward_vec = source.MVAnalyse(blksize=16, isb = true,  pel=2, search=2, idx=1)
forward_vec = source.MVAnalyse(blksize=16, isb = false, pel=2, search=2, idx=1)
source.MVFlowFps(backward_vec, forward_vec, num=2*FramerateNumerator(source), \
   den=FramerateDenominator(source), mask=1, idx=1)
distributor()
What is the problem with the second script? Try SetMtmode(2) instead of MT()
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 25th November 2007, 00:46   #751  |  Link
Delerue
Registered User
 
Join Date: Jun 2005
Posts: 365
Quote:
Originally Posted by tsp View Post
Yes with two threads it looks like it hits a limit about 60 % cpu utilization. Just double the number of threads will fix that (at least it did for me from 60% to 90 %).
try this version:
[code]
Wow! That's for real! I tried with your script and the CPU usage indeed goes beyond 90%. But there is some problems with four threads here when the CPU usage is high (but still not 100%). I noticed some stutter, glitches and other weird things, specially in scenes with too much camera movements. So I tried 5 threads, and it worked perfectly.

About the other script, I'll try later.

Thanks once more.
Delerue is offline   Reply With Quote
Old 26th November 2007, 20:11   #752  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Livesms from russian forum post report about artifactes with script:
Code:
SetMTmode(2)
AviSource("TV.avi")
AssumeTFF()  
MVDegrain2i(4,0,1)  
TomsMoComp(-1,15,1)  
Crop(8,6,-8,-6).Lanczos4Resize(512,384)
FadeIO(25)
#################################################################################### 
function MVDegrain2i(clip "source", int "overlap", int "dct", int "idx") 
{ 
	overlap=default(overlap,0) 		# величина перекрытия value (от 0 до 4 для blksize=8) 
	dct=default(dct,0) 			# используйте dct=1 для клиров с некоторыми мерцаниями 
	idx=default(idx,1)			# используйте различные idx для разных источников в том же скрипте 
	fields=source.SeparateFields() 	# разделим на поля 
	backward_vec2 = fields.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct) 
	forward_vec2 = fields.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=overlap, idx = idx,dct=dct) 
	backward_vec4 = fields.MVAnalyse(isb = true, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct) 
	forward_vec4 = fields.MVAnalyse(isb = false, delta = 4, pel = 2, overlap=overlap, idx = idx,dct=dct) 
	fields.MVDegrain2(backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=400,idx=idx) 
	Weave() 
}
around 96 frame at:
http://dump.ru/files/n/n53176759/
press "Скачать" button near center under this link
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 26th November 2007, 20:45   #753  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Fizick: Is the sample the result of the above script or the source for the script?

IanB: After some more test the most optimal mode appears to be:
Code:
setmtmode(5)
AVISource("D:\simH264.avi")
setmtmode(2)
changefps(last,last,true)
even with a MJPEG encoded file.
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/
tsp is offline   Reply With Quote
Old 26th November 2007, 22:51   #754  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
@tsp,

So this will results in multiple instances of ChangeFPS all individually asserting linear access apon a gate5 cache into an interlocked AviSource. The gate5 cache will probably be thrashed very hard but it's all pointer shuffling without much bulk data copying, so its okay.
IanB is offline   Reply With Quote
Old 26th November 2007, 23:14   #755  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
tsp,
it is result.
does TomsMoComp(-1,15,1) work with setmtmode?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 27th November 2007, 06:47   #756  |  Link
Livesms
Registered User
 
Livesms's Avatar
 
Join Date: Mar 2006
Posts: 184
tsp
I'll try to upload original video and encoded.
This night I left my Core2Duo E6600 encoding two more video.
I'll check it in a hour...
Livesms is offline   Reply With Quote
Old 27th November 2007, 13:33   #757  |  Link
Livesms
Registered User
 
Livesms's Avatar
 
Join Date: Mar 2006
Posts: 184
Original video is too huge for my GPRS internet connection.

I've done some test and found SetMTMode(2) to work incorrectly with MVDegrain2 script for MVTools documentation.
Simple one thread MVDegrain2 (without MT), MVDegrain2 with SetMTMode(4) and MVDegrain2 with SetMTMode(5) produce output video without any artifacts, while SetMTMode(2) + MVDegrain2 and SetMTMode(5) + AviSource + SetMTMode(2) + MVDegrain2 do have some problems.

So I tried to look closely to SetMTMode(4) and MTi(). Here is the results:
Code:
 Source:
  * FourCC: YV12
  * Frames: 1003
  * Resolution: 512x384
  * Frame rate: 25.000 FPS
 
SetMTMode(2)
Destination:
  * Pass 1/1: Finished in 00:02:44.819 (6.09 FPS)
  * Frames: 1003 (1003 keyframes)
  * Memory: 121 788 MB
 
SetMTMode(4)
Destination:
  * Pass 1/1: Finished in 00:02:48.024 (5.97 FPS)
  * Frames: 1003 (1003 keyframes)
  * Memory: 144 252 MB
 
MTi()
Destination:
  * Pass 1/1: Finished in 00:02:57.276 (5.66 FPS)
  * Frames: 1003 (1003 keyframes)
  * Memory: 160 340 MB
So MTi() uses more memory than any of script using SetMTMode() and gives no speed increase comparing to SetMTMode(2) or SetMTMode(4)
SetMTMode(4) is 2% slower than SetMTMode(2) in my test. 1000 frames clips denoise time increased with 795ms. SetMTMode(4) uses more memory than SetMTMode(2) – 144252MB comparing to 121788MB.

Allowing 2% speed decrease and 15% more memory usage (comparing to SetMTMode(2)), it is necessary to check SetMTMode(4) + MVDegrain2().

In the moment my PC is encoding two more video (2pass xvid) with SetMTMode(2) and SetMTMode(4) in search of errors...

Last edited by Livesms; 27th November 2007 at 13:35.
Livesms is offline   Reply With Quote
Old 27th November 2007, 20:51   #758  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
@Livesms,

Do you have base timings for the above script i.e. without MT?

It is important to know how much boost you are getting. i.e. 3fps to 6fps would be fantastic, 5.9fps to 6.0fps indicates some development work is still needed.
IanB is offline   Reply With Quote
Old 27th November 2007, 22:19   #759  |  Link
tsp
Registered User
 
tsp's Avatar
 
Join Date: Aug 2004
Location: Denmark
Posts: 807
Livesms: Is it the same script Fizick posted?

Fizick: From ToMoComp's source code it looks like it should be compatible with SetMtMode(2) (No non-const global/static variables. Not like the evil Dust filter)

IanB: Even if the cache instance in front of avisource is hit by two thread it is still faster than one thread stalling in front of changefps (that would happend if SetMtMode(5) was placed after changeFPS). I wonder if it would be faster to implement the linear access requirement in the cache instead as it has better track on the last accessed frame (as the two instances of ChangeFPS doesn't share the lastframe variable).
__________________
Get my avisynth filters @ http://www.avisynth.org/tsp/

Last edited by tsp; 27th November 2007 at 22:23.
tsp is offline   Reply With Quote
Old 27th November 2007, 23:09   #760  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,168
@tsp,

Yes ChangeFPS is a hack proof of concept. And it seems to be somewhat successful when abused appropriately.

And yes a proper implementation would probably need to involve the cache or actually be a special case of cache.

Better still AviSource, et el, probably should be internally defending against out of order access. Some time ago I changed code to just reuse the same VFB, the original grabbed a stack of new VFB's as it seeked forwards from a Keyframe, it didn't remember them so it was a wasted effort that just generally trashed the cache. In hindsight a better fix would have been to add code to remember the VFB's. Hmmm!

I don't think always enforcing linear access would be a good idea, however as an optional weapon in the arsenal it's gonna be worthwhile.

Think SelectEven into a huffyuv file versus an Xvid file. With linear access you force the huffyuv to read all the disk blocks instead of just half. With the Xvid file you have to read all the disk blocks regardless and the out of order penalty can be extreme.
IanB is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 11:55.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.