Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 17th October 2007, 22:42   #621  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,496
I've been testing, because I couldn't tell when you first asked. I can't really tell, since the artifacts could be subtle - one wrong frame with high metrics can be covered up by the other good frames. But I haven't really come across any really glaring errors with SetMTMode, so it may be that it really does work okay.

MT() makes it painfully obvious, but they do use very different strategies.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 17th October 2007, 23:29   #622  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
I do not understand, how Setmtmode(2) should work? odd frmes by one thead, even thead by second thread?
If so, you will not get any artifactes, but simply some more different (bad) vectors and denoising.

Somebody must encode some short clip to two uncompressed Avi (with and without Setmtmode). And then compare Avis by subtract().

second question. Is any speed gain with setmtmode?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 17th October 2007, 23:49   #623  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
@Foxyshadis

Thanks for the feedback. I suppose depending on the script it could conceivably simply not do it's job yet not introduce glaring artifacts, as a best/worst case (depending on ones point of view - personally I'd rather have realised/noticed sooner).

@Fizick

I'm using a quad core, so there'll be four threads, however I think your description is essentially correct, although I'm guessing there's no guarantee the allocation between threads would necessarily be precisely 1,2,3,4,1,2,3,4,1,2,3,4,... if for some reason one takes significantly longer than another.

Of course it never occurred to me to try subtract. Will run a short test in a few minutes.

Speedwise, the script I quoted on a dvd resolution huffy pre-render within Megui, gives me nearly 3fps without SetMTMode & just over 11fps with SetMTMode(2,0), thus the speed gain is very nearly linear in the number of threads & cores.

NB Due to a problem in the way Mencoder handles avs input, one must append Distributor() as the list line to the script before giving it to Mencoder. This doesn't apply if using VDub for an encode, say.
mroz is offline   Reply With Quote
Old 17th October 2007, 23:51   #624  |  Link
ChrisW77
Registered User
 
Join Date: Sep 2006
Posts: 72
I used both, but not in the same script.

SetMTMode(2,0)

and

MT("
",2,2)

The problem I had with MT, is where to put the ",2,2) ?
I used it on Interlace material, and wasn't sure whether to put it before or after Weave()

Quote:
second question. Is any speed gain with setmtmode?
I had quite a huge gain using it, sometimes almost double, going from say 8fps MT off to 16fps on.

Last edited by ChrisW77; 17th October 2007 at 23:56. Reason: to mention I used 2 scripts not one.
ChrisW77 is offline   Reply With Quote
Old 18th October 2007, 00:03   #625  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
to be clear, I say about test of some simplest script with mvtools, without any other filters. for example

backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 18th October 2007, 00:07   #626  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Just tested a 9s clip from a nasty dark & foggy noisy scene. Didn't need to use subtract - the files were the same sizes so I checked md5 hashes; they're identical.

I'd question the test, but I just sat here & watched one encode at 10.4fps & the other at 2.9fps.

Comments?

@Fizick: Can you suggest a simple MVTools using script which should create problems with SetMTMode if there are any to be seen?

Edit: Ah, I see you just have. Will use that, thanks. Be back in a few minutes.
mroz is offline   Reply With Quote
Old 18th October 2007, 00:24   #627  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Here's my script:
Code:
SetMTMode(2,0)

DGDecode_mpeg2source("E:\Work\test\FM.d2v", info=3)
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=1)

__film = last
__t0 = __film.trim(120776, 121000)
__t0

Distributor()
For the comparison I comment out the first & last lines.

I've ran the above & then repeated with a different clip from the same dvd.

The first gives me a speedup of factor 3.6395 & the second, 3.6398 (around 24.5fps vs 6.5fps).

In each case, the pair of output files are identical.

Last edited by mroz; 18th October 2007 at 00:27.
mroz is offline   Reply With Quote
Old 18th October 2007, 05:20   #628  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,496
I ran a longer test with a very fast-moving scene and a very high threshold, still no difference except in speed with setmtmode.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 18th October 2007, 05:36   #629  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,173
Code:
global idx = 123
mt("""
global idx = idx + 1
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
""", overlap=8)
As MT "Eval"'s the command with a different input clip for each thread this should work as the Rand() version does without the indeterminacy. Also this probably wants a good amount of overlap so any motion vectors near the join won't get truncated.
IanB is offline   Reply With Quote
Old 18th October 2007, 16:04   #630  |  Link
Spuds
dumber every day
 
Spuds's Avatar
 
Join Date: Dec 2006
Location: Planet Earth
Posts: 154
I spent some time this AM with an VHS copy that had a horse running back and forth, good motion and start stop action for MV to chew on. Cut it down to 2500 frames (standard 720x480i) and encoded to an uncompressed format.

I ran the following

Code:
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
""")
last
in several variations, including idx set to 1, idx set to rand(). non mt mode, with setmtmode(2,2) and inside an MT wrapper. I then took each output and subtracted from the original non MT run of the above code.

In terms of speed:
  • non MT -- 9:19
  • setmtmode(2,2) -- 5:02
  • setmtmode(2,4) -- 4:45 (seen this before where this is faster on my core 2 duo
  • MT() -- 5:18
  • MT() idx = rand() -- 5:24
  • setmtmode(2,2) idx=rand() -- 5:01

Here are some examples of delta frames from the subtract operation.
SETMTMODE(2,2) idx=1

an occasional frame was bad

SETMTMODE(2,2) idx=rand()

same as above, maybe a few less frames but sporadic bad ones

mt() idx=1

This was a mess, 80% of the frames (bottom half) had deltas

mt() idx=rand()

This one was really strange, it was like watching a faint ghost image the entire video, this is just one frame I enhanced in photoshop so you could see all deltas. watching the subtract clip was like seeing an faint embossed image.

setmtmode(2,4) idx=1

lots of these to be found
Spuds is offline   Reply With Quote
Old 18th October 2007, 16:59   #631  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
How big is the source 2500 frame clip? Any chance you can make it available on some file sharing service?

What was the lossless format you encoded to?
mroz is offline   Reply With Quote
Old 18th October 2007, 18:11   #632  |  Link
Spuds
dumber every day
 
Spuds's Avatar
 
Join Date: Dec 2006
Location: Planet Earth
Posts: 154
I used the original lossless format called uncompressed RGB/YCbCr for my output The file is pretty large as is >1G

I did some more testing with the same video but this time ran it through yadif to get a progressive source to see it that made a difference in the above.

Doing the same experiment as above with a progressive source same number of input frames but double framerate. I could not find a delta with setmtmode and idx=1. MT with idx=1 was just a mess, MT with idx=rand was better but pretty much the same as with the interlaced source, lots of noise removal deltas between the frames.

I'll probably go back and do a longer progressive comparison with setmt and see if I can spot a problem.
Spuds is offline   Reply With Quote
Old 18th October 2007, 19:18   #633  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
Spuds, what is your full scipts, including source filter?
Is any differences with SetMTmode before source or after it?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 18th October 2007, 19:21   #634  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,857
SetMTMode should always be before the source is loaded, otherwise it won't work properly.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 18th October 2007, 22:15   #635  |  Link
Spuds
dumber every day
 
Spuds's Avatar
 
Join Date: Dec 2006
Location: Planet Earth
Posts: 154
Here is the script I was running .... obviously I commented and uncommented as required depending on what run I was doing. For kicks I ran it again and setmode(2,4), mt idx=1, mt idx=rand() all repeated as above, setmtmode(2,2) did not . Strange, the first time with setmtmode(2,2) there were 2 or 3 frames out of 2500 that were different, granted thats less than 1/10 of 1% but still.

Code:
setmtmode(2,2)
SetMemoryMax(1024) 
Plugin = "C:\Program Files\AviSynth 2.5\plugins\"
Scripts = "C:\Program Files\AviSynth 2.5\plugins\scripts\"
Videos = "C:\Documents and Settings\All Users\Documents\My Videos\"
#load_stdcall_plugin(plugin + "yadif.dll")

setmtmode(5)
avisource(videos + "colt test.avi")
#assumebff
#yadif(mode=1,order=-1)
setmtmode(2)

#mt("""
#idx=rand()
idx=1
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain1(backward_vec1,forward_vec1,thSAD=400,idx=idx)
#""")
I just ran the below script figuring two forward and backward vectors might be more interesting:
Code:
#setmtmode(2,2)
SetMemoryMax(1024) 
Plugin = "C:\Program Files\AviSynth 2.5\plugins\"
Scripts = "C:\Program Files\AviSynth 2.5\plugins\scripts\"
Videos = "C:\Documents and Settings\All Users\Documents\My Videos\"
#load_stdcall_plugin(plugin + "yadif.dll")

#setmtmode(5)
avisource(videos + "colt test.avi")
#assumebff
#yadif(mode=1,order=-1)
#setmtmode(2)

#mt("""
#idx=rand()
idx=1
backward_vec1 = MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec1 = MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = idx)
backward_vec2 = MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx)
forward_vec2 = MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = idx)
last.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=idx)
#""")
and found a couple of these in setmtmode(2,2) vs no mt mode

Last edited by Spuds; 19th October 2007 at 04:03. Reason: Update
Spuds is offline   Reply With Quote
Old 19th October 2007, 00:16   #636  |  Link
mroz
Registered User
 
mroz's Avatar
 
Join Date: Sep 2006
Posts: 201
Any chance you can compress your source & repeat the tests with that material? Then if they're still the same, which one would expect, upload the source somewhere?

I've only experimented with SetMTMode(2,0) but have now tried interlaced sources; still can't find a difference between SetMT & no SetMT.

/grasping at straws

Are you using the latest versions of the filters?

MT 0.7 1/3/7 and mvtools 1.8.4 22/8/7 here.
mroz is offline   Reply With Quote
Old 19th October 2007, 17:57   #637  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
what about setmtmode(3,2) ?
Fizick is offline   Reply With Quote
Old 19th October 2007, 19:40   #638  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
I've been thinking ... what's the point in multithreading for something like video processing filters? There is no finegrained parallelism (there might be inside the filter, but that's another story) and the amount of code which has to be replicated for everything to run in it's own process isn't that big a deal either.

Code wise starting extra processes and passing objects through shared memory is a little more work ... but you are freed from filters having to be thread safe.
MfA is offline   Reply With Quote
Old 19th October 2007, 22:06   #639  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,496
A complex script (such as anything with mv or mcbob) can easily consume massive amounts of memory, which limits what you can do with multiple processes. It's easy to run out with two script - sometimes just one, heh. The easy way of multiprocessing can already be done, but requires huge temporary lossless files, which makes a lot of people think twice; it's harder to make it work with stats files, bitrate distributions, and so on.

Squid_80 is making a multiprocess version of avisynth to handle 64/32 bit, which could likely be extended to multiple 32-bit processes more performant than tcpserver.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 20th October 2007, 00:18   #640  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
If all the actual framebuffers are shared through shared memory (memory mapped pagefile in windows) the only extra memory you take compared to multithreading is basically an extra instance of the avisynth core for every process ... which in the grand scheme of things doesn't matter much.
MfA is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.