Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th November 2008, 09:11   #1001  |  Link
halsboss
likes to tinker
 
Join Date: Jan 2004
Location: girt by sea
Posts: 635
Thanks. The SetMTMode I'd thought just changed the mode back and forth (and I tended to have them at the start of each block of code, mostly snipped out of that sample)... and I was quite taken with the concept of MT, using it wherever I could since it was supposed to retain temporal position. Will give it a try. Still unsure why a scene change should cause the shudder.

Oh, and it used to be quite a speedy script too, by comparison Pity it didn't work as I'd hoped.
Code:
Source Frames: 4355
original (borked) MT script best profile,fox1 matrix, encoding time: 0:02:56 (176.41 s)  fps: 24.7
non-MT                 script best profile,fox1 matrix, encoding time: 0:03:56 (236.00 s)  fps: 18.5
merlin7777             script best profile,fox1 matrix, encoding time: 0:03:24 (204.48 s)  fps: 21.3

Last edited by halsboss; 20th November 2008 at 09:40.
halsboss is offline   Reply With Quote
Old 20th November 2008, 10:48   #1002  |  Link
Adub
Fighting spam with a fish
 
Adub's Avatar
 
Join Date: Sep 2005
Posts: 2,685
Did my edit fix your problem? If it did, you may be able to squeeze some more speed out of that script, with some minor tweaks here and there.
__________________
FAQs:Bond's AVC/H.264 FAQ
Site:Adubvideo
Adub is offline   Reply With Quote
Old 20th November 2008, 11:05   #1003  |  Link
halsboss
likes to tinker
 
Join Date: Jan 2004
Location: girt by sea
Posts: 635
Yes, thanks Merlin777, it did appear to fix it.

Here's an example of the issue showing out of order frames by a scene change, for that possible bug in MT/HCenc (it's possible it is the script, though... anyone spot the script error?)
3 images with MT
http://rapidshare.com/files/165589313/mt2420.jpg
http://rapidshare.com/files/165589314/mt2421.jpg
http://rapidshare.com/files/165589315/mt2422.jpg




and 3 images with NO MT
http://rapidshare.com/files/165589316/nomt2420.jpg
http://rapidshare.com/files/165589317/nomt2421.jpg
http://rapidshare.com/files/165589318/nomt2422.jpg



Last edited by halsboss; 20th November 2008 at 11:56.
halsboss is offline   Reply With Quote
Old 26th November 2008, 22:30   #1004  |  Link
seggitek
Interested Newbie
 
Join Date: Jun 2007
Location: Sopron, Hungary
Posts: 37
Hello guys. I'm trying to figure out this MT thingy.

Quote:
Originally Posted by tsp View Post
it is rather safe because only one thread is used so it makes no sense only to use mode 5 alone but it should be used for only part of the script that doesn't work with lower modes like mvtools.
Quote:
Originally Posted by AviSynth-wiki
Since rev 0.5, MT() internally calls SetMTMode(5) and restores the old mode afterwards so you don't have to add SetMTMode(5) before MT() and restore the old mode after
Does this mean that the original MT functionality (which splits a frame up into several pieces for several threads) is no longer available?



Could this be the reason, why I experience absolutely no speedup when encoding to XviD (using VirtualDubMod)?
seggitek is offline   Reply With Quote
Old 26th November 2008, 23:07   #1005  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
If your filter chain is just mt("blur(1)",threads=2), you shouldn't expect any realistic speedup. It runs so fast on its own that MT'ing it provides no practical benefit.
__________________
You can't call your encoding speed slow until you start measuring in seconds per frame.
Sagekilla is offline   Reply With Quote
Old 26th November 2008, 23:11   #1006  |  Link
Adub
Fighting spam with a fish
 
Adub's Avatar
 
Join Date: Sep 2005
Posts: 2,685
The split frame processing still exists. That's what MT does. SetMTMode I believe splits up by whole frames.
__________________
FAQs:Bond's AVC/H.264 FAQ
Site:Adubvideo
Adub is offline   Reply With Quote
Old 26th November 2008, 23:22   #1007  |  Link
seggitek
Interested Newbie
 
Join Date: Jun 2007
Location: Sopron, Hungary
Posts: 37
Quote:
Originally Posted by Sagekilla View Post
If your filter chain is just mt("blur(1)",threads=2), you shouldn't expect any realistic speedup. It runs so fast on its own that MT'ing it provides no practical benefit.
Hehe sorry. That is not my real filter chain. The image is copied from the AviSynth wiki. My filter chain is this one:

Code:
# Number of threads for multithreaded encoding
mt_count = 4

# PLUGINS
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\MaskTools.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\Repair.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\RemoveGrain.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\FFT3DFilter.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\mvtools\mvtools.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\ChromaShift.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\VInverse.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\gradfun2db.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\mt_masktools-25.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\TTempSmooth.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\AddGrainC.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\Cnr2.dll")
LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\TIVTC.dll")
#LoadPlugin("E:\MyProgies\Ripping\GordianKnot\AviSynthPlugins\VScope.dll")

# IMPORT
Import("E:\MyProgies\Ripping\GordianKnot\AviSynthIncludes\MCTemporalDenoise.v1.0beta2.avsi")
Import("E:\MyProgies\Ripping\GordianKnot\AviSynthIncludes\LimitedSharpenFaster.avsi")
Import("E:\MyProgies\Ripping\GordianKnot\AviSynthIncludes\GrainFactory3.avsi")
Import("E:\MyProgies\Ripping\GordianKnot\AviSynthIncludes\FixChromaBleeding.avsi")

# Load AVI source clip
AviSource("J:\Ripping\MasqueOfTheRedDeath-1989\capture.avi", true, "RGB24", fourCC="HFYU")

# IVTC
ConvertToYV12()
tfm(pp=7, cthresh=8, chroma=true).tdecimate()
Vinverse()

# Shift chroma info left by 2 pixels
MT("ChromaShift(C=-2)", mt_count, 2)

# DENOISING
MT("FFT3DFilter(sigma=3,plane=3,bt=5)", mt_count, 24)
MCTemporalDenoise(settings="medium",chroma=true,stabilize=true,GPU=false)
MT("""Cnr2("xxx", 4, 5, 255)""", mt_count, 24)   # Remove chroma noise

# Trimming & Cropping
crop(0, 6, -4, -0)
MT("Spline36Resize(512, last.height)", mt_count)
MT("Spline36Resize(last.width, 384)", mt_count, splitvertical=true)

# Correct Basic Video settings
MT("""ColorYUV(off_y=-15, gain_y=+22, cont_v=-12.8, cont_u=-51.2, opt="coring")""", mt_count)
MT("Tweak(bright=0, cont=1, hue=+4.5, sat=0.95, coring=false)", mt_count)

# Add grain
MT("GrainFactory3(g1str=4, g2str=5, g3str=16, g1shrp=30, g2shrp=40, g3shrp=50)", mt_count, 24)
MT("FadeIO2(180)", mt_count)
Quote:
Originally Posted by Merlin7777 View Post
The split frame processing still exists. That's what MT does. SetMTMode I believe splits up by whole frames.
I thought so too, but then I read this:

Quote:
Originally Posted by AviSynth-wiki
Since rev 0.5, MT() internally calls SetMTMode(5) and restores the old mode afterwards so you don't have to add SetMTMode(5) before MT() and restore the old mode after
And as we know from the AviSynth wiki page:

Quote:
Originally Posted by AviSynth-wiki
SetMTmode: ... to enable temporal (that is more than one frame is processed at the same time) multithreading. Mode 5 is the slowest (slower than not using SetMTMode) but should work with all filters that don't require linear frameserving (that is the frames come in order: frame 0,1,2,...,last).
Which should mean this, right?



That's why I am confused.
seggitek is offline   Reply With Quote
Old 27th November 2008, 04:42   #1008  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
You do know you can do a huge chunk of code with triple quotes for MT, right?
__________________
You can't call your encoding speed slow until you start measuring in seconds per frame.
Sagekilla is offline   Reply With Quote
Old 27th November 2008, 10:00   #1009  |  Link
seggitek
Interested Newbie
 
Join Date: Jun 2007
Location: Sopron, Hungary
Posts: 37
Quote:
Originally Posted by Sagekilla View Post
You do know you can do a huge chunk of code with triple quotes for MT, right?
Yes, I saw that opportunity. I can only use that if I need the same amount of frame overlapping for every filter. Which is not the case.

I'm still thinking that MT does not do anything. It calls SetMTMode(5) which is the slowest solution, even slower than not using MT. What sense does this make?

Here are some encoding times for the above script:

VirutalDubMod/Xvid using no MT at all: 18 hours
VirutalDubMod/Xvid using MT() only (= above script): 18 hours
VirutalDubMod/Xvid using SetMTMode(2) only: 16 hours



Can you verify my following thoughts?

AviSynth multithreading has basically nothing to do with multithreading XviD or x264.

You can do one of the following 4:

1.) NO Avisynth MT + NO XviD MT (standard encoding)
2.) Avisynth MT + NO XviD MT
3.) NO Avisynth MT + XviD MT
4.) Avisynth MT + XviD MT

NO Avisynth MT + XviD MT is basically the same as using Avisynth MT + NO XviD MT with SetMTMode() in AviSynth, because they both result in temporal multithreading.

AviSynth MT + XviD MT is quite nonsense, because it uses multiple threads for creating XviD frames whereas every XviD thread in addition is split up into another threads at the level of AviSynth.

Last edited by seggitek; 27th November 2008 at 11:45.
seggitek is offline   Reply With Quote
Old 29th November 2008, 14:45   #1010  |  Link
seggitek
Interested Newbie
 
Join Date: Jun 2007
Location: Sopron, Hungary
Posts: 37
Quote:
Originally Posted by seggitek View Post
Here are some encoding times for the above script:

VirutalDubMod/Xvid using no MT at all: 18 hours
VirutalDubMod/Xvid using MT() only (= above script): 18 hours
VirutalDubMod/Xvid using SetMTMode(2) only: 16 hours

After reading the first about 10 pages of this thread I found the problem. I didn't know that you have to call SetMTMode() at the very beginning of your script even if you are only using MT().

Now it looks like this:

VirutalDubMod/Xvid using no MT at all: 18 hours
VirutalDubMod/Xvid using MT() only (= above script): 16 hours
VirutalDubMod/Xvid using SetMTMode(2) only: 16 hours

Speedup is about 10%. Quite dissapointing. Maybe something is still wrong.
seggitek is offline   Reply With Quote
Old 29th November 2008, 14:49   #1011  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Hollola, Finland
Posts: 4,909
You don't have to call SetMTMode if you use only MT().
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 29th November 2008, 15:54   #1012  |  Link
seggitek
Interested Newbie
 
Join Date: Jun 2007
Location: Sopron, Hungary
Posts: 37
Quote:
Originally Posted by Boulder View Post
You don't have to call SetMTMode if you use only MT().
That would make sense. Thanks you. So this limited speedup is all, because of XviD?
seggitek is offline   Reply With Quote
Old 29th November 2008, 18:04   #1013  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
It's possible. You have some really slow filters running though (FFT3DFilter, MCTemporalDenoise, GrainFactory). There's also a lot of temporal dependenices for the first two, so MT'ing them efficiently is difficult.
__________________
You can't call your encoding speed slow until you start measuring in seconds per frame.
Sagekilla is offline   Reply With Quote
Old 29th November 2008, 22:24   #1014  |  Link
seggitek
Interested Newbie
 
Join Date: Jun 2007
Location: Sopron, Hungary
Posts: 37
Quote:
Originally Posted by Sagekilla View Post
There's also a lot of temporal dependenices for the first two, so MT'ing them efficiently is difficult.
I'm currenctly encoding with HuffYUV and it's as "slow" as XviD. This should mean that it has nothing to do with XviD but with my filter chain's "MTability".
seggitek is offline   Reply With Quote
Old 2nd December 2008, 14:18   #1015  |  Link
halsboss
likes to tinker
 
Join Date: Jan 2004
Location: girt by sea
Posts: 635
Help ! MTi giving me unexpected results

When I use this script and alternate between the MTi line and the one under it, I get different results with what HCEnc thinks is the vertical size !.
Code:
SetMTmode(mode=5,threads=1) # start with mode=5 forAVIsource http://forum.doom9.org/showthread.php?p=1067216#post1067216 
SetMemoryMax(256)  
AviSource("G:\HDTV\capt\capt-type1.08-11-30_20-39.00.avi", audio=false) 
AssumeFPS(25) 
AssumeBFF()  
original=LAST 
# snipped processing , just for testing

# for MTi info refer http://avisynth.org/mediawiki/MT#Technical_info
MTi("original").AssumeBFF() #HC tells me this is 720x1152
#original.AssumeBFF()       #HC tells me this is 720x576

Converttoyv12(interlaced=true) 
SetPlanarLegacyAlignment(True) 
Distributor() # use this when using HC and SetMTmode, per http://forum.doom9.org/showthread.php?p=1063622#post1063622
The doco at http://avisynth.org/mediawiki/MT#Technical_info says
Quote:
* MTi() that creates two threads and let each thread process one field before combining them like this avs function
Code:
function PseudoMTi(clip c,string filter)
{
a=eval("c.AssumeFieldBased().SeparateFields.selecteven()."+filter)
b=eval("c.AssumeFieldBased().SeparateFields.selectodd()."+filter)
interleave(a,b).weave()
}
like the other pseudoscript a and b are executed in parallel. Note that only two threads are created so it will only use two (virtual) cores.

What am I doing wrong ?
halsboss is offline   Reply With Quote
Old 2nd December 2008, 14:35   #1016  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,405
Your "original" is not a filter, but a variable.

Replace "original=LAST" by:
Code:
function original(clip c) { return c }
Gavino is offline   Reply With Quote
Old 2nd December 2008, 15:03   #1017  |  Link
halsboss
likes to tinker
 
Join Date: Jan 2004
Location: girt by sea
Posts: 635
Oh. Thanks. Swapping between the two L1 lines confirmed I get different size results for each.
Code:
original=LAST 
L1 = MTi("original.MVDegrain1i2_insideMTi(doAGC=FALSE)")  
#L1 = MTi("last.MVDegrain1i2_insideMTi(doAGC=FALSE)")  
L1
function AGCiEmbedded(clip srcclp) { 
   inpclp = srcclp 
   #inpclp=inpclp.HDRAGC(coef_gain=1.0, coef_sat=1.0) # default. +coef_gain=brighter decrease to limit. +coef_sat=more saturation  
   #inpclp=inpclp.HDRAGC(coef_gain=0.1, coef_sat=1.0) # default. +coef_gain=brighter decrease to limit. +coef_sat=more saturation  
   #inpclp=inpclp.HDRAGC(coef_gain=0.2, coef_sat=1.0, corrector=0.8, reducer=2.0) # default. +coef_gain=brighter decrease to limit. +coef_sat=more saturation  
   #inpclp=inpclp.HDRAGC(coef_gain=0.2, min_gain=0.1, max_gain=1.0, coef_sat=1.0, corrector=0.8, reducer=2.0) # default. +coef_gain=brighter decrease to limit. +coef_sat=more saturation  
   #inpclp=inpclp.HDRAGC(coef_gain=0.2, min_gain=0.1, max_gain=1.0, coef_sat=1.0, corrector=0.8, reducer=2.0, black_clip=0.5) # default. +coef_gain=brighter decrease to limit. +coef_sat=more saturation  
   inpclp=inpclp.HDRAGC(coef_gain=0.1, min_gain=0.1, max_gain=0.5, coef_sat=0.75, corrector=0.8, reducer=2.0, black_clip=1.0) 
   #inpclp=inpclp.HDRAGC(coef_gain=0.1, min_gain=0.1, max_gain=0.5, coef_sat=1.0, corrector=0.8, reducer=2.0, black_clip=1.0) 
   #inpclp=inpclp.HDRAGC(coef_gain=0.1, min_gain=0.2, max_gain=1.0, coef_sat=1.0, corrector=0.8, reducer=2.0, black_clip=1.0) 
   return inpclp 
} 
function MVDegrain1i2_insideMTi(clip srcclp, int "blksize", int "overlap", int "dct", bool "doAGC") { 
   doAGC=default(doAGC,TRUE) 
# AssumexFF() must have been done prior to calling this function 
   blksize=default(blksize,8) # # 4, 8 or 16 ( default is 8 ). Larger blocks are less sensitive to noise, are faster, but also less accurate. 
   overlap=default(overlap,2) # overlap value (0 to 4 for blksize=8) Must be even and less than block size 
   dct=default(dct,0) # use dct=1 for clip with light flicker 
   fields=srcclp # FIELDS ALREADY SEPARATED BY MTi SO DON'T DO HERE 
   super = fields.MVSuper(pel=2, chroma=true, sharp=2) # half-pixel accuracy of the motion estimation 
   backward_vec2 = super.MVAnalyse(isb = true, delta = 1, blksize=blksize, overlap=overlap, dct=dct, chroma=true) 
   forward_vec2 = super.MVAnalyse(isb = false, delta = 1, blksize=blksize, overlap=overlap, dct=dct, chroma=true) 
   fields=fields.MVDegrain1(super, backward_vec2,forward_vec2,thSAD=400,plane=4) 
   doAGC ? fields.AGCiEmbedded() : fields 
   # FIELDS WILL BE RE-WEAVED by MTi SO DON'T DO IT HERE 
}
I'd thought "original=LAST" was creating a clip variable I could use, like the "L1=" lines were... Can you please point me somewhere which explains the difference ? as they look the same to me.
halsboss is offline   Reply With Quote
Old 2nd December 2008, 15:33   #1018  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,405
Quote:
Originally Posted by halsboss View Post
Oh. Thanks. Swapping between the two L1 lines confirmed I get different size results for each.
Code:
original=LAST 
L1 = MTi("original.MVDegrain1i2_insideMTi(doAGC=FALSE)")  
#L1 = MTi("last.MVDegrain1i2_insideMTi(doAGC=FALSE)")  
L1
I'd thought "original=LAST" was creating a clip variable I could use, like the "L1=" lines were...
It does, but the string parameter to MTi has to represent a filter called without an explicit clip and is evaluated inside Mti.
Both your L1 lines are wrong, the second one works by accident because "last" is not the value at the point of calling MTi, but the value inside MTi when the string is evaluated.

What you actually want is
Code:
L1 = original.MTi("MVDegrain1i2_insideMTi(doAGC=FALSE)")
Gavino is offline   Reply With Quote
Old 3rd December 2008, 12:58   #1019  |  Link
halsboss
likes to tinker
 
Join Date: Jan 2004
Location: girt by sea
Posts: 635
Yes, thankyou, that seems to clear that up. Now for a strange one...

In the script below, the L1, L2, L3 items work fine (un-commenting out , re-commenting-out) but L4 doesn't and throws the error message
Quote:
Script Error: Invalid arguments to function "HDRAGC"
I hadn't expected that error, since L3 worked OK and is functionally equivalent, and L2 works and is functionally equivalent too. Any ideas on why the error occurs ?

Code:
original=LAST
#L1 = original.SeparateFields().HDRAGC(coef_gain=0.1, min_gain=0.1, max_gain=0.5, coef_sat=0.75, corrector=0.8, reducer=2.0, black_clip=1.0).Weave()
#L2 = original.MTi("HDRAGC(coef_gain=0.1, min_gain=0.1, max_gain=0.5, coef_sat=0.75, corrector=0.8, reducer=2.0, black_clip=1.0)")
#L3 = original.SeparateFields()
#L3e = L3.SelectEven().AGCiEmbedded()
#L3o = L3.SelectOdd().AGCiEmbedded()
#L3 = Interleave(L3e,L3o).Weave()
L4 = original.MTi("AGCiEmbedded()")
#L1
#L2
#L3
L4
function AGCiEmbedded(clip srcclp) { 
   inpclp = srcclp 
   # other processing
   inpclp=inpclp.HDRAGC(coef_gain=0.1, min_gain=0.1, max_gain=0.5, coef_sat=0.75, corrector=0.8, reducer=2.0, black_clip=1.0) 
   return inpclp 
}

Last edited by halsboss; 3rd December 2008 at 13:21.
halsboss is offline   Reply With Quote
Old 3rd December 2008, 15:09   #1020  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,405
Very odd. I too would have expected it to work.

What happens if you change the function to
Code:
function AGCiEmbedded(clip srcclp) { 
   return srcclp.HDRAGC(coef_gain=0.1, min_gain=0.1, max_gain=0.5, coef_sat=0.75, corrector=0.8, reducer=2.0, black_clip=1.0) 
}
(Clearly this should make no difference but something strange is happening somewhere.)
Gavino is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:35.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.