Log in

View Full Version : QuEnc 2 Pass Encoding of Slow AVS file

Gerard V
25th June 2006, 05:37
I have used Avisynth to create a video of a show. The script combines edits from 3 video cameras, and does a bit of filtering, some resizing and an amount of denoising. As a result, it processes quite slowly at around 2 or 3 fps if I use virtualdub with a total processing time of approx 25 hours.

I am encoding it to MPEG for a DVD using QuEnc. Since QuEnc takes avs as input, I have fed it the avs for the show. Am I right in thinking that since QuEnc is set to 2 pass encoding, it will read and process the entire AVS script two times (i.e approx 50 hours)?

If that is true, would it have been a better idea to process it once using VirtualDub; save as AVI using a lossless codec such as huffyuv, and then using QuEnc to encode the resulting AVI file (with a 1 line avs script and directshowsource)?

Or asking all that another way, is it considered good practice to create an intermediate AVI file to encode to MPEG with QuEnc if 2 pass encoding is used and the Avisynth script is very slow to process?

And should I have asked this in another forum topic (which one)?
Pookie
25th June 2006, 06:07
That would still make your encode 75 hours long (or in the vicinity of) as you'd still want to do 2-pass on the intermediate AVI for the encoder to determine the optimal bitrate on the frames. Besides, if the AVI gets to be over 2GB in size, you can start running into problems with certain video editing and playback applications. Lagarith is a good losseless codec to use if you ever do need to do this as it compresses somewhat better than Huffy, thus improving your chances of ending up with an AVI less that 2GB in size - normally, you'd only want to do the intermediate AVI file in a case where the encoder won't accept AVIsynth scripts as input.

I'm guessing your bottleneck is the placement of your filters in your script. Post your AviSynth script and I'm sure people will suggest ways to improve performance.:D
foxyshadis
25th June 2006, 06:30
No, 2passing an excurciatingly slow avs script is much slower than 2passing the huffyuv. Since there's some overhead writing and reading the huffyuv, the speedup is less than 50%, but it's close if the script is slow enough.

Let's say the quenc 1st pass takes twenty minutes and second pass takes 10 hours with pretty heavy settings.
Total time normally: 25hrs + 35hrs = 60hrs.
Total time w/ lossless avi: 25hrs + 0hrs + 10 hrs = 35hrs.
Throw in huffy decoding time and you might raise that by an hour or so.

But optimizing such a slow avisynth script is always a good start.
Pookie
25th June 2006, 07:02
I stepped out to get a slice of Pizza and redid the numbers on the walk. Foxyshadis' estimates are more accurate:o
foxyshadis
25th June 2006, 07:30
I hate that "Hmm, wait, what about.... Oh, shoot." feeling when you're off somewhere. :p
Pookie
25th June 2006, 08:10
No doubt. The thing that makes the temporary AVI option impractical in this case is the massive amount of disk space the Huffyuv file(s) would require.
Gerard V
25th June 2006, 08:41
Thanks for the answers above. I’m thinking the intermediate file option will be better, as once the AVI file is created, the slow filters will not be called for a second time.

I stepped out to get a slice of Pizza and redid the numbers on the walk. Foxyshadis' estimates are more accurate

If I understand the math and assume the encoding to huffyuv takes place at about 15fps on my system which is a gut feeling average, but processing the AVS script occurs at 2.5fps for a 2 hour video, then the math seems to me to be:

Without intermediate AVI file
AVS -> MPEG with QuEnc 2 Pass, Pass1 20 hours, Pass2 20 hours plus any QuEnc encoding overhead Total 40 hours plus.

With intermediate AVI file and, say, huffyuv for example
AVS -> AVI with VirtualDub and HuffYuv 20 hours plus
AVI -> MPEG with QuEnc approx 4 hours for each pass = 8 hours, total 28+

But if I could speed up the script – that’d be great. I new at this so there’ll be lots to improve I guess.

I'm guessing your bottleneck is the placement of your filters in your script. Post your AviSynth script and I'm sure people will suggest ways to improve performance.

I’ve posted the script below. But it’s very long so I have left off some of the subscripts. The “cam1_raw.avs” and similar scripts merely take the various files captured from DV for each camera and splice them together into one single video file, with BlankClip added at the start, end and during any tape changes so that the frame number on cam1_raw.avs corresponds to the same frame on cam2_raw.avs and podc_raw.avs for ease of editing.

Cam1_Raw.avs returns 1440 x 1080i via Mpeg2Source from a Sony FX1e HDV Cam
Cam2_Raw.avs returns 720 x 576 via Mpeg2Source from a borrowed Sony PAL hard drive camera
PodC_Raw.avs returns 720 x 576 via AViSOurce froma 3rd Sony DV Camera

I could process the whole script in QuEnc fast enough until I noticed the background was a bit jerky during fast motions when the camera panned with one of the actors. I figured that there was too much noise in the background and much of my MPEG bitrate was wasted coding noise, so I added in the DegrainMedian and Peachsmoother to the function HDV2WS which accounts for most of the video output. This is what has slowed it down to a crawl.

The output is intended to be widescreen hence the cropping etc. Coding to Mpeg2 with the 16:9 option – plays nicely on the standalone DVD player and widescreen TV but there’s that jerkyness I noted.

# THS3_FunctionsDsmooth

# Created by AVSEdit
# Gerard 16/06/2006

function HDV2WS(clip V)

{
AssumeBFF(v)
SeparateFields()
ConvertToYUY2
Degrainmedian(limitY=5,limitUV=7,mode=0)
PeachSmoother(noiseReduction=30, stability=20, spatial=100, dot=FALSE)
ColorYUV(autogain=true)
BicubicResize(720,288)
ConvertToYV12
Odd=SelectOdd()
Evn=SelectEven()
Interleave(odd,Evn)
Weave()
}

function HDV2WSzoom(clip V, int topx, int topy, int wid)

{
topx = Ceil(topx/4)*4
topy = Ceil(topy/4)*4
wid = Ceil(wid/4)*4

hi = Ceil(wid*9/16/4)*4
botx = topx+wid-1440
boty = topy+hi-1080

topy = topy/2
boty = boty/2

AssumeBFF(v)
SeparateFields()
Crop(topx,topy,botx,boty)
# Subtitle(string(topx)+" "+string(topy)+" "+string(botx)+" "+string(boty)+" "+string(wid))
BicubicResize(720,288)
Odd=SelectOdd()
Evn=SelectEven()
Interleave(odd,Evn)
Weave()
}

function PAL2WS(clip V, int topy)

{
AssumeBFF(v)
topy = int((576-408)*topy/400)*2
boty = int((576-topy-408)/4)*2
DGbob(0)
Crop(0,topy*2,0,-boty*2)
Degrainmedian(limitY=5,limitUV=7,mode=0)
Spline36Resize(720,576)
ConvertToYUY2
PeachSmoother(noiseReduction=30, stability=20, spatial=100, dot=FALSE)
ConvertToYV12
AssumeBFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
}

function PAL2WS2(clip V, int topy)

{
AssumeBFF(v)
topy = int((576-408)*topy/400)*2
boty = int((576-topy-408)/4)*2
DGbob(0)
Crop(0,topy*2,0,-boty*2)
Degrainmedian(limitY=5,limitUV=7,mode=0)
Spline36Resize(720,576)
ConvertToYUY2
PeachSmoother(noiseReduction=30, stability=20, spatial=100, dot=FALSE)
ConvertToYV12
AssumeBFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
}


# Created by AVSEdit
# Gerard V 16/06/2006

SetMemoryMax(512)

Import("D:\Video\Edits\THS3\THS3_Chapterinfo.avs") # Merely defines H010s H020s etc.
Import("D:\Video\Edits\THS3\THS3_FunctionsDsmooth.avs")

C1 = Import("D:\Video\Edits\THS3\Cam1_Raw.avs").KillAudio()
C2 = Import("D:\Video\Edits\THS3\Cam2_Raw.avs").KillAudio()
PC = Import("D:\Video\Edits\THS3\PodC_Raw.avs").KillAudio()
A = WavSource("G:\THS3\THS3_AudioMixV2.wav")

C1 = AudioDub(C1,A)
C2 = AudioDub(C2,A)
PC = AudioDub(PC,A)


PartX = C1.Trim(H000s, H000e).HDV2WS()

Part0 = C1.Trim(H010s, H010s+470).HDV2WS().FadeIO0(20)
Part0Cut = C1.Trim(H010s+470,H010e).HDV2WS()

Part = C2.Trim( H020s, 23920).PAL2WS(50)
Part = Part + C1.Trim( 23921, 24720).HDV2WS()
Part = Part + PC.Trim( 24721, 24800).PAL2WS(100)
Part = Part + C1.Trim( 24801, 24900).HDV2WS()
Part = Part + PC.Trim( 24901, 24950).PAL2WS(100)
Part = Part + C1.Trim( 24951, 25930).HDV2WS()
Part1 = Part

Part = C2.Trim(025931,026010).PAL2WS2(000)
Part = Part + C1.Trim(026011,026740).HDV2WS()
Part = Part + C2.Trim(026741,027025).PAL2WS2(040)
Part = Part + C1.Trim(027026,031010).HDV2WS()
Part = Part + PC.Trim(031011,031080).PAL2WS(050)
Part = Part + C2.Trim(031081,031290).PAL2WS2(050)
Part = Part + C1.Trim(031291,032230).HDV2WS()
Part = Part + C2.Trim(032231,032390).PAL2WS2(050)
Part = Part + C1.Trim(032391,033390).HDV2WS()
Part = Part + PC.Trim(033391,033800).PAL2WS(025)
Part = Part + C1.Trim(033801,034220).HDV2WS()
Part = Part + PC.Trim(034221,034550).PAL2WS(050)
Part = Part + C1.Trim(034551,035080).HDV2WS()
Part = Part + PC.Trim(035081,035200).PAL2WS(050)
Part = Part + C1.Trim(035201,036510).HDV2WS()
Part = Part + C2.Trim(036511,036820).PAL2WS2(020)
Part = Part + C1.Trim(036821,037660).HDV2WS()
Part2 = Part


Part = C1.Trim(037661,042050).HDV2WS()
Part = Part + C1.Trim(042051,042390).HDV2WS()
Part = Part + C1.Trim(042391,043330).HDV2WS()
V = C2.Trim(043331,043420)
V = v.Animate(1,Framecount(V), "PAL2WS2", 0, 50)
Part = Part + V
Part = Part + C2.Trim(043421,043530).PAL2WS2(050)
Part = Part + C1.Trim(043531,044040).HDV2WS()
Part3 = Part

Part4Ind = C1.Trim(044041,056530).HDV2WS()

Part = C1.Trim(056531,058170).HDV2WS()
Part = Part + C1.Trim(058171,058580).HDV2WSzoom(360,270,720)
Part = Part + C1.Trim(058581,058690).HDV2WS()
Part = Part + C2.Trim(058691,058920).PAL2WS2(000)
Part = Part + C2.Trim(058921,059550).PAL2WS2(050)
Part = Part + C1.Trim(059551,063480).HDV2WS()
Part = Part + C2.Trim(063481,063770).PAL2WS2(000)
Part = Part + C1.Trim(063771,064300).HDV2WS()
Part = Part + C2.Trim(064301,064400).PAL2WS2(000)
Part = Part + C1.Trim(064401,065410).HDV2WS()
Part5 = Part

Part = C1.Trim(065411,065710).HDV2WS()
Part = Part + C2.Trim(065711,066130).PAL2WS2(000)
Part = Part + C1.Trim(066131,067310).HDV2WS()
Part = Part + C2.Trim(067311,067460).PAL2WS2(050)
Part = Part + C1.Trim(067461,070570).HDV2WS()
Part = Part + C2.Trim(070571,070600).PAL2WS2(050)
Part = Part + C1.Trim(070601,071800).HDV2WS()
Part = Part + C2.Trim(071801,071890).PAL2WS2(075)
Part = Part + C1.Trim(071891,075100).HDV2WS()
Part6 = Part

Part7 = C1.Trim(075101,078600).HDV2WS()
Part8 = C1.Trim(078601,080300).HDV2WS()

Part = C1.Trim(080301,086570).HDV2WS()
Part = Part + PC.Trim(086571,086810).PAL2WS(050)
Part = Part + C1.Trim(086811,088660).HDV2WS()
Part = Part + C2.Trim(088661,088740).PAL2WS2(080)
Part = Part + PC.Trim(088741,089200).PAL2WS(100)
Part = Part + C1.Trim(089201,089530).HDV2WS()
Part9 = Part

Part = C1.Trim(089531,098720).HDV2WS()
Part = Part + PC.Trim(098721,098980).PAL2WS(050)
Part = Part + C1.Trim(098981,100300).HDV2WS()
Part = Part + PC.Trim(100301,100350).PAL2WS(030)
Part = Part + C1.Trim(100351,101220).HDV2WS()
Part = Part + C1.Trim(101221,101850).HDV2WSzoom(180,370,1080)
Part = Part + C1.Trim(101851,108340).HDV2WS()
Part = Part + C2.Trim(108341,108760).PAL2WS2(000)
Part = Part + C1.Trim(108761,110540).HDV2WS()
Part = Part + PC.Trim(110541,110640).PAL2WS(080)
Part = Part + C1.Trim(110641,110800).HDV2WS()
Part = Part + C2.Trim(110801,111050).PAL2WS2(010)
Part = Part + C1.Trim(111051,111700).HDV2WS()
Part10 = Part

Part = C1.Trim(111701,112880).HDV2WS()
Part = Part + C1.Trim(112881,113120).HDV2WSzoom(360,270,720)
Part = Part + C2.Trim(113121,113380).PAL2WS2(040)
Part = Part + C1.Trim(113381,113660).HDV2WS()
Part = Part + C2.Trim(113661,115020).PAL2WS2(050)
Part = Part + C1.Trim(115021,115320).HDV2WS()
Part = Part + C1.Trim(115321,115740).HDV2WSzoom(576,216,864)
Part = Part + C1.Trim(115741,117360).HDV2WS()
Part = Part + PC.Trim(117361,117400).PAL2WS(070)
Part = Part + C1.Trim(117401,118790).HDV2WS()
Part = Part + C2.Trim(118791,118890).PAL2WS2(080)
Part = Part + PC.Trim(118891,119250).PAL2WS(050)
Part = Part + C1.Trim(119251,123430).HDV2WS()
Part = Part + C2.Trim(123431,123570).PAL2WS2(000)
Part = Part + C1.Trim(123571,124500).HDV2WS()
Part = Part + C2.Trim(124501,124920).PAL2WS2(050)
Part = Part + C1.Trim(124921,128700).HDV2WS()
Part11 = Part

Part = C1.Trim(128701,131180).HDV2WS()
Part = Part + C2.Trim(131181,131440).PAL2WS2(050)
Part = Part + C1.Trim(131441,132190).HDV2WSzoom(480,288,960)
Part = Part + C1.Trim(132191,132370).HDV2WS()
Part = Part + C2.Trim(132371,132480).PAL2WS2(060)
Part = Part + C2.Trim(132481,133290).PAL2WS2(070)
Part = Part + C1.Trim(133291,133380).HDV2WS()
Part = Part + C2.Trim(133381,133800).PAL2WS2(080)
Part = Part + C1.Trim(133801,134140).HDV2WS()
Part = Part + C2.Trim(134141,134220).PAL2WS2(090)
Part = Part + C1.Trim(134221,135860).HDV2WS()
Part = Part + C1.Trim(135861,135950).HDV2WS()
Part = Part + C1.Trim(135951,137790).HDV2WS()
Part12 = Part

Part13Brk = C1.Trim(137791,144230).HDV2WS()

Part = C1.Trim(144231,144650).HDV2WS()
Part = Part + PC.Trim(144651,144800).PAL2WS(100)
Part = Part + C1.Trim(144801,147870).HDV2WS()
Part = Part + C2.Trim(147871,148200).PAL2WS2(070)
Part = Part + PC.Trim(148201,148570).PAL2WS(050)
Part = Part + C1.Trim(148571,154000).HDV2WS()
Part = Part + C1.Trim(154001,154100).HDV2WSzoom(0,270,1080)
Part = Part + C1.Trim(154101,157460).HDV2WS()
Part = Part + C2.Trim(157461,157550).PAL2WS2(000)
Part = Part + C1.Trim(157551,157730).HDV2WS()
Part14 = Part

Part15Ind = C1.Trim(157731,164740).HDV2WS()

Part16 = C1.Trim(164741,168810).HDV2WS()

Part = C1.Trim(168811,170100).HDV2WS()
Part = Part + PC.Trim(170101,170220).PAL2WS(075)
Part = Part + C1.Trim(170221,174210).HDV2WS()
Part17 = Part

Part18 = C1.Trim(174211,181980).HDV2WS()

Part19Cut = C1.Trim(181981,182900).HDV2WS()

Part20 = C1.Trim(182901,185030).HDV2WS()

Part = C1.Trim(185031,185910).HDV2WS()
Part = Part + C2.Trim(185911,186250).PAL2WS2(000)
Part = Part + C2.Trim(186251,186500).PAL2WS2(100)
Part = Part + C1.Trim(186501,189360).HDV2WS()
Part21 = Part

Part = C1.Trim(189361,193480).HDV2WS()
Part = Part + C2.Trim(193481,193860).PAL2WS2(050)
Part = Part + C1.Trim(193861,194150).HDV2WS()
Part = Part + PC.Trim(194151,194300).PAL2WS(100)
Part = Part + C1.Trim(194301,195380).HDV2WS()
Part22 = Part

Part23 = C1.Trim(195381,198280).HDV2WS()

Part = C1.Trim(198281,199720).HDV2WS()
Part = Part + C2.Trim(199721,199840).PAL2WS2(020)
V = C2.Trim(199841,199860)
V = v.Animate(1,Framecount(V), "PAL2WS2", 20, 100)
Part = Part + V
Part = Part + C2.Trim(199861,199980).PAL2WS2(100)
Part = Part + PC.Trim(199981,200060).PAL2WS(050)
Part = Part + C1.Trim(200061,201260).HDV2WS()
Part = Part + PC.Trim(201261,201800).PAL2WS(100)
Part = Part + C1.Trim(201801,203510).HDV2WS()
V = PC.Trim(203511,203700)
V = v.Animate(1,Framecount(V), "PAL2WS", 50, 100)
Part = Part + V
Part = Part + C1.Trim(203701,205940).HDV2WS()
Part = Part + C2.Trim(205941,206340).PAL2WS2(100)
Part = Part + C1.Trim(206341,207340).HDV2WS()
Part24 = Part

Part = C1.Trim(207341,209490).HDV2WS()
Part = Part + C2.Trim(209491,209590).PAL2WS2(000).Subtitle("C2 1")
V = c2.Trim(209591,209680)
V = v.Animate(1,Framecount(V), "PAL2WS2", 0, 60).Subtitle("Zoom 1")
Part = Part + V
Part = Part + C2.Trim(209681,209840).PAL2WS2(060).Subtitle("C2 2")
Part = Part + C1.Trim(209841,211020).HDV2WS()
Part = Part + C1.Trim(211021,211350).HDV2WSzoom(360,596,720).Subtitle("Zoom 2")
Part = Part + C1.Trim(211351,211640).HDV2WSzoom(576,216,864).Subtitle("Zoom 3")
Part = Part + C1.Trim(211641,211980).HDV2WS()
Part = Part + C2.Trim(211981,212260).PAL2WS2(050)
Part = Part + C1.Trim(212261,212710).HDV2WS()
Part25 = Part

Part = C1.Trim(212711,220520).HDV2WS()
Part = Part + PC.Trim(220521,220700).PAL2WS(100)
Part = Part + C1.Trim(220701,223835).HDV2WS()
Part26 = Part.FadeOut(25)

AlignedSplice( Part0, Part1, Part2, Part3, Part5, Part6, Part7, Part8, Part9, Part10, Part11, Part12, \
Part14, Part16, Part17, Part18, Part20, Part21, Part22, Part23, Part24, Part25, Part26)
foxyshadis
25th June 2006, 12:01
Peachsmoother is probably like 80-90% of the avisynth speed, with decode, resize, and degrainmedian filling in the rest; so there's not much you can do if you prefer Peach. It's good, it's slow, and that's just how it goes... Must be one incredibly long startup time though.

First pass shouldn't be nearly as slow as second; especially on something this long (3-4 hours or so?) the second pass is probably going to be longer than longcat.

If you have the space and don't intend to need it long, it's easiest and quickest to just go with huffyuv (ffdshow's is a little smaller, but huffy works fine), otherwise there's always lagarith, near-lossless xvid, and other tricks to save space.
Gerard V
25th June 2006, 20:30
Pass 1 will have taken 30 hours by the time i get home tonight it'll have finished and pass 2 will have started. I think I'll just let it run, but next time I'll use an intermediate AVI file. Having let it run so long I am loath to stop it now in case I find myself back at square one. Hopefully this is the last time I'll process the video. Certainly I'll remember to only add the smoothing filters in as the last thing in future.

I tried a number of different filters and compared the results side by side. Peach gave the better result this time in my opinion. Even better with MVtools, but that crashed with an "exception" when I tried to process the whole video so I have put that aside for later investigation. But that would have been even slower still. :o