Oversizing and extremely slow second pass--Gordian Knot+VirtualDub

[sector9]

I'm using, as said, latest version of GK. I'm ripping and encoding some TV DVDs--if it matters, Babylon 5 and Star Trek. Very recently (I don't think I changed any settings--it's possible though) I started having a problem with oversizing--the second pass would result in final sizes 500 Mb-700 Mb (my target being 350 Mb.) I read the post about oversizing, and I will try the solution mentioned there, but I had another problem which wasn't mentioned there--the second passes on these rips that went wrong were obscenely slow--6-10 hours, when normally it can finish it in 1. Does anyone know why this is happening?

[unskinnyboy]

What are your PC specs? Are you using any filters (denoising, sharpening etc)?

[sector9]

PC -- 768 Mb ram, Athlon XP 2100+--like I said, it takes (normally) an hour for the second pass. Filters--I use what it says to in the guide, a "lanczos" filter for sharpness. I'm also using IVTC, if it matters.

[Didée]

Please post the used AviSynth script and XviD settings.

I do not believe that your machine could do one episode in 1 hour. Definetly not together with IVTC.

In contrary: doing IVTC, and letting XviD encode with *all* bells'n'whistles it has, an encoding time of 6 to 10 hours for the 2nd pass on a ~42min episode would be rather typical for your machine ...

[sector9]

No, it's one pass in an hour--about 2 hours for the episodes overall. And I'm quite sure about it--I've done it time and time again.
The first pass settings file:


"Supported_4CC"=dword:00000007
"mode"=dword:00000001
"credits_end"=dword:00000000
"credits_start"=dword:00000000
"motion_search"=dword:00000006
"chromame"=dword:00000001
"bitrate"=dword:000002bc
"desired_size"=dword:0008b290
"use_2pass_bitrate"=dword:00000000
"desired_quant"=dword:00000190
"quant_type"=dword:00000000
"lum_masking"=dword:00000000
"interlacing"=dword:00000000
"qpel"=dword:00000001
"gmc"=dword:00000000
"reduced_resolution"=dword:00000000
"use_bvop"=dword:00000001
"max_bframes"=dword:00000002
"bquant_ratio"=dword:00000096
"bquant_offset"=dword:00000064
"packed"=dword:00000001
"closed_gov"=dword:00000001
"ar_mode"=dword:00000000
"aspect_ratio"=dword:00000000
"par_x"=dword:00000001
"par_y"=dword:00000001
"ar_x"=dword:00000004
"ar_y"=dword:00000003
"num_zones"=dword:00000002
"rc_reaction_delay_factor"=dword:00000010
"rc_averaging_period"=dword:00000064
"rc_buffer"=dword:00000064
"discard1pass"=dword:00000001
"full1pass"=dword:00000000
"keyframe_boost"=dword:0000000a
"kfreduction"=dword:00000014
"kfthreshold"=dword:00000001
"curve_compression_high"=dword:00000000
"curve_compression_low"=dword:00000000
"overflow_control_strength"=dword:00000005
"twopass_max_overflow_improvement"=dword:00000005
"twopass_max_overflow_degradation"=dword:00000005
"container_type"=dword:00000001
"target_size"=dword:000a2800
"subtitle_size"=dword:00000000
"hours"=dword:00000001
"minutes"=dword:0000001e
"seconds"=dword:00000000
"fps"=dword:00000002
"audio_mode"=dword:00000000
"audio_type"=dword:00000000
"audio_rate"=dword:00000080
"audio_size"=dword:00000000
"vhq_mode"=dword:00000001
"cartoon_mode"=dword:00000000
"turbo"=dword:00000001
"max_key_interval"=dword:0000012c
"frame_drop_ratio"=dword:00000000
"min_iquant"=dword:00000001
"max_iquant"=dword:0000001f
"min_pquant"=dword:00000001
"max_pquant"=dword:0000001f
"min_bquant"=dword:00000001
"max_bquant"=dword:0000001f
"trellis_quant"=dword:00000000
"fourcc_used"=dword:00000000
"debug"=dword:00000000
"vop_debug"=dword:00000000
"display_status"=dword:00000001
"Deblock_Y"=dword:00000000
"Deblock_UV"=dword:00000000
"Dering"=dword:00000000
"FilmEffect"=dword:00000000
"profile"="AS @ L5"
"stats"="G:\\DVD\\Rips\\TNGS7D1_VTS_02_PGC4\\st4.stats"
"qmatrix_intra"=hex:08,11,12,13,15,17,19,1b,11,12,13,15,17,19,1b,1c,14,15,16,\
17,18,1a,1c,1e,15,16,17,18,1a,1c,1e,20,16,17,18,1a,1c,1e,20,23,17,18,1a,1c,\
1e,20,23,26,19,1a,1c,1e,20,23,26,29,1b,1c,1e,20,23,26,29,2d
"qmatrix_inter"=hex:10,11,12,13,14,15,16,17,11,12,13,14,15,16,17,18,12,13,14,\
15,16,17,18,19,13,14,15,16,17,18,1a,1b,14,15,16,17,19,1a,1b,1c,15,16,17,18,\
1a,1b,1c,1e,16,17,18,1a,1b,1c,1e,1f,17,18,19,1b,1c,1e,1f,21
"zone0_frame"=dword:00000000
"zone0_mode"=dword:00000000
"zone0_weight"=dword:00000064
"zone0_quant"=dword:000001f4
"zone0_type"=dword:00000000
"zone0_greyscale"=dword:00000000
"zone0_chroma_opt"=dword:00000000
"zone0_bvop_threshold"=dword:00000000
"zone1_frame"=dword:0000f71f
"zone1_mode"=dword:00000001
"zone1_weight"=dword:00000064
"zone1_quant"=dword:000007d0
"zone1_type"=dword:00000000
"zone1_greyscale"=dword:00000000
"zone1_chroma_opt"=dword:00000001
"zone1_bvop_threshold"=dword:00000000

is that what it's supposed to look like?

Second pass:

"Supported_4CC"=dword:00000007
"mode"=dword:00000002
"credits_end"=dword:00000000
"credits_start"=dword:00000000
"motion_search"=dword:00000006
"chromame"=dword:00000001
"bitrate"=dword:000002bc
"desired_size"=dword:000334e0
"use_2pass_bitrate"=dword:00000000
"desired_quant"=dword:00000190
"quant_type"=dword:00000000
"lum_masking"=dword:00000000
"interlacing"=dword:00000000
"qpel"=dword:00000001
"gmc"=dword:00000000
"reduced_resolution"=dword:00000000
"use_bvop"=dword:00000001
"max_bframes"=dword:00000002
"bquant_ratio"=dword:00000096
"bquant_offset"=dword:00000064
"packed"=dword:00000001
"closed_gov"=dword:00000001
"ar_mode"=dword:00000000
"aspect_ratio"=dword:00000000
"par_x"=dword:00000001
"par_y"=dword:00000001
"ar_x"=dword:00000004
"ar_y"=dword:00000003
"num_zones"=dword:00000002
"rc_reaction_delay_factor"=dword:00000010
"rc_averaging_period"=dword:00000064
"rc_buffer"=dword:00000064
"discard1pass"=dword:00000001
"full1pass"=dword:00000000
"keyframe_boost"=dword:0000000a
"kfreduction"=dword:00000014
"kfthreshold"=dword:00000001
"curve_compression_high"=dword:00000000
"curve_compression_low"=dword:00000000
"overflow_control_strength"=dword:00000005
"twopass_max_overflow_improvement"=dword:00000005
"twopass_max_overflow_degradation"=dword:00000005
"container_type"=dword:00000001
"target_size"=dword:000a2800
"subtitle_size"=dword:00000000
"hours"=dword:00000001
"minutes"=dword:0000001e
"seconds"=dword:00000000
"fps"=dword:00000002
"audio_mode"=dword:00000000
"audio_type"=dword:00000000
"audio_rate"=dword:00000080
"audio_size"=dword:00000000
"vhq_mode"=dword:00000001
"cartoon_mode"=dword:00000000
"turbo"=dword:00000001
"max_key_interval"=dword:0000012c
"frame_drop_ratio"=dword:00000000
"min_iquant"=dword:00000001
"max_iquant"=dword:0000001f
"min_pquant"=dword:00000001
"max_pquant"=dword:0000001f
"min_bquant"=dword:00000001
"max_bquant"=dword:0000001f
"trellis_quant"=dword:00000000
"fourcc_used"=dword:00000000
"debug"=dword:00000000
"vop_debug"=dword:00000000
"display_status"=dword:00000001
"Deblock_Y"=dword:00000000
"Deblock_UV"=dword:00000000
"Dering"=dword:00000000
"FilmEffect"=dword:00000000
"profile"="AS @ L5"
"stats"="G:\\DVD\\Rips\\TNGS7D1_VTS_02_PGC4\\st4.stats"
"qmatrix_intra"=hex:08,11,12,13,15,17,19,1b,11,12,13,15,17,19,1b,1c,14,15,16,\
17,18,1a,1c,1e,15,16,17,18,1a,1c,1e,20,16,17,18,1a,1c,1e,20,23,17,18,1a,1c,\
1e,20,23,26,19,1a,1c,1e,20,23,26,29,1b,1c,1e,20,23,26,29,2d
"qmatrix_inter"=hex:10,11,12,13,14,15,16,17,11,12,13,14,15,16,17,18,12,13,14,\
15,16,17,18,19,13,14,15,16,17,18,1a,1b,14,15,16,17,19,1a,1b,1c,15,16,17,18,\
1a,1b,1c,1e,16,17,18,1a,1b,1c,1e,1f,17,18,19,1b,1c,1e,1f,21
"zone0_frame"=dword:00000000
"zone0_mode"=dword:00000000
"zone0_weight"=dword:00000064
"zone0_quant"=dword:000001f4
"zone0_type"=dword:00000000
"zone0_greyscale"=dword:00000000
"zone0_chroma_opt"=dword:00000000
"zone0_bvop_threshold"=dword:00000000
"zone1_frame"=dword:0000f71f
"zone1_mode"=dword:00000001
"zone1_weight"=dword:00000064
"zone1_quant"=dword:000007d0
"zone1_type"=dword:00000000
"zone1_greyscale"=dword:00000000
"zone1_chroma_opt"=dword:00000001
"zone1_bvop_threshold"=dword:00000000

the .avs file:

# Created with Gordian Knot
#
# http://gknot.doom9.org

# PLUGINS
LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\decomb.dll")
#LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\KernelDeInt.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\UnDot.dll")
#LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\dgbob.dll")
#LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\Convolution3d.dll")
#LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\FluxSmooth.dll")
#LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\TomsMoComp.dll")
#LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\VSFilter.dll")
#LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\SimpleResize.dll")

# SOURCE
mpeg2source("G:\DVD\Rips\TNGS7D1_VTS_02_PGC4\st4.d2v")

# TRIM
#trim(startframe,endframe)

# IVTC
Telecide(order=1,guide=1).Decimate()
# or use
#IVTC(44,11,95)
#GreedyHMA(1,0,4,0,0,0,0,0)

# DEINTERLACING (1)
#FieldDeinterlace()
#FieldDeinterlace(blend=false)
#TomsMoComp(1,5,1)

# DEINTERLACING (2)
#KernelDeInt(order=1,sharp=true)
# or maybe
#DGBob(order=1,mode=0)

# DEINTERLACING (3) - special requests
#GreedyHMA(1,0,0,0,0,0,0,0)
#Telecide()
#SeparateFields()

# CROPPING
crop(4,0,706,476)

# SUBTITLES
#VobSub("FileName")

# RESIZING
LanczosResize(544,400)

# DENOISING: choose one combination (or none)
Undot()

# 1) little noise
#Temporalsoften(2,3,3,mode=2,scenechange=6)
#mergechroma(blur(1.3))
#FluxSmoothST(5,7)

# 2) medium noise
#Temporalsoften(3,5,5,mode=2,scenechange=10)
#Convolution3d("moviehq")
#FluxSmoothST(7,7)

# 3) heavy noise
#Temporalsoften(4,8,8,mode=2,scenechange=10)
#Convolution3d("movielq")
#FluxSmoothST(10,15)

# BORDERS
#AddBorders(left,top,right,bottom)

# COMPRESSIBILITY CHECK
# !!!!Snip Size now has to be 14 for use in GKnot!
#SelectRangeEvery(280,14)

# FOOL CCEnc
#empty = BlankClip()
#AudioDub(last,empty)

[sector9]

Oh, and one other thing.

I tried it again with the same settings last night--same problem. But if it matters at all, after the first pass ended just fine, the second pass started just fine. With about 40% done it said estimated time was about 75 min--just about right. It was at that point I went to bed. When I got up in the morning, it was at ~8 hours out of 15 estimated, at which point I killed it. Is this a wrong setting or typical of oversizing? What should I do?

[Koepi]

Does this "always" happen when you go to sleep?

Maybe the sleep-mode kicks in (settings for energy saving at some level instead of "desktop" or switched off) and thus throttles your machine.

Cheers
Koepi

[sector9]

It's happened during the day, as well (I believe.) More importantly, I've encoded dozens of episodes at night just fine.

[sector9]

ok, so I tried one of the damn oversizing episodes again, with the oversizing fix on (min Q2.) It worked--made a ~250 mb file. However, it still took ~4, 5 hours.
I've made a few more rips--those have turned out OK sizewize, but are also taking way longer than I'd like. The first pass is generally going at an OK speed--A little slower than I've gotten in the past (~ 1 hour 15 or the like instead of 50 minutes), but reasonable. The second passes are taking forever--2 hours, 3, once or twice 4. Why is it suddenly doing this? Do you guys know why my second pass encodings might suddenly slow down like hell? Anything I can try? (For example, can someone link me to a guide for doing this (Xvid encoding, using an mp3 audio file, minor cropping/resizing, IVTC, a credits section, and not much else in terms of goodies) using just, say, DVIndex and VirtualDubMod--no GKnot--so I can test that it's not GKnot/the method it uses?)

One thing I'll bet someone will suggest--I tried closing down all my programs during one run. No go. VirtualDubMod was pulling 90% of the CPU time during that second pass, whenever I came down to check, and yet still took several hours.

Any help? Do I need to post my current settings in a new way? Anything else?

[squid_80]

This is a bit of a long shot which will probably amount to nothing: Are you using xvid 1.0.3 (which I think is the one in the gordian knot codec pack) or a 1.1 build? Because those xvid settings you posted are missing a few things for 1.1.

[sector9]

1.0.3. Should I upgrade?

[sector9]

By the by, I went to www.xvid.org to get the latest version--but only source was available, and I don't have the needed tools for compilation. Is there a place where Win32 binaries are available?

[sector9]

OK, I found a link in my Xvid directory, so I got the 1.10 binaries. By the way, though, I noticed something that might be a problem. I went to Gknot->Xvid first pass settings->advanced options->quantizations. all three types are set to 1-31. Should they be stuck at 2?

[niamh]

1-31 has been made default for xvid, so it's normal (too many people complained of undersizing, which is a bit like complaining one has too much money to spend IMO)
As you saw for yourself, sometimes it leads to oversizing indeed. quant 1 is pointless and won't be used 90% of the time, so if you don't mind the occasional undersize(due to saturating the codec in the first place), then you may change it to min = 2
btw what it tells you, is that the target size of 350 mb is way too high and a waste of space (or if you really want a big file, adapt your resolution to something less microscopic than 544 x 400 ;)

as for 2nd pass speed, are you using new filtering? Telecide is none too fast, if you weren't using it before, there might be your difference. If those are really well telecined, (which I doubt, those 2 series I would bet are hybrids) you may get away with force film in DGindex, making it much faster. (run a preview on this before encoding to check motion fluidity / stray combs, because forcefilm is dumb)

PS: 2,3,4 hours is NOT forever, wait till you ever try a really slow script/codec :D

[sector9]

Hmm. I had thought that the entire point of the first pass was that it encoded at a constant Q2 so the second pass had an idea of how to change that. If not, someone want to explain to me why we have a first pass?

I upped the resolution on my encodes (to about 240000 pixels) and tried a few episodes. They worked fine--350 Mb encodes, took about 2 hours and 30 minutes, which I can live with.

The next one stalled. I canceled it after the first pass had been going for 8 hours and was projected to go 8 more.

Why might Xvid just suddenly start to chug in the middle of a encode? Every time, the encodes start out with a time estimate of, oh , 50 minutes. That estimate starts to grow, and generally when they work now (like the first two I did last night) take, oh, 70 minutes, 80 minutes. But sometimes it just keeps spiraling up--to ridiculous amounts in this case. Any ideas why that might happen?

[Koepi]

Blaming XviD all the time isn't the proper solution, I fear it's something else - your setup maybe, some other software can be guilty as well. It's _not_ necessarily XviD's fault.

Maybe you can check if something else is happening - little free space on the target disc maybe, or a virus scanner getting active, or something completely different. Can you do that, can you search your system for anything unusual in that direction?

(On the other hand: try the same with another codec like divx. If it stalls too sometimes you can be sure it isn't a codec which is guilty of the "long time to encode").

Another idea: do you have a P4? Maybe your system gets too hot and CPU throttling gets active.

Cheers
Koepi

[sysKin]

Any chance you ran out of diskspace?

[sector9]

Good point--it might not be Xvid. Out of the other suggestions:
My virus checker is disabled temporarily. That's not the problem. I checked--no tasks are scheduled.

Diskspace could be an issue. The disk--well, that's the thing. All but about 20 gigabytes are full, but the rest kind of fluctuates--depends on how many of the dvd's I'm ripping are currently ripped to hard drive, etc, etc. The lowest I've ever seen it go is five gigabytes free; right now, with two tv episodes ripped, one encoding as we speak, 16 gigabytes free.
But in any case, something suggests it's not diskspace--all the rips are located on my secondary media drive--but my primary hard drive, where, if I remember correctly, my swap file should be located, has like 90 gigabytes free.

This is just weird.

After the 8 hour first pass I mentioned in my last post, I did 3 more episodes--2 of them finished in under 2 hours each, the third finished--but took 4 and 1/2. Part of the annoyance is that I can't reliably tell how, why, or even when a particular episode will or will not stall.

One thing I noticed--as I said, when a pass stalls, it generally starts out and goes a while with a perfectly reasonable time estimate, but then I come back 30 minutes later, and instead of being done, it's telling me 5 hours more are needed. Thing is, whenever it does say that, it gives its current encoding speed as something perfectly reasonable--16-22 frames per second, usually. If that's so, it seems almost like some very short portion or the like just stalls the pass temporarily, then eventually when I get back to it, it's gotten past (maybe) after however many hours and is encoding at a OK speed. If that's so, why might some small part stall it so?

Another possibility with that--something slows it down (perhaps something external) but the mere act of me coming back brings it back to speed--sounds almost like the computer starting up something else, or sleeping--but I highly doubt that, or at least don't see why it would--it's not configured to sleep or powerdown, and I've watched it stall with everything else on the computer besides Gknot and Vdubmod closed out.

[Koepi]

If the framerate is still shown as being usual even the timer on your computer is going nuts. This would happen in either case:
- some technique like speedstep is kicking ing
- CPU throttling is active due to overheated CPU (XviD stresses the CPU more than anything else - other tests may pass without problems, XviD still shows there is a weakness in the system).

Install something like motherboard monitor and watch the temperatures of your CPU/chipset. Even if your system gets slow it should show the same temperature if it's the CPU throttling due to termal reasons.

I also noticed on some systems that this can happen if the HD is close to die. It delivers the data very slowly then (and usually makes weird noises). But then the shown framerate should go down, too. (This can affect only certain regions of the discs. I had a HDD which behaved this way on the last 1GB free - if it passed that mark it slowed down to a crawl and soon later completely died.)

Koepi

[Didée]

This sounds really strange. But, two more points that should be looked into:

- when starting an encoding, also start Window's Task Manager. Look at the memory utilization when a job has slowed down - perhaps there's a memory leak somewhere in the decoding/filtering/encoding chain. This should become visible then, in that any process has an untypical high amount of memory load.

- Generally about the swap file: I found it always beneficial to use a fixed size for the swapfile, i.e. min.size== max.size. By default, the swap file is dynamic, meaning Windows will adjust the size of the swap file acc. to current memory needs. Sometimes this may lead to very strange "effects".
Moreover, if you've worked for a longer time with such a "dynamic" swap file, then it has become heavily f.r.a.g.m.e.n.t.e.d all over the drive ... and unless using a defragmenter much more clever than the built-in one, the only way to de-fragment the swap file is: to completely disable it, make a thorough defragmentation of the drive, then create a new swap file.

[sector9]

I recently had one of the usual slowdown problems--just as usual, going back and using the computer for a minute fixed it. However, I thought to open the virtualdub log--it had in it 20-30 instances of something about equal to:
Dub:I/O thread has not cycled for ten seconds -- possilbe livelock
either that or "processing thread", and with different hex addresses.

This mean anything? (This was during 2nd pass of the encoding.)

[niamh]

The livelock issue hasn't been fixed I think, one solution is to use vdm 1.5.4.1 instead of 1.5.10.1, it helped me when I was getting this error; i.e. I didn't get any livelock with the older version.(haven't got it since reformat actually, so I use the new one again).

[squid_80]

Possible livelock isn't an error. It just means the encode is going rather slow.

[niamh]

I call zero fps a serious flaw at the very least. What you're doing is nitpicking, I'm sorry to say

[squid_80]

Originally posted by niamh
I call zero fps a serious flaw at the very least. What you're doing is nitpicking, I'm sorry to say

You can call it nitpicking if you want, but there's a difference between "actual" livelocks and warnings. A real livelock is when fps drops to 0 and the encoding process halts indefinitely for some reason. From sector9's description the process is not halting completely, just proceeding very slowly. The livelock warnings can be ignored since there is no lock.
There's a big thread about the livelock message here. (http://forum.doom9.org/showthread.php?s=&threadid=62774)

[niamh]

Well, I gave a fix for this, whatever you care to call it.This argument has no place here in any case.It might or it might not work, which is the important point :)