Average Quantizer Encoding: How to get constant quality, but with quant distribution

[Harley Quin]

How do I get an average quantizer of 4 at my encoding? Average, not constant, because I want to make use of quantizer distribution, say between 3 and 6, to spend bits where they are needed and remove them where they're not.

Constant bit per pixel, say 0,180, or constant bitrate, for example 900kbit, do not take into consideration the different compressibility of movies and different frame sizes. It can produce unpredictable quantizer and quality.
When using Zones I can set a quantizer for the Zone, but it produces a constant Quantizer even at 2-pass-encoding.

My workaround so far:
I make a single pass with a constant quantizer of 4, take the resulting bitrate and make a 2 pass encoding with this bitrate in the second pass.
Makes a 3-pass-encoding. Hmmm...
Ok, I could make a compressibility check instead of the pre-single-pass but it takes even more work (and thinking :)) and the testing result is not always close to real encoding.

Any ideas?
Thanks

Harley

[Teegedeck]

Perform the first pass at a constant quantizer (through a zone) that you expect being the average for your second pass.

To get an accurate but still fast first pass, switch off VHQ, lower ME precision to 5, skip GMC and QPel but keep Trellis and AQ. (Don't forget switching them back on for second pass.)

And yep, for good guesses as to that you need Enc for a size-prediction.

[Didée]

You could do like this: Make a 1st-pass with *moderately* faster settings:

1st-pass:
- Define one zone only, @ q4 (plus credits zone, if you want)
- +qpel, +VHQ1, [+AQ], -BVHQ, -GMC, -chromaME

2nd-pass:
- Re-define the zone to weight=1.0 (no change to credits zone if present)
- restrict all quants to min=3, max=6
- eventually: set CurveCompression to hi=25~35, lo=12~18
- destination size == 1st-pass size
- +qpel, +VHQ2~4, [+AQ], [+GMC], +BVHQ, +chromaME


That's it, shortly, if you specifically want a 2-pass encoding with an average quantizer of 4. For 1st-pass only, "Turbo" is allowed to be switched on. (Tho' I don't like it, and for sure not for 2nd pass.)

For a similar procedure with "unknown" average quantizer (i.e. fixed dest.size like 700MB), follow Teegedeck's advice and use Jonny's "enc" application. It allows, similar to a comp.check, to estimate the final encoding size for a fixed quantizer with only doing a short snip through the movie.
Makes one, two or three short shots on the source, each a couple of minutes only, to get a pretty good guess for the average quant for the current case.


@ Teegedeck

I don't think cutting down the used encoding Tools by *so* much is a good thing to do for doing a first pass. It's OK for size prediction through enc, but not for a first-pass.

Mind you, (to my knowledge) there is no more I/P/B-decision during the 2nd-pass. 2nd-pass uses (from the stats file) the exactly same frame types that 1st-pass had worked out. Therefore, if the 1st-pass went with too lame settings, it might have produced suboptimal frametype decisions because of that ... and you'll get the same suboptimal frametype usage in the 2nd pass, no matter if you enable all bells'n'whistles of XviD.

[Teegedeck]

Yes; you're right about VHQ.

And I'm certainly gonna test whether your settings for 1st pass give me an even steadier distribution in 2nd pass. (With the above setup I get a quantizer-distribution like min=3, avg=~3.6, max=5, but still too much fluctuation between 3 and 4. I hitherto put it down to aiming for filesizes that lie exactly between the results that constant quant=3 and constant quant=4 would give me, but maybe... hmm! :)

But GMC for first pass? Come on, it seems to have marginal influence on framesize and (unlike AQ) comes at a big speed-penalty! (And I suppose you mean B-VHQ, not B-Qpel?)

Edit: Hm, did I use to set VHQ=1 for 1st pass in my encodings? I just realized how damn long it has been since my last encode... Anyway, I also realized that XviD's standard settings for 1st pass are really quite rigid:

quoting sysKin (http://forum.doom9.org/showthread.php?s=&postid=564594&#post564594)
Well yes: look here and look at the flags that are disabled in rc_2pass1_before() function.
I think I can summzrize it as: motion precision 1/2/3 (they're the same heh), no chroma motion, no vhq, no brute-force intra prediciton decision (can't disable it with GUI), no trellis. Also no qpel or gmc. "Turbo" becomes active.

[Harley Quin]

Originally posted by Didée

2nd-pass:
- destination size == 1st-pass size


Hmmm...
How do I tell 2nd pass to use 1st pass size (or bitrate) as destination if first pass hasn't been done yet?

Ok, just now I made a q4 first pass with zones and kept it (unchecked "discard"). That way I don't need the single pass q4 which is an improvement, thanks.
But...
I still have to interfere between passes which I'd like to avoid.
I don't mind first pass taking a long time, it may even be full quality. But 52 episodes are here, waiting to be encoded and I'd like to program 104 Jobs and simply wait till they're finished.
I customize so much, I'd like to automate things once in a while... ;)

Greetings

Harley

[Didée]

Originally posted by Teegedeck
But GMC for first pass? Come on, it seems to have marginal influence on framesize and (unlike AQ) comes at a big speed-penalty! (And I suppose you mean B-VHQ, not B-Qpel?)
Na-nana, misunderstanding. There are "+" and "-" signs in the settings I posted:
"+" = Activated, "-" = disabled, "[]" = at one's will, see?

And indeed, I really meant to write B-VHQ, not Bqpel, yes... How do you translate "Freudscher Fehler" to English? :)


Harley Quin -

if you're going to do all 52 episodes at once anyways, then you could as well combine all parts to one big file, make zones starting with an I-frame at the places where each parts starts, make a nice 2-pass encode of it, and split the resulting big file in parts again afterwards. Gives perfect overall bitrate distribution.

Oh, and make a prayer there's no power blackout during the week or two that encoding job will take.

:D

[Harley Quin]

@ Didée

It's "Freudian Slip".
By the way, you're a night noodle, hmmm? But thanks for the replies.

Well, in most cases your workarounds are worth a try, but with my prefered OS that limits filesize to 4 Gig and my prefered container format that limits filesize to 2 Gig, using an ultralarge file is, well, suboptimal :)

My prefered solution for now is to copy each episode into a subdir and make all first passes in a row, each with its own stats file ("./video.pass"), so stats files won't be rewritten with the following first pass. Then I program the second passes - for the second week.

Greetings

Harley

[Beave]

I used to encode lots of Episodes. My prefered method was to make a large file and put 50 white frames in between each Episode using avisynth. This way I did not have to bother setting the exact I-frame position. Making the avs was simple. The codec always sets an I-frame at the beginning an end of that white space. Make the first large file an mkv and later when you cut them you can always use avi again.
I experienced that episodes vary a lot. in extrem cases some were half as big as the others. So it's worth it!
You also don't need to put all 52 Episodes in one file. I usually did half or full seasons in one file.
I even wrote a script-program that did the audio-encoding, cutting, muxing and renaming automaticly for me. Back then the biggest problem was to determine the framenumber of the d2v files. Now with the newer avisynth version and dgindex this shouldn't be a problem anymore.

[Harley Quin]

Convincing. Could you post a script? Hey, this is a nice one: :logfile:
Problem #1: How to merge .d2v-clips? SegmentedAviSource won't work, Segmentedmpeg2source does not exist.
Problem #2: How to create white frames? I found BlankClip but it leaves me with the question how to get it between the episodes as well.
My Avisynth 2.5.5 script file for a single episode consists of:


LoadPlugin("C:\Video\AviSynthPlugins\dgdecode.dll")
LoadPlugin("C:\Video\AviSynthPlugins\decomb.dll")
LoadPlugin("C:\Video\AviSynthPlugins\UnDot.dll")
mpeg2source("D:\Film1\Episode1.d2v")
crop(14,4,692,568)
FieldDeinterlace()
Undot()
mergechroma(blur(1.3))
BicubicResize(512,384,0,0.5)

Thanks a lot

[Didée]

ep1 = Mpeg2Source("path/to/ep1.d2v")
ep2 = Mpeg2Source("path/to/ep2.d2v")
[...]
epX = Mpeg2Source("path/to/epX.d2v")

snip = BlankClip(ep1, color=$FFFFFF) .trim(0,49)

ep1 + snip + ep2 + snip + [...] + epX

[Beave]

Yes, Didée got exactly how I did it. That's the script.

[Harley Quin]

That's a nice trick. Works at first try. ;)
Ahhh, AviSynth. Such a powerful tool, such a non-understandable documentation. Even now, searching the manual again, I could not put my finger on the place where it understandably says:
"You can load multiple clips, define each by its own variable and you can queue them with a '+'."
So again: Thank you!

[sysKin]

Hmmm actually, it's probably worth adding that for xvid, optimal quant distribution IS constant quantizer. By trying what you try you just add noise to this scenario and nothing else.
XviD has no smart bitrate distribution and never had.

[manono]

I'm glad you said that, sysKin. I almost said it myself a few days ago, but decided that, since it wasn't what Harley Quin wanted to hear, to just forget about it.

If you want an average quant 4, and aren't all that particular about the final file sizes, just encode to a constant quant 4. It also has the added benefit of saving you a whole lot of encoding time. That has to be a consideration when you're starting on a 52 episode project.

spend bits where they are needed and remove them where they're not.

That's what constant quant encoding will do for you, and it will do it better than your plan of restricting them between quants 3 and 6.

[Harley Quin]

Yes, you're right, quantizer distribution is minimal:
http://forum.doom9.org/attachment.php?attachmentid=4016&stc=1
It's about 95% the same as in the constant quantizer encoding, which was q3 in this case.
I should have checked that in the first place, but I think I simply expected some sort of second-pass-miracle, like it would take the little amount of precious bits I'm only willing to give and spend them where my eyes want them to be. I will use a single pass fixed quantizer here as well, as it is already my prefered solution at higher bitrate encodings.
Greetings