Rips are oversized without any apparent cause

[Wildfire]

Situation: GordianKnot 0.32.0 beta, VirtualdubMod 1.5.10.1 build 2439, XVid 1.01 (Koepi's build).

XVid first pass settings: quantization H.263; B-Vops: max. consecutive 2, quantizer ratio 1.50, quantizer offset 1.0, closed GOV; Zone options: weight 1.00, chroma optimizer enabled, BVOP sensitivity 0; Advanced options: motion search precision 6, VHQ mode 1, Use chroma motion, frame drop ratio 0, maximum I-frame interval 300, min. I-frame quantizer 1, max. I-frame quantizer 31, min. P-frame quantizer 1, max. P-frame quantizer 31, min. B-frame quantizer 1, max. B-frame quantizer 31

XVid second pass settings: most are the same, with the exception of - 2nd pass settings: I-frame boost 10%, I-frames closer than... (frames) 1, are reduced by... 20%, overflow control strength 5%, max. overflow improvement 5%, max. overflow degradation 5%, high bitrates scene degration 0%, low bitrates scenes improvement 0%.


To be clear: I uninstalled GordianKnot and XVid after resetting everything to default. Then I installed all the latest versions, loaded the defaults again (just to be sure).

What's happening is that my 1CD Star Trek: The Next Generation DVD-rips are all but some coming out way oversized. I've seen 783mb, 930mb, 954mb, etc -- and that's even without the 120mb AC3 audiotrack. The strange thing is, my Star Trek: Voyager rips are all encoded perfectly at 575mb (video only) - just the way I want them. But the TNG-rips are almost always way oversized.

I crop the video to 672x512 (pixel) in GordianKnot in 4:3 ratio settings, applying a kernel deinterlace and the Lanczos resize filter.

I have tried settings to rip for a total filesize of 350mb -- they all come out much smaller than... I've seen a 954mb rip being reduced to some 400mb without any real visible loss in visual quality.


Does anyone have *any* idea at all what's going on here???

By the way, please no remarks like "1CD is nonsense for 45 minutes", it's my choice to go for absolute picture quality. At half a CD, with AC3 audio track, the video bitrate is just too low for my liking. So please, no complaining about whether or not you think a 1CD rip is nonsense for 45 minutes of material.

[Didée]

Originally posted by Wildfire
By the way, please no remarks like "1CD is nonsense for 45 minutes", it's my choice to go for absolute picture quality. At half a CD, with AC3 audio track, the video bitrate is just too low for my liking. So please, no complaining about whether or not you think a 1CD rip is nonsense for 45 minutes of material.
Okay, no remark about that.

I'll also post no remark about the fact that, for the first few seasons of TNG, the foreign localisations actually come shipping with MONO audio, and will not remark that the 120 MB for keeping the AC3 are a bad joke, when one could keep 99.9% of the quality with e.g. a 30 MB OGG audio file.

Furthermore, I'll not put any remark about the fact that "absolute picture quality" was the comment that made my day today, for the humoristic part. If I *would* comment on it, I'd tell you that you're dealing with a crappy FieldBlended FILM->TC->NTSC->FieldBlending->PAL conversion, and that you'll get only blended crap out of Kerneldeint if you feed it with such stuff ... crap in, crap out. I could even go as far as to tell you that there is a script named "Restore24" that offers a fair chance to restore a reasonable clean and fluid 24 fps FILM output from that mistreated source (exept perhaps for the hybrid parts, but that's detail) ...

... IF I would comment on that. But since you asked to not comment, I won't do it for sure :D


Well, for your oversizing problem:

Your settings seem quite reasonable, basically. Exept for one thing: Please set the quantizer ranges to "2-31" instead of "1-31".
This is most probably the cause of your problem. 1st-pass is "too small" (no wonder with TNG - a soft, low-detail source, at least the first seasons on European releases. They compress like black holes), 2nd pass tries to compensate with quant-1's, thereby produces overshoot that's not strongly enough counter-compensated.

Your options:

- Leave "1-31", and set those "overflow improvement / degration" values to "20-20-20"

- set quantizer cappings to "2-31" and use bigger resolution, or a high-bitrate matrix.


- Didée

[Koepi]

Use the search next time. This issue pops up every week and gets answered each week. Kinda annoying.

In your case you saturated the codec, but only by a small amount. That's why ratecontrol goes nuts and produces these uncontrolled huge files (remember, this has been written down here very often. You should use the search.)

The only reasonable option for you is (as you want the huge filesize) to set the overflow values from 5/5/5 % to 20/20/20 %, as Didee already mentioned.

Koepi

[Wildfire]

I have used the search but didn't find anything which looked useful. Perhaps a case of wrongly chosen search terms, I don't know.

Anyway, back on-topic -- the strange thing is my rips used to turn out at approximately the size I wanted them. However, only recently this problem popped up. About at the time I switched from GordianKnot 0.28.x to 0.32.0 beta and from Koepi's 24062003 XVid build to Koepi's XVid 1.01 build...

That's why I thought the problem was caused by the newer versions.

[Koepi]

It is caused by the new xvid version, ratecontrol got rewritten. We tried to choose decent, smooth values for that, but they aren't enough if you want your second pass to be bigger than the first pass.

I suggested something to syskin today, maybe we'll use more "aggressive" settings in those cases by default.

Regards
Koepi

[Wildfire]

Originally posted by Koepi
It is caused by the new xvid version, ratecontrol got rewritten. We tried to choose decent, smooth values for that, but they aren't enough if you want your second pass to be bigger than the first pass.

I suggested something to syskin today, maybe we'll use more "aggressive" settings in those cases by default.

Regards
Koepi

Ah - that clarifies things. No wonder things began to go wrong just when I had installed the new XVid. In effect, I should revert to the default of the old XVid instead of the defaults of the new XVid.

I'll try the settings suggested earlier when I have some time next week.

If I don't forget, I'll make sure to post the results here so you know whether it worked or not.

For now, thanks for the help and explaination.

[nanga parbat]

eh, i wonder how many people ever have a look on their first pass stats...

make it like besweet, pop up some log info or so.

'first pass came out at xxx megabytes'

that'd make much things clear i guess.

[piscator]

Originally posted by Didée
I'll also post no remark about the fact that, for the first few seasons of TNG, the foreign localisations actually come shipping with MONO audio, and will not remark that the 120 MB for keeping the AC3 are a bad joke, when one could keep 99.9% of the quality with e.g. a 30 MB OGG audio file.


Off topic to the original question, but just curious about this remark for the general case.

I suppose it still is impossible to playback multi-channel OGGs over a digital output? You would need a real-time ogg-ac3 transcoder for that... One of my questions out of the old box (http://forum.doom9.org/showthread.php?s=&threadid=52053&highlight=ogg+digital+output) :D So for multi-channel playback, ac3 is still the only option.

greetz,
Piscator

[[Admiral]]

well i was having the same problem (sorta) as the guy that starting this thread.
i was trying to encode a ~800meg svcd mpeg-2 file to a 350meg xvid/mp3 file. at first it kept comming out 450megs, did twice to make sure it wasnt some computer instability error or whatever. so then i did what i used to do back when this would happen when using SBC (where i learned most of my encoding). set the destination size @ 250megs. so then the file came out @ 250 megs. was annoyed, so i came looking for help here. saw this thread, and saw the '20/20/20' sollution. so i tried it. yay, now the file comes out about right (off by about 2 megs but whatever). only problem is now im getting this weird 'blocks' around peoples faces. its a really odd effect, ive never seen it before. its like when the people are talking their face turns into all these seprate boxes that dont move together. it doesnt always happen, but it does happen often.

I was using the Field Deinterlace (-noblend enabled) so i tried using Kernal Deinterlace. seems to actually be worse. I tried using neutral bicubic instead of simple resize, again not much difference. i tried turning off quarter pixel and setting the max consecutive b-frames to 2 instead of 3. still no major difference.
so now i seem to be stuck. ive tried a few different variations that i havnt listed here but its really bothering me. ive been encoding for years and years and ive never come across this problem.

any ideas?

[jon.schaffer]

Hi,

I'd like to add something to this topic.
I already posted about that, you could read that:
http://forum.doom9.org/showthread.php?threadid=75527

the answers are interesting and instructive.

[Admiral]:
could you try the overflow settings I recommended? (0,4,9) and let me know if you get rid of these blocks?



In fact, I do not use these settings anymore, I prefer capping quantizer to 2-31, and, in the case of an undersizing, I then improve the image quality (resolution, sharpness...).

Why, IMHO, use quant 2-31 instead of 1-31?
Because the utilization of quant1 frames allows indeed to correct undersizing (if the bitrate control doesn't get crazy when Q1 frames are too numerous...), but in term of quality, it is not optimal. A quant1 frame take too much bits and lead to more higher quant frames to balance...

It seems to me that an undersized entire movie at Q2 is in general better than a correct-sized movie with quant1 frames... (not in every scenes though, mainly in high-motion scenes... but, it would require more testing)

And so, I think you should try this quantizers capping. You'll probably get an undersized movie in your 1st pass (options: keep the 1st pass, and enable full quality-1st pass - you won't need a second pass), then, if you want to make it bigger, try to improve its quality (you could improve the soundtrack too...)

regards,
Jon

[Koepi]

jon:

your solution makes the user need to be more knowledgeable - we added that because the undersize-case came up too often for "newbies".

So simple solution is like suggested using 20% in all overflow settings, if you know what you're doing better shoot out for better resizing filter like lanczos, use less denoising,...

Regards
Koepi

[jon.schaffer]

Originally posted by Koepi
your solution makes the user need to be more knowledgeable...

...like suggested using 20% in all overflow settings...

Yes.. OK...

in fact, my first aim in this answer was to advise [Admiral] to use the settings 0,4,9 instead of 20,20,20. I tested these and reported it, saying that 20,20,20 led to artifacts (too quick rise and fall of the bitrate from a frame to another? too aggressive values?)... nobody has paied attention to this, so I thought my test was useless or not interesting - nevermind ;).
Here, [Admiral] seems to have noticed the same blocking artifacts that I got: I think it would be interesting to see if 0,4,9 lead, in his case, to the correct filesize AND to a better quality than 20,20,20.

Yours,
Jon

PS - a question to Koepi:
let's take an "undersized" 1st pass. The default settings (Q1->Q31) allow the presence of quant1-frames which "fatten" the movie - OK.
But I noticed the presence of quant1-frames even in movies whose 1st pass was bigger than the targeted size, and thus which needed to "lose"... the problem being that a quant1-frame is very big and this involves some poor quality frames to appear...
=> would it be possible that the codec - when starting the 2nd pass encoding - "read" the targeted size and compare it to the 1st pass size?: if (target) < (1st pass) then no quant1-frame allowed ("auto-capping")????

[[Admiral]]

sorry for not replying jon :)
ill try both of your solutions tonight and see what happens :)
im not really all that noob and i actually do understand what you boyz are talking about (mostly) heh :)
but i was originally getting an OVERSIZED file not a undersized file. so yea. but whatever. ill let you know how it turns out :)

[Koepi]

jon:

in fact, it's like we're doing that automagically already.

The problem with the plain assumption of "first pass bigger than desired size" is misleading - fast first pass creates a file which is too big compared to a full quality first pass, which would turn out undersized(!).
So we're allowing q1 frames at about 90% of first pass size as a rough estimation.

Btw., your values 0/4/9 % don't make any major difference - just take i.e. the default 5/5/5 and set it to 5/5/10 - it would turn out nearly the same as your suggested values. That's why you didn't get responses to that. a single % point "tweaking" there could make sense on a single source (like encoding it 30 times and compare the results and take the best), for another sample these values can be absolutely rotten.


I hope this helps.

Regards
Koepi

[jon.schaffer]

Originally posted by Koepi
..."first pass bigger than desired size" is misleading

Yes, I forgot that... since I always use Full Quality 1st Pass...
BTW, I have planned to make (if I have time) a test to see the impact of fast 1st pass on the result. I don't know if it has been tested already. Would it be interesting? I'll make it ASAP...


So we're allowing q1 frames at about 90% of first pass size as a rough estimation.

So... it's like you have no real other choice...?


Btw., your values 0/4/9 % don't make any major difference...
...a single % point "tweaking" there could make sense on a single source

Yes, maybe indeed. I'll add something:
First, I must precise that I made several tests on different clips (slow motion to very high motion, different sources from noisy to soft...) and chose values which gave always good result. I indeed found numerous values combinations leading to near results in terms of good size prediction, and fewer in terms of prediction + quality. (sorry, I feel obliged to justify me! :D - you must be tired of guys like me!)
That saying, I would rather make these comparisons:
- 5,5,5 (default) and 0,4,9 -> small modification, yes, but sufficient to predict the final size.
- 20,20,20 and 0,4,9 -> bigger difference, allowing in my tests to get rid of these ugly blocks.


...for another sample these values can be absolutely rotten.

Obviously, you're right anyway! ;)
We agree on this: my tests, even if varied, are not at all absolutely representative... It's exactly why I proposed it to [Admiral]: to see if it can be applied in other cases than mine. Do not mistake: I don't want to say "hey guys, the values Koepi advises are shitty!"... I have too much respect for you! (and furtermore, I don't think that at all!)


I hope this helps.

Oh, yes. And so, what a best moment to thank you all "MPEG4 guys" for your great work... THANKS!


Truly yours,
Jon


PS:
[Admiral]: I wait for your results ;)
nota: the fact you had an oversized file (not undersized) is indeed the point. It's the case because your (probably virtually) full quality 1st pass is undersized (too compressible). During the second pass the codec tries its best to "fatten" it, and did it too strongly... thus the oversizing...
You want to know more about it?: use the StatsReader and open your video.pass, you'll get the size ("Size (MB)") of your first pass or do not discard the first pass file (the first pass is made at a constant quantizer of 2.). It should be smaller than your target size. If not, we can presume that you did not use the Full Quality 1st Pass option: some encoding options are then disabled during 1st pass. These options would have created an undersized first pass file...
Hope I'm clear...

Oops, what a PS!

[[Admiral]]

dear jon:
no, i do not use the 'full quality first pass' option. mostly because i was unaware of any benefit, and i WAS aware of the annoying bigass file that i didnt want lol. im used to the old sbc way of only having the output file you want to keep, and having a tiny first pass stats file :/

should i use full quality first pass? and stop discarding the first pass? hmm.

im encoding as i type. ill give you the results (and perhaps a screen shot if it turns out badly again).

[[Admiral]]

success! :D
yay 0,4,9 work great!
now my filesize is only off by 2 megs undersize (woopdie freaking doo)
and i dont have any of those crazy tearing boxes around peoples faces when they talk :)
thanks a lot jon!

[jon.schaffer]

Happy to help! :D

Concerning the Full quality 1st pass, I would say that, before further testings, it depends on the way you encode:
- if you prefer using default settings and only tweak the overflow treatment values, you could go on not using this option and keep the 1st pass discarded.
- if you plan to learn more about quantization and want to try to get even better results, you certainly should try to use this option in addition to a quantizers capping: Q2->Q31 (instead of Q1->Q31). Using this, when you'll get an undersized firt pass, you could keep it without any problem: its quality will be already very good. Then, you could use the saved bits to fatten your clip by improving its quality (higher resolution, sharpness, alternative quantization matrices, etc.). Search the forum: there are plenty of interesting topics about all that...

Jon