Can extreme undersize in first pass lead to extreme overflow in second?

[greycortex]

I have been trying to backup a film called Tomie:Rebirth for the last couple of weeks, and have found that with the last release of XVid 1.0 (not the bugfix release) the file was extremely oversized at 900 megs whereas 625 was requested.
After checking out the log-screens, it appeared that the first pass had an average quant of 2.75, as to be expected, but the second was approaching 1.8. I tried to limit the minimum quants to 2, and I got a movie that didn't look so hot (? beats me why, quant 2 used to look fine to me) but was around 225 megs for a 105 minute film. What I'm wondering is: is there a scaling issue with rate compression if the curve is pushed to far back towards quant 1? I'll have more data tonight, as soon as my next test with the bugfixed 1.0.1 has been run.
I've never seen this type of issue before, so I was curious as to whether the bitrate scaling from quant 2 to quant 1 is much more extreme than any other quant shift.

[*.mp4 guy]

i beleive you must be used to extremely compressiblr movies. most movies at quant 2 will be atleast 1500mb.

[gldblade]

@ *.mp4 guy
No, that's most definitely not why XviD is oversizing. greycortex didn't limit quantizers to 2, only the minimum quantizers.

@ greycortex
What were your settings? Are you sure they were the same between passes? You're probably gonna answer yes, just gotta make sure.

[greycortex]

Same settings on both passes

Qtype: H.263
Adaptive Quant: OFF
Qpel: OFF
GMC: OFF
Max Consec. BVOPS: 2
Quant ratio: 1.5
Quant offset: 1.00
Closed GOV selected
MSP: 6
VHQ: 4
Chroma Motion selected
Max I-Frame Interval: 240
Quants:
I: 1-31
P: 1-31
B: 1-31
No Trellis

2nd Pass: 853 kbps
I've never ever had a movie encode do this before...
:confused:

[Pen-Pen]

now THAT'S weird O_o

try settings quants to 2-31, maybe your movie is making the codec's quantization process crazy (but I highly doubt about that)

EDIT :

sorry, I didn't read your post carefully enough...

there must be some mixup in your codec installation, there is NO WAY that your 2nd pass is larger than the first one, if you cap the quants to 2-31

uninstall & reinstall the codec, hit "load defaults" (who knows ? ^^) and try again

[Manao]

greycortex : first, the oversizing you're experiencing is due to the massive use of quant 1 with default settings for the overflow treatment.

XviD's default settings for overflow are perfect when you're not in a situation of undersizing ( and it is the case here, since you're wanting a 625 MB file whereas the first pass came out at 225 MB ).

If you're in a situation of undersizing and if quant 1 get used, the rate control with default settings tends to have issue as soon as it begins to insert too much frames at Quant 1.

At that point, there are several solutions possible :
* change all the default settings for overflow treatment to 20 % ( as advised here (http://forum.doom9.org/showthread.php?s=&threadid=75527&highlight=oversizing+overflow) by Koepi ). You won't have oversizing anymore.
* Try to increase the resolution, reduce the filtering ( if any ), in order to reduce the compressibility of the movie. In your particular case, it won't be enough, since the first pass was very small, so you'll also have to change averflow settings to 20 %. But I prefer the second solution.

Now, I'm really astonished by the first pass size you're claiming to get. 225 MB for a 1h40 movie is really small, and I wonder whether they was a problem or not. What was your resolution ? What avisynth script were you using. And, by any chance, are you sure that the whole movie was encoded during the first pass ?

Edit : typo

[Koepi]

I'd also recommend using MPEG quantiser if a movie is that compressable, it'll give you some more detail than h263.

@Manao:

:goodpost: I'm enjoying these old-fashioned, "scientifical" posts - these are very seldom nowerdays :)

Regards

[greycortex]

The strange thing about this film is that I can get a first pass bitrate of around 325 kbps easy. I have no filtering, I'm using full resolution, and nothing else that I can think of that would be causing this issue. The entire film does get encoded, and it looks great when I don't have the quants capped, but when I do, it doesn't look that great.
I set it up for MPEG quant type before I left for work.

I realize that the quants at 1 would be causing this problem, I was just wondering how the bitrate scaling was done to take this into account. Is there an easily fit function of quant vs. bitrate that can be estimated with a single datapoint (quant2/bitrate at this compression) that fouls up when quant 1 is used?

If the MPEG doesn't work, I'm going to set my overflow high as Manao suggests. I'll give that a go tonight when I get back home.

I'm used to encoding anime and the usual stuff which has a lot more going on than in this film. Even The Seventh Seal was a more difficult encode. There is a _lot_ of locked-still camera work in this film, with minor movement of the actors, which probably explains the insane compressibility.

[Koepi]

uhm, sorry for being imprecise:

of course i meant _everything_ manao wrote _PLUS_ using mpeg quantisation type.

Sorry for the fuzz.

Regards
Koepi

[greycortex]

That did it! Thanks all!

[thoralf]

hey! this one would fit in the hall of really good encodes (http://forum.doom9.org/showthread.php?threadid=78008) for sure :-)

[manono]

Hi-

You didn't really mean that you did Knockin' On Heaven's Door at 704x224 did you? That was a typo, wasn't it? That's an AR of 3.14:1, while IMDB (http://www.imdb.com/title/tt0119472/dvd) says the R2 German DVD is 1.70:1.

But the Tomie movies are very compressible. You can use a high res and still use the AC3 track. They're dark and nothing much ever happens.

[Didée]

I am scratching my head. What am I doing wrong?
Originally posted by thoralf
[...] target size: 644kBytes)
The final is actually undersized, but hey - who cares.
Targeting a movie for 650k is already hazardous. But then getting it even undersized - that's Voodoo!

Please reveal your magic encoding secret!

:D

- Didée

[Soulhunter]

Yeah, two movies per 3.5" floppy disc !!!

XviD rocks... :D


Bye

[thoralf]

silly me ... the "pal anamorphic" option got somehow checked in gknot.
I ran the whole thing again, even with the right AR (resulting in a video size of 704x304) and thus roughly a third more data to be processed the vast majority of the frames still got quant 2 resp. 4. Never seen a movie like this before ...

Originally posted by Soulhunter
Yeah, two movies per 3.5" floppy disc !!!


We should have kept them ... :-)