Odd XVid bitrate scaling problem?

[SirCanealot]

Hi, all.

I've been having a weird problem with XVid in the last few things I've encoded.
I think, looking at an example which demonstrates the EXACT problem would be best, so here it is:
http://www.williams1.homechoice.co.uk/LoveLoveOP.mkv
I DON'T think this link will work. If it doesn't, hit http://www.williams1.homechoice.co.uk/ and crtl+f through all the random crap for the filename.
Shouldn't be a bandwidth limit on that, but if I get an angry email/letter from my ISP, I'll know not to put large examples up like that again :P

Basically, we've got the shot of the title with the sky moving in the background, at around 0:06 (too lazy to get frame numbers - problem is obvious :P), and the fast-ass moving pan-thingy at around 00:59.
My problem is, XVid is refusing to scale the bitrate in these sections. Insted, it raises Quants up horribly (using FFDShow to look at that).
Now, I was assuming this was an isolated problem (I didn't get the problem witht he title screen at 0:06 in a large 720x540 175 meg encode, just the fast pan), so I simply set a zone for it (assumed that XVid was simply trying to save bits, as it does).
The DivX5 raw this came from exibits none of these problems - in both parts the bitrate raises, and quants are kept low.

However, the problem again cropped up again earlier today when I looked at the second pass I'd made from a DVD - there was a fast-motion part that XVid, insted of jacking up the bitrate and keeping the quant, simply left the bitrate at the same level (again, judging from FFDShow) and raised the quant horribly.
The thing that tripped me about THAT encode, was that I set the ABR at 2000kbs (It's Hajime no Ippo R1 - I wanted to see what a high bitrate did with the little grain left in my light filtering didn't kill), so, with an ABR that high (average quants were 2 for I/P frames, even hitting 1 sometimes), I really hit me in the head a little.

Err, that LoveLove encode would have been done with XVid defaults, except quants set to 2-31, Trellis on, VHQ-BVop on, Cartoon Mode on, VHQ4, max I-Frame 250 frames, and that's it. Everything else = default.
Matrix wise, LoveLove was H.263 and Ippo was HVS-Best; as it's done it with 2 different matrices, I can't really put the blame on them.

And, as to why I'm posting this: I haven't had this problem until recently. I've also encoded many things with this SAME XVid 1.1 version, and I encoded Ippo ages back on XVid 1.0 and didn't get this problem. This is 100% recent, or is it 100% coinsidence?

Anyway, any advice would be great. I don't wanna have to hawk-eye over anything I encode and have to set zones for stuff like this...

Thanks, all.

[sysKin]

Originally posted by SirCanealot
Err, that LoveLove encode would have been done with XVid defaults, except quants set to 2-31, Trellis on, VHQ-BVop on, Cartoon Mode on, VHQ4, max I-Frame 250 frames, and that's it. Everything else = default.
Matrix wise, LoveLove was H.263 and Ippo was HVS-Best; as it's done it with 2 different matrices, I can't really put the blame on them.

And, as to why I'm posting this: I haven't had this problem until recently. I've also encoded many things with this SAME XVid 1.1 version, and I encoded Ippo ages back on XVid 1.0 and didn't get this problem. This is 100% recent, or is it 100% coinsidence?
You never said which xvid version you used :) I can see it's from 1.1 tree and I might guess it's Koepi's. It's important to confirm this because Koepi's build has some modifications, such as VBV switch and alternative Foxer's ratecontrol.
I am guessing they were not active? You said "defaults" but I'm not sure what the defaults are, with respect to Koepi's additions.

Radek

[SirCanealot]

Opps.
Yes, Koepi's 1.1 build.

And, do you want me to take a pic of my second pass settings? I can do that tomorow ^^

[Teegedeck]

Yep, that sounds like VBV's effect to me. (VBV is defaultet in Koepi's 1.1 builds.) It is problematic for high bitrates; for example an HDTV encode at 2nd-pass quant=3 (SixOfNine) is not possible with it. It will use absurdely high quantizers in order to free the buffer and never get near quant=3 even if the result is an undersized 2nd pass. VBV is not the way to go if you try to retain a near-constant quantizer during the whole 2nd pass, including high-bitrate scenes (because, at the end of the day, it acts a bit like the good, old curve-compression).

Try deactivating 'respect VBV buffer' on the 2nd pass tab.

[SirCanealot]

Great stuff.

I'll try turning that off.

Thanks all.

[IvS]

I'm having this same problem when encoding HD videos, with Koepi's 1.1.0 beta 2 build. Extremely high quantization values resulting in a very low quality video.
Try deactivating 'respect VBV buffer' on the 2nd pass tab.
I can't find this option, am I missing something?

[pelle412]

It's built into the chosen profile. Try not to use any DXN profile. Instead, use AS @ L5, or unrestricted.

[IvS]

I already did, I've tried everything, nothing helps this problem.

[IvS]

Well, no answers. Can anyone point me to a 1.1.0 beta2 build with deactivated "respect vbv buffer"? Or whatever's needed to make high resolution encoding possible.

[yaz]

... Can anyone point me to a 1.1.0 beta2 build with deactivated "respect vbv buffer"?just make your choice here (http://celticdruid.no-ip.com/xvid/). they are 'plain' cvs builds. i would suggest the very last one (051016) ;)
... Or whatever's needed to make high resolution encoding possible.dunno what specialty would be needed there but i know these builds work perfectly round dvd resolution for me.

the bests
y

[IvS]

Thanks yaz but this doesn't help either. With high definition resolutions, the requested target bitrate or file size are much lower, the quantizers used are much higher and the resulting quality is horrible.

[Didée]

Just tried a few things some minutes ago, and can not confirm any problem. Used CelticDruid's build from Oct-7, and 2-pass produced fully correct results with 1280'ish resolution, and requested bitrates 10'000, 15'000 and 20'000.

Hint: with HD resolutions, try to not do a "standard" 1st pass, but a custom 1st pass with q=3 or q=4, through zone setting during 1st pass. (But don't forget to switch back to weight=1.0 for 2nd pass!)
And of course, yes, also set profile to "unrestriced".

If that doesn't help, uninstall XviD & reinstall XviD. If that doesn't help either, s-th's b0rked with your system .. in that case de-install, manually clean registry from anything containing "xvid", then re-install.

[yaz]

Thanks yaz but this doesn't help either. With high definition resolutions, the requested target bitrate or file size are much lower, the quantizers used are much higher and the resulting quality is horrible.ehrm ... sorry ... i can't get it. if u increase the resolution while decreasing the target bitrate you can only get crap. it's self-evident. or do i get u wrong ? so. what does 'target bitrate or file size are much lower' mean exactly ? what compression do u try to achieve (2nd pass size / 1st pass size) ? how much is that 'much lower bitrate' exactly ?

the bests
y

[IvS]

yaz: The problem is with high definition resolutions (1280x720/1920x1200). Even if you set the target bitrate in the second pass to 10,000kbps, the resulting quality is very low because of (as stated previously in this thread) the "respect VBV buffer" issue. The quantizers the encoder uses are very high no matter what, the resulting bitrate is way lower than you choose.

Now that I think about this, the "respect VBV buffer" issue may also be related to the problem I was having encoding at very LOW resolutions as well (for cell phones etc), it seems like the same problem.

Didée: I haven't seen any mention of that trick for HD encoding regarding the first pass. Why do I need to mess with zone editing for this, how is the issue solved by doing so?
Edit: Also, doesn't setting a quantizer instead of weight result in lower efficiency?

[yaz]

... Even if you set the target bitrate in the second pass to 10,000kbps, the resulting quality is very low because of (as stated previously in this thread) the "respect VBV buffer" issue
... this, the "respect VBV buffer" issue may also be related to the problem I was having encoding at very LOW resolutions as well (for cell phones etc), it seems like the same problem.if u're right it's a fake of xvid. what u described is not rdo but the simplest panic-reaction. should be checked somehow. pls, try another build (not hacked w/that vbv thingy) and follow the suggestions given here (http://forum.doom9.org/showthread.php?t=87424). if u find the same it's not a vbv issue anyhow.
... I haven't seen any mention of that trick for HD encoding regarding the first pass. Why do I need to mess with zone editing for this, how is the issue solved by doing so?i guess, didée will answer u but so far ... running a 1st pass w/cq setting it to or near to the target average q is always the best choice. it provides the optimal pass file for the 2nd pass. my only problem is how to figure out what'd be the avg q in the second pass before completing the 1st pass ;)

the bests
y

[IvS]

yaz: This is not an issue of a "fake build" or a "panic-reaction" on my part. I've tried several builds already, nothing is wrong with my computer, the issue is reproduceable on other computers, etc, etc, and so on. Test it for yourself.
As Teegedeck mentioned:
Don't use the 'respect VBV' option in two-pass. At these bitrates it would produce absuredly high quantization in second pass in order to empty the buffer.
This is exactly the issue I'm experiencing, most probably because 'respect VBV' is enabled in all the builds. I haven't seen any build for Windows/vfw, with an option to disable it manually.
Regarding the constant quantizer thing, as you said, it's not possible to predict the final result so it's not really a sane option for encoding entire videos. For a "zone", sure.

I'd appreciate any suggestion on how to solve this issue. Maybe someone could make a test compile with disabled "respect VBV"?

[stephanV]

VBV buffer is enabled/disabled by choosing the type of profile. If you choose "unresticted" VBV is disabled.

Perhaps you could upload a small source clip that shows the issue (e.g. to savefile.com), post your exact settings, scripts, etc, so we can try to help/reproduce the behaviour. I haven't seen anything horribly wrong doing HD with XviD

[IvS]

stephenV, what you said was already mentioned, but as I said before, unrestricted doesn't solve the problem. Sorry but I don't know what you're talking about, have you encoded HD clips yourself?
To anyone who'd like to check for themselves, try for example 720p5994_parkrun_ter.yuv from ftp://ftp.ldv.e-technik.tu-muenchen.de/dist/test_sequences/720p/ - exceptionally horrible quality.

[yaz]

This is not an issue of a "fake build" or a "panic-reaction" on my part.of course, i didn't mean u but xvid :) in my view, 'rdo' is for preventing such bitrate a/o quantizer peaks while accepting the vbv restrictions. from your example it seems as if xvid corrected buffer under/overflows in jerks. that's what i call panic-reaction.
however, in some instances such kinda jerks are unavoidable, they can only be smoothed or damped somehow. it'd be interesting to see how high the bitrate would run if the quantizer stayed in a narrow(er) range. u can check it by setting a cq zone for that critical part. (use unrestricted profile !) i bet, u get a huge br peak there.

as regards fake build and hacks, i must apologize. i remembered koepi released a build w/switchable vbv ... but that was a non-public 1.1 prerelease just for testing. afaik, 'official' builds never had such switch. sorry.

the only way to switch off vbv is 'unrestricting'.

the bests
y

[stephanV]

stephenV, what you said was already mentioned, but as I said before, unrestricted doesn't solve the problem.
Then the problem is not VBV. Or the profiles are buggy.

Sorry but I don't know what you're talking about, have you encoded HD clips yourself?
Yes.

But using two pass on a 10 second clip probably wont work too well no.

[Didée]

I haven't seen any mention of that trick for HD encoding regarding the first pass. Why do I need to mess with zone editing for this, how is the issue solved by doing so?
It's not exclusively related to HD encoding. A quick search digged out this (http://forum.doom9.org/showthread.php?t=95157), this (http://forum.doom9.org/showthread.php?t=95103&), this (http://forum.doom9.org/showthread.php?t=89342&) and this (http://forum.doom9.org/showthread.php?t=74182) thread. There are probably more, the search was quite narrow.

A quick 2-pass test, target = 1585 kbps ...

... with standard 1st-pass (@ q2, a.k.a. "fast, cause 'all features disabled' ") :

http://img462.imageshack.us/img462/6657/standardfirstpass5df.png


... and with custom 1st-pass (full-Q, zone=q4) :

http://img462.imageshack.us/img462/2421/customfirstpassatfour6vh.png


Shortly, the former looks poor in several places, and did undershoot by 10%. The latter looks good, and is on-spot. Note that this was a normal PAL source, 16:9 letterboxed. With HD resolutions, and especially when *no filtering* is used, the effect might be even stronger.
Explanation is to be found somewhere in the threads linked above, although yaz brought it to the point already, above.

Edit: Also, doesn't setting a quantizer instead of weight result in lower efficiency?
Nope. The analyse pass deals for, surprise, analysing.

[Didée]

In answer to this (http://forum.doom9.org/showthread.php?p=739532#post739532):

sysKin: this [note: this thread here] is an issue that needs to be fixed. Also note, HD encoding of short clips, for example 10 or 20 seconds, results in very much lower bitrate than stated. The longer the length, the more accurate it is. A clip of several minutes has an accurate bitrate.
No it isn't - surely not for v1.1. It even is hardly a real issue. Knowing how the 2pass scheme actually is working, this behaviour is to be expected: when 2nd-pass destination bitrate is too far away from the q2 (!) 1st-pass, then it's not that easy to achieve correct bitrate scaling instantly. (I gave you links to read up.)
Have an idea where your 2nd pass will be going, and set up the 1st accordingly. If you don't have, you still have the "overflow treatment" settings available to force XviD being more aggressive in reaching the wanted bitrate.

[el divx]

How about making XviD actually compile under MinGW?

[stephanV]

How about making XviD actually compile under MinGW?

worked for me last time i tried... how about giving some compiling errors?

[celtic_druid]

Always worked fine for me. I have only compiled xvidcore lately, but I am sure VfW and dshow would still be ok.

[suxen_drol]

How about making XviD actually compile under MinGW?

xvidcore, vfw and dshow compile correctly here using gcc-3.4.2 and w32api 3.2. please describe the problems you are experiencing, and the version of your compiler and w32api header files.

-- pete

[el divx]

MinGW 5.0.0 with GCC 3.4, w32api 3.5.

And this is what I get when running make(having already ran bootstrap.sh and configure):
$ make
D: =build
/bin/install: Installation: command not found
Unknown command: `Makefile'

CVS commands are:
add Add a new file/directory to the repository
admin Administration front end for rcs
annotate Show last revision where each line was modified
checkout Checkout sources for editing
commit Check files into the repository
diff Show differences between revisions
edit Get ready to edit a watched file
editors See who is editing a watched file
export Export sources from CVS, similar to checkout
history Show repository access history
import Import sources into CVS, using vendor branches
init Create a CVS repository if it doesn't exist
log Print out history information for files
login Prompt for password for authenticating server
logout Removes entry in .cvspass for remote repository
ls List files available from CVS
rannotate Show last revision where each line of module wa modified
rdiff Create 'patch' format diffs between releases
release Indicate that a Module is no longer in use
remove Remove an entry from the repository
rlog Print out history information for a module
rls List files in a module
rtag Add a symbolic tag to a module
status Display status information on checked out files
tag Add a symbolic tag to checked out version of fi s
unedit Undo an edit command
update Bring work tree in sync with repository
version Show current CVS version(s)
watch Set watches
watchers See who is watching a file
(Specify the --help option for a list of other help options)
/bin/install: line 4: syntax error near unexpected token `(C)'
/bin/install: line 4: `Copyright (C) 1994, 1995, 1996, 1999, 2000, 2 1, 2002, 2004, 2005 Free'
make: *** [=build] Error 2
Any ideas?

[celtic_druid]

You ran bootstrap.sh and ./configure first?

[el divx]

Yup. 1st ./bootstrap.sh, then ./configure and then make.

[CruNcher]

@Didée
I tried to be very strict what target bitrate means with my EDP Build could you try it ?

[suxen_drol]

Yup. 1st ./bootstrap.sh, then ./configure and then make.

it appears that the /bin/install command was not detected properly, however for some reason, configure did not raise a warning. please view your "config.log" file to confirm this. my log file states:

> configure:2360: checking for a BSD-compatible install
> configure:2414: result: /bin/install -c

maybe this has something to do with mingw 5.0? as a workarround: if you can find where install is located, just edit your "platform.inc" file and set the INSTALL var appropriately.

-- pete

[el divx]

Tried some path combinations but didn't work.

Oh, I forgot to say that I use MSYS 1.0.11 and have also installed in my system Visual C++ Express Edition 2005 along with the latest available Platform SDK and DirectX SDK Octomber 2005 just in case.

[IvS]

Didée: That issue is definitely an issue, I don't know what makes you say it's not even important enough to fix. Why should other encoders not have this issue? Even x264 which is very new compared to xvid handles short clips properly. Short clips do exist. The user shouldn't have to guess things in that manner and I don't think bypasses like the "overflow treatment" should be accepted as a proper way of dealing with the issue.

[CruNcher]

The user shouldn't have to guess things in that manner and I don't think bypasses like the "overflow treatment" should be accepted as a proper way of dealing with the issue.


*cough* *cough* yeah true words for the moment it's only trying to heal the wounds coused by this problem not fixing the actuall problem that couses them, but XviD isn't the only Mpeg-4 SP/ASP encoder with that problem so it seems the problem is in the architecture itself. I doub't that any SP/ASP encoder available will ever reach the bitrate target as good as H.264 encoders can.