1-pass constant quality for standalone harddisk player [Archive]

View Full Version : 1-pass constant quality for standalone harddisk player

ronnylov

13th April 2007, 13:03

I have a TViX M-4000P player with internal hard drive and wireless 54g connection for streaming.
http://www.tvix.co.kr/Eng/products/HDM4000.aspx

It has a Sigma Design EM8621 chipset:
http://forum.doom9.org/showthread.php?s=&threadid=84003

I have verified that XviD works with or without Qpel, with 1 consecutive b-frame (sometimes I get stuttering at playback when using 2 consecutive b-frames), with or without packed bitstream but not with XviD GMC. Custom Quantizer matrices works too with this player.

I was thinking since this player has an internal hard drive (I have mounted a 320 GB Samsung IDE-drive) I do not have any real size limitations of the encoded files other than that I have discovered that avi files >2GB has seeking problems (fast forward and goto functions locks the player and I have to reboot). I solve this by splitting files to make them smaller. However my goal is reasonably good quality with not unnecessary big file sizes to be able to use the harddrive space effecient.

So my first thought was trying to make a general XviD constant quality setting that works without problems for this player. But sometimes I get stuttery playback, especially at complex scenes and high definition encoding (like 25 fps 1280x720 at q=3.5).

Then I read this thread:
http://forum.doom9.org/showthread.php?t=118419

But the VBV settings seems to be only for second pass. Can I have some kind of VBV problems when doing constant quality 1-pass encoding?

So the main question I want to ask is if it possible to have a "standalone safe" 1-pass constant quality encoding with XviD? Or is 2-pass encoding preferred even if the filesize is not critical, only SAP compability is critical? I think things like maximum bitrate should be possible also with 1-pass encoding (like a consant quality up to the maximum limits of the player capabilities). But those fields seems greyed out in the xvid encoder configuration and it also seems to refer to second pass.

I have not done much XviD encodings before I got this player. I have made some MPEG-2 encodings for DVD and then I mainly use HC encoder nowadays. There I can set constant quantizer and a maximum bitrate limit. I can also check DVD player compability. Now I am trying to do the same thing with XviD, but it is a little more tricky when the player maximum limits are not standardizes like it is for DVD Video...

The reason I like a maximum bitrate setting is that my bandwidth at wireless streaming from my PC is somewhat limited. It seems that around 6 Mbit/s is the real transfer speed when transferring files to the TviX unit with FTP. I think the same limit applies to playing remotes files on the PC from the Tvix player.

Sorry if I sound a little bit confused, but I really feel like an beginner of XviD encoding but I think I have farly good knowledge of video encoding in general anyway. What I want is some geneeral recommendations for this subject and perhaps some friendly advices how to achieve my goal.

plugh

13th April 2007, 17:04

It has a Sigma Design EM8621 chipset:
http://forum.doom9.org/showthread.php?s=&threadid=84003

So my first thought was trying to make a general XviD constant quality setting that works without problems for this player. But sometimes I get stuttery playback, especially at complex scenes and high definition encoding (like 25 fps 1280x720 at q=3.5).

Then I read this thread:
http://forum.doom9.org/showthread.php?t=118419

But the VBV settings seems to be only for second pass. Can I have some kind of VBV problems when doing constant quality 1-pass encoding?
I'm not familiar with that chipset, but in general "yes".

A constant quant encode means you don't care about bitrate; if some particular frame requires 8 megabits to represent at that quant, your chipset needs to be able to process those bits in (basically) one frame period of time.

Every cpu ("chipset") has it's limits...

So the main question I want to ask is if it possible to have a "standalone safe" 1-pass constant quality encoding with XviD? Or is 2-pass encoding preferred even if the filesize is not critical, only SAP compability is critical? I think things like maximum bitrate should be possible also with 1-pass encoding (like a consant quality up to the maximum limits of the player capabilities).
'possible' ? yes
'implimented in xvid' ? no

The problem is, such an algorithm requires a 'on the fly' "oops, let's go back and reencode that frame at a lower quant" capability.

Your only option with xvid is 2-pass -
first pass (which is at constant quant),
second pass recognizing and responding to the "oops".

ronnylov

14th April 2007, 22:46

Thank You plugh! Your explanation makes sense to me.
So I guess that I need to make a first pass at constant quant to get the filesize I need for my wanted quality and then applying that filesize/bitrate to the second pass to have the limitations applied.

I noticed that no matter if I select any of the profiles or unrestricted profile the VBV parameters are greyed aout and I can not manually set the desired maximum peak bitrate I want. I think the idea then is to just choose one of the available profiles and see if the player can play the resulting file.

plugh

15th April 2007, 00:12

Thank You plugh! Your explanation makes sense to me.
So I guess that I need to make a first pass at constant quant to get the filesize I need for my wanted quality and then applying that filesize/bitrate to the second pass to have the limitations applied.
The first pass IS constant quant - that is the only option. In your scenario, I would set the 1st pass options to do 'full quality' and 'keep the file'. Then try playing IT to get an idea of how bad your problem is. If you are lucky, it will play as-is, and you are done :) If there are a few short sections with drops / stutters then do the second pass with the target file size same as the 1st pass output file and appropriate vbv params to clip those peaks. If major sections are completely unplayable, you may need to rethink - lower the resolution, increase b-frames, raise the quant target (ie make a smaller file)...
I noticed that no matter if I select any of the profiles or unrestricted profile the VBV parameters are greyed aout and I can not manually set the desired maximum peak bitrate I want.
The profiles were done wrt the mpeg4 standard and divx compatibility, so the vfw gui design didn't anticipate people using them as a general tool for arbitrary standalone compatibility. I assume you are using my build from the other thread - I included all the profiles I was able to find floating around on the net in addition to the ones I created for my target platform (I've actually revised mine slightly since then).

Determining the parameters for your target is a trial and error process. Start with the 'full quality' const quant first pass, then do second passes to refine the vbv settings. Once you've got them, you are set, but getting them is a chore...

Fortunately, I was able to instrument my target platform so I could get a record of exactly what frames/scenes caused problems. Then by turning on the debug prints embedded in xvid (enabled in my builds), I could see which sections of the first pass 'curve' the various vbv settings would adjust and to what degree. (This happens at the very beginning of the 2nd pass, so you don't have to actually do the full encode). Several iterations and you've got candidate values; let the 2nd pass complete, then test the output file for playability...

I have another build where I hijacked two obsolete fields on the vfw second pass config page to allow direct input of vbv bufsize and maxrate, but it is a hack (not nicely integrated, no checks). The command line encoder "xvid_encraw" available in this forum does allow specifying arbitrary vbv values (and has my 'alternate second pass' vbv code fixes), but I don't think it has the debug prints enabled in that module (it requires a few lines of additional code in encraw).
I think the idea then is to just choose one of the available profiles and see if the player can play the resulting file.
Sure, try out the custom profiles already built in as a starting point and see what happens, but you'll probably want to refine them at some point...

ronnylov

15th April 2007, 00:41

This sounds way too complicated to me. It seems you are matching playback problem areas with vbv values to limit exactly the same areas in the encoder. I don't think I have equipment and skills to do this at the same level as you have done it. It will be more trial and error for me and using existing profiles to find anything useful.

I noticed the XBMC-HD profiles with your modded dll file but the custom matrices and MPEG matrice were greyed out. I think I will start by using them because my player can handle HD XviD. But the custom matrices and MPEG matrice option is greyed out. Can these fields be accessed by using the command line encoder?

plugh

15th April 2007, 03:49

This sounds way too complicated to me. It seems you are matching playback problem areas with vbv values to limit exactly the same areas in the encoder. Basically, yes. I wanted to establish vbv parameters (bufsize, maxrate) that accurately reflected the abilities of my target platform, so that the vbv constraint only affected what it _had_ to...
I don't think I have equipment and skills to do this at the same level as you have done it. It will be more trial and error for me and using existing profiles to find anything useful.The tough part in your case would be recording the problem segments. You'd have to manually note the playtime clock and convert that to frame number ranges so you could check that against the xvid debug output.
I noticed the XBMC-HD profiles with your modded dll file but the custom matrices and MPEG matrice were greyed out. I think I will start by using them because my player can handle HD XviD. But the custom matrices and MPEG matrice option is greyed out. Can these fields be accessed by using the command line encoder?In my analysis of xbmc behaviour, I noted that mpeg and custom matrices incurred an additional cpu load vs h263 quantizer. This additional cpu load reduced the number of macroblocks per second it could process, so I disabled that option (and others) in my 'xbmc-hd' profiles.

By the way, another variable to consider - audio. Don't know how your chipset handles audio decoding, but in my case software audio decoding of various formats presents a cpu load that detracts from the video decoding ability (that varies depending on the format). Just be aware that finely tuned video settings may need adjusting depending upon the associated audio... That's part of the reason for my range of 'xbmc-hd' profiles - the highest one is for digital audio output (ie no audio decoding at all).

In answer to your question - I believe encraw gives you access to *everything*, but I've not used it...

ronnylov

15th April 2007, 10:54

Thank You again!

I did an experiment. I choosed the XBMC-HD hdd profile and under the "Quality preset" I choosed more and changed min quantizer values of all frame types to 3. Then I set up the encoder to a "Two pass - 1st pass", followed by a "Two pass - 2nd pass alt". The bitrate was set to maximum to let the encoder use as high quantizers as possible (which I thought would be 3). The statistic window showed after the finnished encoding that a lot of frames with quantizer value of 2 had been used. The bitrate came out at approximately 15 mbit/s of a 1280x720 encoding and the quality on the computer is great. However I wonder why the min q-values I had set was not respected by the encoder?

I have not tested it yet on my standalone player, it is transferring the files via ftp right now and it hogs some CPU power from the player so I have to wait until it is finnished.

plugh

15th April 2007, 22:00

Not sure I followed all that, so I can't comment.

I think what you are wanting to do is a constant quant 3, vbv constrained encode? What *I* would do -

first pass:

Profile@Level - pick a profile (you mentioned 'xbmc-hd hdd')
Click More, set your b-frame options (You really want *everything* at 3? set quantizer ratio to 1, offset 0)

Encoding type - Twopass - 1st pass
Click More - check Full Quality first pass, uncheck Discard

Zone options - switch to Quantizer mode, set to 3

Quality preset - click More
set Motion options as desired
note: Quantization min/max have no effect on 1st pass

Other Options - check Display Status

Do the 1st pass encode.

Note that the display includes vbv stats. If there were NO violations, doing the second pass with that profile will have NO effect! Your full quality first pass output file is constant quant as you desired, and already satisfies the selected vbv constraint. You are done! If it *does* show violations, you want to do the second pass.

second pass:

Encoding type: Two pass - 2nd pass alt
Click More - Set all values on this page to zero.

Enter target size equal to the 'total size' shown on the first pass xvid stats page. OR, use the calculator, plug in the actual size of your first pass output file, your movie length, frame rate, audio format etc, and let it fill in the video target size.

Zone Options - switch back to Weight 1.0

Quality presets - Set your Quantization *MINS* to 3 and leave the maxes alone.

Now do your second pass.

This time, all the quant 3 frames from your first pass will be recreated, except in those scenes where the vbv constraint was violated - these will be smaller, with a higher quant. Your output file will be smaller as well.

Comments -

In the specifc case of my XBMC-hd profiles, the only difference amongst them is the vbv parameters. Since the vbv parameters are only acted upon during the second pass -> you can change the profile and redo the second pass to change the vbv constraint <- you don't need to redo the first pass.

There is nothing that says you HAVE to zero out all the entries on the 2nd pass config page, or set the quantizer mins, or specify exactly the same 2nd pass output size, if you want to do other adjustments to your first pass result.

For example, one thing I often do is:
Note the total size of all I-VOPs on the first pass stats.
For the second pass, I increase the target size by that amount, set "I-frame boost" to 100%, and I-frame min quant to 1. My second pass output now has basically all quant 1 I-frames, subject to the vbv constraint.

Hope that helps.

ronnylov

16th April 2007, 10:41

Thank you again! It helps a lot!
Basically what I am trying to do is a "constant quality" encoding with the limits accordingly set to my players hardware limitations. Your suggestions are very welcome and I will try it.

Well, my previous test failed (the framerate of playback drops at more complex scenes) so I'll try a more restricted profile and work upwards from there.

What is the relationship between the different vbv parameters "buffer size", "max bitrate" and "max bits over any one second interval"? It seems that max bitrate is the maximum bitrate I can set with the bitrate slider and I think that max bits over any one second interval would be the maximum peak bitrate in the variable bitrate used. But how is the buffer size used?

I mean if I have problems in the playback of high bitrate peaks then it can be too high peak bitrate, or too small buffer size in the player hardware. If I do CBR encodings with different bitrates I can find the maximum sustained bitrate it can play. Perhaps by encoding a test clip with low motion and a short one second action scene in the middle of the video to get a one second bitrate peak I can test the max peak bitrate. But how do I test the maximum buffer size? Is there any tool I can use to find out the peak bitrate?

EDIT: I found this thread regarding the subject of VBV settings:
http://forum.doom9.org/showthread.php?t=122100

I have an idea how to check for framedrops. I can connect the S-VIDEO output of the hardware player to my analoge TV capture card and compare the original source video with the captured video.

plugh

16th April 2007, 14:55

What is the relationship between the different vbv parameters "buffer size", "max bitrate" and "max bits over any one second interval"? It seems that max bitrate is the maximum bitrate I can set with the bitrate slider and I think that max bits over any one second interval would be the maximum peak bitrate in the variable bitrate used. But how is the buffer size used?
I knew that question was coming ;)
To begin, for various reasons just ignore the "max bits over any one second interval". If you are using encraw, set it to zero. The other two are the important ones, and are part of the spec. See http://www.m4if.org/resources/profiles/index.php for the nitty-gritty.

First off, understand that these are not two independant variables. They are used together in an algorithmic model. They form a continuum. The easiest way I've found to think about them is to look at the two extremes of this continuum.

Extreme 1 - a single frame
Take "max bitrate", divide by 8, and divide by the frame rate
Now you have a value in units of 'bytes per frame', call it X
Now lets define the 'current buffer level' (call it Y) and assume it is equal to the 'vbv buffer size' (call it Z)
if current frame size > X+Y you have a violation

note that Y is a dynamic value, whose largest possible value is Z, the 'buffer size'. Y=Z (ie X+Z above) is as good as it gets.

Extreme 2 - a very long run of 'N' frames
Add up all the frame sizes for the long run.
if that Sum > N*X you have a violation
to put it another way Sum/N (an average) should be < X
(note that Z - buffersize - has no significance over the long haul)

Now lets look in between.

Let's say you have a sequence of two frames, each (X+Y) in size. Individually they don't violate extreme#1, but together they violate extreme#2. (X+Y)+(X+Y) > 2*X

Now let's say you have a sequence of two frames, one (X+Y) in size, the next (X-Y) in size. Individually, neither violates extreme#1, and in combination (X+Y)+(X-Y) = 2X, so they are OK.

Now this is a model. You can apply this model for differant purposes. One purpose might be an actual physical memory buffer and a bitrate constrained channel. Players using optical media or network links can have a very real channel constraint. And standalone chipset design/cost constraints might mean there is a very real memory buffer limitation.

On the other hand, your device has a hard disk, so practically speaking you don't have a 'bitrate constrained channel'. Your goal is to constrain processor loading. Now look at my description in this context.

A single very large frame could exceed the ability of the cpu to process it and have it ready when needed - resulting in this frame being 'late' and perhaps even dropping the next frame. And a run of large frames could exceed the ability of the cpu to keep up with the load over time.

So the two parameters describe a constraint on a stream, with the "max bitrate" describing a maximum-average ability and the "buffer size" describing short term variability or elasticity ("headroom").

Putting this into an encoded stream context - The 'very large' I frame at the beginning of a scene oftimes needs the short term elasticity. Sequences of 'large' P-frames followed by 'small' B-frames helps keep your average below the 'long term' envelope. And contiguous sequences of 'largish' I and/or P frames may or may not need a higher quant, depending upon both parameters; if they are just a little bit above the maxbitrate, then they will start eating into your headroom. How long this can be sustained depends upon the buffersize. And the recovery time between peak bursts depends upon your current-average relative to the max-average (the difference between them determines how long it will take for the buffer level to get back to 'buffersize').

Anyway, those are the dynamics. Hope it helps...

I mean if I have problems in the playback of high bitrate peaks then it can be too high peak bitrate, or too small buffer size in the player hardware. Think about this again in light of the above (and just forget about that third parameter - really).

If I do CBR encodings with different bitrates I can find the maximum sustained bitrate it can play. Perhaps by encoding a test clip with low motion and a short one second action scene in the middle of the video to get a one second bitrate peak I can test the max peak bitrate. But how do I test the maximum buffer size? Is there any tool I can use to find out the peak bitrate?Here's an idea - a large I-frame followed by small frames. ie a scene change to a very complex image that doesn't change. At some point, a sufficiently large one will cause a frame drop.
I have an idea how to check for framedrops. I can connect the S-VIDEO output of the hardware player to my analoge TV capture card and compare the original source video with the captured video.

And he's off and running :)

ronnylov

17th April 2007, 22:05

plugh

18th April 2007, 14:45

I do not get min quant of 3 despite of setting that in the quant tab...I can't think of any way for this to occur with the settings I specified.Maybe it is because with the filesize I get in the first pass the average bitrate is already above the profile max limit?That would be extraordinary. The 1st pass stats showed all violations? If your average is above the max, then each of your quant 3 frames is, on average, larger than 100KB (assuming 24fps)The resulting file is much bigger then the set filesize (231 MB instead of 149 MB).Well, at least this is consistent with your first statement.Did I install your mod correctly by just replacing the xvidvfw.dll file that was postedIf it wasn't correct, you wouldn't see profiles or 2nd pass alt option.

Redo it, using the settings I gave, and post snapshots (alt-printscreen) of the xvid status window at the end of your first and second passes. And a bit more info about the clip and the other setting (which profile, etc) would be helpful.

Blue_MiSfit

24th April 2007, 21:54

I'm not sure why you are limiting quantizers. Especially B frames. Their lot in life is to be a higher quantizer (and therefore smaller) than P frames, while still looking good :)

It seems to me that what you need to do is setup a normal 2 pass encode, with the second pass at whatever bitrate and VBV settings you want.

But I confess, you guys are using some special built with an alternate 2 pass method, so maybe that's the way to do it... :)

Just my 2 cents.

~MiSfit

ronnylov

6th May 2007, 00:13

I was using the standard HDTV preset when I got the strange result above but after remaking both passes at a lower bitrate it was OK. I have not done any more testing to find out what went wrong that time.

Right now I am doing some testing of the hardware limitations. I have started with CBR encodings with the unrestricted profile to find out Extreme 2. It seems that the limit of my player is somewhere around 17 Mbit/s and 18 Mbit/s but I have some problems with the complex scenes in the beginning of "BBC Planet Earth Episode 3" that I recorded from SVT HD in Sweden and I am transcoding to Xvid.

Anyway, assuming max average bitrate is 17 Mbit/s it would make the value X=85000 bytes (at 25 fps). XviD status window is showing I-VOP max frame size 500738 bytes (and now I have minimum quantizers set to 1 in my settings). Well I am still a little bit confused of what I should do. Assume I find a maximum frame size that not cause a frame drop when testing a scene change to a complex image that not change. How do I translate it to buffer size?

Can I take the maximum single frame size and subtract it with value X to get the buffer size? If I in this esample have X+Y=X+Z=500738 and X=85000 I guess maximum Z would be 500738-85000=415738.

Let's see. Max bitrate a multiple of 400. 17000000 bit/s is a multiple of 400 (or is it kilobits that would be a multiple of 400). Well still OK even at kilobits/400.

VBV would be a multiple of 16384. 415738/16384= 25.37...
25*16384 = 409600

But 409600 seems too low for buffer size when comparing with the profiles. Can I really create a single frame with a size big enough to fill the whole VBV buffer?

Note that this is just an example, I have not tested a single big frame yet (my encoding example had many big frames).

plugh

6th May 2007, 02:32

Those numbers look quite reasonable/possible to me.
But 409600 seems too low for buffer size when comparing with the profiles. Can I really create a single frame with a size big enough to fill the whole VBV buffer?It isn't just about max frame size, but 'elasticity'. What if you happen to create a sequence of 200KB frames? The amount by which each frame exceeds the maxrate eats into the buffer, and the size of the buffer says how long this can go on. If it goes on for "too long", you'll have to drop a frame to maintain audio-video sync. If it's a "short" burst, followed by frames less than 'maxrate size', the system will recover without drops.

Let's say each of those 200KB frames takes 1.125 'frame-ticks' to decode. By the time you get to the 6th one you are over half a frame-tick 'late' and the system may decide to try to catch up by skipping it. However if four 1.125 tick frames are followed by a couple small/easy 50KB frames that only take say .75 ticks to decode, system catches up and no harm done. The shortest possible 'burst' might be a single 500KB frame (unusual), but longer runs of less extreme oversize frames would also be "ok". Made up numbers, but you should get the drift.

(EDIT: expanded above explanation)

Pick up my latest xvidvfw build here (http://forum.doom9.org/showthread.php?p=999248#post999248) and try the "XBMC-HD low" profile (they have been revised from earlier set) which has vbvmax=17,798,000bps and vbvsiz=4,194,304 bits (512KB).

(You can ignore the "hvs plugin" stuff in that build; it won't affect you at all unless you turn on adaptive quant *and* define some environment variables.)

BTW, those profiles now have mpeg/custom matrix enabled, but be forwarned that non-h263 quantization might increase cpu load. (Definitely does so with xbox/xbmc)

One other comment: using the vbv model to constrain cpu load is an imperfect mapping. You are using a model based upon quantity of bits, but in terms of the cpu load incurred processing them, 'not all bits are created equal'. In practice however, with the typical mix of "stuff" in a video stream, it usually works out ok. Point is, don't expect a set of vbv values that works with one set of encoding options to work with any set.

ronnylov

7th May 2007, 09:30

Thanks again for your help. I will try your new build and see what happens.

During my testing with the unrestricted profile I noticed I have difficulties when encoding an easy encoded part followed by a difficult part (the slow title scene of Planet Earth was followed by a fast introduction with complex nautre sceneries with fishes in rivers, swimming monkey, hunting crocodile and thousands of flying birds). When encoding both slow and fast scenes in the same video clip with CBR 17500 kbit/s I get problems after a while in the fast scenes but when only encoding the fast scenes with CBR 17500 kbit/s I do not get the problems. Is it more demanding for the decoder when going from an easy scene to a complex scene compared to encode a complex scene only?

As you have mentioned earlier regarding audio I have also noticed that different audio formats resuls in different CPU load. AAC and AC3 is more demanding for this player than MP3 and MP2 audio. I think MP2 audio is the lowest demanding format of the ones I have tried and that suits me fine because I mostly make DVB recordings where a MP2 audio track is included (and sometimes with an additional AC3 track). When using MP2 audio I can use higher video bitrate without decoding problems compared with AC3 for instance.

plugh

7th May 2007, 12:41

CBR encoding isn't exact; the quants used to attain your rate with the easy scene probably result in an overshoot at the beginning of the complex scene until encoder reacts and adjusts the quants upward to compensate.

You may have found an interesting test case. Re-encode it using exact same settings, except select that 'low' profile. Watch the xvid status window as it works its way through that section. If that profile is about right, you should see it eat into the buffer, generate some violations, then recover. And assuming you are using virtualdub, the little graph on the Video tab of its status window should show the bump, too.

re: audio - Yes, I've had similar findings with xbox/xbmc.
In order: stereo MP2, stereo ac3, stereo MP3, stereo AAC
Was a little surprised at where AC3 2.0 fell, but since MP3/AAC are more highly compressed formats, requiring more cpu work to reconstitue the audio stream, it makes sense. Of course, the efficiency of the actual software codec used also plays a role.

Kind of fun isn't it, figuring out ways to probe the box's behaviour and drawing conclusions from it...

plugh

7th May 2007, 13:54

By the way - it may a complete non-issue for you, but what kind of cooling does your box have?

xbox has an adjustable-speed fan and thermal sensors, and xbmc uses these to impliment a thermally adjusted fan in software. Playing HD encodes definitely presents a higher cpu load, generating more heat and thus xbmc spins fan faster to compensate. Your Sigma chipset will similarly be generating more heat with this workload. You might want to keep an eye on this aspect.

ronnylov

7th May 2007, 19:22

My box has a 40 mm auto adjusting fan but I can select different spped modes and right now it is set to "normal" but since it is behind a closed glass door in the TV bench I may increase the speed to fast just in case it overheats. Thanks for the tip.

ronnylov

9th May 2007, 22:59

Now I have done some testing with the XBMC HD low profile and it works up to 13 Mbit/s average bitrate but increasing to 13.5 Mbit/s average bitrate or higher causes a q=2 constant quantizer and a real average bitrate of 28 Mbit/s. (when set to 13 Mbit/s the resulting average bitrate was 11.7 mbit/s, that is more reasonable after the VBV-limiter has done its work). I noticed that in the slow title part the quantizer of all frames was q=1 at a setting of 13 mbit/s bitrate and then it was adjusted down in a correct way avoiding VBV overflows.

So it seems that if average bitrate is set too high in the 2nd -alt pass then something is causing a q=2 pass. Now I had not limited the quantizers (minimum was set to 1 and maximum 32) to make it adjust the bitrate as it wants.

So how can I avoid this upper bitrate limit that makes everything go wrong when encoding? Can it be someting like it tries to set a q lower than 1 and thinks "nah that's not possible" and just makes a default 1st pass instead of a proper 2nd pass? I don't know why this happens but apparently it goes wrong when average bitrate is set too high.

plugh

10th May 2007, 16:00

I've been puzzling over your post, and you seem to have a couple of issues, including a misunderstanding...

Xvid only has one CBR mode - "Single Pass, Target Bitrate"

If you choose "Target Bitrate" for the 2nd pass, all it is doing is using that value, multiplied by the movie duration, to get a target size. This is an 'average' rate, not a 'constant' rate.

You said earlier you were doing CBR encodes to probe your box. This says to me "Single Pass, Target Bitrate", and is a neat idea to probe for the maximum sustained bitrate of your box. However, the CBR mechanism isn't perfect and you might get overshoot across a 'easy/hard' scene transition - a case you seemed to have encountered.

I suggested you redo *that* CBR encode using the *same* exact settings (ie redo the "Single Pass, Target Bitrate" encode) with "XBMC-HD low" selected. This will NOT limit or vbv constrain the CBR encode output, but the Xvid Status display would show the VBV compliance of That encode against That profile.

If that particular profile's VBV parameters are 'about right' for your box, then, when it checks your CBR "Single Pass, Target Bitrate" encode, the bulk of the frames should pass 'ok', and the section with the 'easy/hard' transition which you said caused your box some problems should be RECOGNIZED as a problem area by the VBV check - the status display should show the buffer being consumed and violations occurring during that section, and ONLY that section.

Note that nowhere in the above does "2nd pass alt" come into play. What you are using is the VBV compliance check built into the Xvid Status window to evaluate a CBR encode using a given set of vbv parameters. The CBR encode at 17-18mbps that you said worked OK except for one specific section is providing you with a test case to see if a specific set of VBV parameters RECOGNIZES this problem segment. You don't want a set of parameters that clips non-problem sections, nor do you want a set of parameters that accepts sections you know are problematic.

=============================================

With respect to the "2nd pass alt" problem you indicate you are having, I would need for you to capture the debug output to make any progress on this. Set the debug output mask to '0x80' and use 'debugview' utility to capture the '2nd pass alt' debug info to a file. Post a pointer...

EDIT: Also, are you using the xvidvfw update I referenced earlier (with the revised profiles)? I ask, because the 'low' profile in my original release had vbvmax ~13Mbps vs the 'low' in that update has vbvmax ~17.8Mbps...

ronnylov

10th May 2007, 21:05

I've been puzzling over your post, and you seem to have a couple of issues, including a misunderstanding...

Xvid only has one CBR mode - "Single Pass, Target Bitrate"

Sorry to confuse you. I am aware of the difference between the CBR mode and 2-pass encoding. Yes I was making CBR encodes earlier but to use your profile with the limitations of the profile applied I need to do a 2-pass encoding instead (which is VBR), otherwise the violations would remain as you have described earlier. So I was testing "the maximum possible quality within this profile" by first doing the "Twopass - 1st pass" followed directly with "Twopass - 2nd pass alt" with all other settings the same. This is how to do a 2-pass encoding right?

f you choose "Target Bitrate" for the 2nd pass, all it is doing is using that value, multiplied by the movie duration, to get a target size. This is an 'average' rate, not a 'constant' rate.

Yes I know. So first I tried 17500 kbit/s average bitrate just to get something close to maximum limit of my player when the VBV buffer limiting function have done its work. But all I got was this 28 Mbit/s average bitrate in the status window and it looked exactly the same as the first pass encoding (with q=2). Something seems to have gone wrong. I tried 5000 kbit/s with all rest settings the same (I did not remake the first pass this time beqause the same stats file should work). This worked and gave an average bitrate below 5000 kbit/s. Then I slowly increased average bitrate in several 2nd passes and found that just above 13 Mbit/s averge bitrate it did not work anymore. This was what I tried to explain.

You said earlier you were doing CBR encodes to probe your box. This says to me "Single Pass, Target Bitrate", and is a neat idea to probe for the maximum sustained bitrate of your box. However, the CBR mechanism isn't perfect and you might get overshoot across a 'easy/hard' scene transition - a case you seemed to have encountered.

I suggested you redo *that* CBR encode using the *same* exact settings (ie redo the "Single Pass, Target Bitrate" encode) with "XBMC-HD low" selected. This will NOT limit or vbv constrain the CBR encode output, but the Xvid Status display would show the VBV compliance of That encode against That profile.

If that particular profile's VBV parameters are 'about right' for your box, then, when it checks your CBR "Single Pass, Target Bitrate" encode, the bulk of the frames should pass 'ok', and the section with the 'easy/hard' transition which you said caused your box some problems should be RECOGNIZED as a problem area by the VBV check - the status display should show the buffer being consumed and violations occurring during that section, and ONLY that section.

Note that nowhere in the above does "2nd pass alt" come into play. What you are using is the VBV compliance check built into the Xvid Status window to evaluate a CBR encode using a given set of vbv parameters. The CBR encode at 17-18mbps that you said worked OK except for one specific section is providing you with a test case to see if a specific set of VBV parameters RECOGNIZES this problem segment. You don't want a set of parameters that clips non-problem sections, nor do you want a set of parameters that accepts sections you know are problematic.

=============================================

With respect to the "2nd pass alt" problem you indicate you are having, I would need for you to capture the debug output to make any progress on this. Set the debug output mask to '0x80' and use 'debugview' utility to capture the '2nd pass alt' debug info to a file. Post a pointer...

EDIT: Also, are you using the xvidvfw update I referenced earlier (with the revised profiles)? I ask, because the 'low' profile in my original release had vbvmax ~13Mbps vs the 'low' in that update has vbvmax ~17.8Mbps...

I will do some testing and report back. Yes I used the updated xvidvfw and yes it has 17.8 Mbit/s max. The problem is that I can not use above 13 Mbit/s without causing the problems I described.

However, I have noticed that once the VBV buffer has decreased to zero in the stats window it will remain at zero during the rest of the encoding. It does never go up again. However the violations occurs mainly in the problematic section.

ronnylov

10th May 2007, 21:27

Here is the debug view of a 2nd alt pass with average bitrate setting 13528 kbit/s

[880] [xvid rc] -- Frame rate: 25/1 (25.000000fps)
[880] [xvid rc] -- Number of frames: 1999
[880] [xvid rc] -- Target bitrate: 13528000
[880] [xvid rc] -- Target filesize: 135212360
[880] [xvid rc] -- Container Frame overhead: 24
[880] [xvid rc] -- New target filesize after container compensation: 135164384
[880] [xvid rc] -- Min 1st pass IFrame length: 16241
[880] [xvid rc] -- Min 1st pass PFrame length: 1575
[880] [xvid rc] -- Min 1st pass BFrame length: 353
[880] [xvid rc] -- scale_curve_internal first scaler:0.482407
[880] [xvid rc] -- scale_curve_internal second scaler:0.482407
[880] [xvid rc] -- IFrame boost tax ratio:1.00
[880] [xvid rc] -- Assymetric tax ratio:1.00
[880] [xvid rc] initial lengths I=4037638 P=245492425 B=23708542
[880] [xvid rc] invariant portion I=250224 P=5862456 B=363734
[880] [xvid rc] scaled lengths I=2077289 P=121460970 B=11625136
[880] [xvid rc] Underflow detected - Scaling Curve for compliancy.
[880] [xvid rc] Scene 0 (Frames 0-249): VBVfactor 25.958805
[880] [xvid rc] Scene 1 (Frames 250-499): VBVfactor 5.443363
[880] [xvid rc] Scene 2 (Frames 500-548): VBVfactor 1.926652
[880] [xvid rc] Scene 3 (Frames 549-716): VBVfactor 1.086533
[880] [xvid rc] Scene 4 (Frames 717-793): VBVfactor 0.931949
[880] [xvid rc] Scene 5 (Frames 794-845): VBVfactor 1.805798
[880] [xvid rc] Scene 6 (Frames 846-882): VBVfactor 1.553289
[880] [xvid rc] Scene 7 (Frames 883-924): VBVfactor 2.628760
[880] [xvid rc] Scene 8 (Frames 925-998): VBVfactor 0.934547
[880] [xvid rc] Scene 9 (Frames 999-1043): VBVfactor 1.076211
[880] [xvid rc] Scene 10 (Frames 1044-1131): VBVfactor 1.182045
[880] [xvid rc] Scene 11 (Frames 1132-1223): VBVfactor 1.429075
[880] [xvid rc] Scene 12 (Frames 1224-1473): VBVfactor 0.311414
[880] [xvid rc] Scene 13 (Frames 1474-1548): VBVfactor 0.342347
[880] [xvid rc] Scene 14 (Frames 1549-1798): VBVfactor 1.717419
[880] [xvid rc] Scene 15 (Frames 1799-1879): VBVfactor 1.564027
[880] [xvid rc] Scene 16 (Frames 1880-1998): VBVfactor 3.606214
[880] [xvid rc] Moving 47046 kB to avoid buffer underflow, correction factor: 1.91394
[880] [xvid rc] Moving 23464 kB to avoid buffer underflow, correction factor: 2.67679
[880] [xvid rc] Moving 11023 kB to avoid buffer underflow, correction factor: 1.86216
[880] [xvid rc] Moving 8860 kB to avoid buffer underflow, correction factor: 3.79023
[880] [xvid rc] Everything underflowing.
[880] [xvid rc] VBV compliant curve scaling impossible.

EDIT:
And here is XviD Status Window:
http://h1.ripway.com/ronnylov/xvidstatus.JPG

plugh

10th May 2007, 23:20

[880] [xvid rc] -- scale_curve_internal first scaler:0.482407
[880] [xvid rc] -- scale_curve_internal second scaler:0.482407

hmmm. You are requesting the second pass file be less than one half the size of the first pass. Thus,
[880] [xvid rc] initial lengths I=4037638 P=245492425 B=23708542
[880] [xvid rc] invariant portion I=250224 P=5862456 B=363734
[880] [xvid rc] scaled lengths I=2077289 P=121460970 B=11625136
The vbv code examines the scaled down sizes and determines
[880] [xvid rc] Underflow detected - Scaling Curve for compliancy.
Of particular note are
[880] [xvid rc] Scene 12 (Frames 1224-1473): VBVfactor 0.311414
[880] [xvid rc] Scene 13 (Frames 1474-1548): VBVfactor 0.342347
I would guess this is the 'hard' segment you mention - the vbv code wants to shrink those frames to one third of the 'scaled' size (or one sixth of their first pass size)
[880] [xvid rc] Everything underflowing.
[880] [xvid rc] VBV compliant curve scaling impossible.

There is your problem.

When vbv code scales down a segment, it spreads the freed bits across the other "ok" segments (ones with factor greater than one). However, doing so has caused some of THOSE segments to become non-compliant, hence the multiple passes.
[880] [xvid rc] Moving 47046 kB to avoid buffer underflow, correction factor: 1.91394
[880] [xvid rc] Moving 23464 kB to avoid buffer underflow, correction factor: 2.67679
[880] [xvid rc] Moving 11023 kB to avoid buffer underflow, correction factor: 1.86216
[880] [xvid rc] Moving 8860 kB to avoid buffer underflow, correction factor: 3.79023

Your requested target size / average bitrate is at (actually exceeds) the vbv maxrate. If you think of the varying bitrate of a typical encode as mountain ranges, you are attempting to bulldoze the entire mountain range flat, filling the valleys with the 'bits' from the peaks. About as far from a 'constant quant' encode as you can possibly get.

And given this [880] [xvid rc] -- Target bitrate: 13528000I'd guess you are not using the updated dll with the 17.8Mbps vbv maxrate 'low' profile.

Your case has revealed an interesting 'gotcha', howver. The '2nd pass alt' code responds to this situation as follows: status = scale_curve_for_vbv_compliancy(rc, fps);

if (status == 0) {
DPRINTF(XVID_DEBUG_RC, "[xvid rc] VBV compliant curve scaling done.\n");
} else {
DPRINTF(XVID_DEBUG_RC, "[xvid rc] VBV compliant curve scaling impossible.\n");
//del - clean up and bail out if scaling fails
free(rc->keyframe_locations);
free(rc->stats);
free(rc);
return(XVID_ERR_FAIL);
}
It seems that returning an error status does not terminate the encode. Instead, xvidcore just stops using the plugin, but proceeds with the encode. Hence your quant=2 output - no rate-control applied at all! This would require a change to 'core' to fix...

BTW, this also makes the problems you had earlier with the Divx derived 'Highdef' profile clearer. THAT profile has a vbv maxrate of only ~9.7Mbps - really quite a low number.

As to why your clip seems to be demanding such extreme bit-rates? You started off talking about constant quant 3 encodes, the log indicates a factor of 2 to 6 reduction in size from the 1st pass result... SOMETHING is driving your bitrate skyhigh, much higher than I've ever seen for my 720p HD encodes. Are you using a high bitrate custom matrix? Are you trying to do 1080p? You will need to address this if you want to live within your box's 17-18Mbps limit.

plugh

11th May 2007, 00:07

oops - I was so focused on the log, I didn't see your earlier post.

Sorry to confuse you. I am aware of the difference between the CBR mode and 2-pass encoding. Yes I was making CBR encodes earlier but to use your profile with the limitations of the profile applied I need to do a 2-pass encoding instead (which is VBR), otherwise the violations would remain as you have described earlier.

I hope you also understand that the test I proposed was to see if that profile correctly recognizes your 'problem' segment, but leaves the rest of the clip alone. It had nothing to do with producing an output file, it was simply a verification test.

So I was testing "the maximum possible quality within this profile" by first doing the "Twopass - 1st pass" followed directly with "Twopass - 2nd pass alt" with all other settings the same. This is how to do a 2-pass encoding right?

er, uh, in general yes, but...

As an example: I just finished an encode. This was a 1280x528 (2.35:1) 95 minute movie. I did need to apply vbv constraint to this one, as my 'full quality' first pass had a few sections that exceeded my chosen profile constraint (with vbvmax ~21Mbps). I used I/P/B of quant 2/2/3, 1 b-frame, and h263 quantization.

I specified I wanted 2nd pass output same size as first pass. My AVERAGE bitrate for this second pass was "only" 4.44Mbps. My final MAX quants were 3/3/4 (due to the vbv scaled sections - most was still 2/2/3). Would it have increased the quality if I had tried to force the entire movie to be at 21Mbps AVERAGE bitrate (or tried to double the output file size)? Not really. I have done constant quant 1 encodes and compared them to quant 2, and there is very little visible difference. And I question whether I could even _reach_ 21Mbps 'CBR', which is what asking for 21M average with a 21M limit 'says'.

So asking for a 2nd pass AVERAGE that is near your MAX is pointless. Your first pass (assuming you did it as 'Full quality', at the quants you want) defines your quality; it will have some AVERAGE rate, with peak demands. You want to insure those peaks don't exceed the capabilities of your box. Specifying a higher average bitrate than the first pass, or larger file size than the first pass, is just you saying 'I'm not satisfied with the quant 2/2/3 (or whatever) results'.

And besides, the closer your Average is to your Max, the greater the number of sections of the movie that are going to be squeezed by the vbv constraint. (The bulldozing of the mountaintops gets more extreme, while filling all those low bitrate valleys, to reach the average.)

((editorial comment: There is nothing magical about the 2nd pass, there is no A.I. code applying hvs heuristics that increase or maximize quality. It simply applies a number of scaling operations to the frame sizes from the first pass, with a feedback loop doing a running check on desired vs actual size as the 2nd encode proceeds.))

I will do some testing and report back. Yes I used the updated xvidvfw and yes it has 17.8 Mbit/s max. The problem is that I can not use above 13 Mbit/s without causing the problems I described.
Then it seems as though those couple of scenes are SO demanding that they push the rest of the clip out of wack. Possible, but I've never seen this happen before...

However, I have noticed that once the VBV buffer has decreased to zero in the stats window it will remain at zero during the rest of the encoding. It does never go up again. However the violations occurs mainly in the problematic section.
Yes, that field in the display is the "MIN" level for the entire encode. If you want to see the dynamic level, click the "Show me the internals" checkbox. The xx% number is the buffer level.

ronnylov

11th May 2007, 09:02

Thank's again for your detailed answer. Just some short notes here.

First I use only max one consecutive B-frame because in my early testing I had some playback problems when using max 2 b-frames.

In this test I tried a custom matrice that is called Heini_mr. I read somewhere that it was a good one and it seems to require a rather high bitrate (as shown). Probably H.263 is a better choice in this case...

I have used quantizer ratio = 1.62 and quantizer offset = 0.

I have also unticked "packed bitstream".

The source is originating from a DVB transmission of the swedish HDTV channel SVT HD which has a constant bitrate of 16 mbit/s h.264 encoded video 720P at 50 fps. But since every frame was duplicated (the original is probably 25 fps) I have used avisynth SelectEven to reduce it back to 25 fps.

I have not reduced the resolution because my player is supposed to handle XviD up to 720P resolution, The original resolution is 1280x720.

SVT HD is noisier than many other HD channels (I have been told, it is the only HD channel I can receive) and the quality is really good but at the "problem parts" of this video there is still some compression artifacts also on the original.

I think this test clip is some kind of torture clip for me. I have selected this part because it is the one that cause the problems for me.

I am using "xvid.cvs.head.MTK.2007.03.10" and have replaced xvidvfw.dll with your 1.2 version. I also tried a later build but that crached on me when trying this encode but maybe that is what it is supposed to do because of this error? So I had changed back to the build that did not crach but it may have been a mistake. Which build do you recommend? I have dual core CPU and get a great speed increase with the mutithreaded xvid and official 1.1.2 feels too slow because of no multithreading.

I admit I am a little bit confused with everything here and I have not much time to test after the children has fallen asleep and before I get so tired that everything falls apart.
Good I can sit at work and think a little bit in between...

I will go back to test CBR encodings with your HD low profile. One thing I noted was that on the CBR 15000kbit/s encode that played without any problems on my player I got VBV underflows in the encoding (just as you say may happen at CBR encoding). So I beleive I can use a higher VBV buffer setting than HD low profile. But on the other hand I think the maximum sustained bitrate still is somewhere close to 17500 kbit/s because all other "non problematic" sections played perfectly at 17500 kbit/s. Maybe they can sustain even higher but at 20000 kbit/s nothing works so I am thinking of separating the non-problematic scenes to a separate clip, maybe repeating a scene with rather constant frame sizes many times to be sure that the buffer fills up when the maximum sustained bitrate in CBR mode goes too high.

Eventually I will get some results. Then I may be able to apply them using xvid encraw or MeGUI profiles. My goal is then to try get back to my original quest to use this profile of my own to make "constant quality with vbv limits encodings".

So in your previous explanations it seems I should make the first pass using a zone with q="my wanted value", let's say q=3 with the H.263 matrice. And then what to do to apply the limits only where it is needed in the second pass? What average bitrate do I need to set to get the same quality in the parts that not need to be VBV-limited? As you say, not to shave off the mountains to fill the valleys, just shave off as much as needed to make a compliant encoding.

plugh

11th May 2007, 13:59

Thank's again for your detailed answer. Just some short notes here.

In this test I tried a custom matrice that is called Heini_mr. I read somewhere that it was a good one and it seems to require a rather high bitrate (as shown). Probably H.263 is a better choice in this case...

Definitely try h263.

The source is originating from a DVB transmission of the swedish HDTV channel SVT HD which has a constant bitrate of 16 mbit/s h.264 encoded video 720P at 50 fps.

SVT HD is noisier than many other HD channels (I have been told, it is the only HD channel I can receive) and the quality is really good but at the "problem parts" of this video there is still some compression artifacts also on the original.

Probably a factor; you may want to apply deblocking filter. I don't know how you would do this with an h264 source (it's an option/argument to avisynth "mpeg2source", and I use it on overly compressed mpeg2 transport streams).

I am using "xvid.cvs.head.MTK.2007.03.10" and have replaced xvidvfw.dll with your 1.2 version. I also tried a later build but that crached on me when trying this encode but maybe that is what it is supposed to do because of this error? So I had changed back to the build that did not crach but it may have been a mistake. Which build do you recommend? I have dual core CPU and get a great speed increase with the mutithreaded xvid and official 1.1.2 feels too slow because of no multithreading.

I have no recommendation. Use what works for you. You may want to check the initial debug output to verify success, now that you know what you are looking at. You will also see that adjusting vbv parameters changes the vbvfactors for scenes (>1 ok, <1 squeezed). Ideally, you want vbv parameters that give vbvfactor<1 only for the scenes that cause problems on your box.

I'm going to have to dig into that error status thing some more. When I began work on "2nd pass alt" adding this error was one of my first few changes (normal 2nd pass does nothing if vbv fails). Xvid 1.1.0 was the current 'stable release' then and I would swear it terminated the encode. Xvid v1.1.2 also exhibits this funky behaviour, consistant with the code in encoder.c pEnc->plugins[n].func = NULL; /* disable plugins that fail */
if (create->plugins[n].func(NULL, XVID_PLG_CREATE, &pcreate, &pEnc->plugins[n].param) >= 0) {
pEnc->plugins[n].func = create->plugins[n].func;
I'll check the cvs history on that module and see if it has been changed in that area.
I admit I am a little bit confused with everything here and I have not much time to test after the children has fallen asleep and before I get so tired that everything falls apart.
Good I can sit at work and think a little bit in between...

I will go back to test CBR encodings with your HD low profile. One thing I noted was that on the CBR 15000kbit/s encode that played without any problems on my player I got VBV underflows in the encoding (just as you say may happen at CBR encoding). So I beleive I can use a higher VBV buffer setting than HD low profile. But on the other hand I think the maximum sustained bitrate still is somewhere close to 17500 kbit/s because all other "non problematic" sections played perfectly at 17500 kbit/s.

Your two observations are not contradictory. The CBR encode will under- and over-shoot around the chosen rate. If you activate the vdub status video tab, the graph will not be perfectly horizontal.

So in your previous explanations it seems I should make the first pass using a zone with q="my wanted value", let's say q=3 with the H.263 matrice. And then what to do to apply the limits only where it is needed in the second pass? What average bitrate do I need to set to get the same quality in the parts that not need to be VBV-limited? As you say, not to shave off the mountains to fill the valleys, just shave off as much as needed to make a compliant encoding.
As an initial experiment, I would keep it very simple.

deblocking filter, h263 quantization, "low" profile

1st pass - check "Full Quality" uncheck "discard"
(don't bother with quant=3 zone with h263)

Hopefully the gui vbv stats will show violations only in 'hard' section. If so,

2nd pass - specify target size same as 1st pass output,
with all 2nd pass options zero and min I/P/B=2/2/2
(use the built in calculator - for your 2000 frame clip,
target size=1st pass output file size
Container=AVI-Legacy
0 hrs, 1 min, 20 sec, @25fps
Audio - as appropriate)
FYI - in the debug output, scale_curve_internal scaler should be approx 1

plugh

11th May 2007, 17:19

BTW, a handy tool for visualizing the encoder output is the Divx DRF Analyzer (http://www.geocities.com/analyzerdrf/), unfortunately it seems the link to it there is broken. However, you can still get it here (http://www.free-codecs.com/DivX_DRF_Analyzer_download.htm)

You might find comparing graphs of your 1st vs 2nd pass interesting, like I posted here (http://forum.doom9.org/showthread.php?t=118021).

ronnylov

11th May 2007, 21:50

I have identified the hard section of my torture clip.
Frames 1224-1548 are thousands of flying birds (swans) taking off from water, first a zoomed view and then next scene unzoomed to white dots moving around the picture...
I realise this would be hard to encode. It was very obvious when looking at the graphs of DRF Analyzer.

CBR 15000 kbit/s
http://h1.ripway.com/ronnylov/DRFAnalyzeCBR1500.png

And here is a screenshot of source frame number 1302:
http://h1.ripway.com/ronnylov/FlyingBirdsFrame1302.png

Here is another screenshot of source frame 1548:
http://h1.ripway.com/ronnylov/FlyingBirdsFrame1548.png

Imagine all these birds flying...

plugh

11th May 2007, 22:15

WOW!

Yeah, those are the range of frames the vbv code identified in the debug log you posted above.

You can see the overshoot and gradual ramp down as CBR algo reacts (does a pretty good job actually) and then the undershoot and ramp up at the end of the section.

Is that with the high bitrate matrix, or h263?

It will be very interesting to see graphs of 1st and 2nd passes (with h263).

plugh

11th May 2007, 22:54

A few DRF Analyzer tips:

1) I am very suspicious of its analysis of packed B-frame streams. In spite of what the author says on the site, I'm pretty sure it has bugs trying to parse the packed frame.

2) If you happen to be running DebugView, disable capture while DRF Analyzer is making its pass - it spits a boatload of stuff out the debug console.

3) Click/release the frame slider (but don't move it). Now use the keyboard arrow keys to move in fixed size jumps. Makes it much easier to position two graphs the same...

ronnylov

12th May 2007, 21:25

ronnylov

12th May 2007, 23:06

1st pass with h.263 (and VHQ Mode 4 instead of Mode 1):
http://h1.ripway.com/ronnylov/XBHD-lowH2631stPassDRFAnalyze.png

And then 2nd Alt pass with average bitrate set to 12500:
http://h1.ripway.com/ronnylov/XBHD-lowH2632ndAltPass12500DRFAnalyze.PNG

Note the VBV violations in this 2nd alt pass in XviD Status:
http://h1.ripway.com/ronnylov/XBHD-lowH2632ndAltPass12500XvidStatus.PNG

This time the debug log was huge so I have compressed it to a rar file:
http://h1.ripway.com/ronnylov/XBHD-lowH2632ndAltPass12500Debug.rar

EDIT: Despite of the reported VBV underflows the file is still playable in my standalone player without visible problems. Both the previous shown CBR 15000 kbit/s encoding and this 2nd alt pass encoding 12500 kbit/s were playable. The profile is usable in my player but it may be a little bit too aggresive.

plugh

13th May 2007, 03:42

I wish I had a major chunk of that capture to play with. That is a very challenging piece!

Some observations, in no particular order...

You need to do a longer clip. As I said earlier, the vbv code spreads the shaved bits across the rest of the clip. This section represents a large portion of the total size of the 2000 frames, and the rest of the short clip has trouble 'absorbing' that much. It is driving a LOT of frames (including B's!? ) to quant 1.

There is something else odd though; for exampleframe:000248 type:p quant:01 stats:005895 scaled:019601 desired:019576 target:018597 actual:029166 error:+009590 vbvfill:524288 overflow:-1798921
stats:005895 is size of q=2 frame in first pass
actual:029166 is size of q=1 2nd pass frame

I've never seen a frame increase in size by a factor of FIVE going from q=2 to q=1, but you have many many cases of this. This is throwing the scaling logic off, and is also causing the later violations. (The log is interesting as it shows my secondary checks trying to fight this - with only marginal success :( I may need to be more aggressive there.)

It seems to be only the q=1 case that is 'off', as for exampleframe:000506 type:p quant:02 stats:053429 actual:051569
frame:000507 type:b quant:03 stats:003431 actual:004558shows 1st and 2nd pass q=2/3 are staying close. (Differences in preceding frames cause the small difference)

Hmmm. Set your min quants to 2...

Wow. Half a megabyte per P-frame. Lets see, that works out to 500K*8*25 = 100Mbps! Interesting that xvid didn't fall back to all I-frames... Also surprised it didn't put one at, hmm looks like frame 1293, where frame size took the big jump.

Speaking of which, you might want to force one there - create a zone starting at 1293 with the 'keyframe' box checked. This will partition that portion better for the vbv code... (perhaps some other zone based tweaks in that area later as well)

It's not shown in your post, but if I back-calculate it looks like the AVERAGE bitrate for the first pass was about 22.8Mbps. That section is really skewing things with the short clip. When I scan the "stats" numbers in that log from the first pass (ignoring that section), my eyeball average tells me that your REAL average is much less.

Yeah, set min quants to 2, encode say 6000 frames, and maybe force a keyframe at 1293. You might want to try 2nd passes with both the "low" and "mid" profiles, just for fun... You'll probably be able to get the 2nd pass target bitrate a lot closer to the new 1st pass average with the longer clip.

plugh

13th May 2007, 07:13

Man, some very weird stuff with those q=1 B-frames.
frame:000473 type:b quant:01 stats:002145 scaled:006764 desired:006755 target:006417 actual:087708
frame:000475 type:b quant:01 stats:001701 scaled:005460 desired:005453 target:005180 actual:068173

q=3 size 2145, q=1 size 87708 => 41 times larger
q=3 size 1701, q=1 size 68173 => 40 times larger

Actually, contrary to my earlier statement, I think my secondary vbv logic did a pretty fair job, considering what was going on.

Moral of the story: never ever ever allow q=1 B-frames!

Hey ronnylov - did you zero out all the entries on the 2nd pass options page? I'm seeing evidence of the code attempting to boost the size of I-frames by 10 percent.

EDIT: more zone tweaks... create zones at
1293, keyframe checked, b-vop sensitivity -35
1463, keyframe checked, b-vop sensitivity +25
1549, keyframe checked, b-vop sensitivity 0

re: the 1463 zone - (right after the zoom-out?)
might want to try both +25 and -35 and see which gives better visual results.
The mix of P&B (+25) reduces bitrate and thus less squeezing, but all P's (-35) means no q=21/22 frames.

ronnylov

13th May 2007, 19:46

You are right, I have forgotten the part of zeroing out 2nd pass options. I thought default values would be acceptable to use. Seems that there are a lot more things I can test...

I will send you a PM regarding "a large chunk of that capture to play with".

plugh

13th May 2007, 20:37

Well, you certainly don't need any 'boosting' going on at this point ;) and I wanted the overflow stuff turned off to avoid 'muddying the waters'.

I answered your PM. Looking forward to your next set of results, with the above adjustments...

ronnylov

14th May 2007, 08:27

I need to set the zones in both 1st and second pass, right?
A higher B-VOP sensitivity means more B-frames will be used?
What does a negative B-VOP sensitivity do? Less B-VOPs than default setting?

I have not played much with all the xvid options so it is interesting to learn what they really do on a real test clip like this.

plugh

14th May 2007, 12:11

Actually, you only have to set those for the 1st pass (the 2nd pass picks up the frame type from the 1st pass stats file) but leaving them defined for both passes is fine.

Side note: Another reason I prefer 'full quality' 1st pass (and 'turbo' off) - the I/P/B frame-type decision making seems 'better'.

Yes, negative value decreases B's, positive increases them (subject to 'max consecutive' limit).

The b-vop tweaks were just for consistency;
the 1293 section was _almost_ entirely free of B's
the 1463 section was _almost_ fully P/B alternate

My main reason for the zones were placement of the key frames (will become clearer when you see the graphs). The b-vop tweaks just came along for the ride.

ronnylov

18th May 2007, 22:58

I have changed to a longer part of the source file, 5995 frames. The decoding method have changed also from Cyberlink decoder via directshowsource in avisynth to DGAVC. I have got a slight offset in the frames compared to the previous shorter source file but the difficult part is still included.

I created 4 zones in this new file to get new key frames.
Zone 1: Start #1235, key frame, chroma optimizer, BVOPs = -35
Zone 2: Start #1304, key frame, chroma optimizer, BVOPs = -35
Zone 3: Start #1485, key frame, chroma optimizer, BVOPs = 25
Zone 4: Start #1560, key frame, chroma optimizer, BVOPs = 0

(Thinking of it maybe I should have set a zone that starts at frame 0 to use chroma optimizer in the whole clip, I forgot that)

I could not see any difference other than the added key frames, I think the B-frames was the same as before.

I have also changed minimum quantizer to 2 for all frame types and I have nulled all the 2nd pass options.

The good news is that I do not get any VBV violations anymore! I think the zeroing of 2nd pass options sid the trick!

I have also done a lot of trial and error. I tried to set maximum 2nd alt pass average bitrate without causing "everything underflows" error and the result was an input value of 14377 kbit/s, If I set 14378 kbit/s then I get the error... The real bitrate as reported by XviD Status window was around 10150 kbit/s.

Then I tried another 2nd alt pass with average bitrate set to 10200 kbit/s to see what happens but then the XviD status reported 8305 kbit/s. Then I tried a lot of different settings of the bitrate and I came to the conclusion that everything above 11600 kbit/s creates almost identical output files (average bitrate around 10150 kbit/s) and DRFAnalyzer Reports same average quantizer and almost the same statistics on everything else. So my conclusion is that this is the maximum limit this profile can produce.

Just to be sure I made a SSIM Analyze of both 11600 kbit/s nominal bitrate output and the 14377 kbit/s nominal bitrate output. Both files with audio was 305 MB in file size. The 11600 kbit/s had average SSIM = 76.658 and the 14377 kbit/s nominal bitrate gave SSIM = 76.642. I think the difference is too small to be noticed.

So this is interesting I think. To set a maximum quality without wasting too much I should set the mimimum quantizer values to the desired q-value and then set the bitrate as high as possible without creating the "everything underflows" error. Of course I have first selected a profile that suits my hardware like "XBMC HD low" profile.

Previously you told me that I can make a first pass with the desired quantizer set in a zone. So if this full quality first pass does not cause any VBV violations I should use the file output from the first pass. To fix the violations I just make a 2nd alt pass as desribed above and set minimum quantizers to the same as the first pass quantizer and bitrate to highest possible without getting any error.

However it was a little bit tricky to find the bitrate limit where the "everything underflows" occur. Can I use the debug output to find a good value of the bitrate setting if I have set a too high bitrate?

plugh

21st May 2007, 03:27

WARNING: Long Technical post

There is a lot to respond to in your post. I've thought about it off and on the past few days. Here goes...

I have also done a lot of trial and error. I tried to set maximum 2nd alt pass average bitrate without causing "everything underflows" error and the result was an input value of 14377 kbit/s, If I set 14378 kbit/s then I get the error... The real bitrate as reported by XviD Status window was around 10150 kbit/s.

You are paying to much attention to the 2nd pass target bitrate. That config setting is A CONVENIENT FICTION. Here is the relevant code
/* Target filesize = bitrate/8 * numframes / framerate */
rc->target =
((uint64_t)rc->param.bitrate * (uint64_t)rc->num_frames * \
(uint64_t)create->fincr) / \
((uint64_t)create->fbase * 8);

When you input a target bitrate, all it does is convert it to a target filesize. Keep that in mind in the following...

Then I tried another 2nd alt pass with average bitrate set to 10200 kbit/s to see what happens but then the XviD status reported 8305 kbit/s. Then I tried a lot of different settings of the bitrate and I came to the conclusion that everything above 11600 kbit/s creates almost identical output files (average bitrate around 10150 kbit/s) and DRFAnalyzer Reports same average quantizer and almost the same statistics on everything else. So my conclusion is that this is the maximum limit this profile can produce.

The profile isn't your major limiting factor. The quants are.

The good news is that I do not get any VBV violations anymore! I think the zeroing of 2nd pass options sid the trick!

No, it was setting the min quants that did it. Zeroing the 2nd pass opts just eliminates all other scaling operations (will be clearer below) {Note: this only applies to '2nd pass alt' code; zeros give defaults with normal 2nd pass}.

So this is interesting I think. To set a maximum quality without wasting too much I should set the mimimum quantizer values to the desired q-value and then set the bitrate as high as possible without creating the "everything underflows" error. Of course I have first selected a profile that suits my hardware like "XBMC HD low" profile.

Previously you told me that I can make a first pass with the desired quantizer set in a zone. So if this full quality first pass does not cause any VBV violations I should use the file output from the first pass. To fix the violations I just make a 2nd alt pass as desribed above and set minimum quantizers to the same as the first pass quantizer and bitrate to highest possible without getting any error.

However it was a little bit tricky to find the bitrate limit where the "everything underflows" occur. Can I use the debug output to find a good value of the bitrate setting if I have set a too high bitrate?

Ok. Let's walk through this. I am going to assume your target quants are I/P/B=2/2/3; you specified 'Unrestricted' profile (ie no vbv constraints).

You do a full quality first pass. The I/P/B frames were encoded at those specified quants. This yeilded a file of some size (call it 'SIZE1').

Thought experiment #1:

You decide to do a second pass anyway. You zero out the 2nd pass options, and set min quants to 2/2/3. You set 2nd pass target size to SIZE1. What will you get?

You'll get the SAME RESULT as your first pass.

Thought experiment #2:

Same as #1, except you set 2nd pass target size to 2*SIZE1. What will you get?

You'll get the same thing as your first pass, AGAIN.

HUH! you say? Think about it... The 2nd pass reads in the 1st pass stats file, looks at your target size relative to the first pass size, and scales all the 1st pass frames sizes - by a factor of two in this case. The 2nd pass tries to create frames the size you requested by decreasing the quant for them (from q=2 to q=1 in this case). But the min quants limit it to the same quant as the first pass, hence YOU GET EXACTLY THE SAME RESULT. This is your classic 'undersized file' situation - it simply can't make the frames any bigger.

So... If you do a 'full quality' first pass at the quant you want, and it is vbv compliant there is nothing to be gained by doing the second pass. You HAVE exactly what you said you wanted - an encode at specified quants that is vbv compliant. Redoing it won't change anything.

Thought experiment #3:

A bit more complicated - Same as above but with a vbv constraint. The 1st pass stream was evaluated against your chosen profile by xvidvfw and the vbv compliance was shown on the status window. 1st pass WASN'T vbv compliant.

So, you zero out the 2nd pass options, and set min quants to 2/2/3. You set 2nd pass target size to SIZE1. What will you get?

You will end up with a file _smaller_ than SIZE1, with (hopefully) only those sections that were subject to the vbv constraint encoded at higher quants. All other frames will be at the same quant (and basically the same size) as they were from the first pass.

A more subtle 'HUH?' - Why is the file smaller? The VBV code takes bits away from the noncompliant sections and spreads them over the compliant ones, right?

Yes, the code scaled down (increased quant for) the bad frames and scaled up (decreased quant for) the 'ok' ones. But the 'ok' ones are already at your desired quant, and you min quants prevent them from going to q=1 and getting any bigger. Hence try as it might, it can't use those extra, freed-up bits. The bits shaved off the non-compliant sections end up not being used anywhere and hence the file gets smaller.

Thought experiment #4:

Same as #3, except you set 2nd pass target size to 2*SIZE1. What will you get?

If you are lucky, you will get the 'everything underflows' error message.
If you're not lucky...

An even more subtle 'HUH? but, sputter, ...'

Let's put it all together.

Your second pass target is twice the first pass size. Therefore the code scales all the first pass frames sizes up by a factor of two. Then the VBV code examines these scaled up frames. What do you think that scaling is going to do to your VBV compliance?

The sections that weren't compliant originally are now twice as big. Sections that were compliant originally may or may not still be. The VBV code starts scaling all these large non-compliant section back down, freeing up massive quantity of bits, which it spreads over the remaining compliant sections. As these are already at twice their 1st pass size, this probably drives more sections into non-compliance, and hence vbv code makes another pass trying to spread THOSE bits somewhere. As I said, if you are lucky, it'll error out, if not...

So now it starts encoding. It's handed all these large frames sizes. It compares these sizes to the first pass size and computes a quant for each frame - probably q=1 - however the min's prevent that so you get q=2 (back where you started). But wait a minute, what about the vbv stuff?

The sections that originally were non-compliant, that were scaled up due to the target size, then scaled down due to vbv constraint, well you've really jerked those numbers around. And segments that wouldn't have been touched at all if they hadn't been 'targetsize upscaled' or had to absorb all the bits from the other sections, they've been jerked around to. So what do you end up with? Beats me - I try to avoid this situation ;)

OK. Having gone through all this, I'll re-quote you
So this is interesting I think. To set a maximum quality without wasting too much I should set the mimimum quantizer values to the desired q-value and then set the bitrate as high as possible without creating the "everything underflows" error. Of course I have first selected a profile that suits my hardware like "XBMC HD low" profile.

Previously you told me that I can make a first pass with the desired quantizer set in a zone. So if this full quality first pass does not cause any VBV violations I should use the file output from the first pass. To fix the violations I just make a 2nd alt pass as desribed above and set minimum quantizers to the same as the first pass quantizer and bitrate to highest possible without getting any error.

However it was a little bit tricky to find the bitrate limit where the "everything underflows" occur. Can I use the debug output to find a good value of the bitrate setting if I have set a too high bitrate?

I'd have to check, but I believe I've said at least three times in this thread 'set second pass target size same as the first pass output size'. When you were playing with CBR encodes to probe your box to try to establish approximate values for vbv parameters, then 'highest possible bitrate' was relevant. But 'target bitrate' has no relevance for applying vbv constraint to your desired constant quant encode. As shown above, (1) bitrate is just an expression of the filesize and (2) trying to force things to some nebulous 'maximum quality' can be problematic, particularly in the context of vbv constraints.

The reason I say 'set 2nd pass target same as 1st pass output' is because it is the most accurate estimate you have available, particularly if 1st pass was 'full quality' to start with. By using the same size the 2nd pass code will apply a scale factor of 1.0 to the first pass frame sizes, and this in conjunction with the min quants will insure you get the target quants you specified in your first pass (except for the non-compliant sections, which will be scaled 'just enough' to become compliant).

Hopefully the problems you had with your clip are more understandable now. (1) You needed a longer clip, so the 'spread out' bits don't drive other parts into non-compliance. (2) You needed min quants, so the extra bits don't cause q=1 frames (which caused secondary vbv problems, and which you didn't want anyway since that wasn't your specifed target quant). And (3) you need to specify 2nd pass target same as 1st pass output, so there isn't any scaling of the 1st pass sizes - the vbv code gets exactly the frame sizes produced by your constant quant 1st pass, neither larger nor smaller.

BTW, the zones thing - The Xvid VBV code operates on 'scenes'. These scenes are (basically) defined by keyframes. In looking at the graphs you posted, it was apparent there were changes in that segment that xvid did not delimit with a keyframe. There was also a keyframe inserted in the middle of a segment due to the 'max I-frame interval' logic. Thus the vbv code was grouping frames and scaling them in a suboptimal fashion. (In particular the 1248-1293 section was getting lumped in with 1293-1463, driving the earlier frames to much higher quants than they needed to be.) I've run into this before and it's usually not an issue, but that scene was so extreme it warranted special attention to try to preserve some amount of picture quality (I shudder thinking about those q=22 b-frames). Anyway, it wasn't related to the 'bigger (vbv) picture', just my attempt at some localised tweaking... (BTW, I personally don't use 'chroma optimizer' for my HD encodes, as the resolution doesn't warrant it IMHO).

Wow :eek: enough for now, I think

ronnylov

21st May 2007, 10:44

Thank's plugh!

You are a good teacher, I'm learning more and more how it works. I understand that it is the mimimum quantizers I set that limits the quality (in combination with the quantizer matrices). Then the hardware of my player is limiting playback and the VBV limitation is used to apply some limits that my hardware can handle. I am trying to combine "my quality limits" with the "hardware limits" to get an acceptable end result.

When I set minimum quantizers to 2 (on the 2000 frame clip at least) at this resolution will cause a total size that is too high to use as input in the second pass where I got the "everything underflows" error. If I understand you correctly this was because of my short test clip with a major difficult part and not many easy scenes. Better to test with a longer clip to be able to shuffle around bits from the difficult parts to the easier parts.

Now I have tested with "Jahwor 1CD" low bitrate matrice and the 5995 frames test clip. This matrice seems to be more optimised for low bitrates than h.263. I used minimum q=3 on both passes and this time I used the total size as indicated in Xvid status as file size input in 2nd alt pass and it worked! When looking at the results in DRF Analyzer it is only the "many birds at far distance" part of the clip that gets red with q around 13 for the P-frames. Everything else is mainly q=3 P-frames and q=4 B-frames except for some parts with maximum q=5 (these parts had some VBV violations in the first pass)

When playing the encoded video I find it watchable, at least if I not sit too close to the display. It is at about what I should call "a minimum quality that I can live with" and to reduce the file size more then I would downsize the resolution or use some denoising filters because the source is rather noisy.

I tried applying some filters to the trouble section but it did not help much with the quantizers. But when looking at the actual encoded clip at normal playback speed it does not look very terrible in the difficult scene with the flying birds. It is a so short and confusing scene for the eye that I hardly even think about any quality degradation in that part unless I am looking at still pictures. Adding the key frames in the zones did probably help the quality too.

I am thinking of my options here. I can downsize the resolution of the clip the make these "extreme" parts looking better or I can live with it and make most of everything else look good. But if my goal is a constant quality then all parts should look the same and I can reduce the resolution to something like normal DVD resolution or maybe lower to make the toughest parts look the same as the best parts. But my source does not look very good in the toughest parts so it might not be worth it.

Maybe it is time to try the whole source clip and see if there are any more sections that are difficult to encode. If this is the only difficult section I will just ignore it and focus on making the rest of it "close to constant acceptable quality".

Actually now I think I know in general how to use a hardware profile with the 2nd alt pass. I have been focusing on this lately and not so much on testing the absolute limits of my player. If I don't understand how to apply the limits there is no reason to find out the actual limits...

I still have a lot to learn what all xvid options is used for. I have no idea what difference "chroma optimizer" actually do other than I thought colours would be better encoded when enabled.

I have not made any testing with your other XBMC HD profiles yet. I think I will get playback problems when the bitrate limit is higher than around 18000 kbit/s but I will try it and see what happens.

plugh

21st May 2007, 13:22

I used minimum q=3 on both passes and this time I used the total size as indicated in Xvid status as file size input in 2nd alt pass and it worked! When looking at the results in DRF Analyzer it is only the "many birds at far distance" part of the clip that gets red with q around 13 for the P-frames. Everything else is mainly q=3 P-frames and q=4 B-frames except for some parts with maximum q=5 (these parts had some VBV violations in the first pass)

You've got it! From here you can start exploring options and alternatives and vbv params to get the best result still playable by your hardware.

A couple notes:

1st pass doesn't have 'minimum q' - the q you set via zone (or default of 2 if not specified) is the only 'q' used.

2nd pass target size - In the debug output, near the very beginning, you'll see "first scaler"; as a check, this should be approx 1.0 (0.95 - 1.05 is 'ok')

Chroma optimizer - Color info is carried at one fourth resolution of luminance. At lower res, this can produce a visible stair-step effect on edges. This filter smears / dithers color to reduce the effect. At HD res the effect is so small as to not be visible, so I don't enable the filter.

2nd pass options - I-vop boost and curve compression are other scaling tools to play with eventually - many people do. The Overflow stuff is of interest mainly if (1) you are making 2nd pass *smaller* than first pass AND (2) you are aiming for an exact final size. (Using it to enlarge a file is problematic, particularly at low quants.)

ronnylov

13th June 2007, 23:18

It works very well now using your XBMC low profile with my TviX 4000 player when compressing 720P 25 fps to XviD.

However I noticed slight problems when converting 50 fps 720P to XviD at 704x576 resolution and 50fps because I got some dropped frames. I think it may be caused by higher load when outputting double framerate. I thought decreasing the resolution would compensate for that.

I was thinking that some tool that analyze an already encoded file regarding VBV limitation could be useful. Let's say I have a file that I know works without problems then I could analyze it to see what VBV parameters I could use to make this file pass a VBV check. Or let's say I have a file that I would check against a certain hardware profile to know if my player will play it.

Another thing I noticed that there may only be a few seconds of a complete video file that needs to be adjusted to pass the VBV check. A tool that only reencode that parts and nothing else in the second pass could be useful to speed up things. Well, if I learn programming maybe I could make such tools sometime in the future...