lower bitrate than specified in QuEnc

[cfriedl]

I've been encoding some video captures and using QuEnc (versions 0.51 to 0.54). I've also just started using bitrate viewer to check average bitrates in the encoded mpeg2 files. I notice that when I specify bitrates of 5000-6000 kbps (with max set at 8000) I don't get average bitrate higher than about 3500 (with peak about 7500). If I raise the max bitrate to 9000 the average reported by bitrate viewer goes up a little (3600 kbps).

I noticed similar results with other libavcodec based encoders (NuEnc, FreeEnc).

Of course I don't understand these encoders much so perhaps there is a fundamental limitation I'm missing. Is there a setting somewhere that sets ths behaviour or is it more complicated than that? Is there any way to get a higher bitrate with QuEnc?

[dragongodz]

for 1 pass this is easily true but 2 pass should be more accurate.

also the content of the footage needs to considered. such as a plain picture with no motion compresses much smaller than high motion and lots of colour, edges etc etc etc.

can you give more detail about the footage you are encoding from, dimensions, type, length etc ?

[cfriedl]

OK some more detail.

Source is typically free-to-air movies recorded on VHS tape. Then captured at 720x576, reducedby2 (avisynth function), some smoothing with peachsmoother, and a little cropping. The resultant 352x288 video is encoded with Huffyuv so I can encode to mpeg as often as I like to try different things. Length is typically 1.5 to 2 hours.

Results of bitrate viewer for three are:

Peak Average
1. kbps 7621 3628
Q level 2.68 1.0
2. 9922 2988
1.0 1.0
3. 7715 3401
1.56 1.0

Quenc settings: VBR, High Quality, Trellis Quant, 10 bit DC, Max bitrate 8500, average bitrate 5000, 6000, 7000 respectively for the three samples above.

Also the results above are single pass. However I did run two pass in two of the cases and found no significant difference (less than 10% increase in average). Then I blew the files away and didn't record the results (thinking I would solve the problem ... doh!)

In addition I ran a 1 minute segment of case 3 in both single and two pass. Single pass peak was higher than max bitrate and average was about 25% low. Two pass peak was about the same as the peak and about 5% low on average, so much better. Yet this result did not scale up with increasing video length as shown above.

Oddly enough I tried with bbmpeg and got average and peak much closer to target (its single pass I think):

Peak Average
1. kbps 7773 4887
Q level 2.88 1.48
2. 8103 4873
1.0 1.0
3. 8931 6425
2.00 1.24

Unfortunately bbmpeg output seems much softer than Quenc and processing is somewhat slower so I'd like to use Quenc if I can.

Anyway, any comments appreciated.

[dragongodz]

with peachsmoother, and a little cropping. The resultant 352x288 video

so you smooth it(making it more compressable) and then make it vcd size(making it more compressable) and wonder why dvd type bitrates are hard to reach ??????

as has been said many times before, avcodec's rate control is not real great. thats why Peter Cheat has been working on it.

first dont use peachsmoother and you will retain more detail and noise which will help boost bitrate.
you can also try adding some noise with an avisynth filter(theres a few different ones, check the avisynth section) and that should help boost the bitrate aswell if still needed.

[cfriedl]

clearly some things I don't understand about mpeg compression.

I think I understand conceptually how smoothing and small size can reduce bitrate (if what you mean by bitrate is how many bits a sequence of frames is represented by). I guess like making a still image mostly just a few colors so that compression makes it very small due to lots of repeated information.

However what I don't understand is why, if I specify a certain number of bits, the frames are not represented by that many bits regardless of how uniform the image is. Presumably the encoders are saying that you don't get more bits than they think you need, regardless of how many you want. Right?

I have been loosely following Peter Cheats work as my reference to Nuenc suggests. Still lots for me to learn there. However I don't understand what rate control is about. Any simple explanation that can be offered?

Any comment of the bbmpeg results?

Thanks also for the pointers on noise. I guess I always figured noise was a bad thing that I should get rid of before mpeg encoding.

As for the VCD size, I find that encoding at full DVD resolution or even half, just adds lots of processing time for no perceptible gain in visual quality.

btw - thanks for your patience.

[Peter Cheat]

The reason you aren't getting the target size is because at such a low resolution, so many bits are not required and the encoder cannot allocate anymore. Don't use peachsmoother as this is removing detail, and increasing your processing time!. Increase the resolution - there is a quality difference. You get more detail with higher resolution (obviously). For faster encodes use these settings:
Resolution: 720x576 (or maybe 704x576 for faster processing)
Quenc settings: 2-pass, VBR, 10 bit DC, Max bitrate 8500

On my PC, with these settings I get near real-time encoding (~24fps).

Rate control in this case is not the problem - it just cant allocate any more bits because its encoding at the highest quality it can already (well it could, try CBR, this will add useless padding that just wastes space to achieve the bitrate you want...but it is wasting space, not using it efficiently)

[dragongodz]

I guess like making a still image mostly just a few colors so that compression makes it very small due to lots of repeated information.

correct.

However what I don't understand is why, if I specify a certain number of bits, the frames are not represented by that many bits regardless of how uniform the image is. Presumably the encoders are saying that you don't get more bits than they think you need, regardless of how many you want. Right?

yes and no.
say you have a pictures/frame that is totally 1 colourthe encoder will look at that frame and say "this frame is this colour". if that statement is too big for the alotted bits for that frame it will try to say it smaller such as "this colour". however if the statement is not big enough to fit the bits alotted there is a way to get what you want, its called padding. so you would get "this frame is this colour ", as you can see no actuall extra information just filler.

ok that was GREATLY oversimplified(so please nobody complain) but it should give you a very genearl idea.
now i havent tested avcodec's padding but since the rest of the rate control is not good i assume that isnt either. something else for Peter to look at later maybe.

Any comment of the bbmpeg results?

so bbmpeg may be padding out the frames better. or it could also be that bbmpeg is not picking up redundant information aswell. havent tested so dont know.

However I don't understand what rate control is about. Any simple explanation that can be offered?

umm the simplest way i can explain it is, rate control is the control of bits given to frames and/or sequence of frames constrained by limits set such as max, min and average bitrate and any other limitations(such as VBV). hows that ?

I guess I always figured noise was a bad thing that I should get rid of before mpeg encoding.

again yes and no. some people dont like seeing lots of noise, thats fine. however with avcodecs rate control problems ,as you have seen, this can have an adverse effect on encoding size.

[dragongodz]

Rate control in this case is not the problem - it just cant allocate any more bits because its encoding at the highest quality it can already (well it could, try CBR, this will add useless padding that just wastes space to achieve the bitrate you want...but it is wasting space, not using it efficiently)

the rate control it appears isnt adding the padding in this case as it should for VBR aswell. so it actually is still a rate control problem. if you set an average bitrate target then that is what the encoder should be trying to give you.

[Peter Cheat]

Padding is NOT to be added to a VBR stream except in ONE (and only one) case:

Where a MINIMUM bitrate is specified.

Padding removes the meaning of VBR, if a stream is padded to the average bitrate, it suddenly becomes CBR (think about it).

Its not a rate control issue...

EDIT: Changing the motion estimation to ZERO will also increase bitrate and speed up encoding (cant be changed in NuEnc or QuEnc but can be in FreeEnc).

[dragongodz]

Padding removes the meaning of VBR, if a stream is padded to the average bitrate, it suddenly becomes CBR (think about it).

sorry that is bullshit. a stream can still be VBR AND have padding to meet the targeted average bitrate of the whole stream. if the encoder does not try and meet the settings then a setting an average bitrate is pointless.

[Peter Cheat]

Read MPEG-1 and MPEG-2 specs reffering to padding

You can set a MINIMUM bitrate (say 3000kbit/s) but it wont get the bitrate you want. This will also reduce the variability. The only way to GUARANTEE the bitrate you want in this case is to set MINIMUM=MAXIMUM to pad ALL the frames - ie CBR. Do you want a mathematical proof dragongodz?

Also, NuEnc cannot encode to be more than ~2x bigger than its first pass in two pass. Maybe I should make it display an error if you try.

[dragongodz]

NuEnc cannot encode to be more than ~2x bigger than its first pass in two pass. Maybe I should make it display an error if you try.

or maybe you could pad lower size frames where it doesnt break VBV etc. to atleast raise the average closer to the target rate. no forget it, that would just be silly wouldnt it.

[Peter Cheat]

Setting a minimum bitrate basically does this. It is a little more complicated so I won't go into detail, but setting a minimum will boost the bitrate of all the frames below the minumum bitrate, increasing the average. How much depends on the number of frames that were padded.

FYI, both TMPGEnc and CCE behave the same way as libavcodec.

Source: 352x288, 25fps PAL
Target bitrate: 5000kbit/s
Maximum bitrate: 9000kbit/s

CCE came out with an average of ~4000kbit/s
TMPGEnc came out with ~3,500kbit/s

Neither of them padded the stream and came out with undersized files.

[dragongodz]

but setting a minimum will boost the bitrate of all the frames below the minumum bitrate, increasing the average. How much depends on the number of frames that were padded.

yes i know. what i was after the first pass if you see the average bitrate is not achievable effectivly raise min bitrate slightly until the extra bits for the average is attained, or at least got closer to the average.

FYI, both TMPGEnc and CCE behave the same way as libavcodec.

i did a quick test with Tmpgenc and MainConcept aswell. yes they didnt get to the average but they were easily closer.
wouldnt adding something like the above then make your rate control better than theirs.

come to think of it i think Tmpgenc only pads when outputting program streams. havent really used it in a long time so going from memory.

[cfriedl]

dg/pc

thanks for the education. You've given me quite a number of things to try. This is what I love about the web

[cfriedl]

From what I've gleened from your responses I can push the average bitrate of my encodes up but in the end, for my material as it currently stands, this is just padding ... fat, taking up space and doing nothing. I'll get better results in ways you've suggested.

So then, what is the purpose of padding, under any circumstances at all

[dragongodz]

From what I've gleened from your responses I can push the average bitrate of my encodes up but in the end, for my material as it currently stands, this is just padding ... fat, taking up space and doing nothing. I'll get better results in ways you've suggested.

basically yes.

So then, what is the purpose of padding, under any circumstances at all

generally when you want a specific sized file and can not gain it. if you dont mind the file being smaller then it isnt needed.
i can point to the DVD-RB section and point out several threads where people complain the output size is slightly smaller(doesnt quite fill a dvd). so to some it does matter.

[Peter Cheat]

Padding is necessary for some hardware players. Some hardware players can only spin a disc at a constant rate and require a perfectly constant data rate. If frames turn out to be too small to keep up the data rate, padding is added (blank data). Padding is also nescessary when a disc can be spun at a minimum speed (some older DVD players). Apart from that, padding has no use and isn't used.

@dragongodz
The other codecs may have been closer, but that may be due to libavcodec's superior motion compensation. You never add padding unless it is necessary for compatibility. If you were to add padding, it would be an impossible task to determine how much is necessary to get to a required bitrate and the probability of overshooting will be the same as the probability of undershooting. Its best to just leave it alone.