Bitrate allocation modeling

[mikenadia]

Starting a new thread for users that are interested in finding a different model (than optimal Q) for bitrate allocation.
optimal bitrate allocation
http://www.research.ibm.com/journal/rd/434/westerink.html
granularity tests
http://www.research.ibm.com/journal/rd/434/mohsenian.html
Based on the papers above-mentionned, I made a different bitrate allocation model with some assumptions ( probably invalid ones ) but I like the ouput (less need for redist_bias, may be ability to mix prog and inter srgments : Would like to know why it was not recommended to do so in the optimal Q model...).
Will post my oversimplistic, minimalistic model once the discussion has started ,not to influence anybody.
I want to start a discussion, users to look for articles linked to the subject, make their own model (with underlying assumptions,encoder name, description of advantages and drawbacks...), suggest a method of comparaison ...
All models have to be able to be replicated on a simple spreadsheet based on output by either Rb-opt or DVD-RB.

[Sharc]

I would assume that proprietary methods supporting constant perceptual quality - including the handling of difficult scenes and transitions - have alredy been implemented in current well known (non-real-time) encoders in one or the other form. TMPGEnc for example supports "constant quality" mode, and the Q parameter for CCE does not only mean "constant quantization" but it pertains to "contant quality" as well, AFAIK. Similar applies for adaptive matrix features on a per GOP basis. As the subject is strongly mpeg2 encoder related I am not sure if it is really a topic for RB-Opt or DVD-RB.

[mikenadia]

Sorry. If the mod thinks it will be better in the MPEG2 forum, they can move it.
But in the first paper, they talked specifically ( and almost only ) about optimal redistribution among 9 scenes and , based on visual effects, they found out that the optimal Q was not constant among scenes (Q ranging from 6 to 12) .
Even better, they propose a different model ( described on a step-by step basis ) that they think is getting closer to constant visual perception. I like the idea and just adapt it to DVD-5 constraint.
The second paper gave me ideas to improve the first model , so I just share the links.
I thought that one of the purpose of Rb-opt was to look for optimal distribution . I misunderstood .
Thx

[jdobbs]

I think you are looking at it backwards. You are, in redistribution, trying to find a bitrate. Since Q and bitrate are (generally) inversely proportional, the Q is used to find a common required bitrate across a fairly large group of representative GOPs. Then the bitrate is fed to the encoder -- which applies variability within each segment to give the best perceptional presentation while keeping the average bitrate.

[mikenadia]

Last comment for me on the subject.
I am doing exactly doing that (computing bitrates:cf title of this thread) based on different Q per cell ( and therefore bitrates because , as you say, QxBitrates is almost constant), trying to have the same visual perception.
Once you have Q, you have bitrates (and vice-versa).
I do not think that allocating bitrates based on the same Q (you compute an optimal Q and then allocate bitrates) is an optimal hypothesis (therefore the need to compensate the output of the model by such things as redist_threshold to prevent overshooting).
I surrender.
Thx

Edit: explanation of the model in the first paper (that model can be improved upon).
1) for each scene , the testers computed a bitrate at which the encoding looks good (table 2).
2) for each scene, go to Table 1 and deduct the Q that gives that bitrate.
They are ranging from Q= 6 to 12, invalidating the Constant Q hypothesis.
3) if we agree on that (that the Constant Q hypothesis is invalid) , keep reading , otherwise quit.
From table 1 , they compute for each scene by interpolation the Q that will give a bitrate of 4000 Kbps (called Q4).
4) then based on visual tests, they found out that their optimal bitrate allocation is proportionnal to the square root of Q4 (actually power between 0.5 and 0.6) (near fig 9 in the article!).
The fact that p is smaller than 1 makes redist_threshold less ( if not) necessary.

Also note that the more difficult to compress the scene is , the higher the visual optimum Q for that scene is : neither constant nor random ( I am starting to give too many hints!)

P.S. I will appreciate if users making posts in this thread ( even I am the only one) are open (not convinced ) to the idea that non-constant Q models might lead to a satisfactory bitrate allocation. I promise I will not post anymore in threads that are linked to constant Q models and , mods , feel free to move that thread if you wish so.

[Boulder]

CCE's constant quality (=OPV) mode does not use constant quants so it is close to what you are describing.

In any case, if DVD-RB encoded the whole video in one large segment, there would be no need for these maths. The outcome would be determined by the rate control mechanism of the encoder :p

[mikenadia]

You are right. But I am using HC ( not the only one) and was mainly thinking about bitrate allocation among segments of the main movie : I am not even convinced that there is a benefit to reallocate bitrates within the main movie (there must be one because so many people are doing it and have helped ithe "redistribution" development)
And that was why I was using Rb-opt in the first place ( Not a doubt that there is a benefit to reallocate between Movie and Extras ).
I thought that the redistribition function in Rb-opt was made to act like a rate control for constant Q encoders (probably not necessary for CCE encodes).

I' ve been told not to distribute between Movie and Extras at the same time. Now you are telling me not very useful for movie only (better trust the rate control system of the encoder). I am lost. I am going to revert to Rb-opt 0.12.
Waiting for DVD-RB to encode the main movie as one segment (may be it should, I do not know...).
Waiting for Rb-opt for HC encoder.
Waiting for Godot.
No.Going fishing. Less stressful.
Thx for your input.

[Boulder]

I think it's a personal preference whether you use redistribution for feature and extras simultaneously or not. In my projects it depends a lot on the material. If I know that the feature compresses well, I usually have all the stuff in the same redistribution pool.

Of course the optimal route is taken when the encoder has the whole video to process in one segment. The original idea, which eventually turned into bitrate redistribution, emerged from my 2-3 movie/episode per DVD encodes where I used (and sometimes still use) an Excel spreadsheet to calculate the average bitrate for all videos, keeping them at the same quality level. Since DVD-RB wants to process the video in smaller segments, redistribution is a good feature especially when you have a CBR source in which all segments have the same average bitrate.

[jdobbs]

I wouldn't agree with that. It might hold true if you were talking about a small number of frames... but a segment is almost always big enough to have the diversity needed. In my testing early on with DVD-RB I found that the distribution was virtually identical at the same overall bitrate as when you did the entire feature at once (as long as you set the average bitrate proportional to the original). This wasn't just a few tests, either, it was a huge number.

The fact is that all you need is a few thousand frames to give the diversity required. And when you use a multiple of the original bitrate (where there has already been an analysis in the original encoding) you come so close that the advantage of encoding the entire stream is non existent.

But... there is always those poorly encoded originals that use limited VBR or CBR -- and that is the real advantage to REDISTRIBUTION.

[Boulder]

One possible issue with multi-segment encoding with HC is when LUMGAIN is applied. The parameter does nothing in CQ mode so when redistributing, a segment with lots of darkness probably gets a low average bitrate. Then when you use LUMGAIN in the multisegment encoding, those scenes get a (heavily) modified quant matrix which is supposed to allocate more bits to the dark frames. However, there usually is not much extra to allocate because the average bitrate is lowish. When you encode the whole video in one big segment, the bits can always be stolen from the brighter scenes.

I'm not saying that you should implement one-segment encoding..I'm sure you have good reasons for the current method :)

[Sharc]

Given the facts (?) that for CCE the meaning of the Q factor stands for constant quality, and for HC the CQ means constant quantization one may conclude that it would be preferable to use CCE for the redistribution pass - at least in the sense of mikeandias discussion - would there not be the risk with the silent crash. In fact it appears that CCE with its constant quality strategy normally allocates lower average cell bitrates to the end credits than HC - so there is definitely a difference for certain "extreme" scenes between CCE and HC. The difference between CCE and HC is less for "normal" scenes though, provided one uses the Q_final. I haven't done similar tests with QuEnc yet, so I don't know how it would behave.
Still one has to keep in mind that in the 2-pass encode following the redistribution pass, the "momentary" bitrate and Q (quantization) will no longer remain fixed but will vary according to the encoders inherent rules for producing an "optimum result".

[mikenadia]

Using Rb-opt and HCwrap , I can find an optimal Q for the whole DVD.
With the following settings:
Main movie: Fox1 and CQ_BFactor=1.3
Extras: AVAMAT7 and CQ_BFactor=1.5
I have a "Extras" Reduction between 25 % and 33 %.
Otherwise, play with the settings.
This is only to determine bitrate allocation automatically. The actual encoding matrix can be different from Fox1 and AVAMAT7.

The fact that there are still some slight variations for the "real average quantizer" for each cell seems to come from the fact that the actual CQ_Bfactor seems to depend on the source, probably the bitrate and the matrix. I was wondering if the average CQ_Bfactor is known after the first pass, and in the affirmative , can we have an educated guess after a partial first pass (Nb of frames< 100 %: probably not unless they are selected by groups of 4 consecutive frames IPBB ?) and if we can readjust the CQ size to take into account the change in CQ_Bfactor (estimated after CQ pass compared to initial guess).

[Capsbackup]

@mikenadia;
You have been determined by this topic for a long time! :p