Suggestion for a new method of bitrate distribution

[Boulder]

4) The ability to encode the whole VTS at a time, that is, so that it is not split up in cells. I have seen too many DVDs that are encoded as CBR in which case this option would be really useful.To add to this: I just ran into another wonderful CBR encode where the average bitrate is ~3500kbps for every cell IIRC. Definitely not optimal. Here's what I suggest: encode every cell as one-pass (CQ mode in HC, OPV in CCE etc.) with the exact same parameters (same quant, same Q-value etc.) Then use the average bitrates of the sample encodes to determine the "correct" average bitrate for each cell.

I use a similar method for re-encoding my DVB captures so if you need the maths, I can send my Excel spreadsheet to you. I intend to try this approach in DVD-RB as soon as I have the time; you can do it manually using the rebuilder.ecl, the spreadsheet and the segment viewer/editor but it requires a bit of work.

[linx05]

@Boulder
That looks like a really handy way of doing things. It would improve the look of some parts of the movie wouldn't it? The bigger action scenes would get encoded differently to the scenes where nothing really happens.

[Boulder]

Exactly. Fortunately most movies are encoded properly using VBR but many series are CBR. One another problem is that many series have the same avg bitrate for every episode, the method I described can also be used to tackle that, and it doesn't matter whether the episodes are in the same VTS or not.

Simply put it's just doing a compression test and then tweaking the average bitrates accordingly.

[Boulder]

Just got the test encode done, and it does work just fine. A little manual work is needed and some extra time, of course, but it does allow you to correct the average bitrates of the cells when the original is almost or completely CBR, or when every VTS on the disc has the same avg bitrate.

I'll ask jdobbs or wmansir to move the couple of posts discussing this into a new topic since it's not just a request.

[Boulder]

OK, here's the deal.

1. Run the prepare phase in DVD-RB.
2. Encode all the avs files DVD-RB created using the exact same parameters for each file in one-pass mode. That means CQ for HCEnc, OPV for CCE etc.
3. Open the spreadsheet I've attached in this post.
4. Get the average bitrates and number of frames in each encoded cell.
5. Enter the space available for video value in the spreadsheet from DVD-RB's status window or the log and divide it by 1024 to get megabytes.
- Space for Video : 3 724 506KB --> 3637MB
6. Use DVD-RB's segment viewer/editor to edit the average bitrate of each cell. The correct value for each cell is in the corresponding "Avg kbps for cell in DVD-RB" field.
7. Encode and rebuild as usual.

If you need any more cell columns in the spreadsheet, you need to copy an existing column and edit the formula in B21 so that all the columns are included in the sum.

[techreactor]

that was exactly the way I was thinking when I read the subject of this post, but with the difference that can we not use any tool for reading the bitrate for each segment of the source and then using those values and reducing them equally in proportion to the compression required for the final encode ???

The results should be similar theoritically.

[Boulder]

What I've been trying to fix are those nasty CBR encodes where the source avg bitrates cannot be used for any calculations because every cell has nearly the same avg bitrate. The latest one was 4:3 full screen at 3200kbps which made me really angry, just how stupid can the authoring houses be?

My method can also be used in those cases where every VTS has the same avg bitrate but the bitrate demand is not even between them. This applies to many series DVDs.

[Sir Didymus]

Hi Boulder!

Very nice approach (simple and useful)...

Appreciation and thanks for sharing the xls!

Cheers,
SD

[Boulder]

Thanks:)

Hoping that jdobbs will some day add it as an option ;)

[jdobbs]

I'll take a look at putting in a check for CBR and making automatic adjustments when it is found.

[Boulder]

Be warned though that I've never seen anything specifically marked as CBR, everything has been as VBR but the min/max/typical average bitrates have revealed the truth.

[mpiper]

If we are re-encoding the entire movie, why would the bitrate and VBR vs. CBR bitrate matter?

Maybe I am missing the point on this, but I use CCE with it set to 2 passes for almost all my Rebuilder work. I have always assumed that rebuilder reads the AVWS files and generates its own bitrate using my settings (DVD-5 size, 10-15 VBR_Bias, 12-16 Quality Prec.)

So why would it matter if the original source was CBR or VBR with a single bitrate? Am I missing a point here?

Thanks,
Mike

[Boulder]

It does matter because every cell would have almost the same average bitrate - regardless of the amount of motion etc. which all contribute to bitrate demand of the cell. Opening credits on a black background need very little bitrate compared to a high-action scene. The method I described shifts the bits to the cells that demand more bitrate.

[mpiper]

Boulder, before I add anything else, let me say that I'm not arguing with you. I am honestly trying to understand the exact nature of how this works.

I amk the multimedia nerd for my department, which does multimedia and CBTs for our company, so I try to understand what I am talking about for future discussions.

So, here goes my questions (lol)

My understanding for multipass encoding is that:
A) The encoder generates a keyframe, using the selected matrix and allocates the midrange of bits, the next frame (intraframe) encodes only the changed information, using the intraframe matrix, using a number between the midrange and the lowest bitrate, and so on. (The avagerage is kept at a running total and extra bits are stolen or added as needed to maintain the average off from the maximum and minimum bitrates, more to high motion, less to the low motion.
B) The second pass, the encoder compares the original frame to the frame generated the first time. If the 1st pass was too high for avg bitrate, encoder is trying to find frames to steal from. If 1st pass was too low, encoder is trying to find frames to add to. If the frame is matching, bits may be removed to lower avg. If frame is not matching, bits will be added, as much as allowable based on max rate and average rate.

With that said (and it may not be accurate), a two pass encode at settings of 1700,2450,5500 would generate a first pass where every part of the video would be at 2450, while the second pass would shift/add/subtract bits so that some sections would be around 1700-2000, some parts around 2400-2500, and high motion parts would be 4000-5500. (Third and more passes would refine the bit shift by comaring again against the original until maximum match for datarate is reached.)

My understanding of your earlier statements (possibly also wrong) is that the second pass would result in 2450 for all parts of the movie, even when VBR is chosen, after the second pass. If this is true, wouldn't this be a case of a bad encoder, not a need for extra steps?

My impression of the Rebuilder add-on RB-Opt(?) that allowed changing individual groups was: if you wanted to push the limits further than the encoder was willing to go (shrink the size and quality to crap for the credits, etc.) to allow the main feature to have the maximum possible bitrate.

I have never really gone back and checked my datarate of encoded DVDs, so I have no basis for my arguement, except my understanding of the process. So if I am mistaken, please point out my errors so I can better understand (and explain to others) how this process works.

Thanks to everyone!
Mike

[Boulder]

Very simply put: DVD-RB splits the video in cells (usually a chapter?). It takes the average bitrate of each cell and uses that as a base for the calculations. DVD-RB handles each cell as a separate unit, it doesn't borrow or allocate bits from/to other cells.

If the authoring house has decided to go for CBR or almost CBR, every cell has (almost) the same average bitrate. For a VBR encode, the cells do not have the same average bitrate because the bits have been allocated more intelligently.

Now consider a regular action movie which has its slow and fast scenes. If both scenes (which usually are a separate chapter) have the same average bitrate, the fast action one will probably look terrible, or at least worse than the slow one.

The idea is to run a constant quality compressibility check pass which tells you directly which cells need more bits than others to keep the quality at the same level in each cell. The spreadsheet I wrote a long time ago uses the constant quality pass to calculate the average bitrate you need to set in DVD-RB for the cells.

Any questions, just ask :)

[mpiper]

So your saying I was roughly correct in my understanding, but by splitting each cell, there is no average datarate to use, so each gets slammed to that average since it is such a small sample size. Correct?

OK. Now I see the big problem.

So your process allows us to see what was encoded, calculate the optimum bitrate distribution and then set it for final encoding?

I don't have a sample DVD to try handy, but how does the spreadsheet know which groups to add or subtract datarate from? Or do I make a judgement call and then the rest auto adjust?

Mike

[mpiper]

After putting fake data into your spreadsheet (I used a CBR value to simulate what we are discussion), I see a new datarate at the bottom of each column, but each one is the same (i.e. CBR).

Are we assuming that running OPV will give us a VBR datarate set, which we can then use to set the VBR rates for multipass encoding?

Just checking on how this is working.

[Boulder]

Using OPV or CQ will give a proper VBR curve (despite the length of the cells!) because the cells are encoded with constant quality mode. Well, constant quant is not exactly constant quality but it is very close to that.

[jdobbs]

One problem with a reencode of CBR is that the best quality you are going to get is whatever bitrate was fixed in the original. In other words, anything that reencodes at a higher rate then the original would be wasting bits on (most probably) reproducing artifacts. So about the best you can do is reencode with VBR with the maximum bitrate set to the constant one found in the CBR. That's what DVD-RB does now. The problem is that it will likely come out smalller than the original (high demand hits the CBR rate barrier, low demand uses less).

[Boulder]

If you use a sub-optimal bitrate curve to re-encode, the results cannot be optimal. That's the problem I'm trying to address. Encode a blocky frame with a low bitrate - you get a frame with even more blocks. Encode a blocky frame with the highest bitrate you can afford without the rest of the video suffering - lowest possible damage done.

I've got one really terrible looking intro scene which I'm going to test my method on. It may take 3-4 weeks before I get to finish the test as I've got very hectic times ahead at work but if I'm lucky, I'll have them done early next week.

[jdobbs]

Encode a blocky frame with a low bitrate - you get a frame with even more blocks. In addition -- if you try to encode a frame that has blocks caused by the decoding at CBR limits -- the sharp edges of the blocks make it harder to reencode (because the encoder is trying to reproduce them as perfectly as possible) -- and you end up needing even more bitrate.

So the worse the original is decoded -- the more bitrate it takes to reencode (attempting to reproduce garbage). Unfortunately the encoder can't distinguish the macroblocks from the good parts.

[Boulder]

Another thing related to a new bitrate distribution: you should be allowed to set the maximum bitrate to any (sane) value. I've got one concert DVD where the avg bitrates between cells have very little variance and to add to that, the max bitrate is always ~6000-6500kbps.

EDIT: it would also be useful when you need to sharpen the source, for example 2010: The Year We Make Contact is one of those. Letterboxed 4:3 video and a very fuzzy one too..SeeSaw is a must there.

[Sharc]

....you should be allowed to set the maximum bitrate to any (sane) value.....


I think that can be done (at your own risk) by inserting

max_bitrate=nnnn

where nnnn = kbps, in the rebuilder.ini.
See DVD-RB Help / Hidden settings.

[Boulder]

Nope, if the value found in the source is lower than the value you set in the ini-file, the original maximum bitrate is used.

[jdobbs]

Yes. The original maximum is used to limit it. But, since you can't make a reencode better than the original -- any bitrate greater than the source maximum would be, by definition, the application of bitrate to retain new artifacts.

[Boulder]

But if you alter the source for example by sharpening, you need those extra bits at times. Many older DVD releases are quite poorly mastered and sometimes need quite serious actions. Of course, this doesn't apply to Joe Average, but there are quite a lot of us who like to use those advanced techniques.

[jdobbs]

Good point. You can override that limit by adding:

DISABLE_DYNAMIC_PEAK=1

to the "[Options]" area of REBUILDER.INI.

You have to be careful, though. If the maximum it is set too high, the audio reintegration might run into timing problems. No matter what you set for max -- DVD-RB will still enforce a limit [9800Kbs - audio_bitrate] though.

[Sharc]

yes, I remember, original bitrate is retained as a means to prevent stutters ....

[Boulder]

Good point. You can override that limit by adding:

DISABLE_DYNAMIC_PEAK=1

to the "[Options]" area of REBUILDER.INI.

You have to be careful, though. If it is set too high, the audio reintegration might run into timing problems.Thanks :) I usually use 8000-8500kbps to avoid playback problems, depending on the audio tracks.

[Boulder]

OK, I'll try to make the process a bit clearer. To achieve that, I made a small FAQ:

1) Why did you come up with this silly method?

I originally began developing it when I used to cram two movies per DVD. I quickly noticed that the average quant levels between the two movies were very uneven - one would have ~3.5 whereas the other would have ~6.0. This is, of course, not a good thing so I began thinking about how I could fix the situation. One thread by DDogg (http://forum.doom9.org/showthread.php?t=74141) in the DVD2SVCD forum actually helped me come up with the initial idea.

2) What is the purpose of the method?

- to shift bits from less demanding segments to those that need more
- to avoid the problems caused by using the original bitrate distribution especially if the original source is CBR or near CBR
- to maintain equal quality throughout the whole video

3) What do you do the OPV/CQ pass for?

The OPV/CQ encode acts only as a compressibility test. Every segment is encoded using the exact same parameters so a bitrate-hungry segment will have a higher average bitrate than a segment that doesn't need as much to keep the same quality level. This means that from the results of the compressibility test encodes, we can calculate a bitrate ratio which can then be used to redistribute the available space for video between the segments.

After the compressibility tests have been run and the bits have been redistributed between segments, the encoding process continues just like in any normal DVD-RB project.

For fastest possible processing, you can run the redistribution pass without any filtering.

I have not tested the effects of different Q/quant values. I usually use Q30-40 in CCE and q4-5 in HC for doing the pass. IMO the most important thing is that the bitrate stays below the specified maximum because when the bitrate gets close to the maximum, the quants need to be raised. This can prevent the redistribution from being optimal.

4) What are the pros and cons of the method?

pros:
- equal quality throughout the whole video (and DVD when all VTSs are in the same redistribution pool)
- all the pros of OPV/CQ encode apply but you get accurate filesizes

cons:
- the redistribution pass takes extra time

5) How do you know that the method works?

Lately I've been capturing a lot of DVB stuff which I then re-encode as they usually need quite heavy filtering. I often put 3-4 episodes per DVD. I've used the method for determining the average bitrate per episode so that the quality would remain constant. When all the episodes have been encoded and I've checked the log, the average quant levels have been roughly the same. The variance has usually been in the range of 0.1-0.2 units which is a very small difference IMO. If I had encoded every episode at the same average bitrate, the variance would have been much larger.

If you have any questions, please don't hesitate to ask.

[gurkan]

Using the q value to redistribuate the bitrates will produce an outcome with an "even" quality throughout, right? So this method won't help in regards of, for example, getting higher bitrates to segments with detail and less to segments with high action, except for the usual vbr in the individual segments?

Don't know if that made any sense. ;)

[jdobbs]

Tell me, why wouldn't you simply do a Q-based multi-pass encode? It works as you've recommended... but instead of doing a standard multipass DVD-RB run after the Q is complete -- you could use the Q-pass generated (except for CCE Basic) .VAF file and recalculate the bitrate (for each segment based on the sizes of the Q-pass output) to get the required result. That way you'd only do 2 passes total.

For most encodes (except the CBR ones you mentioned) you'll probably find you'll get very close to what you got from DVD-RB originally, though. The difference would be that instead of going through all the segments once doing 2 passes, you'd go through them twice doing 1 pass each time (Q followed by VBR).

[Boulder]

Using the q value to redistribuate the bitrates will produce an outcome with an "even" quality throughout, right? So this method won't help in regards of, for example, getting higher bitrates to segments with detail and less to segments with high action, except for the usual vbr in the individual segments?

Don't know if that made any sense. ;)The method will shift the bits toward high-action segments by nature, because they need more bits for the same quantizer than low-motion ones.

@jdobbs: I've never tested it that way..one problem that immediately comes to my mind is that the Q used in the redistribution pass might not be accurate enough and the sizing pass would then not yield optimal results. Besides, you couldn't use HC which is a major disadvantage IMO.

I'm not that sure about getting similar results than in the original DVD. The three discs I've done recently have had much more variance in the average bitrates than in the original source. Chances are that they have originally been encoded using some kind of bias towards CBR whereas OPV/CQ doesn't take any of that into consideration.

[gurkan]

@jdobbs

AFAIK, it's not possible to use opv on seamlessly branched dvd´s. Would the predictability that this opv/vbr-method introduces enable this ability?
I've got this horrible pal-version of The Abyss, dir. cut that only looks good if one does a vbr on the whole unsegmented shebang.. I've only managed to do this in big3, and they can't handle the branching.

[jdobbs]

Use with ILVU would depend on whether it is seamless branching or angles, but generally the answer is "No." ILVU needs to be very accurately reproduced in a very specific fashion.

I've done the Abyss using standard 2 pass encoding without any issues. It's the NTSC Version -- but it also is the same ILVU nightmare... I'm working from memory, though.

[gurkan]

@boulder
Do you have any suggestions as to what quantizer characteristics and vbr bias settings would produce the most opv-like outcome? Might be wrong, but i don't remember those being available for opv.

[Boulder]

Bias should be zero because OPV has no such parameter. I cannot say anything about quantizer characteristics, it probably is neither an multipass-only nor OPV-only parameter but it affects both the same way.

[gurkan]

Wouldn't setting bias to zero just produce segments with individual cbr's? Since opv doesn't do segments I'm having a hard time to see how this would be equal.

[Boulder]

I'm not sure if I'm following you. What are you trying to achieve and how?

[gurkan]

Sorry, got it all wrong.
Nevermind.

[Sharc]

..... For most encodes (except the CBR ones you mentioned) you'll probably find you'll get very close to what you got from DVD-RB originally, though.......

Hmm ... , not sure.
See attached example for comparison. Quite some difference for some of the segments. I used CCE, avamat6 matrix and initial Q=9 for redistribution.

[jdobbs]

You may want to try it without changing the matrix and see if it has an effect...

Also, the second column sure seems to have a much more significant amount of swing in bitrate between segments. Is this from a commercial source or a homemade video? The small differences in the DVD-RB column looks a lot like a realtime VBR encode (like from a home DVD recorder).

[Boulder]

Also another note: did you use 100% for redistribution? It will naturally give the most accurate results. I never use any other values than 100% myself.

[Sharc]

I extended the tests using different matrices (requiring different Q factors to aproximate the target size). See attached file.

I think it is the bitrate redistribution that makes OPV encodes often "look at least as good or better than" standard VBR multipass encodes.

[jdobbs]

I personally wince when I hear that. I have doubts that OPV ever exceeds multiple pass quality... although it can come close to the same.

[robot1]

@Sharc
have you done any visual comparison, between the backup you get with standard DVD-RB and the redistributed ones? (obviously same conditions, same matrices)

[~bT~]

Sorry if its off topic but I have a small question.

Is it poss to do a 1pass encode using the bitrates from RB-Opt after bitrate distribution?

If so, how? Edit the .ecl?

[Sharc]

@jdobbs:
Commercial DVD - The Departed.
Actually I was motivated for the tests by an earlier post by nashcity about the amazing results he obtained with OPV for that movie ... I wondered why it should be better than multipass VBR .... then came along the post by Boulder discussing the redistribution .....

@boulder:
I used 25% sample size for these comparisons. For a real encode I would probably spend some more time and increase the sample size, just to be on the safer side.

@robot:
I cannot watch the differnet encodes simultaneously. But I compared some frames (I know, not very meaningful). I noticed for example less macroblocking in a segement which obtained a significant increse in bitrate after redistribution (using CCE Default matrix, which is softer than the original matrix).

Really a strong combo DVD-RB and RB-Opt :)

[jdobbs]

I cannot watch the differnet encodes simultaneously. But I compared some frames (I know, not very meaningful). I noticed for example less macroblocking in a segement which obtained a significant increse in bitrate after redistribution (using CCE Default matrix, which is softer than the original matrix).As might be expected... but what about the ones where the bitrate goes down? For every plus in bitrate distribution there is also a minus. In order for the new distribution to be better -- you have to also conclude that the original (professionally authored) bitrate distribution was done poorly. With the exception of CBR sources (or realtime VBR) I just can't buy into that assumption (yet).

The biggest argument I have against redistribution is that you are encoding a previously compressed (MPEG) source. The reallocation is just as likely to represent more bits being allocated to reproduce errors (from the first compression). I've seen this in testing.

I'm not against new ideas... I just like to be sure they are correct before joining in... :)

[Sharc]

I personally wince when I hear that. I have doubts that OPV ever exceeds multiple pass quality... although it can come close to the same.

I am not sure here.
Imagine an original complex cell (segment) which looks good when watching it in the original format (bitrate). After rebuilding, the avearge bitrate is lowered to say 65%, and although the VBR multipass will allocate the bits within this segment for best quality, the final encoded quality of this complex cell may become marginal.
OPV in contrast will steal bits from less demanding segments (still maintaing a constant Q there) and give it to the demanding segment which eventually migth receive an average bitrate close to the original => OPV can indeed look better than multipass VBR.
Does this make sense?

[Boulder]

Sorry if its off topic but I have a small question.

Is it poss to do a 1pass encode using the bitrates from RB-Opt after bitrate distribution?

If so, how? Edit the .ecl?Do the redistribution, then switch to OPV in RB-Opt and let it do the prediction etc.

[~bT~]

Do the redistribution, then switch to OPV in RB-Opt and let it do the prediction etc.

OK, that got me thinking. If I do the OPV 1st to find the Q. Then enter that Q to do the resdistribution. Then switch to OPV again. Can I enter the Q I found earlier or is it best to do it again?

EDIT: Or is it straight redistribution and then OPV?

[jdobbs]

I am not sure here.
Imagine an original complex cell (segment) which looks good when watching it in the original format (bitrate). After rebuilding, the avearge bitrate is lowered to say 65%, and although the VBR multipass will allocate the bits within this segment for best quality, the final encoded quality of this complex cell may become marginal.
OPV in contrast will steal bits from less demanding segments (still maintaing a constant Q there) and give it to the demanding segment which eventually migth receive an average bitrate close to the original => OPV can indeed look better than multipass VBR.
Does this make sense?But that distribution between segments has already been done -- in the original encode.

[Boulder]

@jdobbs: The authors are often not that professional. I've seen a lot of crap, especially in localized (i.e. Scandinavian) releases. The MacGyver season boxes are one good example..encoded at CBR and every episode has the same bitrate. Guess how good that looks when the source tapes are from the 80s? :p

[Boulder]

OK, that got me thinking. If I do the OPV 1st to find the Q. Then enter that Q to do the resdistribution. Then switch to OPV again. Can I enter the Q I found earlier or is it best to do it again?

EDIT: Or is it straight redistribution and then OPV?I haven't done any tests whether you should predict the Q which you should use in redistribution. When I use CCE, I simply use a value between 30 and 40. The fastest route is to use a generic value in redistribution and then do prediction against the new bitrates.

[Sharc]

As might be expected... but what about the ones where the bitrate goes down? For every plus in bitrate distribution there is also a minus. In order for the new distribution to be better -- you have to also conclude that the original (professionally authored) bitrate distribution was done poorly. With the exception of CBR sources (or realtime VBR) I just can't buy into that assumption (yet).

The biggest argument I have against redistribution is that you are encoding a previously compressed (MPEG) source. The reallocation is just as likely to represent more bits being allocated to reproduce errors (from the first compression). I've seen this in testing.

I'm not against new ideas... I just like to be sure they are correct before joining in... :)

I basically agree, in particular what relates to poorly mastered originals with suboptimal bitrate distribution. The suboptimal distribution may not be noticeable when being played at the original rate.
However, the original has probably been encoded with a different encoding engine which most likely had a different characteristic (quantization, bitrate allocation, motion estimation etc.) and hence the original bitrate distribution may be suboptimal for the encoder which is used for the backup. The "Q" is a highly nonlinear process. I am not an expert though, just speculating.

[~bT~]

I haven't done any tests whether you should predict the Q which you should use in redistribution. When I use CCE, I simply use a value between 30 and 40. The fastest route is to use a generic value in redistribution and then do prediction against the new bitrates.

cheers mate. gonna try it out tonite. :thanks:

[Sharc]

But that distribution between segments has already been done -- in the original encode.
Yes, but the encoder for the backup knows nothing about the past or how the original has been encoded. It just gets frames served one by one (as good or as poor the original is) and tries to make the best out of it given a constant Q for the entire movie (OPV) or an average bitrate on a cell per cell basis (VBR multipass).
I feel OPV gives better results when the original bitrate distribution is "skewed" or when complex scenes are marginal in the original, and less demanding scenes got abundant bits wasted.

[Boulder]

Like I've said, the method tries to maintain a constant quality throughout the video. Using the original segment bitrates means that you must trust the authors, and my experiences say that many times they don't know that much about MPEG2 encoding.

Also one point is that the original encode possibly has a bias towards CBR. The redistribution will nullify its effect completely - if you use any bias in your backup encode without redistributing the bits, you'll push the bitrate curve even closer to CBR.

[jdobbs]

Yes, but the encoder for the backup knows nothing about the past or how the original has been encoded. It just gets frames served one by one (as good or as poor the original is) and tries to make the best out of it given a constant Q for the entire movie (OPV) or an average bitrate on a cell per cell basis (VBR multipass).
I feel OPV gives better results when the original bitrate distribution is "skewed" or when complex scenes are marginal in the original, and less demanding scenes got abundant bits wasted. But DVD-RB does, because it scanned the original before the encoding starts -- and it's knowledge is reflected in the bitrates provided to the encoder. Lot's of testing has been done related to that argument... I very strongly disagree.

[gurkan]

But that distribution between segments has already been done -- in the original encode.

You can't expect the bitcurve to be identical in proportion between a dvd9 and a dvd5. What's passable in one bitrate isn't neceserely passable at another. Also, the movie has changed and new artefacts have been introduced. So the movie we're about to encode has to be considered new raw-material, not to be confused with the movie that was originally encoded.
So therefor, the original bitcurve that was used for a movie without artefacts and macroblocks no longer is useable.
To quote you from another post:
Unfortunately the encoder can't distinguish the macroblocks from the good parts
Where bits will be used to retain unwanted macroblocks also wanted details will be retained. Keep the same low bitrate as the original and both macroblocks AND details we want to keep will get worse, because the encoder can't see the difference. If you can afford a higher bitrate to retain macroblocks and details, use it. A low bitrate will only increase the macroblocks.

[jdobbs]

Ok this is turning into a debate and a lot of what I'd consider incorrect "facts" are being thrown about. I'll just have to disagree and leave it at that. I'll probably add something along these lines into DVD-RB for sources that are CBR or limited VBR (like DVD Recorder or digital broadcast realtime encoded input), because I think it has merit there. But the bitrate curve argument was finished three years ago after a lot of testing and proving.

[gurkan]

Ok. Sorry if I offended you. That debate took place before my time.
But, if you do, would it be possible to put some alternative in the rebuilder.ini for us who happen to disagree? It would be nice to have the choice to ruin my backups. :)

[Sharc]

@jdobbs
...But DVD-RB does, because it scanned the original before the encoding starts -- and it's knowledge is reflected in the bitrates provided to the encoder. Lot's of testing has been done related to that argument... I very strongly disagree.

May be I misunderstand it, but see for example cell no. 38 (= end credits) in my table (Example_1_.zip in previous post). At the reduction level of 62% DVD-RB allocates to cell 38 an average bitrate of 3305 kbps. I understand that in the subsequent VBR multipass encode this segment would be encoded at this average bitrate of 3305 kbps, with lower and higher swings according to the VBR mechanism and frame complexity. Still the average would aim at 3305 kbps for cell 38. Correct?

When doing a redistribution, the new average bitrate is reduced to 608 kbps for cell 38, because the Q-controlled encoder decided that 608 kbps is enougth for maintaining the desired "quality" given by the Q value, with lower and higher swings according to the actual frame complexity, but still maintainig an average of 608 kbps for cell 38. Means that cell 38 consumes less bits, and the free bits are available for redistribution to other more demanding cells/segments that will move closer towards the original bitrate, hence reducing the bitrate reduction for the demanding segment, which in turn should improve the quality of the encode (similar to blanking some extras for freeing up space).
Correct, or do I misunderstand something?

I agree that there is a risk that the bitrate reallocation might in some cases be abused for more accurately reproducing artefacts and noise of the original (poor) encode. This is definitely not what we want. But still, even the segments that benefit from the redistribution will remain below the original bitrate.

[Boulder]

@everyone: I intend to run a series of tests. The first test DVD will be Predator SE R2. It contains some rather still scenes and some with a lot of motion (trying to encode a jungle scene is a real test to any encoder). I try to post samples from a normal backup and a redistributed bitrate backup as soon as possible. I'm going to test 4-5 different kinds of R2 DVDs to get a good sample variance.

[gurkan]

@boulder
Nice.
A suggestion would be to pick out the two segments that differs the most between normal dvd-rb and redist. One where destrib is bigger than normal and one where normal is bigger than destrib. To see if the quality-gain in one segment justifys the quality-loss in another.

[jdobbs]

Ok. Sorry if I offended you. That debate took place before my time.
But, if you do, would it be possible to put some alternative in the rebuilder.ini for us who happen to disagree? It would be nice to have the choice to ruin my backups. :)We who seek the truth are never offended! If we can make encodes better -- I'm all for it. But I don't want to build false expectations...

The one thing to remember in all this is that for every bit you add to one section of the output -- you are subtracting a bit from another. If the sharp edges of a macroblock requires more bits to keep it sharp.... where are you sacrificing quality in order to do that? The only thing I can guarantee is that you are sacrificing somewhere.

Another thing to keep in mind.... Q for the most part means quantization. Quantization changes with different matrices... so you can't compare the resultant bitrate of one matrix to that of another.

[gurkan]

Yep. Changing the matrix between the projects would render the test useless.

[Boulder]

Another thing to keep in mind.... Q for the most part means quantization. Quantization changes with different matrices... so you can't compare the resultant bitrate of one matrix to that of another.But you are not supposed to look directly at the OPV encodes but the bitrates that are calculated from them..this means the final result should be roughly the same no matter what matrix you use. Of course, different matrices treat the frequencies differently, but it should apply on a general level.

It may take a while before I get the samples to you as I work the weeks and commute 3 hrs every day..I've also got two little kids so spare time is something I often don't have :) If anyone knows a good place where to upload the sample videos once I get them done, please post links here.

[Sharc]

Another thing to keep in mind.... Q for the most part means quantization. Quantization changes with different matrices... so you can't compare the resultant bitrate of one matrix to that of another.

Yes. The meaning of the Q factor in CCE is perhaps a little obscure. It relates to quantization and it relates to some other form of "quality" as I understand. In my example the value of Q for obtaining a DVD-5 target size - as a basis for the redistribution - ranged from Q=9 for Avamat6 to Q=58 for the original matrix (which is approx. CCE Default matrix coefficients divided by 2).

Looking forward to Boulder's samples.

[jdobbs]

Here's how I did my original tests on this subject of original allocation: I encoded the entire movie at a single average VBR bitrate, allowing the encoder to allocate across the entire 2 hours or so. I then did the same movie using DVD-RB and the original bitrate distribution across segments that it picks up during scanning. The maximum bitrate was identical on both. On the vast majority of titles I found that the bitrate curves of the two methods followed each other across the entire movie very, very closely. That drew me to the conclusion that the original VBR method (on the commercial source) took the entire stream into consideration on the vast majority of discs, and that the allocation using the two methods was roughly equivalent.

[Boulder]

That's not the same as what we're doing here so it looks like I really need to do the tests :)

The DVDs I chose are:

Predator SE
The Mask (the Jim Carrey movie)
Ed Wood
Keeping Up Appearances season 5 disc 1
The Two Towers EE disc 1
EDIT: Lapinlahden Linnut, Kuudesti laukeava (a Finnish TV series) - very low avg bitrate for an interlaced source, blocky video at times

I shall use CCE, vaf + 2-pass. No filters except for ColorMatrix when DVD-RB decides to put it there. CCE standard matrix in all bitrates, adaptive quant matrices off, DC prec 9 bits, GOP length 15 frames. Max bitrate at 8000kbps (because CCE is known to spike at times, also that's why the 3-pass), dynamic max bitrate allowed. Redistribution will be done at 100% sample size.

I'll run the projects with non-redistributed bitrates and redistributed ones and then try to choose the samples the best I can. I admit I'm biased but I'll try to look for both good and bad decisions.

[gurkan]

The maximum bitrate was identical on both. On the vast majority of titles I found that the bitrate curves of the two methods followed each other across the entire movie very, very closely.

Did you find the minority of titles, where they didn't follow eachother, better or worse without original bitrate distribution?

[~bT~]

@ Boulder

www.maxupload.com

[gurkan]

@Boulder
I guess you'e going to use 100% sample for the opv redistribution. If so, you may want to add that to the spec to make it more clear.

[Sharc]

Just a note:
I found little difference in the calculated redistributed bitrates for 25% sample size or 100% sample size, respectively. So in practice some time can be saved using smaller sample sizes, like 20 ...30% IMO. The deviations were less than 3% in my test cases.

[gurkan]

@Sharc
I'ld have to disagree. Using less than 100% would introduce chance into to the equation. There is no way to know that those 30% can represent the remaining 70% as well.
IMHO opinion it's imposible to draw any kind of conclusion from such a test.
Why be 97% sure you're right when you can be 100% sure? :)

[Boulder]

Yes, I'll be using 100% to avoid any errors in the calculations the best I can.

[Sharc]

I did some more analysis with my example and conclude as follows:

1. The cells that got more bits allocated did not really improve in quality. These were mainly scenes/frames which suffered from (slight) macroblocking in the original encode. So the bits were apparently primarily abused to preserve the sharpness of the macroblock edges, without substantilly retaining other details better.
2. The cells from which bits were stolen did not noticeably deteriorate. Apparently these cells got more than adequate bits allocated in the original encode.
3. The conclusion is that the original is probably suboptimum regarding bitrate distribution, means too flat. Complex scenes should have been given higher average bitrates, simple scenes would probably have tolerated lower average bitrates. => It was not possible to recover that "damage" by doing a bitrate re-allocation in the re-encode.

So much for my example. I expected actually better results when seeing the large swing after doing the redistribution. In this case it seems simply to indicate that the original is probably suboptimum. Again, this applies for my example. I am looking forward to other people's findings.

[jdobbs]

I'm planning to add an option for redistribution in v1.26. I'm thinking of its use mainly for realtime VBR encodes (like DVD Recorders) and CBR.

3. The conclusion is that the original is probably suboptimum regarding bitrate distribution, means too flat. Complex scenes should have been given higher average bitrates, simple scenes would probably have tolerated lower average bitrates. => It was not possible to recover that "damage" by doing a bitrate re-allocation in the re-encode.They were probably encoded with a high bias.

[Boulder]

I did some more analysis with my example and conclude as follows:

1. The cells that got more bits allocated did not really improve in quality. These were mainly scenes/frames which suffered from (slight) macroblocking in the original encode. So the bits were apparently primarily abused to preserve the sharpness of the macroblock edges, without substantilly retaining other details better.
2. The cells from which bits were stolen did not noticeably deteriorate. Apparently these cells got more than adequate bits allocated in the original encode.
3. The conclusion is that the original is probably suboptimum regarding bitrate distribution, means too flat. Complex scenes should have been given higher average bitrates, simple scenes would probably have tolerated lower average bitrates. => It was not possible to recover that "damage" by doing a bitrate re-allocation in the re-encode.

So much for my example. I expected actually better results when seeing the large swing after doing the redistribution. In this case it seems simply to indicate that the original is probably suboptimum. Again, this applies for my example. I am looking forward to other people's findings.Interesting findings, thanks for taking the time to analyse your samples. The results are mostly what I expected - trying to minimize the damage (i.e. prevent more macroblocking in already blocky frames) when re-encoding. The method is not the Holy Grail but a handy tool IMO.

Did you do any comparison between the redistibuted encode and a regular DVD-RB encode?

I'll get the last two regular encodes done tonight, then I get to doing the redistribution passes.

[Sharc]

Did you do any comparison between the redistibuted encode and a regular DVD-RB encode?


Yes, but only with very few I frames, so the comparison is perhaps not statistically relevant. In fact the results of the standard 2-pass VBR and the redistributed are virtually identical. Damage remains damage. I might, for final conclusions, also have to consider B and P frames, though.
At least I can say that the redistribution did not introduce more damage, but it is difficult to attest it a real benefit - in my case and as per present state of the (still limited and incomplete) analysis.

[Boulder]

Still, if the method doesn't improve the quality of the final result, it doesn't justify the extra time needed to do the redistribution pass.

Then again, it's difficult to say when the bits should be redistributed and when not..I just noticed before I headed for work that one test DVD is encoded with VBR but every segment has been encoded individually and every segment has the same average bitrate. It will be interesting to see the comparison between a regular and a redistributed bitrate encode. In theory, the redistributed one should look a whole lot better.

[Sharc]

I'm planning to add an option for redistribution in v1.26. I'm thinking of its use mainly for realtime VBR encodes (like DVD Recorders) and CBR.

Bitrate redistribution may perhaps also have its benefits when filters are applied (e.g. for noise removal, softening or sharpening). I have however not run any tests yet in this respect.

[Fishman0919]

Here's something from a while ago... I would use this way for encoding with cce 2.50 and 2.66 before I started using DVD-RB back around ver .20 'ish. I couldn't find on this doom9 but here it is from the /turkish.doom9.org


http://turkish.doom9.org/index.html?/mpg/cce-advanced.htm

[Boulder]

Here's one screenshot comparison. You need to zoom in to see the difference well..

In the first pair, pay attention to Legolas's face. There's some chroma noise there in the RB picture but not in the redist one.

In the second pair, look at the sky near Aragorn's face. There's macroblocking in the RB picture, which somehow disappears in the redist picture.

The first pair was taken from a segment in which the redist one has much more bits - 5665kbps compared to 4524kbps that the RB one has.

The second pair was taken from a segment in which redist had 3094kbps and the RB one had 3835kbps.

I need you to tell me which video sample looks better upon playback. Download link : http://maxupload.com/D58429D2. Pay attention to the scene change.

The pics are here : http://maxupload.com/26DF01F1

More later...

[Sharc]

I'm planning to add an option for redistribution in v1.26. I'm thinking of its use mainly for realtime VBR encodes (like DVD Recorders) and CBR.

They were probably encoded with a high bias.

I did a new test with my sample:

- adding the parameter cpu=3 in the mpeg2source("................",cpu=3) as a filter against macroblocks
- then applying the redistribution

The bitrate redistribution - see attached file, yellow marked columns - is very similar to the one without using the cpu=3, however the macroblocks were significantly less prominent, and I did not notice a loss in useful details.

@jdobbs:
I wonder if it would make sense to include an option in Rebuilder 1.26 for inserting the cpu=x parameter in the mpeg2source() line. I think it could be useful even for standard VBR encoding, to prevent abusing the bits for the reproduction of sharp macroblock edges in non-perfect originals.
There are other macroblock filters, but I found the cpu=3 in dgdecode very effective and fast.

[Boulder]

There is the option, MPEG2SOURCE_OPTS under [Options]. It was added recently. For CPU=3, you would use MPEG2SOURCE_OPTS=cpu=3..though I would use CPU=4 ;) I actually always use CPU=4 in all my encodes. It increases compressibility by a substantial amount but it's still very gentle to details.

[Sharc]

:thanks:
I was not aware that this option is already available!

[jdobbs]

In the second pair, look at the sky near Aragorn's face. There's macroblocking in the RB picture, which somehow disappears in the redist picture.

The second pair was taken from a segment in which redist had 3094kbps and the RB one had 3835kbps.I gotta be honest... I'm of the opinion that there's some kind of flaw in your testing method. I think you'd have to agree that you can't, no matter how hard you try, get a better picture by reducing the bitrate. The best goal you can expect would be to get one that is either as good, or not noticably worse.

[Boulder]

I can upload those two segments and you can see for yourself..it will take several hours though. I'll also doublecheck myself.

Maybe CCE is simply smoothing the image as the bitrate is lower, I wondered about that myself too you know.

[~bT~]

I can confirm that the redist method does wonders.

I encoded the same movie, once using normal settings in RB and the other using redist with the other settings the same.

Problem is, I cant back my findings up as I had to get rid of the files to make space on my HD for a new project.

Anyway, i'll be sticking to redist method. Thanks Boulder!!

[Sharc]

I need you to tell me which video sample looks better upon playback.
The pics are here : http://maxupload.com/26DF01F1


Nice job.

Here my vote:

Pictures:
Pair 1: redistributed is better
Pair 2: about the same, depending where we look at
=> Overall I give the redistributed a PLUS

Movie:
I give the file ttt_3.demuxed.m2v a PLUS
=> less macroblocks, less encoding artefacts (less mosquitos), less color spillover

[Boulder]

I snipped a short clip of that dark scene for you to assess. That scene might be something that should also be tested with HC.

http://maxupload.com/20AA3D83

[gurkan]

ttt_3.demuxed.m2v looks slightly better in my eyes as well.

out of curiosity:
what's your vbr bias and qual prec(quantizer characterisitics) for these encodes?

[Sharc]

I snipped a short clip of that dark scene for you to assess. That scene might be something that should also be tested with HC.


I cannot really give a preference this time, from a viewing point of view. So at the end I would say the winner is the one with the smaller file size .....

[jdobbs]

I don't want to sound too harsh. I even like to think that I always keep an open mind... but, if anyone believes that I will somehow be convinced that lowering bitrate can improve picture -- it's really a lost cause. Assuming no flaw in the encoder, and the same parameters being fed to the it, I will stick firmly to the limitations of the laws of mathmatics and physics.

I've seen some very compelling testimony that an alien spacecraft crashed at Roswell... but that doesn't mean I take it as fact.

Of course that's just me... everyone is entitled to his/her opinion -- and RB-Opt is right there for use if people want to redistribute. :)

[Sharc]

... but, if anyone believes that I will somehow be convinced that lowering bitrate can improve picture -- it's really a lost cause. Assuming no flaw in the encoder, and the same parameters being fed to the it, I will stick firmly to the limitations of the laws of mathmatics and physics.

I very much agree, but given this a fact, one would also have to admit that if the stolen bits are thrown into another cell which then gets less compressed there is at least a chance that this cell will improve .... from the very same argumentation :D . The question is just if the benefit is greater than the loss or vice versa.

[jdobbs]

I have no doubts that adding bits will improve an encode. So when you redistribute, all the cells that went up could theoretically improve as long as they don't go higher than the original bitrate. No argument there. In fact the only argument involved in that side of the equation might as to whether it is noticable -- but for the sake of argument, let's assume it is.

That brings us to the other side of the equation. The exact same logic would dictate that every cell that lost bitrate would have to degrade. Period. No room argument. Fact. Again the argument might be made, again, as to whether it is noticable.... but it ends there. Saying it can improve only invalidates the argument. It is like saying 5-2=4.... you can write it on paper, you can say it out loud, you can even post it in on a forum. But you can't, no matter how hard you try, make it true.

That doesn't make redistribution wrong... it is possible that the improvements made on the "up" side of the equation can more noticably (but subjectively) improve the viewing experience than the "down" side degrades it. That's feasible -- and is more what I thought I might hear as an justification for the method.

Something I noticed, though... when you save the settings with RB-Opt the minimum bitrate is changed from 300 to 0. So in order to make a one-to-one comparison, you'd have to do the same in the regular RB encode. I'm not sure if there are other differences. RB sets it to 300 to prevent "hangs" on certain sources when using CCE.

[Sharc]

Exactly. Fully agree!

I
Something I noticed, though... when you save the settings with RB-Opt the minimum bitrate is changed from 300 to 0. So in order to make a one-to-one comparison, you'd have to do the same in the regular RB encode. I'm not sure if there are other differences. RB sets it to 300 to prevent "hangs" on certain sources when using CCE.

Interesting point. I was not aware of it. I think it is the same as with OPV in this respect (?).

Anyway, I found this debate more interesting and challenging than discussing how to fill the last 10 MB on a DVD-5.

And finally, with regard to the spacecraft at Rosewell, I am probably the one who would go there to check with my own eyes if it's true or not ..... :)

[robot1]

Something I noticed, though... when you save the settings with RB-Opt the minimum bitrate is changed from 300 to 0.
That would be a bug :(
I'm trying to reproduce it right now.
[OT] Have you solved the problem you talk in the RB-Opt thread?

[jdobbs]

That would be a bugSorry, don't waste your time... I found the problem and it was in between the keyboard and the seat.

[Boulder]

VBR bias was set to 0 and quantizer characteristics were 16.

I'm going to encode that dark scene with HC with the original, RB-calculated avg bitrate and the redistributed one. I have a feeling that CCE does some smoothing - as you can see from those sample videos I posted, the redistributed one uses a higher quantizer which in theory should degrade the image more. IIRC the original quant in those parts of the frame was 5 and the redistributed one had 7 or so.

EDIT: Oh, in that first video sample, the file ttt_3.demuxed.m2v is from the redistributed encode. While looking at the samples, I also noticed how bad CCE's scene change detection is..no wonder that scene change looks so bad as it's P-B. It takes a few frames to get to the next I-frame which will fix the video again.

[Boulder]

That doesn't make redistribution wrong... it is possible that the improvements made on the "up" side of the equation can more noticably (but subjectively) improve the viewing experience than the "down" side degrades it. That's feasible -- and is more what I thought I might hear as an justification for the method.This is something I'm trying to research here as well. As we're talking about MPEG2 encoding which deals a lot with psychovisuals, evaluating results is not that easy.

Still, encoding with HC shows (to my eyes) that the encode with a lower avg bitrate looks better, at least in per frame analysis. See the bottom left corner, the macroblocking is less evident in the redistributed encode (avg 3094kbps) than in the original one (3835kbps). Aragorn's face gets a little smoother though, but I don't know if it's good or bad in this case.

There's also pics of Excel graphs showing the bitrate distribution for three movies I have in the test. It shows that the biggest difference seems to be some kind of bias. The original bitrates are rather flat compared to the redistributed ones.

http://maxupload.com/3BA18D0E

[jdobbs]

I think the best way to do it is to use AVISYNTH's StackVertical() function. Then the two sources can play the same frames at the same time on the same screen so you can do a realtime full-motion comparison. I like to slow it down a little with AssumeFrameRate() to make it easier to see.

Of course the result will be subjective and you always have a tendency to see what you were expecting to see -- but in MPEG pretty much everything is subjective.

[Boulder]

Too bad it won't fit on my display, 720x576 two times is a bit too much :( TTT might just make it if I get rid of the borders.

[jdobbs]

Ok. I just did some testing with StackVertical and AVISYNTH comparing some encoding I did of X-Men 3. In segment 16 of the main movie I noticed that the redistributed one is smaller than the original RB one (BR 2613 as opposed to the original 3313).

The difference is immediately noticable -- but not in blocking etc... the lower bitrate that was created by redistribution is very obviously missing detail, especially in darker areas of the screen. If you try something similar, I think you'll see the same.

You can also try StackHorizontal().

[Boulder]

I'll try to find a good scene to compare. Ed Wood had one in which the difference between the two average bitrates was quite big, I'll check how it looks.

[gurkan]

@jdobbs
Did you find that the increase of detail, etc, in other segments justified the loss of detail in segment 16 (and presumably others)? Or were they not as noticeble?

[Rippraff]

Sorry guys, but I can't follow your discussions anymore.
I have no doubts that adding bits will improve an encode. So when you redistribute, all the cells that went up could theoretically improve as long as they don't go higher than the original bitrate. No argument there. In fact the only argument involved in that side of the equation might as to whether it is noticable -- but for the sake of argument, let's assume it is.

That brings us to the other side of the equation. The exact same logic would dictate that every cell that lost bitrate would have to degrade. Period. No room argument. Fact. Again the argument might be made, again, as to whether it is noticable.... but it ends there. Saying it can improve only invalidates the argument.
I totally agree with this, but this is only the bitrate side. And for high quality originally encoded discs this is also valid for the quality side.

But what about originally badly encoded or CBR dvds? In these cases I can imagine that bitrate redistribution is a good idea.
Say you have a credits segment which would look fine with an average bitrate of 1000 and you have a CBR encode with 3000. Of course you would lose bitrate with redistribution but I doubt you would notice a quality drop for this example.

@Boulder
Having said this, I'm wondering why you just chose LOTR as an example which isn't CBR nor badly encoded (and I guess the same goes for X-Men 3).
And frankly speaking if I have to zoom in to see a difference then it isn't worth to spent extra work IMHO.
No offence intended, your testings are much appreciated but I'ld like to see differences on CBR sources. Unfortunately I don't have such material.

Cu Rippraff

[jdobbs]

As I said earlier, it may be possible you may get an improvement on CBR or VBR that is encoded with a limited scope (like DVD Recorders). But even then you really couldn't do any better than the original CBR bitrate. So if the CBR was set at, say, 3000Kbs... then any segment that came in higher (in the redistribution) than 3000Kbs would only be using the extra bits to try to recreate the errors of the original CBR encode (at any point in the segment).

The problems with CBR comes on the high demand side. Sometimes in CBR bits are wasted on scenes that don't need the bitrate... and there's not enough for scenes that could use it. But once it's encoded, it's too late to recover the high end -- it's already lost in the first encode.

If, though, your new encode had an average VBR bitrate of less than the original CBR rate (as you might see on a DVD-9 to DVD-5 conversion) -- you could probably do a better job of keeping the original CBR quality (even at the lower bitrate) by using redistribution.

[jdobbs]

Ok. I just did some testing with StackVertical and AVISYNTH comparing some encoding I did of X-Men 3. In segment 16 of the main movie I noticed that the redistributed one is smaller than the original RB one (BR 2613 as opposed to the original 3313).

The difference is immediately noticable -- but not in blocking etc... the lower bitrate that was created by redistribution is very obviously missing detail, especially in darker areas of the screen. If you try something similar, I think you'll see the same.

You can also try StackHorizontal().Just an update. I've gone through some more of this... and it doesn't seem to be as obvious in some other scenes.

[Boulder]

The reason why I also tested VBR stuff is that I wanted to see whether redistribution would be useful only with badly encoded sources (=bad bitrate distribution). As you can see, I've got a couple of different DVDs to test. The last one is CBR, it's even labeled as such in the m2v stream! And it sure looks great..over 3hrs of interlaced video shot in the late 80s on a DVD9 at a constant bitrate:mad:

What I don't (yet) understand is the fact jdobbs points out : that you shouldn't use a higher avg bitrate than what the original has. I have lots of DVB captures which have a very limited bitrate range, and I intend to encode some video at the same avg bitrate that the original video has and also at a higher bitrate and see what happens.

[Sharc]

What I don't (yet) understand is the fact jdobbs points out : that you shouldn't use a higher avg bitrate than what the original has.

At least in one case this seems plausible:
Assume that the studio used the same encoder as we do for our re-encoding => Why should our re-encode produce a better result just by boosting the bitrate above the original encode? It cannot recover details that have already been lost by the original studio encode.

I am not sure however if there exists a chance for improvement when the studio encoder and the one which we use for the recoding are different, as they probably have non-identical caracteristics. Similar when we are using filters (eg sharpening) for the recode. This is very speculative, though.

At the end, we may still have the problem to decide a priori if to go for redistribution or not.

[gurkan]

What I don't (yet) understand is the fact jdobbs points out : that you shouldn't use a higher avg bitrate than what the original has.

That's what gets me aswell. And sure, it can't recover details that have already been lost by the original studio encode. But it would retain the details that haven't been lost.

To me it's fairly simple. 1 x 0,5 x 0,5 = 0,25
1 x 0,5 x 0,75 = 0,375

Where 1 = lossless quality
0,5 = original bitrate
0,75 = new redist bitrate

And by the way: 1 x 0,5 x 0,5 != 0,5

It is like saying 5-2=4.... you can write it on paper, you can say it out loud, you can even post it in on a forum. But you can't, no matter how hard you try, make it true.
Sorry, but you keep bringing it up, jdobbs. I just can't seem to let all these "facts" pass me by undisputed. ;)

[Sharc]

....... your testings are much appreciated but I'ld like to see differences on CBR sources. Unfortunately I don't have such material.
Cu Rippraff

Perhaps you have this one you could try with: Children of Men. Not CBR, but "almost CBR". See attachement.

[Boulder]

In any case, it is not as simple as not allowing to go over the original average bitrate.

There is the fact that encodes have been done using different quantization matrices, different encoders and different parameters for the encoders. The quant matrix alone makes things much more complicated, and that is why I do not believe that the bitrate should be restricted to the average of the original one. And as was already pointed out earlier, the material we are working on is not the same as what was used in the original encode. This adds more complexity to the equation.

Nevertheless, I find this discussion a good one, informative and also very civil. Let's try to keep it that way :)

[Sharc]

And here a case with a surprising similarity - I already began to doubt that such examples do exist.
No strong argument for redistribution in this case, but still: the bits of the long end credits are given to the movie.

[Rippraff]

As I said earlier, it may be possible you may get an improvement on CBR or VBR that is encoded with a limited scope (like DVD Recorders). But even then you really couldn't do any better than the original CBR bitrate. So if the CBR was set at, say, 3000Kbs... then any segment that came in higher (in the redistribution) than 3000Kbs would only be using the extra bits to try to recreate the errors of the original CBR encode (at any point in the segment).

The problems with CBR comes on the high demand side. Sometimes in CBR bits are wasted on scenes that don't need the bitrate... and there's not enough for scenes that could use it. But once it's encoded, it's too late to recover the high end -- it's already lost in the first encode.

If, though, your new encode had an average VBR bitrate of less than the original CBR rate (as you might see on a DVD-9 to DVD-5 conversion) -- you could probably do a better job of keeping the original CBR quality (even at the lower bitrate) by using redistribution.
Couldn't agree more. :)

Of course if they messed things up with the first encode you couldn't get better with a reencode (except useful filters but this is a completely different story ;)).
Sticking with the 3000 example I see the benefits with an intelligent redistribution that you hopefully stay around 3000 in high demanding segments in your backup.
But this implements that you must have other segments where bitrate can be saved.

And for the higher than original bitrate discussion, as we say here "you can't turn crap into gold". ;)
In my eyes it's logical that bitrates above original wouldn't help.

Cu Rippraff

[jdobbs]

Perhaps you have this one you could try with: Children of Men. Not CBR, but "almost CBR". See attachement.
That's (your graph in post #116) exactly the type of encode I believe this can help. But it's pathetic that the original is encoded in that way. The bad part is: I think those spikes above the original bitrate are all wasted bits... you just can't get back what has already been lost.

[jdobbs]

That's what gets me aswell. And sure, it can't recover details that have already been lost by the original studio encode. But it would retain the details that haven't been lost.

To me it's fairly simple. 1 x 0,5 x 0,5 = 0,25
1 x 0,5 x 0,75 = 0,375

Where 1 = lossless quality
0,5 = original bitrate
0,75 = new redist bitrate

And by the way: 1 x 0,5 x 0,5 != 0,5


Sorry, but you keep bringing it up, jdobbs. I just can't seem to let all these "facts" pass me by undisputed. ;)Sorry, but I just can't agree that it works that way. If I wanted to keep the "best" available detail -- I'd just copy that portion unmodified. What would the peak bitrate be? It would be the exact same as the original CBR -- with no spikes above.

I guess the only way I could imagine it any other way would be if my current encoder isn't very good, and it takes more bitrate to get the same picture. But I don't think that is the case with any of the DVD-RB supported encoders.

[jdobbs]

Does anyone have a good example of CBR or limited VBR encoding on an NTSC source? I'd sure like to get one for testing.

[Sharc]

That's (your graph in post #116) exactly the type of encode I believe this can help. .......The bad part is: I think those spikes above the original bitrate are all wasted bits... you just can't get back what has already been lost.

I think there is a misunderstanding here:
None of the peaks exceeds the original! The flat dark blue line represents the bitrate of the standard 2-pass VBR reduced bitrate of 65.4% according to DVD-9 to DVD-5 reduction.
The DVD original is therefore 100/65.4 higher, means it is in the 7200 ... 7800 range.
See attached graph (yellow = original) for clarification.

[jdobbs]

Oh. Yes, I definitely misunderstood that. That example is probably the best case I've seen for where redistributed bitrates can give you better output. Man, those may be CBR, but they are certainly high bitrates on the original.


@all

Here's an interesting test you may want to try:

1. Take a single segment and do a redistribution type pass on it.

2. Then encode it at a ridculously low bitrate -- something that you know is going to cause blocking and pixelation.

3. Then take that terrible output and do a redistribution type pass on it.

It's likely that you're going to find a higher bitrate requirement for reproducing the blocky, pixelated source than the original. But if you redo it at the same peak bitrate it won't look much different in the next run.

[edited] Added the third and fourth sentences after seeing the chart.

[Boulder]

@all: I'm currently running the RB encode on a CBR source and will run the redist encode after that. I should have some samples for you tomorrow if my time just allows.

I'll also run a test encode (original avg and a higher avg) on a DVB capture which uses a very limited VBR range and has a lowish average bitrate for the content.

[Sharc]

.....That example is probably the best case I've seen for where redistributed bitrates can give you better output......


I would say IMBO that the redistribution (or even basic OPV) produced visually better results in this case.
The great reduction was mainly applied - as expected - to dark or "flat" scenes. Maybe, if I would deeply analyze the dark scenes on a frame per frame basis I might discover some loss of detail and higher quant - perhaps. But would this loss of faint details in dark scenes be noticebale when watching the movie? At the other hand, bright and complex scenes have been boosted - means shifted closer to the original DVD bitrate (but still not exceeding the original). I would assume - without overanalyzing - that in these clear scenes any deficiencies are more visible and annoying than some loss of details or higher quant in dark scenes. The encoder (CCE) made hence a good decision how to reallocate the bits. It is possible however that another encoder might have taken a different decision.

[Sharc]

And here another example (Munich) with a rather flat distribution in the original.
I selected this one because it was cited elsewhere in this forum as an example for OPV producing "better quality" than multipass-VBR, and because people are looking for CBR type of sources. Yet this is en example for PAL.
I am always using the cpu=4 parameter in the mpeg2source, in order to avoid that the bitrate for the redistribution is primarily driven by residual macroblock edges in the original.

[Boulder]

Keeping Up Appearances:
http://img267.imageshack.us/img267/4000/kuapl4.th.gif (http://img267.imageshack.us/my.php?image=kuapl4.gif)
Note that the org bitrate graph means the one set by DVD-RB, the same applies to all graphs I created.

The following frames are from the field separated video as it's interlaced.

Here's one frame from segment 6 (higher bitrate compared to the original), first normal, then redistributed:
http://img267.imageshack.us/img267/5395/kuarb1nt8.th.png (http://img267.imageshack.us/my.php?image=kuarb1nt8.png)
http://img267.imageshack.us/img267/4605/kuaredist1yo9.th.png (http://img267.imageshack.us/my.php?image=kuaredist1yo9.png)

One frame from segment 39 (lower bitrate compared to the original), the same order:
http://img267.imageshack.us/img267/5329/kuarb2ky4.th.png (http://img267.imageshack.us/my.php?image=kuarb2ky4.png)
http://img76.imageshack.us/img76/7403/kuaredist2ft3.th.png (http://img76.imageshack.us/my.php?image=kuaredist2ft3.png)

In the first pair, the added bits really show up. The other pair shows the slight degradation when the bitrate is lowered, it was the worst case I could find when I quickly went through hi-motion scenes in that segment.

The question is as always, do the pros beat the cons? The bitrate redistribution method is designed to provide a constant quality level throughout the video. That means some parts will get more bits, some parts less. I guess everyone should answer the question himself, no one can tell you the right answer because there isn't one. I myself prefer constant quality but that's just my opinion.

[gurkan]

@jdobbs
Ok, this thickhead finally gets what you mean. A higher avg than the original would equal a bigger filesized encoding with
worse quality, so anything other than a straight copy of the segment would be rather stupid.
Case closed for me, at least. :)

A nice touch would be to have those copied segments excluded from the redist. calculations and add the remaining
over-the-avg-bits from those segments to the rest. Maybe throw in a 95% threshold as well.

[archaeo]

This idea - of adding bitrate above and beyond what the original source contained - reminds me of a debate on an old doom9 thread regarding the application of a filter called 'addnoise' by SansGrip. His idea for the filter was to add 'noise', or bitrate, during the encode to reduce DCT blocking. The thread is here: http://forum.doom9.org/showthread.php?s=&threadid=37135&pagenumber=1

Obviously the filter did not use Q as a means to distribute bitrate as does Boulder's idea, but the principle is the same in that bitrate is added to a frame/cell in an attempt to 'improve' on low bitrate sources. This seems similar to what people are exploring here in regard to determining the benefits of upping bitrate. As I recall, the debate was left somewhat open, with no firm conclusion.

[Sharc]

I am trying to find a rule as to when to apply bitrate redistribution.
Here my suggestion:

1. The compression factor of the standard reduction should be (<70%).
=> This leaves enough room for corrective bitrate swing after redistribution without exceeding the original bitrate,

AND

2. The variation of the original distribution should be (<15%).
=> The original bitrate distribution should be rather "flat".
Note: First and last cell to be excluded in the evaluation as these may contain intros and credits, or we may just disregard the "min" and "max" cells.

3. When applying redistribution, the inclusion of cpu=3 or 4 in the mpeg2source is recommended for suppression of macroblocks in the source.

The parameters in ( ) are for discussion, or may be user selectable.

I would expect that in these cases the redistribution produces equal or better results than the standard VBR multipass.

What do others think?

[Boulder]

Looks like a good basis to me. Actually I was going to ask jdobbs if he could add a graph in the prepare phase which would show the variance of average bitrates as the scanning progresses. One could use that as a helper item too.

[archaeo]

I would expect that in these cases the redistribution produces equal or better results than the standard VBR multipass.

What do others think?


Equal maybe, but better?
As I posted in the above thread, my question remains: Does the addition of bitrate to a cell above the original source level really just add 'noise' to the cell? There's no doubt that subtracting bitrate from a cell that can give it up and placing it into another that is below original bitrate levels is helpful, but the idea of a bitrate level going 'above' the original level is questionable. Perhaps, (as the sansgrip filter attempted) it can help in reducing blocking, but it will never be 'better' than the original bitrate level.

[Sharc]

Equal maybe, but better?
As I posted in the above thread, my question remains: Does the addition of bitrate to a cell above the original source level really just add 'noise' to the cell? There's no doubt that subtracting bitrate from a cell that can give it up and placing it into another that is below original bitrate levels is helpful, but the idea of a bitrate level going 'above' the original level is questionable. Perhaps, (as the sansgrip filter attempted) it can help in reducing blocking, but it will never be 'better' than the original bitrate level.

May be there is a misunderstanding here :
I am definitely not pleading for going above the original bitrate, and I am not expecting a "equal or better than the original" quality, but a "equal or better than standard multipass VBR" quality. This is possible without exceeding the original bitrate. In other words, I am not aiming at improving a poor original, we rather try to keep the perceived loss in quality due to the re-encode to the minimum by means of the proposed redistribution.

That may be well possible for cases similar to the one I showed in post #123. Up to now I normaly went straight to OPV in such cases, because I felt that even with an OPV undersize of say 100 .... 150 MB the result looked better - thanks to the redistribution of OPV. Just my personal perception, based on experience with CCE which uses its proprietary Q strategy. Actually I found more cases with flat distributions (or high bias) plus high average bitrates than I expected, maybe because on a DVD-9 there is simply enough space .....

Opposed to basic OPV, the new proposed method for redistribution eliminates the uncertainty with the file size, and in addition it will optimize the redistributed segement one by one in the subsequent multipass encode. Hence it combines the benefits of OPV for originals with a flat distribution (CBR or close to CBR) with the segment internal optimization of multipass VBR - at the expense of a slightly increased processing time.

Improving a poor original by throwing more bits than the original to some scenes is a different story, and I am very sceptic in this respect. The only case I imagine that this could make sense is in the context of sharpening filters.

[jdobbs]

Again I'd ask -- we need to look at VBR characteristics within the segments, not just the segment average bitrates to determine whether there actually is a "near CBR" source. Many discs, especially those without a lot of high action scenes will naturally have average bitrates that are close, the larger the segments, the more likely this is the case. But they may have extreme swings in bitrate at the VBR level (within the segments). Also, when using OPV for redistribution you are relying heavily on quantization to determine "quality" -- which is closely related, but isn't necessarily one-to-one.

I reiterate -- I think this is a good idea or CBR or limited VBR sources combined with limiting the upper bitrate boundary, but I'd caution against using it in most other cases. We need a little expectation managment applied here.

[Boulder]

Improving a poor original by throwing more bits than the original to some scenes is a different story, and I am very sceptic in this respect. The only case I imagine that this could make sense is in the context of sharpening filters.I would actually consider any filtering as another factor in the equation. I usually do at least light filtering, sometimes rather heavy motion compensated denoising if the source is really noisy. Many of you even use as strong filters as Deen. In these cases, I heavily suspect that the original bitrate should be forgotten as we're on a whole new playground. And I still state that whenever you don't use the exact same matrix as was used in the original, you can't say that the original bitrate really matters that much ;)

I've got some more samples encoded but probably don't have the time to check them tonight.

[Sharc]

......And I still state that whenever you don't use the exact same matrix as was used in the original, you can't say that the original bitrate really matters that much ;)


Hmm..., I am not sure if the Matrix matters that much. See for example the tests with different matrices in my post #44 or #87 (approval pending). The redistributed bitrates didn't change that much, nor did they deliver totally different swing pictures. In one example I also used the strong deen() filter. Basically all samples followed a similar redistributed swing pattern. Of course the initial Q changed (to approximate a DVD-5 size).

[Sharc]

Again I'd ask -- we need to look at VBR characteristics within the segments, not just the segment average bitrates to determine whether there actually is a "near CBR" source.


Good point. I might have to keep an eye on this as well.

[Boulder]

Hmm..., I am not sure if the Matrix matters that much. See for example the tests with different matrices in my post #44 or #87 (approval pending). The redistributed bitrates didn't change that much, nor did they deliver totally different swing pictures. In one example I also used the strong deen() filter. Basically all samples followed a similar redistributed swing pattern. Of course the initial Q changed (to approximate a DVD-5 size).But let's consider an avg of 3000kbps with the standard matrix and with Fox Home Entertainment. There's a huge difference, I'd say that FHE will look much worse at that bitrate and with a normal, 1.78:1 or 1.33:1 source. You would definitely need a higher bitrate if you are to use it.

Once again, this should be tested as well :)

[Boulder]

Here are some sample screenshots from the DVB capture re-encode. They clearly show that going above the original average bitrate is not useless. The original average bitrate was 2698kbps. I used the same in one encode, then encoded the other with an average of 4000kbps. The max bitrate was 8000kbps in both cases. The same matrix is used in all three clips. They are once again taken from field-separated video as the original is interlaced.

Original:
http://img168.imageshack.us/img168/4579/caleorg1fq9.th.png (http://img168.imageshack.us/my.php?image=caleorg1fq9.png)
Encoded at the same average:
http://img508.imageshack.us/img508/9597/calelow1ja1.th.png (http://img508.imageshack.us/my.php?image=calelow1ja1.png)
Encoded at an average of 4000kbps:
http://img253.imageshack.us/img253/6735/calehi1sf1.th.png (http://img253.imageshack.us/my.php?image=calehi1sf1.png)


Original:
http://img508.imageshack.us/img508/3271/caleorg2dj0.th.png (http://img508.imageshack.us/my.php?image=caleorg2dj0.png)
Encoded at the same average:
http://img253.imageshack.us/img253/3467/calelow2vl5.th.png (http://img253.imageshack.us/my.php?image=calelow2vl5.png)
Encoded at an average of 4000kbps:
http://img168.imageshack.us/img168/175/calehi2ig0.th.png (http://img168.imageshack.us/my.php?image=calehi2ig0.png)


Original:
http://img148.imageshack.us/img148/4199/caleorg3qc3.th.png (http://img148.imageshack.us/my.php?image=caleorg3qc3.png)
Encoded at the same average:
http://img299.imageshack.us/img299/982/calelow3ai7.th.png (http://img299.imageshack.us/my.php?image=calelow3ai7.png)
Encoded at an average of 4000kbps:
http://img168.imageshack.us/img168/1971/calehi3wp7.th.png (http://img168.imageshack.us/my.php?image=calehi3wp7.png)


I'll try to find the time to evaluate my CBR re-encode results when I get home. The bitrate variance in the redistribution is very high, ranging from 713kbps to 5189kbps. It'll still look like crap though as the average bitrate is way too low. If I was to do a real backup, it would be on a DVD9 in this case.

[G_M_C]

Here are some sample screenshots from the DVB capture re-encode. They clearly show that going above the original average bitrate is not useless. [...]

Let's not forget that there are simpler reasons when going above the original bitrate is not useless. Think for instance when you use the filter-editor, and put a slight sharpener in there (or produce something much more complicated). The script will then produce more details, and a higher bitrate could be called for. Especially when you do something like that, and also choose one of the custom matrices that are available a higher bitrate than the original is called for.

(or when you use DVD-RB as a "mediator-tool" like me; I use it to generate scrips, and than change source to the resised-to-DVD-size HD DVD/BR version. My backups than need more bitrate for shure :P. Anyway, when you can align the two versions, you'll end up with a better backup, than the original ever was. And YES it works ! But this routine of mine is not the reason why DVD-RB should change ;) )

[~bT~]

Do the redistribution, then switch to OPV in RB-Opt and let it do the prediction etc.
Just wanted to let you know that the way u suggested doesn't work so well. Final output is over/under sized.
Doing the OPV to find Q, then redist using Q found in OPV and then just entering the Q value in OPV brings it nearest to target.

[Boulder]

It should not be over- or undersized in either case, or if there is under/oversizing, it should be the same regardless of the way you do the redistribution. This is because RB-Opt should use the exact same values for available video space in the calculations. I think you should post the issue in the RB-Opt thread so that robot1 will surely notice it.

[~bT~]

It should not be over- or undersized in either case, or if there is under/oversizing, it should be the same regardless of the way you do the redistribution. This is because RB-Opt should use the exact same values for available video space in the calculations. I think you should post the issue in the RB-Opt thread so that robot1 will surely notice it.

hmmm, maybe im doing something wrong..:thanks:

[Sharc]

@boulder

To my eyes, increasing the bitrate to 4000 - ie above the original - reduced the macroblocking compared to the "recoded at the same avg" somewhat, but did not improve the already blocky original. The original is to my eyes the best quality, so it should be copied 1:1 for a real backup copy.
I assume that you set the bitrate yourself to avg 4000 / max 8000, ie it has not been set by the redistribution process. Still I would expect that the macroblocks in the original would drive the redistribution to similar high bitrates with similar results. I feel it would be better to limit the redistribution to not exceed the original or just to make a 1:1 copy, as the stolen bits from other scenes would probably make those scenes look unacceptably poor (?)

[Boulder]

Well, I've actually never talked about bringing back any details. I've said in many places that I intend to minimize the damage done by re-encoding. If I was to re-encode that video (a DVB capture), I would definitely not do that without some serious filtering - and that includes proper deblocking.

The problem is, like I've said many times, that the quality is not constant on many DVDs, some segments may look great at the cost of some segments looking bad. The redistribution method attempts to obtain constant quality throughout the video, no matter how long it is. Doing this I'm hoping that the stolen bitrate is not noticable where it was taken from but that it improves the segments where it is reallocated to.

[Sharc]

ok, but what irritated me a little is that the high bitrate did not bring back the quality of the original DVB capture, and I was wondering how the quality of those frames where the bits were stolen from would look like. I was mainly questioning the balance of the plus vs the minus - in this example.:)

[Boulder]

I didn't try to look for any minuses or pluses in that one, I have yet to author the DVD where that video will end up in. But as my recent samples have showed, there is a chance that the stolen bitrate will not be noticable but it can still improve the scenes where the bitrate has been raised. Seeing your bitrate graphs and mine, a big issue is probably the bias setting that has been used in the original encode. IMO it's not optimal to use a biased (i.e. flat) bitrate curve to determine the bitrate distribution.

You can never get back what you once lose in lossy encoding but you can try to get by :) I'd still say that I can make the re-encode of that clip look better than the original, blocky mess. It's not even hard as it's so bad looking ;)

[Sharc]

I didn't try to look for any minuses or pluses in that one, I have yet to author the DVD where that video will end up in. But as my recent samples have showed, there is a chance that the stolen bitrate will not be noticable but it can still improve the scenes where the bitrate has been raised.

Yes, understood.
But this brings me back to the question as to what to compare with what when exceeding the original bitrate. I think in this case we should compare with the original as a reference rather than with the re-encoded standard VBR. If exceeding the bitrate does not improve the quality of the original the best is to copy the original untouched rather than boosting the bitrate by stealing the bits from other scenes. No ciriticism on the experiment, though. Just an interpretation of the results.
Opposite, when we stay below the original bitrate with the redistribution, it makes sense to compare the redistributed clip with the standard VBR.

Seeing your bitrate graphs and mine, a big issue is probably the bias setting that has been used in the original encode.

Agree that a high bias in the original is an indicator for a potential benefit of using redistribution. Perhaps it is tempting to go for a high bias when there is ample space on a DVD-9. Means there is room to lift the average bitrate for less complex scenes. No harm for the original, but perhaps suboptimum for a subsequent linear reduction in DVD-5 backups.

[Boulder]

Yes, understood.
But this brings me back to the question as to what to compare with what when exceeding the original bitrate. I think in this case we should compare with the original as a reference rather than with the re-encoded standard VBR. If exceeding the bitrate does not improve the quality of the original the best is to copy the original untouched rather than boosting the bitrate by stealing the bits from other scenes. No ciriticism on the experiment, though. Just an interpretation of the results.
Opposite, when we stay below the original bitrate with the redistribution, it makes sense to compare the redistributed clip with the standard VBR.Unfortunately there is no way to compare but a pair of eyes :( Also the option to use the original segment would apply only when you don't filter at all, deblocking and all included. However, IMHO the cases where the average bitrate of the original source is exceeded are very rare and usually apply only with CBR and limited VBR range sources.

[Boulder]

Just an update..a quick look over the CBR re-encode shows that the redistribution method does improve the overall quality of the backup over the regular method. The disc I chose is not a really good sample as the bitrate is just too low to get a high quality backup no matter what you do. I think I'll try redoing the test on another CBR source I've got.

[Sharc]

Also the option to use the original segment would apply only when you don't filter at all, deblocking and all included.

I fully agree. Filtering opens up another dimension. It becomes even more prominent when different filters are applied to the individual cells. In such cases redistribution may almost become a must.

[Boulder]

Here's some screenshots from the CBR re-encode. The redistributed encode screenshots were taken from segments that had the highest (the first two pairs) and lowest (the last pair) average bitrate of all. The lowest one was 2588kbps and the highest one 5894kbps. The original CBR bitrate was 3725kbps (after reduction).

RB
http://img53.imageshack.us/img53/366/holmesrb1ri1.th.png (http://img53.imageshack.us/my.php?image=holmesrb1ri1.png)
Redist
http://img510.imageshack.us/img510/6310/holmesredist1mi4.th.png (http://img510.imageshack.us/my.php?image=holmesredist1mi4.png)

RB
http://img510.imageshack.us/img510/7413/holmesrb2cl2.th.png (http://img510.imageshack.us/my.php?image=holmesrb2cl2.png)
Redist
http://img53.imageshack.us/img53/328/holmesredist2vl9.th.png (http://img53.imageshack.us/my.php?image=holmesredist2vl9.png)

RB
http://img523.imageshack.us/img523/7901/holmesrb3xl3.th.png (http://img523.imageshack.us/my.php?image=holmesrb3xl3.png)
Redist
http://img510.imageshack.us/img510/114/holmesredist3tv3.th.png (http://img510.imageshack.us/my.php?image=holmesredist3tv3.png)

[~bT~]

Some of those RB snaps look ugly..

[Boulder]

The source is quite far from perfect ;) But it's a good representation of a rather sloppy DVD transfer which are unfortunately quite common.

[archaeo]

The redistributed examples look noticeably better to me. I was paying particular attention to the last landscape scene, where you'd usually expect to see some pixellating in those washed out areas. The redistributed example looks smoother, and without loss of detail, while the RB example is starting to pixellate. I also noticed that the color seems to be more saturated in the high bitrate redistributed examples.

[jdobbs]

I'll be honest with you... maybe it's my monitor -- but I'm not seeing a whole lot of difference. Maybe in the cheekbones on the "face" picture. I have doubts anyone would be able to pick out which is which in a full motion double-blind playback.

[Edit] Ok... now that I overlay them and switch back-and-forth, there definitely is some improvement. But what about scenes from the segments where the bitrate dropped? You have to be unbiased and do a tit-for-tat or it really doesn't say anything.

You say these were taken from a CBR source?

[jdobbs]

Also, can you post the average bitrate used for encoding next to each of the pictures so we can know what we're comparing? I already know a higher bitrate will look better, and I'm guessing that each of the "redist" examples were encoding at a higher bitrate. Not too sure about the last one, though, because the difference isn't really easily seen.

We also have the old problem of compressing a compressed image (into .PNG) for posting -- but I don't know how we'll ever get around that.

Are the pictures shown comparisons of I-FRAME to I-FRAME so we know they are equal as well? No I-FRAME to B-FRAME comparisons, I etc, as we know those would not be equal in quality.

I'm not trying to rag on the process, as I'm pretty sure it will improve CBR, I just want to make sure there is something substantive involved before I add it as an option.

[gurkan]

@boulder
It would be nice to see the original frames as well.
What I interpret as mosquito-noice in the rb-encode might be the actual details from the original.

Edit:
I'm especially thinking of the lowest one. How the redist frame with a lesser avg bitrate than rb can look better is beyond me.

[Boulder]

Also, can you post the average bitrate used for encoding next to each of the pictures so we can know what we're comparing? I already know a higher bitrate will look better, and I'm guessing that each of the "redist" examples were encoding at a higher bitrate. Not too sure about the last one, though, because the difference isn't really easily seen.

We also have the old problem of compressing a compressed image (into .PNG) for posting -- but I don't know how we'll ever get around that.

Are the pictures shown comparisons of I-FRAME to I-FRAME so we know they are equal as well? No I-FRAME to B-FRAME comparisons, I etc, as we know those would not be equal in quality.

I'm not trying to rag on the process, as I'm pretty sure it will improve CBR, I just want to make sure there is something substantive involved before I add it as an option.What comes to using PNG, it is lossless so it doesn't affect the comparison.

I took those screenshots using an Avisynth script so I can't say if they are I-, P- or B-frames. I'll try checking those frames with VDubMod and see if they correspond. At least they should be the same in those two encodes as they were done using the same encoder.

I'll have you some more samples later today, including the original frames etc. Today's Mother's Day and Finland's playing in the ice hockey WC finals so I'm quite busy already ;)

EDIT: IIRC the last pair was taken from a segment which had a lower avg bitrate than the one RB decided to use. Nevertheless, that will be doublechecked too.

[Boulder]

It was hard, but I did find some representative frames that were I-frames in the RB encode, the redistributed one and in the original.

The first comparison set is from a segment which had an average bitrate of 2683kbps when redistributed. The second comparison set is from a segment that had an average bitrate of 3192kbps after redistribution. The original RB-determined bitrate was 3725kbps (after the reduction had been calculated). The original video is CBR.

I used the parameter CPU=4 in the test encodes as I forgot to disable it in rebuilder.ini. I did a normal encode after the last tests so I had put it there. That means that in theory, the blocking should be less severe in the re-encoded frames.

Redist at avg 2683kbps, RB at avg 3725kbps, original CBR ~6593kbps
The original I-frame:
http://img246.imageshack.us/img246/4097/holmesorgiframe1bh0.th.png (http://img246.imageshack.us/my.php?image=holmesorgiframe1bh0.png)
The original I-frame with CPU=4:
http://img523.imageshack.us/img523/3610/holmesorgiframe1cpu4iv7.th.png (http://img523.imageshack.us/my.php?image=holmesorgiframe1cpu4iv7.png)
The I-frame from a one-click DVD-RB encode:
http://img182.imageshack.us/img182/6466/holmesrbiframe1ey8.th.png (http://img182.imageshack.us/my.php?image=holmesrbiframe1ey8.png)
The I-frame from a redistributed bitrate encode:
http://img246.imageshack.us/img246/9703/holmesredistiframe1ng5.th.png (http://img246.imageshack.us/my.php?image=holmesredistiframe1ng5.png)


Redist at avg 3192kbps, RB at avg 3725kbps, original CBR ~6593kbps
The original I-frame:
http://img246.imageshack.us/img246/6838/holmesorgiframe2sh4.th.png (http://img246.imageshack.us/my.php?image=holmesorgiframe2sh4.png)
The original I-frame with CPU=4:
http://img408.imageshack.us/img408/4222/holmesorgiframe2cpu4xl7.th.png (http://img408.imageshack.us/my.php?image=holmesorgiframe2cpu4xl7.png)
The I-frame from a one-click DVD-RB encode:
http://img182.imageshack.us/img182/3104/holmesrbiframe2us2.th.png (http://img182.imageshack.us/my.php?image=holmesrbiframe2us2.png)
The I-frame from a redistributed bitrate encode:
http://img179.imageshack.us/img179/7199/holmesredistiframe2si5.th.png (http://img179.imageshack.us/my.php?image=holmesredistiframe2si5.png)

Here's the bitrate distribution graph:
http://img246.imageshack.us/img246/4455/holmeslr1.th.gif (http://img246.imageshack.us/my.php?image=holmeslr1.gif)

[Sharc]

Do I understand corectly:
- Original CBR = ??? (how much?)
- RB standard reduced avg bitrate = 3725kbps (from DVD-9 to DVD-5 reduction)
- Redistributed avg cell bitrate for example 1 = 2683 kbps
- Redistributed avg cell bitrate for example 2 = 3192 kbps

Was cpu=4 also in effect for the capturing of the original I-frames?

Added:
To my eyes, the redistributed sample frames look slightly better than the standard VBR reencoded frames - i.e. less pixelation without sacrificing useful details or sharpness - although the redistributed avg cell rate was below the standard 2-pass VBR bitrate of 3725 kbps - and probably significantly (?) below the original CBR bitrate.

I am not sure however which one would look "crispier" when watching ....

[Boulder]

The original CBR bitrate is ~6593kbps. I know it sounds a lot but the source is very noisy and the video is wobbly like those tapes from the 80s tend to be.

I added the original+CPU=4 images to my previous post.

[Sharc]

Attached an exampe with constant average cell bitrate, yet not CBR, because the swing within cells is significant.

Findings:
1) Those (short) cells that got significantly boosted (containing scenes with slight blocking in the original, that's why I used cpu=4 ) were slightly less noisy compared to the standard 2-pass VBR. Very similar to the pictures in posts #153 and #161.
2) In those cells from which bits were stolen - mainly dark scenes - I couldn't discover an obvious deterioriation - some frames looked even slightly less noisy than the standard VBR encode.
3) I would say that for this example redistribution did "the lowest possible damage" to the original, although the differences to standard 2-pass VBR are subtle and - to a certain extent - a matter of personal taste as well.

The maximum bitrate was still limited by RB default according to the default DISABLE_DYNAMIC_PEAK=0.


Added:
Impact of filter "fluxsmoothST(5,5)" on redistribution.

[Boulder]

I added the bitrate graph for that Sherlock Holmes DVD. It's disc 3 from The Return of Sherlock Holmes DVD box if anyone's interested.

[Sharc]

I added to post#164 a graph showing the impact of filtering -- fluxsmoothST(5,5) -- to the redistribution, for comparison.
Interesting to note that the filtering pushed the peaks of cells 1,7,13 even higher.

[Sharc]

Just another benefit of redistribution:

Redistribution can prevent encoder saturation -- with eventual undersizing -- by stealing bits from saturated scenes and throwing these bits to more demanding cells.

Undersizing due to encoder saturation can also be avoided by enabling dynamic matrix switching (CCEAQM=1), however this requires 3-pass VBR and has the tendency to enhance noise for the frames that get an aggressive high bitrate matirx allotted.

[jdobbs]

Maybe I'm missing something, but, why do you believe "CCEAQM=1" require 3 passes? I use it with two passes all the time.

[Sharc]

Maybe I am wrong, but I thought that there is a difference between CCE SP trial and CCE Basic.
CCE SP trial requires 3 passes: 1 vaf + 2 encoding for CCEAQM becoming effective. I beleve that this has been stated in a few other posts in this forum, if I remember correctly (?)

[Boulder]

Two passes with AQM often end up in sizing problems. It seems that the feature doesn't always compensate the change of matrix well enough in the first pass after the vaf file creation.

EDIT: AQM is used even when you do 2 passes (vaf + 1 pass).

[Sharc]

Thanks for clarification. Sorry for the confusion.

[jdobbs]

Interesting, I've never had a problem.

[Boulder]

It's most common with sources where you would undersize greatly without AQM. Even with AQM enabled, some undersizing usually occurs in the second pass but the third pass gets you close to the target. That's one reason why I recommend using three passes with AQM.

[jdobbs]

Hmm... I've tried on a few discs that undersized without it -- and they always seem to size ok for me with CCEAQM=1 set, and I rarely ever do more than 2 passes (VAF + 1). I guess it depends on the circumstances.

[Sharc]

Hmm... I've tried on a few discs that undersized without it -- and they always seem to size ok for me with CCEAQM=1 set, ......

Would be interesting to see if redistribution -- without AQM -- would hit the target size as well for these discs.
I am a little sceptic when AQM kicks in sharper matrices for the reencodes than those used in the original disc. It may just enhance the noise.

Added:
..... but this brings me back to my old wish that Rebuilder should adapt its matrix selection -- as defined in the Options -- according to the actual redistributed or tweaked cell bitrates.
I am not sure however how to solve the priority issue between settings done by RB-Opt (as "CCE settings") and selection made by Rebuilder based on bitrate....

[Sharc]

Here a comparison of 2 different encoders: CCE and Quenc, with default matrix each. The results are very similar.

Is there an (easy) way to find out if the bitrate redistribution is primarily driven by encoding artefacts of the original disc?

[Sharc]

The question as to what extent mpeg artefacts of the original disc would influence the redistribution is still open, IMHO. There have been concerns that the bitrate of the redistribution is mainly driven by the mpeg imperfections of the original encode.
I made the following test to provide some evidence that this is most probably not the case: I did a second redistribution from the redistributed backup and compared the results.
1) Original disc
2) Standard 2-pass VBR
3) Redistributed backup of 1)
4) 2nd redistribution (= redistribution of 3)

Because curves 3 and 4 are much the same I would conclude that the redistribution is not driven by encoding artefacts. If artefacts would be the main drivers for redistribution I would expect significant differences between curves 3 and 4.

[Sharc]

Let's not forget that there are simpler reasons when going above the original bitrate is not useless. Think for instance when you use the filter-editor, and put a slight sharpener in there (or produce something much more complicated). The script will then produce more details, and a higher bitrate could be called for.

I did some test with sharpening filters (SeeSaw) and softening filters (fluxsmooth, deen). To my surprise the sharpening or softening did not have a dramatic impact on the basic -- ie unfiltered -- redistribution. It appears that the redistribution is mainly driven by the native characteristics of the original.

[robot1]

I did some test with sharpening filters (SeeSaw) and softening filters (fluxsmooth, deen). To my surprise the sharpening or softening did not have a dramatic impact on the basic -- ie unfiltered -- redistribution. It appears that the redistribution is mainly driven by the native characteristics of the original.
It's a great finding. I didn't expect this result.
Thank you for your tests.

[Boulder]

That is interesting indeed. I guess that the amount of motion remains more or less the same unless you blur the image heavily. That would explain why there is not much difference when scaled to the same level.

[Sharc]

That is interesting indeed. I guess that the amount of motion remains more or less the same....
Probably, yes.
Here some plots. There are some differences due to the filtering, but these are insignificant compared to the basic swing change.

[gurkan]

@sharc
I would like to conduct some testings of my own.
What program are you using to create those nice comprehensible plots?

[Sharc]

You may use any spreadsheet office application which comes with options for producing charts.

[gurkan]

Ok. Guess I can't avoid learning excel any longer. ;)

[Sharc]

Some findings regarding matrices and Q-factors, based on various tests with CCE and "normal" VBR disks.

1) Using the same matrix as in the original title, there exists a (typically low) Q factor for which the redistribution yields a profile that is fairly close to the profile of the original disk. This Q factor would -- applied to direct OPV - result in a size which is also similar to the original (DVD-9). This test indicates that the bitrate distribution of the original disk can be almost "reproduced" by selecting the original matrix and the appropriate Q factor.

2) When increasing the Q (i.e. reducing the file size towards a DVD-5), the deviation of the redistributed profile from the original profile - or downscaled according to the reduction level -- becomes more and more prominent. The swing becomes larger.

3) When selecting Q for different matrices such as to yield the same target size (e.g. DVD-5), the resulting redistribution profiles become very similar and largely independent from the matrix.

The graph shows the influence of Q on the redistribution profile, using the same matrix (FHE) as in the original title.

Q=5 approximates the (downscaled) original distribution profile closely.
Q=35 corresponds to the DVD-5 target size.
Q=64 indicates that there is no significant difference to Q=35. There seems to exist a kind of "saturation".

Finally, it seems to be most appropriate to base the redistribution on a Q-factor which approximates the target size. This Q-value is readily delivered by RB-Opt, even for a particular matrix selection (using CCE settings).

Sidenote:
If we assume that the bitrate distribution profile of the original disk is perfect and we can reproduce it as in 1), one may conclude that using higher Q factors for more compression will also produce optimum results. 1) + 2) are IMHO an indication that redistribution may produce optimum quality even for "normal" VBR disks ......... :rolleyes:

[jdobbs]

In my testing of the routines I've added to DVD-RB for redistribution I've noticed that trying to do small percentage (10% and smaller) samples on some segments really screws up the quality. For example, If you have a small segment that starts blank and then fades into a picture -- the output size at a fixed Q with sampling cuts the calculated "needed sectors" exceptionally low -- and I get blockiness in the output.

Have you guys experienced this?

[dirio49]

I have seen this.
The movie i tried had a very small cell/segment. and it was lowered too much.
Maybe if the segment is too small exlude it form the redistribution.
BTW I was testing with like 1 -3% (hey i have a very slow computer, and time is money ;) )

[Sharc]

In all my tests I selected the sample size 25%, and I have not experienced the problem with blockiness. I did observe few poor estimates of the cell bitrate for very small cells though, but luckily on the high side (waste of bits).

I therefore suggest to include a rule for the sample size which looks something like:
Sample size=Max(a;b).
a=sample size in % (e.g. 20%)
b=number of frames (e.g. 500)
a and b could be hidden settings in the rebuilder.ini

If the entire cell is less than 500 frames, all frames of the cell should be included in the sample. Very small cells (<200 frames) may be copied intact or may get the standard multipass VBR percentage reduction.

[Sharc]

In my testing of the routines I've added to DVD-RB for redistribution I've noticed that trying to do small percentage (10% and smaller) samples on some segments really screws up the quality. For example, If you have a small segment that starts blank and then fades into a picture -- the output size at a fixed Q with sampling cuts the calculated "needed sectors" exceptionally low -- and I get blockiness in the output.

Have you guys experienced this?

Another observation:
I have experienced that for a very low Q / high bitrate matrix (FHE) combination, CCE may silently abort during the cell sampling, continuing with the next cell.
The aborted cell will finally have a very low bitrate allocated to it, especially when the abortion happened at the beginning of the cell.
I don't know the reason for the silent abortion. Perhaps it is related to an excessively high bitrate which is beyond the mpeg/DVD standard. The problem is that the abortion is silent, means you will not notice it during the redistribution process, but only when finally watching the blocky cell.

As I said, I only observed this for unreasonable low Q/high bitrate matrix combinations. I have never observed it for Q factors that approximate the DVD-5 target size, irrespective of the matrix.

I mentioned this also at the end of this post:
http://forum.doom9.org/showthread.php?p=1012090#post1012090

[gurkan]

Out of curiosity:
Has anyone made a comparison between a redist. encode and a "real" unsegmented back-to-back vbr encode?

[jdobbs]

I've done a lot of analysis in which I did bitrate allocation comparisons between the standard DVD-RB dynamic bitrate allocation (the default) and methods in which you do a complete encode of the feature and reintegrate into the DVD -- and interestingly there was very little deviation in the curves.

I haven't had time to do a lot of comparisons using the redistributed methods... but there's plenty of background if you read this thread end-to-end.

[gurkan]

Was just a thought. Since unsegmented encoding and redist. encoding doesn't take the original bitrates of the segments in mind.
It would be interesting to see what differentiates these two similar processes. Especially since the q-pass only provides the bitrates to the segments.
It's still a regular vbr being done, if i'm not mistaken, when it comes to the actual encoding.

[Boulder]

If the segment length in frames is large enough, redistributed segmented encoding should yield pretty much the same results as unsegmented multipass encoding.

[gurkan]

Ok. Thanks for the info, Boulder.

[Sharc]

...... therefore suggest to include a rule for the sample size which looks something like:
Sample size=Max(a;b).
a=sample size in % (e.g. 20%)
b=number of frames (e.g. 500)
a and b could be hidden settings in the rebuilder.ini

If the entire cell is less than 500 frames, all frames of the cell should be included in the sample. Very small cells (<200 frames) may be copied intact or may get the standard multipass VBR percentage reduction.

@jdobbs:
Does the current version 1.26.0 already include a protection against erratic bitrate allocation to small cells? The default sample size is 10% which seems to be rather low for small cells -- I still set it to 25% for that reason.

[jdobbs]

Yes. It looks at the source and if it is a small segment (less than 500 frames) it automatically ups the sample size to 25% (unless it is already greater than 25%).

[Sharc]

This certainly mitigates the situation, I am however not sure if stepping up to 25% will solve the problem in all cases, as I have seen tiny cells of less than few hundred frames. 25% of say 600 frames makes only 150 frames which I still consider insufficient for a reliable bitrate estimation.

What I actually meant with my proposal is that the sample size should take the larger of either the stated percentage (for example the default 10% or whatever is specified in the rebuilder.ini), or 500 frames. So in the case of a 600 frames cell, 500 would be included in the sample. In the case of a 7000 frames cell, 10% of 7000 = 700 frames would be taken.

In cases when the cell has even less than 500 frames, all frames should be included. Tiny cells should be excluded from the redistribution and copied intact, or perhaps encoded at the standard multipass average reduction factor.
The smallest cell I have come across was 28 frames only, btw.

Maybe my 500 frames for the lower limit are pessimistic, that's why I thought this number could be included in the rebuilder.ini as the second parameter.

[jdobbs]

I can certainly modify it -- but in all the tests I did it seems to have worked well. I understood what you'd suggested, but I considered this to be a better alternative. 125 frames out of 500 should be plenty of sampling to get a good mix for distributing it -- especially when it grabs them 12 or 15 at a time throughout the entirety of the stream. Actually I'd made the decision to use 500 frames as the breakpoint even before you'd suggested that -- which was an interesting coincidence.

[Sharc]

ok, understood. Thanks for your explanation. Makes sense.

[jdobbs]

By the way, thanks for all your testing and posting the bitrate analysis. The data you showed using RB-Opt was one of the main reasons I decided to add this feature to baseline DVD-RB. My hat is also off to Boulder and (of course) robot1.

:thanks: