RB v1.26 bitrate redistribution

[Sharc]

As the principal question is still when to go for redistribution in case of "regular VBR disks", I personally tend towards saying "always", because my expereince with redistribution has been "equal or better than regular multipass", in particular since the handling of tiny cells has been addressed.

I attach here an example (WTC) where the swing increase by redistribution is actually huge. I was most suspicious about the qaulity of cell 9, 10 with bitrates in the range of 1000 kbps using CCE at this extraordinary low bitrate. To my surprise I didn't find any noticeable deficiencies in these (dark) scenes when watching the encode. However a loss of sharpness/details is noticeable when making a zoomed picture-by-picture comparison with the original (I did not compare with a standard VBR re-encode though).

I used one single matrix (similar to CCE default) throughout the process i.e. for Q estimation, redistribution pass, and final encoding. I used RB-Opt for redistribution. I will repeat with RB 1.26 but I wouldn't expect meaningful differences.

Noteworthy perhaps is the matrix of the original, which I consider to be quite exceptional:

Intra Luma and Chroma Matrix
8 5 7 9 11 13 14 17
5 6 8 11 13 14 17 18
7 9 10 11 14 17 17 19
9 10 13 13 14 17 18 127
10 13 13 14 16 17 127 127
13 13 14 16 17 20 127 127
13 13 14 17 19 23 127 127
13 14 17 19 127 127 127 127

NonIntra Luma and Chroma Matrix
8 5 7 9 10 10 11 11
5 6 9 10 10 11 11 12
7 9 10 10 11 11 12 12
9 10 10 11 11 12 13 127
10 10 11 11 12 13 127 127
10 11 11 12 13 13 127 127
11 11 12 13 13 14 127 127
11 12 12 13 127 127 127 127

[Sharc]

..... I'm wondering if it may be best to force the "Main Matrix" matrix across the board when a redistribution was done.........

I have doubts as to whether the matrix originally selected by DVD-RB should be used in the redistribution with RB-Opt either, because there are some significant bitrate differences. I'm not sure if Robot1 changes it, but I don't think so.

Anyway, I found that the redistribution profile becomes widely independent from any selected matrix as long as we set the "Base_Q" to a value that would yield a DVD-5 target size in OPV for the given matrix.

@jdobbs:
I understand that the Base_Q -- without iteration -- is used for the redistribution pass, correct? In my tests this Base_Q normally deviated significantly from the OPV value that would yield a DVD-5, even for the standard CCE default matrix. Perhaps something to look into.

[jdobbs]

Base_Q woud not be the final Q value, it would be the start point. In order to get the final figure I would have to do all the prediction passes -- and that just isn't justified in this circumstance, because the difference in redistribution would be negligible.

[kumi]

I think DVD-RB 1.26 redistribution with HC 0.21 has a problem when calculating BaseQ for matrices other than the encoder default. The calculated BaseQ for a redistribution using a high-bitrate matrix (the one last posted by Sharc) is 21, but the calculated BaseQ in RBOpt for that same matrix is 43. The curve seems to return to normal in DVD-RB when I use the HC default MPEG matrix:

http://img508.imageshack.us/img508/3430/hc021redistdvdrbvsrboptxi2.png

EDIT: I see now that archaeo already posted about this problem.

[Sharc]

As reported earlier in this thread and in the other thread about redistribution, DVD-RB and RB-Opt use different methods for the estimation of the Q for redistribution:
- DVD-RB uses the initial Q estimate (= Base_Q) of the OPV method without further iteration.
- RB-Opt uses the final Q value after some iterations which takes more time but ends with the Q which corresponds to the DVD-5 target size for the applied matrix.

Depending on the matrix the 2 methods can deliver substantially different values for the Q, which in turn may result in substantially different redistribution profiles, typically for - but not limited to - the case of high bitrate matrices. For example a low Q can even produce a redistribution profile which is very similar to the standard multipass profile. There are however cases, e.g. for "average" matrices, where the resulting profiles are similar for both methods because there exists a kind of "saturation" for the Q where the distribution profile becomes insensitive of the Q.

I personally would prefer if DVD-RB would have the option of using the "final Q" after few iterations rather than the Base_Q.
May be this is something for optimization.

Added:
Anyway, in your example with an average bitrate in the order of 5000 kbps I don't expect you will get any benefit from redistribution, whatever method/profile you apply.

[kumi]

As reported earlier in this thread and in the other thread about redistribution, DVD-RB and RB-Opt use different methods for the estimation of the Q for redistribution

I see, so you're saying that the deviations between "DVDRB Hi BaseQ 21" and "DVDRB MPEG BaseQ 21" in my graph are a result of the DVDRB estimation routines, and not a bug?



I personally would prefer if DVD-RB would have the option of using the "final Q" after few iterations rather than the Base_Q.

Is there any possible benefit from not performing additional iterations, besides greater speed?

[Sharc]

I see, so you're saying that the deviations between "DVDRB Hi BaseQ 21" and "DVDRB MPEG BaseQ 21" in my graph are a result of the DVDRB estimation routines, and not a bug?
Exactly, that's what I believe. As a test you may set the Q of RB-Opt to the DVD-RB value and you will most likely get the same profile with RB-Opt as with DVD-RB.

Is there any possible benefit from not performing additional iterations, besides greater speed
In my opinion no, but may be I don't see the full picture.

Added: I should mention that my experience is with CCE rather than HC.

[Sharc]

When the redistributed bitrate of a segment exceeds the bitrate of the original disk, the segment is marked for "no reencode".
Will the filters defined in the Filter Editor be skipped in this case?

Edit:
Sorry, I just noticed that the similar question has already been put in this thread (in the context of possible undersizing):
http://forum.doom9.org/showthread.php?p=1016359#post1016359

[jdobbs]

[Removed]

[jdobbs]

After much testing I'm satisfied that this method doesn't hurt the picture, and can improve it under some circumstances. The only bug report I have is related to ILVU, and I've corrected that for the next version -- so I'm going to remove it from beta status.

:)

[acrespo]

Can I use redistibute encode with CCE Basic?
What's happen if I enable redistribute encode and use CCE Basic?
I asking this because as I know, CCE Basic don't support OPV mode.

[Boulder]

@jdobbs: regarding CCE Basic, would it be a reasonable option to allow using one encoder for redistribution and other for the final encode? In this case, HC would do the redistribution phase and CCE Basic would then do the final phase.

[jdobbs]

I'd have to do some comparisons of the bitrate distribution for CCE and HC. One would hope they'd be close -- but I haven't tried it to see.

[kumi]

I've compared CCE, HC and ProCoder on 4 discs and the redistributed bitrates of HC and CCE were essentially identical in all four cases. This was with RBOpt, not DVD-Rebuilder, mind you.

EDIT: I can see the value for ProCoder projects, if jdobbs ever adds support for that. The CQ scale or whatever it is in ProCoder causes some pretty wonky redistributions, at times.

[Sharc]

I did a comparison between CCE and QuEnc. The redistribution profiles were virtually identical. I used the default matrix.
See this post:
http://forum.doom9.org/showthread.php?p=1005914#post1005914

[laserfan]

I need someone to tell me in simple terms what this is about and when/how I would use it.

I thought it had something to do with converting DVDs that were encoded at a set-and-forget constant bitrate, and so DVD-RB should leave certain scenes alone to not detract further from the borderline they were encoded at.

Am I close? If not, what the heck is this about? How does one determine which DVDs to use it with, or does one just select the option and leave it on all the time?

I don't THINK I'm stupid, but I have not followed this discussion closely enough to appreciate it... :confused:

[kumi]

Here's how I understand it:

Redistribution re-calculates the optimal bitrates on a segment-to-segment basis, using a constant Q factor as the base. And since there is a (admittedly loose) correlation between "Q" and perceived visual quality, an ideal redistribution will result in a DVD5 where all segments look equally "good" to your eyes.

Now, the DVD studio that mastered your DVD9 has a larger pool of bits to work with than you do. At DVD9 sizes, swings in the video's Q affect perceived visual quality less than at DVD5 sizes. Because of this, mirroring the original bitrate distribution with a VBR re-encode might result in greater perceived visual quality inconsistencies from segment to segment. Bitrate redistribution "equalizes" these inconsistencies, (ideally) resulting in a output that is at maximum quality, throughout the entire movie.

I really can't think of any cases where redistribution could actually be worse than VBR. Maybe someone else can.

[jdobbs]

On typical sources I personally can't tell the difference, on or off. But I guess the bottom line is: I haven't seen it hurt the output on any sources yet -- but I have seen it help. On CBR or limited VBR sources it seems to make a significant difference. So I use it on all my encodes now.

As to whether you add the "redist_All=1" parameter, that's still debatable, because it could conceivably steal bits from the main movie in order to bring the visual quality of the extras to the same level as the feature.

[blutach]

Just a quick question - does multiple encoder processes not work with the "redistribution pass"? Just trying it for the first time (albeit on a source which would not be improved 1 iota) and it seems to take a while using a single processor.

Regards

[jdobbs]

No. The way they are called is different. Redistribution will only do one segment at a time.

[laserfan]

I just tried my first backup w/1.26.1, BD selected, and Full Movie/Menus/Extras and have some questions. At the end of the Prepare stage, a file is written called REDISTRIBUTION.TXT wherein it appears to show before & after bitrates by segment. But following the Prepare stage, I used the Editor to blank a bunch of Extras, and then re-assign the saved space to the Feature.

It all appears to have worked perfectly i.e. it exactly hit the DVD-5 target size but I wonder: After I blank some extras and give the feature more space, doesn't that render the REDISTRIBUTION.TXT information invalid? Or does RB just change all the target bitrates based on the ratios in the original analysis? Or am I misunderstanding the process (very possible!). :) :confused:

[jdobbs]

It will reallocate the recovered space just like it always did -- but the new distribution will be used as the baseline. So the benefit of the redistribution is retained.

[jdobbs]

[Remove - wrong thread]

[laserfan]

...the benefit of the redistribution is retained.:thanks:

[jdobbs]

Just as an add-on note to substantiate some of the experiences reported on redistribution. The list below is the result of an encode I did on an old 1960's series recently. Notice that the distribution for all the segments was virtually identical in the original encode... probably fine when you have a lot of bandwidth (DVD-9) to play with. But running redistribution showed a significant difference in the required bitrate -- and the picture is visibly better.

This is an example of where this method shines. I've decided that I'm personally going to just leave this option enabled all the time -- that way I don't have to do any kind of preliminary analysis... just click-and-go.

Hat's off to Boulder for exploring this concept, and to Sharc and others for testing/proving... even though I was originally a little resistant to the idea, I've very pleased with the result.

:thanks:

BITRATE APPLIED TO FEATURE SEGMENTS (SEGMENT ORIGINAL REDISTRIBUTED)
(Created using a 10% sample)

V01000000001001 1,799 Kbs 2,331 Kbs
V01000100001002 1,798 Kbs 1,845 Kbs
V01000200001003 1,798 Kbs 2,391 Kbs
V01000300001004 1,798 Kbs 2,008 Kbs
V01000400001005 1,798 Kbs 1,949 Kbs
V02000000001001 1,798 Kbs 1,999 Kbs
V02000100001002 1,798 Kbs 2,404 Kbs
V02000200001003 1,798 Kbs 2,439 Kbs
V02000300001004 1,798 Kbs 1,647 Kbs
V02000400001005 1,797 Kbs 2,296 Kbs
V03000000001001 1,798 Kbs 2,117 Kbs
V03000100001002 1,798 Kbs 1,800 Kbs
V03000200001003 1,798 Kbs 1,115 Kbs
V03000300001004 1,798 Kbs 1,456 Kbs
V03000400001005 1,798 Kbs 1,091 Kbs
V04000000001001 1,799 Kbs 1,885 Kbs
V04000100001002 1,797 Kbs 1,390 Kbs
V04000200001003 1,798 Kbs 2,085 Kbs
V04000300001004 1,798 Kbs 2,046 Kbs
V04000400001005 1,798 Kbs 1,222 Kbs

[archaeo]

Hat's off to Boulder for exploring this concept, and to Sharc and others for testing/proving... even though I was originally a little resistant to the idea, I've very pleased with the result.

:thanks:


I second that. :)
I have found that this new method has definitely improved the CBR stuff I occasionally work with, and have noted similar improvements with a few of the 'standard fare' dvd sources.

[Boulder]

Thank you for the compliments. I'd like say a big thank you to robot1, who did the dirty work and turned the idea into actual code and made the clumsy spreadsheet process unneeded. Without his work, it would have taken a long time to prove anything :) Also thanks go to the testers, especially Sharc has been very active with his graphs and screenshots - I myself have been buried in real work most of the time so I couldn't have done everything myself.

[jdobbs]

I'd like say a big thank you to robot1, who did the dirty work and turned the idea into actual code and made the clumsy spreadsheet process unneeded. Hear, hear!

:thanks:

[laserfan]

I just wanna say thanks too for the new Sticky FAQ entry; some of us need a little extra spoon-feeding to "get with the program"!

:thanks:

[laserfan]

I so enjoyed the preceding "group hug" that I forgot to ask my question!!!?!! :) I've noticed looking at my Sopranos backups that each episode's opening sequence (always the same, lots & lots of very fast movement and quick-cutting) can look blocky as hell, particularly when the backup is from a 4-episode disc. But then once TonyS slams the door on his SUV and this week's program starts, it looks fine, just fine, with none of the blockiness from the credit sequence.

I have thought about re-doing them to remove the openings (then again, will I ever watch them...). But here's a theoretical for this group re: backing-up a disc like this leaving the credits alone: wouldn't the new BR option best be turned-OFF for such discs i.e. the option would allocate additional bits to the credit sequence (where we don't need/want it)?

I know, I should just experiment with it and maybe I will if it keeps raining like it has... ;)

[gurkan]

wouldn't the new BR option best be turned-OFF for such discs i.e. the option would allocate additional bits to the credit sequence (where we don't need/want it)?

Redistribution tries to achieve the same visual quality throughout the whole movie/series.
If you would like to have a better/worse quality than average in a specific section, do a redistribution, increase/decrease
the quality for preferred segments and then reallocate extra bits to main feature.
This might create a more even quality in the remaining parts.
Of course there's no need to do this if you're already satisfied with the regular rb-encoding.

[linx05]

But if you do a redistribution and then change the quality on segments THEN allocate the extra bits to the main feature, wouldn't all the work done in the redistribution phase be a waste of time?

[jdobbs]

Yes, so you probably don't want to do that within the segments of the feature. But you could, after redistribution, reallocate bits from extras and they would be used in a way consistent with the redistribution. You could redistribute bits from something like the credits if you wanted to... I personally wouldn't, though.

[TBL]

First of all, :thanks: to all parties involved in developing this feature!

I ran across a disc that has following information:

- Reduction Level for DVD-5: 56,7%
- Overall Bitrate : 2 550Kbs
- Space for Video : 4 002 486KB
- Redistributing using Base_Q: 41
- HIGH/LOW/TYPICAL Bitrates: 3 658/2 028/2 550 Kbs

Redistribution.txt:
BITRATE APPLIED TO FEATURE SEGMENTS (SEGMENT ORIGINAL REDISTRIBUTED)
(Created using a 10% sample)

V01000000001001 2 946 Kbs 3 058 Kbs
V01000100001002 2 881 Kbs 2 794 Kbs
V01000200001003 2 597 Kbs 2 273 Kbs
V02000000001001 2 694 Kbs 3 012 Kbs
V02000100001002 2 714 Kbs 3 234 Kbs
V02000200001003 2 278 Kbs 2 397 Kbs
V03000000001001 2 375 Kbs 2 083 Kbs
V03000100001002 2 423 Kbs 2 063 Kbs
V03000200001003 2 229 Kbs 2 028 Kbs
V04000000001001 2 949 Kbs 3 658 Kbs
V04000100001002 2 861 Kbs 3 648 Kbs
V04000200001003 2 706 Kbs 3 240 Kbs
V05000000001001 2 291 Kbs 2 116 Kbs
V05000100001002 2 335 Kbs 2 092 Kbs
V05000200001003 2 455 Kbs 2 143 Kbs
V06000000001001 2 449 Kbs 2 170 Kbs
V06000100001002 2 472 Kbs 2 178 Kbs
V06000200001003 2 346 Kbs 2 038 Kbs

It is an episodic disc (pretty small local release), the episodes are about 36 minutes long and split only into 3 parts. The average bitrate is quite low, but there isn't really anything to remove, I already shrinked the menus. The question is, should I split the episodes to smaller parts and second guess the production studio or keep the episodes as they are (which one do you think makes less damage)?

[blutach]

You've already got quite a variation between high and low. As I understand it, it may not be worth the extra time taken to run the redist in these circumstances (which you have already done).

In any event, the BR looks pretty low. I'd split the disks.

Regards

[Sharc]

You've already got quite a variation between high and low. As I understand it, it may not be worth the extra time taken to run the redist in these circumstances.

This is no reason to skip redistribution, in my experience.
You may find a high variation of the swing in the original and still get quite a different profile after redistribution, depending on the source material and the combo of the matrix and the quant.
One should also not forget that a linear reduction of the bitrate reduces the swing by the same percentage (say 65%) , the absolute motion and picture content however remain the same for the original and the re-encode.
The extra time redistribution takes is low with sample sizes of 10%, so why should one bother too much?

[blutach]

The extra time redistribution takes is low with sample sizes of 10%, so why should one bother too much?Thanks for the info Sharc. I did a couple of redists and one I can remember took 77 minutes to get thru the prepare (using HCEnc on best, FHE matrix on what appeared to be a CBR source encoded at 8mbps and I think Base Q was 20). This must not be normal. Am I missing a variable somewhere?

Regards

[Boulder]

Do you do any filtering? It might also be that buffer underflows are being fixed constantly in HC, q2 is very low so the bitrate will be very high.

[Sharc]

The 77 minutes appear long to me, however I am not yet very experienced with HC.
Perhaps you check the Redist_Percent=10 under the [Options] in rebuilder.ini.
If you should have filters applied they will also add up in time during the redistribution process.

[jdobbs]

I do 10% redistributions, and I typically see it taking an extra 5-6 minutes in PREPARE.

[blutach]

This might be my issue - there is no redist_percent in my INI file :eek: (Also, I do not use filters)

Perhaps I should add a line. 10% is considered sufficient? I saw the initial thoughts of Boulder saying 100% was best, but this view is superceded now?

I'll do another test and revert. I was pretty sure though that it wasn't encoding the lot. 77 minutes would have been way too fast for a single processor.

Regards

[jdobbs]

It should default to 10% if it isn't there... but I'll check to be sure.

My testing doesn't show a significant difference in the allocation between 10% and higher amounts. But with 100% you can be absolutely sure it is allocated exactly. I personally think 10% is plenty.

[blutach]

OK - I think I can see the issue. Just doing a prepare now on a real easy souce (fits into a DVD-5).

HC021 is struggling along at 4-5 fps!!!! :( (a normal encode gets 35 fps)

Anyone else seeing this?

Regards

[jdobbs]

I see some slowdown with HC, but not that much. It's probably because of scene detection. When running through a sampling it will in most cases determine a new scene every 12 frames. Maybe I should turn scene detection off for the redistribution passes when using HC?

I've done most of my REDISTRIBUTION testing with CCE SP.

[blutach]

I fed the AVS into HC itself and changed scene detection to off. It is going about the same speed as it does (maybe a fraction slower) when doing the 2% sampling pass of CQ mode.

EDIT: The encoder races along at 45 fps with a redist AVS in CQ mode. See encoder log attached. So, it's not the scene change detection, I don't think.

EDIT2: Turning on the VBV check in the encoder slows things down a lot! Need more experimentation but this could be the issue. I note that with VBV off, the quant is as selected. But if it is on, it is nowhere near that selected.

Regards

[jdobbs]

Yeah, I looked at scene detection -- and it doesn't seem to have an effect. I'll do some testing with the VBV.

[Sharc]

Off the records:
May be your findings will eventually help me to to a redistribution with HC at all. I reported my weird problem earlier: HC cannot write the sample file, seems to be something odd with the script. Very obscure ..... I assumed that something is possibly odd with my system, and simply gave up and restricted my tests to CCE.

Edit:
Maybe it's related to what is discussed here:
http://forum.doom9.org/showthread.php?p=1012413#post1012413

[jdobbs]

The VBV point is interesting. I tested it on one of my systems and it had no impact at all -- then on another (older) one it seemed to speed things up. The redistribution seems to be identical whether it's on or off -- so I've turned it off during redistribution (with HC) for the next release.

[laserfan]

HC021 is struggling along at 4-5 fps!!!! :( (a normal encode gets 35 fps)I have one going right now avg 33fps current 45fps. I just have "Best" for HC and all defaults.

[Boulder]

Yeah, I looked at scene detection -- and it doesn't seem to have an effect. I'll do some testing with the VBV.You'll probably want to disable scene change detection when the redist percent is not 100. Otherwise the result will not be as accurate as it could be, there'll be a scene change pretty much after every GOP. If you use autoGOP together with scene change detection and don't use 100% redistribution, the GOP structures will be far from what they would be in regular encode.

Would it also be possible to add the "disable all filtering" option and the option to choose the profile to use in HC for redistribution? The disable all filtering option should also remove any cpu=xx statement from MPEG2Source. I use the "normal" profile for redistribution in Rb-Opt as it is somewhat faster. I don't think the results differ much from "best". One might even test using "fast" - the most important thing is in any case that all segments are treated the same way.