RB-Opt v0.23 BETA a tool to change titles bitrate, CCE parameters and AVS scripts.

[kumi]

I just edited my post with my DGIndex results. Maybe your bitrate discrepancy is caused by the way DVD-RB uses a common framerate to ensure hybrid discs come out OK.

Edit: also, you might want to check if DGIndex is misreporting with a normal, less-than-100%-Reduction segment. Just encode it straight from DVD-RB and check the reported bitrate against the vbr_brate_avg value.

Edit 2:Testing the .M2V created from an ENCODE phase might not be enough: you might have to complete REBUILD phase and then demux the .M2V. Then compare it against the original .M2V (demuxed from the source disc.)

[Sharc]

I just did your "Edit1" and found no discrepancy ....

I also made a comparison between the *.VOBs of the original and the REBUILD (assuming that DGIndex reports the video bitrate and not the muxed bitrate).

Edit: Mmmm..., I was probably using version RB-Opt v0.30 or 0.31 in this case. Could be that the reduction has not yet been implemented in the earlier versions..... Sorry for the confusion.

[robot1]

Thank you for your tests.
I've found the bug: there is a line of code in which i set the bitrate without checking for the max. I'm testing right now, and will post a new version tomorrow.

[Sharc]

Thanks robot1. No hurry. Time to go to bed now .....

[Sharc]

RB-Opt v0.32b:
The max_reduction is reset to 100 each time I restart RB-Opt, means the setting in the RBOpt.ini is not permanent. Is this so by intention?

The DVD Estimated Size under Global Options seems to be wrong (too low).

[robot1]

RB-Opt v0.32b:
The max_reduction is reset to 100 each time I restart RB-Opt, means the setting in the RBOpt.ini is not permanent. Is this so by intention?

No, it should be saved, and RB-Opt doesn't modify it in my tests.
Probably it's a dumb question, but have you edited the .ini before running RB-Opt?


The DVD Estimated Size under Global Options seems to be wrong (too low).
The DVD Estimated size in Global options is the same reported in the main window. It's an estimate, so it can't be exact, but I'd like to receive feedbacks on the final size of the encoded project.

[Sharc]

Probably it's a dumb question, but have you edited the .ini before running RB-Opt?
At least I thought so. I will doublecheck next time.....


The DVD Estimated size in Global options is the same reported in the main window.
Hmm... I don't get any size info in the main widow since v0.30 or so (?)
In the Global options I get 1609769 sectors. I noticed this "undersize" before, but the final size came out correctly(about 2245000).
Anyway, I will wait for the current encode to finalize.

[kumi]

I've noticed this discrepancy also. Three examples (these are values reported by both programs):

Encode DVDRB DVDRB RBOpt RBOpt
Type Size Compression% Est. Size Compression%
-----------------------------------------------------
VBR 4.32GB 67.9% 4.30GB 100%
RBOpt 4.18GB 65.5% 4.16GB 96.46%

VBR 4.32GB 67.9% 4.30GB 100%
RBOpt 4.26GB 66.9% 4.24GB 98.51%

VBR 4.32GB 52.7% 3.56GB 100%
RBOpt 4.32GB 52.7% 3.56GB 100%


I'm still waiting for my encodes to finish so I can say whether the size is actually misreported or not.

[Sharc]

Here my final results:
DVD-RB in Editor Window: 4.34 GB (= 4444 MB)
RB-Opt in Global Options: 3184 MB, 1609769 target sectors
NERO burning: 4448 MB

So the final size of the encode was perfect.
The error seems to be in the RB-Opt reporting.

[Sharc]

@robot1:
Suggestion: The "min_redist_frames=" parameter (or a similar one) should perhaps also be applied to the Q estimation. Otherwise very few frames may be grabbed from small VOB-IDs for the Q estimate.

[robot1]

@robot1:
Suggestion: The "min_redist_frames=" parameter (or a similar one) should perhaps also be applied to the Q estimation. Otherwise very few frames may be grabbed from small VOB-IDs for the Q estimate.
It's a good suggestion. I will correct this.

RB-Opt in Global Options: 3184 MB, 1609769 target sectors
NERO burning: 4448 MB


Could you send me the rebuilder.inf and rebuilder.ecl (and, if changed, also the .original) to my email? The email is in the readme file.


I've noticed this discrepancy also. Three examples (these are values reported by both programs):


Encode DVDRB DVDRB RBOpt RBOpt
Type Size Compression% Est. Size Compression%
-----------------------------------------------------
VBR 4.32GB 67.9% 4.30GB 100%
RBOpt 4.18GB 65.5% 4.16GB 96.46%


I'm not sure about the second row. In the first one you report the size estimated by DVD-RB and RBOpt. What is in the second one?

[kumi]

Sorry, that chart wasn't very clear. The first row are the reported sizes of a D2VAVS created straight from DVD-Rebuilder in VBR mode. The second row are the reported sizes of the same D2VAVS after redistributing bitrates in RBOpt.

I finally finished three test projects. I used a PREPARE phase in DVD-RB, then RBOpt to redistribute bitrates. Then I recorded the estimated sizes in DVD-RB and RBOpt.
Estimated Estimated Actual
Size in Size in Size of
DVD-RB RBOpt Output
-------------------------------
4.18GB 4.16GB 4.18GB
4.26GB 4.24GB 4.26GB
4.32GB 3.56GB 4.32GB

It seems that DVD-RB reports final sizes more accurately than RBOpt, in line with what Sharc was experiencing.





Something I'm regularly encountering with redistribution, in ProCoder and HC projects, is a chronic undersizing of the output. In the above table, you can the size of the first project was 4.18GB. I've even seen one ProCoder project come out to 4.07GB.

In none of these cases was the encoder saturated, so I guess the reason is the lack of granularity of the Q scale? ProCoder CQ seems to be especially flaky in this regard.

Since you're on a roll, robot1, I would like to request a fix for the undersizing issue ;) Maybe RBOpt could do something like this:

1/ Before redistribution, RBOpt estimates size. For example: 4.32GB.
2/ User initiates redistribution on some segments.
3/ RBOpt calculates the final output size is now less than 4.32GB.
4/ RBOpt increases the redistributed segments' bitrates by a certain percentage, to bring the estimated size back to 4.32GB.

[Sharc]

@kumi:
The reason for my "undersizing" could be traced to my preprocessing: Removing of Extras, and 25% reduction on the remaining Extras which RB-Opt didn't account for correctly.
A cosmetic reporting problem only. as it did not affect the final size of the encode. Robot1 is going to fix it.

[kumi]

Yes I understand. For me, RBOpt reports too-low "Estimated Size" values with 00% Steal Space settings, and no preprocessing. But yeah, you're right: it doesn't affect actual output size.

But apart from that cosmetic problem, there is the other undersizing issue in my last post, that does result in actual undersized output. Have you ever seen one of your redistributed projects come out undersized?

[robot1]

The size estimation routine in RB-Opt was written many versions ago, and it should work well with the free version of DVD-RB.
Nowdays, with DVd-RB Pro, it's clear that the size estimation is worse than the one in DVD-RB (I'm working to fix it), but, as mentioned by Shark, it's an'estetical problem.
The internal routines don't take that size in account, but the Total Video Size %, which should always be 100 and is related to the DVD-RB calculated output.

The undersizing problem could be caused by the reduction going over 100% in redistribution.
Could you check in Rebuilder.inf if there are segments with Src_Video_Sectors lower than Video_Sectors?
If this case applies, the new version should take care of that problem.

[Boulder]

Would it be possible to have the CQ_BFACTOR and CQ_PFACTOR options in the redistribution phase? It would be useful IMHO, since the normal 2-pass encode usually shows that I- and P-frame quantization is nearly the same while the B-frames have a somewhat higher quantizer, the multiplier being usually 1.5-2.0.

[robot1]

Would it be possible to have the CQ_BFACTOR and CQ_PFACTOR options in the redistribution phase? It would be useful IMHO, since the normal 2-pass encode usually shows that I- and P-frame quantization is nearly the same while the B-frames have a somewhat higher quantizer, the multiplier being usually 1.5-2.0.No problem to add those parameters.
What should be the default values, according to your experience?

[Boulder]

I would probably use 1.05 for PFACTOR and 1.6 for BFACTOR, they are close to what I've seen lately. The actual ratio depends a lot on the material but I'd say these ones are from an average encode.

[kumi]

I have 2 requests for future RBOpt versions, I hope you can consider them:


First:
Add a no_reencode_threshold directive to rbopt.ini. So, for example, if rbopt.ini contains no_reencode_threshold=95, any segment with a calculated redistributed bitrate of 95% or more would have Action=3 set in REBUILDER.INF. no_reencode_threshold=0 would disable this feature.


Second:
Provide a way to export (and import) a project's segment bitrates. I ask for this because calculating redistribution with ProCoder 2 is a dicey affair. Sometimes the redistributed bitrate values are very different to those calculated with HC or CCE (or QuEnc, apparently (http://forum.doom9.org/showthread.php?p=1018585#post1018585)). It would be helpful if we could redistribute bitrates with HC/CCE/QuEnc, change the DVD-Rebuilder project type to ProCoder, and then apply the previously calculated values with RBOpt. A possible workflow would be:

1/ Run a PREPARE phase (VBR) in HC Mode.
2/ Redistribute bitrates in RBOpt.
3/ Use the new "Export bitrates" RBOpt function.
4/ Switch the DVDRB Mode to ProCoder, and re-run PREPARE phase.
5/ Use the new "Import bitrates" RBOpt function with the file created from step 3/.

AFAIK, you could probably pull this off by recording just four variables from the .ECL/.INF files:

vbr_brate_avg
Action
Reduction
Video_Sectors

[Fishman0919]

I have 2 requests for future RBOpt versions, I hope you can consider them:


First:
Add a no_reencode_threshold directive to rbopt.ini. So, for example, if rbopt.ini contains no_reencode_threshold=95, any segment with a calculated redistributed bitrate of 95% or more would have Action=3 set in REBUILDER.INF. no_reencode_threshold=0 would disable this feature.


Second:
Provide a way to export (and import) a project's segment bitrates. I ask for this because calculating redistribution with ProCoder 2 is a dicey affair. Sometimes the redistributed bitrate values are very different to those calculated with HC or CCE (or QuEnc, apparently (http://forum.doom9.org/showthread.php?p=1018585#post1018585)). It would be helpful if we could redistribute bitrates with HC/CCE/QuEnc, change the DVD-Rebuilder project type to ProCoder, and then apply the previously calculated values with RBOpt. A possible workflow would be:

1/ Run a PREPARE phase (VBR) in HC Mode.
2/ Redistribute bitrates in RBOpt.
3/ Use the new "Export bitrates" RBOpt function.
4/ Switch the DVDRB Mode to ProCoder, and re-run PREPARE phase.
5/ Use the new "Import bitrates" RBOpt function with the file created from step 3/.

AFAIK, you could probably pull this off by recording just four variables from the .ECL/.INF files:

vbr_brate_avg
Action
Reduction
Video_Sectors

Correct me if I'm wrong... but one of big benefits to redistribution is using the info provided by the encoder for that movie... see how that encoder deal with the video... if you are using HC,QuEnc or CCE to get the info. Who's to say that would be how Procoder would deal with it. Would it real be any benefit?

Just my 2 cents

[kumi]

I'm asking for the second feature because I get wierd filesizes sometimes in ProCoder CQ mode. My comparisons with HC and CCE on a few redistributions have shown ProCoder (@ Q=1) always results in wilder bitrate swings. I can live with that, and perhaps that's just the optimal configuration for ProCoder. But in one my projects, a segment resulted in an extremely low bitrate; completely different from HC and CCE. And sure enough, it looked like crap. I'll make sure and post an example next time I encounter this.

I just don't trust CQ mode. I would rather live with marginally suboptimal bitrate distributions, that occasionally encounter a segment with extremely high or low bitrates.

[Boulder]

It shouldn't matter whether you use CCE, HC or QuEnc to do the redistribution as Sharc had noticed in his tests. In fact, when I do the redistribution manually (when dealing with my DVB captures), I use CCE for doing the redistribution pass and then HC for the final encode. The reason why I do this is that CCE is somewhat faster which will save me some time.

[robot1]

RB-Opt v. 0.33 beta (http://www.savefile.com/files/843117)

Changelog:
Better Size estimation with DVD-RB Pro
Fixed the max_reduction setting in RBOpt.ini
Added HC_PFACTOR (default 1.00 - suggested 1.05) and HC_BFACTOR (default 1.00 - suggested 1.6) options for HC through RBOpt.ini
Used the min_redist_sample value also in Q search phase (to avoid bad predictions on very small VobID)

[robot1]

I have 2 requests for future RBOpt versions, I hope you can consider them:

First:
Add a no_reencode_threshold directive to rbopt.ini. So, for example, if rbopt.ini contains no_reencode_threshold=95, any segment with a calculated redistributed bitrate of 95% or more would have Action=3 set in REBUILDER.INF. no_reencode_threshold=0 would disable this feature.

I could add a no_reencode_threshold, acting in this way:
if the segment reduction is over the threshold, and there are no filters (the .avs is standard), I could set it as No reencode in the rebuilder.inf


Second:
Provide a way to export (and import) a project's segment bitrates. I ask for this because calculating redistribution with ProCoder 2 is a dicey affair. Sometimes the redistributed bitrate values are very different to those calculated with HC or CCE (or QuEnc, apparently (http://forum.doom9.org/showthread.php?p=1018585#post1018585)). It would be helpful if we could redistribute bitrates with HC/CCE/QuEnc, change the DVD-Rebuilder project type to ProCoder, and then apply the previously calculated values with RBOpt. A possible workflow would be:

1/ Run a PREPARE phase (VBR) in HC Mode.
2/ Redistribute bitrates in RBOpt.
3/ Use the new "Export bitrates" RBOpt function.
4/ Switch the DVDRB Mode to ProCoder, and re-run PREPARE phase.
5/ Use the new "Import bitrates" RBOpt function with the file created from step 3/.

AFAIK, you could probably pull this off by recording just four variables from the .ECL/.INF files:

vbr_brate_avg
Action
Reduction
Video_Sectors
It's a complex process. I'd use a
redistribuite_with_HC
(or redistribuite_with_QuEnc)
switch in RBOpt.ini
If the switch is on, and the encoder is CCE Basic or Procoder (and possibly QuEnc and AQE), it uses HC for redistribution.
I'd use HC because I think it's CQ mode is better than the QuEnc one.
Feedback welcome.

[kumi]

I could add a no_reencode_threshold, acting in this way:
if the segment reduction is over the threshold, and there are no filters (the .avs is standard), I could set it as No reencode in the rebuilder.inf

Hmm, my DVD-RB Filter Editor is always populated with #comments and P:, I:, E:, F: conditionals. Not all of the PREPARE phase .avs files would have filters applied, but they would still have extra lines added to them. If there's no work-around for that it's no big deal, I think what you suggested would be fail-safe.



I'd use a redistribute_with_HC (or redistribute_with_QuEnc) switch in RBOpt.ini

I hesitated to ask for something like that, since I thought it would cause you too much work to switch all the different parameters in the .ECL/.INF files.

I also thought of asking for RBOpt to automatically "remember" the last set of bitrates after clicking "Save Settings". And then having an additional function to "Restore Last Bitrates". IMHO that might be more flexible than an rbopt.ini setting.

[kumi]

A silly question: in the OPV Prediction window, what is the purpose of the "Check Quality" and "Save" buttons? They always seem to be grayed out and inactive.

Oh, and what is the "Safety Margin" for? :stupid:

[Sharc]

Check Quality:
You can watch the encoded sample and then decide to go ahead - or not - with the OPV encode.

edit: ... and did you enter the path for the Media Player in the Setup?

Safety Margin:
You can set it to 1% or so if you are afraid of possible OPV oversizing :devil:

[Sharc]

.... Add a no_reencode_threshold directive to rbopt.ini.....
Just for my understanding: Is there another reason for this apart from saving time?

[kumi]

Ahh... I forgot about the media player path setup. Thanks.


Is there another reason for this apart from saving time?

Well, that is my main concern. Especially if robot1 can code it in a manner that avoids the tedious process of adjusting a bunch of segment bitrates downwards to meet the TargetSectors value after nudging certain segments up to 100% (i.e. no re-encode.)

[robot1]

Hmm, my DVD-RB Filter Editor is always populated with #comments and P:, I:, E:, F: conditionals. Not all of the PREPARE phase .avs files would have filters applied, but they would still have extra lines added to them. If there's no work-around for that it's no big deal, I think what you suggested would be fail-safe.
It's easy to ignore comments.
A standard .avs should have only:
comments
loadplugins
mpeg2source
trim
converttoyuy2
AudioDub(BlankClip())
If there are other filters, than the clip has to be encoded.




I'd use a redistribute_with_HC (or redistribute_with_QuEnc) switch in RBOpt.ini

I hesitated to ask for something like that, since I thought it would cause you too much work to switch all the different parameters in the .ECL/.INF files.

I also thought of asking for RBOpt to automatically "remember" the last set of bitrates after clicking "Save Settings". And then having an additional function to "Restore Last Bitrates". IMHO that might be more flexible than an rbopt.ini setting.
It's much easier my solution... really simple to implement (and I think solves the problem with the encoders unable to perform a CQ encoding).


Well, that is my main concern. Especially if robot1 can code it in a manner that avoids the tedious process of adjusting a bunch of segment bitrates downwards to meet the TargetSectors value after nudging certain segments up to 100% (i.e. no re-encode.)
(When there are no bugs) the program always adapts the size of the other segments (set as "Autosize"), to meet the TargetSectors.

[kumi]

You're the boss :) Any way you choose to implement redistribute_with_whatever would be very much appreciated. Thanks for listening.

Oh, call me stupid... I just discovered last night what Autosize does. :p Duhhh...

I still think a no_reencode_threshold, and/or a "No Reencode" checkbox in the GUI would be useful.

[Sharc]

In he standard script the mpeg2source() is presently not editable in order to protect it from unintentional changes.
Could it be made editable (as an option), for example to allow insertion of the cpu=x parameter?

[robot1]

In he standard script the mpeg2source() is presently not editable in order to protect it from unintentional changes.
Could it be made editable (as an option), for example to allow insertion of the cpu=x parameter?
Yes, I had the same issue.
Will be in next version.

[Sharc]

Thanks, robot1.
It has really been your RB-Opt which has encouraged and enabled me to do all the tests about redistribution and eventually to prove to myself and demonstrate to others the benefit of Boulder's suggestion. :)
:thanks:

[dynamis]

silly question, but i gotta ask! does this mean that with Bitrate Redistribution i could do less vbr passes, say 2, instead of 3? :) my thinking is some of the calculations of how much bitrate to put where has been done already. or does Bitrate Allocation just determine how much to allow for a particular segment to keep video quality consistent across all segments?

also, since i never seem to know completely how OPV and multi-pass work, i will ask: does it matter that Bitrate Redistribution cannot be done with an OPV encode? i care more about quality than filesize, but if i can save some time..... should multi-pass with Bitrate Redistribution give me better results than OPV in most cases?

thanks robot1, Boulder, Sharc, and others for all your work to make my backups look better!

[Boulder]

If CCE's adaptive quant matrices feature is enabled, I would use 3 passes, otherwise two. The redistribution feature only redistributes the bits between the different segments, not inside them.

Doing an OPV encode is pretty much the same as doing the redistribution followed by a multipass VBR encode. The difference is that the resulting filesize is accurate with the redistribution, which is usually not the case in OPV encodes.

[stereo]

@Boulder

Do you mean that if CCEAQM=0, you would use the initial .vaf pass + 2 passes, meaning 3 passes altogether?

[Boulder]

If CCEAQM=0, I would do vaf+1 pass so the total is two passes. If CCEAQM=1, then vaf+2 passes=3 passes.

[stereo]

OK, thanks, Boulder.

I normally do 1+3 vbr passes (I almost never use CCEAQM=1), but last night I was playing around with RB-Opt 0.33 (just learning how to use the new features), and like dynamis I was wondering if my normal 1+3 passes hadn't become somewhat overkill.

[stereo]

Sorry, this is probably a stupid question:

After doing the initial linking of VobIDs in the main (first) window, the Q prediction (second window) and the redistribution passes (third window), I get the message "=== Done. Press OK to save the results. Warning: original bitrates will be overwritten". I then press OK, and the redistribution window closes, sending me back to the first RB-Opt window. My question is: Should I still press "Save settings" or just press "Exit" in this first window?

[Boulder]

You should save the settings.

[robot1]

... I get the message "=== Done. Press OK to save the results. Warning: original bitrates will be overwritten". I then press OK, and the redistribution window closes, sending me back to the first RB-Opt window. My question is: Should I still press "Save settings" or just press "Exit" in this first window?
I will change the text for next version.

[stereo]

OK, thanks both of you.

I figured, I should save, and thats what I did. I just wasn't sure.

So far, the new RB-Opt seems great, but I'll have to do some more testing / viewing to see if the encodes are improved.

[dynamis]

...The redistribution feature only redistributes the bits between the different segments, not inside them...

so if the total bitrate of VTS 2 (main feature) is 3900Kbps and VTS 2 (extras) is 2400, the total average of each will still be the same, 3900 and 2400? will bitrate go from one VTS to another? i've seen menus and extras come out with higher bitrates after redistribution and started to wonder where that extra bitrate was coming from because the total avg bitrates were now higher. is that normal? is that good? :P

also i was wondering why CCEAQM=1 would need the extra pass. i just read about CCEAQM in another thread. i've never used it before, but if i understand correctly, it can provide better video quality at the cost of compatibility with certain yard sale dvd players :P i'm thinking about using this together with Bitrate Distriburion, so i would like to know how the extra pass help things. thanks.

[Boulder]

so if the total bitrate of VTS 2 (main feature) is 3900Kbps and VTS 2 (extras) is 2400, the total average of each will still be the same, 3900 and 2400? will bitrate go from one VTS to another? i've seen menus and extras come out with higher bitrates after redistribution and started to wonder where that extra bitrate was coming from because the total avg bitrates were now higher. is that normal? is that good? :PIf both VTSs are included in the same redistribution pass, the average bitrates will change. If they are redistributed separately, the average bitrates will remain the same.
also i was wondering why CCEAQM=1 would need the extra pass. i just read about CCEAQM in another thread. i've never used it before, but if i understand correctly, it can provide better video quality at the cost of compatibility with certain yard sale dvd players :P i'm thinking about using this together with Bitrate Distriburion, so i would like to know how the extra pass help things. thanks.IMHO, the adaptive quant matrices feature needs the extra pass because the first actual pass after the vaf file creation will have a lot of changes in the matrices. I do not think that CCE can do an optimal distribution of matrices in the first pass and do an optimal allocation of bits at the same time. This is why I feel it should be given the second VBR pass - to do some fine adjustments. Also, in some cases when the compressibility of the video is very high, using only one VBR pass causes undersizing which is fixed by a second VBR pass.

[dynamis]

wow.. thanks Boulder! one last question, if i may, before i try some encodes. in your opinion, should menus or extras be redistributed? i'm thinking it depends... i've only tried calculations on a few movies, but so far i've kinda noticed that movies with lower Q factors seem to give bits to menus and extras and those with higher Q factors seem to take bits away. but then again... i rarely watch extras. i just throw them in if i have room and usually shrink them down with dvd shrink leaving the movie untouched. thank you.

[Boulder]

I usually uncheck Autosized for all menus and lower their avg bitrates and then do a redistribution pass with all menu-related items selected - or simply use MenuShrink on them.

Using extras in the same redistribution pass as the main movie really depends on your decision, if you rarely watch them, I don't think that it makes much sense to have the extras there. They usually need more bits than the main movie because they are interlaced.

[dynamis]

i will check out MenuShrink. thanks again, Boulder.

[Sharc]

IMHO, the adaptive quant matrices feature needs the extra pass because the first actual pass after the vaf file creation will have a lot of changes in the matrices......

I have seen frequent changes of the matrix in standard multipass VBR mode with CCEAQM=1. Did you or anyone experience frequent changes happening in redistribution mode as well? I wonder because the redistribution sets the segment bitrates according the the "Q-plus-matrix combo" and hence I wouldn't expect frequent changes happening during the encode afterwards.
I have so far not been using CCEAQM=1 in redustribution mode at all because I felt that changing the matrix dynamically would somehow conflict with the constant Q idea. So I doubt if CCEAQM=1 will have any benefit- but as I said I have not made any comparative tests.

[Boulder]

The adaptive quant matrices are not used in OPV mode, and this applies to HC's LUMGAIN option as well. However, I don't think it is a problem to use either option in the multipass encode, I've been using LUMGAIN 4 all the time.

The redistribution just makes the quality equal, the encoder can then decide what to do with the bits. For example, HC's LUMGAIN mode lowers the quant matrices coefficients in darker GOPs whereas CCE apparently makes its decisions based on the amount of motion (not sure though).