RB-Opt v0.23 BETA a tool to change titles bitrate, CCE parameters and AVS scripts.

[Insomniak4700]

Here is the new build:
RB-Opt v. 0.29 beta (http://www.savefile.com/files/801164)


Thanks robot1! While I was trying the last version yesterday I was wondering how much of an impact not using the same Q matrix would have! I can compare the two now!

[TomBrooklyn]

Hi,
Nice work robot1. I didn't want to read through all 16 pages of this thread, but I read in the beginning that the first beta was not supporting HC. Is it usable with HC now? Regards, TomBk

[Boulder]

Hi robot1,

here are some small comments regarding using HC:

1) When doing bitrate redistribution and you choose to calculate the optimal quantizer to use, the "ignore filters" option is not considered in the prediction phase. (this probably applies to all encoders)

2) In the forementioned prediction phase, you probably should use a 15-2 GOP if the source is PAL and autoGOP is selected in DVD-RB. In NTSC, 12-2 is probably the most accurate choice.

3) In the actual redistribution phase, you should use the GOP settings from DVD-RB like before, now it looks like 12-2 is used there too.

[robot1]

Hi robot1,

here are some small comments regarding using HC:

1) When doing bitrate redistribution and you choose to calculate the optimal quantizer to use, the "ignore filters" option is not considered in the prediction phase. (this probably applies to all encoders)

2) In the forementioned prediction phase, you probably should use a 15-2 GOP if the source is PAL and autoGOP is selected in DVD-RB. In NTSC, 12-2 is probably the most accurate choice.

3) In the actual redistribution phase, you should use the GOP settings from DVD-RB like before, now it looks like 12-2 is used there too.
These adjustments will be in next version.

[robot1]

Hi,
Nice work robot1. I didn't want to read through all 16 pages of this thread, but I read in the beginning that the first beta was not supporting HC. Is it usable with HC now? Regards, TomBk
RB-Opt works with HC, but some advanced options are not available for this encoder.

[tom942]

This question goes to Boulder or anyone that can help me.

I was reading in the help of Ammon82's AutoQ that 12 was better because it is related to the framerate,and also I've seen that when HC does CQ One pass, it selects 12-2, so why is better for 15 gop than 12?. Could you tell me where can I find info related to this?.

And why is better 10 bits of DC precission than 8 or 9?. I've done several test with SP and HC, and I've seen the image improves with 10,but, why?.

Sorry for the off-topic, if so, I'll move it to other place in the forum.

Greetings.

[Boulder]

I think you should post your question as a new thread in the MPEG2 encoding forum as it's a general one and doesn't deal directly with RB-Opt.

In general, a longer GOP compresses better because there are fewer I-frames. Then again, a long GOP may affect quality because there are more frames that are "constructed" using other frames as a reference. See http://en.wikipedia.org/wiki/Video_compression_picture_types for detailed info.

I very rarely use 12-frame GOPs, only when having encoder saturation. I use autoGOP with a maximum of 15 frames 99% of the time.

I'll leave the DC precision part for someone else to answer.

[tom942]

With affect quality, you mean a bit worse?.

What do you mean with encoder saturation?.

I move it to the other forum. BTW, thank you for the answer :).

[kumi]

@Boulder:

I read somewhere that pulldowned sources need special care when choosing GOP. Can you tell me how to determine when a 12-frame GOP is _necessary_?

[~bT~]

^ this thread may answer your questions: http://forum.doom9.org/showthread.php?t=124824

[kumi]

Thank you for that link. But I'm still not sure how to tell when the output of DVD Rebuilder is going to use rff flags. Where do I look?

[jdobbs]

If the source is 23.976. DVD-RB will reimplement the same TFF/RFF sequences as the original.

[tom942]

@robot1

Could you add a log with the old and new bitrate distribution in a similar way that DVD-RB does to compare the final results?

:thanks:

[robot1]

Here is:
RB-Opt v. 0.30 (http://www.savefile.com/files/814252)

Changelog:
Redistribution can now involve multiple VOBIDs (useful for episodic discs)
Support for HC Lumgain and Autogop parameters, through settings in RBOpt.ini
In redisitribution process, if checked "disable all filters", filters are ignored also in the Q search phase
Changed the final log in the redisitribution, to compare original and redistributed values
Improved Q finding precision for HC encoder

As always, feedbacks are welcome.

[~bT~]

^ Much appreciated update. :thanks:

[tom942]

:thanks: for the update robot 1 :)

[kumi]

Thanks a lot for version 0.30, robot1 :)

I have a few questions about bitrate redistribution, my apologies for using a list format:

1/ Does HC_LUMGAIN > 0 play along nicely with RBOpt's bitrate redistribution?

2/ Will bitrate redistribution work with ProCoder projects? I know I've read somewhere that OPV mode in ProCoder is broken... but can't the redistribution be done with HC or CCE?

3/ When using filters that affect Q (sharpening, denoising etc.) is it advisable to use "disable all filters"?

4/ Sharc mentioned on page 30 of this thread that "a small sample size of 0.5% to 1% is sufficient", but the RBOpt default value is set to 20%. Is it safe to use a sample size of 1% here?

[Sharc]

Let me comment on 4/:

There are 2 different %-age settings:
a) One relates to the redistribution of the segments
b) One relates to the estimation of Q

For a) I normally select 25% because this is in relation to the number of frames of the individual segments.

For b) I normally select 0.5% to 1% because this is in relation to the number of frames of the entire movie (or all linked VOB-IDs, respectively).

Generally speaking, I select the percentages for each case such as to grab a sufficient number of frames. I consider about 500 ... 1000 frames sufficient. Therefore the 25% for a) and the 1% for b).

[robot1]

1/ Does HC_LUMGAIN > 0 play along nicely with RBOpt's bitrate redistribution?

I haven't tested it much, and I'm not sure if this setting is important in OPV encoding or only in VRB encoding. Anyway, I'd set LUMGAIN at the same value used in DVD-RB.


2/ Will bitrate redistribution work with ProCoder projects? I know I've read somewhere that OPV mode in ProCoder is broken... but can't the redistribution be done with HC or CCE?

Redistribution works with EclPRO, only if you are using ProCoder 2.
Anyway, I'd use always Q=1, and the results are a bit different from the ones you get with other encoders. Need more (visual) tests for a better answer.


3/ When using filters that affect Q (sharpening, denoising etc.) is it advisable to use "disable all filters"?
There are tests in another thread. If I remember well, the results are similiar with filters enabled or disabled. If time is not an issue, leave all filters enabled.


4/ Sharc mentioned on page 30 of this thread that "a small sample size of 0.5% to 1% is sufficient", but the RBOpt default value is set to 20%. Is it safe to use a sample size of 1% here?
See the previous post by Sharc. I would not go under 10% for redistribution.

[kumi]

Thanks, Sharc & robot1.

I haven't tested it much, and I'm not sure if this setting is importantHave you tested, then, with CCE's AQM? This and LUMGAIN both operate in a similar fashion if I'm not mistaken, so if you have experience with AQM+redistribution it's probably valid to extrapolate to LUMGAIN+redistribution.


Redistribution works with EclPRO, only if you are using ProCoder 2. Anyway, I'd use always Q=1I see, so I should use a Q factor of 1, and disable "Calculate optimal Q"? I'd be happy to do some ProCoder 2 testing, if I can determine the right way to go about it.

[Boulder]

If I remember correctly, CCE's adaptive quant matrices feature is not applied when you do an OPV encode. I personally wouldn't use LUMGAIN in the redistribution at all unless a 100% sample size is used in the redistribution phase. Otherwise it'll add an uncertainty factor in the process - we are doing a constant quant encode in which the matrix has a key role what comes to final filesize and thus the final bitrate distribution among the segments.

ProCoder's problem is that the Q scale is very bad, even at the maximum quality, you often undersize. That's why you should use Q=1 and not predict it (prediction is mostly a complete waste of time in ProCoder).

[kumi]

Thanks for the clarification Boulder.

I've been trying to test the redistribute function, but I am running into a problem with RBOpt. It freezes after encoding the penultimate segment, with RBOpt.exe pegged at 100% CPU usage.

For reference, I am using RBOpt 0.30, DVD-RB 1.26 Pro and HC 0.21. I use Windows XP SP1 with an AMD Sempron 2600+ and 512MB RAM, with absolutely the bare minimum of services and running programs.

I open the Redistribute tab, and use these settings:
Two VobIDs selected on the left: VTS 1 - Vob-ID 1 and VTS 1 - Vob-ID 2
Sample size: 20%
Q Factor: 20
Threads: 1
Launch Encoder Minimized: yes
Calculate Optimal Q: no
Ignore all filters: no

I've also tried with Calculate Q on, Ignore all filters on, and encoder default matrix set in DVDRB. It's always the same story: RBOpt finishes encoding the next to last segment, and then the "Bitrate Redistribution" windows becomes "Not Responding" and freezes with 100% CPU. The last lines in the log read:
=== Completed encoding VTS 1 VobID 2 segment 11
Size of the segment: 38442 KB
Segment bitrate: 4952

It never starts the final VTS 1 VobID 2 segment 12. What could be the problem?

[Boulder]

Have you tried with minimized encoder window unchecked?

By the way, I'm not even sure if LUMGAIN is considered when you do a constant quant encode..I've never checked the matrices of a constant quant encode with DGIndex.

[robot1]

It never starts the final VTS 1 VobID 2 segment 12. What could be the problem?
I will check.
Meanwhile, could you test with RB-Opt 0.29 and with 0.30 selecting one VobID?
Thank you.

[kumi]

OK, in RBOpt 0.29, the redistribution completes successfully.

I ran RBOpt 0.30 twice (both times with "launch encoder minimized" unchecked):

Running it on only Vob-ID 1: Success
Running it on only Vob-ID 2: Freezes after penultimate segment completes.

So it appears the problem is related to the last segment of VTS 1 - Vob-ID 2 (a small cell with only 49 frames), and only in RBOpt version 0.30.

Here's my REBUILDER.INF:
http://pastebin.ca/574797

And my REBUILDER.ECL:
http://pastebin.ca/574799

[robot1]

OK, in RBOpt 0.29, the redistribution completes successfully.

I ran RBOpt 0.30 twice (both times with "launch encoder minimized" unchecked):

Running it on only Vob-ID 1: Success
Running it on only Vob-ID 2: Freezes after penultimate segment completes.

So it appears the problem is related to the last segment of VTS 1 - Vob-ID 2 (a small cell with only 49 frames), and only in RBOpt version 0.30.

Thank you for your tests.
I've PM'ed you a test version - hope to have solved the bug.

[kumi]

robot1: *ding* You've got mail

On an unrelated and somewhat unimportant note, is it possible for RBOpt to consume less RAM on startup? It eats up ~240MB of my 512MB system total, which is quite a burden when it spawns HC (which eats up another ~200MB).

EDIT: Nevermind, memory usage is fine at ~15MB, I was looking at the wrong column.

[robot1]

Solved the bug reported by Kumi
RB-Opt v. 0.31 beta (http://www.savefile.com/files/822335)

Added through RBOpt.ini support in redistribution for:
HC_Profile (0 fast - 1 normal - 2 best) -default best
QuEnc_Qual (0 normal - 1 high) - default high
QuEnc_Trellis (0 off - 1 on) - default off

[tom942]

@robot1

I don't know if it's a bug, but when HC starts the Q value prediction it sets DC precision to 12 :eek:. Later, when it starts to encode it sets it to 11 (the max value in HC, I suppose).

Is it possible that RB-OPT takes the value that is set in DVD-RB or set it through RBOPT.ini?

:thanks:

[kumi]

I'm seeing the same behaviour: both the Q prediction HC .ini files and the bitrate prediction item0.ini files both have:

*DC_PREC 12

But the HC GUI shows "dc prec 11". My DVDRB DC Prec is set to "Auto".

[kumi]

I ran a redistribution on a disc with 10 short films (1 per VTS, 1 per segment) and noticed segment 2's new bitrate is quite a bit higher than the original DVD9:

http://img261.imageshack.us/img261/3746/imagebl5.png



I'm not sure what to make of this. Common sense tells me it's more desirable to subtract bits from the source, rather than add bits ;)

[robot1]

I'm seeing the same behaviour: both the Q prediction HC .ini files and the bitrate prediction item0.ini files both have:

*DC_PREC 12

But the HC GUI shows "dc prec 11". My DVDRB DC Prec is set to "Auto".
Could you send me rebuilder.inf and rebuilder.ecl?
Thank you.

[robot1]

I'm not sure what to make of this. Common sense tells me it's more desirable to subtract bits from the source, rather than add bits ;)
http://forum.doom9.org/showthread.php?p=1014831#post1014831

The distribution graph is interesting, but we need visual test to check if redistribution gives a better visual result than a standard vbr encoding.

[kumi]

Could you send me rebuilder.inf and rebuilder.ecl?Sure thing. FYI: when I encode this project from DVD-RB as VBR, there is no *DC_PREC directive in the intermediate HC.INI files created, and the HC GUI reports "dc prec 9" for all of the segments.

REBUILDER.ECL
http://pastebin.ca/579264

RFBUILDER.INF
http://pastebin.ca/579270



we need visual test to check if redistribution gives a better visual result than a standard vbr encoding.I will run some comparisons and report my findings. Is the max_reduction variable in rbopt.ini responsible for limiting bitrates to a percentage of the original?

EDIT: Nope, I guess it isn't. Also, I just noticed that DVD-RB won't accept +100% compression levels on any segments, and seems to set any that have "overshoot" to 100% compression when you start an ENCODE phase. But the saved bits aren't redistributed back to the other segments, so I think this is a problem (and might cause undersizing?)

[robot1]

Download here:
RB-Opt v. 0.32 beta (http://www.savefile.com/files/826263)

Fixed the bug reported by Tom942 and Kumi: in some circustancies, the DC Precision for HC was wrong.
Added in RBOpt.ini support for HC_AutoDC (0 Disabled - 1 Enabled)

[robot1]

I will run some comparisons and report my findings. Is the max_reduction variable in rbopt.ini responsible for limiting bitrates to a percentage of the original?

EDIT: Nope, I guess it isn't. Also, I just noticed that DVD-RB won't accept +100% compression levels on any segments, and seems to set any that have "overshoot" to 100% compression when you start an ENCODE phase. But the saved bits aren't redistributed back to the other segments, so I think this is a problem (and might cause undersizing?)
You have to change also a setting in the Rebuilder.ini
Anyway, as you stated above,
Common sense tells me it's more desirable to subtract bits from the source, rather than add bits
I suggest you to leave
max_reduction=100
in RB-Opt.ini, so you don't have problems, and will get the right size because RB-Opt redistribuite the saved bits.
I tell you more: if I get a 100% reduction, I'd tweak the cell, lowering its bitrate, to give more room to other cells. Else, you can set the cell at "no compression" in DVD-RB

[Sharc]

EDIT: ...... But the saved bits aren't redistributed back to the other segments, so I think this is a problem (and might cause undersizing?)
I think this is on jdobbs' agenda:
http://forum.doom9.org/showthread.php?p=1014800#post1014800

[Sharc]

.....I tell you more: if I get a 100% reduction, I'd tweak the cell, lowering its bitrate, to give more room to other cells.
Or set max_reduction=90, for example, to have it done automatically?

[robot1]

Or set max_reduction=90, for example, to have it done automatically?
Probably it should work, but I haven't tried.
max_reduction is a very beta setting, and I haven't played much with it.

You have to check the exact reduction (and bitrate) in the tweak cell dialog, because the log in the redistribution reports the calculated bitrate. The check against max_reduction is performed later, when you press the OK button.

[Sharc]

I see, thanks. I will try.

[kumi]

robot1, thanks a lot for version 0.32 :cool:

I finished a visual comparison of a CBR-ish DVD, pitting HC VBR versus RBOpt redistributed (the graph I posted above). The source is Korean Academy of Film Arts - My Beautiful Short Stories Vol. II, 'Naui Areumdaun Danpyeon Vol.II' (2007) R0:

As opposed to other 8 segments, segments 1 and 2 are shot on old handhelds (8mm, perhaps) with tons of noise, grain, and film speckles/dust. The VBR versions are positively swimming in blocks, compared to the redistributed versions.

Looking at segment 8, where VBR bitrate minus redistributed bitrate is greatest, the VBR version of segment 8 is a bit better than the redistributed version. A little bit less mosquito noise and macroblocking. The other seven segments I consider pretty much a tie.

So huge thumbs up for redistribution on this particular disc, one with a high variance in original video quality from segment to segment. I look forward to testing now on more common VBR-encoded one-movie type discs.



I suggest you to leave max_reduction=100 in RB-Opt.ini, so you don't have problems, and will get the right size because RB-Opt redistribuite the saved bits.OK, I understand now. Redistribution writes to the Reduction= and Video_Sectors= in REBUILDER.INF. If a segment overshoots the original bitrate, RBOpt redistributes the saved bits exactly like you said. The sum of Video_Sectors remains about the same before and after redistribution, which was my clue it's saving bits correctly.

It's confusing, because the values reported in the Bitrate redistribution log do not account for the Reduction= safety mechanism. Could you add some extra info in the log to segments that get 'Reduced'?

More importantly, why doesn't RBOpt write the correct vbr_brate_avg values to REBUILDER.ECL? In cases of overshot bitrate, the values before reduction are written, instead of the actual, compensated bitrates:

Actual rebuilder.ini Percent
Segment Bitrate vbr_brate_avg Deviation
------------------------------------------
01 5251 5251 0%
02 5724 6495 +13%
03 3069 3069 0%
04 2880 2880 0%
05 2972 2972 0%
06 3688 3688 0%
07 3023 3023 0%
08 2360 2360 0%
09 3749 3749 0%
10 2503 2503 0%

Can this be done automatically by RBOpt?


- - -
P.S.: I noticed RBOpt adds lines containing "no_move=0" to REBUILDER.ECL. Just curious, what does that do?

[Sharc]

@kumi
Just for my understanding: Is "Actual bitrate" in your table the bitrate of the final ecnode? Means for segment 2 the encoding was done correctly at the reduced (original, 100%) bitrate, and the +13% overshoot is only a matter of the rebuilder.ecl?

[kumi]

Yes, exactly.

[tom942]

Thanks robot1. I'm going to try it.

[robot1]

OK, I understand now. Redistribution writes to the Reduction= and Video_Sectors= in REBUILDER.INF. If a segment overshoots the original bitrate, RBOpt redistributes the saved bits exactly like you said. The sum of Video_Sectors remains about the same before and after redistribution, which was my clue it's saving bits correctly.

It's confusing, because the values reported in the Bitrate redistribution log do not account for the Reduction= safety mechanism. Could you add some extra info in the log to segments that get 'Reduced'?
Yes, you are right. I haven't changed the log because I want to show what would be the ideal bitrate for a given Q for that cell.

More importantly, why doesn't RBOpt write the correct vbr_brate_avg values to REBUILDER.ECL? In cases of overshot bitrate, the values before reduction are written, instead of the actual, compensated bitrates:

Actual rebuilder.ini Percent
Segment Bitrate vbr_brate_avg Deviation
------------------------------------------
01 5251 5251 0%
02 5724 6495 +13%
03 3069 3069 0%
04 2880 2880 0%
05 2972 2972 0%
06 3688 3688 0%
07 3023 3023 0%
08 2360 2360 0%
09 3749 3749 0%
10 2503 2503 0%

Can this be done automatically by RBOpt?

RB-Opt should write the same bitrate in Rebuilder.ecl and Rebuilder.inf. Do you find different avg_bitrates? Can you post Rebuilder.inf/ecl and the .original ones?


- - -
P.S.: I noticed RBOpt adds lines containing "no_move=0" to REBUILDER.ECL. Just curious, what does that do?
It's a swicth for CCE SP 2.70.

[kumi]

Here you go, .ECL/.INF before and after redistribution:

http://www.savefile.com/files/828723

[tom942]

I think I found a little bug.

I've been testing CCE SP OPV and when you set sample size to 3% and try for instance with HVS-Better o Jawor's 2CD matrices, CCE fails and reports a Q=1. Meanwhile if you set it to 2% or 4%, the prediction works fine.

[Sharc]

It seems to me as if the reduction to the original bitrate (100%) does not work as it should when the redistributed bitrate exceeds the original.
For example, I got the following values (max_reduction=100) for cell 5:

Original: 5668 (measured with DGIndex)
Redistributed: 6368 (from log and rebuilder.ecl)
Actual encoded: 6161 (measured with DGIndex)

There seems to have been some reduction from 6368 to 6161, but not to 5668 (100%) as I would have expected.

I did a new redistribution pass with max_reduction=90. After pressing OK at the end of the redistribution pass the rebuilder.ecl still indicated vbr_brate_avg=6368. I did not do the encode, though.

[kumi]

I'm not seeing a discrepancy when I measure bitrates with Elecard StreamEye or when I simply look at .M2V file sizes:

vbr_brate_avg Original Original Encoded
deviation DVD9 .m2v DVD9 .m2v .m2v Original Re-encoded
from 100% Bitrate Bitrate Bitrate DVD9 .m2v .m2v
Segment Reduction (Elecard) (DVD-RB) (Elecard) Filesize Filesize
-------------------------------------------------------------------------------
02 +13% 5732 5724 5719 382MB 387MB

I think DGIndex uses a different method of bitrate estimation than DVD-Rebuilder. Have you compared the original DVD9 .M2V filesize to the re-encoded one? In my case they're very similar, and so are the bitrates as reported by Elecard.

EDIT: Hmm, in my case DGIndex 1.4.9 reports 100% bitrate (5723), not 113% bitrate (6111).

[Sharc]

Thanks for the hint. May be it's related to the method of bitrate measurement. I will check again with other methods.
Still, what makes me wonder a little is that I measured the original disk and the final RB-encode using the same method (DGIndex) and got different results.