Undersizing with redistribution

[proghack]

Using redistribution, if force reencode isn't enabled (I don't use filters), final size is less than expected because some segments are not compressed.
It's fair that segments with calculated bitrate higher than original are not to be encoded, but to avoid undersizing such segments should be excluded from redistribution.

I think redistribution algorithm should be iterative.
Step 1: encode segments with constant quality (like it is done now);
Step 2: calculate bitrates;
Step 3: go to step 2 excluding segments with calculated bitrate higher than original, until no more segments are over their original bitrate.

If force reencode option is used all that is not needed, of course.

[jdobbs]

Good point. I'll make some adjustments. I will say, though, that I would have thought sections that are larger than the original would have been rare on DVD-9 to DVD-5 conversions.

[proghack]

I agree, in fact in my case compression is about 90%. If compression ratio was lower it would be more difficult to find a segment with bitrate so high.

[jdobbs]

Interesting. Do you still have the REDISTRIBUTION.TXT file? I'd be interested in seeing if there was much of a bitrate change from the original. You wouldn't normally think so, seeing as how it is only shrinking to 90% of its original size.

[kumi]

I ran into this problem with a disc containing short stories (link (http://www.seoulselection.com/shopping_dvd_view.html?pid=1249)). The way robot1 deals with it in RBOpt seems to be to increase the other segments by a fixed multiplier:
http://img525.imageshack.us/img525/6735/image2yc1.png


Is a second estimation pass really necessary?

[proghack]

Is a second estimation pass really necessary?

Absolutely yes. Some segments that had a "valid" bitrate in first estimation pass could exceed original bitrate in second estimation pass, so those segments should be excluded from next step calculations. And on third pass other segments could be excluded and so on. It's a non-linear relation and so it can't be done with a linear operation (multiplication by a factor).
Anyway, looking at the graph, it seems that RB-Opt works well (there are no other segments exceeding original bitrate)

@jdobbs: here is my REDISTRIBUTION.TXT

BITRATE APPLIED TO FEATURE SEGMENTS (SEGMENT ORIGINAL REDISTRIBUTED)
(Created using a 10% sample)

V03000000001001 1.548 Kbs 774 Kbs
V03000100002001 4.573 Kbs 4.687 Kbs
V03000200003001 4.731 Kbs 5.031 Kbs
V03000300003002 4.160 Kbs 4.984 Kbs
V03000400003003 3.993 Kbs 4.727 Kbs
V03000500004001 4.162 Kbs 5.888 Kbs
V03000600004002 4.074 Kbs 5.279 Kbs
V03000700005001 3.960 Kbs 3.581 Kbs
V03000800005002 4.004 Kbs 3.849 Kbs
V03000900005003 4.177 Kbs 4.027 Kbs
V03001000006001 4.079 Kbs 4.449 Kbs
V03001100006002 4.074 Kbs 4.984 Kbs
V03001200007001 3.993 Kbs 3.204 Kbs
V03001300007002 3.685 Kbs 2.815 Kbs
V03001400007003 4.531 Kbs 4.524 Kbs
V03001500008001 4.073 Kbs 3.477 Kbs
V03001600008002 4.080 Kbs 4.063 Kbs
V03001700008003 4.076 Kbs 5.180 Kbs
V03001800009001 3.913 Kbs 5.404 Kbs
V03001900009002 4.140 Kbs 5.851 Kbs
V03002000009003 4.280 Kbs 7.130 Kbs
V03002100010001 4.078 Kbs 3.706 Kbs
V03002200010002 4.076 Kbs 3.836 Kbs
V03002300010003 4.075 Kbs 3.228 Kbs
V03002400011001 4.076 Kbs 3.861 Kbs
V03002500011002 3.762 Kbs 5.899 Kbs
V03002600011003 4.192 Kbs 6.143 Kbs
V03002700012001 4.073 Kbs 3.933 Kbs
V03002800012002 4.070 Kbs 4.086 Kbs
V03002900012003 4.108 Kbs 3.848 Kbs
V03003000012004 3.961 Kbs 4.045 Kbs
V03003100012005 4.235 Kbs 4.750 Kbs
V03003200013001 4.073 Kbs 3.877 Kbs
V03003300014001 4.000 Kbs 4.246 Kbs
V03003400014002 4.310 Kbs 5.369 Kbs
V03003500014003 4.108 Kbs 4.917 Kbs
V03003600015001 4.076 Kbs 5.067 Kbs
V03003700015002 4.548 Kbs 4.693 Kbs
V03003800015003 4.075 Kbs 4.631 Kbs
V03003900015004 4.077 Kbs 5.179 Kbs
V03004000016001 4.466 Kbs 6.213 Kbs
V03004100016002 4.050 Kbs 6.154 Kbs
V03004200016003 3.378 Kbs 3.488 Kbs
V03004300017001 4.076 Kbs 3.604 Kbs
V03004400017002 4.069 Kbs 3.742 Kbs
V03004500017003 4.074 Kbs 3.695 Kbs
V03004600018001 4.307 Kbs 6.210 Kbs
V03004700018002 3.802 Kbs 5.121 Kbs
V03004800018003 3.788 Kbs 4.566 Kbs
V03004900019001 4.070 Kbs 4.192 Kbs
V03005000019002 4.077 Kbs 3.432 Kbs
V03005100019003 4.074 Kbs 2.947 Kbs
V03005200019004 4.075 Kbs 4.822 Kbs
V03005300020001 4.245 Kbs 3.637 Kbs
V03005400020002 3.660 Kbs 3.317 Kbs
V03005500020003 3.863 Kbs 3.658 Kbs
V03005600021001 4.073 Kbs 4.678 Kbs
V03005700021002 4.071 Kbs 3.889 Kbs
V03005800021003 3.991 Kbs 2.290 Kbs
V03005900022001 1.935 Kbs 774 Kbs
V03006000023001 1.935 Kbs 774 Kbs
V03006100024001 1.935 Kbs 774 Kbs
V03006200025001 1.935 Kbs 774 Kbs
V03006300026001 1.935 Kbs 774 Kbs
V03006400027001 1.935 Kbs 774 Kbs
V03006500028001 1.935 Kbs 774 Kbs
V03006600029001 1.935 Kbs 774 Kbs
V03006700030001 1.935 Kbs 774 Kbs
V03006800031001 1.935 Kbs 774 Kbs
V03006900032001 1.935 Kbs 774 Kbs
V03007000033001 1.935 Kbs 774 Kbs
V03007100034001 1.935 Kbs 774 Kbs
V03007200035001 1.548 Kbs 774 Kbs


I remember that segments over original bitrate are 3, 4, 5, 6, 10, 11...

[jdobbs]

Just to ensure I understand correctly. Then when the ENCODE phase was accomplished, those segments (3, 4, 5...) were extracted rather than encoded because their bitrate was higher than the original? That was (probably) the reason for the undersize?

[proghack]

Exactly

[Sharc]

Same here.
- Undersizing (4.29 instead of 4.34 GB) when redistributed segments exceed the original bitrate
- Discrepancy between the bitrates reported in the redistribution.txt and the Editor (or rebuilder.ecl).
See also:
http://forum.doom9.org/showthread.php?p=1016015#post1016015

[jdobbs]

Cool. I can fix that.

[proghack]

Has this problem been fixed? In version 1.26.5 it is still there.
Let's have this example (all segments have the same length for simplicity):

Seg 1: 3000 bps
Seg 2: 3000 bps
Seg 3: 3000 bps

After normal calculation, in order to fit DVD-5 we have:

Seg 1: 2500 bps
Seg 2: 2500 bps
Seg 3: 2500 bps

After redistribution we have (for example):

Seg 1: 2900 bps
Seg 2: 3500 bps
Seg 3: 1100 bps

Now segment 2 would be larger than original (3000 bps) so DVD-RB extracts that segment rather than compress it (correctly), but doing so there would be undersizing. To avoid this, 500 bps saved should be distribuited between segment 1 and segment 3, based on segment lengths. What we have is:

Seg 1: 3150 bps
Seg 2: 3000 bps
Seg 3: 1350 bps

Now segment 1 would be too large, so we would repeat previous step using 150 bps exceeded in segment 3. Finally:

Seg 1: 3000 bps
Seg 2: 3000 bps
Seg 3: 1500 bps

The extra steps needed to move bps to other segments are very simple and don't need any compressions, it's only algorithmic.
Now I have to do it manually. Could it be implemented?
Thanks.
P.S.: if there are not segments exceeding original bitrate or force reencoding is active, final size is correct.