Suggestion for a new method of bitrate distribution

[Boulder]

Just an update..a quick look over the CBR re-encode shows that the redistribution method does improve the overall quality of the backup over the regular method. The disc I chose is not a really good sample as the bitrate is just too low to get a high quality backup no matter what you do. I think I'll try redoing the test on another CBR source I've got.

[Sharc]

Also the option to use the original segment would apply only when you don't filter at all, deblocking and all included.

I fully agree. Filtering opens up another dimension. It becomes even more prominent when different filters are applied to the individual cells. In such cases redistribution may almost become a must.

[Boulder]

Here's some screenshots from the CBR re-encode. The redistributed encode screenshots were taken from segments that had the highest (the first two pairs) and lowest (the last pair) average bitrate of all. The lowest one was 2588kbps and the highest one 5894kbps. The original CBR bitrate was 3725kbps (after reduction).

RB
http://img53.imageshack.us/img53/366/holmesrb1ri1.th.png (http://img53.imageshack.us/my.php?image=holmesrb1ri1.png)
Redist
http://img510.imageshack.us/img510/6310/holmesredist1mi4.th.png (http://img510.imageshack.us/my.php?image=holmesredist1mi4.png)

RB
http://img510.imageshack.us/img510/7413/holmesrb2cl2.th.png (http://img510.imageshack.us/my.php?image=holmesrb2cl2.png)
Redist
http://img53.imageshack.us/img53/328/holmesredist2vl9.th.png (http://img53.imageshack.us/my.php?image=holmesredist2vl9.png)

RB
http://img523.imageshack.us/img523/7901/holmesrb3xl3.th.png (http://img523.imageshack.us/my.php?image=holmesrb3xl3.png)
Redist
http://img510.imageshack.us/img510/114/holmesredist3tv3.th.png (http://img510.imageshack.us/my.php?image=holmesredist3tv3.png)

[~bT~]

Some of those RB snaps look ugly..

[Boulder]

The source is quite far from perfect ;) But it's a good representation of a rather sloppy DVD transfer which are unfortunately quite common.

[archaeo]

The redistributed examples look noticeably better to me. I was paying particular attention to the last landscape scene, where you'd usually expect to see some pixellating in those washed out areas. The redistributed example looks smoother, and without loss of detail, while the RB example is starting to pixellate. I also noticed that the color seems to be more saturated in the high bitrate redistributed examples.

[jdobbs]

I'll be honest with you... maybe it's my monitor -- but I'm not seeing a whole lot of difference. Maybe in the cheekbones on the "face" picture. I have doubts anyone would be able to pick out which is which in a full motion double-blind playback.

[Edit] Ok... now that I overlay them and switch back-and-forth, there definitely is some improvement. But what about scenes from the segments where the bitrate dropped? You have to be unbiased and do a tit-for-tat or it really doesn't say anything.

You say these were taken from a CBR source?

[jdobbs]

Also, can you post the average bitrate used for encoding next to each of the pictures so we can know what we're comparing? I already know a higher bitrate will look better, and I'm guessing that each of the "redist" examples were encoding at a higher bitrate. Not too sure about the last one, though, because the difference isn't really easily seen.

We also have the old problem of compressing a compressed image (into .PNG) for posting -- but I don't know how we'll ever get around that.

Are the pictures shown comparisons of I-FRAME to I-FRAME so we know they are equal as well? No I-FRAME to B-FRAME comparisons, I etc, as we know those would not be equal in quality.

I'm not trying to rag on the process, as I'm pretty sure it will improve CBR, I just want to make sure there is something substantive involved before I add it as an option.

[gurkan]

@boulder
It would be nice to see the original frames as well.
What I interpret as mosquito-noice in the rb-encode might be the actual details from the original.

Edit:
I'm especially thinking of the lowest one. How the redist frame with a lesser avg bitrate than rb can look better is beyond me.

[Boulder]

Also, can you post the average bitrate used for encoding next to each of the pictures so we can know what we're comparing? I already know a higher bitrate will look better, and I'm guessing that each of the "redist" examples were encoding at a higher bitrate. Not too sure about the last one, though, because the difference isn't really easily seen.

We also have the old problem of compressing a compressed image (into .PNG) for posting -- but I don't know how we'll ever get around that.

Are the pictures shown comparisons of I-FRAME to I-FRAME so we know they are equal as well? No I-FRAME to B-FRAME comparisons, I etc, as we know those would not be equal in quality.

I'm not trying to rag on the process, as I'm pretty sure it will improve CBR, I just want to make sure there is something substantive involved before I add it as an option.What comes to using PNG, it is lossless so it doesn't affect the comparison.

I took those screenshots using an Avisynth script so I can't say if they are I-, P- or B-frames. I'll try checking those frames with VDubMod and see if they correspond. At least they should be the same in those two encodes as they were done using the same encoder.

I'll have you some more samples later today, including the original frames etc. Today's Mother's Day and Finland's playing in the ice hockey WC finals so I'm quite busy already ;)

EDIT: IIRC the last pair was taken from a segment which had a lower avg bitrate than the one RB decided to use. Nevertheless, that will be doublechecked too.

[Boulder]

It was hard, but I did find some representative frames that were I-frames in the RB encode, the redistributed one and in the original.

The first comparison set is from a segment which had an average bitrate of 2683kbps when redistributed. The second comparison set is from a segment that had an average bitrate of 3192kbps after redistribution. The original RB-determined bitrate was 3725kbps (after the reduction had been calculated). The original video is CBR.

I used the parameter CPU=4 in the test encodes as I forgot to disable it in rebuilder.ini. I did a normal encode after the last tests so I had put it there. That means that in theory, the blocking should be less severe in the re-encoded frames.

Redist at avg 2683kbps, RB at avg 3725kbps, original CBR ~6593kbps
The original I-frame:
http://img246.imageshack.us/img246/4097/holmesorgiframe1bh0.th.png (http://img246.imageshack.us/my.php?image=holmesorgiframe1bh0.png)
The original I-frame with CPU=4:
http://img523.imageshack.us/img523/3610/holmesorgiframe1cpu4iv7.th.png (http://img523.imageshack.us/my.php?image=holmesorgiframe1cpu4iv7.png)
The I-frame from a one-click DVD-RB encode:
http://img182.imageshack.us/img182/6466/holmesrbiframe1ey8.th.png (http://img182.imageshack.us/my.php?image=holmesrbiframe1ey8.png)
The I-frame from a redistributed bitrate encode:
http://img246.imageshack.us/img246/9703/holmesredistiframe1ng5.th.png (http://img246.imageshack.us/my.php?image=holmesredistiframe1ng5.png)


Redist at avg 3192kbps, RB at avg 3725kbps, original CBR ~6593kbps
The original I-frame:
http://img246.imageshack.us/img246/6838/holmesorgiframe2sh4.th.png (http://img246.imageshack.us/my.php?image=holmesorgiframe2sh4.png)
The original I-frame with CPU=4:
http://img408.imageshack.us/img408/4222/holmesorgiframe2cpu4xl7.th.png (http://img408.imageshack.us/my.php?image=holmesorgiframe2cpu4xl7.png)
The I-frame from a one-click DVD-RB encode:
http://img182.imageshack.us/img182/3104/holmesrbiframe2us2.th.png (http://img182.imageshack.us/my.php?image=holmesrbiframe2us2.png)
The I-frame from a redistributed bitrate encode:
http://img179.imageshack.us/img179/7199/holmesredistiframe2si5.th.png (http://img179.imageshack.us/my.php?image=holmesredistiframe2si5.png)

Here's the bitrate distribution graph:
http://img246.imageshack.us/img246/4455/holmeslr1.th.gif (http://img246.imageshack.us/my.php?image=holmeslr1.gif)

[Sharc]

Do I understand corectly:
- Original CBR = ??? (how much?)
- RB standard reduced avg bitrate = 3725kbps (from DVD-9 to DVD-5 reduction)
- Redistributed avg cell bitrate for example 1 = 2683 kbps
- Redistributed avg cell bitrate for example 2 = 3192 kbps

Was cpu=4 also in effect for the capturing of the original I-frames?

Added:
To my eyes, the redistributed sample frames look slightly better than the standard VBR reencoded frames - i.e. less pixelation without sacrificing useful details or sharpness - although the redistributed avg cell rate was below the standard 2-pass VBR bitrate of 3725 kbps - and probably significantly (?) below the original CBR bitrate.

I am not sure however which one would look "crispier" when watching ....

[Boulder]

The original CBR bitrate is ~6593kbps. I know it sounds a lot but the source is very noisy and the video is wobbly like those tapes from the 80s tend to be.

I added the original+CPU=4 images to my previous post.

[Sharc]

Attached an exampe with constant average cell bitrate, yet not CBR, because the swing within cells is significant.

Findings:
1) Those (short) cells that got significantly boosted (containing scenes with slight blocking in the original, that's why I used cpu=4 ) were slightly less noisy compared to the standard 2-pass VBR. Very similar to the pictures in posts #153 and #161.
2) In those cells from which bits were stolen - mainly dark scenes - I couldn't discover an obvious deterioriation - some frames looked even slightly less noisy than the standard VBR encode.
3) I would say that for this example redistribution did "the lowest possible damage" to the original, although the differences to standard 2-pass VBR are subtle and - to a certain extent - a matter of personal taste as well.

The maximum bitrate was still limited by RB default according to the default DISABLE_DYNAMIC_PEAK=0.


Added:
Impact of filter "fluxsmoothST(5,5)" on redistribution.

[Boulder]

I added the bitrate graph for that Sherlock Holmes DVD. It's disc 3 from The Return of Sherlock Holmes DVD box if anyone's interested.

[Sharc]

I added to post#164 a graph showing the impact of filtering -- fluxsmoothST(5,5) -- to the redistribution, for comparison.
Interesting to note that the filtering pushed the peaks of cells 1,7,13 even higher.

[Sharc]

Just another benefit of redistribution:

Redistribution can prevent encoder saturation -- with eventual undersizing -- by stealing bits from saturated scenes and throwing these bits to more demanding cells.

Undersizing due to encoder saturation can also be avoided by enabling dynamic matrix switching (CCEAQM=1), however this requires 3-pass VBR and has the tendency to enhance noise for the frames that get an aggressive high bitrate matirx allotted.

[jdobbs]

Maybe I'm missing something, but, why do you believe "CCEAQM=1" require 3 passes? I use it with two passes all the time.

[Sharc]

Maybe I am wrong, but I thought that there is a difference between CCE SP trial and CCE Basic.
CCE SP trial requires 3 passes: 1 vaf + 2 encoding for CCEAQM becoming effective. I beleve that this has been stated in a few other posts in this forum, if I remember correctly (?)

[Boulder]

Two passes with AQM often end up in sizing problems. It seems that the feature doesn't always compensate the change of matrix well enough in the first pass after the vaf file creation.

EDIT: AQM is used even when you do 2 passes (vaf + 1 pass).

[Sharc]

Thanks for clarification. Sorry for the confusion.

[jdobbs]

Interesting, I've never had a problem.

[Boulder]

It's most common with sources where you would undersize greatly without AQM. Even with AQM enabled, some undersizing usually occurs in the second pass but the third pass gets you close to the target. That's one reason why I recommend using three passes with AQM.

[jdobbs]

Hmm... I've tried on a few discs that undersized without it -- and they always seem to size ok for me with CCEAQM=1 set, and I rarely ever do more than 2 passes (VAF + 1). I guess it depends on the circumstances.

[Sharc]

Hmm... I've tried on a few discs that undersized without it -- and they always seem to size ok for me with CCEAQM=1 set, ......

Would be interesting to see if redistribution -- without AQM -- would hit the target size as well for these discs.
I am a little sceptic when AQM kicks in sharper matrices for the reencodes than those used in the original disc. It may just enhance the noise.

Added:
..... but this brings me back to my old wish that Rebuilder should adapt its matrix selection -- as defined in the Options -- according to the actual redistributed or tweaked cell bitrates.
I am not sure however how to solve the priority issue between settings done by RB-Opt (as "CCE settings") and selection made by Rebuilder based on bitrate....

[Sharc]

Here a comparison of 2 different encoders: CCE and Quenc, with default matrix each. The results are very similar.

Is there an (easy) way to find out if the bitrate redistribution is primarily driven by encoding artefacts of the original disc?

[Sharc]

The question as to what extent mpeg artefacts of the original disc would influence the redistribution is still open, IMHO. There have been concerns that the bitrate of the redistribution is mainly driven by the mpeg imperfections of the original encode.
I made the following test to provide some evidence that this is most probably not the case: I did a second redistribution from the redistributed backup and compared the results.
1) Original disc
2) Standard 2-pass VBR
3) Redistributed backup of 1)
4) 2nd redistribution (= redistribution of 3)

Because curves 3 and 4 are much the same I would conclude that the redistribution is not driven by encoding artefacts. If artefacts would be the main drivers for redistribution I would expect significant differences between curves 3 and 4.

[Sharc]

Let's not forget that there are simpler reasons when going above the original bitrate is not useless. Think for instance when you use the filter-editor, and put a slight sharpener in there (or produce something much more complicated). The script will then produce more details, and a higher bitrate could be called for.

I did some test with sharpening filters (SeeSaw) and softening filters (fluxsmooth, deen). To my surprise the sharpening or softening did not have a dramatic impact on the basic -- ie unfiltered -- redistribution. It appears that the redistribution is mainly driven by the native characteristics of the original.

[robot1]

I did some test with sharpening filters (SeeSaw) and softening filters (fluxsmooth, deen). To my surprise the sharpening or softening did not have a dramatic impact on the basic -- ie unfiltered -- redistribution. It appears that the redistribution is mainly driven by the native characteristics of the original.
It's a great finding. I didn't expect this result.
Thank you for your tests.

[Boulder]

That is interesting indeed. I guess that the amount of motion remains more or less the same unless you blur the image heavily. That would explain why there is not much difference when scaled to the same level.

[Sharc]

That is interesting indeed. I guess that the amount of motion remains more or less the same....
Probably, yes.
Here some plots. There are some differences due to the filtering, but these are insignificant compared to the basic swing change.

[gurkan]

@sharc
I would like to conduct some testings of my own.
What program are you using to create those nice comprehensible plots?

[Sharc]

You may use any spreadsheet office application which comes with options for producing charts.

[gurkan]

Ok. Guess I can't avoid learning excel any longer. ;)

[Sharc]

Some findings regarding matrices and Q-factors, based on various tests with CCE and "normal" VBR disks.

1) Using the same matrix as in the original title, there exists a (typically low) Q factor for which the redistribution yields a profile that is fairly close to the profile of the original disk. This Q factor would -- applied to direct OPV - result in a size which is also similar to the original (DVD-9). This test indicates that the bitrate distribution of the original disk can be almost "reproduced" by selecting the original matrix and the appropriate Q factor.

2) When increasing the Q (i.e. reducing the file size towards a DVD-5), the deviation of the redistributed profile from the original profile - or downscaled according to the reduction level -- becomes more and more prominent. The swing becomes larger.

3) When selecting Q for different matrices such as to yield the same target size (e.g. DVD-5), the resulting redistribution profiles become very similar and largely independent from the matrix.

The graph shows the influence of Q on the redistribution profile, using the same matrix (FHE) as in the original title.

Q=5 approximates the (downscaled) original distribution profile closely.
Q=35 corresponds to the DVD-5 target size.
Q=64 indicates that there is no significant difference to Q=35. There seems to exist a kind of "saturation".

Finally, it seems to be most appropriate to base the redistribution on a Q-factor which approximates the target size. This Q-value is readily delivered by RB-Opt, even for a particular matrix selection (using CCE settings).

Sidenote:
If we assume that the bitrate distribution profile of the original disk is perfect and we can reproduce it as in 1), one may conclude that using higher Q factors for more compression will also produce optimum results. 1) + 2) are IMHO an indication that redistribution may produce optimum quality even for "normal" VBR disks ......... :rolleyes:

[jdobbs]

In my testing of the routines I've added to DVD-RB for redistribution I've noticed that trying to do small percentage (10% and smaller) samples on some segments really screws up the quality. For example, If you have a small segment that starts blank and then fades into a picture -- the output size at a fixed Q with sampling cuts the calculated "needed sectors" exceptionally low -- and I get blockiness in the output.

Have you guys experienced this?

[dirio49]

I have seen this.
The movie i tried had a very small cell/segment. and it was lowered too much.
Maybe if the segment is too small exlude it form the redistribution.
BTW I was testing with like 1 -3% (hey i have a very slow computer, and time is money ;) )

[Sharc]

In all my tests I selected the sample size 25%, and I have not experienced the problem with blockiness. I did observe few poor estimates of the cell bitrate for very small cells though, but luckily on the high side (waste of bits).

I therefore suggest to include a rule for the sample size which looks something like:
Sample size=Max(a;b).
a=sample size in % (e.g. 20%)
b=number of frames (e.g. 500)
a and b could be hidden settings in the rebuilder.ini

If the entire cell is less than 500 frames, all frames of the cell should be included in the sample. Very small cells (<200 frames) may be copied intact or may get the standard multipass VBR percentage reduction.

[Sharc]

In my testing of the routines I've added to DVD-RB for redistribution I've noticed that trying to do small percentage (10% and smaller) samples on some segments really screws up the quality. For example, If you have a small segment that starts blank and then fades into a picture -- the output size at a fixed Q with sampling cuts the calculated "needed sectors" exceptionally low -- and I get blockiness in the output.

Have you guys experienced this?

Another observation:
I have experienced that for a very low Q / high bitrate matrix (FHE) combination, CCE may silently abort during the cell sampling, continuing with the next cell.
The aborted cell will finally have a very low bitrate allocated to it, especially when the abortion happened at the beginning of the cell.
I don't know the reason for the silent abortion. Perhaps it is related to an excessively high bitrate which is beyond the mpeg/DVD standard. The problem is that the abortion is silent, means you will not notice it during the redistribution process, but only when finally watching the blocky cell.

As I said, I only observed this for unreasonable low Q/high bitrate matrix combinations. I have never observed it for Q factors that approximate the DVD-5 target size, irrespective of the matrix.

I mentioned this also at the end of this post:
http://forum.doom9.org/showthread.php?p=1012090#post1012090

[gurkan]

Out of curiosity:
Has anyone made a comparison between a redist. encode and a "real" unsegmented back-to-back vbr encode?

[jdobbs]

I've done a lot of analysis in which I did bitrate allocation comparisons between the standard DVD-RB dynamic bitrate allocation (the default) and methods in which you do a complete encode of the feature and reintegrate into the DVD -- and interestingly there was very little deviation in the curves.

I haven't had time to do a lot of comparisons using the redistributed methods... but there's plenty of background if you read this thread end-to-end.

[gurkan]

Was just a thought. Since unsegmented encoding and redist. encoding doesn't take the original bitrates of the segments in mind.
It would be interesting to see what differentiates these two similar processes. Especially since the q-pass only provides the bitrates to the segments.
It's still a regular vbr being done, if i'm not mistaken, when it comes to the actual encoding.

[Boulder]

If the segment length in frames is large enough, redistributed segmented encoding should yield pretty much the same results as unsegmented multipass encoding.

[gurkan]

Ok. Thanks for the info, Boulder.

[Sharc]

...... therefore suggest to include a rule for the sample size which looks something like:
Sample size=Max(a;b).
a=sample size in % (e.g. 20%)
b=number of frames (e.g. 500)
a and b could be hidden settings in the rebuilder.ini

If the entire cell is less than 500 frames, all frames of the cell should be included in the sample. Very small cells (<200 frames) may be copied intact or may get the standard multipass VBR percentage reduction.

@jdobbs:
Does the current version 1.26.0 already include a protection against erratic bitrate allocation to small cells? The default sample size is 10% which seems to be rather low for small cells -- I still set it to 25% for that reason.

[jdobbs]

Yes. It looks at the source and if it is a small segment (less than 500 frames) it automatically ups the sample size to 25% (unless it is already greater than 25%).

[Sharc]

This certainly mitigates the situation, I am however not sure if stepping up to 25% will solve the problem in all cases, as I have seen tiny cells of less than few hundred frames. 25% of say 600 frames makes only 150 frames which I still consider insufficient for a reliable bitrate estimation.

What I actually meant with my proposal is that the sample size should take the larger of either the stated percentage (for example the default 10% or whatever is specified in the rebuilder.ini), or 500 frames. So in the case of a 600 frames cell, 500 would be included in the sample. In the case of a 7000 frames cell, 10% of 7000 = 700 frames would be taken.

In cases when the cell has even less than 500 frames, all frames should be included. Tiny cells should be excluded from the redistribution and copied intact, or perhaps encoded at the standard multipass average reduction factor.
The smallest cell I have come across was 28 frames only, btw.

Maybe my 500 frames for the lower limit are pessimistic, that's why I thought this number could be included in the rebuilder.ini as the second parameter.

[jdobbs]

I can certainly modify it -- but in all the tests I did it seems to have worked well. I understood what you'd suggested, but I considered this to be a better alternative. 125 frames out of 500 should be plenty of sampling to get a good mix for distributing it -- especially when it grabs them 12 or 15 at a time throughout the entirety of the stream. Actually I'd made the decision to use 500 frames as the breakpoint even before you'd suggested that -- which was an interesting coincidence.

[Sharc]

ok, understood. Thanks for your explanation. Makes sense.

[jdobbs]

By the way, thanks for all your testing and posting the bitrate analysis. The data you showed using RB-Opt was one of the main reasons I decided to add this feature to baseline DVD-RB. My hat is also off to Boulder and (of course) robot1.

:thanks: