X-pass D2S RoBa

[telemike]

Can we start this new thread to discuss how/why to use X-pass D2S RoBa?

[DDogg]

First see my windbag post in the CCE forum 'cause that is what it discusses. http://forum.doom9.org/showthread.php?s=&threadid=59850 then lets do it, ok?

[telemike]

What would happen if I set the internal Roba to 1-pass?

[r6d2]

DVD2SVCD would do 2 passes. One with OPV and one with VBR.

[DDogg]

Note to r6d2 and tylo. Work in progress and a reality check is needed from both of you :) I'll edit it until it is correct

One with OPV and one with VBR. Or to put that in slightly different words, One quality based pass would be done that enforces the quality level you have set via the desired "Q" value. Then a sizing pass follows that uses the existing VAF but tweaks it to create the requested file size.

In the D2s internal process, if you have set the desired Q to a level that is incompatible with the target size, D2S will make the decision to add another disk. That was what RoBa was originally all about. Keeping quality and automatically incrementing disks so as to keep that quality.

[r6d2, reality check please]
Here is where it gets slightly complicated. There is a potential problem in 1.2.1 Build 2 and earlier versions. As I understand it, the D2S internal process uses certain rules that Bach established early on that allowed up to a 50% size correction before adding another disk. They have since been more refined to a maximum of 10% size correction, but Dvd2Svcd/dvd uses the earlier set of rules as of 1.2.1 Build 2.

To work around this small oddity, the user needs to be more precise and input a "Q" that would produce a target size within 10% of the desired size to fill the media selected. Then the internal RoBa process will always get it right.

Calculating this needed Q is not all that hard, but it requires several samples to be run manually and some special scripting and other techniques that would be classed as advanced user level which might bewilder the casual user. If he ever gets to it, D2S said he would add automatic prediction of the appropriate Q sometime in the future. In the meantime tylo added the X-pass D2S RoBa method to his plug-in to deal with this for all level of users.

Enter tylo's D2SRoBa plug-in with X-pass D2S RoBa to the stage and the discussion.

[tylo, reality check please]
Those complicated prediction methods is exactly what the X-RoBa section of tylo's plug-in can do. See his HP for a better understanding of quality based methods and sizing based methods.

Auto mode (quality based):
If you chose Auto, It predicts how many disks will be needed to "hold" that Q and that Q value is fed back to the D2S internal RoBa Process (only if X-pass D2S RoBa mode is checked) so that the sizing pass has to just barely correct the quality based encode method to make sure the file is not too over-sized or undersized (no more than 10% correction). Note: If X-pass D2S RoBa mode has not been checked then only a OPV is used with the predicted Q. You get what you get size wise. Sometimes very close to exact, and sometimes not as the sampling method cannot by its nature be perfect.

Specific number of disks mode (size based):
If the user specifies a certain number of disks as the target then Tylo predicts a Q that will create a correct file size to fill the specified disk or disks. Again, because prediction has an inherent potential of sizing error, if X-pass D2S RoBa mode is checked, that predicted Q is then fed back into the internal Dvd2svcd/DVD RoBa Process and the second pass gets the size just right.

Please note that in sizing mode this Q is all about size and not quality. If you specify a 3 hour movie on 1 disk, it will attempt to do it continually lowering quality to do so. IMO, most users would be better served by selecting the auto or the "?" mode which will present you the number of disks to use as an option to chose from. Allowing a Q of higher than Q 30-40 is not advised if you want good quality. You can always try CVD instead of SVCD if size is more important than quality.

Note: IMHO, the sizing mode should be considered an advanced user mode for people using compression filtering to achieve a certain Q target and should seldom be used by anybody else. Experienced users making SVCDs who have a good feel for length of source and compression characteristics versus target size might like the "?". Most all other SVCD makers should just use "Auto".

DVD makers should generally use sizing mode and set disks to 1 and disk size to match what they used in DVD2SVCD/DVD. Then they should monitor the Q level and make sure it does not exceed whatever they deed minimum quality, like 25-32. Adding some light filtering in the avisynth script and doing a restart should drop Q level easily given the luxury of bitrate available without adversely effecting the video quality.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Concept of a Conditional Second pass:
If you read the above you may find yourself realizing the second sizing pass is only needed, IF and only if, the first OPV pass using the predicted Q size has more sizing error than you are willing to accept. That introduces the concept of a conditional where the second pass would only be done IF the size of the first pass exceeds a certain acceptable percentage. I believe this hybridized "Conditional RoBa" is the final answer for all SVCD and DVD encoding and have discussed it with both Tylo and the author of Dvd2svcd/DVD.

To work towards the goal of "Conditional RoBA", let me throw out a statement and see if anybody disagrees with it, and if so, why.

The second pass is only for sizing. IMHO, it does nothing else. In fact, I think one could argue that the second pass may actually slightly damage quality somewhat as it introduces sizing and Bitrate methods into a quantizer based quality process.

[telemike]

Thanks for explaning that DDogg :) Clears up most of my questions about it.

I think I will stick to 1-pass OPV for encoding time (I can't afford a second pass) and so far I have had good results with disk capacity.

Is there a way to fool the internal Roba to do only 1 pass?

[DDogg]

Is there a way to fool the internal Roba to do only 1 pass? The internal RoBa is *always* at least two passes.Later: Yes, you can sorta fool it but it is such a pain in the backside as not to be worth the aggravation.

[r6d2]

Originally posted by telemike
Is there a way to fool the internal Roba to do only 1 pass?
@telemike, it makes no sense to use internal RoBa with 0 passes in VBR mode. This would be just like sticking to OPV.

As DDogg throughly explained, the second pass is only to fit the target size to the expected value, in case the error of OPV prediction was higher than what you are willing to accept.
Originally posted by DDogg
The second pass is only for sizing. IMHO, it does nothing else. In fact, I think one could argue that the second pass may actually slightly damage quality somewhat as it introduces sizing and Bitrate methods into a quantizer based quality process.
Those who would argue that understand little about CCE VBR process. Please take a look at chapter 3.3.4 of the CCE manual, which I paraphrase here:

Multipass VBR indeed does a 1 pass VAF file creation internally, either on CBR or OPV mode. If the second pass (actual first VBR pass) would screw up the encode, why would they do it?

On subsequent passes, VBR rereads the VAF file and redistributes the available BR, supposedly converging to perfection.

The point is that the better the initial VAF is, the better the encode will be with less passes. OPV with the adequate Q is very likely to give you a good VAF file faster, since it gives the next pass an already leveled file to work with.

Now back to RoBa:

The second pass uses the VAF file from the first OPV pass (and only that, it does not use the MPV file), and it operates like this:

1. Underestimation (OPV's MPV is shorter than the target size)

The VBR pass reencodes from scratch, without recreating a VAF file as it normally would, and spread the additional BR amongst the frames which need it to enhance the quality of the encode.

2. Overestimation (OPV's MPV is larger than the target size)

Same as above, but instead of spreading it removes where there is more than enough to put it where it is needed. In this case the overall quality, measured as the average quantization will of course be a little lower than in the OPV's MPV, but in any case will be adequate to the Q.factor entered, and better than the one on the sample.

Experience shows that prediction underestimates more often than it overestimates, particularly if you use short clips of 1 GOP length. This typically inserts entropy on the sample which was not in the original source, making the sample over compressed to keep the Q.factor entered and hence predicting shorter MPVs.

The bottom line is, the second pass will not screw up your encode, it will improve it, provided that your desired quality commanded the first OPV pass and you chose a Q.factor value which yields a BR within the +/-10% target BR range. Even a 1% sample can give you that.

In my tests, prediction moves on the +/-5% range with GOPs=1, D2Sroba's default.

Regarding DDogg's reality check, I fully support the given statement (addressed to me).

[DDogg]

Multipass VBR indeed does a 1 pass VAF file creation internally, either on CBR or OPV mode. If the second pass (actual first VBR pass) would screw up the encode, why would they do it?Erm, I not so convinced that the quote above makes a very compelling argument and that the next quote would even logically follow:The bottom line is, the second pass will not screw up your encode, it will improve it, provided that your desired quality commanded the first OPV pass and you chose a Q.factor value which yields a BR within the +/-10% target BR range. Even a 1% sample can give you that.But, I also don't disagree with it or you. I do think the alternate discussion point could be made that OPV has the ability to create the VAF because sizing in OPV is a known problem and they added the ability to reuse the VAF because of it. Your points above seem to combine pure virgin quantizer based OPV with bitrate based VBR as if they were the same thing and that is where I am hung up as I don't think they are.

Your statement implies the second pass is and can only be of benefit to the quality of the encode. It would seem to me that the *possibility* that the second sizing pass could negatively impact quality, albeit a minuscule amount, could also be true very easily.

I guess that would take an answer from CCE which I doubt is going to happen.

r6d2, I wonder if you would then refute this: Assuming a Quantizer based encode and a 2 pass VBR encode of exactly the same size and time duration I believe the Q based encode will yield the same, perhaps better, but never less quality than that of the 2-pass VBR. If I understand what you said above, I think you are saying the 2-pass VBR would be inherently better? That it *must* be better? Please share your thoughts.

[jsquare]

Hey guys, what we need is a VQR (Variable Q. Rate) method.

Comparing the 2 methods(my opinion):

OPV makes better looking encodes, depending on the source and how much could be compressed, frustrating choice when the final size is not the desired one, I wish to find a way to use the VAF file for the proper Q.Factor(that will make it a 2-Pass method)

2-Pass VBR is great in filling up your discs and distributing the bitrate, but it fails when the average bitrate is below certain point especially with full screen movies, was my only choice until D2SRoBa came alone, but I'm going back to 2-PVBR out the frustrations with OPV on some of my lastest encodes.

[r6d2]

Originally posted by DDogg
[B]Erm, I not so convinced that the quote above makes a very compelling argument and that the next quote would even logically follow:But, I also don't disagree with it or you.
You still don't buy it, do you? :) Well, CCE is not likely going to answer, but perhaps god will. Why don't you ask him and tell us?

You seem to think that OPV and VBR are very different animals. Actually I think they are not that different in nature, just different roads to get to Rome.

Encoding is always about bitrate. Frames get low quantization when they have enough bitrate and high when they don't.

The problem an encoder faces is how to give each frame the right amount of bitrate to hopefully have the least quantization possible.

When limited in space, as we are, if you start with VBR the encoder just follows the instructions given:

First, it does a VAF creation pass in CBR mode, using Avg as the BR.

Then, maintaining an average Avg, limited by Max and Min, move the BR slider according to Bias on each frame to get quantization as a result. On each additional pass, the encoder will subsequently reuse the information on the VAF file to relocate BR where it is most needed, converging to an optimum where it cannot be further improved. In this process the encoder may completely ignore the previous pass MPV. All it needs is the VAF file.

This process at the end tends to equalize the quantization of the output.

If on the other hand you start with OPV, the encoder fixes the quantization level first and then calculates the corresponding BR for each frame. In our case, OPV is also limited in space, since we are trying to get an exact size, the same size we expect on Multipass.

At the end, both methods converge to the same ideal result. Just OPV+1VBR does is faster.

If you still don't by my stuff, rest assured that I did not make it up ;). Reading the Appendix A of the CCE manual is enlightening!
I do think the alternate discussion point could be made that OPV has the ability to create the VAF because sizing in OPV is a known problem and they added the ability to reuse the VAF because of it.
Certainly. But you don't need to reuse the VAF or do more passes when you are not limited in space. In this case you don't limit the Max to mere SVCD specs. You can go for 9800 if you want. There is a market for OPV, and that’s why it exists.

Suppose you are the kind of guy who wants to encode always at the same quality, regardless of space constrains. Let's say you have a big disk and like to collect movies as you collect MP3. Or let's say you want to fit several SVCD-frames movies on a DVD, (a big floppy disk).

Suppose you decided that Q.factor = 30 is good enough for you. (Actually, you should decide which Q you want depending on the output device you'll use. For regular TVs, 40 would still be great IMHO).

In this case, OPV is the way to go.
Your points above seem to combine pure virgin quantizer based OPV with bitrate based VBR as if they were the same thing and that is where I am hung up as I don't think they are.
Well, we don't think the same, you see, and I don't want to turn this into a religious topic. But the logic of my reasoning is here. The CCE manual too. If anyone can tear it apart, welcome. I like to learn.

It would seem to me that the *possibility* that the second sizing pass could negatively impact quality, albeit a minuscule amount, could also be true very easily.
It does, certainly. Remember it's all about bitrate. When the OPV pass oversizes, the BR allocated (bits/duration) is higher than the target size, hence the second pass will have less BR and hence higher quantization.

The VBR pass will have to shrink a bit the output to get the expected result. But then again this has nothing to do with the nature of the second pass. It has to do with the imprecision of the sample.

Let's try to see it more clear. Suppose a Q exists which will deliver the exact output file size you want. (This is a loose assumption, since unlike TMPGEnc, Q.factor is an integer and not a rational number. But let’s suppose there is one). Then a fairy godmother sends you an e-mail telling you that Q for the movie at your hands.

You do the OPV with this Q, and you end up with the file size you expected, right? Then the second or subsequent passes would not improve the quality and are not needed at all!

So, if you know the real Q for the movie, the best possible output quality is OPV. The second VBR pass is for us full-disk freaks only. Best of both worlds.
r6d2, I wonder if you would then refute this: Assuming a Quantizer based encode and a 2 pass VBR encode of exactly the same size and time duration I believe the Q based encode will yield the same, perhaps better, but never less quality than that of the 2-pass VBR. If I understand what you said above, I think you are saying the 2-pass VBR would be inherently better? That it *must* be better?
Absolutely not! I never said that! I cannot refute something I agree with! A pure Multipass 2 will be lower in quality to the OPV, since the VAF creation is done in CBR mode!

I just said that with OPV+1VBR you get the expected quality much faster that pure n-pass.

Gee! I think we should include this whole thread in the Complete Idiot’s Guide. :D

[DDogg]

Well I did ask Bach and he replied:

"I agree with you, but I don't think you will introduce a noticeable damage. It would be mainly a waste of time."

Which really was all I was saying. My position is a very simple: that the second pass is a waste of time unless needed for sizing. and that seems to be echoed in your quote which is what I thought we were discussing:So, if you know the real Q for the movie, the best possible output quality is OPV. The second VBR pass is for us full-disk freaks only. Best of both worlds.

[r6d2]

Originally posted by DDogg
Which really was all I was saying. My position is a very simple: that the second pass is a waste of time unless needed for sizing.
Can't argue with god ;), but I agree with you. Just remember: it's all about bitrate.

The conditional second pass we've talked about if wasted space is X% or more (in undersized OPV) will improve BR in X%, therefore lowering Q. But since the OPV already gave you at least the quality you expect, it's just a matter of whether you want to leave money on the table.

The question you have to ask yourself is how much BR (What's my X%) are you willing to waste at the cost of a avoiding second pass, which with heavy filtering can indeed be a PITA.

On oversize, the second pass is mandatory.

[DDogg]

Just remember: it's all about bitrate.OK, good, I think we have converged (more or less) in that we both agree the second pass on undersize would, theoretically at least make the encode slightly better as it is in fact increasing bitrate.

Conversely, the second pass to correct oversize must by its nature, negatively effect quality (very slightly) as it is reducing bitrate.

From a practical standpoint, in either of the situations above, the effect on quality is minuscule and more from a theoretical standpoint than anything that could ever be noticed.

All the above assumes a very small correction in the range of 5%-10%.

So hybridizing RoBa to only conditionally do the second pass on x% sizing error is definitely the way to go and gives the speed oriented folks what they want in that they can elect to accept a user configurable x% sizing error (whatever they can live with) as the trade-off for not doing the second pass. People with faster machines, excess time, or hung up about full disks can set that error much lower or in fact to 0%.

r6d2, good discussion and reality check. I just wanted to make damn sure we had it right before asking tylo or D2S to implement this. Anyway, I would imagine either of them would have an option to always do second pass for those folks that thought it necessary.

[telemike]

Bingo!

If I set the error to 25% for gross errors it would be just what I need!


So hybridizing RoBa to only conditionally do the second pass on x% sizing error is definitely the way to go and gives the speed oriented folks what they want in that they can elect to accept a user configurable x% sizing error (whatever they can live with) as the trade-off for not doing the second pass. People with faster machines, excess time, or hung up about full disks can set that error much lower or in fact to 0%.
:)

[tylo]

Nice discussion, guys, and thanks telemike for starting this. I've started implementation, but I also have some bugfixing to do. In addition, there is a need for some GUI overhauling, so it will take a few days.

Implementation thoughts:
- We don't need the 'X-pass RoBa (D2S)' mode anymore. The '1-pass RoBa' mode will cover it, because it will do the conditional extra pass. Should we call it '1-2 Pass RoBa'? Suggestions?

- I think we need both 'X% below' and 'Y% above' limits for the conditional. Many will tolerate higher overestimation than under.
Underestimation (OPV's MPV is shorter than the target size)
r6d2: "Don't underestimate me, I'm bigger than you think". Did you mean Overestimation? ;)

/Add: Dont think this was up in the discussion: Would it in some cases be better to do the conditional extra pass in OPV-mode with an modified Q? At that point we have a good idea how much one Q step would change the final size?

[homerjay]

hi tylo - i for one would still like the vanilla 1 pass left as it is with maybe a radio button for the conditional second pass for those that want it

i use an adjust q of -0.8 to guarantee a 1cd conversion of up to an hour and 40 minutes

cheers

hj

[DDogg]

/Add: Dont think this was up in the discussion: Would it in some cases be better to do the conditional extra pass in OPV-mode with an modified Q? At that point we have a good idea how much one Q step would change the final size? I've been thinking about this deceptively simple little question you bring up :) I don't think it would be a good idea to substitue this for the standard vbr sizing pass but this might be an interesting optional method to help you/us build some more data.

It is certainly intriguing, but only if you could get final file size quite accurate. You said, "At that point we have a good idea how much one Q step would change the final size". I would ask you how close do you think you could get it? 1-2%?

[tylo]

@homerjay: Sure, I meant that. It will be possible to force only one pass.
@DDogg:
You said, "At that point we have a good idea how much one Q step would change the final size". I would ask you how close do you think you could get it? 1-2%?
We know how large the sampling error is for the chosen Q: real mpv size / estimated mpv (from test sample). I think it is correct to assume that this sampling error factor is ~equal for all the samples (i.e. independent of Q). This means that for the other samples (e.g. for Q's close by), we can compute pretty dead on how big the real sizes would become:

Example Q=30:
es30: estimated mpv size
rs30: final real mpv size
cf = rs30 / es30: error correction factor ~equal for all samples

Now, if we during Q-search made a sample for Q29, we have :
ces29 = es29 * cf. (ces: corrected estimated size).

I dunno how close ces29 will be to the real size (rs29), but I think very close. You can try it manually.

Another thing:
For a two CD encode, one Q step means:
for Qs around 32: about 20MB
for Qs around 16: about 45MB
(Please check yourself)

This means that for Qs about 30, we can achieve as little as 10MB below full last CD on average.

From your quantization discussion, an encode that e.g. corrected Q with 2, would be even better than using VBR with exact bitrate as second pass, even though the OPV is not completely filling the last CD (but very close).

[telemike]

I would like to see as the conditionals for a second pass:

If it will go oversize by 10% (if less than 10% it will cut credits at muxing) and if undersized by %25 then fill the disk.

[DDogg]

Ultimately I hope you might consider both as an option so we can get some real world data. We always wants more and more, don't we? :)

GUI
Sizing Method > Adjust Q, Adjust Bitrate

[tylo]

OK, the Sizing Method, with 'Adjusting Q' won't come in the next release, but the 'Adjusting Bitrate' will. Actually, it's going real well - conditional sizing pass works like a charm :D. I've ironed out a few more bugs, and done a few smaller fixes that makes the operation go real slick (not all apply when using CCE 2.50):

- It only breaks D2S once, as it used to. (even slicker too) - the additional break I intoduced also caused some problems.
- You can now minimize DVD2SVCD & CCE without problems
- No longer uses MouseClick emulations (was less robust)
- Simpler : only two modes (RoBa or bitrate tweakin)
- D2S does not restore 'MN GOP' value when switching back to SVCD from DVD output (I have reported it). The plugin fixes this for now.

Bugs fixed:
- A bug that caused DDogg to think he'd smoked crack.
- I think I found the sizing bug too, DDogg :). You'll just have to try the new version - probably tomorrow. Only left now is the GUI...

BTW: I tried to add a 'Silent' mode, i.e. no Status window, and D2S & CCE going mimimized. The only problem is that EclCCE cannot be started minimized in batch, although he has support for it through the GUI. /Edit: I will ask RB if he can add a '-minimized' command line option.

Cheers

[DDogg]

tylo, you are making me mad! :) Everytime I think I am about to "kick" this encoding addiction, you come up with more good stuff!:D