Yet another discussion on the uniqueness of RoBa

[DDogg]

Note: Thread split. Please note the read counter is not accurate as a split thread inherits the number of reads from the one it was spawned from.

Originally posted by bobwillis
it looks excellent considering the average bitrate is only 2.66Mbps! [kidding around with bobwillis a little] erm, this may kick off a discussion. I hope so :-) (if so I'll move it to a new thread)

Average bitrate is irrelevant to quality in CCE quantizer based encodes. The source complexity solely dictates the amount of quality that can be "held" in a given bitrate. Hey, Bob, whatcha think of 'dem apples?

[r6d2]

I'm pretty sure I read a reply from bobwillis to DDogg's riddle, and now it is not here. Forum problem or did you delete your post, bob?

[bobwillis]

I deleted my initial post, since I was (still am) confused by DDogg's dare I say it English :D

Hi DDogg,

I agree with the statement in bold wholeheartedly. For instance, some material that is long (> 3hrs) and therefore low bitrate (<3 Mbps) looks garbage e.g. Lawrence of Arabia, Shackleton etc. These were high Q encodes (significantly over 40). However, other long material for instance JFK directors cut looked great (Q was in the thirties).

Quote, "The source complexity solely dictates the amount of quality that can be "held" in a given bitrate."

From the above quote, and my 'Schindler's list' experiment, I infer that KISS reduces the 'source complexity', and it seems to do it without being detrimental to the image.

As for 'dem apples, I prefer bananas :D

Regards,
Bob

[DDogg]

I prefer bananas I still forget my friendly and mischievous colloquialisms do not always translate well. Anyway, I don't speak English, only Texan :-)

The statement in bold was part of a guide I had started (and since deleted) for novices who had expressed the desire to understand the finer points of why OPV is radically different from Multipass. That part is not too difficult to explain.

What is difficult to explain is why the RoBa process used by D2SRoBa, which happens to use CCE-OPV as part of it, is just as radically different from standard CCE-OPV and is far more than just a filesize prediction method bolted on to CCE OPV (which is what a lot of people think).

I thought starting a discussion might somehow help me solve my problem of expressing these differences. Bob, you, R6D2, and manolito are real good guys to discuss things with :-)

Here are a few more discussion points if you guys are interested. If so, let me know if you agree with this set of facts:

1> As a byproduct of the prediction pass, the process actually "sees" and measures the complexity of the source in advance of the encode. Because of this unique forward knowledge of source complexity, the prediction pass can tell us not only the predicted filesize/average bitrate, but also the predicted quality, in advance of the full encode.

2> Filesize is changed by directly adjusting the level of requested quality, not by directly changing average bitrate.

I think these are fairly clearcut. But I am bogging down in the following #3. Its kind of a mouthful, to say the least.

3> Although average Bitrate is changed as a natural byproduct of quality adjustment, the average bitrate is largely irrelevant to quality. This is because only the complexity of the source dictates the amount of quality that can be held by any given bitrate.

4> Primary description of available CCE process:

Multipass - Convert source into given size. Quality is secondary to size.
OPV - Convert source into a given constant quality. Size is secondary to quality.
RoBa - Convert source to a predicted quality within a predicted size. Size is secondary to quality via an approximate variance of +/-2.5 %

Well see how that sits with you guys. If you decide to reply, look for me to spawn a new thread soon after.

[r6d2]

Originally posted by DDogg
I still forget my friendly and mischievous colloquialisms do not always translate well.Uh...? Then it was not about bananas and apples? :confused:

:D:D:D
The statement in bold was part of a guide I had started (and since deleted)Why did you delete it? For a moment I thought it was a bit too babylonian :) but you did not get any negative feedback, did you?

I wonder if my Idiot's Guide is not a good enough explanation of RoBa or if I should write a Non-Idiots guide to be challenged by guys like jdobbs, a declared multipasser, or the very same Mozart (aka The return of the Jedi) :).

I've tried to talk the less intrusively way I can into the DVD-Rebuilder thread to suggest the benefits of RoBa and CVD encoding of extras, but with little luck so far. I'm pretty sure jdobbs would not read a guide named "Idiot's Guide" ;)
What is difficult to explain is why the RoBa process used by D2SRoBa, which happens to use CCE-OPV as part of it, is just as radically different from standard CCE-OPV and is far more than just a filesize prediction method bolted on to CCE OPV (which is what a lot of people think).Me included, BTW. I think it is an indeed clever, eficient and fast method, but not really that different from plain CCE OPV. It's pretty much like CQ mode in DVD2SVCD with TMPGEnc, IMHO.

In fact I think manolito's requant feature has indeed produced a quantum leap from a plain prediction tool. I even suggested Tylo to think about renaming the tool to D2Srobama. :)
I thought starting a discussion might somehow help me solve my problem of expressing these differences.Be my guest! I'd love to get a different view on the matter!

Your 1, 2, 3 and 4 points are quite correct, but what does it make them so different from pure CCE OPV, when you know in advance the proper Q?

/Add:
Please don't get me wrong. D2Sroba has lots of features not in DVD2SVCD's CQ mode, but my comments above apply to RoBa itself, just like DDogg's points, not specifically to Tylo's great implementation.

[DDogg]

r6d2 wrote - "Why did you delete it?"
Because I was attempting to use a materials analogy to describe some of the benefits of RoBa and decided it was just not working well. I have not given up on doing a novice guide to OPV/MP/Q/RoBa/D2SRoBa, but part of the reason for this discussion is to get a better handle on what it should say.

r6d2 wrote - "Me included, BTW. I think it is an indeed clever, efficient and fast method, but not really that different from plain CCE OPV. It's pretty much like CQ mode in DVD2SVCD with TMPGEnc, IMHO."
Your statement surprises me. Since it is coming from the guy who helped make sense of the Newton Method of calculus used in D2SRoBa's Prediction, it makes we wonder if: A) - I am smoking crack, B) - you are smoking crack, or C) - if you are trying to help set up a better focus on the discussion topic. Either way, you identify the most important point, so I'll try to speak further to it.

RoBa is radically different from standard OPV because, AFAIK, it is the only MPEG2 encoding method that pre-computes the compression index of the source, and then uses that compression index to compute [provide]the most efficient bitrate that will contain the user specified quality level. As a byproduct, the filesize happens to also get predicted.

Gee, r6d2, you mean you did not know that RoBa did all that? :cool: Oh, sure, I guess that is all rather transparent in the process, nevertheless that it exactly what it does. :-D

[r6d2]

Originally posted by DDogg
Your statement surprises me.Well, Newton was a nice addition to speed and precision, but it does not redefine the method, only optimizes it. Other suggestions I did, like the audio after video, the conditional filtering, the CBR threshold, the cut last image setting, the output image equalization of CBR, are probably cute too. Then jonny came up with a prediction enhancer idea which became a new useful parameter. You did yourself some additions also. IIRC, you were the first one to suggest the conditional second pass. (I may be forgetting other contributors/contributions, so please excuse me if I am.)

I also did some suggestions, probably not that good, than never made it into the tool. Like the iteration on the resizer, or the iteration on the frame size, for both of which I provided easy support via FACAR. Also the image equalization for plain undersized OPV did not make it.

All those things were indeed great to an extent, but I don’t think we deserve our initials on the tool name, so to speak. By doing all this, we helped Tylo to optimize the implementation, not to redefine the method. Let me see if I can make my point straight:

[HISTORY MODE ON]
RoBa was initially like this: use Multipass-1 VBR (1 VAF pass + 1 VBR pass) but instead of the simple CBR done by CCE on that VAF creation pass with normal VBR, use OPV with desired quality. That’s it, no more than that.

On another completely different lane, we had Tylo running for OPV to fill CDs. As a result, D2Sroba ended up mixing both approaches and letting RoBa use the most suitable Q, which is something Bach also suggested as a nice to have, but did not materialize in his approach how to get it.
[HISTORY MODE OFF]

So, what we have so far in this analysis is nothing more and nothing less, as far as RoBa itself is concerned, the equivalent of CQ mode for TMPGEnc. Agreed? (please again don’t get me wrong since I’m not talking about D2Sroba’s cuteness, but the plain RoBa method fully implemented as such.)

(I sort of visualize the CAPS mode coming, so I try to be cautious.) :)

So, to answer your question, maybe we are both smoking crack, but this is the story as I recall it.

Then it came manolito, with his compressed domain sizing pass. Now that is, IMHO, a quantum leap. It redefines the method, not the implementation. IIRC, nobody had thought of that, not even Bach, much less Robshot.
RoBa is radically different from standard OPV because, AFAIK, it is the only MPEG2 encoding method that pre-computes the compression index of the source, and then uses that compression index to compute the most efficient bitrate that will contain the user specified quality level. As a byproduct, the filesize happens to also get predicted.You seem to be mixing original RoBa with D2Sroba, but that’s not the point. IMHO there is a huge incorrectness in your statement. D2Sroba does not actually compute the best BR for the encode. It computes the best Q for the encode, provided a minimum Q you want to achieve. BR is just a side effect, an instrument to have the piece fit into adequate media. BR is not even a parameter input to CCE in OPV. It could be called “units of media” instead, or “number of boxes that fit in a truckload” on your materials analogy. What matters is the Q, as much as only CQ is relevant in TMPGEnc.

I know this may seem like just a semantics topic, since actually a stream with variable BR is what we all know is what is ultimately encoded. But, as I expect to have elaborated in this post, is it not a minor semantics discussion, IMHO.
Gee, r6d2, you mean you did not know that RoBa did all that? :cool: Oh, sure, I guess that is all rather transparent in the process, nevertheless that it exactly what it does. :-DWell, as pointed out above, I don’t think I knew that RoBa did what you state, and in fact I still don’t. :)

(Please, no CAPS mode.) :D:D:D

This difference between our conceptions may originate from you being a more size-oriented user of D2Sroba, so BR is more familiar and intuitive to your grey matter. As I am more of an Auto mode user, my brain bark is more related to Q. I don’t know which line of thought is more suitable to newbies. I used mine in my Idiot’s Guide, with positive feedback so far, but as I said before, non-Idiots like you surely prefer a closer and finer look. ;)

[DDogg]

D2Sroba does not actually compute the best BR for the encode. Yeah, r6d2, I think I know that :rolleyes:. Change "compute" TO "provide".

ok, back to the other:

Sure, TMPG CQ mode coupled with size prediction does the same thing. In theory, any constant quantizer based encoding process that pre-measures the compressibility of a source prior to encoding has these abilities. However, from a practical standpoint CCE OPV and TMPG are the only ones I know of (so far) that have enough inherent linearity, and a fine enough range in the stipulated quantizer, to actually be used in this way. That is to say, where one can change the quantizer in fine enough increments to accurately and reliably adjust final filesize.

Of course the point we were discussing, or at least the one that I was trying to get to, is whether a process that pre-measures the compressibility of the source and then uses that knowledge as part of the encoding process, is unique in itself, or whether it just a hybrid form of constant quantizer based encoding. In my opinion, it is a distinctly new and unique process that stands alone because unlike OPV, this process is aware of the compressibility of the source (this seems to be something you are having trouble with). Since no other MPEG2 encoding method does this (AFAIK), in my mind that makes it unique.

One thing I did decide from this discussion - It is not something that a newbie needs to worry about, so I'm not going to spend any more time doing so either.

[r6d2]

Originally posted by DDogg
Yeah, r6d2, I think I know that :rolleyes:. Change "compute" TO "provide".Oh, then please change "incorrectness" to "loose language". :D
In my opinion, it is a distinctly new and unique process that stands alone because unlike OPV, this process is aware of the compressibility of the source Since no other MPEG2 encoding method does this...Originally posted by r6d2
but what does it make them so different from pure CCE OPV, when you know in advance the proper Q?Originally posted by DDogg
... this seems to be something you are having trouble with.Am I? :confused:

[DDogg]

I don't think spending any more time on this is going to accomplish anything, but thanks for your time.

[r6d2]

You're welcome.

[onesoul]

Originally posted by r6d2
I wonder if my Idiot's Guide is not a good enough explanation of RoBa or if I should write a Non-Idiots guide to be challenged by guys like jdobbs, a declared multipasser, or the very same Mozart (aka The return of the Jedi) :).

I've tried to talk the less intrusively way I can into the DVD-Rebuilder thread to suggest the benefits of RoBa and CVD encoding of extras, but with little luck so far. I'm pretty sure jdobbs would not read a guide named "Idiot's Guide" ;)

- The bitrate calculating algorithm can now examine the original allocation by Cell and assign bitrates to each cell consistent with that of the original DVD. This essentially uses the original DVD as a "first pass" and gives more bandwidth to cells that need it. This could (theoretically) result in better quality. This "dynamic" bitrate allocation can be enabled by selecting "Dynamically Assign Cell Bitrates" from the "Modes" menu.
What I am trying to point is that although jdobbs not being into roba, this, I think, clever feature he introduced to dvd-rb could decrease the necessary passes to achieve equivalent source quality.

[jsoto]

Hi guys,

May be this is not the thread to discuss my question, but, it seems to me the best one...Anyway, this is the question I'd like to have your expert comments:

I'm using D2SRoBa for the main movie and standard OPV (that means no D2SRoBa, with a fixed Q of 35-40) for the extras, usually in CVD resolution (as you know, similar to SVCD). But in DVD I do not have the restriction of 2756 kbps max bitrate, so I can allow CCE to go over this (up to 8000-9000). And here is my surprise: depending on the movie, sometimes CCE goes up to 3000-4000 average (so, now, I'm applying similar SVCD bitrate restrictions). Please note 4:3 DAR (no borders) and noisy extras are usual in DVDs.

I still have to find time to play with this a little, but, if I understood well, Q factor (which is perfectly provided by D2SRoBa) isn't "the unique quality factor". You need to take into account the bitrate constraints, that means:
- if you encode two different movies at the same Q, you cannot say they are encoded at the same quality.
- you can get the same final size, same input, with diferent Q changing the bitrate constraints.

Any comments?

jsoto

BTW, using KISS the bitrate is reduced significantly. I'm using it by default in all extras (still not in main movie, but...). Thanks, r6d2 for your "Poor Man's DVD" post.

[r6d2]

Originally posted by jsoto
I still have to find time to play with this a little, but, if I understood well, Q factor (which is perfectly provided by D2SRoBa) isn't "the unique quality factor". You need to take into account the bitrate constraintsYou are right. The explanation of this peculiar behavior is in this post (http://forum.doom9.org/showthread.php?s=&postid=360573&highlight=Bach+forget#post360573), quoting in turn one of Bach's brilliant posts which was lost.

In fact, if you restrict Max to specs you'll get the same Q, but with a higher quantization average. As you can see from the quoted post, Q is indeed constrained by Max and Min, which is not a restriction actually with DVD BRs, even though strictly speaking, you can get uncompliant streams if you let BR go beyond 9-Mpbs.

This would be a nice addition to the Idiot's Guide.

(I've not been around lately, but could not resist to pick up the "expert" glove.) :)

[DDogg]

Jsoto, I'm trying to read between the lines of your post to understand what you want to do. Could you state more clearly what your goal is and what you are trying to accomplish?

I have been spending a lot of time thinking about smart bitrate allocation when several encodes must share space within a defined space like a dvd, which I think it what you are wanting to do?

If you get a chance, read this condensation (http://forum.doom9.org/showthread.php?s=&threadid=74141) and look at the spreadsheet. I'll also attach the derived bitrate calculator sheet to this post. You can get a lot of benefits by using an OPV sample pass, deriving the bitrate and then using it in multipass to achieve an exact size target. The average quantizer value are slightly better than OPV and the filesize will be exact.

If you are doing something similar and want to discuss it let me know.

Attachment moved to this thread (http://forum.doom9.org/showthread.php?s=&threadid=74141).

[jsoto]

Thanks both for your links and comments, very useful, as always.
DDogg,

You can get a lot of benefits by using an OPV sample pass, deriving the bitrate and then using it in multipass to achieve an exact size target. The average quantizer value are slightly better than OPV and the filesize will be exact.

I agree, you can get the benefit of both methods with your proposal, but if I understood r6d2, the average quantizer value will be influenced by max bitrate constraint.

But this is not what I'd like to discuss. My "problem" is how to take the best decision in terms of quality/size, mainly in the case of extras. Don't get me wrong, it's something what I can live with, I'm only trying to improve my knowledge.

This is the procedure I usually follow in a 1:1 full backup:
A) Delete some stuff.
B) Encode the extras to keep with Half D1 resolution, filtered with Undot()+ Deen(), relatively high Q and bitrate constraints.
C) Encode the main movie with D2SRoBa and full DVD bitrate. May be reencode additional audio tracks at lower bitrate. See the final Q and if it seems too high restart from the beginning trying to save more space.

My concern is in phase B). My target is to reduce at least 50% and even more (I'm using half D1), but, how can I take the best decission in terms of quality, increasing the Q or reducing the max bitrate?. I guess it depends on the source bitrate requirements(borders/no borders; noisy/clean), but I do not have a clear "method" to determine the parameters to use. I'm using 2500 as max bitrate because I came from SVCD world, but in theory I should use 4000/4500 (full DVD resolution uses 8000/9000).

As you can imagine, this concern is just something to discuss. Final quality on extras is not my first priority, but I've found some (mainly noisy) extras which really very low compressibility, I need a lot of bit rate to get a good final result.

Summarizing, my question is: How to determine the bitrate constraint in Half D1 resolution?, and even more...

Could it be a parameter to be modified by D2SRoBa in the case of SVCD? And in DVD? Does it make sense? I believe not, but the fact is that the final size, or the calculated Q for the same size, are affected.

jsoto

[DDogg]

My target is to reduce at least 50% and even more (I'm using half D1), but, how can I take the best decision in terms of quality, increasing the Q or reducing the max bitrate?. I understand now. It's a very interesting question. I don't know the answer, but maybe we three can put our heads together and figure it out. We can't really attempt that until we can understand how to quantify the measurement of the quality, as I think you are saying average quantizer levels as measured in bitrate viewer will not do this. Is this correct? hmm, perhaps SSIM to compare against the original source to find out if a general method is the answer, or if it is source dependent? <Just thinking out loud>

[jsoto]

I'm convinced it is source dependent. What I imagine is:
- Bitrate constraint is frame-wise (according r6d2 quote). So when it is applied, these frames will show lower quality than the others(which are affected by the Q).
- Depending on how often (during the encode) the bitrate constraint is applied, the final quality perception will be degraded in more or less quantity.

But, IMHO, this effect can be measured using peak and average quantization values.

jsoto

[r6d2]

Originally posted by jsoto
But, IMHO, this effect can be measured using peak and average quantization values.I think you would actually need the distribution to tell.

Please don't get my previous post statement wrong. Q is indeed the most easily manageable quality parameter we have. And it works. BR limits are not frame-wise. I think they have a more spec-abiding purpose.

It is very unusual you need high BRs on half D1s. If they are, then probably the source, Full D1, which is DVD BR compliant, has a high quantization. You cannot improve that bad quality source.

My advice is for extras is: use you prefered Q (maybe a not too demanding Q, like 40, for instance.), and a low demanding frame size, like Half-D1 (CVD). You won't save 50%, but you'll save a lot.

(The comparison table on the Idiot's Guide is made with Q=20, Max=unlimited, for different frame sizes, and a typical source. That should give you an idea.)

Then use all the available BR for the main movie.

In all cases leave Max at spec, i.e. 9-Mbps or so.

By constraining Max in Q modes you are just cheating yourself and don't even realize about it.

CVD is (IMHO) the way to go for extras. Remember some of them are pure interlaced or hybrid, so for them to look good use interlaced resizing and preferably use interlaced encoding. Eliminating overscan and using KISS, as you do, helps a lot too.