D2SRoBa Plugin

[DDogg]

Tylo, I have always wondered why you do not offer an option to do a AvBR VBR pass at the end of your process as a conditional based upon user selected criteria. I know my thoughts could not be original so I assume you have it in your mind.

When CCE ends you know, or could know easily, how big the final video encode is as well as the size of the audio and a close estimate of the muxing overhead, so it follows that you know whether the final product is going to be under/oversized and by how much.

It seems then that offering an option based upon a preset error threshold set by the user would be useful. One additional pass would then make the size exact, and in fact would then be the most efficient hybrid RoBa process as it would only be invoked when needed.

Repeating myself, if the prediction was accurate, the sizing pass would not be invoked. If the size is off by XX%, then the time of the one additional run is much less than a complete redo with all the predictive passes and would be part of the same automated process.

Pardon me if this has been brought up directly before. My eyes are about shot, I am afraid.

[tylo]

Tylo, I have always wondered why you do not offer an option to do a AvBR VBR pass at the end of your process as a conditional based upon user selected criteria. I know my thoughts could not be original so I assume you have it in your mind.

Indeed I had, but it slipped my mind again. I've only thought about this beeing useful when the video gets oversized, but surely it's useful when it undersizes too.

Actually, this idea fits nicely in with r6d2's request for making support for the internal DVD2SVCD RoBa feature. For that, I thought of changing the 'Normal Multipass' mode to 'RoBa Mode', or something.

BTW: Would a name change from '1-pass VBR RoBa' To '1-pass Bach' be more appropriate? If you wish, please elaborate more on how you imagine the GUI for all this (naming and boxes).

Now I just need to find the time to do some programming too!

[DDogg]

Well, I have not thought it out very well at all. Just off the top of my head you could change the normal to "Final Sizing Pass" then add a drop down with "On Oversize Only", "Always","Never", and 1,2,3,4,5,6,7,8,9,10 as a percent. I would probably type in 2.5 if that was allowed.

Nothing wrong with 1-Pass RoBa in my mind. Also, I think it would be cleaner for you to do the extra final pass as you do the others instead of invoking any D2S pass.

[kru]

Thank you for a great program Tylo

Here is some suggestions:

-What about encode the audio after the video?
(If there are space left on the CD, the audiobitrate will be greater)

-And a "auto choose bitrate" for the audiobitrate??? (That is about 10 % of the audiobitrate)

-And what about if the videosize is n % less or greater of the calculated filesize, D2SRoBa will encode again, and try too make a better prediction???

[DDogg]

Tylo, some of my index thoughts have started to crystallize while thinking about the feedback I received and while making my reply to Barker here: http://forum.doom9.org/showthread.ph...256#post359256 which shows the bitrate has little to do with the quality of the final encode whereas the compressibility has everything to do with it.

I now think the AI thing I was looking for in your file-size methods could be easily achieved if you would put some special test settings in the INI which basically allowed:

AISKIP_TESTMODE_ON=TRUE
AISKIP 1CD Q-32PassKB > xxxx
AISKIP 2CD Q-32PassKB > xxxxx

Just to be clear, If the Q32 pass was bigger than xxxx then you would instantly negate any more predictive Q passes and increment target media by 1 (meaning 2 becomes the target), if bigger than xxxxx then you would increment target media by 2 (meaning 3 becomes the target). Hmmm, am I getting fuzzy again?

This would save a ton of predictive passes for the file size based methods for the advanced user. Actually, eventually if this pans out, I guess I am asking for an "AI" mode to go along with the AUTO and the "?". For testing purposes, we could just do this in the INI and set the GUI to "1" so as not to cause you a lot of work. What are your thoughts?

[Add] Hmmm, the sample size is a slight bit of sand in the oil. I guess people using this method would understand they would always have to use the same sample size for this to work, or unless you think it viable to extrapolate from a 1% in the INI to whatever sample size they used. I would think that would have a bigger error margin though.[/Add]

[tylo]

I now think the AI thing I was looking for in your file-size methods could be easily achieved if you would put some special test settings in the INI which basically allowed:

AISKIP_TESTMODE_ON=TRUE
AISKIP 1CD Q-32PassKB > xxxx
AISKIP 2CD Q-32PassKB > xxxxx

Just to be clear, If the Q32 pass was bigger than xxxx then you would instantly negate any more predictive Q passes and increment target media by 1 (meaning 2 becomes the target), if bigger than xxxxx then you would increment target media by 2 (meaning 3 becomes the target). Hmmm, am I getting fuzzy again?

No, this is pretty clear to me, and very good idea! In fact it's so easy to put in that i'll do that right away!

This would save a ton of predictive passes for the file size based methods for the advanced user. Actually, eventually if this pans out, I guess I am asking for an "AI" mode to go along with the AUTO and the "?". For testing purposes, we could just do this in the INI and set the GUI to "1" so as not to cause you a lot of work. What are your thoughts?

I don't see how it can be used for the AUTO mode - only the "?". The AUTO mode computes the num CDs directly from the WorstQ-PassKB size - it doesn't test each num CDs...

Hmmm, the sample size is a slight bit of sand in the oil. I guess people using this method would understand they would always have to use the same sample size for this to work, or unless you think it viable to extrapolate from a 1% in the INI to whatever sample size they used. I would think that would have a bigger error margin though.

You are talking about the sample percentage I assume? Yes I think it is viable to just divide the size of the actual percentage used (like normalizing) before testing. It will actually have a lesser error margin (e.g a 2% sample test size / 2 is a more accurate 1% sample pass size, that a true 1% sample pass produces...)

BTW: I am just about to release a new intermediate version now. It has a much improved "?" function already. Adds an extra CD if oversizing before mux and cut (much requested). Also reduces and equals out image sizes if it cannot fill CDs completely.

[r6d2]

Originally posted by tylo
No, this is pretty clear to me, and very good idea! In fact it's so easy to put in that i'll do that right away!

@Tylo
It is easier than you think, I guess. It is already implemented (sorta). I PMed you on this on Aug.13. I post it here now for if there is anyone who whises to try.

My other idea is this. You set your desired Q as worst_q. In my case, 30. You set your really_worst_q on 44, in my case.

When conducting a binary search in auto mode, you start by searching with target=30 (30 is the maximum of your binary search).

When in range mode, or in "?" mode, you only search on the [1, 44] range.

With really_worst_q enforced like this, you just have to put max and min to the spec values [300, 2530], and you'll find the Q faster. More important: the Q will always be adequate to your taste and the outcome BR is just a result you'll have to trust even if it is ridiculous! The max and min parameters can still be used, but this approach completely confines them to spec values.

[DDogg]

It is easier than you think, I guess. It is already implemented (sorta).

Yes, I see what you mean. Also, if one wanted to accept the programs choice of the number of CDs they would be using the Automode anyway. While I still think the technique I mentioned above has merit, I don't think it belongs in Tylo's plug-in, certainly not in sizing mode!

r6d2, changing the subject slightly, do you agree that the second conditional sizing pass is highly desirable? If so, I hope it is implemented as soon as it can be.

[r6d2]

Originally posted by DDogg
r6d2, changing the subject slightly, do you agree that the second conditional sizing pass is highly desirable?

Absolutely. I had suggested some weeks ago to go for a second OPV pass, conditionally:

Quoting myself
You might even improve the method by using the current binary search for a first OPV pass, and then correct with the real Q for the whole movie and obtained BR, using a simple proportion.

Then you do a second OPV with much more precision, and filling the CDs to the edge or getting really, really close (it would be a 100% sample, so to speak ).

But probably it would be better to stick Multipass, as you suggest. It is 0% risk of odd-sizing and with comparable quality if you reuse the VAF.

[DDogg]

Yeah, I remember trying to figure out what you meant when you posted that, but stopped when my eyes started crossing

Like you said, I think the standard BR based pass for the second pass is really the easiest way to get where we want to go.

[tylo]

Ok, I actually started to implement the AISKIP thing, but I suddenly realized that I was wasting my time. (I should have learned by now not to jump on things too fast). Setting such limits translates to the same thing as setting a really_worst_q ultimately - just as r6d2 saw.

I must confess I didn't take the time I should have to understand r6d2 suggestions. The Aug.13. post is the sensible way to implement the "?" functionality. As you (r6d2) suggested in another post (I think), we can start a test with the really_worst_q, and use the size found as sort of AISKIP limits. This is basically what is done with worst_q in Auto mode, but here we go for the optimal (realistic) number of CDs directly, giving no user choices.

- On the other subject, I was thinking of supporting an optional second-pass (multipass), reusing the vaf file, as you mentioned.
- Then support for the internal RoBa method (but finding a good initial Q by binary search).
- Third. If I'm not fed up with the plugin, I may put audio encoding after the video encoding. We can adjust the audio bitrate in various ways, depending on the size of the video created.

New subject: I am thinking about removing the 'Bitrate Tweaking' fuctionality. It doesn't really belong here, and I don't think many is using it. As I have said before, cutting credits with frame selection (or chapters) is often a better alternative. Couldn't the tweaking also be done by an Avisynth script? E.g: Add a filter that make the bitrate drop after a certain number of frames?
I need feedback on this one.

[DDogg]

- On the other subject, I was thinking of supporting an optional second-pass (multipass), reusing the vaf file, as you mentioned.

That adds a lot of bang for the buck as, I think, it would only take a small amount of effort? With the conditional, it turns the plug-in into a true RoBa process, but one that is hybridized as the pass is only done if needed.

- Then support for the internal RoBa method (but finding a good initial Q by binary search).

I think this calls for some thought and discussion. First, I would suggest we forget the plug-in for a moment and concentrate discussion on any weaknesses of the Internal RoBa process of D2S. In other words, is it broken and do you need to fix it? If it is broken in any way, who can best fix it, D2S or an external Plug-in?

Isn't it true that the only weakness of the D2S internal process is if the equivalent bitrate of the initial Q for the OPV pass is over 50% off from the bitrate actually needed? Yes, I am not very clear on that. If that is so, I would think if D2S did just one short measurement pass that would no longer be an issue. [edit] Here is the quote from Bach on that subject

-- Decision: Calculate video average bitrate (CAVG) necessary in order to fit 2 CDs.
c) do multipass reusing vaf from first pass and using CAVG.
WEAKNESS OF ROBA: if CAVG > 2*(1st_pass_mpv_size/time_length) or CAVG < 0.5*(1st_pass_mpv_size/time_length), you MUST recreate the vaf file in order to do the multipass encoding.

[Add:] This brings up a question. For discussions sake, let's say an encode is correctly sized with a OPV Q of 48. Let's also say we have set the first Q pass in D2S internal RoBA to 32. The second bitrate based pass will size it correctly for us and the bitrate will be equivalent to the Q 48 pass. The question then is: would we gain anything if we had used the Q 48 as the Q of the first pass in the internal D2S RoBA process. My understanding is we would not, and in fact may lose quality although it may be the same. Anybody got any comments on this?

- Third. If I'm not fed up with the plug-in, I may put audio encoding after the video encoding. We can adjust the audio bitrate in various ways, depending on the size of the video created.

No comment except to suggest you may be opening a can of worms for yourself and you might end up losing all your hair (assuming you have any ).

I am thinking about removing the 'Bitrate Tweaking' functionality.

Maybe some folks use it. With the D2S frame selection so easy to use, I certainly don't and can't imagine doing so. I do understand your inferred suggestion that some users might not like to cut any of the credits and might prefer to just use a lower bitrate on them. Sounds good, but would it really be worth the effort and would anybody actually use it? Dunno.

What the bitrate tweak section does do is use up a lot of valuable "real estate" on your interface. That space might be better used to more fully differentiate between the two radically different methods available in your plug-in (Quality based and Sizing based). I know that caused me a lot of confusion and I am sure many others.

[r6d2]

Originally posted by DDogg
This brings up a question. For discussions sake, let's say an encode is correctly sized with a OPV Q of 48. Let's also say we have set the first Q pass in D2S internal RoBA to 32. The second bitrate based pass will size it correctly for us and the bitrate will be equivalent to the Q 48 pass. The question then is: would we gain anything if we had used the Q 48 as the Q of the first pass in the internal D2S RoBA process. My understanding is we would not, and in fact may lose quality although it may be the same. Anybody got any comments on this?

What OPV does, attempting to quote a Bach post which unfortunately was not recovered, is this:

1. Encode a frame with desired Q.
2. If resulting BR > MAX, forget Q and BR and adjust to MAX.
3. If resulting BR < MIN, forget Q and BR and adjust to MIN.
4. Repeat from 1.

I tend to think from the above that using the correct Q, or a nearly correct Q from the start will give you a better encoding, since there would be less "forget Q" frames.

Again, I may be terribly wrong. But I invite anyone to rip apart this logic to continue for the search of the Holy Truth

What the bitrate tweak section does do is use up a lot of valuable "real estate" on your interface. That space might be better used [...]

@Tylo, I think (IMHO) the credits stuff adds generality to the tool. If you are concerned about keeping the GUI clean, you might want to implement an "Advanced" button, or even better: follow D2S's equivalent parameter.

Don't worry about using a bigger window either to accomodate for more optios and modes. DDogg and I will give the users, sooner than later, a good guide for no brainers, whose only requisite is being able to understand what you read.

Besides, even though frame selection does the job, it requires you to wait for DVD2AVI to finish, and that is not ideal for multiple batch encoding.

[DDogg]

I tend to think from the above that using the correct Q, or a nearly correct Q from the start will give you a better encoding, since there would be less "forget Q" frames.

This then becomes the pregnant question that we have to get the right answer to. I sent off a PM. Maybe I can get an answer and/or opinion.

Don't worry about using a bigger window either to accomodate for more optios and modes. DDogg and I will give the users, sooner than later, a good guide for no brainers, whose only requisite is being able to understand what you read.

Disagree . Docs have to be read. An interface should be intuitive and not require reading to understand basic functionality, in this case that is defined as two completely different methods combined in such a way as to not be clearly differentiated.

[r6d2]

Originally posted by DDogg
[B]This then becomes the pregnant question that we have to get the right answer to. I sent off a PM. Maybe I can get an answer and/or opinion.

Send my warmest regards to God when you see him

Docs have to be read. An interface should be intuitive and not require reading to understand basic functionality, in this case that is defined as two completely different methods combined in such a way as to not be clearly differentiated.

Disagree . For instance, D2S interface is not intuitive. You must read a guide to learn to tweak the tool. On the other side, D2S (him) managed to set it up in such a way that if you know nothing, dislike to read the docs and even have a hard time with readmes, the default values will do a great job anyway.

Software Engineering must balance usability with powerness. The KISS concept is crucial for starters, but you must also take into account power users and the evolution of your tool. An approach with Advanced mode seems to me the best deal, just like D2S (him) chose to do.

[DDogg]

Software Engineering must balance usability with powerness. The KISS concept is crucial for starters, but you must also take into account power users and the evolution of your tool. An approach with Advanced mode seems to me the best deal, just like D2S (him) chose to do.

I don't disagree with that at all and the advanced tab is a good option if Tylo feels like messing with it. However, you said it yourself. "...Kiss is crucial for starters". When you have two completely opposite methods (Quality based and Sizing based) mixed together "without notice", so to speak, you do not even remotely adhere to the KISS GUI system that basically says 60-75% of the functionality should be readily apparent, usable and understandable within 5-8 minutes of first use. Anyway, Tylo can take the suggestions or leave them, so let's not beat it to death and get too off topic

[r6d2]

Originally posted by DDogg
When you have two completely opposite methods (Quality based and Sizing based) mixed together "without notice"

I agree with you here on the different methods stuff, they are for different audiences. I tend to think that starters are quality oriented (you are just too affraid to screw up your encoding). Then, when you get more confident, you get also mode daredevil and begin to be cheap with CDs.

If Tylo would like to take this approach, I'd default to quality based mode in the eventual new GUI.

[tylo]

I'll respond later. Here's a new beta release.
http://home.no.net/tylo/setup_D2SRoBa248beta.exe

/Edited:
Nothing new in the GUI, or second-pass stuff yet, but:

- creates extra image for the oversized part (often only credits)
- if bitrate > max: equals out images sizes, shows fillrate.
- "?" mode: uses 'really_worst_q' to estimate lowest numCDs to test.
- auto mode: searches only within 0 .. Worst Q
- removed N-M ranges (replaced by "?")
- removed special high/low bitrates limits for "?" => uses D2S min/max
- default really_worst_q=128 now (maybe too high)

Not well tested, so beta - please test.

[DDogg]

r6d2, here is the answer from god (slightly edited)

The quote [from r6d2] is correct. I have tested your hypothesis at the time I formulated the nowadays so called 1pass-Bach method. My conclusion was that the quality of the encoding increases when I do the first pass of (internal) RoBa method using the Q value closer to the correct one. The difference is not high, though. It is most noticeable when you have a lengthy still scene.
The reason? Well, I think only the CCE coders themselves could explain with 100% of sureness (is there such word in english?), but r6d2's guess (the "less forget Q" thing) is a good guess.

Anyway, CCE's manual gives us a hint: "if you change the average bitrate of a given encode by twice or half of its previous value, it is mandatory to create a new .vaf file".

In your example, lets suppose the calculated bitrate for Q=32 is 1.5 times bigger than the target bitrate (the one which you achieve with Q=48). You are not in the region which CCE's claims you should discard the vaf file. However, since there is a previous_bitrate/target_bitrate ratio (<0.5 or >2.0) for which the already created .vaf is simply useless, this means that the farther you get from the previous_bitrate/target_bitrate ratio=1 , the worse the re-usage of your .vaf is. [edited sentence structure - hope I got it right]

This most probably is not a linear correlation, so that if ratio is something between, say, 0.9(?) and 1.1(?) it is not possible to realize any loss.

[r6d2]

@Tylo,

Your safe zone stuff made me start thinking about the application of Newton's method to find roots for an equation and its possible use to D2Sroba finding of Q. Here you are.

We have bitrate as a function of Q, BR(Q), a function which is unknown to us, but we suspect it to be linear in some range and curve in others. What is clear is that it is a continuos function and with a negative slope.

Let's define

f(Q) = BR(Q) - Target_BR

Our goal is to find Q for which f(Q) = 0. Let's define x = Q.

Newton's method states that you can find a root by iterations with the following formula:

x(n+1) = x(n) - f(x(n)) / f'(x(n))

where f'() is the derivative of f(). Since we don't know f(), we cannot calculate f'(). However, f'() can be aproximated by:

           f(x(n)) - f(x(n-1))
f'(x(n)) = -------------------
              x(n) - x(n-1)

If we replace the above formula in Newton's, we have that:

                   x(n) - x(n-1)
x(n+1) = x(n) - -------------------  * f(x(n))
                f(x(n)) - f(x(n-1))

This method requires you to do 2 blind searches before iterating (for instance, n=1 with Worst_Q and n=2 with Really_Worst_Q). Then you stop when the error, calculated as |f(x(n))| < E, is adequate. E can be set to 1% or any value you choose.

It converges very fast. I did some tests using as seeds x(1)=Worst_Q and x(2)=Really_Worst_Q, and the results are incredible. I found the target Q in 4 tries, no matter what seeds I fed in.

I hope this is not too technical, you might want to try it yourself and hopefully implement it instead of binary search. If you have questions please don't hesitate to ask.