Modification and optimization of robshot method and its possible use with DVD2SVCD

[homerjay]

hi telelmike firstly i gave your advise a go and dropped output to a vcd - end result fitted onto 1 cd with 30mb to spare when selecting a 740mb cd size ( THANKS )

but i really want to produce svcd's so i did some more testing and it looks to me like d2sroba isnt calculating the bitrate correctly for 1cd svcd quality i will give egs of my last couple of tests
106 minutes onto 1 800mb with 128kbit audio cd was searching for a bitrate of 843 (bitrate calc says 892)
but when it started encoding it settled on a Q of 63 and a bitrate of 1014 (too high)

so i cancelled out and changed the cd size to 700 just to see what happened overnight
d2sroba log shows

Calculations:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Number of CDs : 1
- Target mpv size : 613427641. (791 kbps)

which is correct but then settles on

Determined Q. : 63
--------------------------------------------------------
Replace values in ecl file:
- opv_q_factor : 63
- opv_brate_max : 915 ==> 2530
- vbr_brate_avg : 6000 ==> 791
--------------------------------------------------------
- Encoding Movie
- 2003-07-24 01:03:41
--------------------------------------------------------
- Actual mpv file size:
24/07/2003 04:19 763151664 Encoded_Video_CCE_NTSC.mpv

hope this info helps

[jsquare]

Same problem here with Saving Private Ryan for 2x800MB SVCDs, I know it's a long movie (2:49) but I can't get D2SRoba v2.4.4 to calculate the correct bitrate, it says target bitrate=1,153 when testing, but when CCE starts the bitrate goes up to 1,650 and end up with a 2nd file that's 1.3GB in size, D2S stops with an error before creating the image for the 2nd CD. I also tried setting the CD size to 700MB with similar results.

Update1:
Tested with VCD resolution 320x240 2x740MB and the bitrate calculation worked fine, but I need SVCD and not VCD settings. It may be a bug, I will try CVD next.

Update2:
Same problem with CVD, CD1=750MB CD2=980MB. The video quality was terrible.

[telemike]

Have you tried the "AUTO" setting and set your worst "Q" level at around 30?

[homerjay]

sorry i think im being dumb
if when it is calculating the Q it settles on 63 for any of the longer 1cd conversions i have been doing
how is setting the Q to 30 going to produce a 1cd encode
i would give it a go but i am in work and currently and left my home pc doing a conversion based on number of cd's 1 sample 3% (have found this helps) and adjust Q of 0.7

*edit 13.10 gmt * well normally when doing a conversion i cant connect in but have managed to and aborted my current attempt at 11% the final size for the mpv before audio added would be 755 mb using a Q of 63 and a bitrate of 928

have set as you have suggested and started it up again ( its now i wish i could recycle the earlier stages up to encoded audio completion )and will report back

[ric1234]

Hi all, here is some feedback on my experience.

I do SVCDs only, PAL/permanent subs mostly, did 6 DVD conversions with D2sRoBa already and overall, quality is really good and the software works very well : Thank you Tylo for it

Small problem however : my last CDs are never filled up, generally I have 750 Mb out of 805 Mb.

I did some Q adjustments but I have to put as much as -7 to fill up the CD, which seems strange to me... Also I tried to increase the movie sample to 4 or 5 % => it does not help... So I was wondering if other people have the same results with D2SRoBa ? Anyone knows how to get closer to fill up last CD ?

Also here is an idea : since there is a risk of overflow, would it be possible, after the final .mpv is created, to check its size BEFORE the "muxing and cutting" part, and if it appears that it is going to overflow last CD, then automatically add 1 CD image for the "cutting" ? This would be great because most of the time, you would just loose some end credits and I'm ok to discard these.

So for instance, you would select 2 CD but end up with 3 images : 805 + 805 + 30 Mb and just discard the last one.

IMHO the most important is to create cd images ALWAYS smaller or equal to max CD size otherwise the whole job is lost...

I have no idea if it's easy or not to do and I know Tylo's on holidays but if anyone wishes to comment...

Ric

[r6d2]

Originally posted by Holomatrix
[B]I have a question on how the '?' feature works. I thought it was supposed to give you an option of picking what Q you want to use if the quesstimated Q was going to be not what you set as worst Q, but when I select '?', my encode just goes all the way through with a Q of 1. Is there away for D2Sroba to automatically select a 2 CD encode if the quesstimated Q is going to higher than what I pick as my worst Q?

@Holomatrix,

Tylo has made extensive explanation on how does "?" work in this thread, but if you're lost please read again the docs (http://home.no.net/tylo/).

Plus it would be good for D2Sroba to ask the user what bitrate to use on a given Q...

I think you are not getting the idea behind D2Sroba. The whole point is you select the quality you want, no the BR or he number of CDs. The program determines both based on you worst Q.

If you're getting weird results, is just because of the old encoding rule: garbage in, garbage out. You have to set up parameters in their range of validity.

I would of liked it if D2Sroba gave me an option to decrease the picked Q and decrease the bitrate on the second movie or poped up a window with various Q/Bitrate option I could choose from based on the movie examination that D2Sroba did.

In "?" mode, it will let you choose only if two Qs will satisfy your requirent. If only one does, it won't ask and just proceed.

@homerjay,

In particular, D2Sroba is not good to make 1 CD rips, since it cannot calculate Qs above 64, and Qs > 60 are not good anyway in terms of quality. Don't use 1 CD rips with D2Sroba, unless the movie is short and you can get a good Q. IMO, Q=30 is a good figure to start testing.

D2Sroba is quality driven.

If you are looking for size and not quality, there are other tools for that, probably using TMPGEnc (Tok, CQmatic, etc.)

Hope this helps.

[r6d2]

Originally posted by jsquare
[B]Same problem here with Saving Private Ryan for 2x800MB SVCDs, I know it's a long movie (2:49) but I can't get D2SRoba v2.4.4 to calculate the correct bitrate

Hi, @jsquare,

I'm under the impression your problem has nothing to do with resolution. I don't know which Q are you using, but if you get a BR of 1100 os so, quality won't be good at high res.

There is no black magic in D2Sroba. It just automates the decisions you would make manually in DVD2SVCD. You wouldn't go for an SVCD of 1100, would you? (BTW, CCE is not good at low BRs, TMPGEnc does a better job at low BRs and using MPEG-1).

Please make sure you're starting from scratch every time you change the res., and pick mode=auto and Q=30. The program will tell you how many CDs you really need. I guess it would be 3.

Hope this helps.

[homerjay]

thanks for your comments r6d2 i appreciate what you are saying but

the results i am getting out despite being slightly too large are very good quality using a combination of filters

i have previously done 4 pass 1cd encodes that compare very highly to multiple cd rips i have seen using this combination but takes over 18 hours so i want to persue this further

hopefully tylo might have some thoughts on his return

* edit * telemike the last settings you suggested just started producing a 2cd rip qith a Q of 16

[Holomatrix]

@r6d2 - "garbage in, garbage out" Yes, that could be what happened. Even though the DVD looks good, I've seen some 3.5 GIG VOB DVD's look better than some 6.5 GIG VOB DVD's.

[r6d2]

Originally posted by homerjay
[B]thanks for your comments r6d2 i appreciate what you are saying but

the results i am getting out despite being slightly too large are very good quality using a combination of filters

Heavy use of filters should increase compressibility a lot indeed. And quality is in the eye of the beholder

But also heavy filtering makes prediction more difficult sometimes. Remember the prediction is based on a sample, and the more filtering you do, the more "unrepresentative" becomes your sample.

OPV tries to keep the quality constant. (Quality here means "most alike with the source", not "absolute" quality.) That may be the reason OPV ends up with a higher file size.

With the use of heavy filtering you are probably reducing the Q a lot, Q again understood as "alike the source" but careful: the source is no longer your original source at this stage, it's your source heavily filtered, a different source. So lower Q will mean "alike my filtered source". That by no means guaranties your source will be "respected".

So, you'll end up with a 1 CD rip of a totally different movie...

i have previously done 4 pass 1cd encodes that compare very highly to multiple cd rips i have seen using this combination but takes over 18 hours so i want to persue this further

Remember that how many hours to put into it will not change the fact that D2Sroba is quality driven. You are aiming for good quality too, but constrained to disk space, so that makes your encodes size driven.

I think you may be using a fork instead of a spoon to eat your soup. You will get it done, but with a lot of effort.

Best regards,

[r6d2]

Originally posted by Holomatrix
@r6d2 - "garbage in, garbage out" Yes, that could be what happened. Even though the DVD looks good, I've seen some 3.5 GIG VOB DVD's look better than some 6.5 GIG VOB DVD's.

Sorry, Holomatrix, I did not mean the DVD was a bad source. I meant maybe you gave not D2Sroba the right parameters... Try as Telemike says, Mode=auto and Q=30.

Q=25 and BR=940 may have another meaning... CCE may be thinking that to you BR is more important than Q.

[r6d2]

Originally posted by ric1234
Small problem however : my last CDs are never filled up, generally I have 750 Mb out of 805 Mb. [...] So I was wondering if other people have the same results with D2SRoBa ? Anyone knows how to get closer to fill up last CD ?

This one should make it into the FAQ .

Ok, let's see. CCE OPV uses a desired Q.factor as input. It does not allow you to specify AVG.BR. So, Tylo uses a binary search to find the Q(sample) for a sample of the movie which will then use to do the whole encode.

This Q(sample), even after binary search, is a guess. It depends on the sample. There is no guaranty it will represent the whole movie. If it does or not depends a lot on the source material. Extensive testing has been made to find a way of getting the best sample possible for any movie, with different GOP sizes, sample sizes, and all these tests have been inconclusive.

The best precision I've got is 1%, but with a sample size that renders OPV useless. I've not given up yet, but this is the state of the art.

But the good side on this is that with the default params, D2Sroba will almost always find a Q(sample) that is higher than the Q.factor (this is because using GOP=1 you have sorta worst case scenario).

So, encoding with that Q, if that Q satisfies you (you may look at the binary search samples on the Video folder), you will get a constant quality throughout all the movie, which will likely not fill you last CD. That is the price you pay for doing just one pass.

Since you're getting the quality you are satisfied with, this should be no problem. Yes, it's not the best quality you could get out of those CDs, but... As Dirty Harry said: You have to ask yourself a question: Do I feel OK paying this price?

Regards,

[ric1234]

@r6d2, well, your explanation is pretty clear, I understand it much better now, thanks.

Originally posted by r6d2
But the good side on this is that with the default params, D2Sroba will almost always find a Q(sample) that is higher than the Q.factor

Ok, this is safe. But now let's say I want to take more risks and reduce the Q so that *maybe* I will overflow my last CD... (for the reasons explained above). What would you think of my idea about the "cutting" ?

since there is a risk of overflow, would it be possible, after the final .mpv is created, to check its size BEFORE the "muxing and cutting" part, and if it appears that it is going to overflow last CD, then automatically add 1 CD image for the "cutting" ? This would be great because most of the time, you would just loose some end credits and I'm ok to discard these.
So for instance, you would select 2 CD but end up with 3 images : 805 + 805 + 30 Mb and just discard the last one.

But this raises one question : which software should do the size check ? D2SRoBa or DVD2SVCD ? Because I wonder if D2SRoBa has not given back the hand to Dvd2svcd at this stage and therefore can not interact with the process anymore... Unless D2SRoBa keeps monitoring the process until the encoding is finished and can squiz a little tweak in the number of CDs before the muxing starts... Or unless @dvd2svcd (who has done so much already) would consider adding this feature in a next version... ?

Thanks and Regards,
Ric

[r6d2]

Originally posted by ric1234
What would you think of my idea about the "cutting" ?

Well, I think it's a great idea. So great, that DDogg claimed he did the very same thing on this very thread, june 12:

Quoting DDogg:
The one pass vbr section of the plugin really works well although I am finding it is difficult to get a completely full second cd. I have been increasing the Q adjust upward to actually force a slight oversize on a two cd encode to cause a third partial cd to be created in the muxing step. That small third cue/bin is normally just a few minutes of titles which I can discard thus allowing the second cd to be full.

Seems that he got the muxed files cut automatically... Did you try it? Doesn't DVD2SVCD cut properly in your case?

Unless D2SRoBa keeps monitoring the process until the encoding is finished and can squiz a little tweak in the number of CDs before the muxing starts... Or unless @dvd2svcd (who has done so much already) would consider adding this feature in a next version... ?

I've asked Tylo to have D2Sroba do some more stuff for DVD2SVCD, for instance, resize the CDs to the expected average size after the encode in order to get equally sized CDs (this is good so you don't end up with a last image of just 15 min).

I don't know if he has had time to do that, since he's on vacation, but yes, D2Sroba may continue to get its hands into DVD2SVCD.

DVD2SVCD has not been updated for a while, I don't know if @DVD2SVCD is still improving it.

Regards,

[ric1234]

Ooops ! Sorry DDogg, that is you idea in fact ! I couldn't remember this post...

Seems that he got the muxed files cut automatically... Did you try it? Doesn't DVD2SVCD cut properly in your case?

Well no, in my case, it doesn't cut properly all by itself.
I tried it twice and each time I ended up with an 800+++ Mb last CD image...

Anyway, I thought a bit further about it and I manage to re-cut the movie after the whole SVCD creation is finished. Here is what I did (that was a 3 CD conversion) :
- First I ran the whole D2SRoBa process with Q adjust = 8. I ended up with 3 images : 805 - 805 - 869 Mb
- Edit "dvd2svcd project file.d2s"
- Found the following lines (don't know which ones are really used) :
[Settings]
Number of images=3
Volumes=3
[MovieInfo]
Images=3
- Replaced 3 by 4 and saved
- Start Dvd2svcd and Recover from "muxing and cutting"
And it worked perfectly, I then ended up with 3 images of 805 Mb plus 1 image of 64 Mb. Luckily, that last sequence was from 10 sec after the start of the end credits !

But I must say Q Adjust=8 is probably too much, I think I should better use Q=6 because that one was *very* close from the loss of the end of the movie... (the sample was 4%, so I assume quite accurate)

Anyway, I really hope the "Adaptive cutting" feature will be added in the future ! That would be great
Meanwhile, if there's an easier way to do it manually, I'd be happy to take it.

Cheers,
Ric

[r6d2]

Originally posted by ric1234
But I must say Q Adjust=8 is probably too much, I think I should better use Q=6 because that one was *very* close from the loss of the end of the movie... (the sample was 4%, so I assume quite accurate)

You really mean 8, not 0.8? Gee! That's pretty unusual. What Tylo does is simply substract this number to the calculated Q (take a look at the log). Normally the Q you need should be around the Q automatically calculated, untweaked. 8 is indeed too big. The only explanation I can find is that the Q was very, very low, and the movie was either maxed out or might have fitted on 2 CDs with a reasonable Q.

[ric1234]

I know, it seems strange to me too to have to Adjust Q so much. This is why I tried on several movies, but I had pretty much always the same results... But again, I think -8 is too much, and I'd better go for -6 (but still it is far from -1.6 proposed in D2SRoBA)

Anyway, here is the log :

Settings:
- CD size : 800
- Number of CD's : 3
- Worst Q. factor : ---
+ One pass VBR Robshot/Bach : 1
- Sample percentage : 4
- Sample GOPs : 1
- Adjust Q. : 8
- Safety Zone : 34
+ Do credits bitrate tweak : 0
- Start time : 0:00:00
- Bitrate : 1000
--------------------------------------------------------
- Movie length : 02:39:42 (239572 frames, 25 fps)
- Audio size est : 383315200. bytes (192 + 128 kbps)
- CD user data size : 833237920. = (800 - 5)*1024*1024 - 380000
- SelectRangeEvery : every 375, select 15 frames
- Num. sample frames : 9585
--------------------------------------------------------
Calculations:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Number of CDs : 3
- Target mpv size : 2059004058. (1718 kbps)
--------------------------------------------------------
- Binary search for Q:
- Estimated mpv size : 2329687909. (Q=32, 1944 kbps, 93208132 sample sz)
- Estimated mpv size : 2057591804. (Q=48, 1717 kbps, 82321880 sample sz)
- Computed mpv size : 2193639856. (Q=40, 1831 kbps, (Q48+Q32)/2)
- Computed mpv size : 2125615830. (Q=44, 1774 kbps, (Q48+Q40)/2)
- Computed mpv size : 2091603829. (Q=46, 1746 kbps, (Q48+Q44)/2)
- Estimated mpv size : 2073434799. (Q=47, 1730 kbps, 82955740 sample sz)
--------------------------------------------------------
- Q Weight : (2059004058. / 24.9944705268649 - 82321880) / (83682662. - 82321880)
- Adjust Q : (48 + 1.0 - 4.15222145488089e-002 - (8)) = 40.9584777854512
- Determined Q. : 40

As you can see, I am in very reasonable Q values and bitrates. With this settings, OPV's running with a Q factor = 40.
A previous test with Q Adjust=1.6 was running OPV at Q factor 47, but I had 50Mb left on last disk. Now at Q=40, the improvement in quality is quite sensible.
Ric

[homerjay]

hello ric1234

just a question for you - what version of cce are you using
i was on 2.50 for above tests but i updated to 2.67 last night and with basic settings it actually estimated the correct size....

could be coincidence - i dont know

[ric1234]

Hello Homerjay,

I am using CCE 2.66... But you're right, this could be a track to explore...
I still have a 2.50, I'll give a try tonight to see how much difference there is with 2.66.

Ric

[r6d2]

Originally posted by ric1234
[B]I know, it seems strange to me too to have to Adjust Q so much. This is why I tried on several movies, but I had pretty much always the same results... But again, I think -8 is too much, and I'd better go for -6 (but still it is far from -1.6 proposed in D2SRoBA)
[...]Now at Q=40, the improvement in quality is quite sensible.

I see your values are "reasonable". However, I have no idea why this happens to you, and since it is on several movies, gee, I'm lost.

I personally had some strange stuff happening once, D2Sroba said Q=48 was good and went for a 1 CD rip. The movie looked not that bad, actually, but something smelled bad to me anyway. Its was an ~80 min movie, 4:3 format, BR was 1100 or so. I found it more suitable for MPEG1 and TMPGEnc.

You know, D2Sroba is magic, in a sense (as Tylo says), but it's not black magic. As a matter of caution, don't just trust anything a software tool will say (Try MS Word's grammar corrector to see what I mean ). DVD encoding is an artisan work. Use your instinct too, and all you already know.

Before I made my choice, I tried different worst Q values, with several movies, and finally I kept 30 as my standard. No trouble ever since.

(It also happened to be the value recommended by Bach, the author of the Roba method).

BTW, Tylo uses 38.