Roba Support?

[hoozdapimp]

I did a search, but didn't come up with much.

Is there any chance that Roba support (1 pass VBR) will be implemented in RB? I think it would be an awesome feature!

[juan0r]

Couldn't agree more. It would surely make me use DVD-RB even more than I do to this date. DVD-RB is a very nice app and I enjoy using it but implementing RoBa-style encoding would make it even better.

Keep up the good work!

[jdobbs]

I'm working on OPV support now. If you look in the last version posted you'll see that there is an disabled OPV option under CCE. It currently works in my tests, but I'm struggling through accurately predicting size right now -- it is all over the place. I wish someone would post an algorithm for predicting size based on the result of the sample.

[hoozdapimp]

have you seen D2SRoba? It is used for OPV in DVD2SVCD...basically it just selects x amound of frames every y frames, and tests different Q settings starting at a specified Q. Depending on how the resultant filesize is versus the expectant, it does another test encode with a different Q based on the file size difference. Here is a sample output using Dvd2Svcd:

- Output type : DVD
- CD size : 1800
- Number of CD's : 1 (max 32)
- Sample percent : 2.0
- Adjust % : 1.2
- Cond. sizing pass : Yes (Transcode if opv sz > 0.3% oversize, and Q <= 40)
(Encode if opv sz < 5.0% undersize, or Q > 40)
--------------------------------------------------------
Calculations:
- 1 CD: video_br=5511 (5511) audio_br=224+0 video_sz=1771692489 fill=100% cbr=0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Using num CDs : 1
- Target mpv BR : 5511 (max 8000) kbps, size=1771692489 bytes
--------------------------------------------------------
Search for Q:
- Sample encode : Q=14: 3259 kbps, err=-40.9%, size=1047719825, sample sz=21000952
- Sample encode : Q=6: 4578 kbps, err=-16.9%, size=1471795433, sample sz=29501308
- Sample encode : Q=1: 5765 kbps, err=4.6%, size=1853432928, sample sz=37151016
- Sample encode : Q=2: 5488 kbps, err=-0.4%, size=1764541424, sample sz=35369236
- Determined Q : 2 = Round(2 + (-0.4 - 1.2)/5)
--------------------------------------------------------
- 2004-05-21 18:37:25
- Start movie OPV encoding (Q 2)
- OPV pass result: 96.9% on target (1717251972 / 1771692489) Speed factor: 1.51


Hope that may help!

[DDogg]

jdobbs, do you actually want to use OPV, or just get the benefits of the quality prediction of OPV sampling? I ask because I did this work (http://forum.doom9.org/showthread.php?s=&threadid=74141) with dvd-rb in mind as I thought it would be easier for you to use a hybrid form of Multipass. Given that many times D2SRoBa uses a second pass for sizing accuracy, my feeling was one OPV sample coupled with 1Pass (VAF+1VBR) might make it more manageable for you.

You can derive the bitrate from the OPV sample and use multipass as normal except that the bitrate will be matched to the source complexity, and you will know what bitrate would be required up front to achieve a certain quality.

[the-warriners]

Sorry to be a newb but could somebody please explain what this thread is all about :scared:

[j_olson]

Originally posted by the-warriners
Sorry to be a newb but could somebody please explain what this thread is all about :scared:
You could look here: http://forum.doom9.org/showthread.php?s=&threadid=64886

[DDogg]

In short it is about doing an encode in 1 pass or maybe 2 max using the special OnePassVbr mode of CCE SP instead of multipass VBR.

OPV encodes to a certain user set quality, whereas Multipass encodes to a user set bitrate. Since there is was no way for a standard multipass user to know in advance what bitrate the source complexity will require, it can happen that after the encode the quality is found to be inferior. There may now be a way around that. See the link I posted.

OPV on the other hand delivers a certain quality, yet does not have any size control to speak of. However, via sampling methods, or other hybrid approaches, one can know the quality that will be delivered before the encode is started within the available space, and know if doing the full encode is even advisable.

From a dvd-rb perspective, this could provide the logic structure to allow certain 'AI' decisions to be made in advance of the full encode without user intervention. In effect a sort of "Smart bitrate control" intelligence where DVD-RB would have the information before the encode even begins that would let it make some intelligent decisions for you in preparing the encode for optimum quality.

Jdobbs, please correct if you feel anything said is incorrect.

[jdobbs]

Originally posted by DDogg
jdobbs, do you actually want to use OPV, or just get the benefits of the quality prediction of OPV sampling? I ask because I did this work (http://forum.doom9.org/showthread.php?s=&threadid=74141) with dvd-rb in mind as I thought it would be easier for you to use a hybrid form of Multipass. Given that many times D2SRoBa uses a second pass for sizing accuracy, my feeling was one OPV sample coupled with 1Pass (VAF+1VBR) might make it more manageable for you.

You can derive the bitrate from the OPV sample and use multipass as normal except that the bitrate will be matched to the source complexity, and you will know what bitrate would be required up front to achieve a certain quality. I want to have an option that actually uses OPV. It's a good way to get speed without sacrificing too much quality. That doesn't mean I can't use the Q prediction for analysis... I've made a lot of progress on this today. I think OPV will be available soon. I've done my own analysis and coding so I'm not sure if it is actually "RoBa" -- but it is at least related (sampled streams used to predict output size using Q).

[jdobbs]

Originally posted by DDogg
In short it is about doing an encode in 1 pass or maybe 2 max using the special OnePassVbr mode of CCE SP instead of multipass VBR.

OPV encodes to a certain user set quality, whereas Multipass encodes to a user set bitrate. Since there is was no way for a standard multipass user to know in advance what bitrate the source complexity will require, it can happen that after the encode the quality is found to be inferior. There may now be a way around that. See the link I posted.

OPV on the other hand delivers a certain quality, yet does not have any size control to speak of. However, via sampling methods, or other hybrid approaches, one can know the quality that will be delivered before the encode is started within the available space, and know if doing the full encode is even advisable.

From a dvd-rb perspective, this could provide the logic structure to allow certain 'AI' decisions to be made in advance of the full encode without user intervention. In effect a sort of "Smart bitrate control" intelligence where DVD-RB would have the information before the encode even begins that would let it make some intelligent decisions for you in preparing the encode for optimum quality.

Jdobbs, please correct if you feel anything said is incorrect. No argument from me. You're the expert on this subject, as I got most of what I know about OPV and prediction from you...

[DDogg]

Wow, sounds like you are making strong progress. Exciting news for sure. Let me know if I can help or test.

Oh, as for the term RoBa, it can mean a lot of things to a lot of people it seems. As you said, in this context it is using predictive sampling to predict a Q that will deliver a final size. An optional second pass based upon programmer, or user criteria can reuse the VAF created by the first pass to correct size, or slightly redistribute bits for "on the edge" encodes. Ideally the second pass would not be used except rarely, unless a user decided to use it as insurance when time was not an issue.

[hoozdapimp]

can't wait! I will be sending a donation shortly :)

[jdobbs]

Sigh... Thought I had it, then I ran 4 movies overnight and all four came out oversized... Be careful, all, if you use the OPV feature in v0.50 -- it needs some work.

[juan0r]

@jdobbs:

Ok, then I'll wait some more before I test it. Was very happy to see the new version out this morning and was just about to do an encode when I saw your post. Keep up the good work, the feature will be a killer when its working properly.

[Noah]

I just did an OPV of a 179MB test clip...got the "This already fits on DVD5" message and ended up with something 1/3 the original size. I know, its not much of a bug since its not a typical scenario, but just thought you should know.

[jdobbs]

Thanks.

[DDogg]

Doesn't matter if it is working perfectly yet. Its an alpha feature in a beta software. The basic framework is there, and it's some good stuff. If possible, I wonder if you would share your analysis method here or via PM. Are you sampling the whole group/VOB/D2v in a conventional way, or doing one of your core processes to create the sample? It is hard for me to trace as the Analysis.ecl will not load. I assume you change quite a few things after it is finished so that it is no longer valid. I ask because the sampled source can't be too short or it will not predict accurately and I'm trying to get a handle on that.

/Edit: yes, I can see it is off quite a bit in this first iteration. I got 5.6 gigs. I think several of us want to see OPV happen and can assist you in anyway you desire.

[jdobbs]

The problem is that I tried it on three DVDs using a straight prediction based upon the sampled values and they all were undersized... so I decided the method always undersized and made a fixed adjustment about an hour before posting. Bad idea.

[jdobbs]

Originally posted by Noah
I just did an OPV of a 179MB test clip...got the "This already fits on DVD5" message and ended up with something 1/3 the original size. I know, its not much of a bug since its not a typical scenario, but just thought you should know. That's really odd. The output size should have been determined by the total available storage of a DVD-R -- which means you should have seen the maximum bitrate... hmm..

[Noah]

I ran it again just to make sure, and got the same result. I also tried changing the "Dynamically Assign Cell Bitrates" option thinking it might not be compatible with OPV, but ended up with exactly the same file size.

Dynamically Assign Cell Bitrates checked:

-----------------
[14:05:58] Phase I, PREPARATION started.
- "One Pass VBR (w/analysis)" mode is enabled.
- VTS_01: 90,585 sectors.
-- Scanning and writing .D2V file
-- Processed 5,071 frames.
-- Building .AVS and .ECL files
- Reduction Level for DVD-5: 100.0%
- Overall Bitrate : 9,000Kbs
- Space for Video : 4,460,030KB
- Analyzing VTS_01 for optimal Q factor.
- Q Value selected: 52
- HIGH/LOW/AVERAGE Cell Bitrates: 6,524/6,524/9,000 Kbs
[14:06:20] Phase I, PREPARATION completed in 1 minutes.
[14:06:32] Phase II ENCODING started
- Creating M2V for VTS_01 segment 00
[14:08:57] Phase II ENCODING completed in 2 minutes.
[14:09:03] Phase III, REBUILD started.
- Copying IFO, BUP, and unaltered files...
- Processing VTS_01
- Rebuilding segment 0 VOBID:1 CELLID:1
- Updating information in NAVPACKS...
- Updated VTS_C_ADT.
- Updated VTS_VOBU_ADMAP.
- Updated IFO: VTS_01_0.IFO
Correcting VTS Sectors...
[14:09:15] Phase III, REBUILD completed in 0 minutes.

Done.

Dynamically Assign Cell Bitrates unchecked:

-----------------
[14:10:03] Phase I, PREPARATION started.
- "One Pass VBR (w/analysis)" mode is enabled.
- VTS_01: 90,585 sectors.
-- Scanning and writing .D2V file
-- Processed 5,071 frames.
-- Building .AVS and .ECL files
- Reduction Level for DVD-5: 100.0%
- Overall Bitrate : 9,000Kbs
- Space for Video : 4,460,030KB
- Analyzing VTS_01 for optimal Q factor.
- Q Value selected: 52
- HIGH/LOW/AVERAGE Cell Bitrates: -1/9,001/9,000 Kbs
[14:10:27] Phase I, PREPARATION completed in 0 minutes.
[14:10:47] Phase II ENCODING started
- Creating M2V for VTS_01 segment 00
[14:13:09] Phase II ENCODING completed in 3 minutes.
[14:13:24] Phase III, REBUILD started.
- Copying IFO, BUP, and unaltered files...
- Processing VTS_01
- Rebuilding segment 0 VOBID:1 CELLID:1
- Updating information in NAVPACKS...
- Updated VTS_C_ADT.
- Updated VTS_VOBU_ADMAP.
- Updated IFO: VTS_01_0.IFO
Correcting VTS Sectors...
[14:13:36] Phase III, REBUILD completed in 0 minutes.

Done.

[geffroman]

Lovin the One Pass speed... I did get back 5.75GB on Little House Season 4 though... Looking forward to on target one pass...

Keep up the good work jdobbs...

[jdobbs]

Originally posted by geffroman
Lovin the One Pass speed... I did get back 5.75GB on Little House Season 4 though... Looking forward to on target one pass...

Keep up the good work jdobbs... How long did it take? I'm getting about 1 hr and 45 minutes to do a movie on my Athlon XP 2400+.

[geffroman]

My test run is not good for speed comparison... It was about 2 hours on a Intel 2.23GHz 533FSB... but My system drive is full and badly fragmented at the momment... I just upgraded to 1TB on the drives so I will move some crap off the main RAID (system) drive and defrag and post some better times later... I know it will be under 2 hours though...

[jdobbs]

Well, I did 4 movies overnight again... better but still (slightly) oversized. I'll make a mod again tonight. At some point this will be nailed down!

[bryanb]

Just did Matrix Reloaded with OPV and got 3.72GB.

Keep up the good work!!


Bryan

Edit:
Also did Apocalypse Now Redux and got 4.36GB.

Both were done with .50a and the last movie took 1hr. 17min. from start to finish. Great speed!!

[DDogg]

Jdobbs, I notice dynamically assign cell bitrates is still available when OPV mode is selected. Does it do anything? If so that confuses me. As for the sampling, I understand you are trying to get it done very quickly and if that works, great! But, I'm still thinking doing a 1% sample on each D2V would be worth it. On a modern machine that should still allow it to be done for all D2V's in around 5 minutes. Since Q is a semi-constant quality method, those results seem like they could then be applied to the individual ecl segments with good results and good accuracy. Last night I used 50a with dynamic turned off and got 5.6 gig again. I guess I could test standard sampling methods using the available D2V's and hand edit, but there are a whale of a lot of ecls, so I think I'll just await a response :)

Also, since the bitrate required by a given OPV Q is dictated by the source complexity, after you get size prediction stabilized, I am thinking you might want to revisit light filtering as a conditional when you do not achieve the target Q you wish. On a fullscreen source, this can drop the bitrate requirements of a given Q by 20 and even 25%. That is a huge ABR reduction. This would require using the filters in the sampling runs of course and might be something you would allow the user to specify and/or approve usage. That would take care of any "purist" concerns.

Presently, D2SRoBa solves for a target Q. If not obtainable, an optional "conditional filter" is inserted after the first standard sampling pass and will normally allow the target Q to be achieved, although the encode will be slowed somewhat due the the additional work of the compression filter.

[jdobbs]

DDogg,

Right now processing is done on both fronts but if you do OPV the dynamic selection has no effect. I compute the bitrate for dynamic allocation even when I do OPV purposely. That way I have the option later of (by changing one byte in the ECL) doing a second VBR pass with the bitrates identified. The idea is that if you are running a movie overnight (say in batch mode), and the prediction sucked (you go over) -- DVD-RB can make a determination to just do another pass (using the VAF from OPV) and make it right. But that's not there yet.

I know I'm boldly going where others have gone before on the OPV methods... but it's interesting so I'm doing it (I gotta have a little fun). As for the 1% sample -- that's interesting. I've found that .1% seems to be just as (poorly) accurate. I'm trying to figure this out, because it defies what I know (admittedly little) of statistical prediction. Both should be more accurate than has proven true. A couple hundred thousand frames is a pretty big universe from which to sample. As for the difference... while I do sometimes get different predictions between the two sample sizes, 1% prediction seems to be farther off target just as often as it is closer.

I'm perplexed how you could have gotten the same size with 0.50a as with 0.50... I took out a constant multiplier (.9) that I'd shoved in at the last minute. I would have thought you'd get at least 10% smaller. I did change a couple of other things also -- so it must be there. I know on my test encodes 0.50 was significantly smaller.

I don't like the idea of filtering. If I'm going to distort the image I've defeated the purpose of this method. Blasphemy!!!!

Anyway... I'm in "play" mode right now. I want to figure out how

[hoozdapimp]

this is just awesome! thanks for all the hard work jdobbs...with this we are nearly getting the speed of DVD Shrink with the quality of CCE! :) :)

[DDogg]

The idea is that if you are running a movie overnight (say in batch mode), and the prediction sucked (you go over) -- DVD-RB can make a determination to just do another pass (using the VAF from OPV) and make it right. But that's not there yet. That is great news. This is the purest form of 2Pass conditional Bach, and allows the best of flexibility so long as the size correction, if needed, is no more than 10%. Else, the VAF may not provide the proper information.while I do sometimes get different predictions between the two sample sizes, 1% prediction seems to be farther off target just as often as it is closer. Hmm, well when using the Newton method, tylo consistently hits within 1-2%. You have the beta FastCCE program he is working on. Would be interesting to see if it returns more accurate results than your early internal methods (as reality check).I'm perplexed how you could have gotten the same size with 0.50a as with 0.50... I took out a constant multiplier (.9) that I'd shoved in at the last minute. I would have thought you'd get at least 10% smaller. I did change a couple of other things also -- so it must be there. I know on my test encodes 0.50 was significantly smaller. Dunno, will try to do the same tonight. Oh, lol, it was that damn "IJob" again. You ever get it back?I don't like the idea of filtering. If I'm going to distort the image I've defeated the purpose of this method. Blasphemy!!!! yeah, yeah...:) As bullheaded as I know you are on this subject :)(I really am smiling), I also know you never refute the truth when presented with it. R6D2 has a script that blends one half of the screen unfiltered and one half with undot().Deen(). Nobody, to my knowledge, has yet been able to see any difference between them. Are you 'up' to taking the equivalent to the Pepsi challenge? :p I don't really know how to do that as a blind taste test. I wonder if there is a way to randomly pick a the side that will use filters in the script and then show which after the user picks the screen he thinks was filtered? Maybe get one of the script gurus in on that.

[jdobbs]

Hmm, well when using the Newton method, tylo consistently hits within 1-2%. You have the beta FastCCE program he is working on. Would be interesting to see if it returns more accurate results than your early internal methods (as reality check). I guess if I get desperate I will have to find out what the Newton method is. It just loses all the fun when someone else gives you the answer... I've been chasing nit-noy bugs for so long I'm getting stir crazy and want to do something interesting.

[jdobbs]

Originally posted by hoozdapimp
this is just awesome! thanks for all the hard work jdobbs...with this we are nearly getting the speed of DVD Shrink with the quality of CCE! :) :) You're welcome. It'll be a lot cooler when I can get it reasonably accurate.

[geffroman]

As soon as One Pass is able to create accurate sizes I propose a NEW ROUND of DONATIONS... Who's with me... I pledge another $20... Common peoples... make this man's efforts worth it... who ever imagined less than 2 HOUR, ONE CLICK, CCE based movie back ups?

Chime in now boys... WHO pledges their next donation for ONE PASS DVD-RB?

[hoozdapimp]

jdobbs, the newton method is a pretty simple idea :)

If you are doing a test clip with Q=50 and the test comes out 40% too small, you simply multiply 50*(0.6) (100% - 40%) and reencode that same clip with Q = 50*0.6 = 30. Then with the Q=30 clip, say your clip comes out 10% too small again. Repeat the process (do a new clip with Q = 30*0.9 = 27. Keep doing this until the filesize is within a specified percentage that you find to be "close enough". I think that is right anyway :) Please feel free to correct me if I'm wrong anyone :)

[wmansir]

I just did The Sound of Music and it came out at 5.99 GBs, Ouch. I checked out the Analysis.avs and it seemed like a poor sample of the overall content, just because of randomness (mostly) there were only 11 segments from the movie and 4-5 of them were almost no motion, or very dark scenes with little motion.


I know you are looking at you sample size, so I don't need to go into that, but one suggestion I have is to cut out the first segment in the 'selectevery' results. Since this is the first X frames of the VTS it is almost always just blank frames, which would skew your results (and perhaps account for your consistent oversizing).

[jdobbs]

Good point.

[marnum]

jdobbs, dvdrb saves lives (at least mine :D ).. thank you for this!

DDogg, I really don't want to get on your nerves, but don't you think the filesize prediction people over at kvcd.net use could solve the OPV under/oversize problem? I mentioned this before (http://forum.doom9.org/showthread.php?threadid=74141&highlight=kvcd+filesize+prediction) when you said you want to create a prediction method based on calculations, not trial and error. The Sampler() plugin is only kvcd's way to make it easier for n00bs, but if I remember correctly it DOES use SelectRangeEvery(). This method works with all avs-compatible encoders that feature quality-mode encoding (since it's avisynth based), and I think it takes ~3-4 passes until the correct filesize is achieved (whole process takes only 5-10 minutes). This would also be the point where AI could be employed, either the Newton method or some other new trick :D

Kvcd's filesize prediction has an error margin of 1-2% if I remember correctly - *shrug* I don't know if people here refuse to take over stuff from kvcd, or if this method doesn't suit dvdrb's needs.. just my 2 cents. If you think that's utter crap I'll be silent.

[ShadowKnight]

I ran a few diffent methods through with the One Pass VBR (w/analysis)mode. I encoded Moulin Rouge Disk 1(of the 2 disc set) R1. I ran one with default options, One with Normal Picture Filter, one with IDCT7, and one with IDCT7 and Normal Picture Filter. And I also did a 5 pass overnight for comparison (the 5 pass took about 8 hours for the encode).
I'm on a 1.8ghz Althlon XP With 512MB Ram. My times were pretty decnt, averaging about 7 Minutes for Prep, 90min For Encode, and 16min for rebuild.
My results so far from using One Pass VBR (w/analysis) have been very good, I think it looks better than the 5 pass actually, however that works. I also chose moulin rouge because I have had a lot of trouble getting it to come out looking clean in past projects with it. I also didn't remove anything but the spanish audio.
Below are the size results from each test:
The Original DVD is 7.93GB
Default, One Pass VBR MODE: 4.35
IDCT7, One Pass VBR MODE: 4.35
Normal Picture Filter, One Pass VBR: 4.25
Normal Picture Filter, IDCT7 One Pass VBR: 4.25
Default Options With 4Pass(4+encode): 4.12

I plan to set up some pictures from this test as well. I'll post with a link to look at them for anyone interested.
Overall I'd say it hit the taget size pretty well, but its the piucture quality that really shines through, and I think that the new mode actually came out looking better, and in much less time. I'm going to take screenshots in critical places and see what kind of comparison I can set up. I'll update more on sizes as I test more movies as well. I think the info is pretty obvious that using a normal filter in cce drops the size a bit in the final product. It'd be nice to use that extra space to increase any low spots, but I havenet noticed anything to complain about as far as quality. I'm very impressed. (In fact the only way I notice much of a difference on my monitor is if I zoom up while I'm watching, and the one pass method tests look better than the 4 pass, or at the very least on par with it). Keep up the great work Jdobbs! You rule!

[DDogg]

I don't know if people here refuse to take over stuff from kvcd, or if this method doesn't suit dvdrb's needs.. just my 2 cents. If you think that's utter crap I'll be silent. Not for me to judge about the first part, but I don't think what your mentioned has anything to do with what is being discussed.

[RickA]

from wmansir :

...one suggestion I have is to cut out the first segment in the 'selectevery' results. Since this is the first X frames of the VTS it is almost always just blank frames, which would skew your results (and perhaps account for your consistent oversizing).

---

Great idea if this means what I think.. don't count the beginning of a movie - introduction part, usually not much happening yet. How about including the ending part of a movie as well? Credits and such rolling, not much action there.



jdobbs I think most females would answer yes that man is wrong by default even if a woman is not around to chastise him about it. ;)

[DDogg]

Maybe I'm just too set in my ways, but 1% is already considered small, so a tenth of one percent sample is hard for me to see how it could give a representative sample of the content. 1% is 12 frames every 1200, whereas a .1% is, in this case from your avs, 12 frames every 12000. I really don't see how that can work. I can see how it is enough to get the general compressibility from the pixel count, but I think it would have great difficulty picking up compression differences caused by high and low action, lighting variation, or many other areas effecting compression.

So what if it takes a few more minutes? You are still saving folks a minimum of 50% time, minus the sampling time, and that is if they choose 2 pass multipass (1 VAF+ 1VBR)

[lab-one]

Tested the 1 pass with analysis on Stuck on You with only unwanted audio removed. Came out at 4.83 GB (which I realize is a known issue). Also, ran the movie in one click mode with 2 pass and deleted the working file from the one pass mode. I thought it was considerably worse than the 2 pass mode.

Think I will wait the 6-8 hours for a quality back-up. Time is irrelevant while I'm asleep.

[jdobbs]

Originally posted by DDogg
Maybe I'm just too set in my ways, but 1% is already considered small, so a tenth of one percent sample is hard for me to see how it could give a representative sample of the content. 1% is 12 frames every 1200, whereas a .1% is, in this case from your avs, 12 frames every 12000. I really don't see how that can work. I can see how it is enough to get the general compressibility from the pixel count, but I think it would have great difficulty picking up compression differences caused by high and low action, lighting variation, or many other areas effecting compression.

So what if it takes a few more minutes? You are still saving folks a minimum of 50% time, minus the sampling time, and that is if they choose 2 pass multipass (1 VAF+ 1VBR) Sorry DDogg, but if you want to make an argument you have to make it with mathmatics... not subjective discussions. If you put together some samples that show that 1% is more accurate than .1% when compared to the final encoded size over a significant number of encodes -- then you have a case and I will agree (you may very possibly be right).

It's been a while, but if I recall correctly, in statistical analysis it isn't the percentage of the sample that counts but instead the size and distribution of the sample. As the universe gets continually larger, the percentage that would be representative gets continually smaller. Let's take LOTR, The Return of the King as an example. There are 289,093 frames that represent the universe. A .1% sample results in 289 frames from 24 locations. That seems to be fairly reasonable.

But, I've changed it anyway ;) In the latest version 0.50b that I've just posted I've increased the sample to .5% and follow this algorithm:

1. Do a rough order of magnitude prediction of Q based on framecount and target size
2. Run a .5% OPV sample using Q and get a predicted size
3. Set Q = Q * (predicted_size / target_size)
4. Repeat 2 and 3 until predicted_size is within 1% of and is less than targetsize. From my tests it seems to typically take 3 passes and about 4 minutes.

It seems to be working alright -- we'll see what comments I get back.

[DDogg]

I've increased the sample to .5% I was actually coming back after playing around a little with just that LOTR DVD and a much shorter one (90 mins) and was going to suggest .5% might make a good compromise. Yeah, I know what you mean about the subjective stuff. It makes me crazy too.

I'll try to test the new version throughly over the next couple of days if I have the HD space. If you did not get it this time, I'm sure you will the next. You are a most tenacious man :)

[hoozdapimp]

good news....I'm not home to test, but looking forward to seeing other people's results to see if they actually work.

[SAPSTAR]

Originally posted by jdobbs
...
1. Do a rough order of magnitude prediction of Q based on framecount and target size
2. Run a .5% OPV sample using Q and get a predicted size
3. Set Q = Q * (predicted_size / target_size)
4. Repeat 2 and 3 until predicted_size is within 1% of and is less than targetsize. From my tests it seems to typically take 3 passes and about 4 minutes.

It seems to be working alright -- we'll see what comments I get back.

Some results for you guys :
Movie : Stargate SG-1 Season 3 Disc 3
0.50a : 3.99 Gb
0.50b : 4.08 Gb

[DDogg]

Jdobbs, LOTR RotK could be a good discussion topic for you and I if you are interested. Presently dvd-rb can't do this movie in an acceptable quality from what I can see.

Nothing wrong with that as it is an oddball for two reasons. First the obvious, it is 3:20 long. Whew! Second, as a double whammy, this particular source requires something like 3253 Kbps to achieve a Q28 unfiltered and uncropped. Your OPV prediction shows Q53/2119 kbps with both AC3 soundtracks and Q47/2314 Kbps with just the one 5.1 AC3. That's probably not going to be up to your own personal standards as I think you said you were fairly particular, as am I.

Here is a good example of where filtering can be wise if done properly. By using D2SRoBa, FACAR, and D2D using CCE with the Bach Matrix I can do this movie with the one 5.1 audio and 2 sub streams at Q25 or less within 4.0 gigs. I think from a quick look that should be equivalent to dvd-rb having to deal with the menus and such within 4350, which of course I can't do (menus), with the above combination. If I use the full 4350 I can hit Q 20 or less if pressed.

This is where we have talked a lot about smart bitrate allocation decisions with dvd-rb, and I would like to get your opinion on this scenario. Which is wiser, an encode at Q 47-53 which is not of good quality, or an encode at Q25 which is excellent and would be difficult to distinguish from the original (imo). I really do understand the purist view, you see. I also understand when that view is less than practical, but that is certainly just a personal opinion :)

If you do decide to play around more with SBA in your fun tinkering time, this movie is a good candidate for using conditionals, whether they be filters, cropping, or whathave you. At least on your own personal version maybe it would be worth seeing just what the difference could be, and whether you had any objection to the quality of the encode. It may be that you would think it ok, especially under the circumstances of a challenging source like this one.

[jdobbs]

I just did the movie last night using 3 pass VBR and it looks very good. What it comes down to in this case and others that are this large is that the best you can do is the best you can do.

[geffroman]

Yup... this is a case where spending the bucks for a dual layer disc will be my choice when time comes...

[TuRiSOft]

I did a few titles w .50a and no one w .50b but (thank to the verbosity) i wos pointed to a thing:
- when I check "Half D1 and BR for Extras" it tends (due to the small size of extras) to assign them Q=1 and still the predicted size is real far from the needed (less than eventhough);
- we all know that a Q=25 encode is real good and then....
- why not assign those titles a minimum Q=25 , assign an amount of the room remaining unused to the MAIN and then , basing our calculation upon the results obtained , assign another Q to the MAIN?

It's only theory......



:)

[DDogg]

"IJob" - 50b - OPV + 1/2-1/2

VIDEO_TS = 4.07 GB (4,379,326,464 bytes)


Transcoding...
-----------------
[22:46:50] Phase I, PREPARATION started.
- "Half-D1/Half Space for Extras" mode is enabled.
- "One Pass VBR (w/analysis)" mode is enabled.
- VTS_02: 189,319 sectors.
-- Scanning and writing .D2V file
-- Processed 12,140 frames.
-- Building .AVS and .ECL files
- VTS_03: 371,554 sectors.
-- Scanning and writing .D2V file
-- Processed 31,248 frames.
-- Building .AVS and .ECL files
- VTS_04: 2,630,468 sectors.
-- Scanning and writing .D2V file
-- Processed 159,007 frames.
-- Building .AVS and .ECL files
- VTS_05: 691,869 sectors.
-- Scanning and writing .D2V file
-- Processed 67,644 frames.
-- Building .AVS and .ECL files
- VTS_06: 66,502 sectors.
-- Scanning and writing .D2V file
-- Processed 4,150 frames.
-- Building .AVS and .ECL files
- Reduction Level for DVD-5: 53.6%
- Overall Bitrate : 2,626Kbs
- Space for Video : 3,665,462KB
- Analyzing VTS_02 for optimal Q factor.
- TargetSize (sectors): 98414
-- Predicted size (sectors) at Q=34: 75673
-- Predicted size (sectors) at Q=26: 88217
-- Predicted size (sectors) at Q=23: 93883
-- Predicted size (sectors) at Q=22: 95704
-- Predicted size (sectors) at Q=21: 97929
- Q Value selected: 21
- Analyzing VTS_03 for optimal Q factor.
- TargetSize (sectors): 189475
-- Predicted size (sectors) at Q=45: 75917
-- Predicted size (sectors) at Q=18: 149229
-- Predicted size (sectors) at Q=14: 175470
-- Predicted size (sectors) at Q=13: 183682
- Q Value selected: 13
- Analyzing VTS_04 for optimal Q factor.
- TargetSize (sectors): 1160568
-- Predicted size (sectors) at Q=38: 883573
-- Predicted size (sectors) at Q=29: 1070888
-- Predicted size (sectors) at Q=27: 1126701
-- Predicted size (sectors) at Q=26: 1157017
- Q Value selected: 26
- Analyzing VTS_05 for optimal Q factor.
- TargetSize (sectors): 352495
-- Predicted size (sectors) at Q=53: 235949
-- Predicted size (sectors) at Q=35: 314665
-- Predicted size (sectors) at Q=31: 341642
-- Predicted size (sectors) at Q=30: 349293
- Q Value selected: 30
- Analyzing VTS_06 for optimal Q factor.
- TargetSize (sectors): 31778
-- Predicted size (sectors) at Q=36: 13142
-- Predicted size (sectors) at Q=15: 21442
-- Predicted size (sectors) at Q=10: 26283
-- Predicted size (sectors) at Q=8: 29742
-- Predicted size (sectors) at Q=7: 31471
- Q Value selected: 7
HALF-D1 Extras increased Main-Movie bitrate by 29.0%
- HIGH/LOW/AVERAGE Cell Bitrates: 4,898/300/2,626 Kbs
[22:54:27] Phase I, PREPARATION completed in 8 minutes.
[00:20:34] Phase II ENCODING completed in 86 minutes.
<snip>

[jdobbs]

I did four movies overnight. No more oversizing. All came in at 4.1xGB. I may want to tweak a little higher, but I may be getting close to the best it's going to get (without significantly more time in sampling).

[DDogg]

A few questions and comments later, but I wanted to acknowledge I completely understand this is an early implementation of RoBa and that it may continue to grow in functionality. I hope so because it looks like you have a great start going on.

On the above log some of the extras are encoded at a higher quality than the main encode. The "Steal space from extras" crashes at the present, but I assume this is going to be fleshed out more? Do you see the time when that section will be made more automatic?

OPV is a useful encoding method, but only if the Q-Value is within an acceptable range. Arguably, one of the biggest strengths of the logic created by RoBa is the ability to predict quality before the encode begins which allows intelligent decisions by the software to take various automatic actions to reduce the Q-Value to an acceptable quality.

Presently I am a tad confused on your design direction regarding RoBa. Do you see it just as an alternative OPV encoding method where 'you get what you get', or do you plan to move toward a more automatic decision making implementation to maximize quality of the main encode. That is, the smart bitrate allocation thing?

If a OPV Q-Value encode gets above 40, some would argue 32, it is better to go with multipass, or take other actions, instead of continuing the encode. I sense we are from different camps on this "other actions" subject from your sparse comments on filtering and such, but whether that is true is hard for me to completely understand as I don't believe you have ever clearly stated what you see as your final direction for RoBa.

I bring this up attempting to get an idea of where you wish to go on the subject and because of the sense that I may be aggravating you with the continual commentary about the Smart Bitrate Allocation AI thing and the use of selective compression filtering and cropping when prediction shows unacceptable quality will result without them.

Since I respect you greatly, causing you any irritation is about the last thing I want to do. Having a better understanding of your direction would be helpful for shutting me up, so it may be well worth your trouble to do so :D

[jdobbs]

@DDogg,

I guess from my point of view there are two different angles here.

From an intellectual point of view I love to discuss points such as using OPV with predictive analysis (I hesitate to use the term RoBa because I'm not really sure I understand what RoBa really is, how close I am to RoBa, and whether I would offend the RoBa method authors). I see a lot of potential for using that information in an AI manner to created the absolute best backups possible.

But from a product point of view... When implementing useful software the esoteric angles have to take a back seat, and the practical takes precedence. The average user doesn't know what Q is, and really doesn't care. They want to make fundamental choices: I want my output fast with decent quality (OPV) or I don't care how long it takes -- I want the best quality that will fit on a DVD-R.

So my goal is to first give the user base what they want... then if I can use the results of that effort to create a better backup, I will. But only in a way that doesn't require a PhD to use.

One of the best uses I can see is pointed out in your example in this thread. If I can release valuable storage space from an extra by knowing that it's Q value is within a certain range and reallocate it to my main movie -- I've just found a way to steal from the rich and give to the poor... But I will only do that as an invisible function.... If I start asking people to decide what Q they want it is the equivalent of an architect asking his banking customer what stress analysis method they'd prefer he use in determining their structural integrity... the answer is "it's your job to figure that out, use the best one"

So I guess my answer is: Practicality first.

[jdobbs]

Oh... as for the filtering. Yes I am very opinionated on this. In my mind there is little difference between getting a low Q and filtering. With filtering you've purposefully decided to distort the picture -- and the Q reports artificially better because you have corrupted the picture before the Q calculation.

The only plus I can see is that at least in filtering you've made the decision as to what and how the picture gets distorted.

[added] What I mean by low Q is "bad Q" -- not a low numeric number (which would indicate "good" Q)

[hoozdapimp]

JDobbs, I understand what you're saying, but is there any chance you would consider just using the exact method Tylo does for DVD2SVCD? It works VERY well, and he might even write a plugin for DVD-RB, or give you an exported dll you could just call to use with your program.

[jdobbs]

Originally posted by hoozdapimp
JDobbs, I understand what you're saying, but is there any chance you would consider just using the exact method Tylo does for DVD2SVCD? It works VERY well, and he might even write a plugin for DVD-RB, or give you an exported dll you could just call to use with your program. I just don't like to steal other people's code and you never really understand something unless you figure it out for yourself.

[hoozdapimp]

Originally posted by jdobbs
I just don't like to steal other people's code and you never really understand something unless you figure it out for yourself.
Oh I agree. Being a programmer myself (not as good as you obviously), I totally understand wanting to figure it out yourself and not wanting to steal other people's code. And who knows, maybe by working on it yourself you'll figure out an even better way of doing it or making it even more accurate. I'm just trying to suggest maybe just talking with him, and maybe that will give you ideas on how to improve, but still writing the code/algorithm yourself....I love the idea of one pass encoding and I'm just eager (probably a little too impatient) for it to work perfectly (almost :)) with DVD-RB...keep up the good work :)

[DDogg]

But I will only do that as an invisible function I completely agree with your thoughts on that. Frankly, I believe we both would like an app where you can have a couple of whiskeys (or three), stick in the DVD and press the "Make me an awesome DVD" button and go snore a little.

To get there does require delving down into the guts. BTW, I completely understand your point about using other people's code. When you start fresh and learn the minute details, it is a good feeling of accomplishment AND you actually know the subject matter in depth, so you can then build more upon it. That is not reinventing the wheel in my book, just a true way to learn something.

So, up to that point you would not hear a peep of disagreement from me. Hehe, here comes the however. I noticed you er, forgot to reply to the Pepsi challenge thing :) I would paraphrase your ex tagline with, "If you can't see any effects of filtering, is it distortion?". I put it that way, because I would never suggest something for compression filtering that you can see. I completely agree with you. However, if you are suggesting that it is better to have an encode with macro-blocks, than one that seems near perfect, I question your intellectual reality :) Especially if you have not actually tried it. Personally I think the "purist" view is fairly antiquated today and is stuck back at the time people did really dumb over-filtering. Believe me when I tell you I am a damn perfectionist when I put a movie on a DVD, else I would use SVCD where I can accept some very minor negatives.

If your suggestion is I do a crap encodes after all these years because I judiciously use filters, then I would suggest we compare encodes of LOTR any day. I'm at a beautiful Q22 and there is no way in hell you can touch that (hoping you are motivated by challenge). I assure you, it looks incredible on my TV as well as a friends 100" projector screen. You cannot tell the difference between the encode and the original DVD whereas the encode without black/overscan crop and light filters is at Q47 and looks very poor. Well perhaps it is an intellectual honest poorness, but poor nevertheless.

I prefer the practical approach,

So I guess my answer is: Practicality first. :D

http://img17.imageshack.us/img17/9749/PDVD_000.jpg

http://img15.imageshack.us/img15/4931/PDVD_008.jpg

http://img15.imageshack.us/img15/3999/PDVD_003.jpg

[ShadowKnight]

@Ddogg

I'm just really starting to delve into making the best quality backups I can... I've used filters in cce (and they seem to do an alright job), and just a bit ago made a new thread and I mentioned something about it. But I'm very curious to hear what methods and filters you use? You filter them before running through dvd-rb right? Could you guide me through your process? Or perhaps just point me to a thread you've mentioned this in before if you have :)
Getting a low Q like that on LOTR3 is awesome, and I'd like to be able to use your method on some of my backups.
I appreciate your help.
Thanks for doing what you do. I have a lot of respect for you sticking around here for so long, driving ideas forward and keeping people informed.

[jdobbs]

@DDogg

Man... you exhaust me. The point I was making is that if you measure Q after a filter... you aren't measuring quality. What Q would you get after running BlankClip()? I choose to stick with my convictions. In this instance I also have the advantage that I own the code...
:)
That doesn't mean I won't use filters. It only means I choose to be right on this one.

BTW -- how the hell could I know whether you do crap encodes? So why would I make that suggestion? You're reading too much into completely innocuous statements.

[DDogg]

Man... you exhaust me. The point I was making is that if you measure Q after a filter... you aren't measuring quality. What Q would you get after running BlankClip()? I choose to stick with my convictions. In this instance I also have the advantage that I own the code... You absolutely do own your code and the project. It is always difficult to know where discussion will be helpful and where it becomes exhausting. It was my wish to stimulate discussion, not tire you out :).

The statement above does show me there is a disconnect in our communication as well as a general misunderstanding about CCE OPV Q mode as well as some of the most basic building blocks of how source complexity dictates the bitrate required to carry, or 'hold' the said complexity.

In closing, I would suggest a simple exercise, do OPV samples of the same source at the same X Q-Value, say Q28, using bilinear, bicubic, and LanczosResize. You will see three different final filesizes.

Do the same thing with something as simple as 8,9, 10 DCT values. Again you will see three different sizes. Do the same with three different matrices and you will get three different sizes.

All things effect the compressibility of an encode in varying degrees, thus they effect the final filesize of the encode. Of course you must include those things when sampling for a Q-Value to fit within a certain filesize. What is not clear about this most basic fundamental of compression, or how am I misunderstanding you?

I sense a major disconnect, or some kind of basic knowledge gap, between us, but I do not wish to exhaust you further. Like I said, I respect you very much and I know you are one heck of a lot smarter than I am :)

[jdobbs]

Sigh... It ain't a contest. But it feels more and more like a debate that will never end. That's what is exhausting. I have nothing but respect for you.

[quantum]

Originally posted by jdobbs
... The point I was making is that if you measure Q after a filter... you aren't measuring quality. What Q would you get after running BlankClip()? That's like a beacon in the storm. I was going to bring that point up some time ago but I didn't. I prefer not to get involved when discussions turn endless. If you get tired of the forum, there is a simple solution. Do not reply. Get back to coding ;)

[DDogg]

Originally posted by jdobbs
Sigh... It ain't a contest. But it feels more and more like a debate that will never end. That's what is exhausting. I have nothing but respect for you. We never had a debate, nor was it my intention to have any type of contest. I just wished to have a two way discussion on a topic that is interesting to me, and I thought to you.

I don't know diddle about the inner structure of a DVD, but do feel like I have a pretty decent handle on compression, and what the incremental flow might be to achieve the best possible encode within a given preset space. Thought I could make a small contribution in that area. I think maybe our heads just work a little different from each others. No problem at all. Best of luck on the project.

PS - I thought we would end up at the practical end of discussing the logic flow of how to do this stuff near invisibly to the user. The "Do an awesome encode for me button".

[edited in header]

[DDogg]

Originally posted by quantum
That's like a beacon in the storm. I was going to bring that point up some time ago but I didn't. I prefer not to get involved when discussions turn endless. If you get tired of the forum, there is a simple solution. Do not reply. Get back to coding
quantum, perhaps you could have that discussion with me? You see the statement makes absolutely no sense to me. Since you comment on it, you must understand it intimately. Perhaps you could explain it to me in detail so I can understand it also? Perhaps commenting in the thread mentioned in my Sig might be a better place. If you wish to refute that work piece with logic I would highly enjoy the discussion (I do love to learn).

[Edited in the header and made a few spelling corrections.]

[r6d2]

jdobbs,

I'm surprised I did not notice this thread before. Very interesting! Maybe that PM you sent me the other day was regarding to this. Was the info I gave you of any use? You seem to like to re-discover the wheel a lot. ;)

Anyway, I don't want to get into a religious discussion about if filters reduce quality or not. I know UnDot+Deen() reduces size without any visual impact I can notice. DVDs are created with lots of redundant information, and there are several ways to reduce that information to keep only what you are able to see. Filtering is only one of them.

For instance, as it has been mentioned on this thread, resizers make an impact on the encode size. Also does frame size. For one, I've found that converting some anamorphic sources to Half-D1 anamorphic does not reduce the visual quality. OTOH, in other cases it does.

Normally I stick to 704 width since that's what my projector is able to handle. And I eliminate the anamorpic aspect because of the same thing.

There is probaby not a "best way" to make a DVD9 to DVD5 copy for the general case, but I've found that in quite a few cases this is near the optimum:

Main movie encoded at 704x___, 4:3. OPV 30
Extras encoded at 352x___, 4:3, VBR setting BR to what it's left after encoding the movie. (Number of passed depending on which Q is estimated for extras).

[dragongodz]

at the risk of getting my ears burnt... :)

i agree that light filtering can be helpful for low bitrate encoding to reduce noise, blocks etc. how much and what specific types is a personal thing which need not be gone in to here and now.

however it seems to me that jdobbs is more interested(rightly) in getting DVD-RB working 100% before considering such possible tweaking. as such we should all respect that and leave this discussion until he is ready or interested in having it.

[Noah]

Originally posted by r6d2
Normally I stick to 704 width since that's what my projector is able to handle. And I eliminate the anamorpic aspect because of the same thing.

There is probaby not a "best way" to make a DVD9 to DVD5 copy for the general case, but I've found that in quite a few cases this is near the optimum:

Main movie encoded at 704x___, 4:3. OPV 30
Extras encoded at 352x___, 4:3, VBR setting BR to what it's left after encoding the movie. (Number of passed depending on which Q is estimated for extras).
For the rest of us who don't have your projector or your eyes, this isn't terribly optimum. This is why I think its best for Rebuilder not to assume anything and just come closest to a pure reencode as possible. If you know what else you want done, there's a lot you can do through the Avisynth scripts.

There are cases where the original DVD is far enough away from a generally accepted optimum, and that's why the 4:3 letterbox -> 16:9 option makes good sense to me, and also why I'd like to see either official support for IVTC or even the possibility of manually scripting IVTC.

[tylo]

Oh, my. This thread was really about RoBa, wasn't it?

jdobbs, I don't know much about how you implemented your Q-estimation routine. I know you use 0.5% sample, which in my experience gives a too high error variance. I use 1% sample which is barely OK. (0-2% estimation error most of the time, but sometimes up to 3%).

If you want, I could give you a simple condensed algorithm how to search for Q. It's not very complicated.

The way D2SRoBa does it is somewhat more advanced because it must deal with the max-bitrate limitation for SVCD. The Q-function normally follows an exponential (almost straight) slope, but near max-bitrate it suddenly bends a lot. In that case I switch from newton-raphson to binary search for Q.

For DVD bitrates, that shouldn't be an issue, and it should be fast and easy to find the closest Q that match the target bitrate. Btw. you have to subtract about 1.2% from the estimated size, when determining the Q. Also check 'Restrict auto I-frame insertion' in CCE (fix_gop_length=1) - it improves the estimation accuracy a little (tips from jonny).

/More technical talk: Another note about the accuracy of the final mpv bitrate (from an OPV encode) versus the original target bitrate:

In D2SRoBa, I compute the current estimated Q-step size in percentage, i.e. 100*(estBR(Q) - estBR(Q+1))/targetBR. This varies from 0.8% to 4%.
Example: if this number is 3%, it means that it is 3% between each Q (in that Q-area), which again means that, in worst case you cannot hit better than 1.5% on target, even if your estimates are absolutely correct. What I'm saying is that this error comes on top of the possible estimation error (e.g. if test samples are not representable).
Anyway, normally this all equals out, and D2SRoBa has proven that the method works excellent.

[jdobbs]

@tylo

What I do now is:

- Estimate a rough order of magnitude (ROM) Q value using:
(framecount / targetsectors) * 256
I pulled that out of the air after a few tests.
- I then start with the ROM Q and do these steps until estimated total sectors is within 1% of target or the Q doesn't change.

1. Do a .5% sample using SelectRangeEvery(2400,12)
2. Get the resulting M2V size and estimate sectors for complete encode
3. Set Q = Q * (estimated_sectors / target_sectors)

As a final step, if the estimated_sectors is greater than target_sectors I set Q = Q + 1

So you are saying estimated_sectors should be (estimated_sectors * .988)?

I also throw out the first 12 frames because they are blank so often.

[jdobbs]

One more question: I don't have to restrict auto I-Frame insertion on the final encode, do I? You're just talking about the predictive encodes?

[tylo]

Here is what I suggest you do:

0. Possible add an ROM_Q_const=256 in your ini file. Advanced users may want to tweak it.

1. Do a sample using SelectRangeEvery(GOP_len * 100/sample_percent, GOP_len) Possible add sample_percent=0.5 in your .ini file for advanced users. (I'd like to use 1.0% sample).

2. Get the resulting M2V size and estimate sectors for complete encode

3. Set Q[i+1] = Q[i] * (F*(estimated_sectors[i] / target_sectors - 1) + 1). Use F=1.35 as default (do experiment). If you try it on DDogg's log (page 3), you will notice that it hits optimal Q on second test for most of the VTS's. You can adaptively update F if it doesn't hit optimal Q on second test:F = (Q[i]/Q[i-1] - 1) / (estimated_sectors[i-1]/estimated_sectors[i] - 1), which makes it very likely to hit optimal Q on the third test. (Again, DDoggs log got F=1.35 on most tests).
So you are saying estimated_sectors should be (estimated_sectors * .988)?No. the other way around. Set adjusted_target_sectors = target_sectors * 1.012, and use that as target. I have recently discovered that GOP length may influence on how large this factor should be. It seems that GOP_len=12 requres a little higher adjustment than GOP_len=15, which is common to use with SVCD.
I also throw out the first 12 frames because they are blank so often.Yes, that seems wise for shorter clips (like extras).
I don't have to restrict auto I-Frame insertion on the final encode, do I? You're just talking about the predictive encodes?Correct.
/Add:As a final step, if the estimated_sectors is greater than target_sectors I set Q = Q + 1Well, actually if you skip the adjusting step (1.2% thing) mentioned, you can also skip doing the Q increase. These two actions will equal out. There is always a small chance of oversizing anyway. (You could do a Rejig pass then, like D2SRoBa...) Good luck.

[jdobbs]

@tylo

I appreciate all the tips. I'm going to implement this and give it some trial spins tonight. This is one of the things I love about Doom9... I can get advice from the master!

[jdobbs]

Originally posted by r6d2
jdobbs,

I'm surprised I did not notice this thread before. Very interesting! Maybe that PM you sent me the other day was regarding to this. Was the info I gave you of any use? You seem to like to re-discover the wheel a lot. ;)

Anyway, I don't want to get into a religious discussion about if filters reduce quality or not. I know UnDot+Deen() reduces size without any visual impact I can notice. DVDs are created with lots of redundant information, and there are several ways to reduce that information to keep only what you are able to see. Filtering is only one of them.

For instance, as it has been mentioned on this thread, resizers make an impact on the encode size. Also does frame size. For one, I've found that converting some anamorphic sources to Half-D1 anamorphic does not reduce the visual quality. OTOH, in other cases it does.

Normally I stick to 704 width since that's what my projector is able to handle. And I eliminate the anamorpic aspect because of the same thing.

There is probaby not a "best way" to make a DVD9 to DVD5 copy for the general case, but I've found that in quite a few cases this is near the optimum:

Main movie encoded at 704x___, 4:3. OPV 30
Extras encoded at 352x___, 4:3, VBR setting BR to what it's left after encoding the movie. (Number of passed depending on which Q is estimated for extras). Same comment to you r6d2, I appreciate your help... just a question -- when you use undot().deen() do you ever use any parameters for deen? I ran a test just to see if I could tell any difference. I couldn't... but I also didn't get any change in resulting Q or visible improvement at a given bitrate. Am I missing something?

[ShadowKnight]

Stupid question, but it needs to be asked...
are the dlls for undot and deen in your avisynth plugins directory?
thats about the only thing I can think of...
when I use undot and deen on a source it yields about 12Q lower than without them. At least with about Q40-50, cause I dont really remember how much improvement the lower ones yielded, but it was definately an improvement.

EDIT: oh and I personally dont use any paramaters. I'm still learning all these filters and its a bit brainracking at first. Can anyone list the applications of the modes of deen, and what the values mean and do. I've been looking all over for a detailed explanation, and I've not been able to find. Anyways... thanks for the avs filter edit option in DVD-RB Jdobbs :) Its working great. Awesome job on this program!

[r6d2]

Originally posted by jdobbs
Same comment to you r6d2, I appreciate your help... just a question -- when you use undot().deen() do you ever use any parameters for deen? Nope, I don't. Mainly because I don't understand s**t about the meaning of the parameters. :) Seriously, what Deen does is a sort of convolution with a certain matrix. I know you can tweak it, but my guess is that marcfd did set up the defaults to the most reasonable, useful, and tested thing, so I'd rather buy his stuff instead of experimenting with something I do not understand.
but I also didn't get any change in resulting Q or visible improvement at a given bitrate. Am I missing something? This I don't buy. I have yet to find a single source where Deen() has little effect on output size. Of course this only makes sense if you use OPV. If you use VBR Multipass the output size will be exactly the same no matter the filtering, so the purpose of using Deen in the first place is kind of useless.

But that's precisely the point. If you use OPV to have the movie encoded with the quantization you want, you get the leftover available BR for extras as a side effect. Some users may not be comfortable with that, but it works for me. Personally, I seldom encode extras since I always have my originals to watch them from time to time. And I use the movie backups (several per DVD) for everyday life.

[DDogg]

Filters used in multipass can be very effective also as they reduce the complexity of the source just like they do for OPV. Yes, they will not effect size because multipass is bitrate based, but by reducing the complexity, the encode will 'fit' more efficiently within the 'bitrate basket'.

In other words, bitrate by and of itself, is irrelevant to the measurement of quality. Like a basket, it is only the vehicle that holds the source complexity. How much material that can be packed in it is based on how complex the shapes of the contents are. Many more small compact square blocks will fit in the basket than irregularly shaped odd blocks. Light filtering, when used carefully, allows a reduction of complexity of the video (just my own theory). That allows a corresponding reduction of bitrate. When you make the shapes more compact, you can use a smaller basket. When the shapes are oddly shaped and off sized, you need a bigger basket to hold them all.

Now, sure, you can also take a hacksaw and hammer to the shapes and get them to fit in an even smaller size, but who wants a basket full of junk. This is analogous to the over filtering that makes so many of the senior people on this forum whacko :)


In short and IMO, each specific source, by the very nature of its individual complexity, requires a certain bitrate to achieve a certain defined quality. When there is not enough bitrate available to match the requirements of the source complexity, inferior quality will result.

In this case using filters to reduce the complexity of the source allows a better match to be achieved to a given bitrate and better quality results.

Where there is ample bitrate to match the source's complexity requirement nothing much is gained from filtering. In fact, the point can be made that you should never use a filter unless tests show the bitrate is not already adequate to carry the desired quality.

[r6d2]

Originally posted by DDogg
In fact, the point can be made that you should never use a filter unless tests show the bitrate is not already adequate to carry the desired quality. Precisely what RoBaConditional does. Pretty much well said, DDogg. :goodpost:

[tylo]

If you are concerned if Undot().Deen() gives visual artifacts, you could try FluxSmooth() (tips I picked up form wmansir). I've used it lately, and it certainly reduces Q quite a bit (haven't really compared with U&D), and is also a little faster. In theory it should have less visual artifacts, because it only works on parts of the picture that has motion (flux), which is where the eyes really can't pick up possible differences.
/Add: could be interesting to try it with SPR (saving private ryan).

[jdobbs]

Where there is ample bitrate to match the source's complexity requirement nothing much is gained from filtering. In fact, the point can be made that you should never use a filter unless tests show the bitrate is not already adequate to carry the desired quality. Completely agree. All filtering is distortion and should be avoided. But sometimes it's better for you as the viewer to decide which distortion is better (e.g. image smoothing from a filter as opposed to gross macroblocks caused by lack of bitrate).

[DDogg]

Yeah, most especially when light filtering is a near invisible form of mathematical distortion not readily apparent to the eye, while macro-blocks are just ugly and very easy to see.

[jdobbs]

DDogg, you're kinda' like Mark Twain's father. It's amazing how much you've learned since I decided to start trusting your opinion. You are stubborn, though ;)


BTW: For anyone who doesn't decode this: it's a compliment

[jdobbs]

@tylo,

Just wanted to let you know that the algorithm above works great. In my tests I hit the correct Q value on the second encode most of the time and have yet to have needed more than three. I'm running tests now to see how closely the end results match the prediction. I will add the INI entries you suggested -- 1% sampling is a lot more palatable when you only have to do two prediction passes.

Thanks for the great advice.

[tylo]

You're welcome, my friend. It's my small donation to your great project. :)

[tylo]

@jdobbs, although you haven't needed more than three sample tests so far, the Newton-Raphson method used in D2SRoBa, may converge safer to the optimal Q (from third test).

Q[i+1] = Q[i] - NRoot(i)*(Q[i] - Q[i-1])/(NRoot(i) - NRoot(i-1))
NRoot(i) : (estimated_sectors[i] - target_sectors)

So you don't need to update F - just use this formula instead.

Again, D2SRoBa uses this because optimal Q for SVCDs is harder to find than for DVDs. So if the current algorithm proves to work fine, you don't need it.

Cheers.

[SansGrip]

Originally posted by tylo
In theory [FluxSmooth] should have less visual artifacts, because it only works on parts of the picture that has motion (flux), which is where the eyes really can't pick up possible differences.
I realise I'm dredging up a month-old thread, but I just thought I'd clarify how FluxSmooth works. The algorithm is exceptionally simple, and is designed to avoid filtering motion when at all possible. It's more or less a median filter: it looks for pixels that are inconsistent with their temporal neighbours and applies smoothing to those pixels only. For example:


Frame Pixel value
1 150
2 175
3 158

FluxSmooth will consider the pixel in frame 2 "noise" and will average it. However:


Frame Pixel value
1 150
2 165
3 170

FluxSmooth will consider the pixel in frame 2 part of an overall trend (an unsophisticated, but quick, motion detection method, really) and will not average it.

It's unsophisticated but in my experience seems to reduce noise quite nicely (not necessarily visibly, depending on settings, but enough to have a positive effect on compressibility) without the usual side-effects of most temporal smoothers.

But I might be biased... ;)