@Jdobbs. Slightly OT question

[DDogg]

J, since you have in depth knowledge, maybe you can answer a couple of hypothetical questions for me that would help for a project I'm working on. This is not really to do with DVD-RB (well maybe), so it is kind of OT. I thought I could catch you here easily.

Primarily, I am curious if the information you achieve from the initial opening of the DVD could allow the extrapolation of the physical size of the M2v in bytes. What I mean is, DVD-RB already shows the full stream size (thanks for that by the way). It seems to me if a bit more information was available you could kinda extrapolate the size of the original video stream (which may prove useful later), but I don't know if:

1> Does the information available on 'opening' the DVD include the bitrate of the AC3s? I notice you don't show that in Rebuilder.
2> Same as 1 for sub streams, although they are not all that substantial in size.

If one knows the full stream size and also knew #1 and #2, they could 'back out' the audio and subs from the full stream size to yield the video stream only size./? If we have that and know the frame-rate, we can derive the ABR of the original video stream (which is what I'm really after).

Not to worry, just looking for you to answer the hypo, not trying to get you to do any work ;-)

Any other overheads come to mind?

Also, if you know of any tool that presently does this without scanning please pass it on.

Thanks, DD

ps - RB, if you read this and know the answer, please chime in.

[RB]

Primarily for my own needs, I modified DVD2AVIdg 1.0.0 so that along with the D2V, it creates a log file that should give you all the information you are after (bitrate and M2V size in bytes amongst other things). Download binary (http://home.t-online.de/home/340044300675/dvd2avirb.zip) and download source code (http://home.t-online.de/home/340044300675/dvd2avirb_src.zip). But you'll have to perform the full "Save Project" to get the log file. By just looking at the IFO and AC3 bitrate info, you can make some educated guesses about the M2V size, but that won't ever be accurate enough, I'm afraid.

[DDogg]

Thanks, RB. I should have checked that as I knew you had done one with a log. It is the closest thing I have found so far, and I take your point that there may be no way to do it instantly (one can still hope). Your D2V is a whole lot easier than demuxing the stream for sure! Funny, your D-ABR calculations and mine agreed perfectly ;)

[jdobbs]

When you say "initial opening" you mean when I'm populating the audio tables, right? What I do there is read data from all the IFO files that are present in source directory. So I can give you any information that can be pulled from them. Video bitrates and in-depth information isn't really available there.

When I do PREPARE, though, I scan the entire set of VTSs (that are large enough to qualify) -- and pretty much have everything you can get from the stream available.

[Nic]

@DDogg:
Not a good way, but a way the new version of ReJig does it, is I demux m2v from the VOBs at even distributed points and demux around 10%. Whatever figure I get from that I multiply by 10. And there is your estimate.
(when I say demux, im not writing the data to disk, just demuxing in the sense that im extracting the data only to see its size)

It's not instant, and it's not deadly accurate. But it is the best way so far I've found....

-Nic

[DDogg]

Thanks for the answers, guys. I'm still just playing around abit. It is starting to seem to me that one can near pre-identify the maximum amount a transcoder can reduce a specific source and still retain a certain known (http://forum.doom9.org/showthread.php?s=&threadid=74141) quality. Trying to test that was a pain the way I was doing it. This helped.

Briefly, if we know the ABR of the original stream and the D-ABR of a known quality sample, it may be that one can pre-calculate the maximum reduction a transcoder can be allowed to use on a particular source complexity and still achieve an approximate quality to the known sample. As said, just messing around a little.

Frankly, I am tired of all the programs that pander to the compulsive tweakers. I think eventually we can have enough AI in a program that will allow a level of quality equal or better to all the tweaking based methods.

Hell, at this stage, I just want to punch a "go" button and drink some whiskey. :)

[jdobbs]

Frankly, I am tired of all the programs that pander to the compulsive tweakers. I think eventually we can have enough AI in a program that will allow a level of quality equal or better to all the tweaking based methods. Amen to that.

[DDogg]

I'm very much looking forward to when you have the big varmints exterminated and the other stuff like multi-angle done. After that, if you are so inclined, I think it will be a lot of fun to discuss different methods to achieve smart bitrate allocation in an automated AI manner using both transcoders and encoders at their optimum targeted level. Maybe it testifies to my lack of a life, but how to do this elegantly seems a fascinating subject.

[jdobbs]

Originally posted by DDogg
I'm very much looking forward to when you have the big varmints exterminated and the other stuff like multi-angle done. After that, if you are so inclined, I think it will be a lot of fun to discuss different methods to achieve smart bitrate allocation in an automated AI manner using both transcoders and encoders at their optimum targeted level. Maybe it testifies to my lack of a life, but how to do this elegantly seems a fascinating subject. The problem is that even if you created a "perfect" copy without any of the tweaks available - someone would want to change a CCE parameter and put in a request. I think when I go to version 1, though, I may want to make the fact that there are so many options (most of them unnecessary) a little less obvious. It just confuses people.

[dragongodz]

I think it will be a lot of fun to discuss different methods to achieve smart bitrate allocation in an automated AI manner using both transcoders and encoders at their optimum targeted level.
yep keep it simple for the majority because even though there are quite a few here who do know what they are doing there are plenty more elsewhere who dont. :)

The problem is that even if you created a "perfect" copy without any of the tweaks available - someone would want to change a CCE parameter and put in a request. I think when I go to version 1, though, I may want to make the fact that there are so many options (most of them unnecessary) a little less obvious. It just confuses people.
i agree whole heartedly with that. the more options you give someone the easier it is for them to screw it up and then blame the program. :D

[DDogg]

Just some fuzzy discussion points regarding automatic smart bitrate allocation:

To keep the 'purists' happy it seems that near all of the bitrate stealing has got to come from the extras and trailers, assuming there are enough to make a difference. For this reason, going the obvious way of doing extras first at a fixed OPV Q, tallying the size and giving what is left to the main may not be the most advantageous way to do the bitrate allocation.

Another way that might work well is doing one known Q 1% sample in the prepare phase. This sample only takes like 2 minutes on modern machines. At completion of 'prepare' we would know:

1> ABR of original video streams.
2> D-ABR of known sample quality for main to target as goal. Q28 is a good goal and might be user selectable.

Note: Comparison of ratio of 1 and 2 may indicate the point that transcoding can be used to achieve HQ results.

#2 is also going to tell how much BR is left for the extra stuff, including menus.

IMO, all the actions to reduce bitrate can come into play for the extras without upsetting the purist view, such as resolution reduction and compression filtering. At some stage, if it is clear that these are not sufficient, the purists would have to deal with near transparent compression filtering like undot().Deen() on the main as that can easily achieve a 20% reduction on a full res encode and is near impossible to distinguish any negative effects of filtering as far as I have ever been able to see.

Because there are many ways in the above method to cause triggers, the user could choose to be notified if certain levels of quality can not be obtained so that the process could be aborted in advance of the time consuming part. That might be an option some would want, others might not use that option as they would know the quality is the best that could be achieved while retaining all their content choices.

The nice part is the user could choose OPV OR multipass as their preferred method for the main encode.

[dragongodz]

the only thing i would disagree outright with is the 1% at the start. that would not be reliable. you would be better off doing at least 1% at several points. so 1% at start + 1% at 20% of video + 1 % at 40% of video + 1% at 60% of video + 1% at 80% of video. giving 5% and allowing for different areas to be different(such as bright sections compared to dark).

since we are talking about extras for that the time in general should be relativly small. it would have to be optional though i supose for those with slower cpus etc.

[DDogg]

the only thing i would disagree outright with is the 1% at the start. that would not be reliable ... hmm, I find it extremely reliable. Perhaps you misunderstood me. I'm speaking of using SelectRangeEvery(1200,12) to create a 1% sample of the whole video stream. Of course it could be larger, but tests seem to indicate that 1% is quite sufficient. Did you get a chance to read the post (http://forum.doom9.org/showthread.php?s=&threadid=74141) I linked at the top of this thread? I've spent nearly two years researching this topic on and off. It is one of the few things that I actually think I know what I am talking about. :)

I'm quite happy to discuss it in excruciating detail if you wish. :)

[wmansir]

@dragongodz
RE: Sampling
In a 100 minute movie 1% would be a full minute of video, obviously. I don't think that much is necessary to get a good reading of the compressibility of a sample. However, I do agree that several clips are better than a 'selectevery' type command because 'selectevery' would probably force too many I frames.

I think a better method would be to use fixed size small clips at many points from the movie/extra. Why should the total length of the video affect the length of each sample? Would a better approach be to use a fixed length sample, that we know is long enough to give a reasonably good representation of the compressibility of that sample, and then the total length would dictate how many samples were taken (with minimum and maximum number of samples)?

EDIT: missed DDogg's reply. I didn't even know about SelectRangeEvery(), that is basiclly what I was refering to above. I was referring to the common SelectEvery() that only grabs single frames when I said it would create too many Iframes.

[dragongodz]

calm guys. :)

yes i made the same mistake wmansir and also what i was trying to convey. so ye we are all on the same wave length now. :D

[jdobbs]

Here's what I've been playing with:

SelectEvery(1200,1,2,3,4,5,6,7,8,9,10,11,12)

That way you get a 12 frame collection of pictures from throughout the entire stream that represents 1% of the whole.

[DDogg]

<Deleted>

Turns out to be just a confusion on SelectEvery and SelectRangeEvery. I guess the SelectRangeEvery command is not all that well known unless you are a "sampler".

[DDogg]

calm guys dragongodz, I hope you did not think I was upset or something :-) Man, not in the slightest. I was just looking forward to a good discussion!

[dragongodz]

DDog - well you did seem a little took back. :)

I've spent nearly two years researching this topic on and off. It is one of the few things that I actually think I know what I am talking about.
discuss it in excruciating detail

you put smilies at the end so thought you wasnt really upset, more shocked. as i said though it was just a misunderstanding of your full methodolgy. if you look at what i said it was taking small samples clips through the whole video which is what you are doing. so yes i agreed with you. :D

[DDogg]

lol, yeah I guess the word "excruciating" is not the best 'friendly' word :) I thought these results might be good to post here in case we get a discussion going about techniques for Smart Bitrate Allocation.
-------------------------------------------------------------------

Some test results showing reduction in file size/bitrate.
Method was OPV Q28 encode of full source.
Comment - This source was already easy to compress, requiring only an ABR of 2135 to achieve a Q28. These results show one of the smaller reductions from using filters I have seen, and to my mind would be classed near worse scenario. The effects on compression by filters have a direct relationship to the amount of pixels encoded. That's a good thing because the reduction of filesize by filtering increases on those sources that cause problems like fullscreen.


Filter = None

EncodeFull.mpv Frames 168718 D-ABR 2361 Size 2,076,347,292 bytes
EncodeHalf.mpv Frames 168718 D-ABR 1221 Size 1,074,429,880 bytes
Reduction in bitrate for 1/2D1 = 48.5 %

Filter = Undot().Deen()

EncodeFull_Filter.mpv Frames 168718 D-ABR 1984 Size 1,744,901,764 bytes
EncodeHalf_Filter.mpv Frames 168718 D-ABR 1131 Size 994,862,224 bytes
Reduction in bitrate for 1/2D1 = 42.9 %

Reduction in bitrate for Full Filtered / Full Unfiltered = 16.0 %
Reduction in bitrate for 1/2D1 Filtered / 1/2D1 Unfiltered = 7.4 %

[dragongodz]

DDogg - just wondering what sort of speed hit using the filter gave you ?
also another thing you could try to help reduce size is a different matrix.

[DDogg]

"DDogg - just wondering what sort of speed hit using the filter gave you ?" - 1.30 RT vs 2.00 RT so that's like 35%. Of course I nearly always use OPV so that's not a bad hit to take on one pass.

"also another thing you could try to help reduce size is a different matrix" - I normally report results to the group using the standard CCE matrix, but for my own encodes I typically use the 'Bach1' matrix which yields a substantial bitrate reduction of 7% on the above full res results.

Another post (http://forum.doom9.org/showthread.php?s=&postid=488038#post488038) on the size effects of matrices.

[dragongodz]

if you get the time it would be interesting to see the results of a combination of filter and matrix aswell. :)

[DDogg]

Here (http://forum.doom9.org/showthread.php?s=&postid=488646#post488646)

[dragongodz]

cool. thanks.