MKV as DVD replacement possible?

[FredThompson]

I'd like to convert a few thousand DVDs into AVC MKV files. My hope is to convert them into a format with data and physical size advantage which should also be somewhat independent of storage medium and playable on a variety of soft and hard decoders such as WD TV, PC, BluRay player, etc. I am not interested in downsizing or deinterlacing at all. The original frame size and interlace/progressive structure must remain.

The DVDs have 2 primary structures:

1) Homemade captures with very simple menus (no chapters, no angles but they DO frequently have CC embedded in the VOB)

2) Full commercial discs which can have very complex structures.

Converting #1 above to AVC en masse is simple. The data will be ripped into various source directories using basic DVD ripping software to yield single-file source, even if the file is longer than the 1G DVD VOB standard. When a batch has been ripped, they'll be encoded with x264. Goal is to use VBR to yield files half the size of the original. This will allow comparison with new captures from DVR (DirecTV with TiVo) of the same file. If a new MPEG2 version of an existing source is significantly more than 2X the size of the new MKV it will probably be encoded and stored as the new baseline. DirecTV varies the bitrate, sometimes dramatically, which can affect quality, of course.

However, IF simple profile MPEG4 can truly give the 2:1 data ratio I seek without perceptible quality loss, I'd prefer to use it for such simple sources and convert the CC into subtitle files to accompany the encodes. Is simple profile MPEG4 capable of this?

Converting #2 is not so simple. In many cases, the menu structure is critical. If at all possible, I'd like to keep all the functions of the DVD control structure such as chapters, angles, audio tracks, subtitles, etc.

I have VERY limited experience with h.264 and mkv. The last time I dabbled with MPEG4 was pre-AVC/VC-1 when the propaganda stated equal quality with MPEG2 in half the size...it didn't pan out.

Now, mkv supports multiple audio and subtitle tracks. There are quite a few threads here about converting commercial SD and HD discs to mkv. What I don't see, or haven't found, is something akin to DVD RB Pro which can take a DVD structure and yield an mkv equivalent, preserving all the user interface of the source DVD. Does such a thing exist? If so, where?

If not, are there any automated ways to take a DVD IFO/VOB set and generate the various components which go into the mkv? The more automation, the better, as long as 2-pass x264 and custom AviSynth filters can be used. In this case, the highly complex DVDs would not be processed, only the simple ones.

[thewebchat]

Why would you want to re-encode something without deinterlacing it? Especially for all those cases of telecine and not actual interlacing. As far as I know, you just need the following:

Decrypt
DGIndex
DVD2MKV chapters thingy
Automagic encoding
Automagic muxing

Looks easy enough. I think MEGUI might even cover most of these.

Edit: Oh, right, if by "all the user interface," you mean DVD menus, have fun with that. MKV menus have been vaporware from the start so the chances of that happening are less than 0.

[FredThompson]

You made a lot of erroneous assumptions.

Interlace /= telecine.

Interlaced video has twice the temporal information of progressive video with the same frame rate.

"Converting" Interlaced video to progressive video destroys the temporal information. No matter how sophisticated the conversion, there are always problems. Try it on a subject which is spinning perpendicular to the camera. Look at high motion subjects on regular and double-rate LCD displays and you'll see a huge difference.

"DVD2MKV chapters thingy" is a concept. I'm looking for a manifested tool for the concept. There is precedent:

http://forum.corecodec.com/viewtopic.php?f=15&t=54
http://forum.corecodec.com/viewtopic.php?f=14&t=1973
http://forum.corecodec.com/viewtopic.php?f=14&t=2966

Given MKV is a container which supports chapters and some rudimentary menus have been seen "in the wild" there is ample reason to believe robust control structures should be possible. MKV is a container, just as ISO or AVi are containers. DVD Menus are nothing more than video "chapters" with some control logic.

I should have stated MKV is not a requirement. Functionality is the goal, MKV is a perceived means to portability.

[nm]

Originally Posted by FredThompson

"Converting" Interlaced video to progressive video destroys the temporal information.

Not if you bob it or use some advanced full-rate deinterlacing filter. Otherwise you'd need an old CRT tube to display those interlaced sources properly. Modern display equipment is progressive.

No matter how sophisticated the conversion, there are always problems.

That's true to some extent, but the better deinterlacers can do a pretty decent job -- better than what you get by letting the cheap LCD TV deinterlace on playback.

If I were you, I'd just rip to full disc images. If some player can't handle them, write a script to encode the video titles to a supported format but keep the original images around.

[creamyhorror]

Originally Posted by FredThompson

Given MKV is a container which supports chapters and some rudimentary menus have been seen "in the wild" there is ample reason to believe robust control structures should be possible. MKV is a container, just as ISO or AVi are containers. DVD Menus are nothing more than video "chapters" with some control logic.

I would like to see some of these rudimentary "menus".

I would also like a tool that extracted all parts of a DVD separately and encoded them all according to settings I specified, but I doubt it exists as yet.

[thewebchat]

After a very brief googling, I found http://www.videohelp.com/tools/ChapterXtractor

I believe that if you add this to the chain I proposed earlier, that essentially leaves us with only the menus missing. Since MKV menu implementation has a 0% chance of ever being finished, as MKV development is essentially dead, I think this is as close as you'll ever get.

[Guest]

Originally Posted by thewebchat

Since MKV menu implementation has a 0% chance of ever being finished, as MKV development is essentially dead

What is your reliable source for this claim?

[creamyhorror]

Originally Posted by thewebchat

After a very brief googling, I found http://www.videohelp.com/tools/ChapterXtractor

Wouldn't DVD-Decrypter extract chapters just the same? (Although I think I've actually used ChapterXtractor before...)

In any case, the kind of chaptering FredThompson was looking for is actually a command structure as well as a playback interface it. Which, as you pointed out, doesn't exist and may not for some time to come.

A command/menu structure would be nice to have, it's true.

[FredThompson]

Originally Posted by nm

Not if you bob it or use some advanced full-rate deinterlacing filter. Otherwise you'd need an old CRT tube to display those interlaced sources properly. Modern display equipment is progressive.

Yes, but that's only a requirement during playback. My goal is to reduce physical space and data size. Bear with me as I explain some of the background thought which led to this post: I was merrily going along using DVD RB Pro and HC but now, the physical size and cost of optical discs has become prohibitive compared to SATA drives. I've been trying to devise a way to keep all my "stuff" while my physical living space is going to be cut in half (I'm getting married.) 25 years of VHS tapes which still occupy a complete wall and another complete wall of discs on spindles have created a problem for this packrat. So..what to do? One option is to rip to ISO as you suggest. But...what if I those can be shrunk in half with virtually no quality loss through current encoding? Things are very different now than they were even 5 years ago. I have more than one computer so once the files are pulled from the source disc, it would be great to re-encode them to half the current size with "unused" computer time. I've also realized that something like the Amahi home server could be used to uncouple the use of the data from its storage medium. I already do something similar with DV. The raw source is encoded to 4:2:2 interlaced MPEG with TMPGEnc at a bitrate of 8,000. The result is about half (maybe less) than the full DV data size with virtually no visual difference. I never filter the DV. MPEG2->MPEG4, in my hoped-for use, would have only some light filtering in an attempt to reduce macroblocks and mosquito noise. Have you seen the dedicated mosquito removal inline filters? Seems to me the same functionality will happen soon enough in GPUs so the goal is reducing the data set size and leaving the heavy lifting for the playback method.

That's true to some extent, but the better deinterlacers can do a pretty decent job -- better than what you get by letting the cheap LCD TV deinterlace on playback.

No, it's true period. Interlaced is 2 interwoven sets of still images representing 50 or ~60 distinct time periods per second. Additionally, each of the two interwoven sets of images is shot from a slightly different view point. My goal is to economically use time (CPU to encode) to take advantage of the current size and cost benefits of hard drive storage. Even with motion-compensated methods I still see glitches if there is high motion but it can be visible in other places as well. There is an episode of the History Channel show "Save Our History" (I think that's the title) which has a house in the background as the camera zooms out. The thin wood pieces which separate the window panes (can never remember what those are called) play havoc with deinterlacing. Anything which is rotating can cause problems. The artificial scan lines effect of alternating black horizontal lines can cause problems. Try it on something like flipping hair or acrobatics or motor sports. All things considered, I'd prefer to compress the data size, not convert known data points into interpolated guesses with half the temporal information. A good way to see the result is to compare interlaced source and a deinterlaced version of the same source on 60 and 120 Hz large displays.

If I were you, I'd just rip to full disc images. If some player can't handle them, write a script to encode the video titles to a supported format but keep the original images around.

I'm keeping the commercial originals in storage. The homemade stuff, which includes capturing the VHS tapes, doesn't have complex menus so AP MPEG4 seems safe enough to me now. If space and cost weren't issues, I'd just keep doing what I'm doing.

Originally Posted by creamyhorror

I would like to see some of these rudimentary "menus".

I'm still looking for a good example. One of the posts I mentioned above has a dead link. My guess is such a menu is a simple branch at the beginning and was only to pick language and subtitle. There's a post at the KMPlayer board stating MPC supports them but KMPlayer does not.

I would also like a tool that extracted all parts of a DVD separately and encoded them all according to settings I specified, but I doubt it exists as yet.

Right. That's what I see. Parsing is handled by a number of programs such as DVDShrink, DVD Stripper, PGCEdit, DVD Rebuilder, etc. BR RB Beta looks very promising but the bluray disc format is still used. There are very few players which will mount an ISO but almost every current player works with VOB. Look at things like the WD TV players. They'll work with mkv but won't mount a virtual disc.

@neuron2, howdy! How's kicks?

edit: I'm having trouble finding the forum link I read earlier today which supposedly linked to an MKV sample file with a menu. The link didn't work but the thread mentioned a simple menu which used the first 2 chapter points of the sample data. I'll keep searching.

AutoMKV Chapters: http://forums.animesuki.com/showthread.php?t=85641
101 things you never knew you could do with Matroska: http://www.mod16.org/hurfdurf/?p=8 and http://forums.animesuki.com/showthread.php?t=59561
maybe the anime subculture will figure this out first...

I guess the easiest option would be to add DVD XML and pointers to MKV ignoring the pesky little detail of licensing. Where's derrow and why hasn't he done what I desire :P

[BetaBoy]

Originally Posted by neuron2

What is your reliable source for this claim?

He is talking out of his a**. We have and continue to be active with Matroska development leading towards 2.0. As I have stated here at D9 a few times we have completed EBML 2.0, which is a complete rewrite and is already included in CorePlayer 1.3.x and our upcoming 2.0 release. We are also working on LibMatroska 2.0 which will also have a new Matroska Parser Library utilizing Core-C (our modular ANSI C language), these are also a complete re-writes. However, we have sloooowed things down a bit for 2.0 to allow 1.x to gain adoption ( no need to mess with what works ), stopped devel? Hell no.

[BetaBoy]

On Matroska menus... As we discussed a while back about them. The proposed menu system is 100% reliant on the means to display them (software and hardware) as the specs state. So while the elements can be defined, the means for playback still needs to know the how/where/what to do with them. So technically you could just stuff a current 'Cocktail' menu in Matroska as long as the means to handle all the 'web' elements are handled in the player.

On DVD Menu Extractor (or DMX as we call it), here it is in SVN. https://svn.matroska.org/svn/matrosk...dMenuXtractor/

[FredThompson]

Thanks for the link. I noticed it's gone from the closed thread in the corecodec forum.

Out of curiosity, is the mkv support in commercial devices such as WD TV a result of direct consultation with CoreCodec or did it just happen? IIRC, the main restriction with DVD and BD control structures is the copyright on the set of functions and resultant licensing fees to support...restriction of competition and an administrative staff...my opinion.

We're looking at a horse and cart situation, yes?

I did notice some discussion in the WD TV hobbyist areas of modified firmware, including discussion of menus in mkv but no realization of the concept. There's also a post in your forum about QuickTime-like menus as HTML with a browser for playback.

Maybe an IFO set with mkv component files instead of VOB might be an interim method for proof of concept but IFO includes absolute pointers based on sectors.

[creamyhorror]

Originally Posted by FredThompson

There is an episode of the History Channel show "Save Our History" (I think that's the title) which has a house in the background as the camera zooms out. The thin wood pieces which separate the window panes (can never remember what those are called) play havoc with deinterlacing.

If you're intending to backup many interlaced DVDs, you're not going to find x264 is as efficient as you hope. x264 is optimized for progressive encoding.

I'm still looking for a good example. One of the posts I mentioned above has a dead link. My guess is such a menu is a simple branch at the beginning and was only to pick language and subtitle.

Well, if you consider a list of timeline-chapters to be a menu, then I guess you'd literally be right. But it's far from an actual GUI menu, which doesn't exist in any form yet.

AutoMKV Chapters: http://forums.animesuki.com/showthread.php?t=85641
101 things you never knew you could do with Matroska: http://www.mod16.org/hurfdurf/?p=8 and http://forums.animesuki.com/showthread.php?t=59561
maybe the anime subculture will figure this out first...

Yes, that's all dealing with chapters and timelines/linking. MPC-HC, for example, presents them in a right-click menu. No specification for graphical display as yet, to say nothing of custom animated menus.

I guess the easiest option would be to add DVD XML and pointers to MKV ignoring the pesky little detail of licensing. Where's derrow and why hasn't he done what I desire :P

It shouldn't be hard to create an open-source equivalent, but I'm far from a dev.

[FredThompson]

Hmm...if you edit a DVD menu to remove stuff like Disney's on-disc games and ignore alpha-blending and logic for play-once portions and such, what is a DVD menu, really? It's a liked list of video clips. Still images are just I-frames. IIRC, DVD Stripper completely fragmented a DVD source into all the individual components. Do that and, essentially, parse only the navigation links into MKV links. Then again, it's also almost 3 AM here so...

[lovelove]

Originally Posted by FredThompson

I'm having trouble finding the forum link I read earlier today which supposedly linked to an MKV sample file with a menu. The link didn't work but the thread mentioned a simple menu which used the first 2 chapter points of the sample data. I'll keep searching.

@FredThompson: The posting you are searching for his here:
http://forum.doom9.org/showthread.ph...801#post768801

[lovelove]

Originally Posted by BetaBoy

On Matroska menus... As we discussed a while back about them. The proposed menu system is 100% reliant on the means to display them (software and hardware) as the specs state. So while the elements can be defined, the means for playback still needs to know the how/where/what to do with them. So technically you could just stuff a current 'Cocktail' menu in Matroska as long as the means to handle all the 'web' elements are handled in the player. [...]

maybe I got your posting wrong, but it sounds like it's the same old story:
applications do not support the standards
and the standard doesn't get finished due to lacking software support.

reminds me of lyrics3 and id3v2.4 specs (which had been out for ~10 years)

[Chetwood]

Originally Posted by FredThompson

Look at things like the WD TV players. They'll work with mkv but won't mount a virtual disc.

I don't know where you got your info from but the original WD TV supported ISO files as does the entire line of their players. Problem is that internal VobSubs in MKVs are problematic, check the stickies on the official forums.

While H.264 yields smaller file size at a given quality (progressive or not) I still use xvid for SD material. MakeMKV keeps chapters but does not encode but I'm pretty sure Staxrip, Handbrake or MeGUI will also transfer chapters to MKV. Thus ripping the main movie with DVDFab and encoding to MKV should be easy enough.

I just don't know why you would need menus any more. The file browser of any player is good enough to select a certain ep of a TV show, why complicate things?

[lovelove]

some DVD have highly elaborated menus which are just part of the artwork
why don't you just accept the premise?

[Chetwood]

Because there's no container that properly could handle them. Why dont you just accept that premise? Besides, players like WD TV display jpgs as thumbnails or even screen sized pictures which would allow for keeping some of this artwork. Feel free to complicate things as much as you like.

[lovelove]

There are people who can't distinguish between a still image and a full interactive menu with video, sound, links, multimedia design, etc.

Originally Posted by Chetwood

Because there's no container that properly could handle them. Why dont you just accept that premise?

... And there are also people who can't distinguidh between a premise and a conclusion.
And possibly a wrong one, that is.

Originally Posted by http://www.matroska.org/technical/menu/index.html

Matroska Menu Specifications
[...]
What we'll try to have is a system that can do almost everything done on a DVD, or more, or better, or drop the unused features if necessary."