Future of video containers

[Demi-god]

Seems to me there are four containers for video to choose from: AVI, OGM, MP4 and Matroska. I am trying to determine what the advantages are for each of them. Avi can be ruled out because of its technical shortcomings. OGM is the one I use now because it allows subtitles to be muxed with the video and audio. MP4 is an enigma to me. What are the advantages of this format? I haven't used it so far because it does not support muxing subtitles. Matroska is something I've never used and support for it seems scarce as of yet, but I hear it's supposed to be the technically superior to OGM. Now, I'm asking for someone answer to a few questions for me:

- what will the MP4 format be good for?
- why are ogm and matroska being developed separately? After all they are both Open Source projects.
- is there any chance standalones will ever support either ogm or matroska? If so, which one?

[Atamido]

1. MP4 is a decent container with lots of backing. It is likely to recieve support from stand alone players in the future because of the backing of large companies. MP4 was designed specificaly for MPEG in mind.

2. Ogm and Matroska have different purposes. Or rather, Ogg and Matroska have different purposes. Ogg was designed solely for streaming, with optimization for low bitrate. Matroska was designed mainly for editing/storage. Ogm was made because Xiph never got around to defining how exactly to put video in Ogg. Ogm is not yet open source, doesn't have any specifications, and the man who designed it and the DS filter has disappeared.

3. I don't know about Ogm, because of its unstable development. Matroska support is likely (I would guess within a year, but thats just opinion) once the feature set and tools have stabelized. But that support will only initially come from non-mainstream players. Though once they support it, it shouldn't take to much longer for others to catch on.

[ChristianHJW]

- MP4 is a very powerful container, but the official specs will be limited to a ( small ) number of video and audio codecs, and there are no plans for subs AFAIK. Biggest advantage of MP4 is that it will certainly get hardware support, although nobody can really say on what media, pixel resolutions and bitrates this hardware support will be focussing.

- OGM could become a part of the official Ogg Theora specs, and then live on. Unfortunately the Ogg developers haven not made any announcements yet with respect to the precise scope of Ogg Theora, especially about support for other codecs than Vorbis, Theora and Tarkin ( http://www. theora.org ).

- I personally expect ( maybe i am biased ;) ) that matroska will be the most used container in the OSS world ( and only there !! ) for general video archiving purposes within 3 years from now. Until then many users will recognize that good old AVI is no option for them, as it wont support many modern video and audio codecs by then, and it will be well supported by many different applications and offer great versatility and performance.
Talking about hardware support, i might make the mistake and repeat my saying about '.. hell freezing over ...' as i did when questions about a VfW version of WMV9 arose, but as amirm said, it must have been a very cold day in hell that day when he announced the first beta of this codec, and now its in a very usable state already ;) .... and dont forget, next generation of hardware players, as announced from all major Japanese manufacturers, will be based on a special version of Linux ;) ....

[hans-jürgen]

Originally posted by Demi-god
- what will the MP4 format be good for? Maybe it's a good idea to read through the official FAQs of the MPEG or the M4IF to get a first impression. Or if you would like to look at some practical examples about the possibilities of MPEG-4, you could have a look at the Envivio site (they offer a free MP4 plugin for WMP 6.4 or newer, RealPlayer or older QuickTime versions), especially their sample pages. Osmo4 is another new multimedia player that explores the capabilities of this standard and unique in its approach to the Systems part of MPEG-4 (everything else besides the elementary streams in the MP4 container like video and audio, e.g. Scene Description which also includes subtitles).

http://www.envivio.com/products/etv/sample.html
http://www.comelec.enst.fr/osmo4/

[madoka]

Originally posted by ChristianHJW
- MP4 is a very powerful container, but the official specs will be limited to a ( small ) number of video and audio codecs, and there are no plans for subs AFAIK.
I believe MP4 supports subtitles, for the QuickTime MOV format supports it (since QT 2 IIRC, but no one used it), and MP4 is based on the MOV format.

Which somewhat brings me to the point: what about the MOV format? It's stable, can hold all sorts media, and the format's documented here: http://developer.apple.com/techpubs/quicktime/qtdevdocs/QTFF/qtff.html

The downside could be that Apple may decide to scrap the MOV format in the future and use MP4 natively. I think this would only be a small problem. After all, with AVI's quirks it's still the most popular container format.

[Corby]

MOV is indeed great format, much better than AVI and it has almost all the features of both Ogg and Matroska. But, there was never a good implementation of QuickTime framework for Windows, and there are no free tools. To mux audio and video from various sources and to put on subtitles you have to use QuickTime Pro which is not free. And QuickTime (Pro) for Windows is far from optimal solution - most of the time it's original Mac code working over translation layer that maps Toolbox/Carbon API calls to Win32 API. So, you are basically using an emulator :)

Too bad, because format is very nice. And, yes, it's called MPEG4 these days, so it has bright future :)

[madoka]

Originally posted by Corby
MOV is indeed great format, much better than AVI and it has almost all the features of both Ogg and Matroska. But, there was never a good implementation of QuickTime framework for Windows, and there are no free tools. To mux audio and video from various sources and to put on subtitles you have to use QuickTime Pro which is not free. And QuickTime (Pro) for Windows is far from optimal solution - most of the time it's original Mac code working over translation layer that maps Toolbox/Carbon API calls to Win32 API. So, you are basically using an emulator :)

Too bad, because format is very nice. And, yes, it's called MPEG4 these days, so it has bright future :)
Is it not possible to write the I/O routines from scratch? After all, the container format's documented. Also, I recall such effort has been made for Linux, so perhaps "lifting" some of the code would be beneficial.

[ChristianHJW]

A couple of points to respect when talking about MOV :

1. As mentioned above, there are no opensource libraries to write/read MOV files AFAIK. There may be some in the Linux world though, havent checked.

2. The 'atoms' in MOV/MP4, giving it similar flexibility as EBML is giving to matroska, is patented stuff. Just because nobody is charging for it NOW, doesnt mean this cant be used to kick opensource implementations that are using it without paying MPEG4 license fees if so the MPEG-LA would want to.

3. MOV does support a couple of other codecs, mainly also VfW codecs. But there will definitely be no work on implementing future codecs into it, despite future MPEG4 implementations like h.264 . You really think Tarkin or Theora, just to give one example out of a hundred possible, will ever be officially supported in MOV ? Sure, some outside committee, like we are, could declare new specs for the MOV format and continue working on it, but would they be respected by the mainstream companies making video applications ? In the end the whole thing would lead to user frustration, as there would be support for some new things from certain apps, while others would not do, etc. Having this as an option we should maybe go on hacking all kinds of things into AVI, thats easier IMHO, and does lead to the very same situation, incompatibility and frustration. For matroska its easy : apps will either support it as is, including all options, or they wont .... period !

[Corby]

Originally posted by madoka
Is it not possible to write the I/O routines from scratch? After all, the container format's documented. Also, I recall such effort has been made for Linux, so perhaps "lifting" some of the code would be beneficial.

There is OpenQuickTime project, and 3ivx is using it for it's Direct Show MOV/MP4 parser.

[temporance]

Originally posted by ChristianHJW
2. The 'atoms' in MOV/MP4, giving it similar flexibility as EBML is giving to matroska, is patented stuff. Just because nobody is charging for it NOW, doesnt mean this cant be used to kick opensource implementations that are using it without paying MPEG4 license fees if so the MPEG-LA would want to.Hi ChristianHJW,

The "atoms" are not patented, AFAIK. In any case, all of the patents I've heard about relating to file format could be used against almost any container format. For example: there is a patent concerning interleaving audio and video within a file. This patent covers VOB, AVI, mp4, matroska, in fact any sensible AV file format.

Just because matroska has been designed from a clean sheet doesn't mean that anyone implementing it is immune from patent lawsuits. If MPEG-LA or any other body with patents really wanted to kill matroska, they would find it very easy.

Having said that, I don't think anyone is going to start any legal action over container patents. If someone's making "illegal" use of, say, mp4, it actually helps the owners of the standard that their format is being promoted.

[ChristianHJW]

Originally posted by temporance Just because matroska has been designed from a clean sheet doesn't mean that anyone implementing it is immune from patent lawsuits. If MPEG-LA or any other body with patents really wanted to kill matroska, they would find it very easy.
matroska doesnt use interleaving at all. It contains a huge number of different tracks/streams, of which every of them can be played on its own based on the timestamps it carries. There is no sort of interleaving information in the file.

Having said that, I don't think anyone is going to start any legal action over container patents. If someone's making "illegal" use of, say, mp4, it actually helps the owners of the standard that their format is being promoted ... sure they wouldnt, because they are very much aware that an patent covering such a simple thing as interleaving wouldnt hold a second in front of a judge .... dont forget, most software patents have been applied for to make sure the software that companies are programming does not violate other software patents, companies do know very well their patents will not allow them to sue others doing similar things, and thats not the purpose either ....

[kastro68]

Ogg was designed solely for streaming, with optimization for low bitrate. Matroska was designed mainly for editing/storage.

Does this mean that Matroska would be bad for streaming? Say you downloaded a partial Matroska file that contains a video, would you be able to watch the partial download? What would happen if the file was damaged somewhere in the middle?

And I also checked out the Matroska diagram, and saw there was a section in the matroska container for attachments. Wouldn't this be a nice way to spread viruses, trojans or worms?

Nevertheless, matroska does sound very promising.

PS: What does OSS stand for? I saw ChristianHJW using it somewhere.

[temporance]

Originally posted by ChristianHJW
matroska doesnt use interleaving at all. It contains a huge number of different tracks/streams, of which every of them can be played on its own based on the timestamps it carries. There is no sort of interleaving information in the file.
Unless the audio and video exist in two completely separate chunks, e.g. whole movie's audio = 100MB then whole movie's video = 550MB, then you do use interleaving :)

... sure they wouldnt, because they are very much aware that an patent covering such a simple thing as interleaving wouldnt hold a second in front of a judge ....Such a patent is a very strong one. Even lawyers would be able to understand it. You and I might think it's simple, but this is the sort of thing that's extremely patentable, if there's no prior art.

[robUx4]

Originally posted by kastro68
Does this mean that Matroska would be bad for streaming? Say you downloaded a partial Matroska file that contains a video, would you be able to watch the partial download? What would happen if the file was damaged somewhere in the middle?

You can definitely play a file not fully downloaded.

Originally posted by kastro68
And I also checked out the Matroska diagram, and saw there was a section in the matroska container for attachments. Wouldn't this be a nice way to spread viruses, trojans or worms?

The attachement system can be considereded as TAR, no more no less (it's only missing file attributes yet). So if you consider spreading a TAR file can spread a virus, yes.

Originally posted by kastro68
PS: What does OSS stand for? I saw ChristianHJW using it somewhere.

Open Source Software

[robUx4]

Originally posted by temporance
Such a patent is a very strong one. Even lawyers would be able to understand it. You and I might think it's simple, but this is the sort of thing that's extremely patentable, if there's no prior art.

Could you give us more background on this patent ? Who owns it ? When was it created (ie when will it stop) ? What kind of data interleaving does it cover ?

[temporance]

robUx4,

I can't find it again, but you can search for (US) patents at
http://www.uspto.gov/patft/index.html

It's probably not the only patent you need to consider (if you even care about patents). There are (guess) probably 20-50 US patents that might affect an AV container format. Plus at least as many in other countries.

[Atamido]

Saying that you interleave audio and video data is pretty general. Looking through that database, there are may patents that could cover that. Although, Matroska, as a container could not come under direct pressure itself because it does not require interleaving of data. You could simply store all data in streams in large chunks, although it would be almost impossible to work with this way. I think the more likely target would be any application that writes this data, VirtualDub, Adobe Premier, Windows Media Encoder, etc. All of these use interleaving of audio and video data, so it is unlikely to ever become an issue.

This is getting off topic. It would be possible to enforce a fee for any program that produces MP4 files, as this would be following a spec. Think about the fee associated with distributing MPEG-4 codecs or content that was being pushed. It is unlikely however for someone to sue for patent infringement as there is prior art for basicaly all of them.

Ride the future, just use Matroska. (http://www.matroska.org)

[temporance]

Originally posted by Pamel
I think the more likely target would be any application that writes this data, VirtualDub, Adobe Premier, Windows Media Encoder, etc. All of these use interleaving of audio and video data, so it is unlikely to ever become an issue.But a lot of programs used GIF too... look what happened there! Widespread use doesn't make it legal.
This is getting off topic. It would be possible to enforce a fee for any program that produces MP4 files, as this would be following a spec.....Off topic yes - I find it hard to stop myself when it comes to patents! It's not the ownership of the spec that's important. The spec is only protected by copyright of the words and pictures in the spec. What's important is ownership of inventions that are used in the format. Interleaving was only one example - and there are 1000's of relevant patents out there :(

[Atamido]

Originally posted by temporance
But a lot of programs used GIF too... look what happened there! Widespread use doesn't make it legal.
No, widespread use does not make something legal. However, there is a difference between having a spec that incidentaly complies with an abstract patent, and the full implementation of a spec. In the case of GIF, there were many patents surrounding a full specification. When programs began implementing that full specification, it became easy to prove infringement. Proving infringment on just one abstract patent can be very hard. Proving it when you say you are following a spec that uses many patents is easy.

[Andrey]

I vote for MPEG7 !

[nikthebak]

The new Matroska dsfilter 0.4.0 works great, and tools that work now are all I really need.

I already started encoding to Matroska format, and I plan to keep it that way. Prior to mkv I used ogm, but I've been sceptical towards that format for the reasons already mentioned in this thread.

I just can't make up my mind about the audio format I'm going to use. I use Vorbis currently, but aac would be a nice alternative. I'm thinking of the following dshow graph, with XviD video and aac audio:


mkv file-->Matroska splitter-->ffdshow-->video renderer
\->3ivx audio decoder-->ReClock audio renderer



Oh, it'll be sweet :)

[Sirber]

Anyone can compare ACC with Vorbis?

[Atamido]

Originally posted by Andrey
I vote for MPEG7 !
*Pamel cringes.

MPEG7 will be useful in very large productions for editing purposes, but not for consumer use. It is basicaly a tagging method that can go down to the frame level. For instance, you can label several different items in a single video frame. Scary.

Really, we could add MPEG7 compliance by defining the same type of labeling technique and throwing it in the equvalent of a subtitle track, but noone here would find that useful.

[hans-jürgen]

Originally posted by Sirber
Anyone can compare ACC with Vorbis? Sure... ;)
Or in other words: Do you just want to hear my opinion or do you rather prefer to do some own comparisons, and if yes, at which bitrates for which purposes?

[Demi-god]

Ok, I've talked to a couple encoders and it seems the issue is more like MP4 vs Matroska. The people I talked to didn't know what Matroska was and the general attitude towards it was pretty hostile. It seems MP4 is capable of soft-subtitles although no known program can mux them yet(right?). Also, the best argument I heard was that MP4, being backed by big companies, will surely be supported by standalones whereas there is NO guarantee of any kind that mkv will EVER be playable by standalones. This is a good point(have YOU seen a standalone player capable of playing back vorbis audio? I think not). Anyone care to comment?

[avih]

Originally posted by Demi-god
Ok, I've talked to a couple encoders and it seems the issue is more like MP4 vs Matroska. The people I talked to didn't know what Matroska was and the general attitude towards it was pretty hostile. It seems MP4 is capable of soft-subtitles although no known program can mux them yet(right?). Also, the best argument I heard was that MP4, being backed by big companies, will surely be supported by standalones whereas there is NO guarantee of any kind that mkv will EVER be playable by standalones. This is a good point(have YOU seen a standalone player capable of playing back vorbis audio? I think not). Anyone care to comment?
if Matroska proves usefull and reliable, and many ppl use it, it might have a good chance for stand-alone support. afterall, many stand alone players will probably be just linux machines with software decoders. since it's a cheap and versatile solution. so it's only a software update. i think we'll have many players supporting vorbis as well. give it some time. most ppl today still use mp3/aac but vorbis is gaining attention, and the libs are under bsd-like license...

[Atamido]

I think that the main issue is just gaining popularity. There are several players that support DivX3, and that was definately wasn't a 'standard', or even legal. When these companies decide that they want to support a new type of file, they look at ease of implementation and distribution of use. Currently most people are storing MPEG-4 streams with B-frames in AVI, and that is in no way standard. There isn't a way to officially to that in AVI, so implementations can vary, making it difficult to provide support in a stand alone player. But companies have already started supporting this because of widespread use.

In Matroska we have taken great care to define how things should be stored, with support for any codec in the near future. What this means is that, for instance, any MPEG-4 compliant stream stored properly in Matroska can be played back by a player that supports Matroska and MPEG-4. It also means that any other codec type is easier to support because you only have to look at the spec compliant implementation.

I just commented on storing subtitles in Matroska here. (http://article.gmane.org/gmane.comp.multimedia.matroska.devel/699) I would recommend looking over it if you have interest in this type of thing. At the end I make the point that if streams have been stored properly in Matroksa, most types of editing can be done on the file without even knowing how do decode any of it. How is that for a file format? Appending, cutting, splitting, stretching, adding/removing tracks, tagging, etc. without having a clue how do decode the content.

In MP4, sure there you can put other stuff in it, but not everything. And eventually you are going to end up with hacks akin to B-Frames in AVI because there isn't an official way to store it, and the container wasn't designed to be that flexible.

[hans-jürgen]

Originally posted by Demi-god
It seems MP4 is capable of soft-subtitles although no known program can mux them yet(right?). Also, the best argument I heard was that MP4, being backed by big companies, will surely be supported by standalones whereas there is NO guarantee of any kind that mkv will EVER be playable by standalones. Like I mentioned in the "Sticky" threads, there's a lot of official information available about the rich features of MPEG-4, so if you want to get a first impression, go e.g. to http://mpeg.telecomitalialab.com/faq/mp4-sys/mp4-sys.htm which is the FAQ about MPEG-4 Systems, the part of this standard dealing with all side information besides the elementary video and audio streams in a MP4 container. There are also FAQs and in-depth informations about MPEG-4 in general on the MPEG homepage, so visit their FAQ section, too.

If you want to know more about the backing of big companies, take a look at http://www.m4if.org/ which is the homepage of the MPEG-4 Industry Forum that is concentrating and coordinating the interests of commercial companies with the standardization bodies like MPEG/ISO or ISMA.

Now to your concrete questions: subtitles (like all other side information) will be handled within the Systems part of MPEG-4, and the only free authoring tool for this I know of at the moment is mp4tools from the same guys that released Osmo4, the MPEG-4 Systems player. These are also the same people that are in charge of the Systems documents within the MPEG committee, so they should know what they are doing.

There is also EnvivioTV, a free plugin for WMP 6.4, RealPlayer or QuickTime that is able to understand a lot of MPEG-4 Systems' features. But their authoring tool is not free, you will have to pay for it.

Like someone mentioned recently, QuickTime in itself can use subtitles in the MOV format, so it will probably be able to do this also for MP4 in its v6 Pro version which costs $29.99. But Apple is not one of the leading companies when it comes to standard compliancy or future extensions, so it will probably take some time for them to come up with all or only some of the capabilities of this new multimedia standard.

[ChristianHJW]

Originally posted by Demi-god This is a good point(have YOU seen a standalone player capable of playing back vorbis audio? I think not). Anyone care to comment?
Yes, i did. KiSS DP 450 .... one of the first standalone players coming with SIGMA Design's EM 8500, which can be programmed on an OS called 'µCLinux' ;) ...

[temporance]

Originally posted by Pamel
In MP4, sure there you can put other stuff in it, but not everything. And eventually you are going to end up with hacks akin to B-Frames in AVI because there isn't an official way to store it, and the container wasn't designed to be that flexible. I have to disagree with you, Pamel, here... mp4 is as flexible as, well, a very flexible thing. It was designed that way. New atoms and descriptors can be standardised (either by MPEG or by someone else) to perform new functions.

[Atamido]

The problem with that is that MPEG isn't likely to standardize anything new that they didn't make up. As a large committee it often takes years to make an addition or revision to a set of specs, and nobody wants to wait around that long just to try and get a change implemented for their new codec. And if you aren't part of that committee, then it is likely that they won't put serious consideration into changing the specs to fulfill your needs.

Sure, you or somebody else could create a new ATOM to use, but then it wouldn't be MP4. If the committee doesn't add it to the list, then it isn't part of the spec, and thus not the same file format.

Matroska, on the other hand, has a much smaller, non-profit, committee that can accept changes very quickly. Fortunately, though, I think that they have built enough into the initial specs to prevent that need for a very long time.

[hans-jürgen]

Originally posted by hans-jürgen
Subtitles (like all other side information) will be handled within the Systems part of MPEG-4, and the only free authoring tool for this I know of at the moment is mp4tool from the same guys that released Osmo4, the MPEG-4 Systems player. These are also the same people that are in charge of the Systems documents within the MPEG committee, so they should know what they are doing. I'd like to add that Jean-Claude Dufourd, Osmo4 and mp4tool mastermind, has created a new web forum yesterday for the support of these MPEG-4 Systems applications, both open source now, as far as I know:

http://listes.comelec.enst.fr/wws/arc/mp4tool

The homepage of Osmo4 with download links, test samples, FAQs etc. is:

http://www.comelec.enst.fr/osmo4/

while mp4tool can be found on:

http://www.comelec.enst.fr/~dufourd/mpeg-4/tools.html

And another authoring tool for MPEG-4 content is of course mp4studio from TBX, but it has been very quiet since several weeks now after some reported bugs, so I don't what the current status is. If you want to know what's going on there, you could also ask at their new forum:

http://www.tbx.com.cn:81/forum.jsp?forum=1

[Blueseb]

Originally posted by Demi-god
(have YOU seen a standalone player capable of playing back vorbis audio? I think not)
btw, do you remeber how many years mp3 had to wait to reach standalone players? vorbis is a relatively young format, just wait and see ;)