matroska playback from ffdshow ... was : ffdshow development

[ChristianHJW]

@ milan :

i was planning to send you an email on this, but will try here in this thread before bugging you another way. Our project is heavily depending on ffdshow, thats why we also recommend using it in our playback Guides on our homepage. But to advance from where we are now, we would like to ask you a couple of things :

1. Aspect Ratio correction :

Its working perfectly now with our latest DShow parser filter ( v0.4.3 ) and also with Gabest' matroskasplitter. But the users will have to activate 'use overlay mixer' in the codec setup page. This is no problems for the Pro's, but as mkv should be widely used, may i ask you if its possible to activate this by default ? Are there any reasons why using overlay mixer could cause trouble on some machines ?

2. 'native' MPEG4 mkv files

Currently all mkv video streams are using our so-called 'VfW compatibility mode', so they are basically direct copies of the AVI video streams, but of course without the AVI framing itself. The codec IDs in the mkv file are set to 'V_VFW/FOURCC' and the complete BITMAPINFOHEADER is copied into the codec private data section in the track header. This way we can ensure that every VfW codec can be used in mkv.
As we want to get rid of all the bad AVI legacies we are trying to motivate people to use matroska's 'native' modes, especially for MPEG4. DaveEL is currently working on an app called 'avs2matroska', which is NOT the same as avs2avi with mkv support, as it will not use XviD VCM codec but link to xvid.dll ( now statically, later dynamically ). With a first alpha of this tool DaveEL could produce the first 'native' MPEG mkv video stream this weekend, but :

We couldnt play it !! :( :( ....

We were trying to map our codec IDs for native MPEG4, basically V_MPEG4/ISO/ASP and V_MPEG4/ISO/SP to the FourCC 'mp4v' , as used by you if you connect your filter to the 3ivX parser for MP4 files. Unfortunately this wouldnt work, as apparently they are hadnling timestamps different than we are doing.

After some investigation by Gabest and DaveEL, its becoming obvious that the timestamps the 3ivX MP4 splitter/parser is sending to ffdshow dont have an actual meaning with respect to when the frames have to be displayed, but it more like a syncronisation helper, not more. On a sidenote, this policy will make VFR and error recovery impossible.

In MKV files, but only in native mode, every block has a timestamp indicating its actual display time ! As the blocks are stored in the file in coding order, and not in display order, this means the timestamps of the blocks in a mkv file are not linear. We know this and its a part of our architecture. We insist on doing it this way, as we make editing the file without the help of the codec possible, and error recovery much easier ( reconstructing frames from missing blocks, etc. ).

After this long explanation, here my question to you :

Could you think of supporting 'native' MPEG4 mkv files from FFDshow, by defining a new GUID with us and buffering a few frames internally for decoding, and then to display them depending on their actual timestamp ?

If so, could you join on our IRC dev channel #matroska on irc.corecodec.com during the next time ? Just drop me a short email to chris at matroska dot org , and i will arrange we will be there to talk to you. Thanks in advance

Christian

[shitowax]

Originally posted by ChristianHJW

We were trying to map our codec IDs for native MPEG4, basically V_MPEG4/ISO/ASP and V_MPEG4/ISO/SP to the FourCC 'mp4v' , as used by you if you connect your filter to the 3ivX parser for MP4 files. Unfortunately this wouldnt work, as apparently they are hadnling timestamps different than we are doing.

After some investigation by Gabest and DaveEL, its becoming obvious that the timestamps the 3ivX MP4 splitter/parser is sending to ffdshow dont have an actual meaning with respect to when the frames have to be displayed, but it more like a syncronisation helper, not more. On a sidenote, this policy will make VFR and error recovery impossible.

[/B]

I would be interrested by details of the possible problems with the 3ivx splitter: we already spotted a couple of bugs occuring when seeking in a VFR file, but when playing normally, the splitter should output real VFR timestamps ... And, indeed, I know it does, cause you can already play in synch VFR movie with sound...
The 3ivx splitter, in theory, simply reads the duration of each frame in the mp4 file, converts it in two REFERENCE_TIME and output that ... Is that different from your concept of VFR in matroska ?

[ChristianHJW]

Originally posted by shitowax
I would be interrested by details of the possible problems with the 3ivx splitter: we already spotted a couple of bugs occuring when seeking in a VFR file, but when playing normally, the splitter should output real VFR timestamps ... And, indeed, I know it does, cause you can already play in synch VFR movie with sound...
The 3ivx splitter, in theory, simply reads the duration of each frame in the mp4 file, converts it in two REFERENCE_TIME and output that ... Is that different from your concept of VFR in matroska ?

Hi shitowax,

thanks for replying here, this may help a lot. Gabest found that the timestamps coming from your MP4 parser/splitter filter are always linear increasing, even if b-frames were used in the file, and we dont understand that. Maybe our MP4 test files with b-frames were b0rked ? Well, we have a couple of questions about your splitter/parser, maybe you can help us here :

1. Is you splitter/parser filter aware of b-frames in MP4 yet ?

2. Does the D4D codec use b-frames ? How are b-frames stored in MP4, are they marked ( flagged ) as such, and with a marker to the frame(s) they are referencing ?

3. Do the frames have timestamps in the file, or are those generated/calculated in the MP4 splitter filter, from the frame duration ?

4. If you calculate timestamps, and b-frames are used in the file, would you set the timestamps to each frame as it should be displayed, or according to its location in the file ?

5. For Variable Framerate ( VFR ), if you only have the duration of every frame instaed of a start timestamp + duration ( like in mkv ), you need to have the frame type/ frame order indicated in the MP4 files also somehow ( or you need the codec to tell you ), otherwise you cant calculate the display timestamps correctly ?

Thanks for telling us

[shitowax]

Originally posted by ChristianHJW
Hi shitowax,

thanks for replying here, this may help a lot. Gabest found that the timestamps coming from your MP4 parser/splitter filter are always linear increasing, even if b-frames were used in the file, and we dont understand that. Maybe our MP4 test files with b-frames were b0rked ? Well, we have a couple of questions about your splitter/parser, maybe you can help us here :


I'll try to answer with my understanding of the MPEG and Quicktime world. I'm far from being THE reference in the domain. Now you're warned ;)

Timestamps are always linearly increasing in a constant frame rate movie cause they have to. Of course, in a real VFR movie, they are not. I think our understanding of VFR movie seems to differ somehow: in the MPEG and Quicktime world, VFR movies are movies with variable frame rate : beginning of the movie at 24 fps and end of themovie at 30 fps for example. Adding B frames, packed or not, is not related to this status. After that, you're in theory right : the splitter may output samples with start-timestamp and end-timestamp not correct because of the B frame status of the sample. Well, then it's up to the decoder filter to reorder samples (and that means, of course, retimestamp them as well). The splitter is working at the system level and has not to be aware of the bitstream content.


1. Is you splitter/parser filter aware of b-frames in MP4 yet ?


no, cause it doesn't need to.


2. Does the D4D codec use b-frames ? How are b-frames stored in MP4, are they marked ( flagged ) as such, and with a marker to the frame(s) they are referencing ?


the current (4.04) 3ivx D4 video decoder filter is able to decode B-frame. The released VfW encoder doesn't support B-frame encoding.
B-frame are not specially marked in mp4, cause there is no need to as they are already marked in the video bitstream.


3. Do the frames have timestamps in the file, or are those generated/calculated in the MP4 splitter filter, from the frame duration ?


The Quicktime format, in theory, let you specify both a start time and a duration for each sample. Now, in real-life usage, the first start-time is set and the duration for each sample is stored compressed ala quicktime (instead of coding n times "frame lasts 40 ms", you code "n frames last each 40 ms"). In a VFR scenerio, you would have to code "n frames last each 40 ms" then "m frames last each 33 ms" and so on ...
Knowing that, the 3ivx splitter generates the timestamp from both the duration of frames and the timestamps of the first frames. The 3ivx splitter doesn't currently support correctly funky qt files with concatenated video tracks.


4. If you calculate timestamps, and b-frames are used in the file, would you set the timestamps to each frame as it should be displayed, or according to its location in the file ?
5. For Variable Framerate ( VFR ), if you only have the duration of every frame instaed of a start timestamp + duration ( like in mkv ), you need to have the frame type/ frame order indicated in the MP4 files also somehow ( or you need the codec to tell you ), otherwise you cant calculate the display timestamps correctly ?


See above, the splitter works at the system level, so, it outputs system timestamps. B frames are stored in mp4 in the video bitstream order, not in the display order.
In a more generally way, I don't see why a container would have to know what exactly it contains, as the decoders will know ;)

Hope I didn't write too many errors :)

[ChristianHJW]

Thanks shitowax, your answers were incredibly helpful to us !

Originally posted by shitowax
See above, the splitter works at the system level, so, it outputs system timestamps. B frames are stored in mp4 in the video bitstream order, not in the display order.
In a more generally way, I don't see why a container would have to know what exactly it contains, as the decoders will know ;)

Well, this is true if the container has an 'inferior' importance, and the main info is contained in the video bitstream, like for MPEG4. This is fine as long as the container is tailor-made for the support of a few codecs only.

As for matroska, we were aiming for

- support for EVERY known and future audio and video codec. Those codecs with only a very low level bitstream shall be supported fine also, with all features as other codecs, so the container has to offer a lot of functionalities, such as marking frame types, indicating references, etc.

- file editing should be possible without the help of the codec , or having 'indeep' knowledge about the elementary video/audio stream for every compression format ! Thats why we are convinced that matroska will be a nice alternative for MP4 file creation, as an interims format only used for editing ;). Next VdubMod, to be release this week, will give an impressive example for that, as it will allow you to edit DTS and AAC audio streams that were muxed to any VfW video stream, without the need to have an AAC.acm or DTS.acm codec ;) .....

[shitowax]

Originally posted by ChristianHJW


Well, this is true if the container has an 'inferior' importance, and the main info is contained in the video bitstream, like for MPEG4. This is fine as long as the container is tailor-made for the support of a few codecs only.



hehehe ... I know that it will make reactions, but as you know mp4 is based on quicktime, which is VERY FAR from supporting a few codec only (of course I'm talking of the file format, not the buggy player of Apple). This format is so powerfull that it's the same (with evolution of course) for more than 10 years. You can already put in quicktime file totally funky things like text, still picture, midi, flash, player skin, VRML like mess and so on. Quicktime files were based on track and were already powerfull, but mp4 extends that to objects that you can compose in scenes...

I even think that the qt file format was released before the AVI one =). And it's increadibly easy to edit (indeed much more easily than AVI). Its only problem was/is the buggy-non-free player of Apple.


As for matroska, we were aiming for

- support for EVERY known and future audio and video codec. Those codecs with only a very low level bitstream shall be supported fine also, with all features as other codecs, so the container has to offer a lot of functionalities, such as marking frame types, indicating references, etc.


I don't see how you can hope to support everything if you code very low level frame reference or presentation timestamps ...


- file editing should be possible without the help of the codec , or having 'indeep' knowledge about the elementary video/audio stream for every compression format ! Thats why we are convinced that matroska will be a nice alternative for MP4 file creation, as an interims format only used for editing ;). Next VdubMod, to be release this week, will give an impressive example for that, as it will allow you to edit DTS and AAC audio streams that were muxed to any VfW video stream, without the need to have an AAC.acm or DTS.acm codec ;) .....


I may be wrong, but it's already the case for qt files as chunks are always packed on frame (audio or video) boundaries. Anyway, you have the very good advantage with matroska to be a patent-free alternative to mp4 ... and matroska works very well with 3ivx as well :)

So, keep on the good work :)

[ChristianHJW]

@ DaveEL : can you talk to Swede about copying the last postings into a new thread in 'new A/V formats' section, as discussed on IRC today ?

Originally posted by shitowax
[B]hehehe ... I know that it will make reactions, but as you know mp4 is based on quicktime, which is VERY FAR from supporting a few codec only (of course I'm talking of the file format, not the buggy player of Apple). This format is so powerfull that it's the same (with evolution of course) for more than 10 years. You can already put in quicktime file totally funky things like text, still picture, midi, flash, player skin, VRML like mess and so on. Quicktime files were based on track and were already powerfull, but mp4 extends that to objects that you can compose in scenes...
I even think that the qt file format was released before the AVI one =). And it's increadibly easy to edit (indeed much more easily than AVI). Its only problem was/is the buggy-non-free player of Apple.

Hehehe .. well, i am pretty sure, if it doesnt support frame types and future frame referencing, like mkv does, it wouldnt be able to cope with advanced codec architectures like Ogg Tarkin ( 3D wavelet ) or GLDM's warp codec. It maybe that QT was very powerful for the stuff of the last 15 years, but without definition of some new atoms, as possible for MP4, i am convinced it would fail on a few future codec architectures, but i admit i dont know enough about it to state that.



I don't see how you can hope to support everything if you code very low level frame reference or presentation timestamps ...

We have been thinking through many possible scenarios with codec developers, and it took robux4 about 1 hour to come up with a working and powerful support for h.264 NALU's, when one frame is divided into several packets of different priority, to allow streaming even with low bandwidth ( = lower quality, but plays fluently ).

And in case we need something new, well, EBML as underlying framework gives us the flexibility to define every new low-level element we would need, without breaking old files ;) ....

I may be wrong, but it's already the case for qt files as chunks are always packed on frame (audio or video) boundaries.

If they dont know which frame is a b-frame and what other frames its referencing to, they cant cut/edit the file without risking to break it.

Anyway, you have the very good advantage with matroska to be a patent-free alternative to mp4 ... and matroska works very well with 3ivx as well :) So, keep on the good work :)

Well, if you or Stux plan to add MKV support to any of the 3ivX tools, dont forget our main library is QPL also :D ! So, we could give commercial licenses to your company to add MKV support to your tools if you were interested to do so, just like we have been talking to Ahead for a Nero MKV plugin already recently ;) ....

[Atamido]

As far as containers, I think it is a matter of differing philosophies. There are many advantages to having more of this data stored at the container level, one of the greatest was mentioned above about being able to edit a file without understanding the data.

However, the great disadvantage with storing more data at the container level is that you run the danger of codec being released that has a design which the container cannot properly implement.

The Matroska team has taken great care to try to prevent this from happenning by trying to consider any potential future codec storage techniques. Of course it is impossible to know what will come out in the future, but you can take some well educated guesses. The Matroska team was aiming for at least 10 years of viability, and I would guess that they will hit that.I don't see how you can hope to support everything if you code very low level frame reference or presentation timestamps ... I am not sure how something could be not supported with this system. We allow far-away referencing, so you could reference a frame 1000 frames away. We also allow referencing as many frames as you like. Frames can have overlapping timestamps. And, you can have a single data chunk and say that that data chunk contains frames at these timecodes... Any current codec could be stored in this way, and any codec in the near future should be able to also.

I am sure that a codec will be made eventually that uses concepts so foreign as to completely break the system. But, I don't think that will happen any time soon (at least I hope not ;))

[shitowax]

Originally posted by Pamel
As far as containers, I think it is a matter of differing philosophies. There are many advantages to having more of this data stored at the container level, one of the greatest was mentioned above about being able to edit a file without understanding the data.

However, the great disadvantage with storing more data at the container level is that you run the danger of codec being released that has a design which the container cannot properly implement.

The Matroska team has taken great care to try to prevent this from happenning by trying to consider any potential future codec storage techniques. Of course it is impossible to know what will come out in the future, but you can take some well educated guesses. The Matroska team was aiming for at least 10 years of viability, and I would guess that they will hit that. I am not sure how something could be not supported with this system. We allow far-away referencing, so you could reference a frame 1000 frames away. We also allow referencing as many frames as you like. Frames can have overlapping timestamps. And, you can have a single data chunk and say that that data chunk contains frames at these timecodes... Any current codec could be stored in this way, and any codec in the near future should be able to also.

I am sure that a codec will be made eventually that uses concepts so foreign as to completely break the system. But, I don't think that will happen any time soon (at least I hope not ;))

Just a small remark (I feel a bit like the devil's advocate here =))
Don't you think it's a bit overkill to recode in the container the frame reference contained in the video bitstream ? I don't see any problem with that, but you can't hope to have a small overhead at the same time. And to be honest, all the matroska files I've seen are larger than their mp4 equivalent ... and it's normal as mkv files contains multiple times the same information and mp4 doesn't.

I see what you mean concerning the B frame problem : you're reffering to B frame using the next I frame to be decoded... it's an interresting problem ... no idea if it can be solved or even if it's 100% ISO compliant to produce such bitstreams ... It could be a bug in the MPEG specs =)... Anyway, it's not that important as Studio Profile doesn't use any B frames ;)

[Atamido]

Originally posted by shitowax
Just a small remark (I feel a bit like the devil's advocate here =))
Not a problem. I love playing the devil's advocate as it generally helps resolve issues, or at least helps with understanding them better. (Well, that is until someones feelings get hurt)
Originally posted by shitowax
Don't you think it's a bit overkill to recode in the container the frame reference contained in the video bitstream ? I don't see any problem with that, but you can't hope to have a small overhead at the same time.

Is it overkill? Not for what we are trying to do. Sure the overhead will increase slightly, but it shouldn't be THAT much.

Just compare it to a plugin based software system. When you decide that you want to make a piece of software that is based on plugins, you have to know up front that the software is going to be slower and larger. This is a trade off to adding the flexibility of easier expansion down the road. The same thing was done with MCF. It was an extremely fast and low-overhead design. The reason that much of the team switched to Matroksa was to increase the flexibility of the design at the cost of increased complexity and size. Though it should be noted that the design is still pretty low-overhead, and is more so the larger the file.

[spyder]

I think we may have a problem with using MPEG video for VFR. MPEG video, AFAIK, has a framerate encoded in the bitstream. So the decoder should technically decode it at that framerate.

[ChristianHJW]

Originally posted by shitowax Just a small remark (I feel a bit like the devil's advocate here =))
Don't you think it's a bit overkill to recode in the container the frame reference contained in the video bitstream ? I don't see any problem with that, but you can't hope to have a small overhead at the same time. And to be honest, all the matroska files I've seen are larger than their mp4 equivalent ... and it's normal as mkv files contains multiple times the same information and mp4 doesn't.

Well, look at the alternative : If we dont have our own system, we couldnt achieve file editing without 'indeep knowledge' of the muxed bitstream.

I see what you mean concerning the B frame problem : you're reffering to B frame using the next I frame to be decoded... it's an interresting problem ... no idea if it can be solved or even if it's 100% ISO compliant to produce such bitstreams ... It could be a bug in the MPEG specs =)... Anyway, it's not that important as Studio Profile doesn't use any B frames ;)

b frames can normally only reference to an I before and a P frame after them, or to 2 P frames ( one before one after ), so if you always cut on I frames you should be safe in principal. The nice thing about matroska is that we can cut on p frames also using our frame referencing system, by inserting the I frame from the 1st part into the 2nd part on a new block with the same timestamp as the starting P frame ;) ....

[shitowax]

Originally posted by ChristianHJW
b frames can normally only reference to an I before and a P frame after them, or to 2 P frames ( one before one after ), so if you always cut on I frames you should be safe in principal. The nice thing about matroska is that we can cut on p frames also using our frame referencing system, by inserting the I frame from the 1st part into the 2nd part on a new block with the same timestamp as the starting P frame ;) ....

Sorry ... there is nothing impressive at all in that, and you can already do that with QT for years ;) ... and you can even copy a part of a Sorenson movie (cut on a B or P frame if you want), and paste it wherever you want in an MPEG-4 movie (after a B or P frame if you want) ... It will play perfectly and you won't be able to see that the first part is MPEG-4, the second is Sorenson and the third is MPEG-4. Of course, QT won't display any unwanted frame in the resulting movie.

The only thing I can see that is not possible with QT and may be possible with Matroska (?) is edition of a stream containing B frames refering to the next I frame ... I don't think it's totally forbidden by the norm, but, anyway, it would be a bit stupid for a video codec to produce such bitstream ... but, once again, it was just my 2 cents ;)