DirectShow Media Types for Stereoscopic Content

[pwimmer]

I'm opening this thread to discuss media types and media formats to be used by DirectShow filters for stereoscopic formats.

[pwimmer]

There is a need to specify media types for two categories of stereo content. This posting will be updated periodically to reflect the outcome of the discussions in this thread.


1) Compressed H.264 stereoscopic/multiview content.

The following FOURCCs have been specified and are supported by the MPC HC MP4 and MPEG-TS splitters after applying the 3D patches (http://www.3dtv.at/OpenSource). Upcoming versions of the Haali splitter and CoreAVC will support these media types, too.

The AVC1 FOURCC should be used for 2D content. If a decoder input pin uses the AVC1 FOURCC, the decoder should also offers a second input pin that supports the EMVC FOURCC only.

The AMVC FOURCC should be used for MVC streams where both the base view NALs and the MVC extension NALs (coded slice extension NAL units and prefix NAL units) are delivered on the same DirectShow filter pin. If the AMVC FOURCC is used, there must be a subset SPS and a second PPS in the format block.

A DirectShow splitter filter pin offering the AMVC FOURCC can also offer the AVC1 FOURCC as second media type so that it also connects to legacy 2D decoders. However, when the decoder input pin is connected using the AMVC FOURCC, the decoder must not offer a second pin for EMVC FOURCC media types.

The EMVC FOURCC should be used if the pin delivers MVC extension NALs only. The format block must contain a subset SPS and the corresponding PPS.

NAL units of the base view must be delivered on another pin using the AVC1 FOURCC. Timestamps of the base view and MVC extension samples must be derived from the same clock, else the decoder fails to synchronize them.


2) Uncompressed stereoscopic/multiview content.

The open media format is an extension to Microsoft's DirectShow technology to enable interoperability between stereo-capable DirectShow filters of different vendors. Click here to see the draft of the open media format specification:
DirectShow Open Media Format SourceForge Project (http://openmediaformat.svn.sourceforge.net/viewvc/openmediaformat)
OpenMediaFormat.zip on 3dtv.at server (http://www.3dtv.at/Downloads/OpenMediaFormat.zip)

[pwimmer]

Ad 1) I'm currently working on the MVC decoder and thinking about possible solutions. Things currently evolve...

[pwimmer]

Ad 2)

The Stereoscopic Player that I'm working on already uses a special stereo media type for uncompressed stereo samples sent from the 3dtv.at Stereo Tranformation filter to the 3dtv.at Stereo Renderer filter and from the 3dtv.at Stereo Image Source to the 3dtv.at Stereo Transformation.

Edit: Removed the code to avoid confusion with the new Open Media Format definition. If you are interested in the media format currently used by the Stereoscopic Player, please contact me offlist.

[pwimmer]

I propose a different IMediaSampleEx interface that derives from IMediaSample and adds just two methods:

HRESULT GetPointerEx(BYTE **ppBuffer, int iView);
long GetSizeEx(int iView);

This allows to use separate buffers for each view, which I would prefer over a single buffer for all views.

The format type adds a dwNumViews field to the VIDEOINFOHEADER2

DEFINE_GUID(STEREOLAYOUT_MONOSCOPIC, ...); // Monoscopic
DEFINE_GUID(STEREOLAYOUT_MULTIBUFFERS, ...); // Use StereoMediaSampleEx to get buffers != 0
DEFINE_GUID(STEREOLAYOUT_SIDEBYSIDELEFTFIRST, ...);
DEFINE_GUID(STEREOLAYOUT_SIDEBYSIDERIGHTFIRST, ...);
DEFINE_GUID(STEREOLAYOUT_OVERUNDERLEFTTOP, ...);
DEFINE_GUID(STEREOLAYOUT_OVERUNDERRIGHTTOP, ...);
DEFINE_GUID(STEREOLAYOUT_INTERLACEDLEFTFIRST, ...);
DEFINE_GUID(STEREOLAYOUT_INTERLACEDRIGHTFIRST, ...);
DEFINE_GUID(STEREOLAYOUT_FRAMESQUENTIALLEFTFIRST, ...);
DEFINE_GUID(STEREOLAYOUT_FRAMESQUENTIALRIGHTFIRST, ...);


typedef struct tagSTEREOVIDEOINFOHEADER {
RECT rcSource;
RECT rcTarget;
DWORD dwBitRate;
DWORD dwBitErrorRate;
REFERENCE_TIME AvgTimePerFrame;
DWORD dwInterlaceFlags;
DWORD dwCopyProtectFlags;
DWORD dwPictAspectRatioX;
DWORD dwPictAspectRatioY;
DWORD dwControlFlags;
DWORD dwReserved2;

// Stereo related fields
GUID guidStereoLayout; // See supported STEREOLAYOUT_xxx GUIDs above
DWORD dwNumViews; // Number of views (only valid for STEREOLAYOUT_MULTIBUFFERS)

BITMAPINFOHEADER bmiHeader;
} STEREOVIDEOINFOHEADER;

[madshi]

A few comments:

(1) Let's prepare for *Multi* views and not just stereo. After all it's h264 M(ulti)VC and not h264 S(tereo)VC.

(2) HDMI 1.4a not only knows left/right, but it also supports funny things like "L + Depth". Not sure if the new media type should support such things, too?

(3) I'd like to use the opportunity to add more information to the header structure. Which means that the purpose of the new media type might not be limited to 3D. I'd also like to see the new media type being used for 2D. So maybe we should not even name it 3D or MVC. Maybe we should name it completely different, e.g. "FORMAT_VideoInfo3". However, that name might collide with a future Microsoft format, so maybe we should use "FORMAT_Doom9VideoInfo" or "FORMAT_OpenVideoInfo" or whatever. I don't think there should be "Stereo" in the name. The new format should be stereo capable, but that would only be one benefit of the new media type. I want to have more benefits. And if the new format is used for 2D, the standard IMediaSample interface should suffice.

(4) I strongly suggest reusing the same FOURCC values that also exist for conventional 2D frames, instead of defining new ones. According to Microsoft, there already exists FOURCC definition for YCbCr 4:2:0, 4:2:2 and 4:4:4, with 8bit, 10bit and 16bit each. See here:

http://msdn.microsoft.com/en-us/library/bb970578%28VS.85%29.aspx#_420formats

(5) I don't really like the idea of locking the buffer for different "components". After all, depending on the FOURCC, the buffers may not be planar, and thus locking separate component buffers doesn't always make sense. I think there should be a way to get the pointer/size for every view, but it should be one simple pointer, similar to how the standard IMediaSample interface works. How the data is stored in the buffer depends on the FOURCC.

(6) Let's try to name the methods of the extended IMediaSample interface similar to the original functions. E.g. instead of "LockBuffer" I'd suggest "GetPointerEx/GetSizeEx".

(7) I'd suggest to start the structure with VIDEOINFOHEADER2, and to just append more data to it. This way it would be extremely easy for existing software to add support for the new media type. They'd just need to interpret the structure as VIDEOINFOHEADER2. They'd only need to do extra coding if they want to actually support the new fields of the structure.

(8) Additional fields I want to have in the header, all taken from the h264 specification:

- video_format
- video_full_range_flag
- colour_primaries
- transfer_characteristics
- matrix_coefficients
- chroma_format_idc
- chroma_sample_loc_type_top_field
- chroma_sample_loc_type_bottom_field
- bit_depth_luma
- bit_depth_chroma

Maybe more. The fields above might have to be adjusted to be more flexible. We don't want to limit ourselves to what the h264 spec supports. Other video formats may support more variations than h264.

[nevcairiel]

(7) I'd suggest to start the structure with VIDEOINFOHEADER2, and to just append more data to it. This way it would be extremely easy for existing software to add support for the new media type. They'd just need to interpret the structure as VIDEOINFOHEADER2. They'd only need to do extra coding if they want to actually support the new fields of the structure.


While this idea sounds good in theory, there is one problem with it.
Some formats require codec/format specific data that gets appended after the BITMAPINFOHEADER in VIDEOINFOHEADER2.
If you want to append data to the end of VIDEOINFOHEADER2, you would occupy this space.

I don't see a clear way to preserve compatibility in a way like this without crippling some functionality of the format, tbh.

Another question that i just asked myself..
This new format is mainly designed for data from the decoder to the renderer, not from the source/splitter to the decoder, right?
Would decoders benefit from any additional information we could provide on the splitter side of things?

I'm currently working on a DS source/splitter and just wondering if on this side anything new is required.

[kieranrk]

(1) Let's prepare for *Multi* views and not just stereo. After all it's h264 M(ulti)VC and not h264 S(tereo)VC.


Agreed. Just because Stereo is all the rage now doesn't mean everything else should be excluded.

You should also include Side-by-side MPEG-2 and H.264 broadcasts. There are also 720p120 H.264 3D broadcasts.

There's a special SEI in H.264 for indicating side-by-side 3D but not many people use it yet. Presumably there's something similar in MPEG-2.

[pwimmer]

(1) Let's prepare for *Multi* views and not just stereo. After all it's h264 M(ulti)VC and not h264 S(tereo)VC.


Agreed. The dwNumView field already allows for more than two views.


(2) HDMI 1.4a not only knows left/right, but it also supports funny things like "L + Depth". Not sure if the new media type should support such things, too?


I don't think so. It is not used in practice and would make the new media type much more difficult. The only 2D+Depth files are the ones for Philips Wow 3D displays, but they not only use 2D+Depth but some of them 2D+Depth+Background 2D+Background Depth.

I suggest to define a separate media type for 2D+Depth and do not cover it in our new media type.


(3) I'd like to use the opportunity to add more information to the header structure. Maybe we should name it completely different, e.g. "FORMAT_VideoInfo3". However, that name might collide with a future Microsoft format, so maybe we should use "FORMAT_Doom9VideoInfo" or "FORMAT_OpenVideoInfo" or whatever.


They name can always be changed without breaking binary compatibility. Only in the source code a search&replace must be performed. The GUID is unique in any case, even if there is a collision of the names.

FORMAT_OpenVideoInfo is my personal preference.


(4) I strongly suggest reusing the same FOURCC values that also exist for conventional 2D frames, instead of defining new ones.


Right. I did this because I needed FOURCCs for formats where each plane resides in different formats. I've published this so that everybody can see how the existing Stereoscopic Player works, but it should no be part of the new standard.


(6) Let's try to name the methods of the extended IMediaSample interface similar to the original functions. E.g. instead of "LockBuffer" I'd suggest "GetPointerEx/GetSizeEx".


Agreed. I forgot the GetSizeEx method in my posting above. Will fix it later.


(7) I'd suggest to start the structure with VIDEOINFOHEADER2, and to just append more data to it.


Agreed. That's what I did.


(8) Additional fields I want to have in the header, all taken from the h264 specification:

- video_format
- video_full_range_flag
- colour_primaries
- transfer_characteristics
- matrix_coefficients
- chroma_format_idc
- chroma_sample_loc_type_top_field
- chroma_sample_loc_type_bottom_field
- bit_depth_luma
- bit_depth_chroma



VIDEOINFOHEADER2 already has support for parts of this values. Microsoft's doc says: If the AMCONTROL_COLORINFO_PRESENT flag is set in the dwControlFlags member, you can cast the dwControlFlags value to a DXVA_ExtendedFormat structure to access the extended color information.

http://msdn.microsoft.com/en-us/library/ff563967(VS.85).aspx

[BetaBoy]

I'm bringing in squid_80 and Haali to follow/discuss.

[madshi]

While this idea sounds good in theory, there is one problem with it.
Some formats require codec/format specific data that gets appended after the BITMAPINFOHEADER in VIDEOINFOHEADER2.
If you want to append data to the end of VIDEOINFOHEADER2, you would occupy this space.
True, so it would not be *perfectly* compatible to FORMAT_VideoInfo2, but only "mostly". But isn't 90% compatability better than none at all? Or in other words: What do we lose if we start with VIDEOINFOHEADER2?

This new format is mainly designed for data from the decoder to the renderer, not from the source/splitter to the decoder, right?
Would decoders benefit from any additional information we could provide on the splitter side of things?

I'm currently working on a DS source/splitter and just wondering if on this side anything new is required.
There's been some 3D related discussion on the Matroska mailing list in the last couple of days/weeks, too. So I think this new media type could also be very useful to connect splitters with decoders. If you have any splitter related new fields in mind that would be useful, we can consider them.

You should also include Side-by-side MPEG-2 and H.264 broadcasts. There are also 720p120 H.264 3D broadcasts.
Yes, absolutely.

Agreed. The dwNumView field already allows for more than two views.

[...]

I don't think so. It is not used in practice and would make the new media type much more difficult. The only 2D+Depth files are the ones for Philips Wow 3D displays, but they not only use 2D+Depth but some of them 2D+Depth+Background 2D+Background Depth.
Why would it make the media type more difficult? We could simply add one more "guidStereoLayout" (we should change the name of that field, though) for 2D+Depth etc. The first view index would then be 2D, index 1 would be Depth. I don't see how that would make anything more complicated?

I suggest to define a separate media type for 2D+Depth and do not cover it in our new media type.
Why? I'd have to implement 2 different media types in my renderer, then. That makes things only more complicated. I'd much prefer to have only one new media type which can handle all new formats.

Agreed. That's what I did.
No, you've inserted your new fields *before* the bmiHeader field. As a result your structure definition is not compatible to VIDEOINFOHEADER2.

VIDEOINFOHEADER2 already has support for parts of this values. Microsoft's doc says: If the AMCONTROL_COLORINFO_PRESENT flag is set in the dwControlFlags member, you can cast the dwControlFlags value to a DXVA_ExtendedFormat structure to access the extended color information.

http://msdn.microsoft.com/en-us/library/ff563967(VS.85).aspx
Cool, I didn't know that! However, does any current splitter/decoder actually fill in these values? I doubt it. I think if we add dedicated fields to the structure and make them mandatory (although they may be set to "not specified"), that would very much increase the chance of splitters & decoders actually filling in proper values. Furthermore, the DXVA_ExtendedFormat structure does not contain all the information I'm looking for.

I'm bringing in squid_80 and Haali to follow/discuss.
That's great - thanks!

Here comes my suggestion:

typedef struct tagOPENVIDEOINFOHEADER
{
VIDEOINFOHEADER2 VideoInfoHeader2;
MULTIVIEWHEADER MultiView;
VIDEOPROPSHEADER VideoProps;
DWORD Flags[8]; // Flags[0] & 0x1 = Full Range
} OPENVIDEOINFOHEADER;

typedef struct tagMULTIVIEWHEADER
{
DWORD ViewType; // enum to be defined
DWORD NumViews;
}

typedef struct tagVIDEOPROPS
{
BYTE ColourPrimaries; // enum to be defined
BYTE TransferCharacteristics; // enum to be defined
BYTE MatrixCoefficients; // enum to be defined
BYTE ChromaFormat; // enum to be defined
BYTE ChromaLocTopField; // enum to be defined
BYTE ChromaLocBottomField; // enum to be defined
BYTE LumaBitdepth;
BYTE ChromaBitdepth;
}

[pwimmer]

I've written a C++ header file that includes all the suggestions made so far. The link is in the second message of this thread.

It was possible to use a union including the VIDEOPROPS structure and dwControlFlags (that can be casted to DXVA_ExtendedFormat). This means I didn't break compatibility with the VIDEOPINFOHEADER2. New fields are only required for the stereo stuff.




typedef struct tagVIDEOPROPS
{
BYTE ColourPrimaries; // enum to be defined
BYTE TransferCharacteristics; // enum to be defined
BYTE MatrixCoefficients; // enum to be defined
BYTE ChromaFormat; // enum to be defined
BYTE ChromaLocTopField; // enum to be defined
BYTE ChromaLocBottomField; // enum to be defined
BYTE LumaBitdepth;
BYTE ChromaBitdepth;
}

ColourPrimaries, TransferCharacteristics, MatrixCoefficients, ChromaFormat and FullRange are already covered by DXVA_ExtendedFormat. I just added a few new items to the enumerations. To ensure that we do not have a conflict if Microsoft should add their own items in the future as well, I used the highest possible values for the new items instead of the next available value. ChromaLocTopField and ChromaLocBottomField are imho not required, they are covered by DXVA2_VideoChromaSubsampling. LumaBitdepth and ChromaBitdepth is implicitely defined by the FOURCC, there is no need to store it twice. Consequently, all required fields are already present in DXVA_ExtendedFormat so we can stay compatible.

[madshi]

I've written a C++ header file that includes all the suggestions made so far.
We're working together to define the structures. No offense, but does it make sense to put your name as the copyright holder above it? If at all, shouldn't this thread be the copyright holder?

Redistributions in binary form must reproduce the above copyright
No, thanks!! Your copyright conditions are worse than using media types defined by MS!

It was possible to use a union including the VIDEOPROPS structure and dwControlFlags (that can be casted to DXVA_ExtendedFormat). This means I didn't break compatibility with the VIDEOPINFOHEADER2. New fields are only required for the stereo stuff.
I would still prefer dedicated fields in the header, because this whole logic with "AMCONTROL_COLORINFO_PRESENT" makes the VIDEOPROPS fields look very much optional. I want them to be mandatory. If we make them optional, then once again nobody will bother using them.

LumaBitdepth and ChromaBitdepth is implicitely defined by the FOURCC, there is no need to store it twice.
FOURCC defines 8bit, 10bit or 16bit. Intermediate values are not defined (at least not by MS). Also the existing FOURCCs always define the same bitdepth for Luma and Chroma. The h264 spec allows *any* bitdepth between 8bit and 14bit, and it allows different bitdepths for Luma and Chroma. So let's consider a video track with 9bit Luma and 8bit Chroma. What will a renderer output? Probably 10bit for both Luma and Chroma. But now the video renderer does not know the *native* bitdepth of the source. And that information could be very useful for specific processing algorithms (e.g. anti-banding post processing).

BTW, your structure is still incompatible with VIDEOINFOHEADER2, because the bmiHeader field is in the wrong place, as I said before.

For example, for a 1920 x 1080 video containing squeezed side-by-side format
// the dwPictAspectRatioX should be set to 32 and dwPictAspectRatioY to 9.
This is debatable. The 32:9 aspect ratio would describe what is transfered downstream. But dwPictAspectRatioX/Y actually does not describe what is transfered, but what should ultimately be visible on the screen. And that is 16:9. Whether the side-by-side is squeezed or not is very obvious from the bmiHeader information. Actually it could even be half squeezed, or it could be anamorphically squeezed in addition to the side-by-side squeeze, no problem at all. No need to misuse the dwPictAspectRatioX/Y fields.

// ----------------------------------------------------------------------------
// Stereoscopic layout information
// ----------------------------------------------------------------------------
As said before, please don't use "stereo", please replace it with "Multiview" or something not stereo-specific.

GUID guidStereoLayout; // See supported STEREOLAYOUT_xxx GUIDs above
union {
STEREOLAYOUTPARAMS1 stereoLayoutParams1; // Layout-specific parameters
STEREOLAYOUTPARAMS2 stereoLayoutParams2; // Layout-specific parameters
DWORD dwReserved3[2]; // Set to zero if neither of the STEREOLAYOUTPARAMS structures is used
};
The union solution is somewhat "clever", but I'd prefer a simple flat structure with all fields in it. Just imagine you want to log out the information fields of the media type and the multiview GUID is not known to you (it might be a new GUI defined after your software was written). In this situation you don't know which of the unions is the correct one to use. Because of that I'd prefer a flat structure with all fields in it.

// Video transfer matrix. Values are identical to the
// DXVA2_VideoTransferMatrix and MFVideoTransferMatrix enumerations.
I'd like the media type to have a way to signal that the original source was encoded in RGB and not YCbCr. Maybe we could add a transfer matrix for RGB (basically a passthrough matrix) for that purpose? The reason why I want to know which format the original source had is that my renderer insists on doing all video processing, so it refuses a connection which would result in the decoder doing any kind of conversion. Which means that I'd like to be able to see from the media type information whether the connection would be a "native" connection or not. If the original source was YCbCr, my renderer will not accept an RGB connection and vice versa...

Views must be ordered from left to right. For stereoscopic content (dwNumViews = 2), view 0 is the left and view 1 the right view. For multiview content (dwNumViews > 2), view 0 is the most-left view, view dwViewNum-1 is the most-right view.
If that is the case then why do you have different "STEREOLAYOUT_SIDEBYSIDELEFTFIRST" and "SIDEBYSIDERIGHTFIRST" GUIDs? I'd simply drop that quoted paragraph.

The IOpenMediaSample interface must be supported by media samples if the field guidStereoLayout in the STEREOVIDEOINFOHEADER structure is set to the value STEREOLAYOUT_MULTIBUFFERS.
Why only for MULTIBUFFERS? E.g. for 2DDEPTH IOpenMediaSample would also be needed.

Number of horizontal tiles (only for STEREOLAYOUT_TILEDxxx, zero otherwise)
I'd say "1 otherwise". Makes more sense to me.

[pwimmer]

I've addressed the issues you noticed.

* The license terms are less strict.

* A Doom9 thread cannot be a copyright holder, only individuals. Of course I'll add other contributers as well. Let me know name & email address.

* VIDEOPROPS will always be optional because in many causes the information is unknown. In this case I prefer the value "unknown" instead of forcing a filter to write some mess in these fields. If it is "unknown", it gives the application the chance to the ask the user or apply application-specific defaults.

* You should define new FOURCCs for currently unsupported bit depths. Using existing FOURCCs with wrong bit depths will introduce more problems than it solves because developers will not expect other bit depths than those defined by Microsoft. Is there really content with unusual bit depths? The H.264 spec is very flexible, but that doesn't mean that all possible combinations are used in practice. Lets see the optinion of other people on this issue...

* Using dwPictAspectRatioX and dwPictAspectRatioY is a very convenient way to make squeezed stereo files display properly in 2D media players. Although the output is not stereo but still two views, at least the image is not distored. Imho it is a clean solution, it is pretty much the same situation as with anamorphic 2D content. dwPictAspectRatioX and dwPictAspectRatioY are designed for the purpose of making the video display in the proper aspect ratio, so why not use it for stereo content as well? Anyway, I added an alternative that allows to leave dwPictAspectRatioX and dwPictAspectRatioY untouched: StereoFlags_HalfHorizontalResolution and StereoFlags_HalfVerticalResolution. But I would allow both methods.

* I modified the text to mention "multiview" in addition to "stereoscopic". As multiview content is typically used for autostereoscopic displays, I do not consider the term "stereoscopic" wrong or inappropriate. The specification currently doesn't cover any other use of multiview content than for autostereoscopic displays. If somebody is familiar with other usage scenarios of multiview content (e.g. multi viewpoint video) and has suggestion what could be added to better support such scenarios, then let me know.

* I completely gave up the union stuff in favor of a flat structure.

* The concept of a passthrough matrix is rather confusing and I doubt any filter developer would support it. I added a much simpler feature that allows to find out if a media type is the native one: The flags AdvancedFlags_NativeFormatUnknown, AdvancedFlags_NativeFormatTrue and AdvancedFlags_NativeFormatFalse.

* I have different "STEREOLAYOUT_SIDEBYSIDELEFTFIRST" and "SIDEBYSIDERIGHTFIRST" GUIDs because I want the GUIDs to be as precise as possible. "Views must be ordered from left to right" only applies to the STEREOLAYOUT_MULTIBUFFER GUID. A filter can easily change the order by swapping pointers while it would mean memcpys for other layouts. Thus I only created a single GUID for the multi buffer layout but not for the other layouts. I updated the comments to clarify the issue.

* The latest version contains many new stereo layouts and desciptions for all layouts. Most notably, I added STEREOLAYOUT_UNKNOWN that should be used instead STEREOLAYOUT_MONOSCOPIC if the actual content is unknown.

* The latest version also includes many new fields in the new STEREOLAYOUTPROPS structure, e.g. orientation (rotation and flipping), cropping, parallax adjustment. All these stuff is used and required in practice, e.g. for content recorded with mirror rigs, beam splitter or other optical stereo attachments.

[madshi]

I like some/most of the changes, but I still see some issues. You seem to know a lot more about 3D/multiview related things than I do, so there's not much I can say about that.

(1) Again: Your structure is not compatible with VIDEOINFOHEADER2, because bmiHeader is in the wrong place.

(2) The license terms are much better. But IMHO we don't need any copyright at all. But let's hear what other people have to say about that.

(3) About VIDEOPROPS being optional: The same argument you're using could also be used for the multiview properties - yet all those fields are there and must be filled. There's no "this_is_multiview_content" flag which must be set first to activate the multiview structures. I want the same for VIDEOPROPRS. Of course there should be an "undefined" or "unknown" value for every VIDEOPROPS field. But having to activate the VIDEOPROPRS structure first by setting an additional flag is not good enough for me. If the VIDEOPROPS information is unknown, then filters can set the fields that way. But very often the information is known (or could easily be retrieved) and it's just not set because of lazyness.

(4) I just noticed that one original field I was asking for, namely a "ChromaFormat (4:2:0, 4:2:2, 4:4:4)" is missing. Yeah, I already hear you say: "You can see that from the FOURCC". No, you can't, because an intermediate filter (e.g. ffdshow raw video processor) could already have applied chroma upsampling. Or the decoder could have internally upsampled chroma for e.g. YUY2 output. I want to know what format the original source had, not what the upstream filters have converted the original source data to.

(5) Right now (4) made me think: Maybe there should be 2 VIDEOPROPS structures: One for the native format. And another one for the "current" data format. Something like "NativeVideoProps" and "CurrentVideoProps".

(6) "Using existing FOURCCs with wrong bit depths will introduce more problems than it solves". I never meant to suggest using incorrect bitdepths. Of course if a filter uses a 10bit FOURCC then the data must really be 10bit. But if the original video source was only 9bit then the least significant bit of that 10bit data will always be 0. And that is an important thing to know for the video renderer. It makes no sense at all to define dozens of different FOURCCs for any wild bitdepth combination. No decoder writer would ever implement dozens of FOURCCs for that purpose. Too much programming time needed. Too much time needed for testing. It makes *much* more sense to e.g. always have the filter use a 16bit FOURCC with the video data "upconverted" to 16bit, with additional information fields that indicate how many least significant bits are zeroed out.

(6) About dwPictAspectRatioX/Y. I was just starting to write a long text about how we should set it to 16:9, but then I thought: If there's a video renderer which doesn't really support/know 3D at all, then if we set aspect ratio to 16:9 for "side-by-side (full)" content, the video renderer will draw the image incorrectly. So I have to change my opinion and agree with your original suggestion to use 32:9 for side-by-side (full), for better compatability to non-3D-aware video renderers. Which also means that I vote for removing those half/full flags again.

(6) What happens if the decoder wants to fill the VIDEOPROPS fields, but the content is originally RGB? In that case the current transfer matrix and chroma subsampling options don't fit. The correct transfer matrix for RGB is "passthrough". You could also name it "no matrix (RGB)" or something like that.

(7) I fear that the native flags might not work as intended. E.g. if a decoder programmer finds that his decoder does not connect to my renderer if he sets "NativeFormatFalse", then he might misunderstand that as a bug in my renderer and as a "fix" he may decide to always set the flag to "True" or "Unknown". As a result I'd be screwed. I'd prefer to have decoder programmers fill in the correct values in the VIDEOPROPS structure, then I can check myself what they've done.

So my suggestion would be this:

typedef struct tagOPENVIDEOINFOHEADER {
VIDEOINFOHEADER2 videoInfo2;
VIDEOPROPS nativeVideoProps;
VIDEOPROPS currentVideoProps;
STEREOLAYOUTPROPS stereoLayoutProps;
UINT reserved[8]; // Set to zero when writing and ignore when reading
} OPENVIDEOINFOHEADER;

I also like to suggest using "videoInfo2" like that, because it makes the structure look much smaller and easier to understand. If you list all VIDEOINFOHEADER2 fields one by one in your structure, your structure looks much more intimidating.

[nevcairiel]

You should forget the compat with VIH2. Devs need to modify their code anyway for the new format, and as long as the fields are the same, plus new ones, you can still parse it with the same code.
Having a VIH2 at the top is rather confusing, because the BMI in the VIH2 can actually grow with extradata. How would you handle this? Move it to the end of the OVH? Your compat is already gone, then.

I'm with pwimmer on the basic design, copy VIH2 and add new fields, keep BMI at the end.

[pwimmer]

Having a VIH2 at the top is rather confusing, because the BMI in the VIH2 can actually grow with extradata. How would you handle this? Move it to the end of the OVH? Your compat is already gone, then.


Having the full VIH2 at the top would be nice, but as you say it is not possible because BITMAPINFOHEADER is a variable size structure, so it must be at the end.

All the other fields should be identical, which is the reason why I would like to keep the VIH2-compatible VIDEOPROPS. Anyway, the VIDEOPROPS is big improvement over the dwControlFlags, because it's now obvious what information it contains. Many developers didn't know that dwControlFags can be casted to DXVA_ExtendedFormat. At least this issue is solved, which should increase the chance that the structure ios actually used. By keeping it compatible we even increase the change that developers even use it for VIH2, too.

[pwimmer]

(3) About VIDEOPROPS being optional: The same argument you're using could also be used for the multiview properties - yet all those fields are there and must be filled. There's no "this_is_multiview_content" flag which must be set first to activate the multiview structures. I want the same for VIDEOPROPRS. Of course there should be an "undefined" or "unknown" value for every VIDEOPROPS field. But having to activate the VIDEOPROPRS structure first by setting an additional flag is not good enough for me.


I agree it is not good that there is a separate flag to activate the VIDEOPROPS structure. But that's how Microsoft specified it and since we can reuse the structure, it makes sense to stay compatible.

There is no flag to active the STEREOLAYOUTPROPS, but it is optional, of course. If all byte are set to zero (thus, if the developer ignores it), this defaults to STEREOLAYOUT_UNKNOWN (which is deliberately defined as GUID_NULL). All the other fields in the structure are defined in a way that ensures that 0 always means the default. A renderer that ignores STEREOLAYOUTPROPS and a renderer that obeys STEREOLAYOUTPROPS will output the same image when all fields of STEREOLAYOUTPROPS are zero.


(4) I just noticed that one original field I was asking for, namely a "ChromaFormat (4:2:0, 4:2:2, 4:4:4)" is missing. Yeah, I already hear you say: "You can see that from the FOURCC". No, you can't, because an intermediate filter (e.g. ffdshow raw video processor) could already have applied chroma upsampling. Or the decoder could have internally upsampled chroma for e.g. YUY2 output. I want to know what format the original source had, not what the upstream filters have converted the original source data to.


Ok, now I understand what you mean. But I don't believe the fields are well-thought-out yet. What do you do if the filter delivers 4:4:4 or RGB and says the original chroma format was 4:2:0? You can't do anything, because you do not know how the upsampling was performed. You cannot undo it. It is simply impossible to restore the original data. To find out if it is not the native format, the flags I've already introduced are sufficient.


(6) But if the original video source was only 9bit then the least significant bit of that 10bit data will always be 0.


Actually padding with 0 is the wrong way to convert to higher bit depths. This would make the image slightly darker than it should be.

EDIT: I was wrong. http://msdn.microsoft.com/en-us/library/bb970578%28VS.85%29.aspx#_420formats specifies that bit shifting should be used!


(6) About dwPictAspectRatioX/Y. I was just starting to write a long text about how we should set it to 16:9, but then I thought: If there's a video renderer which doesn't really support/know 3D at all, then if we set aspect ratio to 16:9 for "side-by-side (full)" content, the video renderer will draw the image incorrectly. So I have to change my opinion and agree with your original suggestion to use 32:9 for side-by-side (full), for better compatability to non-3D-aware video renderers. Which also means that I vote for removing those half/full flags again.


There are pros and cons for both approaches. To con of dwPictAspectRatioX/Y is that the display size in a 2D player depends on the decoder being used. A old decoder not supporting the spec would output a squeezed image while a new decoder would output a full-size image. The flags avoid this issue, while still allowing a 3d-enabled renderer/player to properly decode the side-by-side format and display at correct aspect ratio.

Imho, there is no clear winner which approach is better, so I would keep both and let the developer decide.


(6) What happens if the decoder wants to fill the VIDEOPROPS fields, but the content is originally RGB? In that case the current transfer matrix and chroma subsampling options don't fit. The correct transfer matrix for RGB is "passthrough". You could also name it "no matrix (RGB)" or something like that.


For RGB content, filters should write VideoPrimaries_Unknown and VideoTransferMatrix_Unknown; when reading, a filter should ignore videoTransferMatrix and videoPrimaries.

This is the way it is already handled in VIH2 and I see no need to change it.


(7) I fear that the native flags might not work as intended. E.g. if a decoder programmer finds that his decoder does not connect to my renderer if he sets "NativeFormatFalse", then he might misunderstand that as a bug in my renderer and as a "fix" he may decide to always set the flag to "True" or "Unknown". As a result I'd be screwed. I'd prefer to have decoder programmers fill in the correct values in the VIDEOPROPS structure, then I can check myself what they've done.


But if the programmer correctly fills in nativeVideoProps and currentVideoProps, you filter would refuse connection as well (because it is not the native format), resulting in exactly the same problems.

It's more likely that developers are willing to properly set a single flag than a complex structure.

[madshi]

Having a VIH2 at the top is rather confusing, because the BMI in the VIH2 can actually grow with extradata. How would you handle this?
How big can it grow? As far as I can see, the max size would be "RGBQUAD bmiColors[256]", right? So the problem could be solved like this:

typedef struct tagOPENVIDEOINFOHEADER {
VIDEOINFOHEADER2 videoInfo2;
RGBQUAD bmiColors[256];
VIDEOPROPS nativeVideoProps;
VIDEOPROPS currentVideoProps;
STEREOLAYOUTPROPS stereoLayoutProps;
UINT reserved[8]; // Set to zero when writing and ignore when reading
} OPENVIDEOINFOHEADER;
Violà, perfect compatability with VIDEOINFOHEADER2. Or am I missing something?

[madshi]

I agree it is not good that there is a separate flag to activate the VIDEOPROPS structure. But that's how Microsoft specified it and since we can reuse the structure, it makes sense to stay compatible.
You say yourself that many devs didn't know about DXVA_ExtendedFormat. So I don't see a big loss if we don't use that, but define our own fields instead. Using our own fields would not be incompatible, either, because they'd be outside of VIDEOINFOHEADER2.

Ok, now I understand what you mean. But I don't believe the fields are well-thought-out yet. What do you do if the filter delivers 4:4:4 or RGB and says the original chroma format was 4:2:0? You can't do anything, because you do not know how the upsampling was performed. You cannot undo it. It is simply impossible to restore the original data.
Correct. But I can inform the user, saying that the decoder has upsampled chroma and that he should modify the decoder settings to disable the chroma upsampling, or switch to another decoder.

Actually padding with 0 is the wrong way to convert to higher bit depths.

EDIT: I was wrong. http://msdn.microsoft.com/en-us/library/bb970578%28VS.85%29.aspx#_420formats specifies that bit shifting should be used!
Doing anything else would introduce banding, so bit shifting is the only reasonable solution. (Unless you want to go the full mile with floating point and dithering etc).

To con of dwPictAspectRatioX/Y is that the display size in a 2D player depends on the decoder being used. A old decoder not supporting the spec would output a squeezed image while a new decoder would output a full-size image.
I don't understand what you mean here. Can you explain?

For RGB content, filters should write VideoPrimaries_Unknown and VideoTransferMatrix_Unknown
Wrong. The primaries are very important for RGB content, too!!

But if the programmer correctly fills in nativeVideoProps and currentVideoProps, you filter would refuse connection as well (because it is not the native format), resulting in exactly the same problems.
With nativeVideoProps/currentVideoProps, the programmer would have to actually lie to make my filter connect. With the flags you have suggested he simply has to leave it at "0", which is even the default value (provided that he uses memset(0))! Furthermore, if I have nativeVideoProps and currentVideoProps, I can see exactly what the filter chain has done and so I can give exact tips to the end user about what he has to change to improve image quality.

[pwimmer]

How do you like this solution for > 8 bpp?


// 10 bit and 16 bit YUV can contain content that has been converted from 9
// to 10 bit or from 11-15 to 16 bit by using bitshifting. To allow a
// renderers to use optimized algorithms that minimize banding, the following
// fields define the original bit depth of the luminance and chrominance
// channels. If native format is 10 or 16 bit, respectively, the fields can
// either be set to 0 or 10/16. If the native format is unknown or the FOURCC
// is a full range format, which means the maximum value is 2^n-1 and not
// 255*(2^(n-8)), the fields should be set to 0. If only bit shifting has
// been applied on the native data, AdvancedFlags_NativeFormatTrue should be
// set. The following link contains a list of bit-shifted YUV formats:
// http://msdn.microsoft.com/en-us/library/bb970578(v=VS.85).aspx
UINT originalLumaBitCount;
UINT originalChromaBitCount;

[madshi]

How do you like this solution for > 8 bpp?
I like it, but I'd suggest to shorten the comment a bit. How about this?

"10 bit and 16 bit YUV can contain content that has been converted from a lower bitdepth by using bitshifting. To allow the video renderer to use optimized algorithms that minimize banding, the following fields define the original bit depth of the luminance and chrominance channels. These fields should be set (and should only be set) if a lower bitdepth source was upconverted to the output bitdepth by using bitshifting."

[nevcairiel]

How big can it grow? As far as I can see, the max size would be "RGBQUAD bmiColors[256]", right? ?

Not really. Compressed formats send private data in that part thats not specifically limited by any length. For example H264 decoders expect a SPS and a PPS NALU in there.
While i haven't seen a format with really long private data, limiting it here seems wrong.

[madshi]

Not really. Compressed formats send private data in that part thats not specifically limited by any length. For example H264 decoders expect a SPS and a PPS NALU in there.
How do these filters calculate the offset where the private data begins?

[pwimmer]

A new suggestion that replaces the nativeVideoProps and the native format flags: Degradation flags that are passed from the source downstream to the renderer in the OPENVIDEOINFO structure. Each filters adds flags if it performs lossy operations on the content.


// Degradation flags. These flags specify all kinds of degradation that
// occured in the DirectShow filter graph pipeline. Filters should add the
// appropriate flags if they do any modification to the content. Flags
// already set by upstream filters must be preserved. Muxers that combine
// two video streams (e.g. left and right stream to a stereo layout) should
// use binary-or on all input stream flags to create the output stream flags.
typedef enum tagVIDEODEGRADATIONFLAGS {
VideoDegradation_FlagsValid = 0x00000001, // Set if the degradation flags are valid
VideoDegradation_Other = 0x00000002, // A degradation other than the ones below occured
VideoDegradation_Incomplete = 0x00000004, // An upstream filter does not support this flag, so the flags might be incomplete
VideoDegradation_ChromaUpsampled = 0x00000008, // Chroma component upsampled e.g. 4:2:0 to 4:2:2
VideoDegradation_ChromaDownsampled = 0x00000010, // Chroma component downsampled e.g. 4:2:2 to 4:2:0
VideoDegradation_Upsampled = 0x00000020, // Image has been upscaled to higher resolution
VideoDegradation_Downsampled = 0x00000040, // Image has been downscaled to lower resolution
VideoDegradation_Cropped = 0x00000080, // Image has been cropped
VideoDegradation_Rotated = 0x00000100, // Image has been rotated
VideoDegradation_Flipped = 0x00000200, // Image has been flipped
VideoDegradation_PrecisionIncreased = 0x00000400, // The bits per pixels have been increased (do not set if filter can undo the operation by bitshifting, see originalLumaBitCount and originalChromaBitCount)
VideoDegradation_PrecisionDecreased = 0x00000800, // The bits per pixels have been decreased (do not set if YUV is converted to RGB and the bit depth per component, e.g. 8 bpp remains the same)
VideoDegradation_FramerateIncreased = 0x00000800, // Interpolated frames added
VideoDegradation_FramerateDecreased = 0x00001000, // Frames dropped
VideoDegradation_YUVtoRGB = 0x00002000, // Colorspace conversion from YUV to RGB
VideoDegradation_RGBtoYUV = 0x00004000, // Colorspace conversion from RGB to YUV
VideoDegradation_ColorToGrey = 0x00008000, // Chroma channel removed or cleared
VideoDegradation_ColorspaceConversion = 0x00010000, // Colorspace conversion other than YUV to RGB, RGB to YUV or color to grey (e.g. RGB to YCgCo)
VideoDegradation_RemovedAlpha = 0x00020000, // Removed alpha channel
VideoDegradation_PrimariesChanged = 0x00040000, // Image converted to use other primaries
VideoDegradation_TransferFunctionChanged = 0x00080000, // Gamma correction applied
VideoDegradation_VideoProcAmp = 0x00100000, // Brightness, contast, saturation, hue and similar adjustments
VideoDegradation_LossyStereoFormatConversion = 0x00200000, // Loss stereoscopic format conversion, e.g. conversion to checkerboard or anaglyph layouts
VideoDegradation_StereoFormatInterpolation = 0x00400000 // Additional interpolated pixels added during stereo format conversion but existing pixel values remained untouched, e.g. from Sensio to side-by-side layout

// Future flags might be added in the future and must be preserved by filters.
} VIDEODEGRADEATIONFLAGS;

[madshi]

:p I still prefer my suggestion, but I guess I could live with yours. A few comments on those flags:

(1) Rotation (if 90° wise) and flipping is probably lossless? So should we consider that a degradation?
(2) "Cropped" needs a clear definition. E.g. I don't consider applying cropping specified by the h264 bitstream as a degradation, but rather as a must, because without h264 cropping h264 Blu-Rays would be 1920x1088.
(3) "YUV" is usually used for analog content. The correct term for digital data is "YCbCr", AFAIK.
(4) I think there needs to be one more degradation flag for changing of the "transfer matrix"?

I like how you named the flags "degradation" and not "modification" or something. That shows nicely that such changes are usually considered to be a bad idea... :)

[pwimmer]

:p I still prefer my suggestion, but I guess I could live with yours. A few comments on those flags:

(1) Rotation (if 90° wise) and flipping is probably lossless? So should we consider that a degradation?
(2) "Cropped" needs a clear definition. E.g. I don't consider applying cropping specified by the h264 bitstream as a degradation, but rather as a must, because without h264 cropping h264 Blu-Rays would be 1920x1088.
(3) "YUV" is usually used for analog content. The correct term for digital data is "YCbCr", AFAIK.
(4) I think there needs to be one more degradation flag for changing of the "transfer matrix"?

I like how you named the flags "degradation" and not "modification" or something. That shows nicely that such changes are usually considered to be a bad idea... :)

ad 1)

Agreed, the flag should only be set for rotations != 90° and 180°. Maybe it should even be named "resampled" to cover other kinds of distortions as well. Flipping should indeed be removed as it is lossless.

ad 2)

The flag should indeed not be set in the special case 1088 pixels height as no valid content is lost.

ad 3)

Micosoft also uses the term YUV in the DirectShow SDK, so it is not completely wrong. I admit that I'm not sure about the actual difference...

ad 4)

Use VideoDegradation_ColorspaceConversion if the conversion is from one YUV to a different YUV space, or VideoDegradation_YUVtoRGB if the destination is RGB.

[pwimmer]

:p I still prefer my suggestion, but I guess I could live with yours.

The degradation flags are more flexible and also easier to implement by filter developers.

For example, you couldn't detect if a decoder performs contrast/brightness adjustments (which you certainly want to do yourself in your renderer) with your approach, while it is possible with the degredation flags.

[madshi]

The flag should indeed not be set in the special case 1088 pixels height as no valid content is lost.
Why special casing 1088? Should we not "allow" the full h264 cropping to be executed without a degradation flag having to be set? I think the cropping degradation flag should only be set if a filter crops more than the video bitstream asks for.

Micosoft also uses the term YUV in the DirectShow SDK, so it is not completely wrong. I admit that I'm not sure about the actual difference...
I think I was wrong. It's "YPbPr" which is for analog and "YCbCr" for digital. "YUV" seems to be a valid alternative name for "YCbCr". I've confused "YUV" with "YPbPr".

For example, you couldn't detect if a decoder performs contrast/brightness adjustments (which you certainly want to do yourself in your renderer) with your approach, while it is possible with the degredation flags.
True.

[nevcairiel]

How do these filters calculate the offset where the private data begins?

Its just plainly appended after the BMI, and in the BMI is a field that holds the actual size of the thing. (Actually with VIH2 there is a extra null byte between the BMI and the extradata). The formats do not hold color tables, for that matter.

If you want to go with VIH2 as the first member of your new structure, i would do it similar to how MPEG2VIDEOINFO looks, and provide your own place to store extradata (as well as a field that indicates its length)
It would still break compat with VIH2 if the extradata is used, but at least you won't be re-creating the structure, and your concerns of it looking like a "beast" are at least adressed.

[madshi]

If you want to go with VIH2 as the first member of your new structure, i would do it similar to how MPEG2VIDEOINFO looks, and provide your own place to store extradata (as well as a field that indicates its length)
It would still break compat with VIH2 if the extradata is used, but at least you won't be re-creating the structure, and your concerns of it looking like a "beast" are at least adressed.
Ok, I like the MPEG2VIDEOINFO logic. @pwimmer?

[pwimmer]

I think the easiest solution is to place VIH2 at the end of the structure...

[madshi]

Hmmmm... Yes, I like that even better! It's a bit different to what MS is normally doing, but I like it.

[pwimmer]

Updated the header file (link is in second posting).

The structure is very simple now:


typedef struct tagOPENVIDEOINFOHEADER {
EXTRAVIDEOINFO extraVideoInfo; // All extra stuff not part of VIDEOINFOHEADER2
VIDEOINFOHEADER2EX videoInfoHeader2; // Compatible with VIDEOINFOHEADER2
} OPENVIDEOINFOHEADER;



The VIDEOINFOHEADER2EX is compatible with VIDEOINFOHEADER2, it just defines a union that allows easy access to the VIDEOPROPS.

[yesgrey]

We're working together to define the structures. No offense, but does it make sense to put your name as the copyright holder above it? If at all, shouldn't this thread be the copyright holder?
I've addressed the issues you noticed.
* The license terms are less strict.
* A Doom9 thread cannot be a copyright holder, only individuals. Of course I'll add other contributers as well. Let me know name & email address.
Before starting my participation on this I would like to clarify a little further the copyright idea...

In first place, I don't see why it should have a copyright at all... Are you thinking in getting any money from it? If you don't, all this copyright talking might only serve to keep away from it some potential users...

If we really want to create an open source format that would be used by a lot of people, we should target to a completely open format without any copyrights.

Let me show you an example... A while ago, in a doom9 thread, me, madshi, tritical and IanB have defined an open file format for 3D LUTs, and we haven't created any copyright. Recently, we even decided to create a sourceforge project to hold the entire format. You can take a look at it here (http://thr3dlut.sourceforge.net/).

I think that this idea would work better if you would do something similar. A sourceforge project is very simple to create and handle, and then other people involved would also be able to edit and/or update the file format whenever it would be needed.

What do you think?:)

[Wilbert]

Before starting my participation on this I would like to clarify a little further the copyright idea...

In first place, I don't see why it should have a copyright at all... Are you thinking in getting any money from it? If you don't, all this copyright talking might only serve to keep away from it some potential users...
That's not possible this way. Every released work is copyrighted (although that can be implicitly) unless disclaimed otherwise. You just need to release it explicitly in the public domain:

http://www.gnu.org/philosophy/categories.html#PublicDomainSoftware

[yesgrey]

That's not possible this way. Every released work is copyrighted (although that can be implicitly) unless disclaimed otherwise. You just need to release it explicitly in the public domain:

http://www.gnu.org/philosophy/categories.html#PublicDomainSoftware
Thanks for the clarification, I didn't know that.

[madshi]

So where does that leave us?

It would be great if we could get feedback from the commercial guys I've invited to this thread (e.g. CoreAVC, Nero etc). Are you guys ok with the current copyright? Or do you need the spec to be public domain for you to use it?

Personally, I'd prefer if the spec were public domain and if it would be managed through Sourceforge (or a comparable site), with at least 2-3 doom9 devs having edit rights.

[yesgrey]

Personally, I'd prefer if the spec were public domain and if it would be managed through Sourceforge (or a comparable site), with at least 2-3 doom9 devs having edit rights.
I would prefer this too.

[pwimmer]

I would prefer this too.

I'll setup a SourceForce project for it. It is certainly best that the spec is avaiable on a neutral place.

Before I put it on SourceForce, we should agree at least on a final name so that we don't have to rename the SourceForge project later on. Currently, the name is "Open Media Format". As it is not released under one of the typical Open Source licenses but allow unrestricted commercial, I'd rather prefer a name without "Open". Some people could associate it with Open Source, which it isn't. Here are other proposals:

"Extended Video Info"
"Flexible Video Info"
"Advanced Video Info"

[pwimmer]

It would be great if we could get feedback from the commercial guys I've invited to this thread (e.g. CoreAVC, Nero etc).

We need the feedback of the commercial guys indeed. Not only on the copyright issue (which I believe should be fine) but also on the technical aspects. The goal is interoperability between stereo-capable filters of different vendors, which can only be achived if it is adopted by most vendors. I'm also missing features beyond stereoscopy. Up to know, we only support the degradation flags and have improved the video props (gamma, transfer matrix, primaries, ...). Any other ideas?

The very best feedback would be one by Microsoft, in the ideal case we can make the structure compatible with the VIDEOINFOHEADER3 that might be part of Windows 8. I have the feeling that Microsoft will add stereo support in Windows 8, the question is, only in DirectX and Media Foundation or also in DirectShow.

[madshi]

I'll setup a SourceForce project for it.
Great!

Before I put it on SourceForce, we should agree at least on a final name so that we don't have to rename the SourceForge project later on. Currently, the name is "Open Media Format". As it is not released under one of the typical Open Source licenses but allow unrestricted commercial, I'd rather prefer a name without "Open". Some people could associate it with Open Source, which it isn't. Here are other proposals:

"Extended Video Info"
"Flexible Video Info"
"Advanced Video Info"
Hmmmm... That's a tough choice. Let me add a few alternatives:

- Enhanced Video Info
- Detailed Video Info
- Video Info 2010
- Video Info XT (for extended)
- Video Info Pro

It is tempting to use 3D or stereo in the name, but I think we should avoid that, because otherwise many devs will think the new media type is "only for 3D".

[yesgrey]

I vote for "Extended Video Info".

"Flexible" doesn't give the idea that is something new.
"Advanced", after awhile, is note advanced anymore, but simply current usage.
"Detailed" seems to indicate that previously it was just simple video info.
"Enhanced" seems too short, because I feel it's much more than just enhancing...
The 2010 would seem like it's info related only to this year, the XT would seem something related to ATI, and the Pro would seem that previously it wasn't nothing too serious...

Another option would be "Multiple Video Info", but I think I prefer the "Extended", even though it might sound too Microsoftish...

I just tried to explain my feelings about each name, I hope I didn't hurt anybody's feelings... ;)

[nevcairiel]

Open Source doesn't mean it cannot be used by Commercial Products. Just license it as Public Domain or MIT/BSD. The BSD license is a recognized Open Source license that basically allows anyone to use it for anything, they just have to somewhere ship the license notice. Public Domain doesn't require any of this, while its strictly not a license, it does qualify for the definition of open source.
What you think about as "open source" are probably licenses like the GPL that require any project that uses it to also be open source (and non-commercial), but thats only one of the recognized open source licenses, and many licenses actually do not require this. Anyone actually working professionally in this business would hopefully know the difference, and wouldn't be confused by the name.

I would stay with Open Media Format, its the best name yet.

[yesgrey]

I would stay with Open Media Format, its the best name yet.
Agreed.

[BetaBoy]

We need the feedback of the commercial guys indeed. Not only on the copyright issue (which I believe should be fine) but also on the technical aspects..

Give me a chance to ping everyone here internally.

[BetaBoy]

It is tempting to use 3D or stereo in the name, but I think we should avoid that, because otherwise many devs will think the new media type is "only for 3D".
I agree.

[pwimmer]

I would stay with Open Media Format, its the best name yet.

If we keep the word "Open", we should at least rename it to Open Video Info. Open Media Format sounds like a container or codec. Video Info is more appropriate.

[BetaBoy]

madshi and Peter.... I spoke to Haali and he is following this thread. He will help support OMF and what is needed in his splitter. CoreCodec can also provide the splitter source code to Peter as needed to help make it happen.

[madshi]

@BetaBoy,

thank you, I appreciate that. So you Core guys have no specific wishes for further enhancements to the media type record? Are you ok with the current copyright (see header file linked to in the 2nd post)?

[madshi]

I'd like to suggest a small addition. Something like this:

typedef struct tagOPENVIDEOINFOHEADER {
int versionNumber;
EXTRAVIDEOINFO extraVideoInfo; // All extra stuff not part of VIDEOINFOHEADER2
byte reservedSpace [1024];
VIDEOINFOHEADER2EX videoInfoHeader2; // Compatible with VIDEOINFOHEADER2
} OPENVIDEOINFOHEADER;
The new fields "versionNumber" and "reservedSpace" would give us the following advantages:

(1) ability to add further information fields in the future
(2) without breaking compatability to older software (older software would simply ignore the added fields)
(3) and with comfortable direct access to all structure elements.

Not sure if those 2 new fields should be outside or inside of EXTRAVIDEOINFO, that's not important for me. Anyway.

[nevcairiel]

Thats quite alot of reserved space. I know there are some reserved DWORD fields in other headers, do you really expect to use 1kb of data in the future?
On the version thing.. One version number might be good, for compatible changes. However, incompatible changes should be done with a new format GUID to avoid any problems whatsoever. Also doesn't require any custom code by implementing filters to reject the connection.

[madshi]

Thats quite alot of reserved space. I know there are some reserved DWORD fields in other headers, do you really expect to use 1kb of data in the future?
I've no idea. Maybe 1kb is overkill. But then: Isn't it better to reserve too much memory instead of later finding out that we were too cautious? Always remember how Bill Gates thought 640kb of RAM were enough.

On the version thing.. One version number might be good, for compatible changes. However, incompatible changes should be done with a new format GUID to avoid any problems whatsoever. Also doesn't require any custom code by implementing filters to reject the connection.
Well, yes, probably you're right. So let's use only one version number.

[nevcairiel]

Always remember how Bill Gates thought 640kb of RAM were enough.


Thats just a myth, he never said that =p

The amount of reserved memory doesn't really concern me anyway, 1kb isn't what i would call "wasted space", its not like we alloc this structure constantly all over again, its usually a one time deal.
It just seemed like alot, considering most headers barely scratch 150 bytes in size.

[madshi]

Thats just a myth, he never said that =p
He claims himself that he never said that. But I'm not convinced. But anyway, that's another topic... :D

[BetaBoy]

I am talking to Peter and Haali to getting them together to chat more about it.

[yesgrey]

I've just reviewed the spec posted on post #2 and took some notes about it. I don't know if this project is still alive, but here they are hoping they would still be helpful.

I'll setup a SourceForce project for it. It is certainly best that the spec is avaiable on a neutral place.

Before I put it on SourceForce, we should agree at least on a final name so that we don't have to rename the SourceForge project later on. Currently, the name is "Open Media Format".
I would stay with Open Media Format, its the best name yet.
If we keep the word "Open", we should at least rename it to Open Video Info. Open Media Format sounds like a container or codec. Video Info is more appropriate.
Did you setup the SourceForce project, or the name is still not clear?

My first note is exactly about the name. In the intro text (lines 1 to 42), sometimes you refer to it as "Open Video Info", and other times as "Open Media Format". Is that intended, or should it always be "Open Video Info"?

For the other notes I will indicate the line where the problem is, what the problem is, and then my suggestion.

Line 106:
// STEREOLAYOUT_RIGHT are >>>indetend<<< for content that contains only one view but
should be: intended.

Line 269:
// MFVideoChromaSubsampling >>>enumarations<<<.
should be: enumerations.

Line 312:
VideoPrimaries_BT709 = 0x02, // ITU-R BT.709, also used for >>sRGB and scRGB<<
should be: sRGB, scRGB and xvYCC

Lines 313 and 314:
VideoPrimaries_BT470_2_SysM = 0x03, // ITU-R >>BT.470-4<< System M (NTSC), original NTSC primaries
VideoPrimaries_BT470_2_SysBG = 0x04, // ITU-R >>BT.470-4<< System B,G (PAL)
should be: BT.470-6

Line 341:
VideoTransferFunction_22_709_Sym = 0x0B, // >>>Gamma 2.2 curve with a linear range in the low range (IEC 61966-2-4)<<
should be: Gamma 2.2 curve with a linear segment in the low range and origin symmetry (accepts negative values), used by xvYCC

Lines 345:
>>>VideoTransferFunction_22_BT1361 = 0x1E, // Gamma 2.2 curve with a linear range in the low range, used in BT1361 extended colour gamut system<<<
This transfer function is for BT1361 extended. The BT1361 is the same as BT.709, so I suggest a different name and a slightly different description:
VideoTransferFunction_22_1361_Neg = 0x1E, // Gamma 2.2 curve with a linear segment in the low range (accepts negative values), used in BT1361 extended colour gamut system
Note: if name this one as _Neg, maybe it would be preferable to also name the "VideoTransferFunction_22_709_Sym" as "_Neg" instead of "_Sym".

Also, since its value is 0x1E it should come before "VideoTransferFunction_26" whose value is 0x1F.

Lines 310 to 345:
Are used several different expressions: "used for", "used by" and "used in".
I think we should use always the same, the one which is more correct.

Lines 475 and 476:
VIDEOPROPS videoProps;
DWORD dwControlFlags;
In the VIDEOINFOHEADER2 struct dwControlFlags cames first, shouldn't it be the same here?

Lines 499:
typedef enum tagVIDEODEGRADATIONFLAGS {
I suggest another degradation flag:
VideoDegradation_TransferMatrixChanged = 0x00800000, // Image converted to use another transfer matrix

Lines 525:
} VIDEODEGRA>>>DEA<<<TIONFLAGS;
should be: DA

Lines 541:
UINT de>>>gre<<<dationFlags; // See VIDEODEGRA>>>DEA<<<TIONFLAGS enumeration for possible values
should be: "gra" and "DA"

Lines 555:
// The OPENVIDEOINFOHEADER is >>>similiar<<< to the VIDEOINFOHEADER2 structure but
should be: similar

[madshi]

Thank you, yesgrey - appreciate that!

@pwimmer and @BetaBoy, can we finalize this spec? Any more input from your side? @pwimmer, any comments on my suggestion from my post #51?

I'd love to have this spec available as v1.0, so that I could soon start to add support for it to my video renderer.

[pwimmer]

yesgreg, thank you very much for your excellent input!

Give me a few days and I'll update the files.

Fyi, the H.264 MVC decoder I've been working on during the last few weeks will simply use multiple pins for multiple views. I didn't implement the spec discussed on this thread yet.

[7000feet]

from the mvc programmers manual:

1 General Information
The JMVC (Joint Multiview Video Coding) software is
the reference software for the Multiview Video Coding
(MVC) project of the Joint Video Team (JVT) of the
ISO/IEC Moving Pictures Experts Group (MPEG) and the
ITU-T Video Coding Experts Group (VCEG). Since the MVC
project is still under development, the JMVC Software
as is also under development and changes frequently.
The JMVC software is written in C++ and is provided
as source code. Section 1.1 describes how the JMVC
software can be obtained via a CVS server. Information
about the structure of the CVS repository is presented
in section 1.2. Section 1.3 describes how the JMVC
software can be build on Win32 and Linux platforms, and
section 1.4 gives basic information about the
binaries that are contained in the JMVC software package.

1.1 Accessing the latest JMVC Software
In order to keep track of the changes in software
development and to always provide an up-to-date
version of the JMVC software, a CVS server for the
JMVC software has been set up at the Rheinisch-Westfälische
Technische Hochschule (RWTH) Aachen. The CVS server
can be accessed using WinCVS or any other CVS client.
The server is configured to allow read access only using
the parameters specified in Table 1. Write access to the
JMVC software server is restricted to the JMVC
software coordinators group.

Table 1: CVS access parameters

authentication: pserver
host address: garcon.ient.rwth-aachen.de
path: /cvs/jvt
user name: jvtuser
password: jvt.Amd.2
module name: jmvc

Example 1 shows how the JMVC software can be accessed
by using a command line CVS client.
Example 1: Accessing the JMVC software with a command
line CVS client.

cvs –d :pserver:jvtuser:jvt.Amd.2@garcon.ient.rwth-aachen.de:/cvs/jvt login
cvs –d :pserver:jvtuser@garcon.ient.rwth-aachen.de:/cvs/jvt checkout jmvc

[madshi]

@pwimmer,

any chance to get this finalized soon? I'd like to start implementing this in my video renderer soon.

Also please check out my suggestion in comment #51. I think it would be a useful addition.

[mariner]

Greetings pwimmer and madshi.

Came across this JVC 3D mvc (http://av.watch.impress.co.jp/video/avw/docs/424/208/ez0027.mp4) clip over here (http://av.watch.impress.co.jp/docs/series/zooma/20110202_424208.htm).

Will the new decoder/splitter be able to handle it?

Best regards.

[pwimmer]

I've updated the spec.

Thanks again yesgrey for your great and detailed suggestions, I've adopted all of them.

The spec also includes some "reserved" fields for future use. I didn't add the version field, because I believe it's more useful and error-proof that future fields should be defined in a way that a value of zero means "not used", while a value other than zero should be interpreted by a filter. The spec requests that the currently unused fields should be set to zero when writing.

[madshi]

Thanks, pwimmer. I'm pretty happy with the header now.

Could you please add me to the copyright? (Mathias Rauen, madshi (at) gmail (dot) com). Also it would be great if you could create a sourceforge project sooner or later, but both of these things are not urgent at all.

[yesgrey]

I've updated the spec.

Thanks again yesgrey for your great and detailed suggestions
You're welcome. :)

I've adopted all of them.
Not quite... you've missed two:
My first note is exactly about the name. In the intro text (lines 1 to 42), sometimes you refer to it as "Open Video Info", and other times as "Open Media Format". Is that intended, or should it always be "Open Video Info"?
About the name, any conclusion?

Lines 541:
UINT de>>>gre<<<dationFlags; // See VIDEODEGRA>>>DEA<<<TIONFLAGS enumeration for possible values
should be: "gra" and "DA"
You've missed the variable name, which is still degredationFlags instead of degradationFlags.
Now it's in line 543.

And I've just found one more:
Line 40:
// The open media format is discussed on the >>>follow<<< Doom9 thread and
should be: "following"

[pwimmer]

The name of the project is "Open Media Format", which comprises several definitions of new data types, many of them are named after existing types with the "Open" prefix. Examples: IOpenMediaSample extends Microsoft's IMediaSample, OPENVIDEOINFOHEADER supersedes VIDEOINFOHEADER2. Although there is a OPENVIDEOINFOHEADER structure, it is just one out of many new types. We have choosen "Open Media Format" as project name because it is a generic one. Parts of the specification like the IOpenMediaSample might be use for other types than video in the future.

And finally, we also have a SourceForge project:
http://openmediaformat.svn.sourceforge.net/viewvc/openmediaformat

[madshi]

Wonderful, thanks for all the work you've put in this, pwimmer, I appreciate it!

Could you add me as sourceforge committer/admin? I will not do any changes on my own, I'll run everything through you. But I think it would be good to have a backup, just in case something ever happens to you.

[pwimmer]

Could you add me as sourceforge committer/admin?

Done.

[pwimmer]

There's one think I'm worried about: The current specification is based on the ideas of just a few people. There are certainly many more features that should be added. Maybe we should keep the format more generic, not defining any fields for stereo or degredation at all, but just define space for a property/value table that would be fully generic? We would pre-define some properties (for the fields that we have in the spec right now), but everybody else could also define his own private properties.

The obvious advantage is that it would never be necessary to update the format structure. To add new features, only new properties must be defined. There are also disadvantages, however: Accessing fields is not as straightforward anymore, because you have to search the property table and the format block would not have a fixed size. Some helper function can simplify accessing properties, of course.

A table item could have the following form:


typedef struct TOpenMediaFormatItem {
GUID guidProperty;
union {
BOOL boolValue;
BYTE byteValue;
INT intValue;
UINT uintValue;
GUID guidValue;
...
}
} TOpenMediaFormatItem;

Instead of the union, a VARIANT could be used, however, it must not contain pointers because it would make it difficult to copy the format block.

[madshi]

Maybe we should keep the format more generic, not defining any fields for stereo or degredation at all, but just define space for a property/value table that would be fully generic? We would pre-define some properties (for the fields that we have in the spec right now), but everybody else could also define his own private properties.
I like the general idea of a fully flexible format. However, I'm afraid of making it too complicated. I think with the current OpenMediaFormat spec we've found a very simple structure which is very easy to understand and very straightforward to use. No pointer arithmetic necessary. No danger of buffer overruns. 99% compatible with VIDEOINFOHEADER2. All these factors will make it very easy for programmers to add support for OpenMediaFormat.

I think we should extend the current spec a bit to add your idea of fully flexible items. But I would really hate to remove any of the already defined fields. I think they're all important and should stay in the non-flexible part. How would you like this?

typedef struct tagOPENVIDEOINFOHEADER {
EXTRAVIDEOINFO extraVideoInfo; // A couple of added fields
FLEXVIDEOITEMS flexVideoItems[32]; // can be used to store anything
VIDEOINFOHEADER2EX videoInfoHeader2; // Compatible with VIDEOINFOHEADER2
} OPENVIDEOINFOHEADER;
This would keep the basic structure simple. One item could then look e.g. like this:
typedef struct tagFLEXVIDEOITEM {
GUID guidProperty;
union {
BOOL boolValue;
BYTE byteValue;
INT intValue;
UINT uintValue;
GUID guidValue;
WCHAR strValue[MAX_PATH];
BYTE binaryValue[512];
}
} FLEXVIDEOITEM;
We could later define some common "FLEXVIDEOITEM" properties, when needed. But devs could also use these flex fields to transport private data between their filters.

What do you think?

[madshi]

P.S: I would be ok with moving the stereo stuff to FLEXVIDEOITEMs, if you prefer it that way. But I would really like the other 4 fields in EXTRAVIDEOINFO to stay, because if we make them "flexible", too, most devs probably won't ever use them. Having them spelled out in EXTRAVIDEOINFO gives them higher priority/importance. Moving the stereo stuff to FLEXVIDEOITEMs might make sense because these fields are really only useful for stereo movies, but not for the majority of movies. The other flags in EXTRAVIDEOINFO are useful for every movie, though.

[yesgrey]

The name of the project is "Open Media Format", which comprises several definitions of new data types, many of them are named after existing types with the "Open" prefix. Examples: IOpenMediaSample extends Microsoft's IMediaSample, OPENVIDEOINFOHEADER supersedes VIDEOINFOHEADER2. Although there is a OPENVIDEOINFOHEADER structure, it is just one out of many new types. We have choosen "Open Media Format" as project name because it is a generic one. Parts of the specification like the IOpenMediaSample might be use for other types than video in the future.
It makes sense. I think it might be a good idea to put this explanation on the sourceforge site...

And finally, we also have a SourceForge project
Great. It would be a good idea to also put the file in the Files area, so it would be easier for people to get it. If you want to add me as commiter/Admin feel free to do it

Thanks for adding me to the copyright. My e-mail is: yesgrey (at) gmx (dot) com

[pwimmer]

P.S: I would be ok with moving the stereo stuff to FLEXVIDEOITEMs, if you prefer it that way. But I would really like the other 4 fields in EXTRAVIDEOINFO to stay, because if we make them "flexible", too, most devs probably won't ever use them. Having them spelled out in EXTRAVIDEOINFO gives them higher priority/importance. Moving the stereo stuff to FLEXVIDEOITEMs might make sense because these fields are really only useful for stereo movies, but not for the majority of movies. The other flags in EXTRAVIDEOINFO are useful for every movie, though.

I have no problem to move the stereo stuff to the FLEXVIDEOITEMs, but I would move everything else to FLEXVIDEOITEMs as well.

Easy access to a flag doesn't mean that all developers use it correctly. I'm afraid that most developers will set it to zero, or even worse, write some garbage to it.

I see two options: Moving nothing or all to FLEXVIDEOITEMs. In any case, I don't want any string or other large data in FLEXVIDEOITEMs. The GUID type should be the largest supported type, else it would bloat FLEXVIDEOITEMs too much.

[madshi]

I see two options: Moving nothing or all to FLEXVIDEOITEMs.
Then let's move nothing to FLEXVIDEOITEMs. That's my preferred solution, anyway.

In any case, I don't want any string or other large data in FLEXVIDEOITEMs. The GUID type should be the largest supported type, else it would bloat FLEXVIDEOITEMs too much.
In what way would it hurt to allow larger types? As far as I can see it, larger types would allow even greater extension flexibility, without making the structure any more complicated.

Are you so worried about consuming a few KBs of memory? If you're totally opposed to that, how about a compromise? E.g. I'd be happy enough with maybe 64 ANSI chars / 32 WIDE chars. That would increase one FLEXVIDEOITEM to a size of 64 byte. That's not so much. A GUID consumes 16 bytes, too. If memory consumption is so important to you, we could replace the guidProperty GUID with a simple dword. Would save 12 bytes... :p

[pwimmer]

I just see no need to make the data field that large. 16 bytes are fine to hold all numeric data types, including GUIDs.

The format block should only contain data that is important during format negotiations. Other data, including large ones like language, stream name or descriptive text shouldn't be in the format block but retrieved via an interface like IPropertyBag.

The property itself must be a GUID so that it is unique. If it is a (small) number, we would need an authority that assigns properties. I don't want to create a mess like the FOURCC system. Therefore, we use GUIDs.

[madshi]

The format block should only contain data that is important during format negotiations.
The format block already contains information which is not important during format negotiations, e.g. dwPictAspectRatioX/Y or parallaxX/Y.

Let me give you an example of where string properties would be useful: Let's suppose a new MPEG5 video standard is created, supporting different profiles, just like MPEG4 does. And let's suppose that decoders do not support all profiles, just as most h264 decoders today do not support all h264 profiles. Let's further suppose the profiles in MPEG5 are not specified by numbers, like in MPEG4, but instead by strings. In this situation it would be quite useful to be able to transport the profile name through a FLEXVIDEOITEM string property. Of course this is a constructed situation. But my point is: We do not know for sure whether string properties might be useful in the future or not. So not supporting string properties would unnecessarily limit flexibility.

You seem to like GUIDs a lot. You use them to differ between stereo modes and between flex properties. Some future video, audio or subtitle codecs might use strings instead of GUIDs. We can't know that. Actually, personally, I prefer strings to GUIDs, because with strings the name already tells you something. With GUIDs, the number doesn't tell you anything. You have to lookup the GUID first before you know what it means. E.g. if instead of "GUID layout" we would use "CHAR layout[32]", we could use the real layout names like "unknown", "monoscopic", "left", "right", instead of some cryptic GUIDs. Same flexibility, easier to debug. Because of this reason I personally prefer strings to GUIDs, and I think allowing GUIDs as FLEXVIDEOITEM data type, but not strings, would mean that we design the Open Media Format to your personal preferences (using GUIDs), instead of allowing the greatest possible flexibility.

Even if you don't agree with me, please tell me: What would it *hurt* to allow strings as a data type? Where's the big disadvantage? I don't see any worth mentioning. As I said before, it doesn't have to be MAX_PATH.

[Loomis]

Guids rawk. Strings blow.

With strings you have to worry about upper/lower case issues, with leading and trailing spaces, with locale issues, with string length and termination issues, and with name collisions.


We do not know for sure whether string properties might be useful in the future or not. So not supporting string properties would unnecessarily limit flexibility.That type of reasoning is absurd, and an insult to anyone skilled in critical thinking.


Some future video, audio or subtitle codecs might use strings instead of GUIDs. We can't know that.Fear meme.

In the future the Son of Man might return in the clouds with His myriad of angels, and execute judgment on all of us. And in that case this whole guid v. string issue will be moot.

[madshi]

With strings you have to worry about upper/lower case issues, with leading and trailing spaces, with locale issues, with string length and termination issues, and with name collisions.
These are all well understood "problems".

That type of reasoning is absurd, and an insult to anyone skilled in critical thinking.
??? Is that the style you're discussing in? How about discussing with real arguments?

Loomis, you also fail to answer a question I've asked pwimmer twice:

What would it *hurt* to allow strings as an additional optional data type? Where's the big disadvantage?

[madshi]

I've done some more thinking:

(1) We don't really know right now what purpose the "flex" items will be used for. The very purpose of these items is that we want them to be useful for virtually *any* (reasonable) purpose. So what data types might make sense? I think in most cases a single int or dword will do. But I could also imagine that points (2 ints), rects (4 ints) or palettes (16 dwords) might be used. I don't really see why GUIDs would be useful as a datatype, more than strings. But I think we should allow as many datatypes are could be eventually useful. With the current FLEX structure, storing rects or palettes would be very painful to transport, because e.g. for a rect you'd have to define 4 GUIDs and you'd have to consume 4 flex items.

(2) The current FLEX specification is not very debug friendly. MPC-HC outputs detailed pin information, but in order to output it in a nicely readable way, MPC-HC would need to know which kind of data is stored in the FLEX items.

Let's imagine a new video codec stores crop windows for different aspect ratios. It might make sense to store these in FLEX items. With the current structure MPC-HC would output:

FLEXITEM[0]:
property: {403eed06-a58f-4eb8-a2bd-a48ecfda4fa0}
value: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
FLEXITEM[1]:
property: {a0a3f800-194d-4b4d-a299-4919b2e24002}
value: 8C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
FLEXITEM[2]:
property: {47c57add-a3e3-4cc6-9f74-7f447d0c9a99}
value: 80 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00
FLEXITEM[3]:
property: {114957c5-8978-4317-8d1d-cf1d07ae25ab}
value: AC 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00
I think you'll agree that this is painful to read and hard to understand. You'd have to lookup the GUID to find out what they means. And if you've bad luck, the GUID might not be easily available, anywhere.

So I'd suggest to change the FLEX items to this:

typedef struct tagFLEXVIDEOITEM {
CHAR strProperty[32];
FLEXTYPEENUM enumValueType;
INT intValueCount;
union {
BOOL boolValue[16];
BYTE byteValue[64];
INT intValue[16];
UINT uintValue[16];
GUID guidValue[4];
CHAR ansiStrValue[64];
WCHAR wideStrValue[32];
}
} FLEXVIDEOITEM;
I understand that using a string as the key field has its problems, but in this case I prefer it to make debugging easier, because due to FLEX items being open for extension by anyone without any specification, the FLEX GUIDs would in most cases not be known to anyone. With my suggested changes, MPC-HC would output in its pin information:


FLEXITEM[0]:
property: scope_crop_rect
valueType: int
valueCount: 4
value: [0, 140, 1920, 940]
Isn't that *much* better than consuming 4 flex fields with 4 unknown GUIDs?

[nevcairiel]

Having dealt with Media Foundation a bit some time ago, i can't help but being reminded of IMFAttribute (http://msdn.microsoft.com/en-us/library/ms704598%28v=VS.85%29.aspx). MF got rid of the concept of static media types, and instead the filters exchange these attribute stores, which can contain an unlimited amount of data. (See here (http://msdn.microsoft.com/en-us/library/aa376629%28v=VS.85%29.aspx) for a list of attributes defined by MS)

Now, this uses GUIDs as keys, and i don't think thats too complicated.
The only problem to take care of is documenting, but anyone using the Open Media Type (was that what we're calling it now?), should know where to find that documentation, and everyone should be asked to submit the GUIDs for any custom attributes to you, so you can add them to the documentation. (This would equally be required for strings as keys, even if i can read the identifier, i can't be sure that i'm interpreting it 100% correct without documentation)

Going further, mostly focusing on madshi's suggested structure here:

An extra field for "type" is just overhead for debugging, it does not serve a real purpose for the filters implementing it, they will know to check for flex item with id "foobar", and expect it to have type DWORD. Same mostly goes for the "count" field, in most cases you'll know to expect exactly 4 values, and if the corner case would come up where there is a dynamic number of items, you could just have the first item be the count variable.

I would not add those just to get better debug output. The tool should be improved if its lacking, not the format being forced to compensate.

Finally, alot of the types in the union are redundant.

- BOOL is an alias for INT (at least in C/C++)
- UINT and INT are both 32-bit types and can be converted freely between each other.
- CHAR and BYTE both are 8-bit types (again only sign being different)
- WCHAR is meh. I would vote for only allowing CHAR, if someone needs to transport locale-sensitive data, convert it into a Multi-Byte UTF-8 String and put it into CHAR/BYTE

After slimming that down, i would keep:
- INT, for any form of number or bitmask (including BOOL) -- possibly make this a DWORD instead, as its commonly used as a number type in DShow
- BYTE, for binary data, including strings.
- GUID

Additionally, i could potentially see a floating point type being added as well, not sure if that would ever be used, however.

I do however like having the arrays of values in the flex type.

[madshi]

Thanks for your input, nevcairiel, I appreciate that. Using only ansi chars is ok with me (thanks to UTF8). I personally would like to see "type" and "count" items even though they're mainly useful only for debugging. But I can live without them if there's a majority for that. I like the idea to add doubles. Any opinion on max ansi string length? Maybe 64 chars?

IMFAttribute does indeed look similar in concept. Interesting is that IMFAttribute does support strings and also byte arrays and doubles. It also supports IUnknown, which I find bad because anything pointer like is asking for trouble, IMHO.

Any further comments, anybody? Would be nice to have this finally completed.

[nevcairiel]

A GUID is 16bytes, isn't it? If you allow 4 GUIDs in the union, having 64 bytes for the char/byte field makes sense. Should try to keep the arrays equal in size, targeting for 64 bytes is a good goal.

[nevcairiel]

I've been thinking about implementing the OPENVIDEOINFOHEADER in my video decoder (not 3D, just 2D, but still) to export transfer matrix and video primaries to the renderer, if it wants that information.

Is anyone still working on implementing the latest suggestions? Is anyone else using this? Is it dead? :o

[madshi]

I'd like to use it, but we need to officially finalize the spec first. @pwimmer, are you there?

[pwimmer]

I've updated the specification for H.264 MVC media types in the first posting of this thread.

[madshi]

You *are* still alive it seems. What is your comment on the last couple of posts regarding the new flex items?

About those FourCC media types listed on the first page: Why do you use the "H264" FourCC (AnnexB format)? Why not using "AVC1" (NAL format), which is used by most splitters these days?

[pwimmer]

Is anyone still working on implementing the latest suggestions? Is anyone else using this? Is it dead? :o

Nobody is currently working on the spec for OPENVIDEOINFOHEADER and it is not used so far by any DirectShow filter.

The project is not dead, but for the H.264 MVC decoder it turned out to be the better choice to have two separate output pins, thus there was no need to finalize the spec quickly.

The H.264 MVC FOURCCs (AMVC, EMVC) are already used by MPC HC (patches required) and will be used by the next releases of the Haali splitter and the CoreAVC decoder.

[pwimmer]

About those FourCC media types listed on the first page: Why do you use the "H264" FourCC (AnnexB format)? Why not using "AVC1" (NAL format), which is used by most splitters these days?

My mistake, I've corrected it. The implementations used AVC1 from the beginning.

[madshi]

Nobody is currently working on the spec for OPENVIDEOINFOHEADER and it is not used so far by any DirectShow filter.

The project is not dead, but for the H.264 MVC decoder it turned out to be the better choice to have two separate output pins, thus there was no need to finalize the spec quickly.
Are you still interested in the OPENVIDEOINFOHEADER spec? If not, I can take over, because I am still interested. I'd really like to finally finalize the spec, so I can add support for it to my video renderer.

[pwimmer]

Are you still interested in the OPENVIDEOINFOHEADER spec? If not, I can take over, because I am still interested. I'd really like to finally finalize the spec, so I can add support for it to my video renderer.

I am interested.

Let me summarize the outstanding decisions based on the recent suggestions. I've also added my optinion, let's see what others say...

1) Should the format be extensible?

Yes.

2) Should we use a GUID to identify extensions?

Yes.

3) What should be the max. data size per extension?

16 bytes (the size of a GUID). I don't see any reason to allow strings in the format block.

4) Should there be a type flag?

No, it is redundant, the GUID implicitely specifies the type.

5) Should there be a fixed number of extensions?

I'm not sure. A fixed size format block would be easier to deal with, a variable size block would be more flexible.

6) Should there be a non-flexible part?

I would put either everything or nothing in the flexible part.

[pwimmer]

Btw, we have some time to finalize the spec because I don't want to make any final decision before the first beta of Windows 8 has been released and we see what Microsoft has added to DirectShow.

[nevcairiel]

I doubt Microsoft will still extend DirectShow, their money is on Media Foundation.
I'm secretly hoping they kill DirectShow, even when it means nothing will work anymore, it would for the first time be a clean cut. :p

[madshi]

Ok, let me sum up the opinions of you, nevcairiel and me, let me also extend the list of decisions a bit:

1) Should the format be extensible?

[pwimmer] Yes.
[nevcairiel] Yes.
[madshi] Yes.
Yes.
[clsid] Yes.
[Matt] Yes.

[B]2) What data type should be used to identify extensions?

[pwimmer] GUID.
[nevcairiel] GUID.
[madshi] String (for better debugging).
GUID, if registered in the OS, String otherwise.
[Matt] GUID or String, no preference noted.

[B]3) What should be the max. data size per extension?

[pwimmer] 16.
[nevcairiel] 64.
[madshi] 64.
[Matt] "Worrying about size is silly. So allocate more space than you think you'll need."

4) What data types should be allowed in the extension?

[pwimmer] BOOL, BYTE, INT, UINT, GUID.
[nevcairiel] DWORD (= int), BYTE (= char), GUID, float or double.
[madshi] BOOL, BYTE, CHAR, INT, DWORD, GUID, double.
[Matt] int64 instead of int/byte, double instead of float, no comment on GUID and "char" for strings

5) Should the extension use a single item (e.g. "BOOL boolValue"), or should it use arrays (e.g. "BOOL boolValue[64]")?

[pwimmer] array (?).
[nevcairiel] array.
[madshi] array.
[Matt] array.

6) Should there be a type flag and an item count?

[pwimmer] No.
[nevcairiel] No.
[madshi] Yes. For better debugging.
[Matt] No.

7) Should there be a fixed number of extensions?

[pwimmer] No.
[nevcairiel] No.
[madshi] Yes.
Yes.
[clsid] Yes.
[Matt] Yes, if there's an advantage to it. No otherwise.

[B]8) Should there be a non-flexible part?

[pwimmer] ?
[nevcairiel] ?
[madshi] ?

Can you clarify what exactly you mean with question 8)? I'd like the fixed parts to stay as they are and only use the flexible parts for future extensions.

[nevcairiel]

If you use GUID extensions, they are by definition variable, having a fixed number in there would be kinda odd, so put me down for no on those.

I agree with madshi on 8. There can easily be fixed parts for the very common types, like outlined in the definition already, and have flexible GUID identified parts for future extensions.

[pwimmer]

I agree with madshi on 8. There can easily be fixed parts for the very common types, like outlined in the definition already, and have flexible GUID identified parts for future extensions.

Basically I agree, but which are the very common fields? It would be an endless discussion, because each developer has different priorities.

If the structure has a variable size, I don't see any need to have fixed parts anymore. Helper functions to access the parameters can make it nearly as easy to access the variable part as the fixed part.

[madshi]

If you use GUID extensions, they are by definition variable, having a fixed number in there would be kinda odd, so put me down for no on those.
Oh, I just see that I mis-stated my opinion on that one. I think having a fixed number of "flex items" would be preferable, because it would allow very easy use of the whole structure. I could then simply use:

- openInfoHeader->extraVideoInfo.someField
- openInfoHeader->flexItems[i].someField
- openInfoHeader->videoInfoHeader2.someField;

If we allow the numbers of flex items to be dynamic, the videoInfoHeader2 structure will not have a fixed position in the overall structure, anymore, making adressing more complicated. Of course we could use helper functions, but it would still be more complicated. Especially for non-C++ users. IMHO e.g. 16 or maybe 32 flex items should be plenty for future extensibility.

Basically I agree, but which are the very common fields? It would be an endless discussion, because each developer has different priorities.
We had already pretty much agreed on the "final" structure with all fields in it etc. Only then you had the idea to add flexible items. So now my opinion is that we should keep the original "final" structure as it is, and just add flexible items for future expansion capability. There'd be no need for discussion, we'd just leave the original structure as it is with no changes. So the "common" fields are already defined.

[Blight]

Madshi asked me to give my feedback on the process, so here goes:

As a programmer, I mostly care about 'easy to code, flexible for the future'.
So with that view in mind, I recommend calling the structure "OPENVIDEOINFOHEADER1" (note the '1').

I would prefer a fixed number of extensions, but keep the structure simple enough so if/when "OPENVIDEOINFOHEADER2" is introduced, you can easily add extensions, requiring minimal code changes.

With regards to GUID vs. Strings. Are you going to register the GUID with the system, or are you just using them internally? If you plan to register with the system and possibly allow for custom values (saved in the registry), GUID may be the way to go.
If you're only planning to use the GUID internally, it may actually be clearer to use strings (possibly with some pre-defined structure).

[pwimmer]

The variable size of VIDEOINFOHEADER2 requires that it is at the end of our structure, making it inconventient to add a variable size property block before.

However, VIDEOINFOHEADER2 usually has a fixed size unless there is a color palette. I doubt anybody is using a color palette these days, so I suggest to assume VIDEOINFOHEADER2 has a fixed size. Under this assumtion, we could define the following:



typedef struct tagOPENMEDIAPROPERTY {
GUID guidProperty;
union {
BOOL boolValue[4];
BYTE byteValue[16];
INT intValue[4];
UINT uintValue[4];
GUID guidValue;
LONGLONG longlongValue[2];
ULONGLONG ulonglongValue[2];
FLOAT floatValue[4];
DOUBLE doubleValue[2];
};
} OPENMEDIAPROPERTY;

typedef struct tagOPENVIDEOINFOHEADER {
VIDEOINFOHEADER2EX videoInfoHeader2; // Compatible with VIDEOINFOHEADER2 (no color palette allowed)
EXTRAVIDEOINFO extraVideoInfo; // All extra stuff not part of VIDEOINFOHEADER2
UINT propertyCount;
OPENMEDIAPROPERTY properties[];
} OPENVIDEOINFOHEADER;

[madshi]

In the last couple of days/weeks I've added support to my video renderer for some compressed video. For compressed video, the bitstream sequence headers are usually stored in the media type information, following VIDEOINFOHEADER2, or more commonly, following MPEG2VIDEOINFO. See here:

http://msdn.microsoft.com/en-us/library/dd390707%28v=vs.85%29.aspx

MPEG2VIDEOINFO already defines that VIDEOINFOHEADER2 can not have a color palette. Now the big question is: Do we want Open Media Format to be used for transporting compressed video bitstream, too? If so, we may need to alternatively support VIDEOINFOHEADER2 or MPEG2VIDEOINFO. Furthermore the sequence headers will then follow. So that again would require VIDEOINFOHEADER2 (or MPEG2VIDEOINFO) to be the last element in the structure.

Thoughts?

[nevcairiel]

I'm really not a fan of that pure-dynamic OPENVIDEOINFOHEADER.

Anyhow, if it should be used for compressed data, it does need some sort of space for a arbitrary length sequence header. This doesn't necessarily have to be right after VIH2/MP2VI - although it would make things easier - it could be like in MP2VI, just at the end of the wrapping structure. The more complicated question is how that can be easily combined with flex-elements.

[madshi]

I'm still voting for a fixed number of flex items (maybe 16 or 32) in the middle of the structure, to be zeroed out if not used. FWIW, VMR and EVR always blow up any media type to more than 1kb. So having 16 * 64 = 1024 bytes reserved for the flex items should really be no problem at all.

[clsid]

I didn't read the whole discussion and I am far from an expert on this, so feel free to ignore me. Here are some of my thoughts.

The base of the structure as defined in the spec should have a fixed size as that is easier to work with. As I understand the 'extensions' are there to add additional non-standard features in the future. Limiting extensions to a fixed number shouldn't be much of a problem. If the amount becomes insufficient, which would likely only occur in very specific apps, or maybe never at all, then you could just group together some uncommon extensions in one new larger extension. The base should contain a list of extensions, plus the data size of each extension, and maybe if needed also the offset from beginning of the structure. Then an app can easily find the data belonging to each extension. Use the ones it understand, and skip the ones it doesn't support. Seems natural to use a GUI as a fixed size unique identifier for extensions.

[madshi]

Thanks for your comment, clsid.

[JanWillem32]

I took some time to read trough this topic carefully. I looked up several concepts, implementations and a lot more stuff.
I do still have some questions.
The only real things I'm working on at the moment are a software subtitle texture renderer, two video renderers on top of a vanilla EVR mixer, one video renderer on top of a vanilla VMR-9 mixer and some accessory items for those. I'm reviewing this from the perspective that the renderers I'm working on need a lot of work to ever produce quality output with decent efficiency.
Because the mixer parts I'm using currently are pretty much vanilla and not very suited to the renderers attached to them, I don't mind replacing them at all. The thing is, DirectShow is old, Meda Foundation is new. I'm unfortunately the only one who is actively writing big code changes for the renderers I'm working on right now. From my perspective, I can just as easily use vanilla Meda Foundation code for the mixers as the pretty much vanilla DirectShow code the renderers are using right now, and work from that point to improve video mixer parts.
What good would a new iteration of a DirectShow interface bring like this? What Media Foundation parts don't cover the items proposed in this new format?

I also saw the references to support DCI media packages. I know that the DCI format is pretty much set in stone and doesn't allow much variants, so the header for the DCI packages doesn't store much data. (The formats are luckily also very incompatible with any consumer video format.) Just to be sure, are there any parts of the DCI media package header format, or some hidden data that's not in that header, that could matter for this implementation?

[nevcairiel]

MediaFoundation has by design a very flexible layout (storing key-value pairs with GUIDs as keys and a wide variety of value-types), so this whole design is not required for MediaFoundation, its only to get DirectShow up-to-par for the coming time, because its not going away any time soon.

PS:
EVR is a Media Foundation tech, thats why internally you can/have to use alot of the MF things - however externally, it still has to look like DirectShow if you want to use it in a DirectShow player - so you cannot use the Media Foundation Media-Type concept, if thats what you were kind of implying.

[Matt Ashland]

madshi asked me to weigh in on this discussion. The smartest guys in the room have already been here, but I'll post my thoughts anyway :P

First, let me say that J. River will be happy to support this format specification in Media Center however it gets implemented.

As for the technicals, my preference is something fully flexible. If it were done properly, it could be used to solve the problem of video tagging. I realize storing actors, box art, etc. is outside the scope of this conversation, but it's a gaping hole in the video world compared to audio. If there were a standard block of data to describe everything about a video, including the metadata, it would be wonderful.

With that said, the current discussion is focused more tightly on the video format information only. For this, a fixed structure with some number of FLEXVIDEOITEM objects seems like a reasonable compromise.

[madshi]

Thanks for your input, Matt, I appreciate it!

Would you be willing to post your opinion about the questions in the following post?

http://forum.doom9.org/showpost.php?p=1515930&postcount=93

About video tagging: Maybe it would make sense for nevcairiel and you to discuss ways for LAV Splitter to export whatever tagging the various container formats support? LAV Splitter could eventually export a new interface which you could use to retrieve that kind of information. @nevcairiel, what do you think? To be honest, I don't really know, though, how much video tagging any of the current containers support. Probably MKV has the broadest support for that kind of thing? Not sure. In any case, transporting things like cover art etc via the media type information would go too far, IMHO. For these kind of things I'd really prefer having the splitter export an interface.

[nevcairiel]

MKV and MP4 can contain numerous tags, MP4s do sometimes, with MKVs its pretty rare. An interface to export this metadata would surely be possible, ffmpeg supports reading it in a multitude of containers and maps it into a set of common metadata keys.

[pwimmer]

Do we want Open Media Format to be used for transporting compressed video bitstream, too? If so, we may need to alternatively support VIDEOINFOHEADER2 or MPEG2VIDEOINFO. Furthermore the sequence headers will then follow. So that again would require VIDEOINFOHEADER2 (or MPEG2VIDEOINFO) to be the last element in the structure.
Thoughts?

I don't think it is necessary for compressed formats.

[Matt Ashland]

@Specific questions about video header

1) Should the format be extensible?:
Yes


2) Should we use a GUID to identify extensions?:
GUIDs are good because they're a fixed size and they never collide. But strings are easier to understand (for a user and programmer). If you decide to use strings instead, please specify a naming style because key naming is a mess in some tag formats like Ogg tags.


3) What should be the max. data size per extension?
Worrying about size is silly. A modern movie is 30 GB. So allocate more space than you think you'll need. I also like the idea of a flexible sized format block so extensions could be any size.


4) What data types should be allowed in the extension?
Don't bother with low precision types. If you have double, you don't need float. And if you have int64, you don't need int32, int16, uint16, byte, boolean, etc.


5) Should the extension use a single item (e.g. "BOOL boolValue"), or should it use arrays (e.g. "BOOL boolValue[64]")?
It's harmless to use arrays, and might be useful for types like a rectangle. But it might be good form to use binary data (with a structure define) if you're specifying an array or other complicated structure.


6) Should there be a type flag and an item count?
I wouldn't add the type flag. The key (GUID or string) is sufficient. Duplication just creates more ways for the person writing the format block to make a mistake.


7) Should there be a fixed number of extensions?
Is there some clear advantage to a fixed size header? If not, just put a stack of extensions at the end of the format block with a count or byte size in the header.


8) Should there be a non-flexible part?
Yes. You could make the non-flexible part just another extension, and require that extension to be present. But I think this may be getting too smart, especially if you're requiring fixed size extensions which limits extensions anyway.


@Tagging

I'd like to see a simple tag format that could be used in a variety of containers. Tag format folks often get too smart for their own good (XMP, ID3v2, etc.) when a simple string-to-string map (either in binary or as XML) is sufficient. You're probably right that it's outside the scope of this conversation. However, we do have the right cast of characters here to make something really useful for users. Currently J. River has to use sidecar XML files, which is gross. If anyone here ever wants to tackle this topic, J. River would be glad to help where possible.

[pwimmer]

Thinking over the the pros and cons of the flexible part, my preference would be to have only VIH2 and a flexible part, nothing else.

Let me explain why: The current draft contains a lots of fields for stereoscopic media, some of them would rarely be used. I don't want the structure to be bloated with such stuff. After all, many people have no interest in the stereo stuff, the same applies for the degradation flags. I don't think we should design the structure for the few of us participating in this discussion, but rather keep the Open Media Format more generic and generally applicable. To avoid any future limitations, we should not limit the number of flexible items. I know that an unlimited number of flex-items (or properties as I call them) is more difficult from a developers point of view, but it is the only way to keep the format truly extensible.

The OpenMediaFormat.h header should just contain the general definitions, while all the application-specifc stuff should be moved to application-specific headers, e.g. OpenMediaFormatStereo.h and OpenMediaFormatDegradation.h. New application-specific headers can be added at any time.

An OpenMediaFormatHelpers.c file can contain helper functions to simplify accessing the properties.

OpenMediaFormat.h would basically be reduced to the following definitions. Imho, small is beautiful.



typedef struct tagOPENMEDIAPROPERTY {
GUID guidProperty;
union {
BOOL boolValue[4];
BYTE byteValue[16];
INT intValue[4];
UINT uintValue[4];
GUID guidValue;
LONGLONG longlongValue[2];
ULONGLONG ulonglongValue[2];
FLOAT floatValue[4];
DOUBLE doubleValue[2];
};
} OPENMEDIAPROPERTY;

typedef struct tagOPENVIDEOINFOHEADER {
VIDEOINFOHEADER2EX videoInfoHeader2; // Compatible with VIDEOINFOHEADER2 (no color palette allowed)
UINT propertyCount;
OPENMEDIAPROPERTY properties[];
} OPENVIDEOINFOHEADER;

[BetaBoy]

Matt Ashland... Thank you for the support.

All.... As things settle down with the spec over the next few weeks, we are gonna gonna have it listed on www.OpenMediaFormat.com. I'll work with Peter, Madshi and the rest of you, but the thoughts are to design the site like we did with Matroska.org to make it simple to follow the specs and any proposals. We can leave the discussions here on D9, as this seems to work best for everyone.

[pwimmer]

Agreed, Doom9 is a good place for the discussion. Lets continue the discussion for a few weeks, and lets see if some proposals are widely accepted.

As I mentioned before, we are not in a hurry because we should not finalize the spec before Microsoft has released the Windows 8 SDK Beta. Even if they do not add features to DirectShow anymore, they will improve Media Foundation, and I want to keep as much of the Open Media Format compatible with Media Foundation definitions as possible. I just want to see what Microsoft does before deciding on own extensions for DirectShow.

[madshi]

@Matt, thanks again!

@BetaBoy, it's nice that you'll support this, too, even with its own homepage! :)

@pwimmer, there's still the problem with bitstream video headers. We will probably even have to support MPEG2VIDEOINFO instead of VIDEOINFOHEADER2 for bitstream connections. See the definition of MPEG2VIDEOINFO to see how the video bitstream headers are stored. This is used for all MPEG2, VC-1 and h264 connections.

[nevcairiel]

I kind of agree that its not necessarily required for compressed bitstreams, they have all the extra info in their sequence data. That would also solve the problem with the MPEG2VIDEOINFO.

[madshi]

I kind of agree that its not necessarily required for compressed bitstreams, they have all the extra info in their sequence data. That would also solve the problem with the MPEG2VIDEOINFO.
I'm not sure I understand what you mean. What is "it" in the "its not necessarily required"?

[nevcairiel]

The OPENVIDEOINFOHEADER of course.
I thought that was kind of clear, seeing how this thread is about it and all. :d

[madshi]

Oh, I somehow missed pwimmer's post you were refering to.

So, one question is: "Is OPENVIDEOINFOHEADER necessary for compressed formats"? Maybe not. But I think the better question is: "Do we want to limit OPENVIDEOINFOHEADER so that it doesn't work (well) for bitstream connections"? I'm not sure if that is a good idea. Isn't that exactly the opposite of the whole "flexible" idea? One purpose of the "flex" items could be to transport additional information that is stored in the container up the filter chain. If OPENVIDEOINFOHEADER is not to be used for compressed formats, then the chain is broken, and OPENVIDEOINFOHEADER cannot be used for that purpose, anymore.

E.g. what happens if the stereo "separation" information is stored in the container? How can the splitter forward this information to the decoder/renderer, if OPENVIDEOINFOHEADER can't be used for compressed formats?

[nevcairiel]

Could just go with VIH2, and if someone needs extra special information thats usually stored in MP2VI, store it in a flex item.
Allowing VIH2 and MP2VI in the same structure is really overcomplicating things, imho.

[madshi]

The length of the sequence headers could be larger than the max size we've defined for a flex item, though.

Ok, so how about going one step further, making the flex items totally flexible, not defining any standard type at all, like this:

typedef struct tagOPENMEDIAPROPERTY
{
GUID identifier;
int offset;
int size;
} OPENMEDIAPROPERTY;

typedef struct tagOPENVIDEOINFOHEADER
{
VIDEOINFOHEADER2EX videoInfoHeader2;
int propertyCount;
OPENMEDIAPROPERTY properties[];
} OPENVIDEOINFOHEADER;
Every flex item would then have its own private type.

Of course this flexibility would come at the cost of easy of use.

[pwimmer]

Oh, I somehow missed pwimmer's post you were refering to.

So, one question is: "Is OPENVIDEOINFOHEADER necessary for compressed formats"? Maybe not. But I think the better question is: "Do we want to limit OPENVIDEOINFOHEADER so that it doesn't work (well) for bitstream connections"? I'm not sure if that is a good idea. Isn't that exactly the opposite of the whole "flexible" idea? One purpose of the "flex" items could be to transport additional information that is stored in the container up the filter chain. If OPENVIDEOINFOHEADER is not to be used for compressed formats, then the chain is broken, and OPENVIDEOINFOHEADER cannot be used for that purpose, anymore.

E.g. what happens if the stereo "separation" information is stored in the container? How can the splitter forward this information to the decoder/renderer, if OPENVIDEOINFOHEADER can't be used for compressed formats?

I would suggest to define OPENMPEG2VIDEOINFO for that purpose if it turns out to be necessary. We might also have to define OPENWAVEFORMAT to extend WAVEFORMATEX if somebody has a need for it. In general, we will define new structures and interfaces for all stuff that is missing in DirectShow. We just start with the OPENVIDEOINFOHEADER because there is definitely a need for it.

[pwimmer]

I've uploaded a new proposal for the fully flexible format without any fixed fields except VIDEOINFOHEADER2.

OpenMediaFormat.zip on 3dtv.at server (http://www.3dtv.at/Downloads/OpenMediaFormat.zip)

The stereo fields are now stored in separate properties. For the degradation information a single property turned out to be sufficient.

To simplify access to the property values I've added further entries to the union of OPENMEDIAPROPERTY, this also elimites any doubt about the type for each property.

[madshi]

I would suggest to define OPENMPEG2VIDEOINFO for that purpose if it turns out to be necessary.
That wouldn't solve the problem, though. The key thing with MPEG2VIDEOINFO is that it contains *dynamically* sized bitstream sequence headers. So we can't put MPEG2VIDEOINFO at the beginning of an OPENMPEG2VIDEOINFO structure. I'd also prefer if we could get along with only one structure for all video related purposes. My suggestion would be to:

(1) Either always use MPEG2VIDEOINFO instead of VIDEOINFOHEADER2. The additional MPEG2VIDEOINFO fields would then simply be set to zero for uncompressed video connections. For this to work, MPEG2VIDEOINFO would probably have to be at the end of the OPENVIDEOINFOHEADER structure.

(2) Or stick to VIDEOINFOHEADER2 and store the additional fields and the sequence headers of MPEG2VIDEOINFO into a flex item. This would only be possible, though, if we don't limit the max size of a flex item, because there's no law that specifies how long the sequence headers may get.

We might also have to define OPENWAVEFORMAT to extend WAVEFORMATEX if somebody has a need for it.
You mean defining a separate structure for audio? Ok, we could do that. I don't have enough knowledge to know what is needed for that, though.

I've uploaded a new proposal for the fully flexible format without any fixed fields except VIDEOINFOHEADER2.

OpenMediaFormat.zip on 3dtv.at server (http://www.3dtv.at/Downloads/OpenMediaFormat.zip)

The stereo fields are now stored in separate properties. For the degradation information a single property turned out to be sufficient.

To simplify access to the property values I've added further entries to the union of OPENMEDIAPROPERTY, this also elimites any doubt about the type for each property.
First of all, I don't really like having a dozen of header files for just one format. Let's not make it more complicated than necessary. I'd prefer having the main structure defined in one header file. Ok, definitions for flex items could go into separate header files, though.

If I counted right, "sizeof(STEREOVALUES) == 76". You personally wanted the max size of a flex item to be 16. Now the size of a flex item is defined by sizeof(STEREOVALUES)? That doesn't make much sense to me.

Personally, I would like to see one of the following 2 solutions:

(1) Either we define the exact size of each flex item, and also the exact number of flex items. If we do that, we can still put MPEG2VIDEOINFO at the end of the structure.

(2) Or we make the flex items *totally* flexible, allowing each flex item to have any size it needs (without any limitations). In that case we would need to store an array of flex item offsets and sizes into the OPENVIDEOINFOHEADER. Each flex item would then have its own private structure.

[pwimmer]

If I counted right, "sizeof(STEREOVALUES) == 76". You personally wanted the max size of a flex item to be 16. Now the size of a flex item is defined by sizeof(STEREOVALUES)? That doesn't make much sense to me.

STEREOVALUES has a size of 16 bytes, it is a union, not a struct. STEREOVALUES serves the only purpose to simplify access to OPENMEDIAPROPERTY and to clarify the type for each property.

For example, instead of stereo.numViews you could access intValue[0]. It is an alias for the same memory location that makes the source code more readable. It helps to solve some of the drawbacks of the flexible format. I'll also write some helper functions that allow code like this:

OpenMediaFormatReadProperty(pOpenVideoInfo, OPENMEDIAPROPERTY_NUMVIEWS)->stereo.numViews

[madshi]

Oh, missed the "union", sorry. What about the other parts of my post?

[pwimmer]

(1) Either we define the exact size of each flex item, and also the exact number of flex items. If we do that, we can still put MPEG2VIDEOINFO at the end of the structure.

(2) Or we make the flex items *totally* flexible, allowing each flex item to have any size it needs (without any limitations). In that case we would need to store an array of flex item offsets and sizes into the OPENVIDEOINFOHEADER. Each flex item would then have its own private structure.

Making the flex items totally flexible, thus allowing a variable size for the flex items, makes it nearly impossible to calculate the required size of the structure. Imho this is not possible if we want to keep the Open Media Format useable.

VIDEOINFOHEADER2 and MPEG2VIDEOINFO could be located at the beginning or the end of the structure. Both is possible.

[madshi]

Making the flex items totally flexible, thus allowing a variable size for the flex items, makes it nearly impossible to calculate the required size of the structure. Imho this is not possible if we want to keep the Open Media Format useable.
IMHO we should decide whether we want to use a structure which requires pointer arithmetic or not. If we do, we will need helper functions to improve usability. Once we go dynamic, using pointer arithmetic and helper functions, we can also make the size of the flex items totally dynamic. We could simply add another helper function to calculate the requires size of the structure. IMHO requiring pointer arithmetic, using helper functions, but still sticking to a fixed size for flex items is neither here nor there. I'd vote for either using a totally static structure (fixed number & size of flex items), or a totally dynamic structure (dynamic number & size of flex items).

VIDEOINFOHEADER2 and MPEG2VIDEOINFO could be located at the beginning or the end of the structure. Both is possible.
FWIW, if you put MPEG2VIDEOINFO at the beginning of the structure, the position of the flex items is not fixed, anymore, because the size of MPEG2VIDEOINFO is dynamic.

[pwimmer]

IMHO we should decide whether we want to use a structure which requires pointer arithmetic or not. If we do, we will need helper functions to improve usability. Once we go dynamic, using pointer arithmetic and helper functions, we can also make the size of the flex items totally dynamic. We could simply add another helper function to calculate the requires size of the structure. IMHO requiring pointer arithmetic, using helper functions, but still sticking to a fixed size for flex items is neither here nor there. I'd vote for either using a totally static structure (fixed number & size of flex items), or a totally dynamic structure (dynamic number & size of flex items).

I want to keep it user friendly and allow a command sequence like this one:

OPENVIDEOINFOHEADER *pvi = (OPENVIDEOINFOHEADER *) pMediaType->AllocFormatBuffer(OpenMediaFormatGetVideoInfoHeaderSize(numProperties));

...

OPENMEDIAPROPERTY property;
property.guidProperty = OPENMEDIAPROPERTY_STEREOLAYOUT;
property.stereo.layout = STEREOLAYOUT_SIDEBYSIDELEFTFIRST;

OpenMediaFormatAddProperty(pvi, &property);

property.guidProperty = OPENMEDIAPROPERTY_SEPARATIONX;
property.stereo.separationX = 20;

OpenMediaFormatAddProperty(pvi, &property);

I don't see any need for large properties. The OPENVIDEOINFOHEADER is used for format negotiation and it is unlikely that strings will ever be needed.

Strings like artist, copyright, chapter names and other metadata are not part of the format but should be read via other filter interfaces.

FWIW, if you put MPEG2VIDEOINFO at the beginning of the structure, the position of the flex items is not fixed, anymore, because the size of MPEG2VIDEOINFO is dynamic.

So we put it at the end of the structure. Question is, should we put VIDEOINFOHEADER2 also at the end? This prevents typecasts from OPENVIDEOINFOHEADER to VIDEOINFOHEADER2 , but a simply helper function could solve the problem.

[madshi]

I want to keep it user friendly [...]
Well, then let's return to having a fixed flex item size *and* a fixed number of flex items. That would allow us to have a fully static structure with no dynamic elements (other than MPEG2VIDEOINFO). Advantage: No pointer arithmetic needed at all, no helper functions needed at all.

I don't see any need for large properties. The OPENVIDEOINFOHEADER is used for format negotiation and it is unlikely that strings will ever be needed.

Strings like artist, copyright, chapter names and other metadata are not part of the format but should be read via other filter interfaces.
Well, we have 2 choices: Either we go dynamic and allow anything that could ever be used. Or we go in the other direction and put a priority on ease of use. In the latter case I don't see any need for more than max 16 flex items. Let's make it 32, or even 64, and we should be very safe. No need to allow a dynamic number of items. Going static would noticeably improve ease of use IMHO.

So we put it at the end of the structure. Question is, should we put VIDEOINFOHEADER2 also at the end? This prevents typecasts from OPENVIDEOINFOHEADER to VIDEOINFOHEADER2 , but a simply helper function could solve the problem.
As luck has it, MPEG2VIDEOINFO starts with a VIDEOINFOHEADER2 structure. So IMHO we should simply always use MPEG2VIDEOINFO. Or we could put M2VI and VIH2 into a union at the end of the OPENVIDEOINFOHEADER structure.

[pwimmer]

I neither want the totally fixed nor the all-flexible extreme. I prefer a balanced approach, a compromise between flexibility and ease of use, based on the actual needs.

A fixed number of 16 properties doesn't simplify the use of the structure, you still have to loop thru the properties to find a free slot or a certain property. You would only eliminate the OpenMediaFormatGetVideoInfoHeaderSize helper function, everything else would be as complicated as with a variable number of properties. As a result, a variable number of properties is the better trade-off.

It is quite likely that more that 16 properties are defined in the future. On the other hand, nobody has suggested a property yet that requires more than 16 byte of data.

[madshi]

I neither want the totally fixed nor the all-flexible extreme. I prefer a balanced approach, a compromise between flexibility and ease of use, based on the actual needs.
The problem is that I disagree with your classification of ease of use. I consider your "balanced" approach to be a lot worse in ease of use than the totally fixed solution and not much better than the all-flexible solution.

A fixed number of 16 properties doesn't simplify the use of the structure, you still have to loop thru the properties to find a free slot or a certain property. You would only eliminate the OpenMediaFormatGetVideoInfoHeaderSize helper function, everything else would be as complicated as with a variable number of properties. As a result, a variable number of properties is the better trade-off.
No. If you have one dynamic element in a structure, you can put it at the end, no problem. But if you have 2 dynamic elements, you've got a problem, because the offset of the 2nd element isn't fixed, anymore. MPEG2VIDEOINFO is already dynamic. If you make the number of flex items dynamic, as well, either the flex items or the MPEG2VIDEOINFO structure will lose their fixed offset in the structure, making it necessary to use pointer arithmetic. If the number of flex items is static, both flex items and the MPEG2VIDEOINFO structure will have a static offset in the structure, noticeably improving ease of use.

I consider the necessity to use pointer arithmetic (or helper functions as a replacement) as a noticable drop in ease of use. Looping through an array can easily be done without helper functions. In Delphi you don't normally declare helper functions as part of a structure. In Delphi structures are typically defined static, with any dynamic elements positioned at the very end of the structure. Most Windows structures are also declared like that, having any dynamic elements placed at the end of the structure, while all other elements are static.

IMHO, if we drop ease of use by requiring pointer arithmetic, then we shouldn't stop half way, but go right through to making things fully flexible.

It is quite likely that more that 16 properties are defined in the future.
Defined, yes. But used at the same time? I doubt it. And as I suggested, let's make it 64, then we should be safe. If you insist, I can also live with 256 items, I don't really care much about wasting a couple of bytes.

[madshi]

We need to extend the DXVA_ExtendedFormat definition a bit. Currently it does not support all the values defined by h264. Here's a list of what h264 defines and what Open Media Format allows:

http://forum.doom9.org/showthread.php?p=1519740#post1519740

Look for "[DXVA: -]" to see where we need to extend the DXVA_ExtendedFormat definitions.