DirectShow Media Types for Stereoscopic Content [Archive] - Page 3

clsid

27th July 2011, 16:27

I didn't read the whole discussion and I am far from an expert on this, so feel free to ignore me. Here are some of my thoughts.

The base of the structure as defined in the spec should have a fixed size as that is easier to work with. As I understand the 'extensions' are there to add additional non-standard features in the future. Limiting extensions to a fixed number shouldn't be much of a problem. If the amount becomes insufficient, which would likely only occur in very specific apps, or maybe never at all, then you could just group together some uncommon extensions in one new larger extension. The base should contain a list of extensions, plus the data size of each extension, and maybe if needed also the offset from beginning of the structure. Then an app can easily find the data belonging to each extension. Use the ones it understand, and skip the ones it doesn't support. Seems natural to use a GUI as a fixed size unique identifier for extensions.

madshi

27th July 2011, 16:29

Thanks for your comment, clsid.

JanWillem32

27th July 2011, 20:29

I took some time to read trough this topic carefully. I looked up several concepts, implementations and a lot more stuff.
I do still have some questions.
The only real things I'm working on at the moment are a software subtitle texture renderer, two video renderers on top of a vanilla EVR mixer, one video renderer on top of a vanilla VMR-9 mixer and some accessory items for those. I'm reviewing this from the perspective that the renderers I'm working on need a lot of work to ever produce quality output with decent efficiency.
Because the mixer parts I'm using currently are pretty much vanilla and not very suited to the renderers attached to them, I don't mind replacing them at all. The thing is, DirectShow is old, Meda Foundation is new. I'm unfortunately the only one who is actively writing big code changes for the renderers I'm working on right now. From my perspective, I can just as easily use vanilla Meda Foundation code for the mixers as the pretty much vanilla DirectShow code the renderers are using right now, and work from that point to improve video mixer parts.
What good would a new iteration of a DirectShow interface bring like this? What Media Foundation parts don't cover the items proposed in this new format?

I also saw the references to support DCI media packages. I know that the DCI format is pretty much set in stone and doesn't allow much variants, so the header for the DCI packages doesn't store much data. (The formats are luckily also very incompatible with any consumer video format.) Just to be sure, are there any parts of the DCI media package header format, or some hidden data that's not in that header, that could matter for this implementation?

nevcairiel

27th July 2011, 21:15

MediaFoundation has by design a very flexible layout (storing key-value pairs with GUIDs as keys and a wide variety of value-types), so this whole design is not required for MediaFoundation, its only to get DirectShow up-to-par for the coming time, because its not going away any time soon.

PS:
EVR is a Media Foundation tech, thats why internally you can/have to use alot of the MF things - however externally, it still has to look like DirectShow if you want to use it in a DirectShow player - so you cannot use the Media Foundation Media-Type concept, if thats what you were kind of implying.

Matt Ashland

1st August 2011, 19:24

madshi asked me to weigh in on this discussion. The smartest guys in the room have already been here, but I'll post my thoughts anyway :P

First, let me say that J. River will be happy to support this format specification in Media Center however it gets implemented.

As for the technicals, my preference is something fully flexible. If it were done properly, it could be used to solve the problem of video tagging. I realize storing actors, box art, etc. is outside the scope of this conversation, but it's a gaping hole in the video world compared to audio. If there were a standard block of data to describe everything about a video, including the metadata, it would be wonderful.

With that said, the current discussion is focused more tightly on the video format information only. For this, a fixed structure with some number of FLEXVIDEOITEM objects seems like a reasonable compromise.

madshi

1st August 2011, 19:36

Thanks for your input, Matt, I appreciate it!

Would you be willing to post your opinion about the questions in the following post?

http://forum.doom9.org/showpost.php?p=1515930&postcount=93

About video tagging: Maybe it would make sense for nevcairiel and you to discuss ways for LAV Splitter to export whatever tagging the various container formats support? LAV Splitter could eventually export a new interface which you could use to retrieve that kind of information. @nevcairiel, what do you think? To be honest, I don't really know, though, how much video tagging any of the current containers support. Probably MKV has the broadest support for that kind of thing? Not sure. In any case, transporting things like cover art etc via the media type information would go too far, IMHO. For these kind of things I'd really prefer having the splitter export an interface.

nevcairiel

1st August 2011, 19:39

MKV and MP4 can contain numerous tags, MP4s do sometimes, with MKVs its pretty rare. An interface to export this metadata would surely be possible, ffmpeg supports reading it in a multitude of containers and maps it into a set of common metadata keys.

pwimmer

1st August 2011, 19:45

Do we want Open Media Format to be used for transporting compressed video bitstream, too? If so, we may need to alternatively support VIDEOINFOHEADER2 or MPEG2VIDEOINFO. Furthermore the sequence headers will then follow. So that again would require VIDEOINFOHEADER2 (or MPEG2VIDEOINFO) to be the last element in the structure.
Thoughts?

I don't think it is necessary for compressed formats.

Matt Ashland

1st August 2011, 20:02

@Specific questions about video header

1) Should the format be extensible?:
Yes

2) Should we use a GUID to identify extensions?:
GUIDs are good because they're a fixed size and they never collide. But strings are easier to understand (for a user and programmer). If you decide to use strings instead, please specify a naming style because key naming is a mess in some tag formats like Ogg tags.

3) What should be the max. data size per extension?
Worrying about size is silly. A modern movie is 30 GB. So allocate more space than you think you'll need. I also like the idea of a flexible sized format block so extensions could be any size.

4) What data types should be allowed in the extension?
Don't bother with low precision types. If you have double, you don't need float. And if you have int64, you don't need int32, int16, uint16, byte, boolean, etc.

5) Should the extension use a single item (e.g. "BOOL boolValue"), or should it use arrays (e.g. "BOOL boolValue[64]")?
It's harmless to use arrays, and might be useful for types like a rectangle. But it might be good form to use binary data (with a structure define) if you're specifying an array or other complicated structure.

6) Should there be a type flag and an item count?
I wouldn't add the type flag. The key (GUID or string) is sufficient. Duplication just creates more ways for the person writing the format block to make a mistake.

7) Should there be a fixed number of extensions?
Is there some clear advantage to a fixed size header? If not, just put a stack of extensions at the end of the format block with a count or byte size in the header.

8) Should there be a non-flexible part?
Yes. You could make the non-flexible part just another extension, and require that extension to be present. But I think this may be getting too smart, especially if you're requiring fixed size extensions which limits extensions anyway.

@Tagging

I'd like to see a simple tag format that could be used in a variety of containers. Tag format folks often get too smart for their own good (XMP, ID3v2, etc.) when a simple string-to-string map (either in binary or as XML) is sufficient. You're probably right that it's outside the scope of this conversation. However, we do have the right cast of characters here to make something really useful for users. Currently J. River has to use sidecar XML files, which is gross. If anyone here ever wants to tackle this topic, J. River would be glad to help where possible.

pwimmer

1st August 2011, 20:11

Thinking over the the pros and cons of the flexible part, my preference would be to have only VIH2 and a flexible part, nothing else.

Let me explain why: The current draft contains a lots of fields for stereoscopic media, some of them would rarely be used. I don't want the structure to be bloated with such stuff. After all, many people have no interest in the stereo stuff, the same applies for the degradation flags. I don't think we should design the structure for the few of us participating in this discussion, but rather keep the Open Media Format more generic and generally applicable. To avoid any future limitations, we should not limit the number of flexible items. I know that an unlimited number of flex-items (or properties as I call them) is more difficult from a developers point of view, but it is the only way to keep the format truly extensible.

The OpenMediaFormat.h header should just contain the general definitions, while all the application-specifc stuff should be moved to application-specific headers, e.g. OpenMediaFormatStereo.h and OpenMediaFormatDegradation.h. New application-specific headers can be added at any time.

An OpenMediaFormatHelpers.c file can contain helper functions to simplify accessing the properties.

OpenMediaFormat.h would basically be reduced to the following definitions. Imho, small is beautiful.

typedef struct tagOPENMEDIAPROPERTY {
GUID guidProperty;
union {
BOOL boolValue[4];
BYTE byteValue[16];
INT intValue[4];
UINT uintValue[4];
GUID guidValue;
LONGLONG longlongValue[2];
ULONGLONG ulonglongValue[2];
FLOAT floatValue[4];
DOUBLE doubleValue[2];
};
} OPENMEDIAPROPERTY;

typedef struct tagOPENVIDEOINFOHEADER {
VIDEOINFOHEADER2EX videoInfoHeader2; // Compatible with VIDEOINFOHEADER2 (no color palette allowed)
UINT propertyCount;
OPENMEDIAPROPERTY properties[];
} OPENVIDEOINFOHEADER;

BetaBoy

1st August 2011, 20:16

Matt Ashland... Thank you for the support.

All.... As things settle down with the spec over the next few weeks, we are gonna gonna have it listed on www.OpenMediaFormat.com. I'll work with Peter, Madshi and the rest of you, but the thoughts are to design the site like we did with Matroska.org to make it simple to follow the specs and any proposals. We can leave the discussions here on D9, as this seems to work best for everyone.

pwimmer

1st August 2011, 20:25

Agreed, Doom9 is a good place for the discussion. Lets continue the discussion for a few weeks, and lets see if some proposals are widely accepted.

As I mentioned before, we are not in a hurry because we should not finalize the spec before Microsoft has released the Windows 8 SDK Beta. Even if they do not add features to DirectShow anymore, they will improve Media Foundation, and I want to keep as much of the Open Media Format compatible with Media Foundation definitions as possible. I just want to see what Microsoft does before deciding on own extensions for DirectShow.

madshi

1st August 2011, 20:38

@Matt, thanks again!

@BetaBoy, it's nice that you'll support this, too, even with its own homepage! :)

@pwimmer, there's still the problem with bitstream video headers. We will probably even have to support MPEG2VIDEOINFO instead of VIDEOINFOHEADER2 for bitstream connections. See the definition of MPEG2VIDEOINFO to see how the video bitstream headers are stored. This is used for all MPEG2, VC-1 and h264 connections.

nevcairiel

1st August 2011, 21:40

I kind of agree that its not necessarily required for compressed bitstreams, they have all the extra info in their sequence data. That would also solve the problem with the MPEG2VIDEOINFO.

madshi

1st August 2011, 21:54

I kind of agree that its not necessarily required for compressed bitstreams, they have all the extra info in their sequence data. That would also solve the problem with the MPEG2VIDEOINFO.
I'm not sure I understand what you mean. What is "it" in the "its not necessarily required"?

nevcairiel

1st August 2011, 21:55

The OPENVIDEOINFOHEADER of course.
I thought that was kind of clear, seeing how this thread is about it and all. :d

madshi

1st August 2011, 22:09

nevcairiel

1st August 2011, 22:15

Could just go with VIH2, and if someone needs extra special information thats usually stored in MP2VI, store it in a flex item.
Allowing VIH2 and MP2VI in the same structure is really overcomplicating things, imho.

madshi

1st August 2011, 22:27

The length of the sequence headers could be larger than the max size we've defined for a flex item, though.

Ok, so how about going one step further, making the flex items totally flexible, not defining any standard type at all, like this:

typedef struct tagOPENMEDIAPROPERTY
{
GUID identifier;
int offset;
int size;
} OPENMEDIAPROPERTY;

typedef struct tagOPENVIDEOINFOHEADER
{
VIDEOINFOHEADER2EX videoInfoHeader2;
int propertyCount;
OPENMEDIAPROPERTY properties[];
} OPENVIDEOINFOHEADER;
Every flex item would then have its own private type.

Of course this flexibility would come at the cost of easy of use.

pwimmer

1st August 2011, 22:44

Oh, I somehow missed pwimmer's post you were refering to.

So, one question is: "Is OPENVIDEOINFOHEADER necessary for compressed formats"? Maybe not. But I think the better question is: "Do we want to limit OPENVIDEOINFOHEADER so that it doesn't work (well) for bitstream connections"? I'm not sure if that is a good idea. Isn't that exactly the opposite of the whole "flexible" idea? One purpose of the "flex" items could be to transport additional information that is stored in the container up the filter chain. If OPENVIDEOINFOHEADER is not to be used for compressed formats, then the chain is broken, and OPENVIDEOINFOHEADER cannot be used for that purpose, anymore.

E.g. what happens if the stereo "separation" information is stored in the container? How can the splitter forward this information to the decoder/renderer, if OPENVIDEOINFOHEADER can't be used for compressed formats?

I would suggest to define OPENMPEG2VIDEOINFO for that purpose if it turns out to be necessary. We might also have to define OPENWAVEFORMAT to extend WAVEFORMATEX if somebody has a need for it. In general, we will define new structures and interfaces for all stuff that is missing in DirectShow. We just start with the OPENVIDEOINFOHEADER because there is definitely a need for it.

pwimmer

1st August 2011, 22:48

I've uploaded a new proposal for the fully flexible format without any fixed fields except VIDEOINFOHEADER2.

OpenMediaFormat.zip on 3dtv.at server (http://www.3dtv.at/Downloads/OpenMediaFormat.zip)

The stereo fields are now stored in separate properties. For the degradation information a single property turned out to be sufficient.

To simplify access to the property values I've added further entries to the union of OPENMEDIAPROPERTY, this also elimites any doubt about the type for each property.

madshi

2nd August 2011, 08:23

I would suggest to define OPENMPEG2VIDEOINFO for that purpose if it turns out to be necessary.
That wouldn't solve the problem, though. The key thing with MPEG2VIDEOINFO is that it contains *dynamically* sized bitstream sequence headers. So we can't put MPEG2VIDEOINFO at the beginning of an OPENMPEG2VIDEOINFO structure. I'd also prefer if we could get along with only one structure for all video related purposes. My suggestion would be to:

(1) Either always use MPEG2VIDEOINFO instead of VIDEOINFOHEADER2. The additional MPEG2VIDEOINFO fields would then simply be set to zero for uncompressed video connections. For this to work, MPEG2VIDEOINFO would probably have to be at the end of the OPENVIDEOINFOHEADER structure.

(2) Or stick to VIDEOINFOHEADER2 and store the additional fields and the sequence headers of MPEG2VIDEOINFO into a flex item. This would only be possible, though, if we don't limit the max size of a flex item, because there's no law that specifies how long the sequence headers may get.

We might also have to define OPENWAVEFORMAT to extend WAVEFORMATEX if somebody has a need for it.
You mean defining a separate structure for audio? Ok, we could do that. I don't have enough knowledge to know what is needed for that, though.

I've uploaded a new proposal for the fully flexible format without any fixed fields except VIDEOINFOHEADER2.

OpenMediaFormat.zip on 3dtv.at server (http://www.3dtv.at/Downloads/OpenMediaFormat.zip)

The stereo fields are now stored in separate properties. For the degradation information a single property turned out to be sufficient.

To simplify access to the property values I've added further entries to the union of OPENMEDIAPROPERTY, this also elimites any doubt about the type for each property.
First of all, I don't really like having a dozen of header files for just one format. Let's not make it more complicated than necessary. I'd prefer having the main structure defined in one header file. Ok, definitions for flex items could go into separate header files, though.

If I counted right, "sizeof(STEREOVALUES) == 76". You personally wanted the max size of a flex item to be 16. Now the size of a flex item is defined by sizeof(STEREOVALUES)? That doesn't make much sense to me.

Personally, I would like to see one of the following 2 solutions:

(1) Either we define the exact size of each flex item, and also the exact number of flex items. If we do that, we can still put MPEG2VIDEOINFO at the end of the structure.

(2) Or we make the flex items *totally* flexible, allowing each flex item to have any size it needs (without any limitations). In that case we would need to store an array of flex item offsets and sizes into the OPENVIDEOINFOHEADER. Each flex item would then have its own private structure.

pwimmer

2nd August 2011, 10:21

If I counted right, "sizeof(STEREOVALUES) == 76". You personally wanted the max size of a flex item to be 16. Now the size of a flex item is defined by sizeof(STEREOVALUES)? That doesn't make much sense to me.

STEREOVALUES has a size of 16 bytes, it is a union, not a struct. STEREOVALUES serves the only purpose to simplify access to OPENMEDIAPROPERTY and to clarify the type for each property.

For example, instead of stereo.numViews you could access intValue[0]. It is an alias for the same memory location that makes the source code more readable. It helps to solve some of the drawbacks of the flexible format. I'll also write some helper functions that allow code like this:

OpenMediaFormatReadProperty(pOpenVideoInfo, OPENMEDIAPROPERTY_NUMVIEWS)->stereo.numViews

madshi

2nd August 2011, 11:56

Oh, missed the "union", sorry. What about the other parts of my post?

pwimmer

3rd August 2011, 00:09

(1) Either we define the exact size of each flex item, and also the exact number of flex items. If we do that, we can still put MPEG2VIDEOINFO at the end of the structure.

(2) Or we make the flex items *totally* flexible, allowing each flex item to have any size it needs (without any limitations). In that case we would need to store an array of flex item offsets and sizes into the OPENVIDEOINFOHEADER. Each flex item would then have its own private structure.

Making the flex items totally flexible, thus allowing a variable size for the flex items, makes it nearly impossible to calculate the required size of the structure. Imho this is not possible if we want to keep the Open Media Format useable.

VIDEOINFOHEADER2 and MPEG2VIDEOINFO could be located at the beginning or the end of the structure. Both is possible.

madshi

4th August 2011, 12:02

Making the flex items totally flexible, thus allowing a variable size for the flex items, makes it nearly impossible to calculate the required size of the structure. Imho this is not possible if we want to keep the Open Media Format useable.
IMHO we should decide whether we want to use a structure which requires pointer arithmetic or not. If we do, we will need helper functions to improve usability. Once we go dynamic, using pointer arithmetic and helper functions, we can also make the size of the flex items totally dynamic. We could simply add another helper function to calculate the requires size of the structure. IMHO requiring pointer arithmetic, using helper functions, but still sticking to a fixed size for flex items is neither here nor there. I'd vote for either using a totally static structure (fixed number & size of flex items), or a totally dynamic structure (dynamic number & size of flex items).

VIDEOINFOHEADER2 and MPEG2VIDEOINFO could be located at the beginning or the end of the structure. Both is possible.
FWIW, if you put MPEG2VIDEOINFO at the beginning of the structure, the position of the flex items is not fixed, anymore, because the size of MPEG2VIDEOINFO is dynamic.

pwimmer

4th August 2011, 12:27

IMHO we should decide whether we want to use a structure which requires pointer arithmetic or not. If we do, we will need helper functions to improve usability. Once we go dynamic, using pointer arithmetic and helper functions, we can also make the size of the flex items totally dynamic. We could simply add another helper function to calculate the requires size of the structure. IMHO requiring pointer arithmetic, using helper functions, but still sticking to a fixed size for flex items is neither here nor there. I'd vote for either using a totally static structure (fixed number & size of flex items), or a totally dynamic structure (dynamic number & size of flex items).

I want to keep it user friendly and allow a command sequence like this one:

OPENVIDEOINFOHEADER *pvi = (OPENVIDEOINFOHEADER *) pMediaType->AllocFormatBuffer(OpenMediaFormatGetVideoInfoHeaderSize(numProperties));

...

OPENMEDIAPROPERTY property;
property.guidProperty = OPENMEDIAPROPERTY_STEREOLAYOUT;
property.stereo.layout = STEREOLAYOUT_SIDEBYSIDELEFTFIRST;

OpenMediaFormatAddProperty(pvi, &property);

property.guidProperty = OPENMEDIAPROPERTY_SEPARATIONX;
property.stereo.separationX = 20;

OpenMediaFormatAddProperty(pvi, &property);

I don't see any need for large properties. The OPENVIDEOINFOHEADER is used for format negotiation and it is unlikely that strings will ever be needed.

Strings like artist, copyright, chapter names and other metadata are not part of the format but should be read via other filter interfaces.

FWIW, if you put MPEG2VIDEOINFO at the beginning of the structure, the position of the flex items is not fixed, anymore, because the size of MPEG2VIDEOINFO is dynamic.

So we put it at the end of the structure. Question is, should we put VIDEOINFOHEADER2 also at the end? This prevents typecasts from OPENVIDEOINFOHEADER to VIDEOINFOHEADER2 , but a simply helper function could solve the problem.

madshi

4th August 2011, 14:52

I want to keep it user friendly [...]
Well, then let's return to having a fixed flex item size *and* a fixed number of flex items. That would allow us to have a fully static structure with no dynamic elements (other than MPEG2VIDEOINFO). Advantage: No pointer arithmetic needed at all, no helper functions needed at all.

I don't see any need for large properties. The OPENVIDEOINFOHEADER is used for format negotiation and it is unlikely that strings will ever be needed.

Strings like artist, copyright, chapter names and other metadata are not part of the format but should be read via other filter interfaces.
Well, we have 2 choices: Either we go dynamic and allow anything that could ever be used. Or we go in the other direction and put a priority on ease of use. In the latter case I don't see any need for more than max 16 flex items. Let's make it 32, or even 64, and we should be very safe. No need to allow a dynamic number of items. Going static would noticeably improve ease of use IMHO.

So we put it at the end of the structure. Question is, should we put VIDEOINFOHEADER2 also at the end? This prevents typecasts from OPENVIDEOINFOHEADER to VIDEOINFOHEADER2 , but a simply helper function could solve the problem.
As luck has it, MPEG2VIDEOINFO starts with a VIDEOINFOHEADER2 structure. So IMHO we should simply always use MPEG2VIDEOINFO. Or we could put M2VI and VIH2 into a union at the end of the OPENVIDEOINFOHEADER structure.

pwimmer

4th August 2011, 15:02

I neither want the totally fixed nor the all-flexible extreme. I prefer a balanced approach, a compromise between flexibility and ease of use, based on the actual needs.

A fixed number of 16 properties doesn't simplify the use of the structure, you still have to loop thru the properties to find a free slot or a certain property. You would only eliminate the OpenMediaFormatGetVideoInfoHeaderSize helper function, everything else would be as complicated as with a variable number of properties. As a result, a variable number of properties is the better trade-off.

It is quite likely that more that 16 properties are defined in the future. On the other hand, nobody has suggested a property yet that requires more than 16 byte of data.

madshi

4th August 2011, 15:38

I neither want the totally fixed nor the all-flexible extreme. I prefer a balanced approach, a compromise between flexibility and ease of use, based on the actual needs.
The problem is that I disagree with your classification of ease of use. I consider your "balanced" approach to be a lot worse in ease of use than the totally fixed solution and not much better than the all-flexible solution.

A fixed number of 16 properties doesn't simplify the use of the structure, you still have to loop thru the properties to find a free slot or a certain property. You would only eliminate the OpenMediaFormatGetVideoInfoHeaderSize helper function, everything else would be as complicated as with a variable number of properties. As a result, a variable number of properties is the better trade-off.
No. If you have one dynamic element in a structure, you can put it at the end, no problem. But if you have 2 dynamic elements, you've got a problem, because the offset of the 2nd element isn't fixed, anymore. MPEG2VIDEOINFO is already dynamic. If you make the number of flex items dynamic, as well, either the flex items or the MPEG2VIDEOINFO structure will lose their fixed offset in the structure, making it necessary to use pointer arithmetic. If the number of flex items is static, both flex items and the MPEG2VIDEOINFO structure will have a static offset in the structure, noticeably improving ease of use.

I consider the necessity to use pointer arithmetic (or helper functions as a replacement) as a noticable drop in ease of use. Looping through an array can easily be done without helper functions. In Delphi you don't normally declare helper functions as part of a structure. In Delphi structures are typically defined static, with any dynamic elements positioned at the very end of the structure. Most Windows structures are also declared like that, having any dynamic elements placed at the end of the structure, while all other elements are static.

IMHO, if we drop ease of use by requiring pointer arithmetic, then we shouldn't stop half way, but go right through to making things fully flexible.

It is quite likely that more that 16 properties are defined in the future.
Defined, yes. But used at the same time? I doubt it. And as I suggested, let's make it 64, then we should be safe. If you insist, I can also live with 256 items, I don't really care much about wasting a couple of bytes.