Request: Guide to Understanding Demuxers, Splitters, Decoders, Renderers

[spartan711]

Hi everyone,

I was wondering if someone could do a quick writeup, or point to an existing writeup, of the exact process of a video file goes through when being played. And what the role of each part is, ex. the splitter is responsible for seperating the video, audio, and subtitle streams from each other for decoding. Also, how all of this ties into the frame rate? 23.976 vs. 24.00Hz?

The difference between hard-coded subs and soft subs in these terms would be a great help. Also, There are too many options for each step! There is Haali, Lavsplitter, MadVR renderer, Gabest, Reclock, autofrequency, etc, etc, etc! It makes a person crazy, and is very hard to know the pros and cons of each thing, especially if someone wants PQ over AQ, or bitstreaming audio, etc.

I am grateful to the community for creating such a vast information empire, and the multitude of options! However I would appreciate a nice summary, and I'm sure it would be a great help to many others. Hey maybe it could even get stickied! Thanks for looking.

[dansrfe]

- Demuxers separate a file into its elementary streams
- Splitters "split" the elementary streams, label them and send it to the directshow player for decoding or re-routing to a decoder
- Decoders decode the elementary streams accordingly
- Renderers render, sometimes convert colorspaces and essentially display what you see through the directshow player

Framerate is just a characteristic of a video stream. madVR, latest MPC-HC and ffdshow are all that you need to do everything with max PQ/AQ. How you set them up is based on your preferences.

[Hypernova]

Hard sub is in the video stream itself; it's part of the picture you're seeing. There is no different between a person and the hard sub text in each frame. On the other hand, softsub is save as a text file ie. it's like an .txt file that just got packed together with video, like audio is packed together with video stream. Then, at some point, it got read and render with the picture. This can be done by decoder (eg. ffdshow), or seperate filter (eg. VSFilter).

Maybe the easiest comparison is: Suppose you have a document. You can save it as an image file (jpg, bmp, etc.), or you can save as one of document file (txt, rtf, word document, etc). The former is hardsub, the later is softsub.

[roozhou]

Maybe the easiest comparison is: Suppose you have a document. You can save it as an image file (jpg, bmp, etc.), or you can save as one of document file (txt, rtf, word document, etc). The former is hardsub, the later is softsub.
Wrong. Softsub can either be text or bitmap(e.g. vobsub). The main difference is that hardsub is part of the video image itself while softsub is not.

[nevcairiel]

Demuxers and Splitters are really the same thing. Splitter is a DirectShow term that people keep using for Demuxers.
Its task is taking the file, and splitting (demuxing) it into its individual streams, for example taking the video and sending it to the video decoder, and sending the audio to the audio decoder.

The Decoder is the component that takes the compressed streams, and decodes them into raw data. This can be both audio and video.

The Renderer is then responsible for delivering the decoded data to you. This can also be both audio and video. The Audio Renderer takes the raw PCM Audio data, and sends it over the sound card to your Speakers. The Video Renderer is responsible for showing the video frames on your screen, in the proper order and proper timing.

[Hypernova]

Wrong. Softsub can either be text or bitmap(e.g. vobsub). The main difference is that hardsub is part of the video image itself while softsub is not.

My bad. roozhou's right. :thanks:

[spartan711]

Thanks for all the replies everyone.

So for windows, demuxers = splitters, good to know. I will be trying to understand each part of this process, so please bear with me.
Possible Splitters:
-MPC internal splitters
-FFDshow (splitters + decoders)
-LAVsplitter (new, splitters + decoders)
-Haali
-Shark (I think its splitters + decoders)??
-Gabest??

Is there any reason to use one over the other? Is there any difference? At least in terms of "splitting"?

P.S. - @dansrfe thanks for the tip! I am trying to get into the guts of the thing

[dansrfe]

MPC Internal splitters for TS/PS files, FLV files and some other source splitters are good. Other than that the rest of the decoders should be disabled. ffdshow does not split anything. It's just a huge library of decoders.

Yes there are differences in splitters. For example mpc-hc's internal mpeg splitter (can be ussed for mp3's in this case) does not give the correct duration for a variable bitrate mp3 file. Specifically for this reason I use Gabest's mpasplitter.ax for splitting mp3 and then have it send to ffdshow for decoding.

[nevcairiel]

Thanks for all the replies everyone.

So for windows, demuxers = splitters, good to know. I will be trying to understand each part of this process, so please bear with me.
Possible Splitters:
-MPC internal splitters
-FFDshow (splitters + decoders)
-LAVsplitter (new, splitters + decoders)
-Haali
-Shark (I think its splitters + decoders)??
-Gabest??

Is there any reason to use one over the other? Is there any difference? At least in terms of "splitting"?

P.S. - @dansrfe thanks for the tip! I am trying to get into the guts of the thing

Like dansrfe said, ffdshow is just decoders.
"Shark" is nothing, its just a codec pack by some guy named "Shark", he doesn't provide any splitters or codecs of his own.

[spartan711]

@dansfre, I get the correct duration using v1.5.0.2827 with a V0 mp3 file. Graphstudio shows it goes to the MPEG-I Stream Splitter, then goes to ffdshow audio decoder. EDIT! - You're correct! It shows 8:20... crazy. I don't normally play audio with mpc-hc though, so no worries.

It seems like all of these splitters (Besides haali) is based on FFmpeg. Its interesting that there are so many forks of the project. So in order to get all necessary splitters for video playback, just install latest rev MPC-HC?

[nevcairiel]

Only my LAV Splitter is based on ffmpeg, all other splitters use their own magic. For decoders, they are mostly based on ffmpeg. MPC internal, ffdshow, mine, etc.

[spartan711]

So can splitters enable/disable features? For example, I was looking at the haali splitter, and it is incompatibile with TrueHD, yet is compatible with DTS. So what about his splitter cannot split that audio stream, vs mpc-hc's splitter (is there a name for the internal splitters?) ? If its actually a decoder problem, then I understand. But from what it seems like, its a splitter problem?

[nevcairiel]

Splitters can have problems, too!

[spartan711]

I'm sure... New example. Say you have MULTIPLE splitters on a computer. When playing a file, how does a player decide which splitter to use? Does it use some sort of Profiler Program, such as MediaInfo, that detects video/audio streams, which then lets the player which splitter to use? Or does the player simply use a splitter by default?

[dansrfe]

The player will first check it's own internal collection of splitters. If the specific splitter needed to decode the file being sent to the player is "checked" or "allowed" from the player's own internal splitter library then it will start to split from there and send to whichever decoder is set to decode in the next part of the chain. If none of the "checked" or "allowed" splitters from the player's own internal library are available then the player will refer to what is known as a DirectShow merit list. This merit list is Windows recognized and is used system wide for all DirectShow applications. The splitter suitable for splitting the desired media file that has the highest "merit" from that list is chosen to split the file and is then directed to send the data to the decoder as specified by the player.

[spartan711]

That's a great summary of the process. But I'm still kind of confused on how to player decides which specific codec to use. How does it interact with the file to know what format audio is present?

For example, if a video file has TrueHD audio, and there are three audio splitters - A ac3, B dts, and C truehd, how does the player interact with the file to get the audio codec information so it knows to use splitter C?

[nevcairiel]

The Player does not know what is in the file before it chooses the splitter. All it sees is like "Oh, this is a .mkv file, now which splitter supports .mkv files?" and then loads the splitter. The Splitter then determines what formats are in the file, and exposes them to the player - so if some splitter does not support TrueHD audio in .mkv, the player wouldn't know not to use it. its basically your fault for installing a faulty splitter.
The player (or the DirectShow Filter Graph) can then decide which decoders to use. At this point, the Splitter has told the player which format the Audio is in, and the decoders have the information attached which format they can decode.

[spartan711]

I see. So there is no way to conditionally set a splitter, that's good to know. And there is no way to use multiple splitters at once? Like in the above scenario, this would be helpful if you have a mix of TrueHD and DTSMA files, and one splitter did not support both.

On a side note, does anyone know of a splitter that supports both hd audio codecs, in full 16-24bit and up to 192kHz?

[dansrfe]

ArcSoft HD audio decoder.

[nevcairiel]

You're mixing up splitters and decoders again.

[spartan711]

Wait me or dansrfe? I thought a necessary part of decoding those codecs is detecting those codecs, otherwise how can you hope to decode it without an accurate picture of what it is? I thought it would be necessary to detect the 24 bit vs 16 bits, as detecting as 16 bit would lead to "quality loss". I know xbmc currently converts everything to 16 bit.

edit: I remember reading somewhere in the audioengine thread that part of the reason why bitstreaming/decoding hd audio is so difficult is that no splitter currently detects them properly, i.e. dtsma at a bitrate above 1.5mbps, or 24 vs 16 bit. However, that was probably about a month ago...

@dansfre - I am currently using that right now to play files, although on an analog 2.1 setup. I found that other thread which was a great resource where the filter pack was posted.

[spartan711]

ARGH! I refreshed the page and lost my post.

Essentially, my question boils down to how can you decode something without knowing exactly what it is? That's why I asked about a splitter that supports both truehd and dtsma, because otherwise I shouldn't be able to play them.

I remember reading about a month ago in the audioengine thread at xbmc that the reason bitstreaming/decoding hd audio was so hard is that no splitter currently detects dtsma or truehd streams properly, i.e. the full bitrate and 24 bit audio.

@dansfre - I found the thread in which you were active in regarding the arcsoft decoders, and I have been using them for about a week. Thanks!

EDIT: sorry for double post, didn't realize it went to page 2. Delete if you want mods!

[nevcairiel]

The splitter just needs to detect that there is a Audio Stream, and which general type it is.
Like my LAV Splitter will detect all DTS-HD streams just as DTS, because ffmpeg (which it is based on) does not support DTS-HD. But that doesn't matter, as marking it as DTS is enough for the decoders. The audio itself contains information which DTS profile it contains, what sample rate, and all that information. The decoder just needs to be told that its DTS that its decoding, it doesn't necessarily need to be told that its DTS-HD MA, or DTS-HD HR, or just plain DTS Core, it can detect that from the incoming audio data.

In any case, my LAV Splitter will support all popular formats in all the popular containers, including TrueHD and DTS in MKV, TS and whatever other main-stream containers there are.

For decoding DTS-HD MA, you will have to use something like the ArcSoft HD Audio Decoder, as its the only decoder i know that can fully decode DTS-HD MA and be freely used in other DirectShow players (once you set it up properly, that is, and of course you need to own a copy of TMT3)