MPEG-4 B-frames in AVI/VFW hackery description

[bond]

As i had to describe a bunch of people how b-frames can be placed in .avi/be used during enc/decoding in vfw, i thought it would be better to write some sort of a short doc once and for all, which i could point people too, so here it is:

after reading it you should know:
- what packed bitstream is
- what delay frames are
- what xvid's decoder lag message is
- what all has to be done for keeping outdated .avi and vfw alive


first of all the problem

the "video for windows" (VFW) codec interface (as used in virtualdub(mod)) and its container (AVI) can NOT handle b-frames, and they will never even know that this frame type exists!
therefore it can be said that these are outdated technologies, as they themselves cant handle modern technologies, like b-frames

now there are two possibilites if you want to use b-frames:
1) simply dont use outdated technologies, not able to handle b-frames (and use better technologies like DirectShow and .MP4)
2) use outdated technologies and develop and use hacks

two different hacks

ad 2) atm there are two different hacks existing to make b-frames work with avi and vfw:
a) on the encoder side (called packed bitstream as used by default in xvid and divx5)
b) on the decoder side (as used by xvid, if packed bitstream is disabled)

basics

now if we want to understand the hacks for using b-frames in avi and vfw, we have to know the following:

normally the frames are stored in the container this way:
I P B B
they get displayed this way:
I B B P

VFW and AVI use a "one frame in, one frame out" sheme (thats simply how the technology works), which means that on every frame inputted, you have to output one (on both the encoding and decoding side).
this is not compatible with b-frames, as b-frames are constructed by using two frames at once, the previous and following i/p frame (but vfw/avi dont allow something like "two frames in, one frame out" or so (in contrary to modern "not outdated" technologies, like directshow or .mp4)

now normally (by using up-to-date technology) the decoder would do the following during decoding:
1) display the I-frame
2) now he wants to display the B-frame, for which he needs the previous and next I/P-frames. so, having the already decoded I still ready, he graps the P and therefore now is able to decode the B too (its a "3 frames in, one frame out" situation)
3) the same goes for the second B
4) than he displays the P-frame

hackery when decoding

now lets go on to the hacks:
ad a) as AVI and VFW only can do "one frame in, one frame out", the following workaround is done on the encoder side (called packed bitstream):
the first B gets packed together with the P-frame to one frame:
I P B B becomes
I PB B N
(N is a not coded placeholder frame to still have the same framenumber)

now the decoder does the following:
1) he decodes the I
2) for decoding the first B-frame he needs (as described above) the I and the P
as the first B is packed together with the P, he "officially" only gets only 1 frame (as PB appears as 1 frame), but in fact they are two frames (P + B), now is able to decode the first B
3) as he has the I and P already he is able to decode the second B too
4) than he displays the P

the hack here is that, as avi/vfw is forced to the "one frame in, one frame out" sheme, you circumvent this by feeding it with two frames at one time, by packing them together to one frame
some devs (like michael niedermayer from ffmpeg) have the opinion that packing two frame to one breaks the compliancy of the stream to the mpeg-4 standard! also a normal mpeg-4 decoder, written following the mpeg-4 specs, not knowing packed bitstream, cant decode packed bitstreams


ad b) the second possible hack is that the stream gets stored correctly in the avi (as I P B B) but the hack happens on the decoder side:
1) first the decoder gets the I-frame, but does not display it (instead he displays for example the famous "decoder lag" message of xvid!)
2) in the second step the decoder gets the P as input, but now only displays the I frame (the first frame) - one frame in, one frame out is still followed but with a time lag of 1 frame
3) in the third step the decoder now gets the first B-frame and he already has the needed I and P-frame handy and therefore is able to decode the B
4) as he has the I and P already, he now is also able to decode the second B
5) he displays the P

the hack here is that, as avi/vfw is forced to the "one frame in, one frame out" sheme, you create a lag, which leads to that the sheme is still followed, but the decoder does not output the frame he gets, but the frame before
in contrary to packed bitstream, the stream gets written as the mpeg-4 standard defines it by using this way

hackery when encoding

these are the two hacks which can make it possible to use b-frames in the outdated VFW and AVI on the decoding side
still on the encoding side there is also a hack (as there also is the "one frame in, one frame out" sheme to follow of course):
for encoding a B-frame you need to feed the encoder with two frames:

now the encoder does the following:
1) you feed the encoder with the first frame -> coded as I-frame
2) input second frame -> should get coded as B-frame => not possible as you need a P-Frame too for creating a B
3) third input -> B => also not possible as the P was not there till now
4) fourth input -> coded as P-Frame
5) (now as the I and P are available) => first B gets written
6) (now as the I and P are available) => second B gets written

now as we know AVI and VFW is forced to follow the "one frame in, one frame out" sheme, so what do you think it writes in step 2) and 3) ?
as it has to write something (one in -> one out), it writes so called delay frames (1-byte 0x7f frames)
these delay frames are simply written as VFW simply has to write something! BUT they are not there in the input AND they break the compliancy to the mpeg-4 standard!

so what can we do?
what we always do if AVI and VFW cause troubles -> we hack a workaround:
=> virtualdub(mod) will drop these "delay frames" during encoding, to make it sure that there are no delay frames in the final output stream
BUT the chance is high that other tools able to encode using vfw codecs, will not remove these delay frames, so beaware


AVI and VFW is nice, right?

[Hiro2k]

That's all good and well known (if you can search that is!).

But the better question is, why are we still encoding with VFW/AVI programs? Why doesn't virtualdub(mod) output directly to .mp4 or mkv? Where are the encoding tools that will output native MVK and MP4 files?

[Wilbert]

C'mon you know answers to these questions, so why ask them?

But the better question is, why are we still encoding with VFW/AVI programs?

I didn't know there are hardware players which play MP4+AC3 or MKV+AC3. I didn't know you can mux AC3 in MP4.

Why doesn't virtualdub(mod) output directly to .mp4 or mkv?

Because the dev. didn't program that (I'm not sure whether they have plans for this). Saidly, the dev. of VDubMod seems being stopped.

Where are the encoding tools that will output native MVK and MP4 files?

These will come some day.

About bond's problem. Let me ask the following question to put it into perspective: Suppose we do 2-CD rips (because we want to have the AC3 stream), why do we need b-frames at all?

[P0l1m0rph1c]

Originally posted by Hiro2k
That's all good and well known (if you can search that is!).

But the better question is, why are we still encoding with VFW/AVI programs? Why doesn't virtualdub(mod) output directly to .mp4 or mkv? Where are the encoding tools that will output native MVK and MP4 files?

Well, the reason is, because noone coded something else yet. But i'm sure you are aware of the TCME project, and also of the progress of Gstreamer (now on win32 too). These should be a start point for a new, better framework.

[stephanV]

Originally posted by Hiro2k
But the better question is, why are we still encoding with VFW/AVI programs? Why doesn't virtualdub(mod) output directly to .mp4 or mkv?

Via what API? Here's how Avery Lee feels about DirectShow:

I have done a moderate amount of DirectShow programming, and don't like the API for several reasons. It is fragile and easily broken by bad third-party filters, the documentation is terrible, it is a sea of multithreading race and deadlock problems, and it is much less suited to editing than to playback. I can write a program to pull video frames from an AVI file using VFW AVIFile in about 10 minutes in about a page of code. It is more complex to do so using DirectShow. The situation is even worse for the inverse operation, trying to write raw data to a video file.

Much of the DirectX API is useful because it provides accelerated and featureful support for operations that require kernel support and were not previously available, but the vast majority of functionality in DirectShow is userspace-driven and is thus replicable in custom application code.

Since he has created VirtualDub and thus also has had the biggest hand in VirtualDubMod you can basically forget about it.

Wilbert:
I didn't know there are hardware players which play MP4+AC3 or MKV+AC3. I didn't know you can mux AC3 in MP4.

Theoratically it is posible with private streams. But IIUC such streams will be ignored by most MP4-splitters... video without sound, how nice! (of course there is AAC or something, but that too goes in AVI)

These will come some day.

Bond
AVI and VFW is nice, right?

As it seems it easier to hack AVI/VFW then making use of an already existing API like DirectShow, well yes!

And what alternative do we have anyway? (and do we really, really need one yet?)

EDIT: uhmmmm, nevermind

[bond]

lol, funny in what direction things evolve

this thread wasnt meant as a "what could be done" discussion, but simply as a description on "how its done now"

beaware, no flamewars!

[stephanV]

Originally posted by bond
this thread wasnt meant as a "what could be done" discussion, but simply as a description on "how its done now"

I thought it was one more of your attempts to shows us all how much AVI/VFW "sucks" as this is info is pretty well known or otherwise accesible with some searching (as Hiro2k said). Your last comment about avi and vfw at least suggested so to me.

Packed bitstream is not so bad at all:
a. It works fine in most cases (but show me one thing that will always work; software is buggy by nature). Don't bother with compliancy... most people won't care.
b. It can be removed (e.g. 3ivx-muxer)
c. It allows us to actually use B-frames (or perhaps it doesnt really matter anyway... see your ad b on decoding hacks)

beaware, no flamewars!

Then don't post such explanations with subjective and suggestive comments like 'outdated technology' and 'AVI and VFW is nice, right?'. A nice and dry explanation of the facts would do just as fine.

If you throw out a hook, you'll catch some fish sooner or later.

[bond]

thats what i mean

[stephanV]

well yes, what do you mean?

[Asmodeus]

Originally posted by Hiro2k
Why doesn't virtualdubmod output directly to mkv?

It doesn't Oh my God, my MKV does not egzists

[stephanV]

no it doesnt; it uses the the VFW framework for encoding so the video part of your mkv-file could just as well be a avi-file.

what VDM adds to mkv muxing is e.g. Vorbis sound, chapters and srt/ssa-subtitles of which the subtitles go in AVI too but is just not supported by VDM.

[Asmodeus]

But it outputs mkv so what's the difference

I post - you edit

[stephanV]

yeah, edit-button is my favorite one

anyway, to make things perhaps somewhat clearer: VirtualDubMod in principle only edits/outputs AVI-files and encodes via VFW. However there are a few things that VFW/AVI cannot do that Matroska/DirectShow can do (e.g. variable frame rate). So like I said, the video part of your matroska file is just limited to what AVI and VFW can do. I guess your file is muxed via VFW-compatability mode in Matroska as it doesnt have native MPEG4 support yet...

[Hiro2k]

Of course I know the answer to these questions, it posted it to point out how our current software is lacking in support of these new containers which have been around for more than a year now.

I would also like to point out as bond said no flame wars! There is a difference between an obvious flame and a discusion. The links your provided were useful to to the people who didn't know such things. I know Virutal Dub won't support DSHOW because of Avery´s direction with the project.

Edit:
I was about to respond to Asmodeus's question, but I see that stephanV has beeten me!

That's the reason I think we should change to a better API, there are alot things that could be done much easier and better than with VFW and is supported by the other containers (Variable Framerate to name one).

Edit 2: I see me and Steven were on the same wave length with vfr. lol

[Wilbert]

I think you misunderstood my point (and stephanV's). I was not complaining about this:

Of course I know the answer to these questions, it posted it to point out how our current software is lacking in support of these new containers which have been around for more than a year now.

Of course, this is important to know. Of course, it would be nice if there would be support.

I was complaining about this:

But the better question is, why are we still encoding with VFW/AVI programs?

and this

- what all has to be done for keeping outdated .avi and vfw alive

Suppose you are not a Start Trek fan, nor are you interested in 1-CD rips. What is wrong with avi-vfw encoding (without b-frames) in this case? Is the quality not good?

[stephanV]

Originally posted by Hiro2k
I would also like to point out as bond said no flame wars! There is a difference between an obvious flame and a discusion.

Then where did i flame? bond was obviously (at least for me) trying to diss AVI (again) for no apparant reason; I just said something about it. I'm sorry if that is considered flaming, but you can count on me flaming some more than in the future. (I usually back stuff up with links though).

That's the reason I think we should change to a better API, there are alot things that could be done much easier and better than with VFW and is supported by the other containers (Variable Framerate to name one).

That rises the question is: do we really need that? VFR for example:

a. IMHO on the encoding side it is almost useless and could do more wrong than good
b. it would be very useful for hybrid content (thats not really an opinion, thats almost a fact )

so for a. the answer is 'no', for b. it is 'yes'.

I couldn't agree more with Wilbert: why use all those "modern" technologies if you have no real use for them. What is really wrong with sticking with AVI? Why shouldn't we cap our video in AVI? huffyuv/MJPEG and uncompressed audio work fine right?

I just had to help a guy who had read a guide that let him put XviD and AC3 in matroska via mkvmerge. Admittedly he had misread the guide and didnt know he had to install the matroska pack but this had easily could be done in AVI via AVIMux (and even then matroska would still be an option). It would be nice to have a VirtualDub- and AVIMux-like tools for a better API but the fact remains that this just isnt the case. Complexity has it weaknesses too and one of them is the underdevelopment in tools we are seeing now. AFAIAC i could still hold out with AVI/VFW in most cases. I only use matroska because of the smaller overhead.

To sum up things: the factual review of how b-frames work in AVI is ok, but there is no need to call AVI outdated at all. But if saying that is flaming, well... i don't really care actually.

Remember that you are mods and your opinions are highly valued along many on this board. Be careful before you say one technology is superior to another.

[bond]

Originally posted by stephanV
bond was obviously (at least for me) trying to diss AVI (again) for no apparant reason

striked for ongoing insults against my person

if you cant stand me summarizing the hacks/workarounds needed for using b-frames in avi/vfw and feel insulted because of this, leading to insulting other people and accusing them of whatever comes to your mind, than you better should learn how to deal with your anger and defeat your obsession (taken from your sig) of insulting people, you dont like for whatever reason

thread closed, as it indeed evolved to a flamewar and i dont see in what productive way the discussion should evolve to