Divx/Xvid frame structure (or should I say bitstream?)

[Movie Maniac®]

Hi

I'm looking around to find some informations about how each frame of an Xvid/Divx encoded movie is structured.

I'd like to know what each byte (or bit) means, becouse I'd like to retrieve some important infos from there (such as encoding options like GMC or Qpel).

Can anybody help me in finding these informations?

Please, I've searched the forum and google without any success.

Bye

P.S. any link is welcome. I don't know where to look, if anybody could point his finger to some places, I'd be very happy to have a look.

[esby]

Frame structure is part of the format,
and not of the codec used.

If you have an avi file, the frames will be structured
according the avi spec.
And if you take another container, it will probably change.

If you are asking about the internal (or supposed internal?)
frame structure, that use the codec to decode the frame,
I cannot answer you...

Some encoding options might be found by analyis the stream structure
(eg if there are no B-frames, B-frames were not used :D )

the avi header only contain the infos that have been defined by the avi container definition.

I guess the encoder could add special tag and extra infos in the streams, but since it's not needed for playback, it would be a waste of space...
Better to do it separately, but then, if you don't have the file,
you are still at the same point than before :rolleyes:

esby

[Movie Maniac®]

Originally posted by esby
Frame structure is part of the format,
and not of the codec used.

If you have an avi file, the frames will be structured
according the avi spec. Some encoding options might be found by analyis the stream structure
(eg if there are no B-frames, B-frames were not used :D )

the avi header only contain the infos that have been defined by the avi container definition.


Yes I know this. I've already made an Avi detector that works like Gspot :D

Originally posted by esby
If you are asking about the internal (or supposed internal?)
frame structure, that use the codec to decode the frame,
I cannot answer you...

Yes, I'm searching for this informations...
I mean, I suppose that the encoder puts some flags into each frame, to let the decoder know what to do... I mean for example in case you use GMC or Qpel when compressing...

For example, seems that DRF nalyzer can read some of this things... but I can't really understand the sourcecode since I can't understand C++. I only know some VB6 and php...

However, thanks for youur interest in solving my problem :D

Bye

P.S. hope to receive an help...

[bond]

ffdshow for example can detect if gmc, qpel is used, have a look at it

[Movie Maniac®]

Originally posted by bond
ffdshow for example can detect if gmc, qpel is used, have a look at it

Yes I know...

but what I need is not a tool that can detect those things... I'd like to know how to detect them becouse I want to implement some functions onto my avi detector to show more detailed informations...

That's way I asked :(

Thx

[bond]

well i meant you should have a look how ffdshow detects that stuff

[Movie Maniac®]

Originally posted by bond
well i meant you should have a look how ffdshow detects that stuff

I said I can't understand a single line of code written in C++ :(
I've tried doing this, and I'll keep trying... I was just looking for a little help :(

Anihow
thx for your support

Best Regards

[Sulik]

You're going to have a tough time getting anywhere in this field if you don't understand C++, so you might want to take a look at that first - it's really not that hard.

Weither the file is avi, mp4, mov or anything else, it doesn't really matter much for finding out the MPEG-4 compression headers (just look for the proper start codes).
All of them contain a MPEG-4 elementary video stream, with headers that conform to the MPEG-4 spec (ISO 14496-2).
Short of getting the spec, the next best thing would be to look at existing code (ffdshow or xvid), and look for functions that parse the headers (especially the video object layer header).

[Movie Maniac®]

well

it seems I should learn some C++ programming. I think that hard times are coming for me :(

Well

let me ask a last question before starting.
I went to ffdshow homepage on sourceforge, just to download the source code. But the latest version I could find is dated 2002-06-17.
Is it possible?
And mostly, does it contains what I'm searching for?

I've also downloaded Divx DRFAnalyzer which contains a class called Mpeg4HeaderParser, and it seems to have something to do with what I'm looking for..

Then I downloaded Xvid source code, but it's really to big for me :'(
I don't know where to look in it...

Thanks for your support

[sh0dan]

Originally posted by Movie Maniac®
Then I downloaded Xvid source code, but it's really to big for me :'(
I don't know where to look in it...


My guess would be to start in src/bitstream/*.* - of course starting with "bitstream.c". There are functions for both reading and writing bitstreams.

I'm afraid that is how complex it is - you probably don't need much of the code, since you are only detecting things - not actually decoding it.

[Nic]

Looking at the DRFAnalyzer source code...you are far better using that source code to try and create your prog than using XviD code.

Depending on how good your C++ skills become, trace through the code to get to a VIDOBJLAY_START_CODE (CMPEG4HeaderParser::parseStartCodeVIDOBJLAY) from there you can modify it to read the bits you need.

The Video Object Layer contains elements such as:
aspect_ratio_infp
quarter_sample

GMC info might be more tricky (not sure how that fits in exactly off the top of my head, look up SPRITE_GMC in DRFAnalyzer)...it would help if you got a copy of the MPEG-4 standard...otherwise you won't know what things to parse in parseStartCodeVIDOBJLAY.

Good luck, I'm afraid you might need it ;)

-Nic

ps
Im pretty sure you're going to need to understand C/C++ to do this. Unless you try and do it all from the MPEG-4 spec. Which is easy enough, but the MPEG-4 spec is more tricky to understand than C ;)

[Movie Maniac®]

Originally posted by Nic
Looking at the DRFAnalyzer source code...you are far better using that source code to try and create your prog than using XviD code.

Depending on how good your C++ skills become...
Good luck, I'm afraid you might need it ;)

-Nic

ps
Im pretty sure you're going to need to understand C/C++ to do this. Unless you try and do it all from the MPEG-4 spec. Which is easy enough, but the MPEG-4 spec is more tricky to understand than C ;)

Ok Ok
I've understood. I must practice some c++ before trying to do this... unfortunately I've got to study hard in order to take my graduation before the end of this year. I don't want to grow old before takin' it...

So, it seems that it can wait a bit...
anyhow, I'll let you know any future developments, mostly becouse it seems that I'm going to need your help in the future :D

bye and thx for your support

[halsboss]

I'm looking for the same thing (VB6 orientation, ie it's all I have available). ... how did you go, and do you have any VB6 code?)

[esby]

Any particular reason for using VB6 ?
(I mean 4 years ago, I would have understood, but now...)

esby

[halsboss]

:) no good ones. Only that I'm a dinosaur and it's the only compiler I have.

My objective is to implement sub-functionality of MPEG4Modifier (a C# .net program) ie --unpack and --pack with --always-write, in a non .NET windows environment.

Suggestions welcomed :)

[LoRd_MuldeR]

My objective is to implement sub-functionality of MPEG4Modifier (a C# .net program) ie --unpack and --pack with --always-write, in a non .NET windows environment.

Avidemux can do that and it doesn't require the .NET Framework ;)

[halsboss]

Avidemux can do AVI --unpack --always-write and also --pack --always-write ? YIPPEE ! Thanks !

And there was me poring over mpeg4 spec after using VB6 to decode xvid AVI RIFF documents, trying to figure out what was in the actual data chunks (mpeg4 ASP elementary stream, I presume, and say MP3 audio).

Now, off to look into Avidemux.

Cheers !

PS just for my mental exercise, anyone know of any non-C code or *commented code* or pseudocode to unpack and repack the xvid data stream ?

[LoRd_MuldeR]

Avidemux can do AVI --unpack --always-write and also --pack --always-write ? YIPPEE ! Thanks !

And there was me poring over mpeg4 spec after using VB6 to decode xvid AVI RIFF documents, trying to figure out what was in the actual data chunks (mpeg4 ASP elementary stream, I presume, and say MP3 audio).

Now, off to look into Avidemux.

Cheers !

PS just for my mental exercise, anyone know of any non-C code or *commented code* or pseudocode to unpack and repack the xvid data stream ?

I don't know what "--always-write" is supposed to do, but "--force-unpack" should do what you want. Note that this option effects the next file.

For more info see the Wiki:
http://avidemux.org/admWiki/index.php?title=Command_line_usage

[halsboss]

Sweet. I'll start using that from now on. Cheers.

[halsboss]

Hi, I gather from the wiki that
--autoindex --load "%~1" --output-format AVI --save-unpacked-vop "%~1-unpacked.AVI" --quit will save it as unpacked regardless of the input.
and
--autoindex --load "%~1" --output-format AVI --save-packed-vop "%~1-unpacked.AVI" --quit will save it as packed regardless of the input.

But given I don't know whether the input is packed or unpacked, what does "--force-b-frame" and "--force-unpack" do ?
It only mentions detection, and I don't know if that means
"flag it, even if it isn't there" or "make sure to look for it".

--force-b-frame, Force detection of bframe in next loaded file (no args)
--force-unpack, Force detection of packed vop in next loaded file (no args)


edit: upon reflection, I'm guessing the right commands in a batch file you can drop an AVI onto (XP) are:
@echo on
%~d1
CD "%~dp1"
"C:\software\AVIdemux\avidemux2_cli.exe" --force-b-frame --force-unpack --load "%~1" --output-format AVI --autoindex --audio-map --rebuild-index --save-unpacked-vop "%~1-unpacked.AVI" --quit
REM ----------------------------------------------------------------------------------------------------
"C:\software\AVIdemux\avidemux2_cli.exe" --force-b-frame --force-unpack --load "%~1" --output-format AVI --autoindex --audio-map --rebuild-index --save-packed-vop "%~1-packed.AVI" --quit
REM ----------------------------------------------------------------------------------------------------
pause

[halsboss]

Hi, couple of questions re the CLI ...

This came up; is there a switch to force a time map rebuild as I currently have to type in Y and ENTER to continue.
********************************************************************************
* Question *
* Build Time Map *
* Avidemux detected VBR MP3 audio in this file. For keeping audio/video in sync, time map is needed. Build it now?

You can do it later with "Audio -> Build VBR Time Map". *
********************************************************************************
y
Encoding Video codec : Copy
Encoding Container : AVI
Encoding Audio codec : Copy
Encoding Phase : Saving
Done:97% Frames: 86620/88482 ETA: 00:00:00
The Commandline was
"C:\software\AVIdemux\avidemux2_cli.exe" --force-b-frame --force-unpack --load "%~1" --output-format AVI --autoindex --audio-map --rebuild-index --save-packed-vop "%~1-packed.AVI" --quit

And then these errors came up

scanning timeline
OVR: 732439326 rel:0 lentogo:0 blocklen 2376Grabbed :2624
**PKTZ:READ ERROR
**END OF AUDIO STREAM
MapVBR:Get packet failed
Nb entries in timeline : 147457
checking for B-Frames...
scanning 125 frames
[lavc] Invalid and inefficient vfw-avi packed B frames detected
* Frame 5 is A B frame, flag not ok
* Frame 7 is A B frame, flag not ok
* Frame 9 is A B frame, flag not ok
...
Env override 1 used
Forcing Bframe present and incorrect
[Editor] Decoder says it is vop packed

Mmm this appear to have b-frame...

But the index is not up to date
[Editor] Decoder says it is vop packed
Env override 2 used
Trying to unpack the stream
WARNING*************** Missing one NVOP, dropping one b frame instead at image 6083
WARNING*************** Missing one NVOP, dropping one b frame instead at image 15322
WARNING*************** Missing one NVOP, dropping one b frame instead at image 28401
WARNING*************** Missing one NVOP, dropping one b frame instead at image 38552
WARNING*************** Missing one NVOP, dropping one b frame instead at image 72159
WARNING*************** Missing one NVOP, dropping one b frame instead at image 79106
WARNING*************** Missing one NVOP, dropping one b frame instead at image 86257

Which seems to indicate it dropped at least 7 frames. Any advice on what happened ?

edit: if this helps, MPEG4Modifier shows no errors and has this to say about the source: ---------- Video Info ----------
Packed bitstream: Yes
QPel: No
GMC: No
Interlaced: No
Aspect ratio: Square pixels
Quant type: H.263
FourCC: XVID
User data: DivX503b1393p
XviD0050

I-VOPs: 1179 (1.33%)
P-VOPs: 35285 (39.87%)
B-VOPs: 52026 (58.79%)
S-VOPs: 0 (0.00%)
N-VOPs: 0 (0.00%)

Max consecutive B-VOPs: 2
1 consec: 43.07%
2 consec: 56.93%
--------------------------------