d2v file creation on linux

[d'Oursse]

Hello,

avisynth 3.0 is working a bit on linux. And, now, I would like to test mpeg files. They are not very well supported by gstreamer, or at least, that requires a big big code.

So, one solution is to use .d2v files and the mpeg2dec plugin. Problem : there is no generator of d2v files on linux.

some times ago, Neuron gave me some (unofficial) spec of the d2v files. So I tried to look a bit at what I can do with libmpeg2 (http://libmpeg2.sourceforge.net/)

Result: Nothing... I don't understand how I can use this lib to get what i have to write in the .d2v file...

So, does anybody has some informations about what to do ? Some code ? Or anything else ?

Thank you

PS: I've posted that thread here because I think that Nic or Neuron, who have both worked a lot on dvd2avi, read mostly this forum. But if it has to go to the dev forum, move it there.

PS2: please, don't tell me : "read dvd2avi source code", please don't :D

[Guest]

please, don't tell me : "read dvd2avi source code", please don't OK. Please have a look at the DGIndex source code.

[Leak]

OK. Please have a look at the DGIndex source code.
:D

Anyhow, doesn't the Windows binary work with WINE? That would at least be a workaround until someone ports it to Linux who's not afraid of the source... ;)

(Porting DGDecode probably is easier, as it doesn't try to put anything on screen by itself...)

np: Stuntman5 - Sunbathing At The Bus Stop (Melted Fake Plastic)

[d'Oursse]

it's not a good solution
If I'm asking, it's because i don't want to use wine ;)

[Guest]

I don't see the big deal. You have to parse the MPEG stream and record some stuff. The DGIndex code obviously does that. What's the problem with adapting that code?

[d'Oursse]

the problem is that i've never touched anything about video coding. I mean, deep video coding. I don't know the spec of mpeg1/2, i've never coded an encoder nor a decoder or a mpeg parser, etc...
But i'm looking at your code
thank you

[Guest]

I see. That would complicate understanding/porting anybody's code. I was afraid you were going to launch into a "DVD2AVI code is a spaghetti bowl" rant. :)

The MPEG2 spec is available in the library at my web site.

If you have any specific questions about the code, the spec, or the indexing operation, I'll be happy to assist you.

[d'Oursse]

neuron2: in initial_parse.c, function determine_stream_type, what it the interest of EOF_reached ? I don't see any: it is only set (to 1) and after determine_stream_type is finished, it is set to 0.

[Guest]

It's left over from another of my applications, from which I ported the code. You can ignore it.

[d'Oursse]

ok
now, in pack_parser, I sometimes see _read (file, &buffer, ***). I don't see the point in taking a reference to buffer. Isn't the aim of pack_parser filling buffer ?

[Guest]

ok
now, in pack_parser, I sometimes see _read (file, &buffer, ***). I don't see the point in taking a reference to buffer. Isn't the aim of pack_parser filling buffer ? The & operator is not a dereference operation. The following all evaluate to the same vaue:

buffer
&buffer
&buffer[0]

*Why* does it say &buffer? Probably because I cloned the line when coding from another line with &val and then I just changed val to buffer. I'll clean it up to avoid possible confusion.

[d'Oursse]

The & operator is not a dereference operation. The following all evaluate to the same vaue:

buffer
&buffer
i didn't know that. Is it standard (C89 or C99) notation ? It seems very strange to me. I know that the standard allow buffer as notation for &buffer[0], but that's all. For me, buffer is of type unsigned char *, so &buffer is of type unsigned char **.

Maybe it is a microsoft (tm) convention.

[Guest]

It's standard C language. buffer is an array, not a char pointer, so buffer is an lvalue referring to the array of characters. The result of the unary & operator is a pointer to the object referred to by the lvalue. Thus, &buffer means a pointer to the array of chars, i.e., the address of its first element.

The simple way to say it is that & is the "address of" operator and so &buffer is the address of the buffer, i.e., the address of its first element.

If buffer was declared as a char pointer, then &buffer would have a different meaning:

char *buffer;
buffer = (char *) malloc(256000);
// Here &buffer is not the address of the
// first element of the malloc'ed space.

It's not uncommon to see code showing the &buffer notation. Here's one example:

http://www.securitydocs.com/library/3222

[d'Oursse]

i don't think that buffer is an lvalue. buffer = ... is not valid
but i have found that: http://www.eskimo.com/~scs/C-faq/q6.12.html
edit: also look at 6.7

[Guest]

i don't think that buffer is an lvalue. Of course it is! And of course you can't say buffer =. There are modifiable and unmodifiable lvalues. See the very bottom of this page:

http://www.cepba.upc.es/docs/sgi_doc/SGI_Developer/books/CLanguageRef/sgi_html/ch04.html

[d'Oursse]

http://www.eskimo.com/~scs/C-faq/q6.7.html
(ansi C)

[Guest]

See the link I added in my edit. An array name is an unmodifiable lvalue.

Also, to address your notion that &buffer generates a char **, see 6.3, where it points out the exceptions to the rule:

"An lvalue of type array-of-T which appears in an expression decays (with three exceptions) into a pointer to its first element; the type of the resultant pointer is pointer-to-T.

(The exceptions are when the array is the operand of a sizeof or & operator, or is a string literal initializer for a character array.) "

Anyway, the point is that in C buffer and &buffer evaluate to the same value.

[d'Oursse]

neuron2: for the creation of the .d2v file, only initial_parse.c and mpeg2dec.c are needed, right ?

[Guest]

No, that's not right. initial_parse() is used just once at the first file open to get the stream and MPEG type. Thereafter, getbit.c does the stream parsing. You'll need gethdr.c also.

[d'Oursse]

arg, so much code... thank you (about mpeg2dec.c, it contains at least some code that writes the first lines of the d2v file)

edit : here is a first attempt for getting the stream and mpeg types http://www.iecn.u-nancy.fr/~torri/files/libd2v/
it seems to work for a vob file. I have no other mpeg files. Can you can test it with other mpeg files ?

[Guest]

Sorry, I don't have any Linux machines.

Yeah, there's a lot of code. But there'll be less after you strip it down. :)

[d'Oursse]

I've put the Makefile just for linux users. You just have to compile 3 files. It should not be a big deal.

Well, i try to understand the design (not how to parse) :) and, it's not easy. There are global variables everywhere, that are needed in several files. It's not easy to make a lib from that

[Guest]

Umm, bz2? Do I have to install yet another ZIP utility?

[d'Oursse]

what are Start_Flag and Stop_Flag used for ? More precisely, is it for the gui only ?

[GaveUp]

Umm, bz2? Do I have to install yet another ZIP utility?

bz2 is pretty common (along with gz) in unix environments. WinRAR can handle both formats. It can also handle tar as that's how you typically find archives (.tar.gz, .tar.bz2, .tgz, or .tbz2).

[Leak]

Sorry, I don't have any Linux machines.
If you have a windows machine, there's always coLinux (http://www.colinux.org/) - it'll run a (coLinux-patched) Linux kernel as a Win32 process, with a file system in an image file (with several pre-installed distribution images to choose from) and an emulated network card; it's what I use if I'm in need a Linux system, since I don't want another box standing around.

It's quite handy, and if you either run an X server on windows or a vnc server on the Linux side it'll run X apps, too - and you get all that without the virtualization hoops things like VMWare have to jump through.

Just my 0.02€...

np: Autechre - V-Proc (Advanced Listening Opportunity)

[d'Oursse]

powerarchiver, too
i can use gzip to get a tar.gz
strange that you don't know bzip2. It compress quite more than the zip algo.

[GaveUp]

powerarchiver, too
i can use gzip to get a tar.gz
strange that you don't know bzip2. It compress quite more than the zip algo.

unless you're on a nix environment you almost never encounter bzip2. It's either rar or zip, and frankly rar typically compresses better than just about anything else.

[fccHandler]

Bzip2 is what Avery Lee uses to compress his VirtualDub source code, so I've always had it. I don't recall where I got it, but the one I have is a single 72K command-line executable. It doesn't install anything or use the registry, it just runs. If you like, I can post it.

[Guest]

what are Start_Flag and Stop_Flag used for ? More precisely, is it for the gui only ? They are used for more than the GUI. Start_Flag is an important part of the audio delay calculation code. Stop_Flag is important for gracefully terminating at EOF.

[d'Oursse]

fccHandler : http://gnuwin32.sourceforge.net/packages/bzip2.htm

neuron2: ok, thank you

[d'Oursse]

donald: in getbit.c, line 1488, what is the order of the tests ? I think that it's :

** || ( ** && **)

am i wrong ?

[Guest]

Yes, that's right. && has higher precedence than ||.

http://www.difranco.net/cop2220/op-prec.htm

[d'Oursse]

ok. thank you. I prefer putting some parenthesis, it's clearer.

Also, I think that I need:
1) AC3Dec code
2) wavefs44.c
3) WriteD2VLine in getpic.c

Anything else in getpic.c ?

Also, is it better to use liba52 (http://liba52.sourceforge.net/) instead of the AC3 code in AC3Dec ?

[Guest]

You need the AC3 code only if you need to decode to WAV. I can't advise you on which AC3 decoder to prefer.

ThreadKill() has code to flush out the last line of the D2V file.

[d'Oursse]

in getbits, in Next_Transport_Packet, This_Track is declared and used in DEMUX_AC3 , but its value is never set. Is there a problem with that ?

[Guest]

in getbits, in Next_Transport_Packet, This_Track is declared and used in DEMUX_AC3 , but its value is never set. Is there a problem with that ? If that were true, it would be a problem. Didn't you notice the initializer:

static unsigned int code, flags, VOBCELL_Count, This_Track = 0, MPA_Track;

[d'Oursse]

i didn't noticed it because I have no such initializer in the code i have (http://neuron2.net/dgmpgdec/dgmpgdec140src.zip)

[Guest]

i didn't noticed it because I have no such initializer in the code i have (http://neuron2.net/dgmpgdec/dgmpgdec140src.zip) It's declared static, so it gets initialized to 0 by default. I added the explicit initializer in the 1.4.1 source to avoid a compiler warning.

[Bidoche]

It's declared static, so it gets initialized to 0 by default. warning.Is it 0 because the standard says so, or because your compiler does it ?

[Guest]

Standard.

[Bidoche]

Ok, I didn't knew that one.

[Cyberia]

I realize that this thread is about getting DGIndex to run on Linux, but it also sounds like there is a large scale code cleanup going on. True? Would this be rolled back into the Windows build?

[d'Oursse]

Cyberia: I try to write a library that will parse a mpeg file and write a .d2v file. That's all. No gui, no decoding. I try to write it completely in C89 except the 64 bit types. So it should compile on every system, i think.

I try to clean up the code, and so, I have made some modifications. If neuron wants to use it, then the gui part and the decode part has to be rewritten a bit, i think. Well, maybe I'll try to do that, but only after i (or other people, don't hesitate to help ;)) finish the mpegsource for avisynth 3.0.

neuron: is the source code of dgindex 1.4.1 released somewhere ? I can't find it, and I would like to see the modifications with respect to dgindex 1.4.0. Thank you.

[Guest]

neuron: is the source code of dgindex 1.4.1 released somewhere ? I can't find it, and I would like to see the modifications with respect to dgindex 1.4.0. Thank you. 141b5 source is linked in the main thread. But things are changing very rapidly right now, I warn you. It's not a good time to take a stable snapshot.

[d'Oursse]

neuron2: could you explain me what the variable 'process' if for ?

[Guest]

I would have to write a sizable essay to answer that. Suffice it to say it controls the processing when a play, preview, or save project operation is performed, e.g., it stores the selection points so that the range of the project can be limited. The process variable and all that goes with it is the poorest part of the DVD2AVI legacy. One day I will re-engineer it.

[d'Oursse]

in global.h, MuxFile is declared as an int. It should be *FILE, shouldn't it ?
If yes, then, there are some tests in getbits.c to change

[Guest]

What version are you looking at? It's been FILE *MuxFile for donkey's years.

[d'Oursse]

maybe an old one. I have no possibility to get a newer one for the moment
when do you think that the 1.4.4 version will be final ?

[Guest]

Soon.

[d'Oursse]

about MuxFile, there are some strange tests on it (MuxFile > 0 and MuxFile != (struct _iobuf *) 0xffffffff ). What do you want to do with these tests ? I think that the first one should be != NULL. But the second one ?

Also, in getbit.c, there are sometimes pcm[0] and pcm[This_Track] in Next_Packet (in getbit.c, in Next_Packet). Is it normal ?

[Guest]

MuxFile == 0xffffffff means demuxing is disabled.

MuxFile == 0 is set to enable demuxing but the file handle hasn't yet been opened.

MuxFile > 0 && != 0xffffffff is the valid opened file handle when demuxing.

Yes, as noted in the changes listing for 1.4.4 rc1, Decode AC3 to WAV was broken for tracks 2-8. It has been fixed. All the pcm[0]'s should have been pcm[This_Track].

Version 1.4.4 will be released very soon, at which point you can get the new source code.

[d'Oursse]

ok, thank you. Nevertheless, would it be possible to cast with (FILE *) and not (struct _iobuf *) ? there's a compiler warning with struct _iobuf

right now, i have something that compiles and links fine. I'm trying to write a CLI with that library

[Guest]

Nevertheless, would it be possible to cast with (FILE *) and not (struct _iobuf *)? Sure. I can't remember why I did it that way.

[d'Oursse]

stupid question : there's no main function in the dgindex code ?

a related question : where can I get the general structure of the creation of the d2v file ? (the functions to call, in which order, etc...) I see that there are some in mpeg2dec.c, in the creation of the main window. Is there something elssewhere ?

[Guest]

stupid question : there's no main function in the dgindex code ? All Windows programs begin at WinMain(). Leaving the CLI aside, it just sets up the main window and gets the message loop going. After that, everything is triggered by GUI controls sending messages to the message loop, WndProc().

a related question : where can I get the general structure of the creation of the d2v file ? (the functions to call, in which order, etc...) I see that there are some in mpeg2dec.c, in the creation of the main window. Is there something elsewhere ? I'll give a rough overview:

The user selects menu option Save Project.

WndProc() receives a message at case IDM_SAVE_D2V. The D2V file is opened. The D2V_Flag is set to true. process.locate is set to LOCATE_RIP. A thread is started to run MPEG2Dec. The processing for this GUI message is completed.

The MPEG2Dec thread starts. It first sets up the process structure. Check_Flag is already set from the initial loading of the file, so that big chunk of code conditional on !Check_Flag is skipped. The header and settings sections of the D2V file are now written. The MPEG decoding loop now is entered. Get_Hdr() and Decode_Picture() are called for each picture.

In each call to Get_Hdr(), for each I picture, the file position is calculated and stored in d2v_current.position in the function picture_header(). For all pictures, the TFF/RFF flags and other data are also stored. In Decode_Picture(), when an I picture is seen (except the first one), the previously stored data is written out to the D2V file by calling WriteD2VLine() [or WriteGopLine() in earlier versions]. WriteD2VLine() reorders the information into display order and writes a complete line in the D2V file.

Note that in Decode_Picture(), when D2V_Flag is set, the picture is not actually decoded by calling picture_data(), because there is no need to actually decode the picture. Don't ask me why the original author put the indexing functionality in Decode_Picture()! It doesn't really matter.

At the end of the stream, ThreadKill() is called. This function flushes out the last D2V line, writes the trailer at the end giving FILM percent, and closes the file.

I haven't described the multilayered parsing system that supplies stream data to the decoder. That would require several hours to document properly.

[d'Oursse]

ok, thank you very much

And, I remember another question, that I forgot several days ago :
ptsdifference is always returning false (the cases where it returns true are commented) Is it normal ? If it really returns always false, as it is always called in tests, one can remove it.

[Guest]

Originally I used the return value. If the delay was too large, I didn't put it in the filename. But users wanted to see it in any case, so I commented out the bits needed so as to always include the delay value, no matter how big it is. You can't remove the calls to PTSDifference(), however, because that function writes the variable PTSDiff (it receives a pointer to PTSDiff via a function parameter).

[d'Oursse]

indeed. So, a call to PTSDifference is sufficient, right ? (without the if block)

Also, what's the difference between Frame_Rate and frame_rate (i want to give htem better names)

And is it necessary to create another thread for mpeg2dec ? Is it necessary for windows ? I don't see the need of it

[Guest]

indeed. So, a call to PTSDifference is sufficient, right ? (without the if block) Right.

Also, what's the difference between Frame_Rate and frame_rate (i want to give htem better names) The lower-case one is the actual value read from the MPEG stream. The upper case one may be different if Force Film is enabled:

Frame_Rate = (FO_Flag==FO_FILM) ? frame_rate * 0.8f : frame_rate;

And is it necessary to create another thread for mpeg2dec ? Is it necessary for windows? I don't see the need of it. The main window process must remain responsive to messages, so two threads are required.

[d'Oursse]

are the pat.[h/c] used for the creation of the d2v file ?

[Guest]

are the pat.[h/c] used for the creation of the d2v file ? Not directly. It's used by the "Detect PIDs" menu option to determine the transport PIDs. If you are not demuxing/decoding audio and you already know the video PID, or you require the user to determine and enter it some other way, then pat.* is not needed. But if you are demuxing or decoding audio, you'll need them to determine the audio type for transport streams, uness, again, you require the user to determine and enter that some other way.

[d'Oursse]

As, Donald does not want to talk to me anymore (because of a small sentence that he has misunderstod and upset him, which i really don't understand why, but it's his problem), I will not be able to finish that library in the little spare time that I have and without his help. I'll not work anymore on it.

regards

[Malphas]

That's unfortunate and disappointing.

[Guest]

I stand ready to answer any questions posed to me by anybody regarding DGIndex, so please don't try to make me a whipping boy. I never had anything to do with 'libd2v' and whether it ever does or does not exist.