Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > General > DVD2AVI / DGIndex

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th October 2006, 22:14   #1  |  Link
Daner
Registered User
 
Join Date: Dec 2004
Posts: 24
Parse D2V file syntax

I've searched around and have been unable to find a detailed explanation of how to interpret the file generated by the ParseD2V tool. Here's a sample excerpt:

Code:
[GOP]
61418 [B]: 76733,76733..........2 *
61419 [B]: 76734,76734,76735....3 *
61420 [I]: 76735,76736..........0 *
61421 [B]: 76736,76737,76737....1 *
61422 [B]: 76738,76738..........2 *
61423 [P]: 76739,76739,76740....3 *
61424 [B]: 76740,76741..........0 *
61425 [P]: 76741,76742,76742....1 *
61426 [P]: 76743,76743..........2 *
[GOP]
61427 [I]: 76744,76744..........2
61428 [B]: 76745,76745..........2
61429 [B]: 76746,76746..........2
61430 [P]: 76747,76747..........2
61431 [B]: 76748,76748..........2
61432 [B]: 76749,76749,76750....3 *
61433 [P]: 76750,76751..........0 *
[GOP]
61434 [B]: 76751,76752,76752....1 *
61435 [B]: 76753,76753..........2 *
61436 [I]: 76754,76754,76755....3 *
61437 [B]: 76755,76756..........0 *
61438 [B]: 76756,76757,76757....1 *
61439 [P]: 76758,76758..........2 *
61440 [B]: 76759,76759,76760....3 *
61441 [B]: 76760,76761..........0 *
61442 [P]: 76761,76762,76762....1 *
Now, I understand that [GOP] represents the start of a new group of pictures, I know what I/B/P frames are, and I understand that an asterisk at the end of a line indicates that the frame fits within a 3:2 pulldown pattern.

What I don't understand is the rest. Are the comma-delimited numbers field numbers? Why are there sometimes 3 for a single frame? Do the numbers at the beginning of each line represent the pulled-down frame number? What exactly do the numbers 0, 1, 2, and 3 at the end of each line indicate?
Daner is offline   Reply With Quote
Old 24th October 2006, 22:41   #2  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,923
I'll deconstruct it for you...

61418 [b]: 76733,76733..........2 *

This represents picture number 61418 of the coded MPEG stream (not the display frame numbering, which can be different due to pulldown). It is a B picture. Its fields appear together in display frame 76733. The 2 at the end is a bitmapped indication of the TFF/RFF flags. Bit 0 is RFF, bit 1 is TFF, so 2 = binary 10, which means this picture is TFF and is not pulled down. That is why there are only 2 display frame numbers shown.

61419 [b]: 76734,76734,76735....3 *

This represents picture number 61419. It is a B picture. Two of its fields appear in display frame 76734, and one (pulled down) in 76735. TFF=1 and RFF=1.
Guest is offline   Reply With Quote
Old 31st October 2006, 08:50   #3  |  Link
Daner
Registered User
 
Join Date: Dec 2004
Posts: 24
Thanks, Donald, for your reply. That certainly explains things for me.

Now this raises another question in my mind. In the case of a soft-telecined hybrid DVD source (that is, one which is not reported as 100% VIDEO by DGIndex, but rather contains mostly 3:2 pulled-down material with a few occasional breaks in the pulldown pattern), shouldn't it be possible to exactly reconstruct the proper progressive frame sequence by using this information?

Currently, it appears that DGIndex can either do all FORCE_FILM for the entire sequence, or no FORCE_FILM for the entire sequence, either of which will result in some combed frames when performed on a sequence that is not 100% FILM. Yet, the RFF and TFF flags tell you exactly how to translate between the display frames and the source pictures, so why can't DGIndex make use of this information to perform inverse telecine (FORCE_FILM) on the pulled down material, and a simple deinterlacing filter on the nonfilm material, and then output a timecode file to indicate where the 24fps vs. 30fps frame sequences are, to be used in a variable framerate MKV container?

Last edited by Daner; 3rd November 2006 at 23:39.
Daner is offline   Reply With Quote
Old 4th November 2006, 19:30   #4  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,923
You could cobble that together by doing this:

1. Use Ignore Pulldown Flags for your project.
2. Massage the Parse D2V log into a timecode file.

This will work only for cases where there are good 3:2 sections mixed with video sections. I will consider adding a function to generate the timecode file.
Guest is offline   Reply With Quote
Old 4th November 2006, 23:35   #5  |  Link
Daner
Registered User
 
Join Date: Dec 2004
Posts: 24
Quote:
Originally Posted by neuron2 View Post
1. Use Ignore Pulldown Flags for your project.
2. Massage the Parse D2V log into a timecode file.
I see. So at what point, and through which method, would the IVTC be done on the pulled-down material?
Daner is offline   Reply With Quote
Old 4th November 2006, 23:39   #6  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,923
Quote:
Originally Posted by Daner View Post
So at what point, and through which method, would the IVTC be done on the pulled-down material?
If you ignore the pulldown flags of a pure 3:2 soft telecined section, that is equivalent to a perfect IVTC. The video portions are unaffected. If you want them deinterlaced, you can use some deinterlacer that touches only combed frames, such as FieldDeinterlace(full=false). TDeint() can do that also. I don't see DGIndex as the right place to do deinterlacing.

Last edited by Guest; 5th November 2006 at 03:44.
Guest is offline   Reply With Quote
Old 4th November 2006, 23:46   #7  |  Link
Daner
Registered User
 
Join Date: Dec 2004
Posts: 24
Quote:
Originally Posted by neuron2 View Post
If you ignore the pulldown flags of a pure 3:2 soft telecined section, that is equivalent to a perfect IVTC.
Oh right, that makes sense!

Quote:
Originally Posted by neuron2 View Post
If you want them deinterlaced, you can use some deinterlacer that touches only combed frames, such as FieldDeinterlace(full=false). Deint() can do that also. I don't see DGIndex as the right place to do deinterlacing.
Well the reason I suggested doing it in the DGIndex stages rather than in the AviSynth stages is because DGIndex has access to the RFF and TFF flags, so it knows exactly where the nonfilm sequences are. FieldDeinterlace and Deint, on the other hand, can only rely on image analysis to detect combed frames, so their decisions will always be imperfect.
Daner is offline   Reply With Quote
Old 5th November 2006, 03:46   #8  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,923
The TFF/RFF flags are stored in the D2V file, so they are available. I believe TDeint() has a mode that can read the flags and deinterlace conditionally as you describe, IIRC. You could also use the known frame ranges (from the D2V file) to create an override file to FieldDeinterlace() or TDeint(). Finally, we can't assume that non-film must be interlaced, because it could be full rate progressive. So maybe it is better to use image analysis to determine if frames are actually combed.

Last edited by Guest; 5th November 2006 at 03:50.
Guest is offline   Reply With Quote
Old 11th January 2017, 04:05   #9  |  Link
Katie Boundary
Registered User
 
Katie Boundary's Avatar
 
Join Date: Jan 2015
Posts: 524
I just learned about this function and I am THRILLED. One question: I know what TFF and BFF are, but what the hell is "RFF"?

Quote:
Originally Posted by Daner View Post
Currently, it appears that DGIndex can either do all FORCE_FILM for the entire sequence, or no FORCE_FILM for the entire sequence, either of which will result in some combed frames when performed on a sequence that is not 100% FILM. Yet, the RFF and TFF flags tell you exactly how to translate between the display frames and the source pictures, so why can't DGIndex make use of this information to perform inverse telecine (FORCE_FILM) on the pulled down material, and a simple deinterlacing filter on the nonfilm material, and then output a timecode file to indicate where the 24fps vs. 30fps frame sequences are, to be used in a variable framerate MKV container?
Oh wow, you beat me to this idea by ten years! A fork of DGIndex that does exactly this (but bobbing to 60 fps instead of 30 fps) is going to be one of my first projects when I'm done learning C/C++

EDIT: I just learned something interesting. When parsing a .d2v file, the frames that are marked with * aren't guaranteed to actually be progressive. Correct switching between film and NTSC sections can only be done manually using the parse files as a rough guide.
__________________
If I ask "How do I do X?" or "what happens if I do X?", and X is a very bad thing that no one would ever normally do, assume that I already know this, and that I have Katie reasons for asking anyway.

Last edited by Katie Boundary; 11th January 2017 at 06:30.
Katie Boundary is offline   Reply With Quote
Old 12th January 2017, 12:15   #10  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 5,583
Quote:
what the hell is "RFF"?
I think you'll find that its "Repeat Field Flag".
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 12th January 2017, 13:42   #11  |  Link
videoh
Suspended for forum rule violations
 
Join Date: Jul 2014
Posts: 379
repeat first field
videoh is offline   Reply With Quote
Old 12th January 2017, 14:45   #12  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 5,583
I stand corrected.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 13th February 2017, 04:28   #13  |  Link
Katie Boundary
Registered User
 
Katie Boundary's Avatar
 
Join Date: Jan 2015
Posts: 524
Thanks. New question. In the "fix.txt" files like this...

Quote:
D2V Fix Output

Field order transition: 2 -> 0
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f2 f0 f1 e3 f3 f0 e1 f2 f3 e0 f1 f2 e3
corrected...
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f3 f0 f1 e3 f3 f0 e1 f2 f3 e0 f1 f2 e3

Field order transition: 3 -> 3
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f3 f0 f1 e3 f3 f0 e1 f2 f3 e0 f1 f2 e3
corrected...
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f3 f0 f1 e2 f3 f0 e1 f2 f3 e0 f1 f2 e3
...is any of this information useful for figuring out exactly where the field order transitions happen, so I can figure out whether or not they need to be fixed?
__________________
If I ask "How do I do X?" or "what happens if I do X?", and X is a very bad thing that no one would ever normally do, assume that I already know this, and that I have Katie reasons for asking anyway.
Katie Boundary is offline   Reply With Quote
Old 13th February 2017, 05:22   #14  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 5,583
The section in DGIndex User Manual outlining this is "D2V Format - Data Section ".

Dont think anything intended for human extraction, perhaps some tool exists but not aware of any.
Basically just gives file, and byte offset type stuff, nothing about frame numbers or clip time.

Usually (in my experience), field order transition occur when somebody has added some clip to beginning or end of
clip, eg black leader or trailer, or some kind of credits.
Only guide I have ever used is which file it belongs to (see above mentioned docs) and byte position (ie near end of beginning of file).
More often than not, I would just ignore them (especially at end) and let it fix the problem, or could try to estimate position (based on byte offset)
and investigate after a Separatefields.
Perhaps someone knows of some tool to scan d2v's to give a better frame estimate.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 17th February 2017, 02:17   #15  |  Link
Katie Boundary
Registered User
 
Katie Boundary's Avatar
 
Join Date: Jan 2015
Posts: 524
I figured it out 12 hours ago. The fix file is written in more or less the same format as the D2V files themselves. As per the dgindex instructions, each line roughly corresponds to one GOP, and the first seven information slots per line are for information about that GOP or its i-frame. Each two-character combination after that is a frame. I haven't yet figured out what all of the first characters mean, and I'm not 100% sure that they matter for the purpose of correcting illegal field order transitions. The second characters, however, are always in the range of 0-3, and they often occur in long repeating 0-1-2-3-0-1-2-3-0-1-2-3 patterns, which means that they're the TFF/RFF flags. So, when you have a fix file that says this...

Quote:
Field order transition: 2 -> 0
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f2 f0 f1 e3 f3 f0 e1 f2 f3 e0 f1 f2 e3
corrected...
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f3 f0 f1 e3 f3 f0 e1 f2 f3 e0 f1 f2 e3

Field order transition: 3 -> 3
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f3 f0 f1 e3 f3 f0 e1 f2 f3 e0 f1 f2 e3
corrected...
900 5 0 1086896128 0 3 4 73 70 d1 f2 f3 e0
900 5 0 1087176704 0 3 4 71 72 d2 f3 f0 e1 f2 f3 e0 f1 f3 f0 f1 e2 f3 f0 e1 f2 f3 e0 f1 f2 e3
...what you can do is copy the part that says "1087176704" (or whatever), open up the D2V file in Notepad - fixed or un-fixed, it doesn't matter - use the "find" function to locate the illegal line, and then make manual corrections. You'll often want to do this because DGindex is kind of dumb when choosing what automatic corrections to make. In the above example, this line...

1-2-2-3-0-1-2-3-0-1-2-0-1-3-3-0-1-2-3-0-1-2-3

...got automatically "corrected" to this...

1-2-2-3-0-1-2-3-0-1-3-0-1-2-3-0-1-2-3-0-1-2-3

..when anyone with a functioning brain can see that the result should be this...

1-2-3-0-1-2-3-0-1-2-3-0-1-2-3-0-1-2-3-0-1-2-3

Such corrections can be made without having ANY idea what the images look like or where they are in the video.
__________________
If I ask "How do I do X?" or "what happens if I do X?", and X is a very bad thing that no one would ever normally do, assume that I already know this, and that I have Katie reasons for asking anyway.
Katie Boundary is offline   Reply With Quote
Old 17th February 2017, 21:09   #16  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 3,578
The way I understand it (which could be wrong), for just the underlined section, it's effectively:

1-2-2-3-0-1-2-3-0-1-2-0-1-3-3 = bff - bff+rff - tff - bff - bff+rff - tff+rff - tff+rff

Which was automatically corrected to

1-2-2-3-0-1-2-3-0-1-3-0-1-2-3 = bff - bff+rff - tff+rff - bff - bff+rff - tff - tff+rff

When anyone liking nice orderly patterns can see the result should be:

1-2-3-0-1-2-3-0-1-2-3-0-1-2-3 = bff+rff - tff - tff+rff - bff - bff+rff - tff - tff+rff

And without ANY idea what the images look like or where they are in the video?

Seems dubious to me, but it's not the sort of thing I'd tend to think about. Isn't that what TFM() is for?

Last edited by hello_hello; 17th February 2017 at 21:23.
hello_hello is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.