FieldOrder Problems in DGIndex (fixed in 1.44)

[Amnon82]

Frameserved with DGIndex 1.4.3:
http://amnon82.cybton.com/int/elcid_int_tff_DGIndex.demuxed.png (http://amnon82.cybton.com/int/elcid_int_tff_DGIndex.demuxed.jpg)
click on the picture to see the original screenshot

Frameserved with DVD2AVIdg 1.10:
http://amnon82.cybton.com/int/elcid_int_tff_dvd2avidg.demuxed.png (http://amnon82.cybton.com/int/elcid_int_tff_dvd2avidg.demuxed.jpg)
click on the picture to see the original screenshot

Testencodes + D2V-files (25 MB) (http://www.megaupload.com/?d=Y1JMC4FB)

I used in both encodes AVISYNTH 2.55 and this Script:

#loadplugin("C:\TOOLS\decf\MPEG2Dec3dg.dll")
mpeg2source("D:\Autoq2enc\elcid.d2v")


Here is the ECL I used:

[item]
title=AutoQ2-EclFile
vid_out=1
vaf_out=1
i32_out=0
i32_pulldown_prg_thres=80
i32_pulldown_rpt_val_thres=6
aud_out=0
aud_mode=0
vid_file0=D:\Autoq2enc\elcid_int_tff.m2v
vid_file1=D:\Autoq2enc\elcid_int_tff.m2v
vaf_file=D:\Autoq2enc\elcid_int_tff.vaf
i32_file=C:\cce.i32
aud_file=C:\cce.mpa
file_focused=0
xe_exec_mode=0
ve_exec_mode=0
ve_mpeg1=0
packet_size=2048
ref_tc=0
ref_frm=0
drop_frame=0
frame_rate_idx=3
cbr_brate=6000
opv_q_factor=2
opv_brate_min=8950
opv_brate_max=9500
vbr_brate_avg=9500
vbr_brate_min=8950
vbr_brate_max=9500
vbr_bias=30
vbr_pass=3
create_new_vaf=1
seq_endcode=1
dvd=0
resize=0
interpolation=4
width=720
height=576
half_width=0
half_height=0
uscc=0
top_first=1
offset_line=0
lum_level=0
deinterlace=0
blanking=0
blank_top=0
blank_left=0
blank_right=0
blank_bottom=0
aspect_ratio=3
gop_m=3
gop_nm=4
gop_hdr=12
seq_hdr=1
all_closed_gop=0
fix_gop_length=0
fix_vbv_delay=0
adjust_q_matrix=1
last_frame_should_be_I=0
24p_to_60i=0
411_to_422=0
pulldown_detect=0
pulldown_auto_rebuild=0
letter_box=0
panscan=0
qmat_idx=-1
q_scale_type=1
qmat_name=Matrix_by_AutoQ2
qmat=
8 16 19 22 26 27 29 34
16 16 22 24 27 29 34 35
19 22 26 27 29 34 35 38
22 22 26 27 29 34 35 40
22 26 27 29 32 35 40 48
26 27 29 32 35 40 48 50
26 27 29 35 40 48 50 60
27 29 35 40 48 50 60 62
16 20 24 28 32 36 40 44
20 24 28 32 36 40 44 48
24 28 32 36 40 44 48 52
28 32 36 40 44 48 52 56
32 36 40 44 48 52 56 58
36 40 44 48 52 56 58 60
40 44 48 52 56 58 60 62
44 48 52 56 58 60 62 62
samples_per_sec=48000
stereo=2
brate_idx=7
crc=1
aud_sync=0
pict_name=Matrix_by_AutoQ2
q_char_f=30
progressive=0
alternate_scan=1
letter_box=0
h_filter=0
h_filter_idx=10
h_filter_strength=100
h_filter_idx2=0
h_filter_hpf=0
h_filter_hpf_idx=5
h_filter_hpf_strength=25
h_filter_hpf_idx2=0
v_filter=0
v_filter_val=4
no_move=0
dither=0
dither_Y_max=8
dither_C_max=4
intra_dc_precision_9_max=113
intra_dc_precision_10_max=113
i32_32=0
force_black=0
force_monochrome=0
fill=0
fill_Y_val=16
fill_C_val=128
fill_lines_top=30
fill_lines_bottom=30

[file]
name=C:\Programme\AutoQ2\temp\AQ2FULLENC.avs
stream_no=0
encode_first=0
encode_last=120507
chapter=5624:
chapter=10868:
chapter=16220:
chapter=24656:
chapter=28736:
chapter=37232:
chapter=39620:
chapter=43556:
chapter=55436:
chapter=62564:
chapter=68434:
chapter=72766:
chapter=78466:
chapter=84946:
chapter=98626:
chapter=102406:
chapter=104758:
chapter=108502:


I only changed the D2V-Creator ...

[Guest]

I'm downloading now, but have these comments right now.

I don't want or need the encodes. I need you to give me the unprocessed source clip. I also need to know what your settings were in DGIndex.

EDIT: I see from the D2V file that there is an illegal transition. Please run FixD2V or try again with version 1.4.4 RC1. It shouldn't be necessary to encode. You ought to be able to demonstrate the problem by just serving into VirtualDub.

Finally, I'm not interested in ancient versions and comparisons to them. If you can demonstrate a problem in the current version, then I am all ears.

[Amnon82]

It seems I found the problem I had with 1.44RC1. I used DGDecode.dll of 1.43.
RC1 detected an error in the d2v. I encoded the fixed one and got a bad result. With 1.43 it was ok. I'll try it again with 1.44 RC1 and upload You the d2v-files.

This is what 1.44 RC1 gaves me as info:

Aspect Ratio: 16:9
Frame Size: 720x576
Profile: main@main
Framerate: 25.000 fps
Video Type: PAL
Frame Type: Interlaced
Colorimetry: ITU-R BT470-2
Frame Struct: Frame
Field Order: -
Coded#: 120507
Playback#: 2
Vob Cell ID: 3/8
Bitrate -

D2V Fix Output

Illegal transition: 0 -> 2
900 5 0 10477568 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 e0
d00 5 0 10975232 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
corrected...
900 5 0 10477568 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 e1
d00 5 0 10975232 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2

[Guest]

The source is changing the field order in the middle, which is illegal. You cannot ask your display process to display two successive top fields or two successive bottom fields. When DGIndex corrects this, it may create a single combed frame at the point of the illegal transition, because an extra field is inserted.

When the video is non-pulled-down progressive, the borked field order is irrelevant. In such a case you may prefer to retain the "bad" D2V file to avoid the combed frame.

Ask yourself why your source stream changes field order. Where did this stream come from?

I'll add some further explanation in the DGIndex manual about this.

[Amnon82]

I uploaded another encode: http://www.megaupload.com/?d=N6RXGFL9

I created a D2V from the whole DVD (elcid_144RC1wholedvd.d2v)
This has the Error:

D2V Fix Output

Illegal transition: 0 -> 2
900 5 0 10477568 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 e0
d00 5 0 10975232 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
corrected...
900 5 0 10477568 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 e1
d00 5 0 10975232 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2

I encoded 12% from the beginning of the DVD and abort then the encode.

Then I cutout the scene I upped already to megaupload.

The encoded file is elcid_int_tff_144RC1_partofwholedvd.demuxed.m2v.
It comes with the d2v-file.

Then I cutted the scene from the SourceVOBs directly in DGIndex.
I got the elcid_144RC1_cuttedpart.d2v and encode it with the same settings
as the last encode. This time all is ok.

After that I demuxed the scene form the SourceVOBs.
I got elcid_144RC1_source.d2v and elcid_144RC1_source.demuxed.m2v.

Then the bad d2v-file came into my mind. I encoded it till 16% from the beginning.
I got elcid_int_tff_144RC1_badd2v.d2v and elcid_int_tff_144RC1_badd2v.demuxed.m2v.

In this case the illegal transition was right.

It is the DVD "EL_CID_DIE_LEGENDE" You can buy in germany.

[Guest]

I explicitly said I don't want any encodes. I want a source VOB fragment.

Anyway, I have explained the cause and the solution. Is that not satisfactory for you?

[Amnon82]

After that I demuxed the scene form the SourceVOBs.
I got elcid_144RC1_source.d2v and elcid_144RC1_source.demuxed.m2v.


... I uploaded it. This was the wrong fragment. Only the first Logo has BFF. The rest has TFF.

Anyway, I have explained the cause and the solution. Is that not satisfactory for you?

Yes I'm realy happy about it. Now I know how to tell the fieldorder of the source. 2 is TFF and 0 is BFF. THX for this.

This movie took me more time than others. I encode it now the third time. This time I changed the d2v by hand into this:

d00 5 0 2048 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 444416 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 1032192 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 1626112 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 2213888 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 2803712 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 3395584 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 3985408 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 4573184 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 5167104 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 5754880 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 6344704 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 6936576 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 7526400 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 8114176 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 8708096 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 9295872 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 9885696 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 5 0 10477568 1 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
d00 5 0 10975232 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
d00 5 0 11544576 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2

This was the old:

d00 5 0 2048 1 1 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 444416 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 1032192 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 1626112 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 2213888 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 2803712 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 3395584 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 3985408 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 4573184 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 5167104 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 5754880 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 6344704 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 6936576 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 7526400 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 8114176 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 8708096 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 9295872 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 9885696 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 f0 f0 e0
900 5 0 10477568 1 1 70 70 d0 f0 f0 e0 f0 f0 e0 e1
d00 5 0 10975232 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2
d00 5 0 11544576 2 1 b2 b2 92 b2 b2 a2 b2 b2 a2 b2 b2 a2

I let you know what happend with my encode. Also I upload You the SOURCE.M2V I remuxed form the VOB. If You create a D2V You get the same D2V during the demuxing with DGIndex 1.44 RC1.

I realy don't know why the first logo is bff. Maybe a new protection system ;)

Here the Link to the Source.m2v: http://www.megaupload.com/?d=RVSJ0AFK

Do You need still the VOB?

btw. nice cat (the avatar)

[Guest]

No, I don't need the VOB; the demuxed bit is enough. But anyway, we understand it now. Certainly your fix is fine, too, although if it were in the middle of the movie it would be a little more difficult.

Meow

[Guest]

I've had a major epiphany regarding field order, pulldown, Fix D2V, etc.

I had a clip recently from Yufi, which had an "illegal transition" which was fixed by the Fix D2V process. Yet here we have one that is OK without Fix D2V and becomes broken when the Fix D2V process is applied. This got me thinking about whether these transitions are really illegal. I always believed that they were because they require the display process to display two top or two bottom fields in succession.

Now that we have the single-step capability in DGIndex, I single stepped through Yufi's clip with Honor Pulldown Flags. Despite the "illegal transition", the clip played correctly (no jerking as you'd get with bad field order) after the transition. Yet the D2V served into VirtualDub showed jerking after the transition.

The next experiment I performed was to cut the Yufi clip after the transition. Of course, now it played fine served to VirtualDub.

Then I began thinking about what my company's MPEG decoding cells do. They have to be resistant to such transitions, because they are used in DVD players, and we don't see this effect playing the DVDs. And it's unlikely that a broadcaster's MPEG stream would always have the same field order in the MPEG, which implies that these transitions must occur. Then I thought about how they would be handled. If the decoding process generates a top field when the display process expects a bottom field, the display process just repeats the existing bottom field and then carries on.

So why do we see this effect, i.e., continuing jerkiness after some of these transitions, which is fixed by Fix D2V? It's now clear to me. The DGDecode pulldown process is not correct. The DGIndex pulldown process works correctly but the DGDecode process uses a different algorithm. Looking at its code I see that it uses the initial field order of the clip. But if you have a transition 0->2 the field order changes and the algorithm becomes invalid. All along, Fix D2V was just acting as a band-aid for a deficiency of DGDecode's pulldown algorithm!

So, I will have to fix DGDecode to use the same method as DGIndex. I believe this will remove the need for Fix D2V, but we'll see about that. Note that this issue arises only when there is pulldown and one of these transitions. It's been pretty rare thus far, especially with most processing being done on DVD content. But I think as we move more and more to transport streams, this will become more significant.

Any thoughts or insights about all this would be welcome.

EDIT: The changes turned out to be minor because I was able to salvage the DGDecode algorithm rather than toss it out. I've implemented and tested the changes a little. The thesis appears to be borne out. The pulldown process in DGDecode no longer relies upon the initial field order and Fix D2V is now superfluous. I'll test some more and then make a new release.

[Amnon82]

Thanks for this great replay. Even I understand what You mean. Btw. the third encode works without a framedrop or any other bug.

I hope the little upload helped You a little bit :D

Is it fixed in RC2?

... i changed the title of this thread.

[Guest]

I hope the little upload helped You a little bit. Your thread here was life-changing for me. You're my hero.

But, uh, are you like German or something? You keep upper-casing "You". I'm not God ya know, not even close.

Is it fixed in RC2? Of course. Well, I mean, it's supposed to be. You could tell me. Tell ya what, I'll give you 12 hours to moan. If you don't, I will release 1.4.4 tomorrow morning. And if it's broken, it'll be your fault. Bwahahahahahah. :devil:

[Amnon82]

But, uh, are you like German or something? You keep upper-casing "You". I'm not God ya know, not even close.

What do You mean by this exactly? I'm happy to help people. I'm german. What does it matter? I feel multicultural ;)

I know You're not God, but sometimes Your programs do wonders :D

I'll give you 12 hours to moan. If you don't, I will release 1.4.4 tomorrow morning. And if it's broken, it'll be your fault. Bwahahahahahah.

I didn't had the time to stop You ;) I had to work, so I wasn't able to throw You something into Your way...

... if something brake I'll keep on trying to fix it.

Now I'm downloading Your final.

Go on with Your coding, I do it also :D

[Guest]

What do You mean by this exactly? Well, see, there you go again. You've put "You" instead of "you". If the word started the sentence that would be OK. Usually "God" is always capitalised; hence my jocular reference. I was wondering if maybe there were different capitalisation rules in German.

[scharfis_brain]

I german a capitalized 'Du' (You) is no offense.
It means that you are respecting the person you're communicating with.

It is not a must-rule. In offical it is even forbidden, to use a capitalized 'Du'.
But it is mandatory to capitalize 'Sie' (3rd person plural) in communications.
'Sie' and 'Du' are the same in english (you).
Unknown or respectable persons (your boss etc.) are commonly adrressed with 'Sie', while friends and family etc. are adressed with 'Du'.

If I would write 'DU' (YOU) it could (would) be meant offensive.

[Guest]

Oh, thank You for the clarification. It was never a matter of offense, just curiosity on my part.

[Prodater64]

Well, see, there you go again. You've put "You" instead of "you". If the word started the sentence that would be OK. Usually "God" is always capitalised; hence my jocular reference. I was wondering if maybe there were different capitalisation rules in German.

OT
Im not english native either, and I thought until now that all pro names should be wrote in capital letters (I, You, He, She, We, etc).
END OT

[Cyberia]

I'm not God ya know, not even close.

So... I should stop praying to the great Donald?

What about the shrine I made?




ps: I'm kidding. (about the shrine)

[Amnon82]

Neuron2, I can also write you instead of You, if you wish. I thougth it would be gentle to respect you by writing You ...

[Guest]

Neuron2, I can also write you instead of You, if you wish. I thougth it would be gentle to respect you by writing You ... Certainly, please yourself. As I said, I was just curious about it. And keep the valuable feedback coming!