Strange 30fps/24fps Problem

[CarlEdman]

I've been converting my DVD/Blu-Ray collection to mp4s on my media server for some years and with one recent file, I've encountered a strange problem I've never seen before.

Background:
I rip the DVDs/BRs using MakeMKV and use a homebrew python frontend to automatically transcode, tag, remux as mp4. My homebrew tools use MKVToolnix to extract the tracks, DGIndexNV to index the video files, and avisynth and x264 to reencode.

The particular file I am having this problem with is a SD trailer from a BR movie. The mkv identifies the file's video as 29.97 fps progressive. Visual inspection on the track confirms that this is correct; there are no telecining or interlacing artifacts. The extracted track has the same features. DGIndexNV identifies it as 29.97 fps progressive.

However, the output of x264, and avisynth (with a script that does not do any deinterlacing or teleciding) is too short, compared to the audio files and the original MKV. The output video, coded by my tools as 29.97fps, is exactly 25% too short, strongly suggesting that the output is actually 23.97 fps progressive!

I looked everywhere for a clue on this and the only thing I ran across where these curious lines from the dgindexNV log file:
Coded Number: 3504
Playback Number: 4380
Frame Repeats: 0
Field Repeats: 1752

That playback/coded ratio is (1.25) is certainly just what you would expect in such circumstances. The author appears to have taken a 24fps video and converted it to 30fps, not by telecining, but just by repeating every couple of frames. But I thought I had set DGIndexNV to respect pulldown etc. flags so that it would replay to x264 exactly as the original, so shouldn't the repeated frames have come through?

Relatedly, I looked at the coded/playback number in a bunch of other dgindexnv logs and for most of them, the numbers are reasonable and match. But for a surprising number, the coded number is reasonable (100,000s), but the playback number is 2(!) without any frame or field repeats. For others, both coded and playback number are 0. This files transcode fine, so they are more a matter of curiosity.

PS: If I could ask for one little bugfix in DGIndexNV it would be to stop stealing focus when run in batch mode. If you have scripts doing that in the background while you are working on something else, this is pretty annoying.

[Guest]

However, the output of x264, and avisynth (with a script that does not do any deinterlacing or teleciding) is too short, compared to the audio files and the original MKV. The output video, coded by my tools as 29.97fps, is exactly 25% too short, strongly suggesting that the output is actually 23.97 fps progressive! Please give us the script, because I can't guess what your fieldop setting was.

I looked everywhere for a clue on this and the only thing I ran across where these curious lines from the dgindexNV log file:
Coded Number: 3504
Playback Number: 4380
Frame Repeats: 0
Field Repeats: 1752 So it's pure 3:2 soft telecine. What is the problem?

That playback/coded ratio is (1.25) is certainly just what you would expect in such circumstances. The author appears to have taken a 24fps video and converted it to 30fps, not by telecining, but just by repeating every couple of frames. No, fields are repeated, not frames. That is soft 3:2 telecining.

I thought I had set DGIndexNV to respect pulldown etc. flags so that it would replay to x264 exactly as the original, so shouldn't the repeated frames have come through? The repeated fields will come through if you have fieldop=0. You can open the script in VirtualDub, then single step and you should see the usual 3 progressive and 2 combed frames.

But for a surprising number, the coded number is reasonable (100,000s), but the playback number is 2(!) without any frame or field repeats. For others, both coded and playback number are 0. This files transcode fine, so they are more a matter of curiosity. That sounds very wrong, can you provide a stream? But be aware, you need to do a play/preview operation to see the correct coded and playback numbers for the section you have played. You can't just open the file and expect the playback number to be complete. You can also do a Save Project and then look at the numbers. You haven't told me what you did to see these numbers you report. I can't imagine doing a play/preview/save project and seeing 0 reported for coded and playback. How did you make that happen, and can you please give me a sample that shows that?

PS: If I could ask for one little bugfix in DGIndexNV it would be to stop stealing focus when run in batch mode. If you have scripts doing that in the background while you are working on something else, this is pretty annoying. You'll have to tell me how to duplicate this, as I do not do any batch stuff myself.

[CarlEdman]

Please give us the script, because I can't guess what your fieldop setting was.

The script invokes DGIndexNV like this:
DGIndexNV -i "FILE.264" -o "FILE.dgi" -h -e
The avisynth script was:
SetMTMode(5,6)
SetMemoryMax(1024)
DGSource("FILE.dgi", deinterlace=1, use_pf = true)
ColorMatrix(hints = true, interlaced=false)
KillAudio()
autocrop(threshold=30,wMultOf=2, hMultOf=2,samples=51, mode=0)
SetMTMode(2)
super = MSuper(planar=true)
bv1 = MAnalyse(super, isb = true, delta = 1, blksize=8, overlap=4)
fv1 = MAnalyse(super, isb = false, delta = 1, blksize=8, overlap=4)
bv2 = MAnalyse(super, isb = true, delta = 2, blksize=8, overlap=4)
fv2 = MAnalyse(super, isb = false, delta = 2, blksize=8, overlap=4)
bv3 = MAnalyse(super, isb = true, delta = 3, blksize=8, overlap=4)
fv3 = MAnalyse(super, isb = false, delta = 3, blksize=8, overlap=4)
MDegrain3(super,thSAD=400,planar=true,bv1,fv1,bv2,fv2,bv3,fv3)
Distributor()

You can see that I did not specify the 'fieldop' argument on DGSource. This may be the source of the problem. In my scripts it is a holdover from DGIndex in which the field op was specified in the .ini file, but that option does not appear to exist in the DGIndexNV .ini file.

The repeated fields will come through if you have fieldop=0. You can open the script in VirtualDub, then single step and you should see the usual 3 progressive and 2 combed frames.

That is the weird part. I've seen a ton of telecined video streams over the years and I know how to recognize that 3/2 combing pattern. This file doesn't have it. Which is why I suspected that it was some weird encode that just repeated a frame every now and then, rather than a field.

Seems wrong, can you provide a stream? But be aware, you need to do a play/preview operation to see the playback number. You can't just open the file and expect the playback number to be complete.

Happy to. The usual problem is that the source files are humongous. Just let me know any way to send them that would be convenient to you.

And the numbers are from log files output after a complete .dgi has been made, not figures read-off the status window in realtime.

You'll have to tell me how to duplicate this, as I do not do any batch stuff myself.

It should be pretty easy to replicate. DGIndexNV (and DGIndex, if I remember correctly), even when invoked with '-hide' and '-exit' will steal focus every time. To replicate, just create a script that sleeps for ten seconds, then starts DGIndexNV with these options. After starting the script, click on another window and watch it lose focus when dgindexnv starts.

[Guest]

Please attach the full DGI file. You have deinterlace=1 which is very wrong for a soft telecined file (try deinterlace=0). In fact, IIRC, DGSource() should error out when you try to do that. Show me the DGI file and we will go from there. fieldop should default to 0.

Also, all this MT stuff puts you on your own. Get rid of it for now until we figure out what is going on. I do not support MT stuff. Just have DGSource() in the script and nothing else.

For samples, cut the VOB with DGSplit and post to mediafire.com.

[MeteorRain]

The very first thing is to stop using deinterlace.

Deinterlacing simply eliminates the combing pattern.

[CarlEdman]

Getting rid of the deinterlace=1, which had been in the program for historical reasons and had not been discovered earlier because almost all actually non-progressive content went through a different path, definitely solved the problem. The source file still has some weird, non-telecine mangling going on, but with pulldown flags honored, the output was fine, so I am happy.

[Guest]

Do you plan to follow up with stream(s) and an explanation of how to duplicate for your second issue, where you claim that the coded/playback numbers are wrong? Thank you.

[CarlEdman]

Sure. I was just testing that I could reliably reproduce the bugs. These bugs, if that is what they are, are at worst cosmetic. The files are played just fine by DGSource/etc. It is just the log files that may contain incorrect numbers.

I was able to reproduce the incorrect Playback Number in dgindex . The resulting d2v file and log file are attached. As you can see, the d2v file appears fine, but the log file states, among others:
<QUOTE>...
Coded Number: 78526
Playback Number: 2
Frame Repeats: 0
Field Repeats: 0
...
Remain: FINISH
...</QUOTE>
Dgindex for this example was invoked on the command line:
<QUOTE>dgindex -i "FILE.mpg" -o "FILE" -fo 0 -ia 3 -om 2 -hide -exit</QUOTE>

I was not able to reproduce the dgindexnv misreporting. After every dgindexnv batch execution, my frontend captures the log files, puts the contents into its own log file, and deletes the log file to minimize the directory clutter. The content of the captured log files contain lines like this for multiple sources (though they are correct for most sources and the dgi files play back fine even for these):
<QUOTE>Stream Type: Elementary
Video Type: MPEG2
Profile: esc (-1@0)
Coded Size: 0x0
Display Size: [not specified]
Aspect Ratio: Reserved
Frame Rate: 0.000000 fps
Colorimetry: BT.601 [2]*
Frame Structure:
Frame Type:
Coded Number: 0
Playback Number: 0
Frame Repeats: 0
Field Repeats: 0
Bitrate:
Bitrate (Avg):
Bitrate (Max):
Elapsed:
Remain: 0:00:00
FPS:
Info: Finished!</QUOTE>

But when I manually invoke dgindexNV from the commandline, either in batch mode or in GUI mode, the final logs are accurate every time.

Given this heisenbug nature, I suspect it is probably some sort of race condition. My frontend gets a signal that the dgindexnv subprocess terminated successfully and somehow grabs the log file before dgindexnv was actually able to write out its updated contents. This is probably not a high priority problem.

On a separate issue, were you able to reproduce the focus-stealing issue?

[CarlEdman]

Sure. I was just testing that I could reliably reproduce the bugs. These bugs, if that is what they are, are at worst cosmetic. The files are played just fine by DGSource/etc. It is just the log files that may contain incorrect numbers.

I was able to reproduce the incorrect Playback Number in dgindex . The resulting d2v file and log file are attached. As you can see, the d2v file appears fine, but the log file states, among others:

...
Coded Number: 78526
Playback Number: 2
Frame Repeats: 0
Field Repeats: 0
...
Remain: FINISH
...
Dgindex for this example was invoked on the command line:
dgindex -i "FILE.mpg" -o "FILE" -fo 0 -ia 3 -om 2 -hide -exit

I was not able to reproduce the dgindexnv misreporting. After every dgindexnv batch execution, my frontend captures the log files, puts the contents into its own log file, and deletes the log file to minimize the directory clutter. The content of the captured log files contain lines like this for multiple sources (though they are correct for most sources and the dgi files play back fine even for these):
Stream Type: Elementary
Video Type: MPEG2
Profile: esc (-1@0)
Coded Size: 0x0
Display Size: [not specified]
Aspect Ratio: Reserved
Frame Rate: 0.000000 fps
Colorimetry: BT.601 [2]*
Frame Structure:
Frame Type:
Coded Number: 0
Playback Number: 0
Frame Repeats: 0
Field Repeats: 0
Bitrate:
Bitrate (Avg):
Bitrate (Max):
Elapsed:
Remain: 0:00:00
FPS:
Info: Finished!

But when I manually invoke dgindexNV from the commandline, either in batch mode or in GUI mode, the final logs are accurate every time.

Given this heisenbug nature, I suspect it is probably some sort of race condition. My frontend gets a signal that the dgindexnv subprocess terminated successfully and somehow grabs the log file before dgindexnv was actually able to write out its updated contents. This is probably not a high priority problem.

On a separate issue, were you able to reproduce the focus-stealing issue?

[Guest]

I can't really comment without a stream and explanation for how to duplicate it.

I looked into the focus issue but don't find what I consider any viable solution. Do you have any ideas about it?

[CarlEdman]

I can't really comment without a stream and explanation for how to duplicate it.
Fortunately the source file for that bug is "only" 1.6 gBytes. I'm uploading it to MediaFire right now and will PM you the link as soon as it is up. The reproduce the outcome, just run the command line above on the file when you have it.
dgindex -i "FILE.mpg" -o "FILE" -fo 0 -ia 3 -om 2 -hide -exit

I looked into the focus issue but don't find what I consider any viable solution. Do you have any ideas about it?

I feared as much. For many GUI systems, the startup library code just grabs focus before the application gets a chance to override it and, when the application is converted to also work on CLI, that glue code can't be overridden. Unfortunately, I have no expertise in Windows GUI coding.

[Guest]

Thanks, Carl. I'll wait for your PM.