Two-Frame Audio Desync Possibly Involving DGSource()

[CarlEdman]

First, let me apologize if this turns out to be the wrong forum. I think I am pretty familiar with what goes where in these forums, but this issue is somewhat confusing to me. Right now, my prime suspect is that the issue relates to DGIndexNV/DGSource, so I post here. If--as is quite possible--I am wrong, I'm happy to take the matter somewhere else.

I copy my DVDs and BRs to my home server for streaming in my home. To convert, I use various free tools and a moderately complex python script to automate the process. As the problem in principle could be introduced at any stage, let me give the whole rundown:

1. The disc is extracted to MKV files using MakeMKV
2. The raw streams in the MKV file are extracted using mkvextract and then processed separately according to type and format
3. Audio streams are transcoded to m4a using eac3to and qaac.
4. Video streams are indexed using DGIndexNV, served in an avisynth script using DGSource, fed to x264 to create new .264 files
5. Finally, all the processed component streams are remuxed to mp4 using mp4box

While this appears to have worked well for many years, recently I noticed a slight, but definite audio sync problem made noticeable by the correlation between a sharp sound a screen event in one clip. The video pretty clearly is two frames behind of the audio in the final mp4. While I am not as positive that this occurs with all files--without this correlation it is harder to detect--I fear it may.

Investigating the problem, I found the following facts:

• The desync was not in the original MKV files, played in any of several common video players.
• The desync was in the remuxed MP4 file, again in all players I tried.
• All the involved software uses the latest publicly available binaries.
• There is nothing strange about the clip it is plain 24000/1001 fps 1080p video. There is no interlacing, deinterlacing, telecining, teleciding, etc. involved.
• The audio transcode seems not to be at fault. Muxing the mp4 using the original AC3, rather than the transcoded M4A makes no difference.
• While I usually use a number of AviSynth filters, stripping all but the essential ones out, made no difference. The problem persisted with a script as simple as:
  

  SetMTMode(5,6)
  SetMemoryMax(1024)
  DGSource("clip.dgi", deinterlace=0, use_pf = true)
  ColorMatrix(hints = true, interlaced=false)
  KillAudio()
  autocrop(threshold=30,wMultOf=2, hMultOf=2,samples=51, mode=0)
  SetMTMode(2)
  Distributor()

• Nor is there anything unusual in the avisynth invocation or x264 call:

  avs2pipemod -y4mp "script.avs"
  | x264 --demuxer y4m - --preset veryslow --tune film --crf 19.0 --fps 24000/1001 --sar 1:1 --non-deterministic --no-fast-pskip --no-dct-decimate --profile high --level 4.0 --output "out.264"

Furthermore, I found the following clues:

• Going to specific recognizable frame in the original MKV and then finding the same frame in the final MP4, shows that MP4 frame is offset by 2. That is, e.g., what was frame 23,556 in the MKV is apparently frame 23,558 in the mp4.
• However, using ffprobe to analyze the number of frames in (1) the original MKV, (2) the extracted .264, (3) the transcoded .264, and (4) the remuxed mp4 yields exactly the same number: 23,834.

One hypothetical cause that would be consistent with all of the above would be if DGSource added two dummy frames at the beginning and chopped off two frames at the end. However, I have never heard of that happening and I would imagine it would be more generally known.

Can anybody offer any insight into this behavior or a possible workaround more scientific than putting "Trim(2,0)" into all my scripts?

[Groucho2004]

Just a couple of notes about your workflow:

1. DGIndexNV can load BD playlists. The whole Makemkv multiplex / demultiplex procedure just stresses your hard drive and takes a lot of time.

2. With your sample script, using Avisynth MT and multi-threading is overkill and adds unnecessary complexity (and the way you place the SetMTMode(2) call is completely wrong and probably makes it even slower).

3. You can also skip the piping step and feed the script directly to x264.

Since DGIndexNV also detects audio delays, your problem will probably go away by simplifying your workflow.

[videoh]

To simplify things for troubleshooting:

1. As Grouch2004 suggests, open the playlist (or VOBs) directly and make the project, demuxing the audio.

2. Make a simple script like this:

vid=DGSource("file.dgi")
aud=???("audfile").DelayAudio(xxx) # use the requisite audio source filter with xxx being the delay reported in the audio filename
AudioDub(vid,aud)

3. Play the AVS script in VirtualDub. Is it out of sync? If not, then something else in the workflow is at fault. If yes, then what disk is it? Knowing that, I can try to reproduce it.

There are some rare disks with a small segment of blank audio at the start (after which the PTS resets to 0) that confuses the DGIndexNV delay determination. We can investigate that if you find out of sync in the above scenario.

[CarlEdman]

Originally Posted by Groucho2004

1. DGIndexNV can load BD playlists. The whole Makemkv multiplex / demultiplex procedure just stresses your hard drive and takes a lot of time.

That is probably true, but I find that MakeMKV (which I run interactively to select playlists and streams) does a better job on many BRs that DGIndexNV (which is then run in batch mode from my frontend).

Originally Posted by Groucho2004

2. With your sample script, using Avisynth MT and multi-threading is overkill and adds unnecessary complexity (and the way you place the SetMTMode(2) call is completely wrong and probably makes it even slower).

Again, true, but the AVS script I posted was a stripped down version without all the filters that benefit greatly from the SetMTMode places just right.

3. You can also skip the piping step and feed the script directly to x264.

Not really, as I find that 64-bit x264 is substantially faster on my workstation than the 32-bit version, but many filters I use do not have a stable 64-bit version (or didn't last I checked), so I have to do the avisynth in 32 bits and pipe it in.

Originally Posted by videoh

Play the [simplified] AVS script in VirtualDub. Is it out of sync? If not, then something else in the workflow is at fault.

That is the next thing I tried. And indeed the simplified script run in VirtualDub has the correct frames. So does the full script when run in VirtualDub. So all my filters and DGSource are good. The problem occurs when feeding the avs2pipemod output to x264.

So it appears that either x264 or avs2pipemod insert the two extra frames (and I have been able to find the 2 extra frames on several different 1080p24 sources now). Given that I cannot get around using avs2pipemod without switching to 32-bit x264 or dumping many of my filters, what should I do?

Using a raw format in the pipe (rather than y2mp) makes no difference. My x264 x64 --version is:

x264 0.146.2538 121396c
(libswscale 3.0.0)
(libavformat 56.16.0)
built on Feb 24 2015, gcc: 4.9.2
x264 configuration: --bit-depth=8 --chroma-format=all
libx264 configuration: --bit-depth=8 --chroma-format=all
x264 license: GPL version 2 or later
libswscale/libavformat license: LGPL version 2.1 or later

[stax76]

If avs2pipemod inserts the frames then you can try alternative piping methods like this:

ffmpeg -i "avs_file" -f yuv4mpegpipe -pix_fmt yuv420p - |

[videoh]

I'm not familiar with piping or why you need it. I can only guess that you are doing all your Avisynth stuff in 32 bits and then feeding to 64 bit x264. I would wonder if the gains from 64 bit x264 are really so large to force you through this. Doesn't piping have significant overhead that may negate any advantage? If I have misunderstood, can you please straighten me out?

Also, of course, try stax76's suggestion. He seems to know about piping.

[CarlEdman]

Originally Posted by videoh

I'm not familiar with piping or why you need it. I can only guess that you are doing all your Avisynth stuff in 32 bits and then feeding to 64 bit x264.

That is right. Sorry for not being more clear on this.

I would wonder if the gains from 64 bit x264 are really so large to force you through this. Doesn't piping have significant overhead that may negate any advantage? If I have misunderstood, can you please straighten me out?

It has been a while since I did benchmarks, but I remember when switching that I got double-digit percentage increases in speed. When you have encodes running 10 or more hours that is pretty significant.

Piping--unless the OS has been screwed up worse than any currently widely used one or you spend a lot of cycles converting data to/from pipeable format (which with raw or nearly raw files, you don't)--costs almost nothing and can even speed up a series of single-threaded programs over a single-threaded monolithic app.

Also, of course, try stax76's suggestion. He seems to know about piping.

I will as soon as I dig up a version of ffmpeg which likes my AviSynth 2.5.8 32-bit filters (which neither my installed 64-bit ffmpeg or the first 32-bit ffmpeg I found does).

What I did try was running the same full script with the same options directly in x264 32-bit. That avoided the extra frames. The problem really is pinpointed to the pipe (but could be either end).

[videoh]

Thank you for the explanation. Several years ago I tried piping and found significant overhead making it basically unfeasible. But that was in WinXP 32-bit days and things may have changed since then. And of course I could have implemented it poorly. I still have the code, so I think I'll give it a whirl again just for fun.

[CarlEdman]

Ok, one more update: I found a frame-accurate player for y4mps and had a look at the output of avs2pipemod. That one is correct (i.e., having the same frames as the input file). That means two things: (1) x264 has bug with piped-in raw files; (2) switching to a different avisynth pipe providers would not make a difference.

[videoh]

Did you try HolyWu's idea to remove --non-deterministic?

[Asmodian]

It would be a very odd bug that would add frames with non-deterministic. That only allows threads to use information from other threads as soon as they finish a block, instead of only when they will have finished the block for sure. I have never had an issue with non-deterministic and I usually use it.

[Groucho2004]

I suggest using this version of avs2yuv for piping.

Syntax example:

avs2yuv "script.avs" -o - | x264 - --crf 16 --frames 154247 --output "test.264" --demuxer y4m

Do not use Distributor() in the script, the tool adds the call.

[CarlEdman]

Ok, I ran into some problems with my software tools while trying to reduce the issue to a simple, reproducible case. To pinpoint the problem I need to be able to look at video streams on a frame-accurate basis. Unfortunately, all the tools I tried either (a) don't allow you to look at particular specified frame (VLC, WMP, QT, ffplay) or (b) they don't give consistent results.

For example, I identified a specific frame in an MKV video I converted to MP4. In PotPlayer the frame was 822 in the original MKV, but 820 in the final MP4. In MPC-BE, the same frame was 823 in the original MKV, but 825 in the final MP4!

Can anybody recommend software player with consistent, accurate(!) frame searches, so I can have some confidence that any inaccuracies were introduced by the conversion process, not player bugs? It would have to work for MKVs and MP4s at least, though if it could also do it for .264s and .y4mps that would make pining down the source of the problem much easier.

[videoh]

DGDecNV: First index the stream and then serve it into VirtualDub via Avisynth. DGDecNV is frame accurate.

[CarlEdman]

OK, it is quite an impression of Inspector Clouseau that I have offered in this thread. At first, I knew that it was DGSource() was at fault. Then, well... I knew that avs2pipemod was at fault. Then, well... I knew that the x264 stream reader was at fault. Now... I just hope that you were as amused as I'm embarrassed.

At videoh's suggestion, I put both the original MKV and the final MP4 through DGIndexNV and fed them into VirtualDub from an avisynth script consisting just of DGSource(). Then I identified both the first non-blank frame and a specific identifiable frame at about 43 seconds in both the MKV and the MP4. The results were entirely consistent.

In VirtualDub, for both the MKV and the MP4, the first non-blank frame was frame 24 (@1001ms) and the later frame was frame 1029 (@42918 ms). This is actually a relief because it means that my transcodes were actually correct all along and I don't need to redo or patch thousands of MP4s on my server!

Then I tried both the MKV and the MP4 in my usual software player of choice, PotPlayer.

In PotPlayer x64 1.6.54266, the original MKV had its first non-blank frame at #24 (@1001ms) and the later frame at #1028 (@42909 ms). While the latter frame number is off by one, the result is fine as the desync is only an unnoticeable 9 ms. The MP4 was much worse. The first non-blank frame was at #21 (@876ms) and the later frame was at #1026 (@42793 ms). This is pretty bad; not only are the frame numbers both off, the video had gotten desynced from the audio by over 100ms in both cases.

In MPC-BE x64 1.4.5b364, the original MKV had its first non-blank frame at #23 (@959ms) and the later frame at #1028 (@42876 ms). That is not great, but at least the desync stays within 40ms (or about one frame). But in the MP4, the first non-blank frame was at #27 (@1126 ms) and the later frame was at #1032 (@43042 ms). Again, the audio got desynced from the video by a very noticeable over 100 ms!

At this point, I'll probably post my conclusion and a link to this thread in the software players forum. But if anybody can suggest another software player here that does not screw over MP4s so badly, I'd love to hear it.

[stax76]

Could it possibly be the same issue currently discussed in the ffmpegvideo and megui thread?

[Emulgator]

Just for these occasions I decided to simply run my test encodes with ShowFrameNumber(),
applied at up to 6 positions in my more extended scripts,
for pinpointing where I would end, just in case of doubt, and for any customer (me included).
Had interesting results back then, but this would be OT.

[wonkey_monkey]

I have only skimmed this thread so apologies if my information is irrelevant.

I have noticed two-frame sync issues in the past when using DGIndex/mpeg2source (as opposed to the NV versions).

I have suspected it was something to do with leading B-frames in the MPEG video. Once, when I ripped The Matrix DVD, I found that when loading it into VirtualDub-MPEG2, it reported the first two frames as B-frames (non-keyframes) with the first I-frame (keyframe) coming at frame 2.

It was my suspicion that DGIndex was ignoring the two B-frames and starting from the first I-frame, and that's why the audio was late. Could it be something similar with DGIndexNV?

One thing that makes this issue so hard to track down is that the human brain is geared towards ignoring audio delays. After all, it only takes 25m to "delay" audio by two frames in the real world. In contrast, video delays seem far easier to spot.

David

[videoh]

Both DGDec and DGDecNV are designed to properly handle leading B frames. If anyone has a sample that produces an async I'd be happy to look into it. There are some causes other than leading B frames.

[mandarinka]

I noticed that some my encodes had "extra" frames leading to desync too. It happened few times in the last and this year to me, and it seems the trouble starts in dgdecode (mpeg2, DVD source), but it is a random occurrence, not replicable.

What I found by comparing the source and output was that at some frame, there would be a visually observable corruption and that frame gets duplicated, increasing the total framecount of clip and creating desynchronization. It was not two such frames, I was only getting one at a time, but it could happen in multiple places in video.

It is a weird bug, because it happens with the same dgdecode/dgindex binaries that I have been using for years. I was using avisynth 2.6 alpha 3 (not sure now, I upgraded few months ago) and Windows 8.1 64, and AFAIK latest public binaries of dgdecode/dgindex...

Not sure if it is the same bug but it might be if the codebase is common... also no idea what could be done to track this bug, since it is random.