Joints 1.0.2: joining files with unequal audio/video

[Turtleggjp]

I'm (neuron2) adding a link to Joints here to make it easy to find for newcomers to the thread:

http://neuron2.net/misc/joints102.zip

-----

Originally Posted by neuron2

It could only be to add frames at the end of your files. With the workaround, yes, they get added at different spots but the desync can never be more than one frame at any given time, which is not perceptible.

And to be honest, developing a proper solution for this is very low on my priority list.

I think you're looking at solving this problem the wrong way. The way I do it with my AVCHD camcorder is by removing audio frames instead of adding (repeating) video frames. I did my usual calculations that I do when I have a series of clips to the 3 sample clips he posted. Here are my results:

Clip1:
Video Frames: 3780 (126.126 sec)
Audio Frames: 3942 (126.144 sec)
Excess Audio: ~18ms <-Greater than 16ms (half of one frame) so DROP one audio frame!
Audio is now ~14ms too short.

Clip2:
Video Frames: 1290 (43.043 sec)
Audio Frames: 1346 (43.072 sec)
Excess Audio: ~28ms <-WAIT! Audio is already 14ms behind.
Adjust for previous clip: ~14ms <-Less than 16ms (half of one frame) so Leave this one alone!
Audio is now ~14ms too long.

Clip3:
Video Frames: 810 (27.027 sec)
Audio Frames: 845 (27.040 sec)
Excess Audio: ~13ms <-WAIT! Audio is already 14ms ahead.
Adjust for previous clip: ~27ms <-Greater than 16ms (half of one frame) so DROP one audio frame!
Audio is now ~5ms too short

Presto! Your clips are never more than 16ms out of sync either way. I've been doing this for a little over a year now with my camcorder's footage, and it works great. In fact, I just finished a trip that had about 271 clips.

I use a spreadsheet in Excel to help me set everything up, including using DGAVCIndex to index all my clips and produce an audio file for each one. Using the .dga file for each clip, I can determine the number of video frames. I then run AC3 Cutter on all the audio clips, and it reports the number of audio frames. Once I have this information, all the information I posted above can be calculated. I then have tsMuxeR join all my clips together (with unsyncronized audio) and then have AC3 Cutter go in and trim out all the audio frames that I determined need to be cut out. I can then run tsMuxeR again, but this time have it not use the audio in my clips, but instead use the audio file created by AC3 Cutter. I now have a .M2TS file with all my videos joined together, and with syncronized audio.

If this sounds like a complicated process, well yes it is. I consider it more of a proof-of-concept. Eventually, I'd like to either make up a little tool that would be able to generate the synced audio track, given a series of clips, or else hope that someone like Neuron2 would be willing to add code to his tools to create the synced audio track. All you need to know to do the calculations is:

1. Number of video frames per clip (Indexer already knows this)
2. Video frame rate (Indexer already knows this)
3. Number of audio frames per clip (this could be trickier)
4. Audio frame rate (31.25 fps for AC3 audio, the most common in AVCHD camcorders).

By creating a "camcorder mode" in the indexer, AC3 audio can be assumed for now until more codecs start showing up. That might help figuring out #3.

What do you think Neuron2? I'm willing to help you on this if you need any more info.

Matt

[Guest]

Yes, I understand that cutting audio is an option, but I assumed it may create objectionable pops.

But anyway, sure, it's possible. First, I'd prefer to do this with a preprocessing tool rather than in the DGNV tools. But second, as I said, it's very low on the priority list for me, especially since I have an acceptable (to me) workaround that doesn't involve the massive effort you are going through.

[Turtleggjp]

Originally Posted by neuron2

Yes, I understand that cutting audio is an option, but I assumed it may create objectionable pops.

I think I understand what you mean by pops. I want to clarify that my method is only useful for camcorder files, not seamless branching Blu Rays like Ratatouille. Files on a seamless branching Blu Ray should naturally combine together (perhaps with a video frame repeat or two) and the audio should not be cut. With camcorders though, the clips do not naturally combine together (since they are usually created from two different scenes) and so there is already a "pop" at the junction.

One other thing, I can think of a couple different types of "pops." One is caused by having a complete audio frame missing, which simply results in missing data. This can be tricky to hear if the audio is quiet. Even if it is loud, it may take careful listening to hear. These "pops" are not too bad, but should probably still be avoided. The other kind of "pop" occurs when a program blindly trims audio data as if it were LPCM. This can result in a very loud pop or chirp, since the compressed data has now been altered significantly. These types of "pops" should definitely be avoided.

This is why I suggested the "camcorder mode" option. My method works for camcorder files, like the ones posted, but should not be used for seamless branching Blu Rays.

Originally Posted by neuron2

But anyway, sure, it's possible. First, I'd prefer to do this with a preprocessing tool rather than in the DGNV tools. But second, as I said, it's very low on the priority list for me, especially since I have an acceptable (to me) workaround that doesn't involve the massive effort you are going through.

Ok, I'll see what I can do about a preprocessing tool. I've wanted to do that for a while now, as I think it would be very useful for people with AVCHD camcorders. You'll have to be patient though, I'm not a highly experienced developer like Neuron2...

EDIT: BTW, it was only a massive effort creating the Excel file that I use as a template. Now, it's just a moderate effort.

[Guest]

How do you automate the actual audio frame cuts? Does AC3 Cutter take a cut list of some sort? What is the link to this cutter tool?

I'm starting to get a little interested.

[rack04]

Originally Posted by neuron2

How do you automate the actual audio frame cuts? Does AC3 Cutter take a cut list of some sort? What is the link to this cutter tool?

I'm starting to get a little interested.

Speaking of cut list, is there a way to translate the trim/cut points (pressing [ and then ]) specified in the dga/v/m files to a cut list to trim the audio at the same points?

For example:

RANGE 1110441 4582562

[Guest]

First things first. We're thinking along the same lines, but I want to know about this cutter thing first.

[Turtleggjp]

Found it:

http://forum.doom9.org/showthread.php?t=92325

I actually use a modified version that fixes a couple issues with his parsing of the cut list. I've been using it for a couple years now to cut the audio on my captured TV shows.

Basically, it takes a VirtualDub style .vcf file as its cut list, which my Excel spreadsheet generates for me. For example, here is the one it generated for those 3 sample clips:

VirtualDub.subset.AddRange(0,3941);
VirtualDub.subset.AddRange(3942,1346);
VirtualDub.subset.AddRange(5288,844);

[Guest]

Reading that thread, the author says it's buggy and shouldn't be used!

Who made your modifed version of it? Is it also buggy? Can we have the source code?

What is the syntax of the AddRange() function?

I'm sure people will ask: What about AAC, LPCM, and MPEG audio?

[Turtleggjp]

Originally Posted by neuron2

Reading that thread, the author says it's buggy and shouldn't be used!

I think he was refering to the way he determined which audio frames to take, which was buggy. I figured out how to do it right, and modified the code, which leads to...

Originally Posted by neuron2

Who made your modifed version of it? Is it also buggy? Can we have the source code?

I modified it, as far as I know it is not buggy, and yes here is the source code.

Originally Posted by neuron2

What is the syntax of the AddRange() function?

AddRange(StartFrame, NumberOfFrames), just like VirtualDub.

Originally Posted by neuron2

I'm sure people will ask: What about AAC, LPCM, and MPEG audio?

Good question. Do you know of any camcorders that use these? As I said, my method is so far only for AVCHD camcorder clips, not Blu Rays.

Bottom line, AC3 cutter does have a lot of extra stuff in it that I never need, like patching bitrates to a higher one in order to maintain constant bitrate. I basically threw this method together with what I had and filled in the gaps where needed. I figured your tools already know how to pull the audio files out of .M2TS files, so it's just a matter of looking at the audio and counting how many frames are in it. With a little analysis, you can determine when to throw out a frame when joining files together.

Really bottom line, this works great for me, and I would be happy to share what I know to get it working for others as well.

EDIT: Most of my modifications to the code are in the "parseVCF" function in the "ac3functions.c" file. I seem to have commented it pretty well, so I hope you can follow what I did.

[Guest]

Thanks, Matt. I think I may make something out of this along the lines of the preprocessing utility we discussed above. Automatic cut list generation is a must.

[Turtleggjp]

Absolutely. That's basically what my Excel spreadsheet does is generate that cut list. Prior to that though, it also creates a batch file that calls DGAVCIndex for each clip, and then another batch file that runs AC3 Cutter on all the audio clips created by the first batch file (to have it report number of audio frames for each one). Once the data from these two runs is gathered into my spreadsheet, it can make the cutlist. After that, AC3 Cutter does the actual cutting after tsMuxeR has (blindly) joined all the clips together and then demuxed the audio track.

[Guest]

OK, first phase completed. I have a working GUI that takes a file list and determines the info needed for the cuts. Tested it successfully on the camcorder files and on Ratatouille M2TS files. The app is tentatively named "Joints".

Next step: integrate the cutter code so that the GUI app can directly correct the already demuxed audio file. Of course that will make the whole thing open source.

[Guest]

Matt, I've run into a problem with your code. The syntax you give above for your AddRange calls passes the audio frame numbers, e.g. 3942 as the start of the second range. But it appears that these are interpreted as video frame numbers in the code:

FrameTime = 1 / vfps;

Time = ((sFrame + nFrames) * FrameTime) - dDelay + Sync;
RoundUp = ((Time / AC3Time) - (long)(Time / AC3Time)) >= .5;
tPtr->endFrame = ((long)(Time / AC3Time)) + RoundUp;

You can see that Time is calculated using the video frame rate! Also, doesn't VirtualDub expect video frame numbering in those calls?

So, please explain. I do not see any reason for the cutter to know about the video frame rate if it is receiving cut data referred to audio frame numbering.

I can fix it to use the audio frame numbering as it should but thought I better mention it so you are aware that you may have been doing things wrongly.

[Turtleggjp]

Originally Posted by neuron2

Matt, I've run into a problem with your code. The syntax you give above for your AddRange calls passes the audio frame numbers, e.g. 3942 as the start of the second range. But it appears that these are interpreted as video frame numbers in the code:

FrameTime = 1 / vfps;

Time = ((sFrame + nFrames) * FrameTime) - dDelay + Sync;
RoundUp = ((Time / AC3Time) - (long)(Time / AC3Time)) >= .5;
tPtr->endFrame = ((long)(Time / AC3Time)) + RoundUp;

You can see that Time is calculated using the video frame rate! Also, doesn't VirtualDub expect video frame numbering in those calls?

So, please explain. I do not see any reason for the cutter to know about the video frame rate if it is receiving cut data referred to audio frame numbering.

I can fix it to use the audio frame numbering as it should but thought I better mention it so you are aware that you may have been doing things wrongly.

I thought of this last night. What's going on is that this program was originally designed to work with video frames as cut points, not audio frames. This is because the .vcf files were supposed to come from VirtualDub, which is talking about video frames with its AddRange function. In fact, the code I fixed deals with this conversion from video frame rate to the audio frame rate. This fix is almost unnecessary for the application we are talking about, since when I call this program, I specify a frame rate of 31.25 (AC3 frame rate). This is supposed to make the conversion from video frame rate to audio frame rate a simple 1:1 conversion. I say almost unnecessary, because the program does not recognize the simplicity of the conversion and did all its calculations anyway. Due to the bugs in the code, it would occasionally miss cut the audio by a frame here and there. I built a checking function into my Excel spreadsheet to catch these errors, so I could correct them (This got to be tedious sometimes). Once I corrected the code, these errors have never happend again, but my code still does not do a simple 1:1 mapping of the cut points when a frame rate of 31.25 is specified.

I made my fixes to the code in order to cut audio on HD captures long before I had my AVCHD camcorder. When I finally got my camcorder and realized what was happening when I tried to merge my clips together, I realized I needed a way to cut frames out. This led me back to AC3 Cutter. So as you can see, it's a classic case of using a program for something that it was not exactly intended to be used for.

Anyway, to answer your question, you can probably ignore the code that seems to be converting video frames to audio frames. Whenever I run this program for this purpose, the two are the same.

Matt

EDIT: Here is a sample of one of my command lines:

"E:\Apps\AC3 Cutter v0.4\AC3 Cutter.exe" -fr 31.25 -vcf "E:\Video Camera\AVCHD\090828 - Lake Basin\090828 \
- Lake Basin.vcf" -o "E:\Video Camera\AVCHD\090828 - Lake Basin\090828 - Lake Basin.txt" \
 "Q:\090828 - Lake Basin.track_4352.ac3" "E:\Video Camera\AVCHD\090828 - Lake Basin\090828 - Lake Basin.ac3"

[Guest]

Whoa, you lie about the frame rate!

That's real hacky. I'll have to clean it all up. I assume you are not a professional software developer.

[Turtleggjp]

Originally Posted by neuron2

Whoa, you lie about the frame rate!

That's real hacky. I'll have to clean it all up. I assume you are not a professional software developer.

Yes it is. You can probably just throw out all that code that converts video frames to audio frames, since you are already calculating the audio frames that need to be taken.

Like I said, this program was not designed for this particular application. Turned out, I was able to get it to work pretty well, and then it turned out that my fixes made it work perfectly. I've been meaning to write a tool like you are doing, but I feel fortunate that I was able to port my "code" over from Excel into a C++ program. As you have guessed, no I am not a professional software developer (yet...).

[Guest]

OK, I have fixed up the cutter. It is still separate and my GUI writes a cut list.

I applied it to the Ratatouille BD and the result was perfect sync. The process removed 512ms (16 frames) of audio as needed.

I will integrate the cutter into the GUI and then make a beta release.

Matt, I want to commend you and thank you for bringing this idea and process to the fore.

[Guest]

It's all done and integrated into one convenient GUI.

You first demux the AC3 audio as usual. Then you open Joints, specify the source transport files, select the AC3 file, set your PIDs and frame rate, and then hit Process. When it finishes you have a trimmed AC3 file. A cut list is also written.

I just have to write the help file and then you can have it.

[Guest]

Here is the first version. Source code will be released after some initial feedback.

http://neuron2.net/misc/joints100.zip

[Guest]

Now with VC-1 video support:

http://neuron2.net/misc/joints101.zip