NHMLTransform

[Aktan]

Based on the work of zmi from this thread http://forum.doom9.org/showthread.php?t=112199, I have created my own converter since his application seemed inaccurate. I think my version is more accurate, and I hope it will be useful to someone.

NHMLTransform

Converts a NHML to a variable frame rate with the help of Matroska Version 2
Timecodes and optionally adds a delay to an audio NHML to match the delay
introduced by the conversion.


Usage:

NHMLTransform -i <input-file> -t <mkv-timecodes-v2-file> -o <output-file>

Where:

<input-file> : Path to NHML file to convert
<mkv-timecodes-v2-file> : Path to Matroska timecodes (v2) to read from
<output-file> : Path to NHML file to output to

Optional:

-nt <int> : Sets a new timeScale (overrides -f)
-f <int> : Divides the timeScale by this much to keep DTS
numbers under the maximum value MP4Box supports (a
bug with MP4Box)
-ia <input-audio-nhml> : Path to the audio NHML file to fix (requires -oa)
-oa <output-audio-nhml> : Path to the audio NHML file to output to (requires
-ia)

Reverse conversion (NHML to Matroska timecodes (v2):

-ot <out-mkv-timecodes> : Path to Matroska timecodes (v2) file to output to
(requires -i)

Notes:

- To retreive a NHML file out of a MP4, use the following command:

MP4Box.exe -nhml <track number> <MP4 file>

- While the NHML specs define DTS values as unsigned integer, MP4Box
currently has a bug and only allows a maximum value of a signed integer.
The workaround is to lower the timeScale value in the input NHML. The
optional -f parameter does this for you. The setting I usually do is
divide the timeScale by 10, 100, 1000, etc until the DTS values are under
signed integer maximum. This will make the times slightly off, but it's
close enough.


Limitations:

Due to the way I create variable frame rate, the program may introduce a
video delay in which the audio might not be synced if the delay is large
enough. A way to fix this is to add the delay to the audio also. This
functionality is included with this program.


Thanks to zmi for giving me a starting point.

Changelog:

2011/07/26:
-Rewrote the conversion part better to not rely on the source NHML to be of
constant framerate.

2011/07/22:
-Added the ability to set a new timeScale.

2011/02/13:
-Fixed a (possible?) bug with sorting the timecodes.

2011/02/04:
-Added the ability to do the reverse conversion.

2011/01/14:
-Fixed a bug in reading Matroska timecodes that assumes only 1
line of comments at the top.
-Added the ability to fix the audio NHML with the delay
introduced.

2010/10/29:
-Added some simple error checking.

2010/10/10:
-Fixed a small mistake of leaving debug information there.

2010/09/24:
-Fixed a bug to account for weird timestamp delays in frames.

2010/06/10:
-Fixed a bug not accounting all b-frames when finding the max offset.

2010/03/11:
-Fixed to support bigger delays in offsets.

2010/01/20:
-Fixed a lot of rounding errors.

Download: http://www.mediafire.com/?kc7trb4s3yll0q0

[Velocity 7]

Hello,

I tried using NHMLTransform on an mp4 I downloaded from Justin.tv. My source was an .flv file with a h264 and MP3 streams. I used FLV Extract to get the video, audio, and timecode v2 file, and then used MP4Box to build a temporary mp4 with the video. Afterward, I extracted the NHML file and used NHMLTransform on it with the timecodes, and then attempted to insert it. However, I get the following:

NHML import - Stream Type Visual - ObjectTypeIndication 0x21
Error importing testnhml.nhml: Bad Parameter

My commands were the following:

mp4box -nhml 1 test.mp4
NHMLTransform.bat -i test_track1.nhml -t timecodes.txt -o testnhml.nhml
mp4box -add testnhml.nhml -new video.mp4

Any ideas?

[Aktan]

Can I have the timecodes file and the original NHML file to see what's going on?

As of right now, I have no clue, but maybe it's due to the DTS timestamps being too large.

[Velocity 7]

Here you go:

http://www.mediafire.com/?frwpgrh1rwbm4sd

[Aktan]

Okay I took a look. It's weird. Your MKV timecodes has 3 frames at time zero. This is causing the error for MP4Box. Also your MKV timecodes has times for 75161 frames, while your NHML has listed only 75160 frames. What I would do to try is, modify your MKV timecodes by removing one of the zero time code, and changing the last zero timecode to one and try it again. Basically like this:

# timecode format v2
0
1
33
66
133
200
...

I also noticed there seems to be lack of B-Frames in your encode. At least the NHML suggest this.

[Velocity 7]

Looks like that did the trick. Thanks!

[Aktan]

I just thought of something else. I still don't like the fact that there are 3 frames with time 0 in the MKV timecodes. You should check the video carefully and see if it sometimes playback weirdly. If it does, it may mean the timecodes are offset for each frame by 1. You can also tell if it's off if the sound seems consistently to be a little bit off. A way to fix this is to get rid of the extra 0s in the MKV timecode and add a new made-up timecode at the end. For example:

# timecode format v2
0
33
66
133
...
4407173
4407240
4407307
4407374 <-- made up timecode

[sneaker_ger]

Several leading zeros are perfectly fine in mkv. It's its way of applying negative delays. I've seen it quite often with flv, but I don't know the reason for that.

[Velocity 7]

Yeah, after making the modifications Aktan suggested, things are working smoothly now. Sound is actually in sync for my files so I'm good to go here.

[Aktan]

So which method did you use? Remove all extra zeros in the beginning and make up a new last timecode, or remove 1 zero from the begining and change 1 zero to time one?

[Velocity 7]

Removed one of the zeroes and changed another zero to 1.

However I also had other FLV files which caused two leading 0's, which only required a change and no removal due to frame count being correct.

[Aktan]

Removed one of the zeroes and changed another zero to 1.

However I also had other FLV files which caused two leading 0's, which only required a change and no removal due to frame count being correct.

Just wondering. Did you try the other way at all?

[Velocity 7]

I hadn't really, but I could try it later when I have some time.

EDIT: Just checked. I have three files; A.mp4, B.mp4, and C.flv. A is the one with the first modifications, and B is the one with the second modifications. C is the original source file. I opened them all under AviSynth using FFMpegSource and found that A is 2 frames behind C and B is 1 frame behind C.

EDIT 2: Actually it may not matter much at all, because VFR will sometimes cause A to be ahead of C or behind, same with B.

EDIT 3: For your reference, you may look at this StackVertical() result sample (sample.avi, 67.3 MB). A, B, and C are stacked together from top to bottom.

http://www.mediafire.com/?leljexqesv5uupg

[Aktan]

Wow I just noticed your edits. Too bad I don't get notified on edits, I'll take a look. Also I want to note that I put up a newer version that checks to make sure the amount of frames in the NHML file matches the amount of timecodes in the MKV times file.

[Aktan]

I took a look at the sample, and I don't like the fact that it's sometimes off. One should be very close to like the original. I don't know how accurate FLV Extract is. If you don't mind, could you provide me a sample FLV?

[Velocity 7]

Here you go: http://www.mediafire.com/?04c8ciw5in8x2ag

[Aktan]

Okay, I took a little look today and notice FFmpegSource also can output the timecodes. The timecodes FFmpegSource outputs looks to be better, so right now I suggest you use those timecodes instead. The param is timecodes="<filename>". Try it out!

I'll keep looking into it, but right now I don't trust the timecodes FLV Extract gives.

[Aktan]

Well, I guess I was wrong. The timecodes from FFmpegSource has a lot of duplicate timecodes. So that can't be right. I'll keep doing more research.

[Aktan]

Okay. After looking at the .FLV file with an hex editor and following the FLV specs as specified on this (http://osflash.org/flv) website, I found out that the timecodes FLV Extract gives are more correct! Apparently the correct method you should use is method one. That is, remove extra zeros if there are too many timecodes listed and change one of the zero to one. For example:

# timecode format v2
0
0
0
33
66
133
200
...

to

# timecode format v2
0
1
33
66
133
200
...

The extra zero is weird since it seems to be part of the video data, but not really. The frame is too small. Maybe it was just the header or something.

Well there you have it. I had fun learning how FLV works =)

[Selur]

@aktan: could you extend you application to convert between timecode v2 and .nhml files?

[Aktan]

@aktan: could you extend you application to convert between timecode v2 and .nhml files?

Well the original intent of this program was to modify a CFR NHML to VFR NHML using MKV timecodes v2. If I understand you correctly, you would like for me to extend the program so that it can take a VFR NHML as input and output MKV timecodes v2?

I should mention a way you can already convert a VFR NHML back to MKV timecodes v2 is using MKVToolnix. Just make a VFR MP4 out of the NHML, convert the MP4 to MKV using mkvmerge, and then extract the timecodes v2 using mkvextract.

[Selur]

If I understand you correctly, you would like for me to extend the program so that it can take a VFR NHML as input and output MKV timecodes v2?
Yup, that would be nice. :)

I should mention a way you can already convert a VFR NHML back to MKV timecodes v2 is using MKVToolnix. Just make a VFR MP4 out of the NHML, convert the MP4 to MKV using mkvmerge, and then extract the timecodes v2 using mkvextract.
I know. But it would be nice if 'NHMLTransform' would be able to do it. :)

Cu Selur

[Aktan]

Okay then, will add that feature along with a bug fix on reading MKV timecodes v2.

[some dude]

In your read me it states

Limitations:

Due to the way I create variable frame rate, the program may introduce a
video delay in which the audio might not be synced if the delay is large
enough. A way to fix this is to add the delay to the audio also. I plan
on updating the program to also convert the audio NHML to fix this if there
is a large enough interest.

I'm interested in that in case you still plan on updating it. Of course I haven't tried the program yet so I don't even know if the delay is noticeable.

[Aktan]

Sure, I'll do all three things. It should be done by the end of the week (need to find some time). I've seen the video delay become 40 frames ahead of the audio. Though I must say it was mainly due to the framerate being extremely variable, with it going down to 3 FPS.

[Aktan]

So I've fixed a small bug and added the ability to fix audio NHML. Your request, Selur, is next. Should be early next week as my weekend is kind of booked. Sorry.

[Selur]

No, problem. I'm happy that you accepted the request at all. :)

Thanks!

[Aktan]

Just an update. It is pretty much done. I just need to clean up and add some instructions. Sorry for the delay. I kind of forgot/slacked/had important real life things going on.

[Selur]

No problem, happy to know that you are still on it. :)

[Aktan]

There, the functionality has been added. Do report any bugs you find.

[Selur]

I will thanks!

[Selur]

just wondering: would it be possible to use the output of MP4Box when called with -dts test.mp4 (i.e. MP4Box.exe -dts -std d:\video_vfr.mp4) instead of the nhml file? (or are there some additional infos missing?)
(would be nice to avoid the MP4Box -nhml call, since it always additionally extracts the videostream, and it seems like there's no way to specify where the output files will be created)

Cu Selur

[Aktan]

I just finally noticed this post. Sorry. It seem the email notification is broken, so I had no idea. I'll take a look tomorrow, but my current answer is, I don't know.

[Aktan]

Looking at it a bit, I don't think it is possible. I will look more n a few days.

[Aktan]

Finally answering this question. The answer is no, not possible, since the output file is missing information like the frame size and if the frame is a seek point. I guess one would be able to just use this information if one modifies the MP4 directly and change only the timestamp information.

[Selur]

Thanks for the info.
I realized that it's possible with mp4fpsmod (http://forum.doom9.org/showthread.php?p=1503592) but forgot so post here. :)

[VFR maniac]

Finally answering this question. The answer is no, not possible, since the output file is missing information like the frame size and if the frame is a seek point. I guess one would be able to just use this information if one modifies the MP4 directly and change only the timestamp information.

"change only the timestamp information"?
Haha, it's not preferable since ISO base media/QuickTime file format has boxes that relate with timestamps.
Also, "only timestamp changing" breaks media interleaving, this causes poor reading of file access.
(Well, though "extreme composition delay + correcting this delay by EditBox" also brings about this, considering this is trivial since EditBox can pick arbitrary portion of media.)
So, reconstructing whole movie is needed to convert into VFR material.

BTW, L-SMASH (http://repo.or.cz/w/L-SMASH.git) project now includes timeline editor.
I'll support importing timecode v1 and import auto timebase generation from x264cli later.

[Aktan]

Thanks for the info.
I realized that it's possible with mp4fpsmod (http://forum.doom9.org/showthread.php?p=1503592) but forgot so post here. :)

Interesting tool. Thanks for the link!

[Aktan]

"change only the timestamp information"?
Haha, it's not preferable since ISO base media/QuickTime file format has boxes that relate with timestamps.
Also, "only timestamp changing" breaks media interleaving, this causes poor reading of file access.
(Well, though "extreme composition delay + correcting this delay by EditBox" also brings about this, considering this is trivial since EditBox can pick arbitrary portion of media.)
So, reconstructing whole movie is needed to convert into VFR material.

BTW, L-SMASH (http://repo.or.cz/w/L-SMASH.git) project now includes timeline editor.
I'll support importing timecode v1 and import auto timebase generation from x264cli later.

I'm no expert in MP4 format, but from what I have read, I thought the DTS (stts) and CTS (ctts) are just separate boxes. Since I have not read about media interleaving, I guess I had no idea. Thanks for the information.

[Dogway]

I have the same problem as Velocity 7, I have 93 zeroes, I did try removing all zeroes except 2, and rename them 0 and 1 (this gives me an error of mismatch on amount of timecodes), and I also tried removing one zero, and changing the last zero to 1, that gives me the error:
NHML import - Stream Type Visual - ObjectTypeIndication 0x21
Error importing testnhml.nhml: Bad Parameter

I also tried tc2mp4Mod, it tells me: There is Invalid TimeStamp!!

These are my both timecodes in case you can see something strange
http://www.mediafire.com/?n7511w165fiebvp

[b66pak]

try mp4fpsmod (http://forum.doom9.org/showthread.php?t=161386)...
_

[Dogway]

Yes, I also tried mp4fpsmod, but it outputs txt timecodes where NHMLTransform needs nhml format....

[b66pak]

what are you trying to do? (be exact)...
_

L.E. if you try to insert timecodes into a mp4 file and fail use mediafire to upload you timecode file and post the link...also type "mp4box -info your_video.mp4" and post the result here...
_

[Dogway]

I downloaded a flv from ustream and I'm trying to remux it to mp4. My timecodes are in my previous post

This is the info of my mp4 without timecodes:

* Movie Info *
Timescale 600 - Duration 03:01:20.498
Fragmented File no - 2 track(s)
File Brand isom - version 1
Created: GMT Fri Nov 04 18:10:36 2011

File has root IOD
Scene PL 0xff - Graphics PL 0xff - OD PL 0xff
Visual PL: AVC/H264 Profile (0x15)
Audio PL: AAC Profile @ Level 2 (0x29)
No streams included in root OD

iTunes Info:
Encoder Software: Yamb 2.1.0.0 http://yamb.unite-video.com

Track # 1 Info - TrackID 1 - TimeScale 11981 - Duration 03:01:20.227
Media Info: Language "Undetermined" - Type "vide:avc1" - 130356 samples
MPEG-4 Config: Visual Stream - ObjectTypeIndication 0x21
AVC/H264 Video - Visual Size 640 x 480 - Profile Baseline @ Level 3
NAL Unit length bits: 32
Self-synchronized

Track # 2 Info - TrackID 2 - TimeScale 44100 - Duration 03:01:20.499
Media Info: Language "Undetermined" - Type "soun:mp4a" - 468584 samples
MPEG-4 Config: Audio Stream - ObjectTypeIndication 0x40
MPEG-4 Audio AAC LC - 2 Channel(s) - SampleRate 44100
Synchronized on stream 1

edit:
I thought the problem could come from the mp4 muxing, so I took out yamb form the equation and remuxed manually with mp4box at 11.9808 (indicated by FLVExtract)

* Movie Info *
Timescale 600 - Duration 03:01:20.498
Fragmented File no - 2 track(s)
File Brand isom - version 1
Created: GMT Sat Nov 05 16:29:09 2011

File has root IOD
Scene PL 0xff - Graphics PL 0xff - OD PL 0xff
Visual PL: AVC/H264 Profile (0x15)
Audio PL: AAC Profile @ Level 2 (0x29)
No streams included in root OD

Track # 1 Info - TrackID 1 - TimeScale 11981 - Duration 03:01:20.227
Media Info: Language "Undetermined" - Type "vide:avc1" - 130356 samples
Visual Track layout: x=0 y=0 width=640 height=480
MPEG-4 Config: Visual Stream - ObjectTypeIndication 0x21
AVC/H264 Video - Visual Size 640 x 480
AVC Info: 1 SPS - 1 PPS - Profile Baseline @ Level 3
NAL Unit length bits: 32
Self-synchronized

Track # 2 Info - TrackID 2 - TimeScale 44100 - Duration 03:01:20.499
Media Info: Language "Undetermined" - Type "soun:mp4a" - 468584 samples
MPEG-4 Config: Audio Stream - ObjectTypeIndication 0x40
MPEG-4 Audio AAC LC - 2 Channel(s) - SampleRate 44100
Synchronized on stream 1

It doesn't solve anything...

[b66pak]

use this timecode file:

http://www.mediafire.com/?056vag7x7yi5sx4

with this line:

mp4fpsmod -t new-flv-time.txt -c -o final.mp4 my_file.mp4
_

[Dogway]

GOOD! It kinda worked, nice to know mp4fpsmod accepts .txt timecodes, I was only using it to export timecodes not for muxing, but the problem it has is it makes my aac audio unseekable.
I also don't know what you changed exactly to the timecodes, you substituted the 93 zeroes by times separated each by 3 (ms?), but you also removed the first three non-zero lines; 145,145,245.

[b66pak]

first create the mp4:

mp4box -flat -add my_file.h264 -new my_file.mp4

then add the timecodes:

mp4fpsmod -t new-flv-time.txt -c -o final.mp4 my_file.mp4

finally add the audio:

mp4box -add final.mp4 -add audio.aac -new final_mux.mp4
_

[Dogway]

Whoaaa thanks a bunch, with this my day is complete hehe. Thanks for the VFR initiation. Could you explain me why you used 3 ms interleaving and removed the first three timecodes, just in case I face the same issue again?

[b66pak]

you can't have the same timecode for more than one frame in a mp4 file...so 0,0,0,0,0,0,0,0,0,0.....145,145 is invalid...also the number of timecodes in the timecode file must much the number of frames in the mp4 (this is the reason for "mp4box -info")...for your file: 130356 samples in mp4 but 130357 values in timecode file...this is also invalid...so all i have to do is delete one of the first zeros and resample all the rest of the first 99 values...
_

[Aktan]

Thanks for the help, b66pak. I really should take a look at mp4fpsmod someday as it seems simpler than my tool.