Problem with MP4Box

[R!tman]

Hi all,

last night I did a 2-pass encoding of a movie using mencoder under linux. As I want to mux it into a matroska container, I tried to convert it to .mp4 first, as I did not want to use "--engage allow_avc_in_vfw_mode" with mkvmerge. Btw, the movie is encoded in x264.

So I did this (I guess here is the error):
$ MP4Box -aviraw video 2ndpass.x264
[avilib] Reconstructing index...Extracting AVI video (format h264) to 2ndpass_video.h264
After that I use
$ MP4Box -add 2ndpass_video.h264 movie.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.000 FPS
Adjusting AVC SizeLength to 24 bits67/100)
Import results: 95606 samples - Slices: 2574 I 79708 P 108930 B - 1 SEI - 2244 IDR
Stream uses B-slice references - max frame delay 2
Saving to movie.mp4: 0.500 secs Interleaving
to create an .mp4 file. But this is not the complete movie, only a little more than one hour, instead of 3h20mins (see the number of frames, also later for 1st pass).

I tried several versions of gpac, including the latest cvs; none worked. But what I found very interesting was that it did work with the file from the first pass.

Here are a few details:
$ MP4Box -aviraw video 1stpass.x264
Extracting AVI video (format h264) to 1stpass_video.h264
and
$ MP4Box -add 1stpass_video.h264 movie.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.000 FPS
Import results: 299608 samples - Slices: 8390 I 246676 P 344150 B - 1 SEI - 7322 IDR
Stream uses B-slice references - max frame delay 2
Saving movie.mp4: 0.500 secs Interleaving
As you can see "[avilib] Reconstructing index..." only shows up for the second pass.

Last but not least, I want to list only the differences in the x264 encoding options of the two passes:

firstpass:me=2:subq=4:frameref=2:trellis=1:pass=1
second pass:me=3:subq=5:frameref=3:trellis=2:pass=2:bitrate=2250
Any help is greatly appreciated!

:thanks: R!tman

[nm]

Maybe you are hitting some 2 GB file size limit, either in the filesystem if you are using an old FAT partition, or as a bug in MP4Box/gpac. You don't seem to be specifying bitrate for the first pass (you should give it the same bitrate as in the second pass, by the way), so the file is probably significantly smaller.

[R!tman]

Maybe you are hitting some 2 GB file size limit, either in the filesystem if you are using an old FAT partition, or as a bug in MP4Box/gpac. You don't seem to be specifying bitrate for the first pass (you should give it the same bitrate as in the second pass, by the way), so the file is probably significantly smaller.
nm, thank you for your answer! Indeed the 1st pass file is much smaller as you can see here:
$ du -sh *.x264
918M 1stpass.x264
3.2G 2ndpass.x264
As I am using reiserfs, I doubt it's the filesystem.
I will try to reencode the movie this night with "bitrate" in both passes. But this will most probably not solve the problem, as for MP4Box the faulty one seems to be the 2nd pass, not the 1st.

[GodofaGap]

mencoder cannot output raw h264?

[R!tman]

mencoder cannot output raw h264?
GodofaGap, well, I just learned it can using "-of rawvideo".

Unfortunately, a quick test with
mencoder -o 2ndpass_new.x264 -ovc copy -nosound -of rawvideo 2ndpass.x264
did not work either.

But, as I will have to reencode the whole thing again anyway (because I forgot to include "bitrate" in the first pass), I hope this will solve the problem.

On the other hand, this still does not solve the problem MP4Box has with that file.



Directly adding the raw video produces this:
$ MP4Box -add 2ndpass_new.x264 movie.mp4
(Feature Not Supported)Unknown input file type
Error importing 2ndpass_new.x264: Feature Not Supported
And a new encoding like this
mencoder -of rawvideo -o test.avi -ovc x264 -x264encopts bitrate=2250 -nosound -ss 00:02:00 -endpos 00:00:10 2ndpass.x264
results in this (latest gpac-cvs and gpac-0.4.1):
$ MP4Box -add test.avi movie.mp4
*** glibc detected *** double free or corruption (top): 0x000000000051c470 ***
Aborted

[R!tman]

As I mangaged to get my hands on a friend's M$ (windows, that is) laptop, I did some testing there with MP4Box.exe. I used the same version as I did with linux, which is
$ MP4Box -version
MP4Box - GPAC version 0.4.1-DEV
GPAC Copyright: (c) Jean Le Feuvre 2000-2005
(c) ENST 2005-200X
Creating an .mp4 file under windows and playing it with mplayer in Linux resulted (after the first hour of the movie, which works fine) in a whole bunch of these errors:
[h264 @ 0xb1dd10]AVC: Consumed only 13622 bytes instead of 8119477
[h264 @ 0xb1dd10]Unknown NAL code: 15
The "NAL codes" I have seen are: 0, 14, 15, 20, 21, 22, 23, 25, 26, 29

Furthermore, the raw video created with windows is bigger (3220MB) than the one from Linux (1023MB). Trying to create an .mp4 file in Linux with the raw video file created in windows results in this error:
$ MP4Box -add 2ndpass_video.h264 video.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.000 FPS
Adjusting AVC SizeLength to 24 bits21/100)
MP4Box: media_import.c:3578: gf_import_h264: Assertion `nal_start' failed.
Aborted
I hope some more information will help, and not scare you off...

When creating the raw video file in windows (the one that is bigger than in linux), I do not get "[avilib] Reconstructing index...".

This is what I get when creating the .mp4 file in windows (from the raw video also created in windows)
MP4Box.exe -add 2ndpass_video.h264 video2.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.000 FPS
Adjusting AVC SizeLength to 24 bits21/100)
Import results: 299608 samples - Slices: 8390 I 246676 P 344150 B - 1 SEI - 7322
IDR
Stream uses B-slice references - max frame delay 2
Saving to video2.mp4: 0.500 secs Interleaving
And the last thing: trying to create an .mp4 file in windows from a raw video created directly from mplayer (using -vf rawvideo) in linux also did not work.

HELP! :(

[R!tman]

Hi, it's me again!

This night I reencoded the movie again, this time with "bitrate" also in the first pass. Now, MP4Box has troubles with the files from both passes. They are both 3.2GB now.
Peculiar is that both fail at 1023MB, which made me think about the RIFF chunks every 1023MB mplayer creates. I found something in the forum:
http://forum.doom9.org/archive/index.php/t-89842.html

Unfortunately, the solution posted there
mencoder -ovc copy -nosound -noodml -o 2ndpass_noodml.x264 2ndpass.x264
does not work for me.

Using that, the raw video file is also 3.2GB, but creating an .mp4 file from it fails at 63%. 63% of 3.2GB is approximately 2024MB - interesting...

Anyone?

[nm]

So you didn't get the raw output from Mencoder working since you are still using AVI?

[R!tman]

So you didn't get the raw output from Mencoder working since you are still using AVI?
nm, no that's not how it is.

I DID get the raw output from mencoder working, but I could not use that file together with MP4Box.

This is what I used to get raw output for an example file
$ mencoder -of rawvideo dvd://3 -ovc x264 -x264encopts bitrate=2000 -nosound -o 3.raw
And this is what I got with MP3Box
$ MP4Box -add 3.raw asdf.mp4
(Feature Not Supported)Unknown input file type
Error importing 3.raw: Feature Not Supported
So, as MP4Box does not play nicely with raw output from mencoder I am back on the original task. That's why I still use avi output and try to get MP4Box to work.

[GodofaGap]

mp4box does file type detection by extension IIRC (yes, pretty lame :( ).

Rename the file to 3.264 or 3.h264.

[nm]

As is said in the MP4Box manual, you need to use the proper file extension for the raw files: .h264, .h26L, .264 or .26L. Otherwise MP4Box doesn't know what kind of files it is dealing with.

[R!tman]

mp4box does file type detection by extension IIRC (yes, pretty lame :( ).

Rename the file to 3.264 or 3.h264.
As is said in the MP4Box manual, you need to use the proper file extension for the raw files: .h264, .h26L, .264 or .26L. Otherwise MP4Box doesn't know what kind of files it is dealing with.
@GodofaGap, nm: You are both right. MP4Box checks extensions.
@me: RTFM!

Thanks to you, I can now use mplayer output directly with MP4Box. Unfortunately, I still get this error:
$ MP4Box -add 2ndpass.h264 video.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.000 FPS
MP4Box: media_import.c:3578: gf_import_h264: Assertion `nal_start' failed.
Aborted
It is aborted at ~63% again. I assume it has something to do with the 2GB which is approximately 63% of 3.2GB (filesize of the raw video).

I am pretty sure it's not the filesystem, otherwise I would hardly have the raw video file.

Any ideas, again?

[GodofaGap]

I know mplayer has a --enable-largefiles configure flag (for files larger than 2GB)... perhaps that is it?

[R!tman]

I know mplayer has a --enable-largefiles configure flag (for files larger than 2GB)... perhaps that is it?
GodofaGap, this already is part of my compile flags.

[GodofaGap]

Then I'm out of ideas regarding that...

I assume small files (<2GB) work? You could also try to pipe from mencoder to x264-cli perhaps? A bit tedious... I know...

[MeteorRain]

hm... could we encoded it to 2 parts, and join them when produce MKV? the append function in mkvmerge should work...

[R!tman]

Then I'm out of ideas regarding that...

I assume small files (<2GB) work? You could also try to pipe from mencoder to x264-cli perhaps? A bit tedious... I know...
GodofaGap, yes, smaller files work. I just tried a 2068MB file, encoded with "-noodml -of rawvideo". It fails at 99%. I am pretty sure now it's that 2GB thing.

Question remains: What causes that? Mpayer or MP4Box?
hm... could we encoded it to 2 parts, and join them when produce MKV? the append function in mkvmerge should work...
MeteorRain, actually, the movie is made up of 48 different smaller ones. It took me a hell of a lot time to put them together into one single video and, most importantly, in sync with the two sound streams. Furthermore, I joined the small file to one big in order to achieve a better overall bitrate usage of the whole movie.
(Imaging joining an action movie and a cartoon, each encoded individually with the same bitrate. The action movie would have the same bitrate, but lower quality than the cartoon. I hope you know what I mean...)
In essence, I will only reencode in two different parts if it's absolutly neccessary.
But I will consider, better try out right now, to cut the movie, create an .mp4 file and mux everything into .mkv.

Well, after dinner :-).

EDIT: I have to admin that I'd like a working MP4Box or mplayer (whatever causes the problem) much better than having to split and merge into .mkv...

[bond]

try another muxer, like mp4creator and avc2avi and check if those work, cause then we see if its a mp4box bug or not

moved

[R!tman]

try another muxer, like mp4creator and avc2avi and check if those work, cause then we see if its a mp4box bug or not

moved
bond, actually, I did try mp4creator. But as I did not manage to get it to work at all, I thought it was just stupid me not understanding the mp4creator syntax. So I dropped it. This is what I tested:

With the raw video stream:
$ rm test.mp4 && mp4creator -use64bits -rate=25 -verbose=2 2ndpass_noodml_raw.mp4v test.mp4
Write: type ftyp
Write: majorBrand = mp42
Write: minorVersion = 0 (0x00000000)
Write: Write: brand = mp42
brand = isom
Write: finished ftyp
mp4creator: buffer overflow, invalid video stream?
Segmentation fault
In the man page it says
...
These original media files are known as elementary stream (ES)
files in MPEG parlance.
...
I did not really know what that meant, but I did this
mencoder -of mpeg -ovc copy -nosound -o 2ndpass_noodml_mpeg.mp4v 2ndpass.x264
to ensure the movie would be in an mpeg container.
I got this then:
$ rm test.mp4 && mp4creator -verbose=2 -use64bits -rate=25 2ndpass_noodml_mpeg.mp4v test.mp4
Write: type ftyp
Write: majorBrand = mp42
Write: minorVersion = 0 (0x00000000)
Write: Write: brand = mp42
brand = isom
Write: finished ftyp
mp4creator: no VOSH header found in MPEG-4 video.
This can cause problems with players other than mp4player.
mp4creator: No VO header found in mpeg-4 video.
This can cause problems with players other than mp4player
mp4creator: fatal: No VOL header found in mpeg-4 video stream
Write: type moov
Write: type mvhd
Write: version = 0 (0x00)
Write: flags = 0 (0x000000)
Write: creationTime = 3224865677 (0xc0378b8d)
Write: modificationTime = 3224865677 (0xc0378b8d)
Write: timeScale = 90000 (0x00015f90)
Write: duration = 0 (0x00000000)
Write: rate = 1.000000
Write: volume = 1.000000
Write: reserved1 = <70 bytes>
00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00
00 00 00 00 00 00 00 00 00 00 40 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00
Write: nextTrackId = 1 (0x00000001)
Write: finished mvhd
Write: type iods
Write: version = 0 (0x00)
Write: flags = 0 (0x000000)
Write: objectDescriptorId = 1 (0x001) <10 bits>
URLFlag = 0 (0x0) <1 bits>
includeInlineProfileLevelFlag = 0 (0x0) <1 bits>
reserved = 15 (0xf) <4 bits>
ODProfileLevelId = 255 (0xff)
sceneProfileLevelId = 255 (0xff)
audioProfileLevelId = 255 (0xff)
visualProfileLevelId = 255 (0xff)
graphicsProfileLevelId = 255 (0xff)
esIds
ociDescr
ipmpDescrPtr
extDescr
Write: finished iods
Write: finished moov
Maybe it IS just me, please post a correct command line then.

I have not tested avc2avi, yet. This, I will do right now...

[GodofaGap]

I think mp4creator thinks your file is MPEG4 ASP... why did you change the extension to mp4v? Just use the raw file with a normal extension (e.g. .264)...

[R!tman]

I think mp4creator thinks your file is MPEG4 ASP... why did you change the extension to mp4v? Just use the raw file with a normal extension (e.g. .264)...
GodofaGap, this time I read the manual ;-).
There, it says:
...
-c, -create=<file>
Create a media track using <file>. File must have either .mp3 .aac .divx or .mp4v exten-
sion.
...
Otherwise, I get
$ mp4creator 2ndpass_raw.264 test.mp4
mp4creator: unknown file type
FYI, I have just finished my test with avc2avi. Is it normal this program takes so long? Well, I only found it as .exe, so maybe it's only M$ ;-).
But the important part is that it worked. Although I needed to use "-idx" to skip forward and backward within the movie (using mplayer), I could see every part (although not good until the next keyframe). As you can see here, the number of frames also corresponds to a movie of about 3h20mins.
avc2avi.exe -i 2ndpass.264 -o 2ndpass.avi
avi file written
- codec: h264
- size: 704x416
- fps: 25.000
- frames: 299608
So, what's up next? Shall I write an email to Jean Le Feuvre and report that gpac has a bug?

[R!tman]

I have to make an addition.

Once more, I went over everything I did to get MP4Box to work.

I am pretty sure now, that I cannot extract the raw video from the mplayer avi output, neither with (error at 1023MB due to RIFF chunks, I believe) or without odml (error at 2GB). This happens in linux and windows.

What I also know that does not work is to add a raw video directly from mplayer into an .mp4 file IN LINUX. It causes this error (again, something about 2GB):
$ MP4Box -add 2ndpass_raw.264 test_linux.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.000 FPS
MP4Box: media_import.c:3578: gf_import_h264: Assertion `nal_start' failed.
Aborted
But I had not tried this on windows. This indeed works! I could get a fully functional .mp4 file from it.

I am not sure how to proceed now, as it works with windows. The linux versions I tried were gpac-cvs-0.1-r2 from kpex's tarball and the latest release from portage (0.4.1_pre20060122), both on amd64 and x86. The windows version was 0.4.1.

[bond]

raw avc streams need to have the .264 extension
raw asp streams need to have the .m4v extension

try the following commandline:
mp4creator -c raw.264 -rate 25 mp4creator.mp4

[R!tman]

raw avc streams need to have the .264 extension
raw asp streams need to have the .m4v extension

try the following commandline:
mp4creator -c raw.264 -rate 25 mp4creator.mp4
bond, first I wanted to hint you to the extract of mp4creator's man page in the first post of the second page. That man page was for mpeg4ip-1.1. Well, the man page has not changed for mpeg4ip-1.4.1, which I tried out before posting here, but the output of the command you posted has :-). With 1.1 it did not work ("unknown filetype", see above), but now it works:
$ mp4creator -c raw.264 -rate 25 mp4creator.mp4
Error decoding sei message
Although I get this "Error decoding sei message" (whatever that means), the output .mp4 file is just fine.

I guess we discovered that MP4Box has a problem importing big raw files, at least the linux version. I don't know what's with MP4Box regarding the extraction of the raw video from avi, but right now I don't really care about that. I am happy that we got this working, thank you all very much!

Of course, I would like to offer my help in resolving that bug with MP4Box. Do you think it is appropriate to inform the author of gpac of this?

[bond]

two things:

1) post the link to the mp4box compile you used
2) make sure you have twice the space of your file on your disc. mp4box does some sort of "twopass" muxing for being able to optimize the file and use the smallest headers possible aso and this needs twice the space which maybe can be a problem with very big files

[R!tman]

two things:

1) post the link to the mp4box compile you used
2) make sure you have twice the space of your file on your disc. mp4box does some sort of "twopass" muxing for being able to optimize the file and use the smallest headers possible aso and this needs twice the space which maybe can be a problem with very big files
bond,

1) I did a usual "emerge gpac" on my gentoo system, so, I compiled it myself. Of course with the correct keywords set, as it is still marked testing. I mainly used gpac-0.4.1_pre20060122. See

http://packages.gentoo.org/search/?sstring=gpac
http://www.gentoo-portage.com/media-video/gpac

On the second link, click on "view". There you can see the ebuild and its standart compile option.
My own compiler flags are "-O2 -pipe -march=k8", so quite conservative.

2) I think this speaks for itself ;-)
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md/1 447G 334G 113G 75% /
udev 1006M 372K 1006M 1% /dev

[bond]

22.1 is outdated, there has been a 4gb and more problem fixed since then

check out the latest gpac cvs and compile this

[R!tman]

22.1 is outdated, there has been a 4gb and more problem fixed since then

check out the latest gpac cvs and compile this
bond, using kpex's cvs-ebuild (https://kpex.no-ip.org/kpex-media/media-video/gpac-cvs/gpac-cvs-0.1-r2.ebuild) for gentoo, I installed gpac-cvs. But I still get same problem.
$ MP4Box -add raw.264 test.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.0000 FPS
Importing: |============ | (63/100)
until here everything looks fine, but it never went beyond 63%. Instead it turn into this:
$ MP4Box -add raw.264 test.mp4
Adjusting AVC SizeLength to 16 bits
AVC-H264 import - frame size 704 x 416 at 25.0000 FPS
MP4Box: media_import.c:3638: gf_import_h264: Assertion `nal_start' failed.
Aborted

[bond]

sure you used latest cvs? and not some 2 weeks old sources or so?

[R!tman]

sure you used latest cvs? and not some 2 weeks old sources or so?
bond, I am pretty sure I have the latest cvs. But I have to admit, I am no big fan of cvs and usually try to avoid it. Therefore my knowledge of cvs is very limited.
I always guessed using cvs would automatically intall the latest version.

This is what happens when I install gpac-cvs:
# emerge gpac-cvs 1>gpac-cvs_stout.emerge 2>gpac-cvs_sterr.emerge
For better readability, I did not post these two files here.
gpac-cvs_stout.emerge (http://n.ethz.ch/student/homartin/download/gpac-cvs_stout.emerge)
gpac-cvs_sterr.emerge (http://n.ethz.ch/student/homartin/download/gpac-cvs_sterr.emerge)

[bond]

well if the gpac code you used is older than 1st march you are propably seeing the bug that has been fixed already

if you are using newer code you have found a new bug and therefore make a bug report on the gpac bug tracker plz

[R!tman]

well if the gpac code you used is older than 1st march you are propably seeing the bug that has been fixed already

if you are using newer code you have found a new bug and therefore make a bug report on the gpac bug tracker plz
bond, as I was using cvs, I am pretty sure it was the latest version.

I took a look at the gpac bugs, and tried to find the "4GB bug" you mentioned. The only one I found that fit was this one:

1397776 RAW AVC import failure * 2006-01-05 06:27 5 Open nobody nobody (http://sourceforge.net/tracker/?group_id=84101&atid=571738)

As you can see, it is still open. I guess that is the issue.

[bond]

hm the guy used a bitrate of 724, i dont think he will get above 4gb with that?

you maybe want to start a new bug report?

[R!tman]

hm the guy used a bitrate of 724, i dont think he will get above 4gb with that?

you maybe want to start a new bug report?
bond, you're probably right, with that bitrate he would have to have a looong movie :-).

I created a new bug report and included a link to this topic.