x264 mp4 output borked (rev 295) (mkv fixed)

[stephanV]

No problems with that here...

[Doom9]

(x264 --> mp4) --> mkv = failThis works like a charm over here, using Sharktooth's 292a build and mkvtoolnix 1.5.6. It sounds more and more like a problem at your end with each test somebody else makes ;)

[MeteorRain]

Hey, did anyone test my sample?

and i met more problems now:

i compressed a lot of avi files, simply using avisource. x264 just compressed about half of the file, and stopped and said successfully compressed but the result file is obviously completely broken.

all jobs failed except only one works.......

see attachment for avs&log

[Sirber]

This works like a charm over here, using Sharktooth's 292a build and mkvtoolnix 1.5.6. It sounds more and more like a problem at your end with each test somebody else makes ;)Well, I don't get it. I'm using all latest builds and it fails. MPC dies on me.

+ EBML head
+ Segment, size 108949416
|+ Seek head (subentries will be skipped)
|+ EbmlVoid (size: 4027)
|+ Segment information
| + Timecode scale: 1000000
| + Muxing application: libebml v0.7.5 + libmatroska v0.7.7
| + Writing application: mkvmerge v1.5.6 ('Breathe me') built on Sep 7 2005 18:18:22
| + Duration: 1731.512s (00:28:51.512000000)
| + Date: Sat Sep 17 10:39:14 2005 UTC
| + Segment UID: 0xa2 0xd9 0x45 0x6c 0xef 0x5e 0xe8 0x01 0xbb 0x3a 0x2c 0xa9 0xd
9 0xef 0x62 0xc9
|+ Segment tracks
| + A track
| + Track number: 1
| + Track UID: 905473676
| + Track type: video
| + Default flag: 1
| + Forced flag: 0
| + Lacing flag: 0
| + MinCache: 1
| + Timecode scale: 1.000000
| + Max BlockAddition ID: 0
| + Codec ID: V_MPEG4/ISO/AVC
| + CodecPrivate, length 38
| + Default duration: 10.347ms (96.646 fps for a video track)
| + Language: und
| + Video track
| + Pixel width: 704
| + Pixel height: 400
| + Display width: 704
| + Display height: 400
| + A track
| + Track number: 2
| + Track UID: 3167492503
| + Track type: audio
| + Default flag: 1
| + Forced flag: 0
| + Lacing flag: 1
| + MinCache: 0
| + Timecode scale: 1.000000
| + Max BlockAddition ID: 0
| + Codec ID: A_AAC/MPEG4/LC/SBR
| + Default duration: 46.440ms (21.533 fps for a video track)
| + Language: und
| + Audio track
| + Sampling frequency: 22050.000000
| + Channels: 2
| + Output sampling frequency: 44100.000000
|+ EbmlVoid (size: 1024)
|+ Cluster

[Sirber]

MP4 in MKV, it seems that it's libavcodec.dll that crash.

[stephanV]

@Meteorain:

Yes, it crashes here too in both mplayer (latest CVS) and DirectShow. So it is either a bug in x264 or libav. Of course this raises the question: what settings did you use?(AVS scipt and commandline)

(Don't post them as attachment please)

[movax]

We had reports of borked libavcodec_dec.dll with similar troubles in the last CCCP, solved my going to the latest CVS non-dec version.

@Sirber, I think it has to be something at your end...I just used 292A to complete your (x264 --> mp4) --> mkv test as well.

[Sirber]

Where could I get such a build?

[MeteorRain]

@stephanV

let me just post the avs and log for another sample (meet exact the same problem i bet)

avisource("[raw]DearS 01 (DVD 640x480 DivX5.11 QB95 24fps).avi",false)Next job job1 is a video job. encoder commandline:
"H:\mp4\x264.exe" --qp 23 --ref 4 --bframes 10 --b-pyramid --nf --weightb --analyse all --8x8dct --ipratio 1.0 --pbratio 1.0 --me umh --progress --no-psnr --output "I:\hd200\disk2\subgroup\DearS\1.mp4" "I:\hd200\disk2\subgroup\DearS\1.avs"
successfully set up video encoder and callbacks for job job1
----------------------------------------------------------------------------------------------------------

Log for job job1

avis [info]: 640x480 @ 23.98 fps (34049 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
mp4 [info]: initial delay 2000000 (scale 23976023)
This is a CQ job so there's no desired bitrate. Obtained video bitrate: 179.386766222332 kbit/s
----------------------------------------------------------------------------------------------------------The above is the corrupt sample.



And an OK sample:
avisource("[raw]DearS 10 (DVD 640x480 DivX5.11 QB95 24fps).avi",false)Next job job10 is a video job. encoder commandline:
"H:\mp4\x264.exe" --qp 23 --ref 4 --bframes 10 --b-pyramid --nf --weightb --analyse all --8x8dct --ipratio 1.0 --pbratio 1.0 --me umh --progress --no-psnr --output "I:\hd200\disk2\subgroup\DearS\10.mp4" "I:\hd200\disk2\subgroup\DearS\10.avs"
successfully set up video encoder and callbacks for job job10
----------------------------------------------------------------------------------------------------------

Log for job job10

avis [info]: 640x480 @ 23.98 fps (34046 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
mp4 [info]: initial delay 250 (scale 2997)
x264 [info]: slice I:314 Avg QP:23.00 Avg size: 25089
x264 [info]: slice P:15512 Avg QP:23.00 Avg size: 8242
x264 [info]: slice B:18220 Avg QP:23.00 Avg size: 2028
x264 [info]: slice I Avg I4x4:39.8% I8x8:31.4% I16x16:28.7%
x264 [info]: slice P Avg I4x4:3.4% I8x8:3.8% I16x16:4.2% P:32.0% P8x8:6.1% PSKIP:50.4%
x264 [info]: slice B Avg I4x4:0.4% I8x8:0.3% I16x16:0.3% P:10.5% B:2.7% B8x8:1.2% DIRECT:3.6% BSKIP:81.0%
x264 [info]: 8x8 transform intra:32.6% inter:33.4%
x264 [info]: kb/s:972.8

Actual bitrate after encoding without container overhead: -105575.16
This is a CQ job so there's no desired bitrate. Obtained video bitrate: 974.550695947835 kbit/s
----------------------------------------------------------------------------------------------------------

[stephanV]

Well... you're settings are a bit ackward, but that should never cause a crash of course...

Maybe its the combination of many b-frames and b-pyramid that libav doesnt like... do you have the Nero decoder to test? (I guess not)

[Haali]

Looks like the problem is in integer overflow when AVI dwScale is too high (>23 millions in this case).

[stephanV]

Haali is obviously better at guessing than me :)

http://savefile.com/files/2005757 <--- test file

AVS script:
Avisource("test.avi")
assumefps(3.142)
x264 commandline:
x264 -o test.mp4 test.avs

Resultant mp4 will crash. Resultant mkv will play, but can't be remuxed with MKVmerge and remuxes weird with AMG.

You probably can use any weird non-integer frame rate to emulate the problem (just make sure you use enough frames too).

[movax]

Non-integer framerate...so 23,967 and the other standard rates will bork that test file? /me wonders what went into that file

[stephanV]

it doesnt b0rk the test file, it will probably b0rk any file, the file itself is nothing special.

[movax]

Ahkay, I think I see what you're getting at, x264 output to mp4 borks as a result of AssumeFPS(non-integer) in the input AVS.

[Sirber]

All my files are use "non-integer fps" and are "b0rked", unless I use RAW. I think we are on the path of bug killing :D

[Doom9]

when AVI dwScale is too high (>23 millions in this case).but which integer would overflow? the framerate is a double (uint dwRate / uint dwScale). However, a dwScale of 23 million is unusual.. a proper 23.976 fps file has a dwRate of 23976 and a dwScale of 1000, a 29.97 fps one would have 29970 and 1000 respectively.

So it's not so much about non integer FPS streams, but possibly those that have weird dwScale (and dwRate?) values. Here's what I get if I add assumefps(3.142) to a perfectly working 23.976 fps script: dwRate = 13178503, dwScale = 4194304. I'm encoding now to see what the results are (I used that same script without assumefps and successfully created a properly working mp4 and mkv (that can be remuxed) from it))

Those that can debug MeGUI.. open a script, set a breakpoint in AviReader.cs at the line
framerate = (double)streamInfo.dwRate/(double)streamInfo.dwScale;
and check out dwRate and dwScale for clips that don't work.

[stephanV]

but which integer would overflow? the framerate is a double (uint dwRate / uint dwScale). However, a dwScale of 23 million is unusual.. a proper 23.976 fps file has a dwRate of 23976 and a dwScale of 1000, a 29.97 fps one would have 29970 and 1000 respectively.
Not exactly since the right frame rates for NTSC are 30000/1001 and 24000/1001 IIRC. But that doesn't change that it shouldn't use such a weird number if it is using dwScale as scaling factor, but it seems to me it isn't doing that at all.

[Haali]

but which integer would overflow?
x264 calculates frame_number*dwScale in one place

[Doom9]

1001 is used on DVD (probably stems from the fact that film is 24fps and if you do 3:2 pulldown you have to use drop frames so it'll be 29.97) VFW seems to be using the more straightforward values.

Either way, not surprisingly my mp4 ended up being corrupt as well and mkvmerge complained upon importing:
mkvmerge v1.5.6 ('Breathe me') built on Sep 7 2005 18:18:02
'D:\DVDs\KILL_BILL_VOL2\VIDEO_TS\chapter1.mp4': Using the Quicktime/MP4 demultiplexer.
Warning: 'D:\DVDs\KILL_BILL_VOL2\VIDEO_TS\chapter1.mp4' track 1: The AVC video track is missing the 'CTTS' atom for frame timecode offsets. However, AVC/h.264 allows frames to have more than the traditional one (for P frames) or two (for B frames) references to other frames. The timecodes for such frames will be out-of-order, and the 'CTTS' atom is needed for getting the timecodes right. As it is missing the timecodes for this track might be wrong. You should watch the resulting file and make sure that it looks like you expected it to.

[Sirber]

I made a little test and the (x264 --> mp4) doesn't work, it crash. if I sue MKV instead, output work but is unmergable in another mkv.

[edit]

Source FPS: 23.976

[MeteorRain]

well, i should mention, the log that i post yesterday indicated that the encoding process stoped unexpectly. that means, the x264 crashed on encoding.

i'm thinking of: source fps 23.976 maybe 23.976023. but we usually see a %.3f result, so we can't figure out the difference between 23.976 or 23.976023

23.976 = 23976 / 1000 = 2997 / 125
23.976023 ~=~ 24000 / 1001

(IMHO) the avi should have a fps value 24000 / 1001 but had a value 23976023 / 1000000, which makes it overflow again and again.

[Doom9]

before anyone posts here again saying something doesn't work, make damned sure you post your dwRate and dwScale values.

(IMHO) the avi should have a fps value 24000 / 1001I doubt AVIFile would round, that's why it uses those big values.

[Sirber]

how can I find it (whitout MeGUI :))?

[MeteorRain]

before anyone posts here again saying something doesn't work, make damned sure you post your dwRate and dwScale values. how to get the 2 values?
btw, i tried adding assumefps(23.976), how can i write the 2 values? and the result file corrupt too!

I doubt AVIFile would round, that's why it uses those big values.
how to resolve it? >_<

[bob0r]

For Siber's problem, could gpac 0.3.x or 0.4.x make any difference?
Or was that already tested? Sharktooth uses 0.4.x, i use whatever the latest x264 svn supports, which for these builds is 0.3.x.

[Sirber]

My problems started with 0.3x and continued with 0.4x. I can't find the old thread about that :(

[Doom9]

how to get the 2 values?debug megui the way I wrote.. or extract the code to a separate app. I could provide one if necessary, but it'll still be .NET of course.

whitout MeGUIOpen an AVI File in Delphi and read those values?

[Sirber]

I think I can get them both. Gonna try tonight.

[Doom9]

r295 | robux4 | 2005-09-20 18:18:23 +0200 (Tue, 20 Sep 2005) | 1 line

fps patch by HaaliMight have something to do with it but looking at the source changes I'm not so sure. Still, those AVIs with these values are pretty darned weird.

[Sirber]

So, if I get it correctly, some borked tools produced those AVI which bork x264?

[Sirber]

old: pic.i_pts = i_frame * param->i_fps_den;
new: pic.i_pts = (int64_t)i_frame * param->i_fps_den;
From the changelog using 64bit integer should fix the problem for most case. I will do some tests tonight.

[stephanV]

Nope, that didnt fix it. :(

[Haali]

This may be not enough to make mp4 output work, but matroska should be fine now.

[stephanV]

Aaaah yes. Very good. :)

[Sirber]

I'm not asking more :D
Testing...

[Sirber]

@Haali

Problem fixed for MKV. Many thanks!!!!!! :D

Produced MKV is 100% valid.

[MeteorRain]

@Haali

Problem fixed for MKV. Many thanks!!!!!! :D

Produced MKV is 100% valid.
is it still ok when you remux the mkv with audio using mmg?

[Sirber]

yep, all #1. :D I waited for this for months! :D :D :D

I'll be able to continu my work on RealAnime LE. :D

[bob0r]

Now for .mp4, does x264 need to change code, or does gpac need to change code?

[leowai]

Now for .mp4, does x264 need to change code, or does gpac need to change code?
@Haali

Problem fixed for MKV. Many thanks!!!!!! :D

Produced MKV is 100% valid.
From my point of view, since x264 raw-> mp4 works using MP4Box, it's more to the bug in porting x264 with gpac's mp4 output support.

Correct me if I'm wrong. Thanks.

Just curious about this: What's the different between x264 encoder and MP4Box when creating a *.mp4 output? If they are same (using same function), does it means that's the problem in x264 using gpac's mp4 output function?

[Sirber]

For (x264 raw-> mp4), you need to know the FPS, which cound have been altered by user avisynth script, and bork the result MP4. It's best to have it handeled by x264 directly.

[bob0r]

For (x264 raw-> mp4), you need to know the FPS, which cound have been altered by user avisynth script, and bork the result MP4. It's best to have it handeled by x264 directly.

Hmm yes ofcourse.

psp.avs:

LoadPlugin("i:\cap\_x264\_dgdecode.dll")
LoadPlugin("i:\cap\_x264\_decomb521.dll")
LoadPlugin("i:\cap\_x264\_mpasource.dll")
v=mpeg2Source("i:\cap\_x264\cap.d2v")
a=mpasource("i:\cap\_x264\cap.mpa")
audiodub(v,a)
fielddeinterlace()
crop(4,2,712,574)
LanczosResize(368,208)
ChangeFPS(29.97)


psp.bat:

start /belownormal /b /w x264 --progress --pass 1 --me dia --ref 1 --subme 1 --analyse none --output NUL psp.avs
start /belownormal /b /w x264 --progress --pass 2 --bitrate 512 --ref 1 --subme 6 --analyse all --output video.264 psp.avs
start /belownormal /b /w MP4Box.exe -fps 29.97 -add video.264 output.mp4
start /belownormal /b /w MP4Box.exe -fps 29.97 -add audio.aac output.mp4
start /belownormal /b /w ATOMChanger.exe output.mp4 MAQ00264.MP4 AtomAVC.ini


I'll try setting the FPS in the x264.exe line too; --fps <float|rational> Specify framerate

--fps 29.97, and then maybe it "fixes" the .mp4 file, ill try

Note the _detected_ fps here:

start /belownormal /b /w x264 --progress --pass 1 --me dia --ref 1 --subme 1 --analyse none --output NUL psp.avs
avis [info]: 368x208 @ 29.97 fps (9539 frames)



If i remove ChangeFPS(29.97) from the .avs and --fps 29.97 via x264.exe, will x264 then convert the 25FPS PAL to 29.97?

[movax]

I think both will certainly convert the framerate, but the telecining/assorted processes that have been done to the PAL video can't be easily fixed by a simple framerate change. (e.g., a 24fps source to 25fps PAL is just speeded up, this is common, but then converting that to 29,97 would be akin to telecining 24fps video, I'm kind of sure.)

[bob0r]

@movax
Ah yes, well the videos are for Sony PSP, and they work fine with MP4Box, its purely about a working .mp4 now, converting the right way may come later (unless it influences the .mp4 output)

I see i have made a small mistake; avis [info]: 368x208 @ 25.00 fps (7957 frames) is just how many FPS the .avs file is outputting.
I am now trying --fps via x264.exe and ill send the files to my friend to test on his PSP.

avs change 29.97 FPS > .mp4 PSP does not work
avs change 29.97 FPS + x264.exe --fps 29.97 = Needs to be tested
avs 25 FPS (original source) + x264.exe --fps 29.97 = Needs to be tested
... will let you know. (If all fails on PSP, ill remux the audio/video with MP4Box)
Thanks.

Edit:
It seems x264.exe --fps 29.97 does NOT change the video fps, i wonder what it does do :)
Or is it the same as MP4box, just to set the .mp4 FPS?
In that case, .avs changefps 29.97 + x264.exe .mp4 output --fps 29.97, should create a valid file?

[Sirber]

PSP encoding is kinda OT...

[bob0r]

PSP encoding is kinda OT...

Not really, it can be used perfectly for .mp4 output testing.

I just encoded:
.avs ChangeFPS(29.97) and x264.exe --fps 29.97, and indeed the .mp4 output file is fucked, so adding --fps 29.97 is indeed b0rked.

Atomchanger.exe goes besoik on this output.mp4!

[bond]

seems jeanlf fixed this framerate problem with gpac:
I just fixed 64-bit ts support in isomedia and other parts of GPAC. You should now be able to import with pretty much any kind of timescales and dts - just checked it with a file lasting 4 days with only 31 frames.can sharktooth or celtic_druid plz make a new compile with latest gpac and anyone who had a crashing sample file test whether it is now fixed?

[Kurtnoise]

You can grab my last compile here (http://kurtnoise.free.fr/mp4tools/MP4Box_20051203.zip)...

[Sharktooth]

that should fix the problems in x264 too...

EDIT: i've just updated gpac and i get tons of undefined references... and obviously it doesnt compile.