x264: AQ + RDO + 8x8dct + bitrate mode => crash

[berrinam]

I have been trying to encode the Harry Potter trailer as hpn described in this post (http://forum.doom9.org/showthread.php?p=727140#post727140). However, x264 consistently crashed. I have been able to narrow it down to the three features mentioned in the title, and it doesn't appear to crash in CQ mode. I have tried it on two computers as well as other sources, and it consistently crashes.

Symptoms: x264 will run for a while, seemingly fine, then it will stop without any error message, but not printing any final stats as it normally does. The resulting file is unplayable. With the commandline given below, it always stops at frame 175.

CPUs:
1. P4 3GHz with HT.
2. Mobile P4 (laptop) 1.6GHz.

To reproduce:
The source, including the avs file, is identical to what hpn described, where I linked to above. The various commandlines I used which resulted in this crash all included AQ, RDO, 8x8dct, and weren't CQ mode (I haven't tried --crf). Note: only together do the options produce a crash. Individually, or in pairs, they work fine. One example which crashes is:
c:\x264 --bitrate 700 --aq --subme 6 --analyse all --8x8dct --progress --output "c:\hp\hp.mp4" "c:\hp\trailer.avs"

Has anyone else come across this problem? The avisynth file encodes fine with xvid/mencoder, and by getting rid of any individual feature from the x264 encode, it also works fine.

@edit: I have tried using the --no-asm flag, and it appeared to have no effect on the problem.

[ChronoCross]

I wasn't able to reporduce it on my machine. Anyone else?

[Japhsoncross]

i also have met this problem, and even if i stop before crash, there're gray blocks can be found in the encoded frames.

[bond]

you could also try encoding with exactly the same settings to .264 or .mkv and report if it crashes too plz

[berrinam]

Results are exactly the same with .264 and .mkv output as with .mp4 -- it still crashes at frame 175.

[Sharktooth]

i was unable to reproduce the crash too... berrinam, what build are you using?

[hpn]

I can confirm the crash with your command line. The build is x264-Lite_r333E by Sharktooth.

G:\333_test>x264 --bitrate 700 --aq --subme 6 --analyse all --8x8dct --progress --output s.mp4 b.avs
avis [info]: 960x400 @ 23.98 fps (3223 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 0 (scale 2500000)
encoded frames: 149/3223 (4.6%), 6.79 fps

with --subme 7 it crashes at frame 155

The crash occurs only when I use --aq, so it may be a bug in x264 or the raw .264 or the .mp4 produced by MP4Box.exe is borked. These mov trailers are hard to deal with, so I'll try the same "--aq" command line with some of the other trailers on this list (http://forum.doom9.org/showthread.php?p=724530#post724530) and see if it will crash again

[Sharktooth]

try rev334 and builds from x264.nl too.
...maybe it's the RDO patch that crashes...

[unmei]

Symptoms: x264 will run for a while, seemingly fine, then it will stop without any error message, but not printing any final stats as it normally does. The resulting file is unplayable.
I had something similar with --crf when i encoded only 252 frames it would seemingly stop at frame 37, but actually the clip was encoded all the way and played correctly - only the CLI output stopped there and did not display the stats at the end [it hanged and had to be escaped with ctrl+break]. Encoding the same thing, same parameters but only with AVS trim modified to encode twice the frames it worked flawlessly. So in this case (not bitrate-mode) it seems only the be a problem of too short of an input clip (but might still be related ;)).
It was with --aq, --8x8dct and --subme 7 in revision 333E as well. I didn't investigate further because giving it a little longer clip was OK (and 252 frame clips are not exactly what you encode to keep).

[hpn]

try rev334 and builds from x264.nl too.
...maybe it's the RDO patch that crashes...
I tried with your rev334 and this time it reports
"unknown option ( )" and then exits.

Seems --aq is the problem cause without it everything encodes fine. I've never used --aq before, so I don't know if the problem is new due to the latest patches/changes or it has been around for a while. I also just compiled 334 from SVN without any extra patches and get the same error: --aq -> unknown option ( )

[hpn]

I found the problem. In the SVN rev334 there is still no --aq option at all. It is now only present in the ST patched builds, plus in rev334 it takes -strength and -sensitivity to work:

--aq-strength <float> Amount to adjust QP by AQ:
0.0 => no AQ,
1.1 => strong AQ [0.50]
--aq-sensitivity <float> "Flatness" threshold to trigger AQ [15.0]
5: applies to almost all blocks
22: only flat blocks

With rev334 by ST and --aq-strength 0.5 this command still crashes at frame 155:

x264 --bitrate 700 --aq-strength 0.5 --subme 6 --analyse all --8x8dct --progress --output b.mp4 b.avs
encoded frames: 155/3223 (4.8%), 5.59 fps

EDIT: another test with --aq-strength 0.1 works! so the problem is how much -strength is applied. I guess the --aq code needs some more tweaking (that's why it's not yet committed to the SVN). It's also a good idea ST to remove the "Adaptive quantization V3" patch from 334 until it's tested and confirmed working under all conditions. RD patch V2 seems to work fine and --subme 7 produces a nice 0.2 PSRN gain, so I quess it will be soon committed to the SVN.

EDIT 2: I'm unable to reproduce the "--aq-strength 0.5" crash with a regular DVD encode (4500 frames test avs encodes fine, I can't confirm about a full movie), so the crash also depends on the source.

[Sharktooth]

--aq does no longer exists. with the new patch, just set --aq-strength to something different than 0 and aq gets enabled.

[MeteorRain]

at floor #7 in this post:
G:\333_test>x264 --bitrate 700 --aq --subme 6 --analyse all --8x8dct --progress --output s.mp4 b.avs
avis [info]: 960x400 @ 23.98 fps (3223 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE 3DNow!
mp4 [info]: initial delay 0 (scale 2500000)
encoded frames: 149/3223 (4.6%), 6.79 fps
Did you guys see the line below?
mp4 [info]: initial delay 0 (scale 2500000)

[berrinam]

Thanks for pointing that out, MeteorRain. Adding assumefps(25) at the end of my avisynth script seems to fix the problem.

[bond]

but your stream is 23.976fps right?

still its funny that we again see problems with the 23.976 framerate in mp4 but also with raw and mkv!!!

[MeteorRain]

OH MY!

berrinam:
see this thread!

[bond]

i reproduced the crash
with strength 1.0 it crashed at frame 155
with strenght 0.7 it crashed at frame 158
with strength 0.5 it crashed at frame 179
with strength 0.1 it worked fine

adding assumefps(23.976) with strenght 1.0 showed exactly the same crash
adding assumefps(2997,125) with strenght 1.0 showed exactly the same crash
adding assumefps(24000,1001) with strenght 1.0 showed exactly the same crash

adding assumefps(25) with strenght 1.0 crashed at frame 175

using directshowsource(fps=25) instead of assumefps(25) with strenght 1.0 crashed at frame 172 (haali+ffdshow seem to pass less frames in this case tough)


its exactly the same with mkv, raw and mp4!

i used the same cmdl as berrinam and hpn with sharktooths 334a build and haalis media parser 11 oct (its funny that the tray icon is displayed twice when encoding!?) and cd's ffdshow 13 oct

[Sharktooth]

The AQ patch is broken.

[ChronoCross]

The AQ patch is broken.

I have to disagree with you there sharktooth. It's running fine in my configuration. Been running a 23 minute episode through MeGUI in 3 pass mode and it finished properly. The framerate of my encode is 23.997fps in native mp4.

[bond]

ChronoCross, read the crash reports of 3 different people in this thread again ;)

[ChronoCross]

ChronoCross, read the crash reports of 3 different people in this thread again ;)

yeah I did but I mean isn't it strange that it's broken for some...and not for others? Maybe there is another underlying issue.

[bond]

yeah I did but I mean isn't it strange that it's broken for some...and not for others? Maybe there is another underlying issue.if you would read the posts in detail you will see that aq borks on some settings and on others not...

[hpn]

A few more observations: I tried the same --aq-strength encode with the King Kong (http://movies.apple.com/movies/universal/king_kong/king_kong_h1080p.mov) trailer and now the crash is at frame 400. King Kong has even bigger resolution 1920x1080p. These crashes tend to occur with HD content and not likely with a normal DVD, at least It doesn't crash with my test DVD encode.

I also just upgraded the Haali splitter to the latest 25/10/2005 build and now when I try to encode the avs I get the following error:

avis [error]: unsupported input format (DIB )
could not open input file 'b.avs'

Downgrading to a previous Haali build (06/10/2005) fixes this error, but of course the Harry Potter avs -> x264.exe encode keeps crashing at frame 175 (the new Haali seems buggy! EDIT: The bug was just fixed in the next version). I also tried to circumvent the haali splitter during the encode, so from the avs I created an uncompressed "hp.avi with y4m stream inside (a big 1.9GB file cause I couldn't remember how/if possible to pipe from avs2yuv.exe to x264.exe directly :D)

Because this avi is uncompressed it means that when I encode it I should get identical encode as if I encode from the avs, but this time the haali splitter will not be triggered:

x264 --bitrate 700 --aq-strength 0.5 --subme 6 --analyse all --8x8dct --progress --output hp.mp4 hp.avi

Again crash at frame 175 but this time the crash triggers the VS debugger:

Unhandled exception at 0x00432d56 in x264.exe: 0xC0000094: Integer division by zero: disassembly:

00432D4D xor edx,edx
00432D4F mov eax,dword ptr [esp+2A50h]
error ->00432D56 div eax,dword ptr [ebx+18h]
00432D59 mov edx,ebx
00432D5B inc eax
00432D5C mov dword ptr [esp+64h],eax ]

And the Call Stack window reports:
[Frames below may be incorrect and/or missing, no symbols loaded for x264.exe]

I have no idea what all these things mean except for the obvious division by zero that happens at run-time.

EDIT: Haali just fixed the splitter and posted 25/10/2005 "second edition", so no need to install the old haali version now