x264 r1041 - Segmentation Fault

[LoRd_MuldeR]

I just compiled x264 r1041 as a DLL to use it with Avidemux. Did everything as I do it all the time.

But the build will crash right after the encoding process starts:

Program received signal SIGSEGV, Segmentation fault.
[Switching to thread 4236.0xf7c]
0x67619a45 in block_residual_write_cabac (h=<value optimized out>,
    cb=0xb83d380, i_ctxBlockCat=<value optimized out>, i_idx=1, l=0x5d28ec0,
    i_count=16) at ./common/bs.h:232
232             return x264_ue_size_tab[(val+1)>>8] + 16;
(gdb) bt
#0  0x67619a45 in block_residual_write_cabac (h=<value optimized out>,
    cb=0xb83d380, i_ctxBlockCat=<value optimized out>, i_idx=1, l=0x5d28ec0,
    i_count=16) at ./common/bs.h:232
#1  0x67629d50 in x264_rd_cost_part (h=0x5d272c0, i_lambda2=9289, i4=1,
    i_pixel=6) at encoder/cabac.c:1109
#2  0x67635761 in x264_me_refine_qpel_rd (h=0x5d272c0, m=0xb83e2f0,
    i_lambda2=9289, i4=1, i_list=0) at encoder/me.c:1000
#3  0x6762f9e6 in x264_macroblock_analyse (h=0x5d272c0)
    at encoder/analyse.c:2415
#4  0x676573d5 in x264_slice_write (h=0x5d272c0) at encoder/encoder.c:1177
#5  0x6766e5a4 in x264_stack_align () from d:\Avidemux 2.4\libx264-65.dll
#6  0x676530cd in x264_slices_write (h=0x5d272c0) at encoder/encoder.c:1326
#7  0x611812fa in ptw32_threadStart@4 () from d:\Avidemux 2.4\pthreadGC2.dll
#8  0x77bcb530 in msvcrt!_endthreadex () from C:\WINDOWS\syswow64\msvcrt.dll
#9  0x7d4dfe21 in KERNEL32!GetConsoleOutputCP ()
   from C:\WINDOWS\syswow64\kernel32.dll
#10 0x09301230 in ?? ()
#11 0x00000000 in ?? ()
(gdb)

Compiler is GCC 4.3.2-tdm-1, which used to work fine up to (including) r1038. I did not use any custom CFLAGS for the debug build.

No patches applied, except for the alignment fix (as required to get a working DLL). Any ideas?

[J_Darnley]

I am also experiencing a crash, but not with a debug build, that runs just fine. Using: x264 --crf 18 -b3 -m9 -r5 --me umh -8 -t1 -A all --mixed-refs -w --b-pyramid --direct auto --no-fast-pskip D:/Avisynth/fam-guy-ptv.y4m --progress -o NUL
causes a crash on the 5th frame. It happens with subme 8 or 9. Is it the same for you LoRd_MuldeR?

I will post more when I have more. I am also waiting for a reply from Dark Shikari or pengvado on the IRC channel.

[EDIT] --no-cabac seems to make it all better

[burfadel]

Since the build on www.x264.nl isn't updated to 1041 yet, I used the 1041 build from techouse (www.x264.tk). It crashed almost straight away for me as well, it did get to the 40th frame though.

On testing, --no-cabac works, and with cabac its only subme 8 and 9 that don't work.

[LoRd_MuldeR]

So I assume the current git version is borked...

[Sharktooth]

seems so. stick with r1040 until devs fix it.

[LoRd_MuldeR]

Originally Posted by Sharktooth

seems so. stick with r1040 until it gets fixed.

Aye aye, Sir

[kemuri-_9]

i was playing around with the config settings, since i was able to get a debug build working just fine (so it's not just the code but how it's compiled as well).

I nailed it down to:
w/ -fomit-frame-pointer = crash
w/o = fine

[ACoolie]

Originally Posted by kemuri-_9

i was playing around with the config settings, since i was able to get a debug build working just fine (so it's not just the code but how it's compiled as well).

I nailed it down to:
w/ -fomit-frame-pointer = crash
w/o = fine

I'm getting no crashes with:
CFLAGS=-O4 -ffast-math -Wall -I. -falign-loops=16 -mdynamic-no-pic -DHAVE_MMX -DARCH_X86 -DSYS_MACOSX -DHAVE_PTHREAD -s -fomit-frame-pointer

[kemuri-_9]

i tried with the -falign-loops=16 and it still crashes,
we don't have the -mdynamic-no-pic on windows.
so it might just be isolated to windows as well.

Edit:
Tried it on an available linux box, and it works there as well.
So it does seem to be isolated to windows.

Edit Edit:
Crashes when compiled with MSVC9 as well.
so this definitely seems to be a secluded windows problem.

[komisar]

On my "patched" 1041-build no crashes.

CFLAGS=-O4 -ffast-math -DNDEBUG -Wall -I. -march=xxx -mtune=xxx -mno-cygwin -mwin32 -m32 -fno-zero-initialized-in-bss -fno-strict-aliasing -Wno-array-bounds -Wno-uninitialized -Wno-unused-variable -DHAVE_MMX -DARCH_X86 -DSYS_MINGW -DPTW32_STATIC_LIB -DHAVE_PTHREAD -s -fomit-frame-pointer

gcc version 4.3.3 20081126 (prerelease) (GCC)
Clear profiled 1041 build (without patches) compiled with some CFLAGS: http://komisar.gin.by/test/x264.1041.test_crash.exe
On

x264 [info]: using cpu capabilities: MMX2 SSE2Slow

no crashes with any settings for me...

[burfadel]

Originally Posted by komisar

On my "patched" 1041-build no crashes.gcc version 4.3.3 20081126 (prerelease) (GCC)
Clear 1041 build (without patches) compiled with some CFLAGS: http://komisar.gin.by/test/x264.1041.test_crash.exe

Both your test build and your normal builds still crash for me with the same settings, 1040 and below work fine! I really think it is because the output has changed for CABAC due to the changes in the CABAC code, which is stated in the build release. The changes affect RDO, which make sense since subme 8 and 9 deal with RD refinement. RD mode decision 6 & 7 work fine.

At a guess I'd assume all platforms are affected, just that on non-windows platform instead of a crash there may something else that happens (momentary pause, quality degradation). I did notice fiddling with the settings that I could get to different frame numbers. It usually crashed at the 40th frame, but when the settings were changed at the 40th frame there was a pause, then it kept going for a short while until it crashed.

[Dark Shikari]

Oh joy, a Heisenbug!

I tested this patch heavily and I cannot replicate this crash.

Fortunately, I have my minions on #x264dev to help me find what the problem is.

Edit: Fixed and committed. That was fast, 20 minutes.

[bob0r]

[burfadel]

Thanks for the fix, works flawlessly now

[LoRd_MuldeR]

Confirmed

By the way: What does "10L" stand for?

[akupenguin]

Originally Posted by LoRd_MuldeR

By the way: What does "10L" stand for?

10 liters of cola. This means:
(a) "I must have been asleep when I wrote that, need more caffeine."
(b) One of the MPlayer developers didn't like cola, and we joked about assigning him some as punishment after one time he committed a nasty bug that should have been easily spotted.

[LoRd_MuldeR]

Originally Posted by akupenguin

10 liters of cola. This means:
(a) "I must have been asleep when I wrote that, need more caffeine."
(b) One of the MPlayer developers didn't like cola, and we joked about assigning him some as punishment after one time he committed a nasty bug that should have been easily spotted.

I see