x264 r839 crashes

[Japhsoncross]

I use the following video input and command and I sometimes got x264 crash at the first 11 frames encoded on the 1st pass.

video: this is not the video i first found crash, but this crashes as well, and it's easy to reproduce ;)
colorbars()
converttoyv12()
trim(0, 100)

cmd:
R839_x264.exe --keyint 300 --min-keyint 1 --bframes 5 --b-pyramid --ref 16 --deblock 0:0 --bitrate 350 --pass 1 --stats "stats.log" --partitions all --direct auto --weightb --me esa --merange 16 --subme 7 --b-rdo --mixed-refs --bime --8x8dct --no-fast-pskip --no-dct-decimate --direct-8x8 1 --threads 12 --thread-input --output NUL "1.avs" --progress --trellis 2

R839_x264.exe --keyint 300 --min-keyint 1 --bframes 5 --b-pyramid --ref 16 --deblock 0:0 --bitrate 350 --pass 2 --stats "stats.log" --partitions all --direct auto --weightb --me esa --merange 16 --subme 7 --b-rdo --mixed-refs --bime --8x8dct --no-fast-pskip --no-dct-decimate --direct-8x8 1 --threads 12 --thread-input --output "Rev830 VAQ 350.mkv" "1.avs" --progress --trellis 2


I first found this issue on R830_fgo patched version, I was first testing with some small size video, and that's why i use such overkill settings. it sometimes crashed, and I thought that would be a broken patch, and I changed to R839 from x264.nl, still crashed. Then I guessed it could be a hardware issue, I was runing Phenom9750+RD780+2G DDR2 800X4 @Windows XP 32bit, 2G for system memory, and 6G ramdisk for x264 encoding with fast&silent I/O. I move the whole folder to another PC Athlon64X2 4200+, nForce560, 1G DDR2 800X2, without ramdisk, fresh WinXP 32bit installed, and it never crashes, but when I change the --threads from auto to 6, since Phenom is QuadCore, the auto threads equals 6, and it begins to crashes sometimes at first pass the 11th frame sometimes, which was exactlly the same as what I've saw with Phenom9750. I move again the folder to an Intel E2180 PC, it crashes as well.


I hope these information could be useful for you.

[Dark Shikari]

I use the following video input and command and I sometimes got x264 crash at the first 11 frames encoded on the 1st pass.Since the number of frames read in before encoding any is equal to (bframes + threads - 1), that means it actually didn't encode anything; it crashed the instant encoding began.

Any chance you could post a gdb backtrace, or at least something like "disass $pc-50 $pc+50", which will show us the assembly at the point of the crash (even if you didn't use a debug build of x264)? If the assembly is from a handwritten assembly function, we'll recognize it and possibly be able to find the problem.

Of course, a gdb log from a debug build with full symbols would be even better.

[burfadel]

Where would one find up-to-date debug x264 builds?

[Dark Shikari]

Where would one find up-to-date debug x264 builds?You'll have to make your own. Grab a copy of cygwin or mingw. Of course, even if you don't have a debug build, it'd be nice to get an assembly dump as I stated.

[Japhsoncross]

i don't know if this is what you want:

00428F5E psubw mm0,mm7
00428F61 psubw mm1,mm6
00428F64 pxor mm2,mm2
00428F67 psubw mm2,mm0
00428F6A pmaxsw mm0,mm2
00428F6D pxor mm3,mm3
00428F70 psubw mm3,mm1
00428F73 pmaxsw mm1,mm3
00428F76 paddw mm0,mm1
00428F79 paddusw mm0,mmword ptr [ebx]
00428F7C movq mm4,mm5
00428F7F psubusw mm4,mm0
00428F82 packsswb mm4,mm4
00428F85 movd dword ptr [esi],mm4
00428F88 add ecx,8
00428F8B add ebx,8
00428F8E add esi,4
00428F91 sub eax,4
00428F94 jg 00428F57
00428F96 jmp 00429450
00428F9B nop
00428F9C lea esi,[esi]
00428FA0 push ebx
00428FA1 push esi
00428FA2 push edi
00428FA3 push ebp
00428FA4 mov eax,dword ptr [esp+14h]
00428FA8 mov ecx,dword ptr [esp+18h]
00428FAC mov edx,dword ptr [esp+1Ch]
00428FB0 mov ebx,dword ptr [esp+20h]
00428FB4 pshufw mm7,mmword ptr [eax],0
00428FB8 pshufw mm6,mmword ptr [esp+2Ch],0
00428FBE mov ebp,esp
00428FC0 mov eax,dword ptr [esp+28h]
00428FC4 sub esp,eax
00428FC6 sub esp,7
00428FC9 and esp,0FFFFFFF0h
00428FCC mov esi,esp
00428FCE shl edx,1
00428FD0 movq mm0,mmword ptr [ecx]
00428FD3 movq mm1,mmword ptr [ecx+8]
00428FD7 psubw mm0,mm7
00428FDA psubw mm1,mm7
00428FDD pxor mm2,mm2
00428FE0 psubw mm2,mm0
00428FE3 pmaxsw mm0,mm2
00428FE6 pxor mm3,mm3
00428FE9 psubw mm3,mm1
00428FEC pmaxsw mm1,mm3
--------------->00428FEF paddusw mm0,mmword ptr [ebx]
00428FF2 paddusw mm1,mmword ptr [ebx+8]
00428FF6 movq mm4,mm6
00428FF9 movq mm5,mm6
00428FFC psubusw mm4,mm0
00428FFF psubusw mm5,mm1
00429002 packsswb mm4,mm5
00429005 movq mmword ptr [esi],mm4
00429008 add ecx,10h
0042900B add ebx,10h
0042900E add esi,8
00429011 sub eax,8
00429014 jg 00428FD0
00429016 jmp 00429450
0042901B nop
0042901C lea esi,[esi]
00429020 push ebx
00429021 push esi
00429022 push edi
00429023 push ebp
00429024 mov eax,dword ptr [esp+14h]
00429028 mov ecx,dword ptr [esp+18h]
0042902C mov edx,dword ptr [esp+1Ch]
00429030 mov ebx,dword ptr [esp+20h]
00429034 movaps xmm4,xmmword ptr [eax]
00429037 pshuflw xmm7,xmm4,0
0042903C pshuflw xmm6,xmm4,0AAh
00429041 pshufhw xmm5,xmm4,0
00429046 pshufhw xmm4,xmm4,0AAh
0042904B punpcklqdq xmm7,xmm7
0042904F punpcklqdq xmm6,xmm6
00429053 punpckhqdq xmm5,xmm5
00429057 punpckhqdq xmm4,xmm4
0042905B mov ebp,esp
0042905D mov eax,dword ptr [esp+28h]
00429061 sub esp,eax
00429063 sub esp,7
00429066 and esp,0FFFFFFF0h
00429069 mov esi,esp
0042906B shl edx,1
0042906D movups xmm0,xmmword ptr [ecx]
00429070 movups xmm1,xmmword ptr [ecx+10h]
00429074 psubw xmm0,xmm7
00429078 psubw xmm1,xmm6
0042907C pxor xmm2,xmm2
00429080 psubw xmm2,xmm0
00429084 pmaxsw xmm0,xmm2
00429088 pxor xmm3,xmm3
0042908C psubw xmm3,xmm1
00429090 pmaxsw xmm1,xmm3
00429094 movups xmm2,xmmword ptr [ecx+edx]

[Dark Shikari]

Odd, that looks like its crashing inside a SATD function unless I'm mistaken. So my instincts were right; it is a hand-written assembly function.

I'll have to see what Akupenguin thinks of this.

[akupenguin]

Odd, that looks like its crashing inside a SATD function unless I'm mistaken.
x264_pixel_ads1_mmxext

[Dark Shikari]

Well that means the crash should go away if you don't use ESA, but this is a problem nonetheless.