crash from gcc double free corruption of scratch_buffer in lookahead?

[bitbytebit]

When I encode an mpeg2 file (guessing it's any type of file) from a pvr500 Hauppauge card, with the following x264 library version and settings, if I use a resolution of 854x480 I get the crash and backtrace listed below. If I change just the resolution to 640x480 I don't have any problems. Seems like the scratch_buffer in the lookahead.c file being freed causes this, not sure why just the wider 16x9 resolution makes this occur.


Writing library/Version : core 85 r1442 781d300

Encoding settings : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / sliced_threads=0 / nr=100 / decimate=1 / mbaff=0 / constrained_intra=0 / bframes=6 / b_pyramid=0 / b_adapt=1 / b_bias=0 / direct=3 / wpredb=1 / wpredp=2 / keyint=300 / keyint_min=30 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00



#0 0xffffe424 in __kernel_vsyscall ()
#1 0x08632d99 in raise ()
#2 0x085f55e4 in abort ()
#3 0x08609b3b in __libc_message ()
#4 0x0860e639 in malloc_printerr ()
#5 0x0860fd76 in free ()
#6 0x085737fb in x264_free (p=0x0) at common/common.c:802
#7 0x085813a5 in x264_lookahead_delete (h=0x2) at encoder/lookahead.c:175
#8 0x0857713f in x264_encoder_close (h=0x8c4a7d0) at encoder/encoder.c:2658
#9 0x0804f085 in ngtc_X264_close (avctx=0x9184660) at libx264.c:170
#10 0x081033d9 in avcodec_close (avctx=0x9184660) at libavcodec/utils.c:685
#11 0x08050310 in stop_muxer (cs=0xbff771f0) at encoder.c:182
#12 0x080620ff in main (argc=11, argv=0xbff7a1c4) at ngtc.c:4172


Thanks,
Chris

[bitbytebit]

definitely looks weird, I just went through and checked the allocation and deallocation of the scratch_buffer for threads in x264, and doesn't seem like there's any problems in how it's coded yet it does this double free corruption. This is the complete backtrace output I get, and simply altering the width of the output from 640 to 854 causes this to occur. The input is 720x480 mpeg2 with a 16:9 aspect recorded from HDTV using a pvr500 capture card. Version x264.79.1354 didn't have this issue. If I comment out the code that frees the scratch_buffer for each thread, in both the lookahead and close functions, this problem goes away (and I'm guessing may leak memory, but maybe not if it's really already freed?).

*** glibc detected *** /home/ckennedy/workspace/ngtc/ngtc: double free or corruption (out): 0x09406150 ***
======= Backtrace: =========
[0x860e639]
[0x860fd76]
[0x85737fb]
[0x85813a5]
[0x857713f]
[0x83c6889]
[0x80620ff]
[0x85f0ce0]
[0x8048151]
======= Memory map: ========
08048000-0879a000 r-xp 00000000 08:03 2908474 /home/ckennedy/workspace/ngtc/ngtc
0879a000-087ae000 rw-p 00752000 08:03 2908474 /home/ckennedy/workspace/ngtc/ngtc
087ae000-0a7b0000 rw-p 00000000 00:00 0 [heap]
b1144000-b1542000 rw-p 00000000 00:00 0
b15d3000-b17d2000 rw-p 00000000 00:00 0
b2064000-b217d000 rw-p 00000000 00:00 0
b297e000-b2a97000 rw-p 00000000 00:00 0
b3298000-b4428000 rw-p 00000000 00:00 0
b4528000-b5097000 rw-p 00000000 00:00 0
b5100000-b5121000 rw-p 00000000 00:00 0
b5121000-b5200000 ---p 00000000 00:00 0
b52f0000-b5ecc000 rw-p 00000000 00:00 0
b5ecc000-b5ecd000 ---p 00000000 00:00 0
b5ecd000-b66cd000 rw-p 00000000 00:00 0
b674d000-b674e000 ---p 00000000 00:00 0
b674e000-b6f4e000 rw-p 00000000 00:00 0
b6f4e000-b6f4f000 ---p 00000000 00:00 0
b6f4f000-b774f000 rw-p 00000000 00:00 0
b774f000-b7750000 ---p 00000000 00:00 0
b7750000-b7f51000 rw-p 00000000 00:00 0
bf956000-bf96b000 rw-p 00000000 00:00 0 [stack]
ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x8c40850 (LWP 20993)]
0xffffe424 in __kernel_vsyscall ()
(gdb) backtrace
#0 0xffffe424 in __kernel_vsyscall ()
#1 0x08632d99 in raise ()
#2 0x085f55e4 in abort ()
#3 0x08609b3b in __libc_message ()
#4 0x0860e639 in malloc_printerr ()
#5 0x0860fd76 in free ()
#6 0x085737fb in x264_free (p=0x0) at common/common.c:802
#7 0x085813a5 in x264_lookahead_delete (h=0x2) at encoder/lookahead.c:175
#8 0x0857713f in x264_encoder_close (h=0x8c4a7d0) at encoder/encoder.c:2658
#9 0x083c6889 in X264_close (avctx=0x9184660) at libavcodec/libx264.c:143
#10 0x081033d9 in avcodec_close (avctx=0x9184660) at libavcodec/utils.c:685
#11 0x08050310 in stop_muxer (cs=0xbf966270) at encoder.c:182
#12 0x080620ff in main (argc=9, argv=0xbf969244) at ngtc.c:4172

[Dark Shikari]

"Corruption" can also mean there was an invalid write. Try valgrind.

[bitbytebit]

Valgrind does complain about invalid read's and also invalid writes (bunch of these happen)....


==21844== Invalid read of size 4
==21844== at 0x860FD80: free (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85EFF5A: search_object (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85F011A: _Unwind_Find_FDE (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85ED8CC: uw_frame_state_for (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85EE0AF: uw_init_context_1 (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85EE1E0: _Unwind_Backtrace (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x8620F15: backtrace (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x8609B51: __libc_message (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x860E638: malloc_printerr (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x860FD75: free (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85737FA: x264_free (common.c:802)
==21844== by 0x85813A4: x264_lookahead_delete (lookahead.c:175)
==21844== Address 0x6e00010 is not stack'd, malloc'd or (recently) free'd
==21844==
==21844== Invalid write of size 4
==21844== at 0x85EFF64: search_object (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85F011A: _Unwind_Find_FDE (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85ED8CC: uw_frame_state_for (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85EE0AF: uw_init_context_1 (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85EE1E0: _Unwind_Backtrace (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x8620F15: backtrace (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x8609B51: __libc_message (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x860E638: malloc_printerr (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x860FD75: free (in /home/ckennedy/workspace/ngtc/ngtc)
==21844== by 0x85737FA: x264_free (common.c:802)
==21844== by 0x85813A4: x264_lookahead_delete (lookahead.c:175)
==21844== by 0x857713E: x264_encoder_close (encoder.c:2658)
==21844== Address 0x6e00468 is not stack'd, malloc'd or (recently) free'd


Not sure what that means though, guessing the scratch buffer is either still in use or something like that.

[Dark Shikari]

Valgrind does complain about invalid read's and also invalid writes (bunch of these happen)....Those aren't errors inside x264, so I can't say anything about them. Post the whole valgrind log for functions that are actually in x264.

[bitbytebit]

Here's the output from valgrind, this hopefully is better.


http://pastebin.org/95465

[Dark Shikari]

Try this:

Replace, in encoder/encoder.c

int buf_hpel = (h->param.i_width+48) * sizeof(int16_t);

with

int buf_hpel = (h->fdec->i_width[0]+48) * sizeof(int16_t);

I think this will fix it.

[bitbytebit]

Yep, that seems to fix it.

Thanks,
Chris

[Dark Shikari]

Fixed locally.