FFmpegSource

[Chikuzen]

I tested Rumbah's sample with libav-win32-pthreads-20110610.7z as follows.
(Luca distributes both w32threads-build and pthreads-build for 32bit Windows)

ffmpeg -threads 4 -i crashtest.avi -vcodec ffvhuff -acodec copy re_crashtest.avi

ffplay -threads 4 crashtest.avi

and ffmpeg/ffplay processed the clip without a crash.

[Rumbah]

It should be noted that the file is no H264 but H263. I don't know if libav even supports multithread decoding for ASP.

[qyot27]

I was looking at the crash I previously mentioned here:
http://forum.doom9.org/showthread.ph...23#post1503123

And I found out a couple more things. Most importantly, the crash doesn't occur - even on 13MB+ files - if I specifically force it to use lavf as the demuxer. ffmsindex -m matroska causes_crash.mkv (which I assume is what is used for 'default' as well) crashes with the error from the Visual C runtime dll. Is there any off chance that the C-interface could be trying to load Haali's splitter (or there being code in the matroska demuxer that doesn't play nice with msvcrt.dll when GCC is used to compile it?), which is making the crash occur under the circumstances that it does?

I do think it may be related to the header's Segment size being read as 'unknown' (in mkvinfo) in these samples that are crashing - that with however resources are allocated to handle indexing, it having an unknown size is causing a memory leak or that it's requesting too much and crashing when it runs out of resources. This would make it crash with fairly small files on my nearly-decade-old computer, but not on setups with more resources (and especially not on 64-bit, with the raised RAM limits that provides). 13MB would be a limit of my particular hardware; for those more powerful computers, much much larger samples would be needed before it would show up. Is there any way to predict ahead of time what would be a suitable filesize on a given setup?


I did attempt to run a backtrace in gdb with builds that had debug enabled, but the only thing it got back was the address in msvcrt.dll where the crash occurs:

$ gdb ffmsindex
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mingw32".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from c:\Program Files\AviSynth 2.5\plugins/ffmsindex.exe...done.
(gdb) set pagination 0
(gdb) run causes_crash.mkv
Starting program: c:\Program Files\AviSynth 2.5\plugins/ffmsindex.exe causes_crash.mkv
[New Thread 2436.0xb4]
Indexing, please wait... 0%
Program received signal SIGSEGV, Segmentation fault.
0x77c3554a in msvcrt!_abnormal_termination () from C:\WINDOWS\system32\msvcrt.dll
(gdb) bt
#0  0x77c3554a in msvcrt!_abnormal_termination () from C:\WINDOWS\system32\msvcrt.dll
(gdb)

[TheRyuu]

ffms2-r473.7z
ffms2-r473-avs64.7z

Changes since last build:
Updated libav to b203f65.
Fixed open failures with some Vorbis files.

Originally Posted by qyot27

...snip...

I can't reproduce it. Do my compiles also have the issue? Or is it specific to your builds (if so maybe there's something wrong with your build system/chain).

[qyot27]

Originally Posted by TheRyuu

I can't reproduce it. Do my compiles also have the issue? Or is it specific to your builds (if so maybe there's something wrong with your build system/chain).

The 32-bit builds are all Visual Studio builds, though. Visual Studio builds don't exhibit the crash. I don't have a 64-bit Windows environment to test under, and depending on the circumstances, the file sizes would potentially need to be huge to run afoul of it (as even --preset ultrafast --tune zerolatency could generate MKVs that do this, it wouldn't be that hard to get an oversized file on better computers, but whether that would need to be a couple hundred megs or a couple dozen gigs, I don't know). I did see it happen under Wine on this computer too, but I can't remember how large my swap partition is. It didn't happen - with my builds - on a Vista-running laptop I have access to, but then it also has 6x the amount of RAM my computer does. A different computer also displayed the crash, and it has less RAM than mine does. That computer and mine both run XP.

The build system in question is constructed of the default mingw repository packages from Ubuntu (11.04), with more or less freshly-compiled versions of zlib 1.2.5, bzip2 1.0.6, and a CVS build of pthreads-w32 which is probably from late April. The MSys environment I use on Windows is mainly the prebuilt one from the CCCP Wiki, augmented with komisar's GCC builds and the aforementioned builds of zlib, bzip2, and pthreads-w32.

I did stumble my way around in the source code a couple days ago to see if I could maybe do trial-and-error alterations. I have no idea if it's connected, but in stdiostream.h, the cachesize is defined as the regular 16-bit entry of 65536 bits (8.192 in KB). My paging file's maximum limit is set to 1536 MB, and if 8.192 KB of cache is used per MB of available memory, that works out to approximately 12.6 MB, which is very close (given factors like overhead or whatnot) to that crash/no-crash threshold I was describing.

Of course, even when I raised the paging file's maximum limit to 4096 MB, it didn't stop the crash from occurring. That throws some doubt in there, unless the real key is just that it waits until it tries to use 3x the available physical RAM and then crashes, rather than anything to do with the paging file.


I tried varying up my build options (using ffmpeg's cpudetection, not passing -march, adding -lmsvcrt to FFMS2's --extra-ldflags) to see if anything changed, but that didn't work either.

The actual compilation steps I was using:

git clone git://git.videolan.org/ffmpeg.git
cd ffmpeg
./configure --prefix=$HOME/win32_build --cross-prefix=i586-mingw32msvc- --enable-gpl --enable-version3 \
--enable-postproc --enable-memalign-hack --disable-encoders --disable-muxers --disable-debug --disable-network \
--disable-hwaccels --disable-indevs --disable-outdevs --extra-cflags="-march=pentium3 -DPTW32_STATIC_LIB" \
--target-os=mingw32 --arch=x86
make
make install

cd ..
svn checkout http://ffmpegsource.googlecode.com/svn/branches/avs64 ffms2-cinterface
cd ffms2-cinterface
#if necessary:
svn merge http://ffmpegsource.googlecode.com/svn/trunk

fromdos configure
FFMPEG_LIBS="-L$HOME/win32_build/lib -lswscale -lavformat -lavcodec -lpostproc -lavutil -lavifil32" \
FFMPEG_CFLAGS="-I$HOME/win32_build/include" ./configure --prefix=$HOME/ffms2-avs --cross-prefix=i586-mingw32msvc- \
--host=i686-pc-mingw32 --enable-avs --enable-shared --enable-postproc --extra-cflags="-march=pentium3" \
--extra-ldflags="-lz -lbz2 -lpthreadGC2"
make
make install

[qyot27]

I've actually managed to compile a build that doesn't crash at all.

I'd been noticing the number of warnings issued during the compilation process itself while matroskaparser.c was being worked with. It looked like this:

gcc -O3 -Wall -march=pentium3 -std=gnu99 -s -fomit-frame-pointer -I/home/Stephen/win32_build/include -I. -Iinclude -D_FILE_OFFSET_BITS=64 -DFFMS_USE_POSTPROC -c -o src/core/matroskaparser.o src/core/matroskaparser.c
src/core/matroskaparser.c:936:13: warning: 'IsWritingApp' defined but not used [-Wunused-function]
src/core/matroskaparser.c: In function 'errorjmp':
src/core/matroskaparser.c:390:6: warning: 'll' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c:304:40: note: 'll' was declared here
src/core/matroskaparser.c:353:18: warning: 'neg' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c:304:35: note: 'neg' was declared here
src/core/matroskaparser.c:353:18: warning: 'zero' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c:304:29: note: 'zero' was declared here
src/core/matroskaparser.c:353:18: warning: 'width' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c:304:22: note: 'width' was declared here
src/core/matroskaparser.c: In function 'readFloat':
src/core/matroskaparser.c:842:3: warning: 'f.v' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c:810:11: warning: 'ui' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c: In function 'parseTrackEntry':
src/core/matroskaparser.c:1517:36: warning: 'CompScope' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c: In function 'parseBlockGroup':
src/core/matroskaparser.c:2133:15: warning: 'add_len' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c:2112:34: note: 'add_len' was declared here
src/core/matroskaparser.c:2132:15: warning: 'add_pos' may be used uninitialized in this function [-Wuninitialized]
src/core/matroskaparser.c:2112:25: note: 'add_pos' was declared here

So I decided to play around with the individual command used on that file to see if that affected those warnings. The only thing necessary was to remove -O3, because with that gone, the only warning is the first one about IsWritingApp.

With that, I went into configure and removed the references to -O3 in CXFLAGS and CXXFLAGS. I then did a normal recompile, and the crash didn't occur.

I've not tried to see if -O2 also has this problem, but as I was getting crashes under debug (and that uses -O1, from what configure seems to say), then I'm guessing all of the optimization modes will spark it.

EDIT: Running just that command on matroskaparser.c with the differing modes, yes, all of the optimization modes (apart from -O0/optimization disabled) exhibit those warnings. Those warnings are also the only difference in the compilation messages between it being enabled or disabled.


The full original log is here:
http://pastebin.com/W9XNyWjn

The log after -O3 has been removed from configure:
http://pastebin.com/8dznUGfm

[TheFluff]

Originally Posted by qyot27

With that, I went into configure and removed the references to -O3 in CXFLAGS and CXXFLAGS. I then did a normal recompile, and the crash didn't occur.

Congratulations, it seems you have found a bug in GCC. Apparently it's miscompiling stuff. Try with a different version.

The warnings are almost definitely unrelated to your problem, as is the page file stuff you were talking about in your previous post.

[Abs62]

FFMS2 r473 don't work when it compiled with libav after 603b8bc (16.06.2011). ffmsindex crash on av_open_input_file.

From libav APIchanges:

2011-06-xx - xxxxxxx - lavf 53.2.0 - avformat.h
Add avformat_open_input and avformat_write_header().
Deprecate av_open_input_stream, av_open_input_file,
AVFormatParameters and av_write_header.

[kemuri-_9]

we've just found this out for x264cli, av_open_input_file is crashing due to careless mistakes in av_open_input_file's deprecation.

stay at the revision BEFORE

Deprecate av_open_input_* and remove their uses.

Deprecate the last remaining member of AVFormatParameters.

until either ffms2 is fixed to use the new API or libav/ffmpeg have av_open_input_file stop crashing.

[JEEB]

Given problem got fixed upstream just now, after I herped a derp on #libav-devel.

This and this are the fixing commits.

[CruNcher]

@JEEB
it seems seeking is broken in your x264 build when outputing to .mp4 and using the mpc-hc splitter

[TheFluff]

Originally Posted by CruNcher

@JEEB
it seems seeking is broken in your x264 build when outputing to .mp4 and using the mpc-hc splitter

That sure is a detailed bug report, but I don't see how it's related to ffms2...

[CruNcher]

Hehe i was just trying some other thing (related to ffms2) and accidentally found this issue too and as i found no thread about jeebs build i thought i hijack this for a moment sorry

[TheRyuu]

Originally Posted by Chikuzen

I tested Rumbah's sample with libav-win32-pthreads-20110610.7z as follows.
(Luca distributes both w32threads-build and pthreads-build for 32bit Windows)

ffmpeg -threads 4 -i crashtest.avi -vcodec ffvhuff -acodec copy re_crashtest.avi

ffplay -threads 4 crashtest.avi

and ffmpeg/ffplay processed the clip without a crash.

It does not play it correctly. It freezes and loops some audio.

Edit:
It's cool though, booted up linux to gdb this and ffmpeg encoded it to ffvhuff without issue regardless of threading. So clearly the best thing to do is point fingers at each other and accomplish nothing productive (the issue is almost certainly ffmpeg/libav's fault) so go poke them.

[Abs62]

Ffmsindex is crashed on some files if ffms2 compiled with libav libraries with commit a26ce1e (13.06.11) and next. For example on this file (ffmsindex -t -1 ...).
ffms2-r473.7z from this thread also is crashed.

It looks like function parse_nal_units() in h264_parser.c in libav get wrong data (for example sps.bit_depth_luma=12345) and handle it without verification. It leads to heap corruption and later crash in some other function.

[TheRyuu]

Originally Posted by Abs62

Ffmsindex is crashed on some files if ffms2 compiled with libav libraries with commit a26ce1e (13.06.11) and next. For example on this file (ffmsindex -t -1 ...).
ffms2-r473.7z from this thread also is crashed.

It looks like function parse_nal_units() in h264_parser.c in libav get wrong data (for example sps.bit_depth_luma=12345) and handle it without verification. It leads to heap corruption and later crash in some other function.

The video's not the problem, it's the audio I think. Indexing it without audio works just fine.

I'll check it out a bit more later. (Also inb4rule6)

[Abs62]

The video's not the problem, it's the audio I think. Indexing it without audio works just fine.

Nope. If you reset the mentioned commit the file will be indexed without problem. I think a heap corruption in parse_nal_units() can afterwards lead to crash in any function in any place. It's a hard kind of bugs.

[kemuri-_9]

Thanks for the report, have you reported the issue to the libav/ffmpeg teams?

[Abs62]

No. I haven't contacts with these teams.

[TheRyuu]

Originally Posted by Abs62

No. I haven't contacts with these teams.

I've poked them about it.