[x264] Facepalm: Crash in x264 -- all the recent builds

[Egh]

Great news for x264 developers:

crash found and identified :) Spent couple of hours researching this issue tonight, so here's the info :

Crash was first detected in 1109M x264 sharktooth's build, (x64 version) however was confirmed to exist in any of the recent builds I have got including:

latest 1113 unpatched from x264.nl;
1109M x86 and x64 from Sharktooth;
last available build 1109 from xvidvideo.ru before it went down;
latest debug build i was able to find (1088 from someone) (as a side note I still dont get the fact why most recent debug build is nearly 30 SVN revisions behind, and why it no official debug builds are released)

Luckily enough I was able to find a thread where debug output from x264 was explained (as a side note again however no link to download GDB win32 build given)

And the best thing is? No stack, no registers. And why?

Program exited normally.


However in the progress output file:

avis [info]: 1280x720 @ 23.98 fps (7812 frames)
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 PHADD SSE4.1 Cache64
x264 [info]: profile High, level 3.1

encoded 1541 frames, 10.25 fps, 2605.91 kb/s


Here only :facepalm: would help :)

I have tried several options, and quite reliably identified that it crashes (every time) on several different inputs (all YV12) provided it has > 1542 frames (all 720p, haven't tried anything else yet).

BTW it is a first pass and thus no output file (-o NUL) so developers cannot hide behind MKV or MP4 borked output this time :P

The reason I use zero-pass files is that x264-x64 doesn't work with AVS so I cannot feed video straight from AVS, therefore only pipe or YV12 file.

And if you need parameters for x264, here you go:

x264.exe --progress --threads 2 -p 1 -b 4 -r 2 --subme 5 --stats "y:\1a.stat" -B 2500 -o NUL "e:\@rb01\araw3.avi" 2>1a01.txt


Of course my parameters are typically more complicated even for the first pass, however for tests etc I managed to narrow down to such simplistic case and the bug is still lurking :)

What is more, just now I've tried this

x264.exe --progress --threads 1 -p 1 -b 4 -r 2 --subme 5 --stats "y:\1a.stat" -B 2500 -o NUL "e:\@rb01\araw3.avi" 2>1a01.txt

i.e. no multithreading and crashed after first 1541 frames again :PP

[Dark Shikari]

Here only :facepalm: would help :)Explain? The output file looks entirely correct--if the program had crashed, the final line would not have been printed. If the regular x264 did actually crash, it sounds more like an issue of the debug build working and the non-debug build crashing, which means you need a debug build compiled with all optimizations.

The fact that only 1541 frames were encoded means your input failed, not x264. If x264 had failed, the final status line would never have been printed.

If you need debug builds, ask J_Darnley on #x264.

Also, be careful with the x64 builds; GCC 4.4 is known to miscompile them.

You don't seem to have "identified" the crash at all (saying "it crashed" is not "identifying" it), so don't say that you have until you actually have. You're jumping the gun quite a bit.

[Egh]

Sorry I was doing a bit more tests when you replied, so my first post updated.

Now I've scratched my head a bit etc, and here's the actual explanation:

does x264 support AVI files as an input which are >2GB ?:) As 1542 frames is just next to 2GB in size (in YV12 uncompressed).
Input may end there as you say, however it is x264 responsibility, there's no AVS involved in the chain :)


Also, be careful with the x64 builds; GCC 4.4 is known to miscompile them.


It applies to both x86 and x64 builds, both patched and SVN.

[Dark Shikari]

Sorry I was doing a bit more tests when you replied, so my first post updated.

Now I've scratched my head a bit etc, and here's the actual explanation:

does x264 support AVI files as an input which are >2GB ?:)x264 doesn't support AVI files as input to begin with. I'm not sure how you have decided that it does, done analysis, and concluded from all of this that there was "crashing" going on...


... using an input format that isn't even supported in the first place.

[Egh]

x264 doesn't support AVI files as input to begin with. I'm not sure how you have decided that it does, done analysis, and concluded from all of this that there was "crashing" going on...


... using an input format that isn't even supported in the first place.


x264 core:66 r1113 cc4f807
Syntax: x264 [options] -o outfile infile [widthxheight]

Infile can be raw YUV 4:2:0 (in which case resolution is required),
or YUV4MPEG 4:2:0 (*.y4m),
or AVI or Avisynth if compiled with AVIS support (yes).
Outfile type is selected by filename:
.264 -> Raw bytestream
.mkv -> Matroska
.mp4 -> MP4 if compiled with GPAC support (yes)


Comments? :) especially regarding
or AVI or Avisynth if compiled with AVIS support (yes).

[Dark Shikari]

Comments? :) especially regarding
or AVI or Avisynth if compiled with AVIS support (yes).Theoretically the Avisynth support could also read from AVI files, but there's no guarantee (just as you discovered). There is no AVI-specific code and I'm shocked it works at all.

And seriously, stop misusing the word "crash." If there isn't an exception thrown or an assert failed, it isn't a crash.

(By the way, the original AVI format doesn't support files over 2GB anyways, so I suspect that might be your problem.)

[Egh]

Theoretically the Avisynth support could also read from AVI files, but there's no guarantee (just as you discovered). There is no AVI-specific code and I'm shocked it works at all.

Lol if you asked me I used that AVI input hundreds of SVN revisions ago :) Only smaller fragments of course.

Now question is who added AVS support code and how can I request that someone to enable support for OpenDML AVI files (note the code DOES correctly identify frame number in AVI which is over 2GB, starts the job and when it reaches 2gb limit apparently successfully finishes the job.

In the meantime would be nice to correct --longhelp then :) and at least add warning there that >2GB avis are not supported (if that is the case of course, it is only suggestion at the moment which needs to be verified by someone fluent with the source :)) and maybe even check AVIs for size.....

Also, I hope that raw and y4m files are supported of arbitrary size? To avoid random exits at some frame over 20000 or so ;)


(By the way, the original AVI format doesn't support files over 2GB anyways, so I suspect that might be your problem.)

Of course, but that was the case during Windows 3.1 times, vdub has been able to save to >2GB AVIs for many years already.

[Dark Shikari]

Also, I hope that raw and y4m files are supported of arbitrary size? To avoid random exits at some frame over 20000 or so ;)Indeed, arbitrarily large yuv/y4m is supported.

OpenDML AVI would be a nice feature to add; patches are welcome. The AVI(S) reading code appears quite simple, so it shouldn't be too hard to extend if someone cares about doing it.

[Egh]

Indeed, arbitrarily large yuv/y4m is supported.

OpenDML AVI would be a nice feature to add; patches are welcome. The AVI(S) reading code appears quite simple, so it shouldn't be too hard to extend if someone cares about doing it.

I'll post a request for that in the "current patches thread" I guess.

How long the current code been there? I reckon it could be years....

[Dark Shikari]

I'll post a request for that in the "current patches thread" I guess.

How long the current code been there? I reckon it could be years....Probably. There haven't been non-trivial modifications to muxers.c since I became a developer.

[Manao]

OpenDML AVI would be a nice feature to add; patches are welcome. The AVI(S) reading code appears quite simple, so it shouldn't be too hard to extend if someone cares about doing it.
The AVI code is simple because it uses VFW API which doesn't support OpenDML. Supporting open DML requires you to either use an avi library supporting it or implement its support yourself. In both cases, expect the code to become far more complicated.

[squid_80]

Um, how hard is it to wrap the .avi you're using for input in an .avs script? Tada, problem solved!

[Egh]

Um, how hard is it to wrap the .avi you're using for input in an .avs script? Tada, problem solved!

That means you don't know actual problem why so much effort ;)

Trouble is that my general recommendation is to avoid AVS input in x264, like fire. Of course I could use AVISource() but that would involve AVS into chain and that is what any sane individual tries to avoid in the first place ;)

Truth should be said that AVISource() is apparently safe but with x264 and AVS you can never ever be sure about anything :) Well actually you can -- years pass by and it still crashes... Since last time I was testing this issue I've changed my computer (AMD single core to Intel c2d), my OS (into x64 XP), Avisynth version from 2.5.6 into 2.5.8 including most release candidates on the way, x264 build is several hundreds SVN revisions updated.... And I'm glad to announce AVS crashed in the very same fashion today; again :)

[video_magic]

....
Truth should be said that AVISource() is apparently safe but with x264 and AVS you can never ever be sure about anything :)..

And if you're using an .AVS you know of the OpenDMLSource choice.

[squid_80]

That means you don't know actual problem why so much effort ;)I think you're dodging your original problem (broken avisynth) and making it an x264 issue.

Trouble is that my general recommendation is to avoid AVS input in x264, like fire. Of course I could use AVISource() but that would involve AVS into chain and that is what any sane individual tries to avoid in the first place ;)Any sane individual would choose avisynth's .avi reading code over Microsoft's.

Truth should be said that AVISource() is apparently safe but with x264 and AVS you can never ever be sure about anything :) Well actually you can -- years pass by and it still crashes... Since last time I was testing this issue I've changed my computer (AMD single core to Intel c2d), my OS (into x64 XP), Avisynth version from 2.5.6 into 2.5.8 including most release candidates on the way, x264 build is several hundreds SVN revisions updated.... And I'm glad to announce AVS crashed in the very same fashion today; again :)Then your problem is simple; you either have a bad plugin in avisynth's autoload directory or a bad vfw codec installed that is trying to decompress a stream it knows nothing about. What does GDB say in this case?

video_magic: using OpenDMLSource isn't necessary, avisource will choose OpenDMLSource or AVIFileSource internally after determining which is needed.

[Egh]

I think you're dodging your original problem (broken avisynth) and making it an x264 issue.

Original problem in this thread is undocumented feature regarding AVI support (i.e. in the help it is noted that AVI/AVS are supported however 2GB limit is not mentioned). As I've mentioned already, it does accept OpenDML AVIs and even correctly identifies the frame count.

The issue with broken avisynth is only indirect cause for this thread as I have to use some kind of workaround, either avi or as now due to 2gb input limit I'm forced to use YV420 raw files.


Any sane individual would choose avisynth's .avi reading code over Microsoft's. :confused: why not?? :O


Then your problem is simple; you either have a bad plugin in avisynth's autoload directory or a bad vfw codec installed that is trying to decompress a stream it knows nothing about. What does GDB say in this case?


Well I've described the actual problem with x264+AVS == it works but it is just not reliable. I've seen that on several occasions completely unrelated to each other (like different OS running on different CPU and different versions of AVS, as mentioned above). VfW codecs are nothing to do with it.

It is simply a case of x264 receiving input from AVS for encoding. It can crash on a first pass or on the second, but most important that then x264 goes on with encoding, and if you lucky to be there at that time you can identify it by encoding FPS sky-rocketing. Of course only because it encodes blank frames so entire process basically fails at that point (without any error messages as from x264 POV everything is as normal).

Last time it crashed I was testing a very light script, basically yatta-produced AVS which only used TDeinterlace, TDecimate, FieldHints and DGDecode as external filters and several common internal filters (like resizing etc). Sample to encode was 10 minutes MPEG2 1080p downsized to 720p. Avisynth crashed half-way through...

That's why I typically prefer zero-pass method (also saves encoding time in case of heavy AVS filtering) which is to create a filtered video and then use it for two pass encoding.

P.S. If you suggest to use AVISource() to input video from AVI files to x264, tell me how do I use x64 builds :)

[Dark Shikari]

P.S. If you suggest to use AVISource() to input video from AVI files to x264, tell me how do I use x64 builds :)64-bit Avisynth or avs2yuv?

By the way, here's a general tip: if you're the only one having problems and tens of thousands of others people using Avisynth-based apps like MeGUI are having no such issues, it probably isn't a general problem with Avisynth.

[squid_80]

:confused: why not?? :OBecause Microsoft's AVI reading code is broken and can't read avi files over 2GB. Avisynth's code is a lot more robust.
It is simply a case of x264 receiving input from AVS for encoding. It can crash on a first pass or on the second, but most important that then x264 goes on with encoding, and if you lucky to be there at that time you can identify it by encoding FPS sky-rocketing. Of course only because it encodes blank frames so entire process basically fails at that point (without any error messages as from x264 POV everything is as normal).Now you're changing the story. If x264 encodes all the way to the end, that's not a crash. If the encoded stream ends up being a red error message from avisynth, it's an avisynth problem and nothing to do with x264.

That's why I typically prefer zero-pass method (also saves encoding time in case of heavy AVS filtering) which is to create a filtered video and then use it for two pass encoding. So you can successfully make an .avi out of the script, but feeding it directly to x264 fails? That sounds like memory/cpu failure while under stress.
Regardless, my original suggestion was to use a script with one avisource() line to open the >2GB avi. If the avi file contains raw YV12 frames I find it really hard to believe that avisynth is failing somehow.

P.S. If you suggest to use AVISource() to input video from AVI files to x264, tell me how do I use x64 builds :)Heh, you're asking the person who made the x64 version of avisynth. Personally I use exactly the same script as normal.

[burfadel]

I'm doing some batch encoding at the moment (have been for the last week!) - high settings so it takes a while!

Anyways, the encoding just stopped... it didn't crash, just x264 was literally doing nothing at all at around 84 percent of that particular file. While this could be caused by any number of things, and could be just a once off! just thought I'd mention it in case others have had the same issue. It didn't crash on the first encoded file, it was around the third (the list just started to get on to a whole heap of files encoding at 512x384).

There is not signs of any errors, except for the fact x264 stops and does nothing (and zero cpu usage).

I'm using a vanilla build of 1114 from x264.nl

I've been keeping up to date with the builds normally, and have also done so in the past week. I'm assuming that if there is indeed an error its related to the changes introduced in 1114...

It may just be one of those things that happen too! If it happens again I'll install build 1113 again and see if that helps.

Settings used were:


--crf 24 --keyint 450 --ref 5 --mixed-refs --bframes 6 --b-adapt 2 --b-pyramid --weightb --direct auto --subme 9 --trellis 2 --psy-rd 1:1 --partitions all --8x8dct --me umh --threads auto --progress --no-psnr --no-ssim

On a Core 2 E6600 (which I'm replacing with a Q9400 this week :D)

[Dark Shikari]

I'm doing some batch encoding at the moment (have been for the last week!) - high settings so it takes a while!

Anyways, the encoding just stopped... it didn't crash, just x264 was literally doing nothing at all at around 84 percent of that particular file. While this could be caused by any number of things, and could be just a once off! just thouight I'd mention it in case others have had the same issue. It didn't crash on the first encoded file, it was around the third (a whole heap of files encoding at 512x384).

I'm using a vanilla build of 1114 from x264.nl

I've been keeping up to date with the builds normally, and have also done so in the past week. I'm assuming that if there is indeed and error its related to the changes introduced in 1114...

It may just be one of those things that happen too!No recent changes could cause a deadlock. Most likely the input stopped responding and x264 as a result just paused.

[burfadel]

Ah ok! I thought that may be the case, just thought I'd mention it in case others have found the same thing!

According to virtualdub the source file has no errors thats why I assumed it was x264 as the filter list I have used unchanged for quite a long time. I do understand these things happen though!