crashing towards end of secondpass

[Blue_MiSfit]

Hey all,

Strange problems recently. I have tried twice encoding the Battlestar Galactica miniseries. It seems that x264.exe crashes somewhere in the secondpass. Although I don't know exactly where it quit, I do know that it was later than 50%. A strange bug. Here is my avs and x264cli settings:

AVS:

LoadPlugin("C:\PROGRA~1\GORDIA~1\DGMPGDec\DGDecode.dll")
LoadPlugin("C:\PROGRA~1\GORDIA~1\AviSynthPlugins\removegrainsse2.dll")

mpeg2source("C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.d2v")

removegrain(mode=2)


x264

x264.exe --progress --pass 1 --bitrate 1553 --stats "\x264.stats" --subme 3 --ref 4
--threads 1 --filter -2:-2 --keyint 240 --min-keyint 24 --scenecut 40 --qpmin 10 --qpmax 51 --qpstep 4 --direct temporal --me hex --merange 16 --sar 16:9 --bframes 3 --weightb --b-pyramid --b-bias 0 --ipratio 1.40 --pbratio 1.30 --qcomp 0.60 --analyse p8x8,i8x8,i4x4 --8x8dct --quiet -o "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\galactica" "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.avs"

x264.exe --progress --pass 3 --bitrate 1553 --stats "\x264.stats" --subme 5 --ref 8
--threads 1 --filter -2:-2 --keyint 240 --min-keyint 24 --scenecut 40 --qpmin 10 --qpmax 51 --qpstep 4 --direct temporal --me hex --merange 16 --sar 16:9 --bframes 3 --weightb --b-pyramid --b-bias 0 --ipratio 1.40 --pbratio 1.30 --qcomp 0.60 --analyse all --8x8dct --quiet -o "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\galactica" "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.avs"


I am going for about 1553 kbit, with 8x8dct, 3 bframes, and 8 b-refs. Everything else is maxed out as usual. Aside from being mind numbingly slow (4fps) this seems to work just fine, but two times in a row I have come back to my pc after about 15 hours of encoding to find the program mysteriously closed, and with unusable output files.

I was using x264 rev 280, and avisynth 2.55.

Any ideas? I am going to start my batch file from scratch with Doom9's new full GUI for x264.

Thanks in advance,
-Misfit

[Doom9]

as usual I have to ask why you put default values in the commandline.. it makes things really confusing.

Is the crash reproducible and does it always happen at a specific frame? If so, if you use the trim command to encode only a small part including that particular frame, does it still happen?

Also, I noticed that you're overclocking.. it's one of the main reasons for incorrect behavior in many applications that are used around here. Your CPU is made for 1.8 GHz, you're running it a whopping 25% above specs so I'm not surprised at all that apps crash on you.

[bond]

you have to define an output filetype. you can use .264, .mp4 or .mkv

whats the framerate?

[Blue_MiSfit]

I'm actually not overclocking anymore. Also, my filetype is mp4. I have not really absorbed the x264 settings yet, and am still just using GUIs to create my command lines for me. As far as crashing at a certain point, I'm not really sure where it crashes because as I mentioned I'm not actually at the PC when it crashes. I come back to check on the status and the window has simply exited.

Framerate is NTSC with Force Film - so 23.976 as usual for me.

I'm sure there's logging somewhere. Where is it / how do I enable it?

Thanks for the help. I will try to clean up my scripts.

-MiSfit

[bond]

your commandline doesnt specify the .mp4 extension. if you have got that commandline form a gui than tell the gui developer about it, cause its a bug

and again: whats the framerate of your input stream?

[Doom9]

and am still just using GUIs to create my command lines for me.might I suggest using one of mine next time? The extension problem for instance, you'd never run into this using MeGUI ;) That's not saying you'd not get that crash but the commandline issues should be sorted out at least.

[Blue_MiSfit]

re running encode with same avs (edited previous post to include frame rate information). I created the command lines in MeGUI's special x264 version.


"x264.exe" --pass 1 --bitrate 1553 --stats "2pass.log" --keyint 240 --min-keyint 24 --ref 3 --bframes 2 --b-pyramid --filter -2,-2 --weightb --analyse all --8x8dct --qpstep 1 --sar 16:9 --progress --no-psnr --output NUL "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.avs"

"x264.exe" --pass 2 --bitrate 1553 --stats "2pass.log" --keyint 240 --min-keyint 24 --ref 3 --bframes 2 --b-pyramid --filter -2,-2 --weightb --analyse all --8x8dct --qpstep 1 --sar 16:9 --progress --no-psnr --output "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica New.mp4" "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.avs"


70% done with the firstpass, no problems so far. Temperatures and voltages remain stable. I will report with the results. Assuming it does in fact crash again, how/where can I access the logs that would pinpoint the location?

Thanks
-MiSfit

[bond]

Framerate is NTSC with Force Film - so 23.976 as usual for me.there seems to be a bug with 23.976 fps in x264.exe when outputting to .mp4 and .mkv which leads to a crash while encoding

afaik its not known till now what really causes it, its funny tough that both mkv and mp4 are affected
raw output should work

[Blue_MiSfit]

interesting. what can deal with raw .264 streams? Last I checked Mkvtoolnix cant...

[berrinam]

mp4box and mp4creator can mux .264 into mp4. You can then mux that into mkv with mkvtoolnix if you want.

[Blue_MiSfit]

Okay so I finished the encode after 2 days :) and muxed the resulting 264 file into mp4 with YAMB/mp4box. This mp4 was then muxed into matroska with mkvmerge. There was a strange error - something about an atom missing so the timecodes could potentially be messed up. Regardless, the resulting MKV plays in core media player and media player classic.

Unfortunately, somehow the video is only 16x16 pixels!!!!! Both the MP4 and the MKV have these dimensions, and the MP4 crashes media player classic after a few frames of playback. It seems I have much to learn regarding processing raw h.264 streams.

Any ideas as to why the video was 16x16 when for both passes of x264.exe the video was reported as being 720x480?

Thanks
-MiSfit

[bond]

which version of mp4box did you use? where did you download it?

[Sharktooth]

WARNING: MP4 and MKV outputs are broken for 23.976 FPS.
the bug is in GPAC's m4systems library...

[Sirber]

yeah. took me 5 days of debuging to come to this conclusion :)

[bond]

the bug is in GPAC's m4systems library...how does gpac influence mkv output?

[Sharktooth]

coz probably mkv output needs the gpac library too or takes the output from the gpac functions.

[bond]

coz probably mkv output needs the gpac library too or takes the output from the gpac functions.are you guessing? i would be really surprised if this is really that way

[stephanV]

You can compile x264 with mkv output without GPAC (i do that) so it isnt that. It doesnt rely on GPAC in any way.

what appears to go wrong is this:
avis [info]: 384x208 @ 23.98 fps (1501 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
x264 [info]: slice I:15 Avg QP:14.13 Avg size: 18737 PSNR Mean Y:53.02 U:53.97
V:54.22 Avg:53.32 Global:48.36
x264 [info]: slice P:1486 Avg QP:17.35 Avg size: 2724 PSNR Mean Y:50.46 U:51.44
V:51.57 Avg:50.75 Global:43.98
x264 [info]: slice I Avg I4x4:81.6% I8x8:0.0% I16x16:18.4%
x264 [info]: slice P Avg I4x4:0.7% I8x8:0.0% I16x16:1.0% P:57.8% P8x8:10.3
% PSKIP:30.2%
x264 [info]: PSNR Mean Y:50.48 U:51.47 V:51.59 Avg:50.77 Global:44.01 kb/s:553.1


encoded 1501 frames, 13.11 fps, 445460.49 kb/s

as opposed to:
avis [info]: 384x208 @ 29.97 fps (1501 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
x264 [info]: slice I:15 Avg QP:14.73 Avg size: 17798 PSNR Mean Y:52.56 U:53.59
V:53.82 Avg:52.89 Global:47.53
x264 [info]: slice P:1486 Avg QP:18.48 Avg size: 2205 PSNR Mean Y:49.65 U:50.87
V:50.94 Avg:49.99 Global:42.92
x264 [info]: slice I Avg I4x4:81.1% I8x8:0.0% I16x16:18.9%
x264 [info]: slice P Avg I4x4:0.6% I8x8:0.0% I16x16:1.0% P:56.2% P8x8:9.0%
PSKIP:33.2%
x264 [info]: PSNR Mean Y:49.68 U:50.90 V:50.97 Avg:50.02 Global:42.95 kb/s:566.1


encoded 1501 frames, 13.37 fps, 566.20 kb/s

Something is wrong there...

[edit] before people ask:

"x264.exe -B 500 -o file.mkv file.avs" for both encodes

[Sirber]

one .mkv and one .264?

[stephanV]

no... same command line for both encodes. the difference is in the frame rate

avis [info]: 384x208 @ 23.98 fps (1501 frames)

and

avis [info]: 384x208 @ 29.97 fps (1501 frames)

[Blue_MiSfit]

which version of mp4box did you use? where did you download it?

I downloaded the version that the guy that makes YAMB ( sorry, I forget who ) posted on the first page of his YAMB thread. This installer includes mp4box. Perhaps this is an ancient version...

Regardless I decided to start over. I updated mp4box to the latest version (from the link on the wikipedia entry for mp4box), added
SelectRangeEvery(400,20) to my avs, and ran a 2 pass with the same bitrate. The resulting 101mb 264 file was then muxed into MP4 with YAMB. Behavior was a bit strange, with the progress bar proceeding normally to about 50%, and then restarting. Muxing took about 30 seconds, and the resulting mp4 crashes vlc after a couple of frames. Also, the 16x16 size issue remains!! I may have to *gasp* use avi???

The 264 file can be played with mplayer, although the AR is very messed up (probably my fault, I usually assign one in mkvmerge). It is, however, NOT a stupid 16x16 image. Thus, it seems that x264 is doing its job. Therefore I must conclude that somehow mp4box is b0rking things up here :)

I am at a loss! Thanks for all the help so far everyone!

If anyone knows anything about windows xp media center edition 2005, please check out my thread at http://forum.doom9.org/showthread.php?t=98103


-MiSfit

[bond]

You can compile x264 with mkv output without GPAC (i do that) so it isnt that. It doesnt rely on GPAC in any way.if both .mkv and .mp4 have problems with 23.976 output it makes me think that maybe the encoder is doing something strange with that framerate

The 264 file can be played with mplayer, although the AR is very messed up (probably my fault, I usually assign one in mkvmerge). It is, however, NOT a stupid 16x16 image. Thus, it seems that x264 is doing its job. Therefore I must conclude that somehow mp4box is b0rking things up here :)this makes me think that your stream is already f*ed up. you shouldnt see a strange ar when playing the raw stream

[Sirber]

x264 --> .h264 --> .avi

works #1 for me and solved my problem.

[leowai]

no... same command line for both encodes. the difference is in the frame rate

avis [info]: 384x208 @ 23.98 fps (1501 frames)

and

avis [info]: 384x208 @ 29.97 fps (1501 frames)
Since different framerate reported using same command line...
Could this be the:
1. Bug in DGMPGDec (DGDecode.dll)?
2. AVIS "removegrainsse2.dll" plugin problem?
3. Or AVIS input of x264 problem?

[stephanV]

No, the frame rate change was my doing. If you look at the 23.976 fps one you can see it reports some obscure bit rate in the output info. With 29.97 everything is ok. It's weird, and I don't understand why it does this. I also get negative bit rates reported somewites.

[Blue_MiSfit]

if both .mkv and .mp4 have problems with 23.976 output it makes me think that maybe the encoder is doing something strange with that framerate

this makes me think that your stream is already fucked up. you shouldnt see a strange ar when playing the raw stream

I'm one of those folks who isnt really clear on the distinction between DAR and SAR... am I incorrect in assuming that if I set SAR to 16x9 in x264's commandline, that I should get a nice anamorphic encode that is actually 720x480 but when played will be vertically squished down to the proper 16x9 AR? If not, this would explain the strange AR in mplayer. If so, then something evi is afoot...

Again for reference, my command lines:

"x264.exe" --pass 1 --bitrate 1553 --stats "2pass.log" --keyint 240 --min-keyint 24 --ref 3 --bframes 2 --b-pyramid --filter -2,-2 --weightb --analyse all --8x8dct --qpstep 1 --sar 16:9 --progress --no-psnr --output NUL "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.avs"

"x264.exe" --pass 2 --bitrate 1553 --stats "2pass.log" --keyint 240 --min-keyint 24 --ref 3 --bframes 2 --b-pyramid --filter -2,-2 --weightb --analyse all --8x8dct --qpstep 1 --sar 16:9 --progress --no-psnr --output "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.264" "C:\Documents and Settings\Blue_MiSfit\Desktop\Movies 2\Battlestar Galactica Miniseries\Galactica.avs"


Also - @ leowai,
I am relatively sure that removegrainsse2.dll is a very solid denoiser with no bugs. I have been using it for about 6 months, with various formats - mpeg2, mpeg1, xvid, x264, etc and it has never caused any sort of frame rate issues. I would submit that the heavy development that dgindex has been doing through recently would make it the most likely candidate. Also, I don't know if/how avisynth deals with x264.

hrm.. now that I read stephanV's post - would it sound odd if the firstpass of my encodes ended with a final bitrate of -55000 or something like that? I assumed it was due to the fact that I don't actually write an output file for the firstpass to save space / fragmentation... Is that usual behavior?

Thanks again all..

-MiSfit

[stephanV]

I have no clue how it is related to frame rate, I rather doubt there is a connection. I'm getting wierd bit rates with other frame rates too... I must say I never payed much attention to it till now... I don't know if it causes the crash, coz im having difficulty getting x264 to crash with 23.976...

Something is wrong and one thing is the weird bit rates, but if its related to your problem and if that is related to frame rates... I don't know. Some how I have the feeling there's not an issue with 23.976 as such, as that would have been a problem that should have been observed more often, unless only PAL people use x264... and its not 1, 2, 3 reproducible anyways...

[bond]

sharktooth posted a potential fix to this problem, would be great if the people who reported this problem could try it and see if it fixed the issue

more info here (http://forum.doom9.org/showthread.php?t=98680)