[x264] Crash while encoding 1080p

[Karl Yeurl]

Hello everybody,

I'm experiencing some trouble with x264 while encoding 1080p anime files (duration : 25min, about 34k frames).

I have to warn you : I'm not really an encoding wiz, so I do many things I do not really understand, which means that I (for sure) do many things wrong.
I'm willing to learn, of course, but as far as now, I'm kinda a noob.

The description of my problem : I use meGUI as GUI (but I remplaced meGUI's x264.exe by the last release of x264 downloaded on x264.nl),
and when I try to encode an 1080p anime of 25 minutes, it crashes.

Well, a long time ago, the same presets worked. I could encode without any problems. Some time ago, it just crashed.
And now, it finally returns an error :).

The log says that the error is : "[Error] An error occurred x264 : [error] malloc of size 9142272 failed"

As far as I can tell, it's because an option that wasn't previously there tells now my computer to do things that it can't do.
I must admit that I try to balance my lack of skills by ticking many boxes and encode very slowly to get the best quality I can.

I read some posts here, and regarding to what I understood, it might be the mbtree, but it's just a guess.

One more thing, it doesn't crash with 720p vids, and it doesn't crash with short 1080p vids (I can encode the 1080p trailer without any problem).

There's the useful stuff for those who will understand the thing that happens better than me :

My command line (given by meGUI) :
program --profile high --preset fast --crf 17 --thread-input --deblock 1:1 --bframes 16 --b-adapt 2 --b-pyramid
--ref 16 --vbv-maxrate 40000 --merange 32 --me tesa --subme 9 --partitions all --trellis 2 --psy-rd 0.6:0 --no-dct-decimate
--no-fast-pskip --output "output" "input"

The error returned by the encoder :
[Error] An error occurred x264 : [error] malloc of size 9142272 failed

My config :

CPU : Intel core 2 duo T9300 @ 2.50GHz
RAM : 3Go of DDR2 @ 332.5Mhz
GPU (dunno if it's useful) : NVidia GeForce 8600M GT


The .avs :
I'm not gonna post it right here (it's pretty long), but it returns an audioless vid in YV12.

What I ask you :
I'd like you to tell/explain me what happens (if you need some extra info, just let me know),
and to help me to correct the command line to get rid of the pointless options I chose, to add the useful ones I "forgot".
Just one thing I know, because I heard a skilled encoder talking about it, on this anime, I have to keep the "chroma" (whatever it is :p).

Thanks for reading, thanks (in advance) for helping,
Karl Yeurl.

PS : As you probably noticed, I'm not a native, please forgive the spelling/grammar/other mistakes I (probably) made
(you can send me a PM with corrections if it's really too hard to bare for your eyes :rolleyes:).

EDIT : the formatting really sucked, I had to change that :p.

[LoRd_MuldeR]

[Error] An error occurred x264 : [error] malloc of size 9142272 failed

Memory allocation failed, so obviously you ran out of memory!

If your operating system supports it, upgrade to a x64 build (http://mirror01.x264.nl/x264/64bit/revision1281/x264.exe) of x264. This removes the 2 GB per process memory limit.

And if you can't do that, lower the "--rc-lookahead" value...

(BTW: You seem to use some bizzare settings! You use "--preset fast" but then "--me tesa --subme 9 --partitions all --trellis 2". What the ... ???)

[Dark Shikari]

(BTW: You seem to use some bizzare settings! You use "--preset fast" but then "--me tesa --subme 9 --partitions all --trellis 2". What the ... ???)He's using a completely broken encoding GUI, what do you expect?

I've given up attempting to help MeGUI users since the developers themselves don't seem interested in making anything work properly.

Use Ripbot264, Staxrip, or one of the other dozens of GUIs that actually works.And if you can't do that, lower the "--rc-lookahead" value...The memory usage due to the insane --bframes value, combined with tesa, combined with b-adapt 2, is much more of an issue.

[LoRd_MuldeR]

I've given up attempting to help MeGUI users since the developers themselves don't seem interested in making anything work properly.

Maybe people should grab the latest MeGUI update? :rolleyes:

http://img517.imageshack.us/img517/3954/meguix264.png

[Dark Shikari]

Maybe people should grab the latest MeGUI update? :rolleyes:Still broken. If you check "advanced settings" it adds a boatload of settings that contradict the presets automatically even if you don't do anything else.

And it doesn't help that the only way to get MeGUI updates is to watch all the Doom9 forum threads, find the post in the middle of nowhere that says to change your update server (because apparently the devs don't feel like fixing the old one), and insert it into a config box three levels deep inside the settings interface...

[LoRd_MuldeR]

Still broken. If you check "advanced settings" it adds a boatload of settings that contradict the presets automatically even if you don't do anything else.

You are right. But 99% of all user shouldn't need to touch any "Advanced" options now.

The biggest mistake is that they didn't hide the "Advanced Options" well enough ;)

And it doesn't help that the only way to get MeGUI updates is to watch all the Doom9 forum threads, find the post in the middle of nowhere that says to change your update server (because apparently the devs don't feel like fixing the old one), and insert it into a config box three levels deep inside the settings interface...

Just to make it clear for everybody: The latest MeGUI is on the automatic update server. Unfortunately the URL changed and needs to be adjusted by hand!

http://kurtnoise.free.fr/MeGUI/

[Dark Shikari]

It's on the automatic update server. Unfortunately the URL changed and needs to be adjusted by hand:

http://kurtnoise.free.fr/MeGUI/That doesn't help much the tens of thousands of users who don't read every thread on these forums...

We're going to spend the next 4 years dealing with users with old MeGUIs because of this.

[Karl Yeurl]

First of all, thanks for your really fast answers ! :)

Oh, and I want to apologize, I forgot to tell you that my OS was 32 bits (Windows Server 2003, but it actually does work like XP).

The memory usage due to the insane --bframes value, combined with tesa, combined with b-adapt 2, is much more of an issue.

I told you not to expect a well balanced command line :p.

I set bframes to 6, but decided to keep b-adapt at 2 (and tesa because I don't know what it's for), and it doesn't crash anymore.

I actually didn't know meGUI sucked that much. Well, I'm not against using the CLI, I guess I don't have to adapt too much meGUI's CL to get a working CL in my cmd.exe. If you prefer that instead of using meGUI, well I can at least give it a try :D.

Well, so now that I know why it crashed (since you confirmed my suspicions), please, could you help me getting a decent CL ? Which options should I {dis/en}able ? Are there parameter that are totally useless ?

Also, I'm willing to learn (really), so if you know some reliable links with contents understandable to someone like me who doesn't really get all the technical terms used in encoding, I'd be glad to read it. I already found mencoder's doc about x264, but it doesn't actually explain all the concepts I don't know.

Thanks again for your answers,
Karl Yeurl.

[Dark Shikari]

Well, so now that I know why it crashed (since you confirmed my suspicions), please, could you help me getting a decent CL ? Which options should I {dis/en}able ? Are there parameter that are totally useless ?x264 --help provides most of the information you need ;)

Just use the presets.

[Yoshiyuki Blade]

MeGUI was a good user-friendly tool when x264's options were ambiguous to the average user. But now that x264's presets makes things far easier to understand logically, MeGUI's current state is a mess, although the concept itself looks fine so far. The main issue is that you can't make a command like that simply comes out as: --profile --level --preset --tune --crf (or whatever rate control you want) and any other explicitly-defined options. That requires me to take a detour to the "Advanced Settings" which will flood the entire box, though unintentionally.

[Karl Yeurl]

Well, that's OK it seems it's working. And moreover, I succeeded with my attempt to use the command line. Is that just an impression, or does it go faster than with meGUI ?

Actually, Dark Shikari, I know how to use the presets (I read the helpfile some time ago), but I'd like to understand more x264 than just its presets. Playing with the options might be a good way to do so. :)

@Yoshiyuki Blade : yeah, meGUI is really user-friendly, that's why I began with it. But I didn't know the dev was messy right now. ^^

[Fr4nz]

You are right. But 99% of all user shouldn't need to touch any "Advanced" options now.


Not true Mulder; for example, if you want to make an encode which is blu-ray compliant (bufsize, maxrate, nal-hrd and so on...) you must use advanced settings. But if you do this, you override various preset parameters, which is BAD.

[Blue_MiSfit]

Yup, I just do everything on the CLI now... MeGUI needs to get sorted :(

Thanks for the hard work guys - I don't mean to knock it. MeGUI has been a fantastic tool over the last few years, and I hope it returns to proper usability WRT the new presets system.

~MiSfit

[Karl Yeurl]

Yup, I just do everything on the CLI now... MeGUI needs to get sorted :(

Thanks for the hard work guys - I don't mean to knock it. MeGUI has been a fantastic tool over the last few years, and I hope it returns to proper usability WRT the new presets system.

~MiSfit

meGUI remains a good tool for other encoders than x264, right ?
Or is that the same with XVID & other ones ? ^^

[LoRd_MuldeR]

Not true Mulder; for example, if you want to make an encode which is blu-ray compliant (bufsize, maxrate, nal-hrd and so on...) you must use advanced settings. But if you do this, you override various preset parameters, which is BAD.

Well, of course you must set the VBV parameters, if your target device requires VBV.

But that doesn't mean you can't use presets. And it certainly doesn't "override" any preset parameters! The presets simply don't cover VBV ;)

And there's absolutely nothing "bad" about it. Using a preset and/or a tuning together with the desired VBV values is perfectly fine...

[Fr4nz]

Well, of course you must set the VBV parameters, if your target device requires VBV.

But that doesn't mean you can't use presets. And it certainly doesn't "override" any preset parameters! The presets simply don't cover VBV ;)

Maybe I've expressed wrongly my point: let's suppose you choose a preset/tune, if you also select "Advanced settings" many parameters set by the preset/tune are overriden by MeGUI. Try yourself :D


And there's absolutely nothing "bad" about it. Using a preset and/or a tuning together with the desired VBV values is perfectly fine...

With the latest version (1056) absolutely not...try it!

[LoRd_MuldeR]

Maybe I've expressed wrongly my point: let's suppose you choose a preset/tune, if you also select "Advanced settings" many parameters set by the preset/tune are overriden by MeGUI. Try yourself :D

But that's only a bad/inconsistent implementation in current MeGUI, which hopefully will be fixed one day ;)

It definitely shouldn't be that way in a proper GUI. And it definitely isn't that way if you run x264 from the console.

My idea of a simple x264 GUI looks like this:
http://img204.imageshack.us/img204/4668/launcher.jpg

[Fr4nz]

But that's only a bad/inconsistent implementation in current MeGUI, which hopefully will be fixed one day ;)

It definitely shouldn't be that way in a proper GUI. And it definitely isn't that way if you run x264 from the console.

My idea of a simple x264 GUI looks like this:
http://img204.imageshack.us/img204/4668/launcher.jpg

Yeah yeah, I didn't mean to say that x264 command line parsing is broken obviously!

Anyway I found a little "workaround" in order to use the partially-broken x264 GUI of MeGUI: if you choose tune "default" and preset "medium", then you can activate safely "advanced settings"; from there you can customize your command line :D

PS: Your simple GUI is good ;)

[JohannesL]

Since MeGUI started failing, I've done all my encoding by editing Script.avs and running Encode.bat. Encode.bat uses x264, wavi (for piping audio), oggenc and mkvmerge. Perfect :)

[Rumbah]

The latest StaxRip 1.1.1.9 beta handles the x264 preset stuff pretty perfect.

On the front tab you can select preset and tuning, and on the advanced tabs everything ist changed according to your preset/tuning settings. If you edit the advanced settings, only the things changed from the standard are taken to the command line.
That's the way it should be in MeGui, too (and hopefully will someday).