What are the most RAM intensive settings in x264?

[Chengbin]

Just out of curiosity, what are the most RAM intensive encoding settings in x264?

I got some encodes where the filters take too much RAM, and would crash at the relatively high settings I use (around preset slower). But when I use low settings, like preset ultrafast, it would encode. I'm assuming it is hitting the 32 bit 2GB RAM limit. I don't want to encode a lossless version first then encode it with the settings I like.

[LoRd_MuldeR]

I'd guess ME=TESA with 16 references. Also it appears that the new MB-Tree patch seems to eat a lot of RAM (especially when increasing "--lookahead").

[Chengbin]

Oh, that might be the reason. I'm using mbtree as well.

[MasterNobody]

Just out of curiosity, what are the most RAM intensive settings in x264?
--refs, --threads, --bframes, --b-adapt 2, --me esa (tesa). And of course it depends from source resolution.

[kemuri-_9]

I got some encodes where the filters take too much RAM, and would crash at the relatively high settings I use (around preset slower). But when I use low settings, like preset ultrafast, it would encode. I'm assuming it is hitting the 32 bit 2GB RAM limit. I don't want to encode a lossless version first then encode it with the settings I like.

if you want to be that lazy about it, use avs2yuv to split avs and x264 into separate processes so they each have their own 2GB limits (or pipe to x64 x264 if you are able)

[roozhou]

But when I use low settings, like preset ultrafast, it would encode. I'm assuming it is hitting the 32 bit 2GB RAM limit. I don't want to encode a lossless version first then encode it with the settings I like.

This is not x264's fault but due to avisynth's stupid frame buffer management. By default avisynth buffers frames as many as possible, no matter downstream filters would ever use it. Actually 99% of filters references to no more than 5 previous frames.

The easiest way to solve this is add SetMemoryMax(xx) to your script and I recommend xx=16. Higher values does not bring any speedup at all but eats more memory and causes more cache miss.