x264 eat up 1g memory

[aolu11]

Hi,guys..
I just used x264 to encode a 1080p movie,I did not put anything in avs,just one line,that's the directshowsource,and used megui to encode,and then the memory used went up to 950+m then held there,is that normal?

parameters in mediainfo are something like:
cabac=1 / ref=4 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / constrained_intra=0 / bframes=8 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00

[sneaker_ger]

Yes, nothing to worry about.

[roozhou]

Play that video directly in your media player, then play the one-line .avs in the same media player. Compare the memory and CPU usage and you will be surprised by the magic power of avisynth.
AviSynth always tries its best to waste memory. It fills its buffer with a lot of frames which will never be read again. If you don't use SetMemoryMax at the beginning of the script, DirectShowSource will buffer more and more frames until it eats up all your memory.

[burfadel]

That amount of memory isn't unrealistic for HD encoding with x264.

[LoRd_MuldeR]

There is nothing wrong with Avisynth using the memory as cache and it certainly isn't a "waste". Leaving the memory unused would be the "waste". Caching frames in memory that have already been computed can speed up things a lot, as these same frames might be requested again very soon. And if the requested frame is still in cached in memory, it doesn't have to be decoded and processed again. Keep in mind that even when the encoder accesses the frames in a completely linear fashion, some of the filters used in your script might not! So as long as you don't run into any "out of memory" issues, you shouldn't worry about memory usage and you shouldn't have to use SetMemoryMax() at all. Also: While 32-Bit processes have a 2 GB memory limit, this limit is resolved for 64-Bit processes.

[roozhou]

There is nothing wrong with Avisynth using the memory as cache and it certainly isn't a "waste". Leaving the memory unused would be the "waste". Caching frames in memory that have already been computed can speed up things a lot, as these same frames might be requested again very soon. And if the requested frame is still in cached in memory, it doesn't have to be decoded and processed again. Keep in mind that even when the encoder accesses the frames in a completely linear fashion, some of the filters used in your script might not! So as long as you don't run into any "out of memory" issues, you shouldn't worry about memory usage and you shouldn't have to use SetMemoryMax() at all. Also: While 32-Bit processes have a 2 GB memory limit, this limit is resolved for 64-Bit processes.
A good filter system should be aware of how many frames each filter really needs (at least in linear access mode). The filter should keep the upstream frames (e.g. use smart pointers) when needed and release them immediately when no longer needed.

Even 500 frames is only 20 sec for a 25fps movie so if the user seek 1 minute backwards there is no chance that the target frame is still in cache.

[Wishbringer]

rc_lookahead=40
bframes=8
ref=4

That's all settings that increase usage of memory.
x264 has to store several frames in memory to compare them and to gain best compression out of them.

So it's not a "feature" of avisynth, it's in x264.
Else in taskmanager avisynth would take memory and not x264.

[nm]

Else in taskmanager avisynth would take memory and not x264.
AviSynth runs within the x264 process, so it doesn't get shown separately. Both x264 and AviSynth contribute to the 1 GB number.

[Groucho2004]

Else in taskmanager avisynth would take memory and not x264.

Not true. The calling application allocates the memory through the Avisynth API. You don't "see" Avisynth in Task Manager or Process Explorer.

Edit: nm beat me to it.

[Wishbringer]

I have with HD video a memory usage of 580MB for x264_x64.exe (rc_lookahead=60, but only keyint=48, so somewhat lighter settings than aolu11 with keyint=240)
Because I use avs2yuv to pipe video into x264 it's without AviSynth overhead.
With rc_lookahead=60 and full keyint it could be around 700MB.

Avs2yuv on the other hand uses an increasing amount of memory (see explaination for buffer of avisynth from Lord_Mulder)

So I didn't recognized that avisynth isn't in tasklist.

[aolu11]

Thanks for all the replies,I got it... Since the memory used was good for the encode speed,I'll leave it...anyway I'm not yet ran out of memory..

[skampy]

I'm curious.

Does increasing b-frames slow down the encoding speed in either CRF or 2pass mode? Obviously if you do a first pass with b-adapt=2, more b-frames will slow things down considerably but it seems to me my 2nd passes are slower too.

[nurbs]

It slows down both (first pass in 2-pass mode). Framtype decision runs in a single thread. With CRF you might not have a noticable slowdown if your other settings are so slow that frametype decision isn't a bottleneck. The second pass of a 2-pass encode shouldn't be slower though AFAIK.

[skampy]

So doing CRF with settings subme=9, ref=5, me=umh, merange=32, trellis=2 & rc_lookahead=50, more b-frames won't make a difference in encoding speed (I never have more than 7 as there is very little benefit with any more)?

I'll probably do some more testing before I speculate whether more b-frames slow down the 2nd pass of an encode... I suppose there isn't any logical reason why it should, is there? Frame type decisions are made in the first pass after all..

[AnonCrow]

So doing CRF ... more b-frames won't make a difference in encoding speed ?
Negligible, may in fact make it faster (assuming b-adapt=1 of course)

I'll probably do some more testing before I speculate whether more b-frames slow down the 2nd pass of an encode... any logical reason why it should

The only thing that would really measure is how long it takes to encode P-frames compared to B-frames. To somewhat properly test that, you'd do encodes with b-adapt=0 and varying fixed amounts of b-frames.

[Dark Shikari]

The only thing that would really measure is how long it takes to encode P-frames compared to B-frames.Longer.

[skampy]

Negligible, may in fact make it faster (assuming b-adapt=1 of course)

OK. So with CRF & b-adapt=2, more b-frames will slow things down?

Also, is b-adapt 2 even relevant for the 2nd pass?

The only thing that would really measure is how long it takes to encode P-frames compared to B-frames. To somewhat properly test that, you'd do encodes with b-adapt=0 and varying fixed amounts of b-frames.

So doing tests with increasing number of b-frames with b-adapt=2 with either CRF or 2pass modes would be pointless/useless? I know the 1st pass would be significantly slower but I'm really interested to know if there is any speed impact on the 2nd pass.

And thanks DS for your input, I thought that was the case. Is it inferred, because p-frames take longer to encode than b-frames, that the more allowed b-frames there are this would actually speed things up?

No doubt I'm way off base .. :stupid:

[LoRd_MuldeR]

OK. So with CRF & b-adapt=2, more b-frames will slow things down?

Yes. But remember that "--bframe" only sets an upper limit. Even with a limit as high as 16, x264 will hardly ever use that many consecutive b-frames.

Thus you can safely limit the number of consecutive b-frames to 4 or 5. The difference will usually be negligible, compared to 16.

Also, is b-adapt 2 even relevant for the 2nd pass?

Nope. Frame-type decision is made in the first pass of a 2-Pass encode.

[kemuri-_9]

Yes. But remember that "--bframe" only sets an upper limit. Even with a limit as high as 16, x264 will hardly ever use that many consecutive b-frames.

you of all people should know better than to blindly generalize like this without stating a type of content.

[LoRd_MuldeR]

you of all people should know better than to blindly generalize like this without stating a type of content.

There always are exceptions, of course. Nonetheless it generally makes sense to limit the maximum number of b-frames to the range that is actually used.

(After all, even the "very slow" preset uses only 8 b-frames, "slower" and below uses only 3. "--tune animation" adds two extra though)

[r00t61]

When I encode 1080p at --preset veryslow, my memory usage is around 1.2 GB.

When I encode 720p at --preset veryslow, my memory usage is around 600 MB.

This is with using no filters apart from a source() and possibly a crop/resize().

I'd always assumed this was typical x264 behavior.

[roozhou]

When I encode 1080p at --preset veryslow, my memory usage is around 1.2 GB.

When I encode 720p at --preset veryslow, my memory usage is around 600 MB.

This is with using no filters apart from a source() and possibly a crop/resize().

I'd always assumed this was typical x264 behavior.
You should encode from an uncompressed yuv/y4m file or a pipe instead of using avs script to see a typical x264 behavior.