x264 (overload memory size)

[sarmano]

Hello!

I did not know where to write about an error if it is an error.
During coding video at allocation of accessible memory through a script avisynth
SetMemoryMax (3000)
x264 falls during the moment of accessible memory only 1700Mb.

Is it possible to expand a range of accessible memory?

Regards,
sarmano.

[nm]

There is no need to give AviSynth 3000 MB of memory for any sane video encoding task. The error you get probably means that you simply don't have that much RAM (+swap space) in your machine when AviSynth tries to reserve memory. Try SetMemoryMax (64) if you need to set some value, or just don't specify a maximum.

[sarmano]

I have 8 Gb of RAM.
This sets isn't for high quality, this is for speed of encoding.

I try testing a few configuration of memory and come to conclusion: more memory size - much faster encoding without loss of quality (about x2-x4 speeds аgainst default)

[Sharktooth]

and i guess you have a 64bit processor and OS and avisynth build etc...
otherwise 8gb are useless ;)
btw 3gigz for avisynth is a bit of an overkill and probably it's causing that problem

[sarmano]

I have P4 3.6 only.
OS has a 32-bit mode.

3gigz for avisynth is a bit of an overkill
x264 has crash error at only 1.7 gigz.

btw: SetMemoryMax (1500) - stable (about x1.5-x2 speeds),
[ threads = 2 ]

Sharktooth
Is any command line for x264 to set limits of memory?

[nm]

x264 has crash error at only 1.7 gigz.That is also overkill and I doubt it's x264 that is crashing, but AviSynth.

btw: SetMemoryMax (1500) - stable (about x1.5-x2 speeds),
[ threads = 2 ]
Have you compared that to a lower setting such as SetMemoryMax(150) or only to the default settings?

Is any command line for x264 to set limits of memory?Not really, but x264 doesn't need much memory anyway.

[foxyshadis]

x264 doesn't use the memory, avisynth does, and x264 can't control the allocation patterns of the external functions it calls. You might want to try recompiling avisynth and x264 with support for /3GB. Certain plugins also allocate their own.

Are you running with PAE? Neither avisynth nor x264 support that, so you will never be able to utilize more than 3gb in 32-bit windows.

However, the question remains, why are you doing this? Avisynth rarely needs more than 200 megs, even with complex scripts, but it'll use as much as you give it whether it can address it or not. If you're using tons of mvtools with HD, most of the memory is allocated inside mvtools itself, so you'd have to recompile that with /3gb too.

[sarmano]

nm
You are right this is my mistake.
I compare time of encoding and result:
SetMemoryMax (1500) = SetMemoryMax (150)
:(

foxyshadis
However, the question remains, why are you doing this?

I have tried to load system as much as possible rather its parameters.
At check of quantitative loading memory by programs, in the sum, I use actively memories about 400Mb from available 8Gb.
The majority of programs also do not demand the big loading of memory but when it is a question of video compression, the question rises sharply.

My system works in AHCI a mode without a swаp-file.
All operations working directly through memory without the postponed record to a disk.
I didn't nothing changed for a PAE mode and do not know where more precisely to look it's enabled or not.

[Theliel]

well, a little explication.

32bits architecture can access only to 2^32bits = 4Gb.
PAE avoid this limitation, mapping memory space up to ¿64? dont remember well.

Windows XP (not x64 edition) or Windows Vista x86 have got this limitation as well. In windows are more amuse, because OS only can access to 3gb. If you want use more memory in 32btis architecture, you need your app will be compile and "preparate" to operate with PAE mode. So I dont belive you use your 8gb in the most of times.

If you will use A 64Bits processor and X64 OS, this limitation disappear.
64bits can access to 2^64= a lot of ram :)
But even when you use 64Bits OS and 64Bit CPU, you need your App will be compile and prepare to operate in 64bits, x624, avysinth and any plugins of avysinth use, should be 64bits version.

(in this moment, x264 versiom and avysinth and alot of filter for this are avalible in 64bits)

[ChronoCross]

PAE maps up to 36 bits if I remember correctly however at any time a single application can only access 2GB of memory.

[squid_80]

Someone will have to donate 2x1gb (or larger) ddr2 memory for me to be sure that 64-bit avisynth can access >2gb memory. I currently have 4x512mb with no free slots. Until then I make no claims that 64-bit avisynth/x264 will solve the problem.

[foxyshadis]

You don't need actual memory to test. Just set your swap file to 1-10gb and use three instances of wdresize with a rather large photo for source image. Instant enormous allocations without a ton of swapping, at least until it gets to the second and third iterations. (The original version of wdresize allocated much more still, it might be possible to make a version that breaks win64 altogether. :p)

[squid_80]

You don't need actual memory to test.Sshh. I'm trying to score free ram here.Just set your swap file to 1-10gb and use three instances of wdresize with a rather large photo for source image. Instant enormous allocations without a ton of swapping, at least until it gets to the second and third iterations. (The original version of wdresize allocated much more still, it might be possible to make a version that breaks win64 altogether. :p)Meh, too many plugin dependencies to wade thru and port just for an experiment. Also the resizer instances don't take up nearly as much memory in avisynth64 since they don't use softwire for dynamic assembly.

[Shinigami-Sama]

have you tried the /3gb switch
it allows the applications to use upto 3gb each, else windows tries to limit it to 2gb to allow the OS to have 2gb
it's pretty much required for Exchange...

[squid_80]

Hmmm, running under win x64 both 32-bit and 64-bit x264 are able to use more than 2gb ram. Using setmemorymax(3072) in the script, that's exactly how much is used. Can't seem to push it any higher though.
(There also seems to be a severe handle leak in my win64 pthreads build too.)