Does each filter in avisynth get its own 2GB of RAM?

[Chengbin]

I'm upgrading to 8GB of RAM so I can run some heavy scripts. Since not many filters are written for 64 bit, I have to use 32 bit. So does each filter get 2GB of RAM by itself, or is it 2GB for the entire script?

[thewebchat]

Neither. Windows 32-bit API limits memory to 2GB per process.

[Redsandro]

I did notice that having 30+ clips imported into After Effects through AvsImporter leaves no memory for rendering on a 4GB machine, even though there's still memory left according to taskmanager. In this case I have to render full resolution proxy clips first. Because I can have a few hundred footage clips imported and still render, just not Avisynth ones.

So, although I don't know the details, there's some serious memory issues. You'd think the problem might be in the importer plugin, but both Premiere avisynth import filter 1.95 (http://urchin.earth.li/~tomford/avisynth/) and Premiere CS AVS Importer (http://valion.net/csavs/) show this behaviour, and they are both based on example code (http://neuron2.net/www.math.berkeley.edu/benrg/avisynth-premiere.html) from Neuron2 included with the Avisynth package. So my guess is it's something to do with the way Avisynth works.

I've never seen a single Avisynth clip eat away all my memory though, not even nearly.

[Chengbin]

So basically on a 64 bit system, with 32 bit avisynth and 64 bit x264, the maximum RAM for all avisynth filters is still 2GB?

Does 64bit Avisynth help?

Is there a list of 64 bit filters?

[Zarxrax]

Make sure to use the SetMemoryMax command, and set it to something sane. Avisynth unfortunately really sucks when it comes to allocating memory to more than 1 script, because each script likes to eat up as much of your systems ram as it possibly can. And the more ram you have, the more it wants to take!
So for instance, if your system has about 8gb of ram, and you want to keep 4 of it available for your applications like After effects, that leaves 4gb for your avs scripts. Since you've got 30 scripts, you can afford to give each one about 128mb of ram, so use SetMemoryMax(128).

[Redsandro]

Hey thank you! I didn't know that.

Most of my scripts are just colorspace conversion to rgb with gamma adjustment, because Adobe's MediaCore (Première, After Effects) cuts away supers and my camera does record them. I use them on all my recorded footage items to keep the extra detail in the bright colors (superwhite).

How can I find out how much memory the conversion/gamma/resize costs? Because it's tempting to use SetMemoryMax(1).

[thewebchat]

http://members.optusnet.com.au/squid_80/

Pretty much every 64-bit filter is listed above. I'm sure that, with some assembly coding skills, most filters could be recompiled on 64-bit. The 2 GB memory limit only applies to 32-bit processes, of course (although you can get 3GB with the /3G boot switch).

Also, SetMemoryMax only applies to the memory used by AviSynth's frame cache. It has no effect on the memory used by filters.

[Blue_MiSfit]

Do keep in mind that 64 bit avisynth probably won't work with After Effects :)

I regularly use avs2yuv and 32 bit avisynth - this at least lets me separate the avisynth process from the encoding process. As an added bonus, 64 bit x264 lets me wire up as much RAM as I can feed it, so I can bring on the really hungry stuff, like obscenely high lookahead valus :devil:

~MiSfit

[Redsandro]

For video I use all 32 bit apps, the incompatibility ended up being too confusing :P But how come you can use 32 bit avs with 64 bit x264?

[thewebchat]

32-bit and 64-bit processes can communicate over certain resources, including named pipes, STDIN/OUT, and shared memory. The program "avs2yuv" writes frames on a FIFO basis to STDOUT, which is managed by the operating system, and x264 can then read these frames from STDIN.

For example:
avs2yuv -raw -o - dongs.avs | x264 - 640x480 -o dongs.264

[Chengbin]

So can 64 bit avisynth use 32 bit filters?

[thewebchat]

Come on, try reading for once. Did I list DLL function calls as a way 32-bit and 64-bit processes can communicate? The answer is "of course not."

[Redsandro]

The program "avs2yuv" writes frames on a FIFO basis to STDOUT, which is managed by the operating system, and x264 can then read these frames from STDIN.

Thanks! Now I can use 64 bit x264 as well. I didn't before, because of 32 bit avisynth.

You reckon 64 bit x264 is faster, or is the extra bandwith only for insane settings purposes? Those lookahead values you mention, are they a notable quality difference? Because I'm definitely interested in quality improvements for my encodes. Here's my current approach:

x264.exe --profile high --pass 1 --bitrate 8000 --stats clip.stats --level 4.1 --keyint 24 --min-keyint 2 --ref 1 --no-mixed-refs --b-adapt 2 --direct auto --deblock -1:-1 --subme 2 --trellis 0 --partitions none --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 30000 --vbv-maxrate 40000 --qcomp 0.5 --me dia --thread-input --sar 1:1 --output NUL clip.avs --mvrange 511 --nal-hrd --sar 1:1

x264.exe --profile high --pass 2 --bitrate 8000 --stats clip.stats --level 4.1 --keyint 24 --min-keyint 2 --b-adapt 2 --direct auto --deblock -1:-1 --trellis 2 --partitions p8x8,b8x8,i4x4,i8x8 --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 30000 --vbv-maxrate 40000 --qcomp 0.5 --me umh --thread-input --sar 1:1 --aud --output clip.264 clip.avs --mvrange 511 --nal-hrd --sar 1:1

I am very sorry if this is too off-topic for the topic.



As for the memory we talked about above, I am guessing I could do:

1920x1080x3 = 6220800 ≈ 6 MB

provided that I do processing on a single clip. (no v=clip1 w=clip2...)
Leaving me to conclude that in order to be able to throw around imported avs files like it's nothing, the safest bet would be to include a line like so:

SetMemoryMax(8)

Am I wrong?

[Blue_MiSfit]

No. 64 bit avisynth cannot use 32 bit filters.

32 bit avisynth uses 32 bit filters, and can feed its output to a pipe or buffer, when then passes uncompressed video into 64 bit x264.

Lord_Mulder made a fantastic little GUI for automating this workflow. It's really really awesome :) Check this out for some benchmarks and a download.

http://forum.doom9.org/showthread.php?t=144140

There are substantial gains for extremely high settings, and the extra RAM for tons of --rc-lookahead is quite useful ;)

Also, are you targeting a BluRay / AVCHD disc? If not, you really need to revise your parameters. The small GOPs that BluRay requires aren't optimal for compression (at all), and are only useful for excellent seeking performance.

I usually do something like this for 100% transparent encoding, and DXVA compatibility

x264.exe input.avs --crf 16 --preset veryslow --tune film --level 41 --vbv-maxrate 40000 --vbv-bufsize 40000 --thread-input --output out.mp4

~MiSfit

[Redsandro]

Nice! Looks like 32 vs 64 bit speedwise is extremely marginal though, so I won't bother (because I would have to rewrite all my scripts to pipe), unless there's a notable quality improvement. But I don't think so. 32 bit allows for setting values to improve quality that aren't even notable.

[Chengbin]

Come on, try reading for once. Did I list DLL function calls as a way 32-bit and 64-bit processes can communicate? The answer is "of course not."

I'm not a computer programming expert, sorry if I didn't understand your technical response.

Anyway, that's unforunate. I hope work will be done to recompile existing filters to 64 bit.

[Redsandro]

Makes me curious. What you need so much memory for? Working on IMAX footage?

[Chengbin]

Makes me curious. What you need so much memory for? Working on IMAX footage?

1080p MDegrain multithreaded

Under 32 bit I can't get AnimeIVTC to run for resolutions over SD (and that's without MT)

x264's rc-lookahead 250 for 1080p video takes 3GB alone.

[Redsandro]

I didn't know that specific telecine was so sophisticated to use a lot of resources.
And I didn't know about rc-lookahead. I use MeGUI to construct a command line to put in my scripts, and there is no mention of this parameter anywhere, unless I inadvertently ignore it.
What does it do? I can't really understand from reading the settings page (http://mewiki.project357.com/wiki/X264_Settings).

[Chengbin]

I didn't know that specific telecine was so sophisticated to use a lot of resources.
And I didn't know about rc-lookahead. I use MeGUI to construct a command line to put in my scripts, and there is no mention of this parameter anywhere, unless I inadvertently ignore it.
What does it do? I can't really understand from reading the settings page (http://mewiki.project357.com/wiki/X264_Settings).

Part of mbtree.

rc-lookahead states the number of frames x264 will look ahead. Each look ahead frame is stored in RAM (uncompressed). Values above 60 give diminishing gains. But what the hell, what are we gonna do with 8GB of RAM?

[Redsandro]

Ah, thanks. It is hard to find any documentation on that, but I found this:

What does it actually do?

It tracks the propagation of information from future blocks to past blocks across motion vectors. It could be described as localizing qcomp to act on individual blocks instead of whole scenes. Thus instead of lowering quality in high-complexity scenes (like x264 currently does), it'll only lower quality on the complex part of the scene, while for example a static background will remain high-quality. It also has many other more subtle effects, some potentially negative, most probably not.

So looks like I could use that. Is it still a patch or is it in the official branch of x264?

Can you provide a line of code how you use that with other related settings?

[thewebchat]

Hey, I'm not a programmer either, but even I know that you can't link two binaries built for different architectures together.

[Redsandro]

Okay, about the linking, take it easy on the TS because it's not that obvious. I am not a programmer either, but apart from that, I am the smartest person in the universe and I didn't know either until I found out.
Windows seems to do a good job at playing nice for both 32 and 64 bit apps. So if you don't understand the inner workings, it's not a very strange question.

Just saying. :)

[Blue_MiSfit]

but even I know that you can't link two binaries built for different architectures together.


No, you don't - because plenty of people do it every day :)

This approach works perfectly fine on Windows and most any Linux distro (via WINE).

~MiSfit

[thewebchat]

Uh, Wine runs programs for the x86 architecture on x86 CPUs using x86 code. What is your point. I am pretty sure there are as many instances of people linking AMD64 code to x86 code as there are people linking ARM code in SPARC binaries.

[Zarxrax]

As for the memory we talked about above, I am guessing I could do:

1920x1080x3 = 6220800 ≈ 6 MB

provided that I do processing on a single clip. (no v=clip1 w=clip2...)
Leaving me to conclude that in order to be able to throw around imported avs files like it's nothing, the safest bet would be to include a line like so:

SetMemoryMax(8)

Am I wrong?

I believe that if Avisynth needs more ram, it will take it, even if you told it not to take that much. So SetMemoryMax is more like the amount it should "try" to use. I'm not sure if there is any downside to giving it too little ram. There probably is, though.

[thewebchat]

SetMemoryMax limits the number of previously generated frames that AviSynth can keep in memory. For example, some filters depend on previous/past frames from preceding filters. Generally, lowering the cache size doesn't do anything until you don't have enough room to store all the dependent frames, at which point performance will drop sharply. Increasing the cache size also lets you step backwards (when actually using AVS as a frameserver, not just a video filtering engine) farther without re-rendering frames.