Thanks for the Memory... (memory_max problems)

[zettai]

Nope, this isn't about Lisa Yates, it's a question about memory usage and Avisynth and how this seems to be causing crashes when loading up multiple instances.

Myself and many of my associates regularly edit in Adobe Premiere from mpeg2 using avisynth for importing. This can sometimes include over 20 avs files in one project.

It seems that the default for 'memory_max' is made when there are no frame buffers allocated - on creation of ScriptEnvironment. However, in terms of loading them into Premiere this happens when you load up the project... i.e. at the same time for all 20+ files.


For example: Lets assume that when Premiere loads the timeline and opens the .avs files you have 700Mb of free RAM, in that case it seems that each instance will set 'memory_max' to something close to 175Mb. Now lets assume that you load 20 .avs files that means that AVS thinks it has 3.5Gb of RAM to play with (across all the files). That's a lot :) This tends to explain my personal observation that this problem occurs most on systems with lots of physical ram.


This is just my presumption about how the memory is handled in avisynth but it does seem to explain why when using mulitple instances of mpeg2 importing vob files we have avisynth crashing.


Now, I'm hoping that SetMemoryMax() can fix this in most instances but I'm also worried about the fact that when incoming frame > memory_max then a new buffer could possibly be made regardless of setmemorymax()... just looking at the code, that is.

Is my analysis of how avisynth memory management works correct? If so, is there any way to improve this functionality within avisynth? I was presuming that there was a memory leak in either avisynth or mpeg2dec but it looks like avisynth is designed to work like this but doesn't work well with multiple instances that are loaded together.

Thanks in advance for any insight into this problem.

[sh0dan]

AviSynth does have a basic need for framebuffers, since many filters require several frames at the same time to produce the required results.

If it cannot do this without breaking the "Setmemorymax", it WILL allocate more memory.

The decomb filters for instance require to have at least 5 or 6 framesize images to be able to operate. Many filters have similar requirements.

Furthermore several filters allocates memory to operate, beside the frames they request. Dust, SmoothHiQ are two filters that require a fair amount of memory to operate. The memory the filters allocate is NOT included in the setmemorymax() limit. Setmemorymax() only controls how much memory can be used for internal frames.

[zettai]

That all makes sense, although I'm still concerned (in a general sense) about the crashing itself. Let me explain what happens in greater detail:


A basic avisynth script used, such as a simple MPEG2Source(...) but there are, say, 25 avs scripts being loaded. Avisynth crashes after processing a few hundred frames of something being exported. Is this due to avisynth presuming there is more memory there than there is available or is it simply there not actually being enough memory for the operation?

Given that these scripts work fine on their own I'm curious as to why exporting a video that cuts together scenes from different scripts would cause lack-of-memory issues given that different scripts are rarely being processed at the same time (just in fades, etc...)


Now, I am going to presume that for the most part setmemorymax() will fix issues with multiple scripts that arent procssing frames at the same time. I can understand if it doesnt fix issues to do with serious overuse of memory (such as compositing very large numbers of complicated avs sources into one frame) but for basic cutting operations using multiple avs sources shouldn't really pose much of a problem, should it?

[sh0dan]

I cannot claim to fully understand MPEG2DEC, but I seem to remember that it has much internal buffering - I cannot recall if it is from I-frame to I-frame (mostly 15 frames) or from P-frame to P-frame (usually 2-3 frames).

There is little code to handle "out-of-memory" situations - and I wouldn't wonder if AviSynth or MPEG2DECn would simply crash, if it didn't get the memory it needed.

[zettai]

I'm not debating that possiblity, as I said.

What I don't understand is why having LOTS of avs files causes the problem, given that when exporting from a timeline you are accessing the avs files individually and using a very similar amount of buffering as if you are just using one or two avs files.

I can have projects with 5 or 6 avs files and have no memory issues at all.... increase that to double figures and crashes can occur. However, the complexity of the indivdual frame access has not changed from video to video. In terms of what is being processed during each frame it shouldnt matter that in the entire proect I have 20 sources because only 1 or 2 avs sources are having frames processed at any one time.... (unless I've totally missed the boat on how these buffers work).

So I can only presume it's something to do with how memory is allocated when the avisynth files are loaded into the projcet and not to do with the state of a frame buffer at any one time during an export.


The question, for me, is why there would be an out-of-memory issue when accessing many avisynth files in serial when there isn't a problem when acessing just one or two if there wasn't an allocation problem (as all of the files are always "open" but not all serving frames at any one time)

[Bidoche]

Since VC6 operator new returns NULL when lacking memory, out of memory will always result in a crash.
This should be tested or better use an (good) implementation that throw the bad_alloc exception as the C++ standard suppose.

Besides, avisynth dll is only loaded once into memory and maybe there are collision issues with all the scripts sharing frame memory. I am not sure the memory limit is handled per graph or globally.

[zettai]

This is a post from a colleague of mine, kmv, who would post here but has only just signed up so needs to wait 5 days:



What zettai is alluding to is that the out of memory condition would seem to be a bug, and a very solvable one. From what we can see there are two things going on here that effect the problem:

Point One

ScriptEnvironment::ScriptEnvironment() : at_exit(This()),
function_table(This()), global_var_table(0, 0) {
MEMORYSTATUS memstatus;
GlobalMemoryStatus(&memstatus);
// Minimum 16MB, otherwise available physical memory/4, no maximum
memory_max = max(memstatus.dwAvailPhys / 4,16*1024*1024);
memory_used = 0;

As has been noted in these forums before and can be seen in the code sample below, there is no upper bound on the default for 'memory_max'. This means that the amount of memory "allocated" is directly proportional to the total available RAM in the computer (a quarter of it) when the script loads.



Point Two

LinkedVideoFrameBuffer* ScriptEnvironment::GetFrameBuffer2(int size) {
// Plan A: are we below our memory usage? If so, allocate a new buffer
if (memory_used + size < memory_max)
return NewFrameBuffer(size);
// Plan B: look for an existing buffer of the appropriate size
for (LinkedVideoFrameBuffer* i = video_frame_buffers.prev;
i != &video_frame_buffers; i = i->prev) {
if (i->GetRefcount() == 0 && i->GetDataSize() == size)
return i;
}
// Plan C: allocate a new buffer, regardless of current memory usage
return NewFrameBuffer(size);
}

When a new frame buffer is requested the standard case, "Plan A", is to always create a new buffer if 'memory_max' has not been reached, it is only after that point that any attempt is made to reuse old frame buffers.


Effect

If there are a number of were loaded at the same time (for example: when Premiere loads a project) then they will all get a default 'memory_max' that very close to a quarter of available RAM at that time because no frame buffers have been requested yet.

If you have a machine that has a lot of RAM (lets say 1Gb) it is entirely possible that 700-800Mb will be available when Premiere loads the project meaning each script might get 200Mb.

Now assume there are 20 scripts (or even more), each assuming they have 200Mb to play with - that is a total of 4Gb, lots more than the RAM and I'm willing to bet lots more than the swap too.

If the computer only has 100Mb free then its 25Mb each and a total of 500Mb which will probably fit in most people's swaps.

In other words: the more RAM the computer has the faster AVS will eat it up before attempting to reuse old frame buffers.


Solution?

There are several things that could be done, here are some ideas for what its worth:

* An upper bound on the default 'memory_max' might be a start
* Make reusing frame buffers the default case, or at least have a higher priority
* Since you have overridden the new operator you could check for low memory conditions and throw back to a reuse attempt
* You could even make a statement saying AVS was never intended to process that many files at once and we are on our own - although I don't like this option much

[Bidoche]

An upper bound on the default 'memory_max' might be a startWe probably can have environment instance warn each other to correctly share available memory
Make reusing frame buffers the default case, or at least have a higher priorityProblematic, generally it's cached frames who are reycled, so it will increase cache misses.
That's why plan A goes first.
Since you have overridden the new operator you could check for low memory conditions and throw back to a reuse attempt
It's impossible to globally override new, what I was talking is to go to another STL implementation, like STLPort which I use for 3.0.
You could even make a statement saying AVS was never intended to process that many files at once and we are on our own - although I don't like this option much I don't like it either

[sh0dan]

I very much agree with Bidoche.

The design of AviSynth makes it "memoryhungry".
The only way for you to limit that is to use SetMemoryMax in each script, or otherwise render your files out to temporary huffyuv files.

The very nonlinear nature of avisynth means that it need to cache many frames to be effective.

[zettai]

Response to Bidoche from kmv:





An upper bound on the default 'memory_max' might be a start

We probably can have environment instance warn each other to correctly share available memory


Wouldn't the ideal be that all instances shared the same frame buffer memory pool? I'm sure there are a bunch of thread safe classes floating around for this.




Make reusing frame buffers the default case, or at least have a higher priority

Problematic, generally it's cached frames who are reycled, so it will increase cache misses.
That's why plan A goes first.


It was my impression from the code that the cache was off by default and only turned on when the (apparently) undocumented Cache() function was found in the script.


In any case, coming back to zettai's original question, can we work around the problem for now by always including a SetMemoryMax() function in the scripts with a low value? For example: 10.

We can live without the speed for a while, having it not crash would be worth it.

[sh0dan]

The Cache filter is automatically inserted between all filters, when the filter graph is created. This allows for filters to request past or future frames, so they don't have to be recreated from the source filters.

For 2.5 we tried to implement cache hints, to be better able to control frame caching - this proved however to be a lot more complicated than we expected. The main problem is that multiple filters can request frames from the same cache.

An example:

vid = avisource("file.avi")
vid2 = tweak(vid,...)
mergechroma(vid2, vid.blur())

For each output frame, avisource is requested to deliver the same frame twice. Luckily there is a cache after avisource, that caches this frame, so the the AVI file does not have to be read twice.

A more complicated example:
vid = avisource("file.avi")
vid2 = tweak(vid,...)
vid2 = trim(vid2,5,0)
mergechroma(vid2,vid.blur())
Now the two sources are a few frames apart, so the AviSource cache again help and avoids duplicate reads. This is a bit "non-real-world", but a realistic script could be:
vid = avisource("file.avi")
vid2 = convolution3d(vid,...)
mergechroma(vid2,vid.blur())
Convolution3D need the previous frame, the current and the next one. Without a cache (or with a very limited one), this would run horribly slow.
The idea behind cache hints would be that c3d could send a hint to the avisource cache, that it will at most need three frames cached. But since it isn't possible to see which filter is calling it isn't possible to control that.

[Bidoche]

Wouldn't the ideal be that all instances shared the same frame buffer memory pool?It probably would, but that's not the case.
Actually there is a global linked list holding VideoFrame for their recycling, which is probably unworthy since VideoFrame are small objets. Besides it holds all instances even those who are not electible to recycling (aka linked)...
And there is another linked list of VideoFrameBuffer in each env. Same problem here, it holds linked instances too.

I'm sure there are a bunch of thread safe classes floating around for this. I just noticed that the global VideoFrame recycle_bin is not synchronised, so it may have issues when using multiple scripts... Maybe that's the problem