AviSynth 2.5.8 [Dec 30th] - Release! (was RC-5 [Dec 27th])

[squid_80]

It crashes for me (huffyuv_ccesp, not with ffdshow decoding), only when loading via avisynth. It dies in huffyuv when decoding the last few symbols from the compressed frame. Huffyuv reads up to 4 bytes ahead in the input frame when decoding, it's this load instruction that's crashing:
02C14C58 mov edx,dword ptr [esi+ebx*4]
02C14C5B mov ebx,dword ptr [esi+ebx*4+4] <- FAULT
02C14C5F shld edx,ebx,cl ;pack input bits
The problem is huffyuv is trying to read beyond the end of the input frame, but the input buffer matches the size of the input frame exactly, with no slack allocated. Unfortunately I don't have a build environment setup for avisynth at the moment so I can't provide a definite patch. I did try enlarging dwSuggestedBufferSize by hex-editing the avi file but it had no effect. I'm guessing avisynth checks the index and allocates the input buffer based on the largest frame-size; the fact that in this case that size is exactly divisible by 32 might be part of the problem. In fact if 32 is added to the frame size (1535968), it's cleanly divisible all the way up to 2^12.

[Wilbert]

@Wilbert,

The huffyuv_ccesp-patch_025 from Doom9's download section.

So it does crash for you? or not?
Yes, as squid_80 says. I didn't try ffdshow decoding yet.

[IanB]

Still can't get an Avisynth to crash but I got an old VirtualDub to crash. So I can see what the problem is. It's a HuffYUV bug as Squid found in huffyuv_a.asm@632, in the HUFF_DECOMPRESS macro. It simply runs of the end of the input buffer.

A simple workaround is to add some guard bytes to the file read buffer in AVISource::DecompressFrame(). This code need 4 extra bytes but I decided to add 15 in case there are other smartarse codecs pulling the same trick but doing a read into an XMM reg.

Watch for an AviSynth_080712.exe or latter in the Installer texts language translations thread.

[squid_80]

This code need 4 extra bytes but I decided to add 15 in case there are other smartarse codecs pulling the same trick but doing a read into an XMM reg.While I think of it, xvid does the same thing when decoding b-frames but even worse - it checks ahead for a resync marker and if it finds one, attempts to continue decompression using data following the marker. The likelihood of an erroneous resync marker being found is low but not impossible, virtualdub's "display decompressed output" (http://forums.virtualdub.org/index.php?act=ST&f=15&t=15076) option has a high likelihood of setting it off. So even though I've never seen it cause a crash with avisynth it might be a good idea to fill the bytes immediately following the input data with 0xA5 (or whatever bitstring you choose) before passing the buffer to the codec.

[IanB]

Okay, I will fill with 0xA5's and 1 0x00

[RedDwarf1]

You may have downloaded a new version of AvsTimer, RemoveGrain, ReduceFlicker, or RemoveDirt, all of which require avsrecursion now. If they're in your autoload plugins, then they'll always load with every script, thus erroring every one.

Ah, thanks. That must of been the reason. I did try AvsTimer on one occasion and still had it in the plugins folder as I do tend to copy the plugins folder to new windows installs.

After deleting AvsTimer from the plugin folder it worked fine.

[Revgen]

@Revgen,

:confused: What version are you running? "AVIsource can't combine these files" is not a current message. The validation code for splice is :- if (vi.HasVideo() ^ vi2.HasVideo())
env->ThrowError("Splice: one clip has video and the other doesn't (not allowed)");
if (vi.HasAudio() ^ vi2.HasAudio())
env->ThrowError("Splice: one clip has audio and the other doesn't (not allowed)");
...
if (vi.width != vi2.width || vi.height != vi2.height)
env->ThrowError("Splice: Frame sizes don't match");

if (!vi.IsSameColorspace(vi2))
env->ThrowError("Splice: Video formats don't match");
...
if (fabs(fps_v1-fps_v2) > 0.000001)
env->ThrowError("Splice: Video framerate doesn't match");
...
if (vi.AudioChannels() != vi2.AudioChannels())
env->ThrowError("Splice: The number of audio channels doesn't match");

if (vi.SamplesPerSecond() != vi2.SamplesPerSecond())
env->ThrowError("Splice: The audio of the two clips have different samplerates! Use SSRC()/ResampleAudio()");

That wasn't the exact message, it was one of the ones in your example above. Basically it said that a clip with audio and a clip without audio couldn't be combined. I couldn't remember the exact message at the time.

[squid_80]

I recently upgraded my ram from 2 gigs to 8 gigs and there seems to be a problem somewhere in the core with memory addresses higher than 0x80000000. If I watch task manager, I get this message right before the memory usage reaches 2GB:
CAVIStreamSynth: System exception - Access Violation at 0x4a548612, writing to 0x0

As a workaround I can use setmemorymax(1500) (or some other value below 2048), but that doesn't seem like it guarantees any memory allocated will be below 0x80000000.

[IanB]

@Squid,

I think you will find you have simply exhausted the 2GB address space. Avisynth is absolutely crawling with code that does malloc's and does not check the result before trying to use it.

However I would expect some failures should be from env->NewVideoFrame() which does check, and should return "NewVideoFrame: Returned 0 image pointer!" or "NewPlanarVideoFrame: Returned 0 image pointer!" but no-one ever seems to see these messages.

:Edit: Yes I guess they won't because we should be checking vfb->GetWritePtr() which had the big malloc.

I have already clamped the initial SetMemoryMax to 1GB for the next RC (get a sneak peak from the Installer Translation thread).

And I guess you are now in a position to check if Avisynth survives life in a 3GB enabled host app. I believe the 1.8.* VDub's can be made 3GB aware quite easily.

[AnnaFan777]

after installing 2.58 RC2, I can't use avisource to open my DV files (ntsc).

"could not open video stream in any supported format"

not i can only open them with directshowsource.

windows98, sonydv codec.

any suggestions?

[squid_80]

Indeed, when using virtualdub compiled with /LARGEADDRESSAWARE the memory usage peaks around 2100M with no crash. I should have known since I've had avisynth64 chew up over 5G of memory before, but that's still mostly 2.55 code.

Reading up on GlobalMemoryStatus(), the MSDN documentation seems a bit lacking so I might run some tests. It worries me slightly that -1 is meant to indicate an overflow, but last time I looked a SIZE_T was unsigned.

[IanB]

@AnnaFan777,

Yes there is a bug with opening RGB24 decoded output format. I will fix it for RC3.

In the meantime you could try installing cedocida 2.0 (http://neuron2.net/misc/cedocida020.zip)

[IanB]

Changelist with respect to 2.5.8 RC2: See 1st post for full change list.

Additions:

* Added Polish doc translation (by Slim, Krismen & Co).

Bugfixes:

* Fixed AviSource() decoding to RGB24 logic regression.
* Added workaround for HuffYUV, Xvid reading past end of input buffer.

Changes:

* Test VFB data buffer for NULL malloc's, if so then Throw!
* Initial Memory Max value clamped to 1GB.
* Installer Start menu shortcut names translated.

As usual download from Sourceforge (http://sourceforge.net/project/showfiles.php?group_id=57023&package_id=105994&release_id=616887).
Enjoy! :cool:

Special thanks to the installer translation contributors.

[squid_80]

IanB: I tried out RC3 with virtualdub and the memory max clamp does it's job, but there's a small catch: when virtualdub's play input/play output buttons are used, it seems to open another copy of the input which subsequently chews up another 1Gb. When the total memory used reaches approx. 2Gb vdub freezes and I have to kill it.

Not sure if steps should be taken to fix this, because:
It's easily mitigated by using SetMemoryMax()
It only happens during preview, so a 30 hour encode won't be ruined by the last X frames being replaced with the "System Exception" message

[IanB]

@Squid,

This simple script should prove/disprove the multiple open theory :-BlankCLip()
WriteFileStart("ABC.txt", """Time("%c")""", True)This memory management issue needs some hard thinking. Maybe I should add an env->malloc() call to the API, but the how do you make filter authors use it in preference to the standard calls.

[smok3]

'portable' install seems to be gone?

[IanB]

Select the type of install: "Standalone"

[smok3]

ups, i missed that the first 3 times :/

[squid_80]

Looks like there's something else going on... the script you gave doesn't indicate multiple opening, but there's definitely something strange happening. Scrubbing the timeline shows memory usage increasing up to the setmemorymax value, then previewing increases it to 2x the setmemorymax value (freezing if it reaches 2Gb).

While researching I came across this: _set_new_handler (http://msdn.microsoft.com/en-us/library/a45x8asx.aspx). Whether it's usable or not (to flush/shrink the cache when a malloc fails and subsequently retry), I'm unsure.

[Blue_MiSfit]

I tried RC3 and noticed that Histogram() seems borken...

I reverted to 2.5.7 and it works fine. :D

~MiSfit

[IanB]

@Squid,

Unfortunately _set_new_handler is process global and Avisynth, being a library, should not change such things. Well I could flip it in the AviFile entry and restore it in the exit code, but that does not help the apps that do direct access via IScriptEnvironment.

Does a simple Colorbars() only script exhibit this behaviour? i.e. only has 1 VFB total.

Is this a memory leak outside of the SetMemoryMax controlled scope?



Histogram() seems borkenA little more detail might be nice. Added AudioLevels and Color2 modes to Histogram.
Fixed Histogram() Classic mode pixel values exceeding YUV limits.
Fixed Histogram() chroma plane initialization to 128.

Putting on my psychic hat, I intuit you are complaining about the brightness of the histo points. Previously they started at 0 and ramped up to 255 for a total populaton count of 64 and greater. counts less than 4 were ultra black and counts greater than 59 were ultra white. Now the values are between 16 and 235 proportional to the population.
> const int S2 = ((235-16) << 16) / (vi.width-256);
...
< p[x*2+vi.width*2-512] = min(255, hist[x]*4);
> p[x*2+vi.width*2-512] = max(min(235, ((hist[x]*S2)>>16) + 16), 16);

Might have been nice to discuss when it was first done, instead of waiting until we are doing release candidates :(

[Blue_MiSfit]

@IanB:

You're absolutely right I should have given you some more details. Sorry :)


MPEG2Source("I:\BluRay Rips\Girlgasmic\gg.d2v", cpu=0)

Histogram


Here's what I see with 2.5.7:
http://img507.imageshack.us/img507/7798/257hu9.jpg

And again with 2.5.8 RC3:
http://img511.imageshack.us/img511/9766/258bw6.jpg

FYI - the source is MPEG-2 1440x1080 anamorphic from a BluRay. In both images, the same frame is being decoded, but I fail to see the usefulness of histogram() in 2.5.8 RC3. Perhaps I need to set a parameter to get the "old" behavior of histogram()?

~MiSfit

[halsboss]

Just a lowly wondering ... is an MT http://forum.doom9.org/showthread.php?t=94996 version of v2.5.8 being considered in some way, aligning with this great work ?

[Seraphic-]

Is it recommend to use AviSynth 2.5.8 RC3 or should you still use AviSynth 2.5.7 for now? Thanks

[IanB]

@Seraphic-,

Unless someone finds a show stopper bug, RC3 will be renamed 2.5.8 official. So you had better test it, find and report any unliveable bugs you need fixed.

@Blue_MiSfit,

If you want Histogram fixed you need to find a bug worthy of me doing a RC4. I have significantly enhanced the brightness and scaling of the Histogram graph in the 2.6 stream. I need a more prominent bug before I can consider porting the new code back into the 2.5 stream.

[josey_wells]

I want to propose a change to the way that AviSynth allocates memory. I am running a computer with 8GB of memory running Windows Server 2008 x64. Windows Server will cache a lot of files in memory if available and report a lower free memory available then is really available. I would like to propose that available memory be based on amount of memory in computer, i.e Total Physical Memory.

For example from the V2.5.7 code

int ScriptEnvironment::SetMemoryMax(int mem) {
MEMORYSTATUSEX memstatusex;
__int64 mem_limit;

GlobalMemoryStatusEx(&memstatusex);
memory_max = mem * 1048576i64; // mem as megabytes
if (memory_max < memory_used) memory_max = memory_used; // can't be less than we already have
mem_limit = (__int64)memstatusex.ullTotalPhys - 5242880i64;

...
}

and from script environment

MEMORYSTATUSEX memstatusex;
GlobalMemoryStatusEx(&memstatusex);
// Minimum 16MB, otherwise available physical memory/4, no maximum
if (memstatusex.ullTotalPhys > 64*1024*1024)
memory_max = (__int64)memstatusex.ullTotalPhys >> 2;
else
memory_max = 16777216i64;
}

GlobalMemoryStatusEx was used instead of GlobalMemoryStatus since the non-ex problem supposedly has problems with memory above 4GB according to MSDN documentation

thoughts?

[IanB]

@josey_wells,

What you propose would be suicidal. Increasing the default to 50% of dwAvailPhys, when more than 256MB was available in 2.5.8, caused problems with 32bit processes readily exhausting the 2GB address space. I now clamp the initial default to 1GB. SetMemoryMax() can set the limit higher for those who think they know what they are doing.

I have previously experimented the using dwAvailVirtual instead of dwAvailPhys to restrict SetMemoryMax() calls and it always ended in tears. Either a crash or severe page faulting. Much of the code in Avisynth that does malloc's, foolishly does not check the return pointer for NULL. This inevitably leads to Null pointer exceptions in the least up to severe corruption at the worst. So any change in strategy to reduce protecting users from themselves also needs a fair amount of effort to seek and destroy all the unchecked malloc's that currently infest the code.

SetMemoryMax() sets the limit for the VideoFrameBuffer pool allowed for caching. It does not control or consider memory malloc'd for other uses within Avisynth, the plugins or the hosting application. For the cache to be useful at least 1 GetFrame call somewhere must request a given frame number more than once. If a recalcitrant script needs more memory in its GetFrame chain it will be given whatever it needs above the SetMemoryMax() imposed limit, possibly dieing in the attempt.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

For a library used with 32bit application GlobalMemoryStatus is preferred over GlobalMemoryStatusEx because it does clamp values based on the address space limits for a 32bit process i.e. 2GB (3GB with 4 GT RAM Tuning enabled).

Hint: When using GlobalMemoryStatusEx you must set MEMORYSTATUSEX statex; statex.dwLength = sizeof (statex); before your call.

[josey_wells]

IanB, thank for the explanation. The reason for the post was I kept getting Out Of Memory errors from AviSynth when running scripts with blu-ray processing, i.e. 1920*1080. I tried SetMemoryMax to no avail also.

My solution which seems to be working thus far is to disble large system file cache via a registry entry which frees available memory for programs. This seems to be working thus far.

But something is still amiss since I was getting failures when the program calling the AviSynth was approaching 1.3 GB of usage even though I have plenty more memory still available. I am using AviSynth 2.5.8 RC3 at the moment.

[IanB]

@josey_wells,

Are you explicitly getting this exception?ThrowError("GetFrameBuffer: Returned a VFB with a 0 data pointer!\n"
"size=%d, max=%I64d, used=%I64d\n"
"I think we have run out of memory folks!", size, memory_max, memory_used);Or are you getting some other error. Either way please report all the information.

At 1.3GB you are probably suffering from fragmentation. i.e there is 700Mb free and you want a 3Mb chunk but the biggest free chunk is only 2.9Mb but the good news is that there are 241 of them. This happens when varying size memory blocks are being malloc'd and then only some are free'd. Avisynth tries to avoid this by saving and reusing everything it malloc's, but not all the code adheres to this policy and 3rd party filter mostly do not.

:script: YV12 HD Frames are 3Mb, RGB32 HD Frames are 8Mb. Although inconveniently large L1/L2 cache wise, you still get a lot of them per GB. If you are using MVTools, complain in it's thread. It mallocs hugh amounts of memory for it's idx cache and this is well beyond Avisynth control. Best advice I can offer is to set SetMemoryMax to a lower value and pray.



@All,

Maybe I should revert to the old value, 25% of dwAvailPhys, for the initial SetMemoryMax, at 50%, there seems to be a lot of stuff crashing out.

.

[squid_80]

For a library used with 32bit application GlobalMemoryStatus is preferred over GlobalMemoryStatusEx because it does clamp values based on the address space limits for a 32bit process i.e. 2GB (3GB with 4 GT RAM Tuning enabled).

I did some checking and this isn't always the case. It's true if the pc has between 2Gb and 4Gb of ram but if the machine has >4Gb, the values are clamped to 2^32-1. (This was tested with LARGEADDRESSAWARE:NO explicitly set.)

So 25% of dwAvailPhys would be 1Gb... Same as current RC.

[IanB]

@squid_80,

So are you suggesting I clamp at 512MB as well as restore the init value to 25%.

[josey_wells]

@josey_wells,

Are you explicitly getting this exception?ThrowError("GetFrameBuffer: Returned a VFB with a 0 data pointer!\n"
"size=%d, max=%I64d, used=%I64d\n"
"I think we have run out of memory folks!", size, memory_max, memory_used);Or are you getting some other error. Either way please report all the information.

If you are using MVTools, complain in it's thread. It mallocs h
uge amounts of memory for it's idx cache and this is well beyond Avisynth control. Best advice I can offer is to set SetMemoryMax to a lower value and pray.

Maybe I should revert to the old value, 25% of dwAvailPhys, for the initial SetMemoryMax, at 50%, there seems to be a lot of stuff crashing out.


Yes that was the error message that I was getting. Since disabling the large system cache I have not had this error. Which indicates that too small of buffer was originally allocated because of low free memory.

MVTools - It wouldn't do me any good to complain in that post since I am the one working on an update multithreaded version. :) I have been replacing places in the code were several thousand news were performed with a single new. This should greatly help memory fragmentation.

IMO - I am in agreement with 25% of Available physical memory use the EX function for memory.

[squid_80]

@josey_wells: The LargeSystemCache registry key does not have any affect on x64 versions of windows. Also the cache lives in the kernel's virtual address space, not user space.

[josey_wells]

@josey_wells: The LargeSystemCache registry key does not have any affect on x64 versions of windows. Also the cache lives in the kernel's virtual address space, not user space.

By golly you are correct. Looks like Microsoft has deprecated the registry entry.

However, if I launch the application before a large amount of caching has taken place, i.e. high free memory reported, no problem. However, I haven't had a crash in a while since making a large amount of memory management problems in MvTools.

IMO - I still lean toward the proposed change.

[squid_80]

@squid_80,

So are you suggesting I clamp at 512MB as well as restore the init value to 25%.
Clamp availPhys to 2Gb and init to 25% - yes, same thing.

That's where my vote currently stands, although only because it seems to be the safest all-round fix. What sort of problems did using dwAvailVirtual cause? It would be nice if the cache could shrink/grow somehow based on the curent memory usage.

[IanB]

... problems did using dwAvailVirtual cause?Basicly there was no protection against insane memory_max values. You could always max out the address space with cache buffers, assuming you had enough disk space to VM your way there.

I am now thinking of limiting the default to an even lower value, like 300Mb. Most everyday scripts cannot really make use of even that much cache.

And in addition to the current constraints, I will also limit SetMemoryMax() calls to 75% of dwAvailVirtual, just to be sure. This will limit the total address space usage due to the Avisynth cache to 2.25GB on a 3GB aware application, (1.5GB normally).

As the current infrastructure is, we really need to stay well away from risking exhausting the address space.

[squid_80]

Ah right. I forgot about VM.

Something rather unrelated: Would there be an easy way to make avisource cache implicitly decoded frames? I think this is how it currently works, correct me if I'm wrong: if frame 5 (a delta frame) is requested and frame 3 is the nearest previous keyframe, frames 3, 4 and 5 are all decoded but only 5 is added to the cache. Subsequently if frame 4 is requested, 3 and 4 are decoded again. How feasible would it be to add an avisource parameter (bool recurse or some such) that executed recursive avisource calls back to the previous keyframe, caching each frame as it was decoded?

Failing that idea I guess it could be done with a custom plugin working from an index file created by prescanning the avi. Speed gains from using ICDECOMPRESS_PREROLL would be lost but they're normally negligible anyway (thanks a lot, packed bitstream mode).

[IanB]

Unfortunatly a child pclip cannot poke data into the parent cache pclip and xxxSource's are always at the end of the GetFrame chain.

You need a proxy filter to fetch the requisite frames through a cache instance. Currently I abuse ChangeFPS(last, last, True) to deal with linear access issue, but it is only good for upto 10 frame hops.

[Quark.Fusion]

But something is still amiss since I was getting failures when the program calling the AviSynth was approaching 1.3 GB of usage even though I have plenty more memory still available. I am using AviSynth 2.5.8 RC3 at the moment.
Look how much virtual memory was allocated — used physical memory is usually lower for avisynth, but crash will ocur when trying to allocate more that 2GB of virtual memory.

[Quark.Fusion]

Unfortunatly a child pclip cannot poke data into the parent cache pclip and xxxSource's are always at the end of the GetFrame chain.

You need a proxy filter to fetch the requisite frames through a cache instance. Currently I abuse ChangeFPS(last, last, True) to deal with linear access issue, but it is only good for upto 10 frame hops.

Hmm i don't know about that trick :) Always failing in chaining multiple avisynths with linear source.

What about making OutOfProcess() filter, that will call another avisynth process and process child filter/script there?
Like process=CreateNewProcess("""
clip1=ParentProcessSource(1)
clip2=ParentProcessSource(2)
param=GetParentProcessVar("somevar")
clip=SlowAndMemoryHogFilter(clip1, clip2, param)
EndOutOfProcess(clip)
""")
OutOfProcess(process, clip1, clip2)

result=ConcurrentOOP(process1=process, source1a=clip1, source1b=clip2, \
process2=process, source2a=clip1, source2b=clip2, \
process3=process, source3a=clip1, source3b=clip2, \
process4=process, source4a=clip1, source4b=clip2)
clip1=OOPextract(result, 1)
clip2=OOPextract(result, 2)
clip3=OOPextract(result, 3)
clip4=OOPextract(result, 4)


process=ImportAsNewProcess(avsfile)
clip1=SelectEvery(4,0)
clip2=SelectEvery(4,1)
clip3=SelectEvery(4,2)
clip4=SelectEvery(4,3)
result=ConcurrentOOP(process1=process, source1a=clip1, \
process2=process, source2a=clip2, \
process3=process, source3a=clip3, \
process4=process, source4a=clip4)
clip1=OOPextract(result, 1)
clip2=OOPextract(result, 2)
clip3=OOPextract(result, 3)
clip4=OOPextract(result, 4)
Interleave(clip1, clip2, clip3, clip4)

[Gavino]

Changelist with respect to 2.5.7:
Bugfixes:
* Fixed Parser FunctionTable::TypeMatch() missing compulsory arguments.

Is this the same thing that you are talking about here:
I recently enhanced the parser to handle overloaded function definitions based on the named argument, previously it only used the argument type pattern (ignored the arg names).
This might have a bearing on a problem I am investigating.

[IanB]

@Quark.Fusion,

:search: Posts by squid_80 and IanB discussing calling 32bit plugins from 64bit Avisynth.

@Gavino,

* Fixed Parser FunctionTable::TypeMatch() missing compulsory argumentsThis was a bug where a filter arg definition would declare arg 3 is mandatory, but a script calling the filter without 3 args would slip through unchallenged and crash the filter. I think it was Fizick who reported the bug. Review the CVS history to see what was done.

I recently enhanced the parser to handle overloaded function definitions based on the named argument, previously it only used the argument type pattern (ignored the arg names).Previously for filter arg definitions that only differed in the named arguments, only the 1st definition in memory could not be accessed. i.e for "tom", "cii[dick]" and "tom" "cif[harry]" the "harry" arg version was inaccessible.

This might have a bearing on a problem I am investigating.Don't hoard bugs, share them around. ;)

[Gavino]

Previously for filter arg definitions that only differed in the named arguments, only the 1st definition in memory could not be accessed. i.e for "tom", "cii[dick]" and "tom" "cif[harry]" the "harry" arg version was inaccessible.
...
Don't hoard bugs, share them around. ;)
Unfortunately, it's a problem reported in my own plugin. :(
It works OK on Avisynth 2.58 but not on 2.57, hence my interest in possibly-related changes.
It would make me feel better if I could blame it on a parser bug :devil: (especially a fixed one).

I have a nasty feeling it's related to the issue I raised here and the fix I came up with, which is similar to your "harry" example.

[IanB]

@Gavino,

Post 1174023, "MPEG-4 Encoder GUIs" -> "RipBot264 v1.10.5 - Simple and easy to use GUI -> IPOD . PSP . CONSOLES . BLURAY" #2317 by duramaximus discussing ripping m2ts streams. :confused:

Thread 139206, "Avisynth Development" -> "Calling internal function from plugin with same name" discussing monkey patching. :confused:

Neither issue was a classic "bug". With the first as long as the user called the function correctly there was no problem, if the arg list was incomplete then it didn't complain as expected. With the 2nd I removed a restriction that has always been present.

[Gavino]

Post 1174023 ... :confused:
Whoops, sorry, it should have been 1174203. I've edited the link now.

It seems the approach I took to enable my 'monkey-patching' relies on your parser extension. I didn't realise this was a recent change, so foolishly assumed without retesting that it would work on older versions of Avisynth. :o

[Zep]

But something is still amiss since I was getting failures when the program calling the AviSynth was approaching 1.3 GB of usage even though I have plenty more memory still available. I am using AviSynth 2.5.8 RC3 at the moment.

wow I wish I read this sooner. I see the exact same behavior in 2.5.7 and 2.5.8. When I reach around 1.3 GB things start to go crazy (crashes or freeze or Vdub just instantly quits) even though I have at least 700 megs free of the 2 gig address space (using XP 32 with 4 gig total physical ram) of that VDub process.

[Zep]

@josey_wells,

Are you explicitly getting this exception?ThrowError("GetFrameBuffer: Returned a VFB with a 0 data pointer!\n"
"size=%d, max=%I64d, used=%I64d\n"
"I think we have run out of memory folks!", size, memory_max, memory_used);Or are you getting some other error. Either way please report all the information.

At 1.3GB you are probably suffering from fragmentation. i.e there is 700Mb free and you want a 3Mb chunk but the biggest free chunk is only 2.9Mb but the good news is that there are 241 of them. This happens when varying size memory blocks are being malloc'd and then only some are free'd. Avisynth tries to avoid this by saving and reusing everything it malloc's, but not all the code adheres to this policy and 3rd party filter mostly do not.

.


so there is no way to get access to that 700 in continuous block? I thought most OS had some sort of cleanup call that defrags the memory. I know OSX does with handles i.e. a process uses handles which is a pointer to a pointer so blocks can move since your handle is always pointing to the correct address. if you need speed you can defrag the space then lock down the block(s) the handle(s) point to and deref the handle and access the block as if you have a direct pointer. This lets you do clean up when ever you want by unlocking everything. making a cleanup call then locking everything back down then use deref handles for speed etc...

anyway wish I read this thread sooner :)

[Leak]

so there is no way to get access to that 700 in continuous block?
That's not an OS problem, but a language/runtime problem. C/C++ and MSVC runtime hand out pointers to all memory they allocate, so compacting free memory is a sure-fire way to crash your program.

Java, Managed C++ and a boatload of other languages does Garbage Collection, though, but since AviSynth is C++ no can do.

[Zep]

That's not an OS problem, but a language/runtime problem. C/C++ and MSVC runtime hand out pointers to all memory they allocate, so compacting free memory is a sure-fire way to crash your program.

Java, Managed C++ and a boatload of other languages does Garbage Collection, though, but since AviSynth is C++ no can do.

It is an OS problem. If an OS does not supply low level managers then all dev tools/languages end up using a non standard way of doing things and their own libs and thus you get the exact problem we have on XP. On OSX all languages have access to handles because it is OS basic functionally. (memory manager is at kernel level of the OS and all request for memory are sent to it) There is no "C++ no can do" which is just crazy.

As for crashing of course if memory content moves and your pointers are made bogus you crash lol Why do you think I brought up handles? :)

[squid_80]

If you do a malloc on OSX you get a pointer returned (if successful), same as on windows. How is this any different?