Optimize function memory handling

[Dogway]

Sorry if Im a bit thick.
Im trying to make up a function to simplify things a little because depending on scene I need to use certain procedures.

If I run this one raw without a function, ram usage goes up to 2140 and render.
Inside a function like the one I show, it reaches 2175 and throws a System Exception (Out of ram I suppose). What's the difference and what is heavy about this? How can I optimize it?



...
setmemorymax(768)
mpg2source...
IVTC
...

trim(6037,6139).do(dejitter=true,stab=true,dups=true)+\
trim(6140,6451).do(dejitter=true,stab=true,dups=true)+\
trim(6452,6540).do(dejitter=true,stab=true,dups=true)+\
trim(6541,6671).do(dejitter=true,stab=true,dups=true)+\
trim(6672,6826).do(dejitter=true,stab=true,dups=true)+\
trim(6827,6922).do(dejitter=true,stab=true,dups=true)+\
trim(6923,6989).do(dejitter=true,stab=true,dups=true)+\
trim(6990,7150).do(dejitter=true,stab=true,dups=true)+\
trim(7151,7293).do(dejitter=true,stab=true,dups=true)+\
trim(7294,7403).do(dejitter=true,stab=true,dups=true)+\
trim(7404,7690).do(dejitter=true,stab=true,dups=true)

function do(clip clp, bool "dejitter",bool "stab",bool "dups"){
clp

# Dejitter
_fft = 6
_thSAD = 600
_bs = 16
_ov = 4
prefilt=clp.FFT3DFilter(sigma=_fft,plane=4,bw=32,bh=32,ow=8,oh=8,bt=5,ncpu=2)
osuper=clp.MSuper()
asuper=prefilt.MSuper()
b3v = MAnalyse(asuper, isb=true, delta=3, blksize=_bs, overlap=_ov)
b2v = MAnalyse(asuper, isb=true, delta=2, blksize=_bs, overlap=_ov)
b1v = MAnalyse(asuper, isb=true, delta=1, blksize=_bs, overlap=_ov)
f1v = MAnalyse(asuper, delta=1, blksize=_bs, overlap=_ov)
f2v = MAnalyse(asuper, delta=2, blksize=_bs, overlap=_ov)
f3v = MAnalyse(asuper, delta=3, blksize=_bs, overlap=_ov)
process = MDeGrain3(osuper, b1v, f1v, b2v, f2v, b3v, f3v, thSAD=_thSAD)
dejit=mt_merge(process, mt_edge("min/max").RemoveGrain(19), U=3,V=3)

# Dering
b=16
s=3
dejitter== true ? dejit.FFT3DFilter(sigma=s,sigma2=s/2,sigma3=s/4,sigma4=s/8,plane=4,bw=b,bh=b,bt=1,ow=b/2,oh=b/2,ncpu=2,sharpen=0.5,wintype=1)
\ : FFT3DFilter(sigma=s,sigma2=s/2,sigma3=s/4,sigma4=s/8,plane=4,bw=b,bh=b,bt=1,ow=b/2,oh=b/2,ncpu=2,sharpen=0.5,wintype=1)
deconv_h = "-1 2 1 -8 20 -31 38 -3 -32 100 -32 -3 38 -31 20 -8 1 2 -1"
mt_convolution (deconv_h, "1", Y=3, U=2, V=2).turnleft.vinverse.turnright.Crop(12, 0, -8, -0)

# Stabilise + Edge Filling
stab==false ? nop : InpaintLogo(last.addborders(8,8,8,8),Mask=msk.converttoyv12)
StabV = (stab==false) ? last : Stabilise(capabsolutemetric(15)).crop(8,8,-8,-8)

# Dup blend
return dups==true ? StabV.Dup(threshold=8,blend=true,copy=true,chroma=false,blksize=8,show=false) : StabV}

[Gavino]

Each function call creates a new instance of each of the underlying filters. Call the function once and do the trims afterwards.

do(dejitter=true,stab=true,dups=true)
trim(6037,6139)+trim(6140,6451)+ ...

If the function changes frame numbers (I haven't examined it in detail), perhaps you need to do it the other way round:
trim(6037,6139)+trim(6140,6451)+ ...
do(dejitter=true,stab=true,dups=true)

[Dogway]

In this case that is an option because everything is set to true, but the intention is to vary them for each trim cut.
But even so the problem would reside in Stabilise(), it doesnt have a scene change detector so it needs the trims. I wouldn't rely on any SCD as it has to be accurate either it would glitch right away.

Still I dont understand why everything is loaded at the same time. Does it try to load everything at start or it just calls it in order when a frame is requested? This is what I thought.

[Gavino]

The filters are all loaded at the start (compile-time), although they only produce frames on demand (at run-time). If a filter has a high load-time memory requirement (eg for static buffers, lookup tables, etc), that will be incurred immediately.

Perhaps you could rearrange your script to call the function just once for each unique set of parameters. Or if Stabilise is a problem, split the function up so that the bulk of the work can be done with a single call to one function and the Stabilise bit in multiple calls to a simpler function.

[Dogway]

Maybe a function inside a function? placing trims in the stabilise stage and reference them for the function call. It still would need some craft work depending on the trim instances I want but Im gonna give it a try. Im also going to read a bit about compile-time load...

edit: senseless if each trim needs different parameters. Im gonna read a bit, thanks for the help

[IanB]

As Gavino said the filters are all loaded at the start (compile-time). So make upto 8 instances then select the frames as required. i.e. ...
FlavTTT=do(dejitter=true,stab=true,dups=true)
FlavTTF=do(dejitter=true,stab=true,dups=false)
FlavTFT=do(dejitter=true,stab=false,dups=true)
...
FlavFFF=do(dejitter=false,stab=false,dups=false)

FlavTTT.trim(6037,6139)+\
FlavFTT.trim(6140,6451)+\
FlavTFF.trim(6452,6540)+\
....

[Dogway]

@IanB: By doing so avisynth hangs up with 60% cpu 1540Mb ram load and I need to terminate it.
I searched a bit through the documentation and this is the most I could find about:

parsing:
http://avisynth.org/mediawiki/The_script_execution_model/Sequence_of_events
functions:
http://avisynth.org/mediawiki/User_functions
http://avisynth.org/mediawiki/User_defined_script_functions

By instinct I have always tried to avoid functions, variables etc and write everything raw, but I would like to learn or read more about it if possible, I guess it has to do mainly because the static buffers and so on.

Another question is in the case of having the function arguments set as false whether it has an impact on the performance in the form of CPU or RAM consumption.
Autoanswer: Yes, it takes RAM space...

[IanB]

You are not taking the hint. Minimise the number of discreet instances of the various filters. Most of your Do function is independent of its argument values. My hint mealy reduced things to a possible 8 sets. With a little thought you really only need 1 instance of most things, 2 of a very few, ....

Functions are just syntactic sugar, calling them 8 times is the same as pasting the contents 8 times.

[Dogway]

I took the hint, I tested to the most austere condition.
FlavTTT=do(dejitter=true,stab=true,dups=true)
FlavTTT.trim(6037,6139)
But it doesn't make any difference (well, a hang up) I did in first place when testing in-function vs out-function filters, by just doing:
trim(6037,6139).do(dejitter=true,stab=true,dups=true)
one single instance too.

edit: I remember it rendered but started crashing after 4 or 5 instances. I take your idea, but it didn't work in this case I dont know why... I will investigate this because a simplier example did work.

About your last comment, I dont understand what is the problem about pasting the function content 8 times, what is the problem of a 500 lines script if everything it does is to read "boolean: false" 499 lines and "return: clip" 1 line?

[Gavino]

About your last comment, I dont understand what is the problem about pasting the function content 8 times, what is the problem of a 500 lines script if everything it does is to read "boolean: false" 499 lines and "return: clip" 1 line?
Nothing.
But your function is not like that, it does a lot of work even when all parameters are false, eg calling MDegrain3 and in particular calling FFT3DFilter twice.

[Dogway]

yes that is true...:rolleyes: well I didnt make them optional because I want them to work in all the trims. But mdegrain? isn't it supposed to be ignored when argument is false? can't variables be hidden under a boolean argument?

[Gavino]

But mdegrain? isn't it supposed to be ignored when argument is false? can't variables be hidden under a boolean argument?
You're right that no frames will be requested from the MDegrain part of the chain when dejitter=false. However, the filters will still be loaded at compile time. To prevent this you would need to make each of the variable assignments conditional, eg
prefilt= dejitter ? clp.FFT3DFilter( ... ) : NOP()
osuper= dejitter ? clp.MSuper() : NOP()
asuper= dejitter ? prefilt.MSuper() : NOP()
...

or use GScript:
if (dejitter) {
...
}

or extract the dejitter code into a separate function that is only called when needed.