Running out of memory when processing parts of a script differently

[zero9999]

i'm running a script that debands parts of a video (1080p, 16-bit stacked) differently, which includes using different masks for protecting details.

Trim(0,23) +\
Trim(24,100).DebandStrong("aamask_fine",false,true) +\
Trim(101,174).DebandLight("aamask_fine",false) +\
Trim(175,196).DebandStrong("aamask_fine",false,true) +\
Trim(197,948).DebandLight("aamask_fine",false) +\
Trim(949,984).DebandStrong("aamask_dull",false,true) +\
Trim(985,1269).DebandLight("aamask_fine",false) +\
Trim(1270,1313).DebandStrong("aamask_fine",true,true) +\
Trim(1314,1629).DebandLight("aamask_fine",false) +\
Trim(1630,1671).DebandStrong("aamask_veryfine",true,true) +\
Trim(1672,1933).DebandLight("aamask_fine",false) +\
Trim(1934,2044).DebandStrong("aamask_fine",true,true) +\

the functions look like this:

function DebandStrong(clip c, string "mask_name", bool "less", bool "addgrain", string "mask_file")
{
mask_name=Default(mask_name,"aamask_fine")
addgrain=Default(addgrain, false)
mask_file=Default(mask_file, "none")
less=Default(less, false)

mask=BlankClip(c,width=c.Width,height=c.Height/2).KillAudio().Subtitle(mask_name)   
mask = (mask_name=="aamask_dull") ? c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.1,c=0.35).tcanny(sigma=1.2, mode=1, plane=7).RemoveGrain(27).mt_lut("x 13 < 0 255 ?" , U=1, V=1).mt_binarize() : mask
mask = (mask_name=="aamask_fine") ? c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.1,c=0.35).tcanny(sigma=1.2, mode=1, plane=7).RemoveGrain(27).mt_lut("x 10 < 0 255 ?" , U=1, V=1)\
.mt_logic(c.DitherPost(mode=6).MinBlur().Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.45).FastlinedarkenMOD(strength=40,thinning=0, luma_cap=255, threshold=0).MinBlur(2).mt_edge(mode="hprewitt"), mode="or").mt_binarize() : mask
mask = (mask_name=="aamask_veryfine") ? c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.1,c=0.35).tcanny(sigma=1.2, mode=1, plane=7).RemoveGrain(27).mt_lut("x 7 < 0 255 ?" , U=1, V=1)\
.mt_logic(c.DitherPost(mode=6).MinBlur().Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.45).FastlinedarkenMOD(strength=40,thinning=0, luma_cap=255, threshold=0).MinBlur(2).mt_edge(mode="hprewitt"), mode="or").mt_binarize() : mask
mask = (mask_name=="aamask_ultrafine") ?  c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.75).tcanny(sigma=1.1, mode=1, plane=7).mt_lut("x 7 < 0 255 ?" , U=1, V=1)\.mt_logic(c.DitherPost(mode=6).MinBlur().Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.45).FastlinedarkenMOD(strength=40,thinning=0, luma_cap=255, threshold=0).MinBlur(2).mt_edge(mode="hprewitt"), mode="or").mt_binarize() : mask
mask = (mask_name=="special") ? ImageSource(mask_file,0,c.FrameCount-1,c.FrameRate).ConvertToYV12(matrix="PC.709").mt_binarize() : mask


debanded= !less ? c.GradFun3(lsb=true, lsb_in=true, thr=0.75, radius=16, smode=2, elast=3, thr_det=2, mask=0) : \
                  c.GradFun3(lsb=true, lsb_in=true, thr=0.55, radius=16, smode=2, elast=2.5, thr_det=2, mask=0)
grainy_debanded = addgrain ? debanded.GrainFL(lsb=true, lsb_in=true, g1str=3, g2str=10, g3str=5, g1shrp=50, g2shrp=60, g3shrp=80, g1size=0.9, g2size=0.7, g3size=0.6, th1=16, th2=24, th3=128, th4=180, g1soft=15, g2soft=15, g3soft=15) : \
                  debanded
debanded=addgrain ? mt_lutxy(debanded, grainy_debanded, expr="x 2 * y 4 * + 6 /") : debanded
merged=dither_merge16_8(debanded,c,mask,luma=false)
return merged
}

(the light variant uses f3kdb instead.

the debanding is run in a separate process (via mp_pipeline) to make use of the full 4GB memory available to a 32-bit Avisynth process.

When called only once (or twice), those functions run just fine, but calling them many times on different video sections causes out of memory errors.
Lower values for SetMemoryMax cause this error:

avs [error]: SharedMemoryClient: Server got error: GetFrameBuffer: Returned a VFB with a 0 data pointer!
size=8601664, max=268435456, used=1693698752
I think we have run out of memory folks! occurred while reading frame 13

Higher values cause different errors related to frame buffer errors or the process just hangs.
I also tried using ApplyRange instead of Trims while building the masks outside of the debanding functions (in a different process) and passing them as clips to the DebandLight()/DebandStrong()... to no avail.
The error happens regardless whether i use mp_pipeline or not and occurs both in AviSynth 2.5.8 (ST/MT) and AviSynth 2.6 (ST/MT).
I tried both piping the script output directly to x264 (via avs4x264mod) and encoding to UtVideo with VirtualDub. Tested this on a machine with 32GB RAM and a machine with 8GB RAM.

Is there any sane way to rearrange the script to avoid clogging up avisynths frame cache without encoding 4+ debanded versions and doing masking/stitching afterwards?

[Didée]

Each call to a filter or function invokes a full incarnation of that filter or funktion. The way you do now, If you stitch one thousands trims, then you (would) have to have one thousand copies of the filter in memory.


Instead: Prepare pre-processed clips as needed, and stitch those together.

vid = last
light = vid.DebandLight("aamask_fine",false)
strong = vid.DebandStrong("aamask_fine",false,true)

vid   .Trim(   0,  23) +\
strong.Trim(  24, 100) +\
light .Trim( 101, 174) +\
strong.Trim( 175, 196) +\
light .Trim( 197, 948) +\
strong.Trim( 949, 984) +\
light .Trim( 985,1269) +\
strong.Trim(1270,1313) +\
light .Trim(1314,1629) +\
strong.Trim(1630,1671) +\
light .Trim(1672,1933) +\
strong.Trim(1934,2044) +\
..etc..

This way you can make one thousand trims, but have only two copies of the filter actually in memory.

[Gavino]

Originally Posted by Didée

Each call to a filter or function invokes a full incarnation of that filter or funktion. The way you do now, If you stitch one thousands trims, then you (would) have to have one thousand copies of the filter in memory.

Instead: Prepare pre-processed clips as needed, and stitch those together.

This way you can make one thousand trims, but have only two copies of the filter actually in memory.

Right strategy, but he needs more than two because the filters are not called with the same arguments each time. You need a separate instance for each combination of (function + arguments).

[mastrboy]

Replaceframessimple might help: http://avisynth.org/stickboy/

[StainlessS]

As might ClipClop().
http://forum.doom9.org/showthread.php?t=162266

EDIT: ClipClop deals with multiple replacement clips (up to 255),
which would seem to be a requirement in this case.

[cretindesalpes]

With the same kind of strategy you could go even further and factorize almost all the code from the Deband* functions. You'll probably end up with everything at the top level or in a single bigger function. Filters in the script will be instantiated only once, not more. Just switch the signal path in your flowgraph with a ClipClop or ReplaceFramesSimple at each switching point. The latter is much easier to use than Trim because you can merge multiple sets of intervals just by concatenating the mapping strings.

[StainlessS]

Originally Posted by cretindesalpes

The latter is much easier to use than Trim because you can merge multiple sets of intervals just by concatenating the mapping strings.

Hi Cretindesalpes, would it be possible to clarify what you meant there, I've spent more than a few few minutes trying to figure it out. Are you saying that both ClipClop and ReplaceFramesSimple are easier than trim or that ReplaceFramesSimple is
easier than ClipClop, You have a unique (almost mystical) way of describing things.

[cretindesalpes]

Originally Posted by StainlessS

Hi Cretindesalpes, would it be possible to clarify what you meant there, I've spent more than a few few minutes trying to figure it out. Are you saying that both ClipClop and ReplaceFramesSimple are easier than trim or that ReplaceFramesSimple is
easier than ClipClop, You have a unique (almost mystical) way of describing things.

Sorry, English is not my native language. I meant that ReplaceFramesSimple is easier than Trim for this kind of job, because you can easily do things like :

r1 = "[123 456] [999 6666]"
r2 = "[789 888] [5000 10000]"

SourceFilter ()
tmp = Filter1 ()
ReplaceFramesSimple (tmp, mappings=r1)
tmp = Filter2 ()
ReplaceFramesSimple (tmp, mappings=r2)
Filter3 ()
tmp = Filter4 ()
ReplaceFramesSimple (tmp, mappings=r1+" "+r2)

And you don't have to worry whether ranges from r1 and r2 are overlapping or not, or their order within the string. With ClipClop it's a bit more complicated because you have to care about clip names/numbers too.

[StainlessS]

@cretindesalpes,
Thanks for your clarification, that is how I should have interpreted it.
Yes, I suppose that you are correct in that only supporting 1 source clip,
you do not need to care what the source clip name or number is, possibly
a case of less is more. Too late to change ClipClop(), to assume clip index
1 if only 1 source clip supplied, as the frame/range specification is so very
loose and flexible in what it will accept, would have required a much tighter
spec (perhaps the 'R' in 'Rx' should not have been made optional).

EDIT: @All,
In Clipclop, it also does not matter about range order in string command nor
overlapping ranges , (as in ReplaceFramesSimple), the later one takes effect.
Also, you can concatenate command/mapping strings using either the end of line
character 'Chr(10)' or the Clipclop soft end of line character ';', whereas in
previous post code sample the space character was used when concatenating
two mapping strings ie 'mappings=r1+" "+r2', in Clipclop you could use eg 'SCmd=r1+";"+r2'.
In both cases, the separation character is just to ensure that the parsers can recognise where
one range/frame ends and the next begins.