Running out of memory when processing parts of a script differently

[zero9999]

i'm running a script that debands parts of a video (1080p, 16-bit stacked) differently, which includes using different masks for protecting details.

Trim(0,23) +\
Trim(24,100).DebandStrong("aamask_fine",false,true) +\
Trim(101,174).DebandLight("aamask_fine",false) +\
Trim(175,196).DebandStrong("aamask_fine",false,true) +\
Trim(197,948).DebandLight("aamask_fine",false) +\
Trim(949,984).DebandStrong("aamask_dull",false,true) +\
Trim(985,1269).DebandLight("aamask_fine",false) +\
Trim(1270,1313).DebandStrong("aamask_fine",true,true) +\
Trim(1314,1629).DebandLight("aamask_fine",false) +\
Trim(1630,1671).DebandStrong("aamask_veryfine",true,true) +\
Trim(1672,1933).DebandLight("aamask_fine",false) +\
Trim(1934,2044).DebandStrong("aamask_fine",true,true) +\

the functions look like this:

function DebandStrong(clip c, string "mask_name", bool "less", bool "addgrain", string "mask_file")
{
mask_name=Default(mask_name,"aamask_fine")
addgrain=Default(addgrain, false)
mask_file=Default(mask_file, "none")
less=Default(less, false)

mask=BlankClip(c,width=c.Width,height=c.Height/2).KillAudio().Subtitle(mask_name)   
mask = (mask_name=="aamask_dull") ? c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.1,c=0.35).tcanny(sigma=1.2, mode=1, plane=7).RemoveGrain(27).mt_lut("x 13 < 0 255 ?" , U=1, V=1).mt_binarize() : mask
mask = (mask_name=="aamask_fine") ? c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.1,c=0.35).tcanny(sigma=1.2, mode=1, plane=7).RemoveGrain(27).mt_lut("x 10 < 0 255 ?" , U=1, V=1)\
.mt_logic(c.DitherPost(mode=6).MinBlur().Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.45).FastlinedarkenMOD(strength=40,thinning=0, luma_cap=255, threshold=0).MinBlur(2).mt_edge(mode="hprewitt"), mode="or").mt_binarize() : mask
mask = (mask_name=="aamask_veryfine") ? c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.1,c=0.35).tcanny(sigma=1.2, mode=1, plane=7).RemoveGrain(27).mt_lut("x 7 < 0 255 ?" , U=1, V=1)\
.mt_logic(c.DitherPost(mode=6).MinBlur().Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.45).FastlinedarkenMOD(strength=40,thinning=0, luma_cap=255, threshold=0).MinBlur(2).mt_edge(mode="hprewitt"), mode="or").mt_binarize() : mask
mask = (mask_name=="aamask_ultrafine") ?  c.DitherPost(mode=6).Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.75).tcanny(sigma=1.1, mode=1, plane=7).mt_lut("x 7 < 0 255 ?" , U=1, V=1)\.mt_logic(c.DitherPost(mode=6).MinBlur().Dither_Luma_Rebuild(lsb=false,S0=1.4,c=0.45).FastlinedarkenMOD(strength=40,thinning=0, luma_cap=255, threshold=0).MinBlur(2).mt_edge(mode="hprewitt"), mode="or").mt_binarize() : mask
mask = (mask_name=="special") ? ImageSource(mask_file,0,c.FrameCount-1,c.FrameRate).ConvertToYV12(matrix="PC.709").mt_binarize() : mask


debanded= !less ? c.GradFun3(lsb=true, lsb_in=true, thr=0.75, radius=16, smode=2, elast=3, thr_det=2, mask=0) : \
                  c.GradFun3(lsb=true, lsb_in=true, thr=0.55, radius=16, smode=2, elast=2.5, thr_det=2, mask=0)
grainy_debanded = addgrain ? debanded.GrainFL(lsb=true, lsb_in=true, g1str=3, g2str=10, g3str=5, g1shrp=50, g2shrp=60, g3shrp=80, g1size=0.9, g2size=0.7, g3size=0.6, th1=16, th2=24, th3=128, th4=180, g1soft=15, g2soft=15, g3soft=15) : \
                  debanded
debanded=addgrain ? mt_lutxy(debanded, grainy_debanded, expr="x 2 * y 4 * + 6 /") : debanded
merged=dither_merge16_8(debanded,c,mask,luma=false)
return merged
}

(the light variant uses f3kdb instead.

the debanding is run in a separate process (via mp_pipeline) to make use of the full 4GB memory available to a 32-bit Avisynth process.

When called only once (or twice), those functions run just fine, but calling them many times on different video sections causes out of memory errors.
Lower values for SetMemoryMax cause this error:

avs [error]: SharedMemoryClient: Server got error: GetFrameBuffer: Returned a VFB with a 0 data pointer!
size=8601664, max=268435456, used=1693698752
I think we have run out of memory folks! occurred while reading frame 13

Higher values cause different errors related to frame buffer errors or the process just hangs.
I also tried using ApplyRange instead of Trims while building the masks outside of the debanding functions (in a different process) and passing them as clips to the DebandLight()/DebandStrong()... to no avail.
The error happens regardless whether i use mp_pipeline or not and occurs both in AviSynth 2.5.8 (ST/MT) and AviSynth 2.6 (ST/MT).
I tried both piping the script output directly to x264 (via avs4x264mod) and encoding to UtVideo with VirtualDub. Tested this on a machine with 32GB RAM and a machine with 8GB RAM.

Is there any sane way to rearrange the script to avoid clogging up avisynths frame cache without encoding 4+ debanded versions and doing masking/stitching afterwards?