Hybrid DVD to VFR - Complex Script Optimization Help / Ram Obliteration

[jack44556677]

I have a dvd source that is a hybrid of many different kinds of pulldowns / source framerates that I want to convert into a progressive x265/x264 of visibly identical quality. That is how the nightmare begins.

I have decided that the best possible approach for this source is to end up with a vfr mkv as a target. No dups, no interlacing, no nonsense. I have also decided that I want this to be the most perfect archival conversion possible, so I don't want any automatic detection of the various sections and the errors such automatic approaches necessarily introduce (TIVTC)

I have been following, generally, hello_hello's approach outlined here : https://forum.doom9.org/showthread.p...27#post1893927 & https://forum.doom9.org/showthread.p...53#post1824253
As well as Overdrive80's approach here :
https://forum.doom9.org/showthread.p...03#post1893903

Although hideously tedious, this approach was working beautifully until I hit a brick wall with ram usage (segmentation faults and c++ exceptions) - even though I have 16GB to play with. I've been using avisynth+ 64 bit v3.7.3 (r4003, 3.7, x86_64)

There seems to be some serious memory leakage problems and or improper memory handling / releasing of caches going on behind the scenes which I am desperately hoping some guru around here will be able to provide solutions, approaches, and/or workarounds for.

At first I thought it was simply too much overhead with the hundreds of qtgmc clips being spawned and spliced, but I have come to suspect that it is in fact the cache of the trim function that is the main culprit. SetCacheMode and SetMemoryMax have negligible impact.

I am using arrays for the input ranges and source framerate types used by the IVTC subroutines (executed using the array loop function ArrayOpFunc), which certainly doesn't help any - and each array is broken up into 80 items because more than that will cause these segmentation faults in and of themselves. I have 12 of them currently.

At the end of the script, as in the examples provided I am roughly using as a template, I do an unaligned splice on all of them and then return that as the final clip.

Does anyone know why this is absolutely demolishing my ram? Why can't the avs script generate the frames, pass them off to the encoder, and then (properly) release the memory used for those previous frames?!

I found this https://github.com/AviSynth/AviSynth...ent-1084862174 detailing a patch for a NoCache function to use to stop trim from caching which is a last ditch longshot. Do you think it is worth compiling from source, or am I making some sort of rookie mistake that can be solved by trying another approach?

I'm trying to avoid processing the source in segments and then splicing it back together in reencoded chunks due to the difficulty with handling the vfr timestamps generation (among other things), but I will do it as a last resort if there is simply no other way.

Any and all help is greatly appreciated!

***UPDATE***

It does not look like the trim cache is the problem. I decided to do some tests to see if I could further pin down what was really responsible.

I've not finished converting all the routines yet - but what I did was convert the script and subfunctions to print out a massive avisynth splice statement (with all the clip segments) rather than actually execute/process them - then pasted that mega-splice (each individual spliced clip including one, and sometimes 2, trim statements) into a clean script. Suddenly the same action that was using around 4gb of ram to process, used only 88mb.

This leads me to conclude that the loop function ArrayOpFunc and arrays (avslib), the multiple user subfunctions, and/or the way they interact with avisynth's memory caching (assuming they do) are really responsible.