Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
|
|
Thread Tools | Search this Thread | Display Modes |
4th January 2024, 06:40 | #1 | Link |
Registered User
Join Date: Jun 2015
Posts: 3
|
Hybrid DVD to VFR - Complex Script Optimization Help / Ram Obliteration
I have a dvd source that is a hybrid of many different kinds of pulldowns / source framerates that I want to convert into a progressive x265/x264 of visibly identical quality. That is how the nightmare begins.
I have decided that the best possible approach for this source is to end up with a vfr mkv as a target. No dups, no interlacing, no nonsense. I have also decided that I want this to be the most perfect archival conversion possible, so I don't want any automatic detection of the various sections and the errors such automatic approaches necessarily introduce (TIVTC) I have been following, generally, hello_hello's approach outlined here : https://forum.doom9.org/showthread.p...27#post1893927 & https://forum.doom9.org/showthread.p...53#post1824253 As well as Overdrive80's approach here : https://forum.doom9.org/showthread.p...03#post1893903 Although hideously tedious, this approach was working beautifully until I hit a brick wall with ram usage (segmentation faults and c++ exceptions) - even though I have 16GB to play with. I've been using avisynth+ 64 bit v3.7.3 (r4003, 3.7, x86_64) There seems to be some serious memory leakage problems and or improper memory handling / releasing of caches going on behind the scenes which I am desperately hoping some guru around here will be able to provide solutions, approaches, and/or workarounds for. At first I thought it was simply too much overhead with the hundreds of qtgmc clips being spawned and spliced, but I have come to suspect that it is in fact the cache of the trim function that is the main culprit. SetCacheMode and SetMemoryMax have negligible impact. I am using arrays for the input ranges and source framerate types used by the IVTC subroutines (executed using the array loop function ArrayOpFunc), which certainly doesn't help any - and each array is broken up into 80 items because more than that will cause these segmentation faults in and of themselves. I have 12 of them currently. At the end of the script, as in the examples provided I am roughly using as a template, I do an unaligned splice on all of them and then return that as the final clip. Does anyone know why this is absolutely demolishing my ram? Why can't the avs script generate the frames, pass them off to the encoder, and then (properly) release the memory used for those previous frames?! I found this https://github.com/AviSynth/AviSynth...ent-1084862174 detailing a patch for a NoCache function to use to stop trim from caching which is a last ditch longshot. Do you think it is worth compiling from source, or am I making some sort of rookie mistake that can be solved by trying another approach? I'm trying to avoid processing the source in segments and then splicing it back together in reencoded chunks due to the difficulty with handling the vfr timestamps generation (among other things), but I will do it as a last resort if there is simply no other way. Any and all help is greatly appreciated! ***UPDATE*** It does not look like the trim cache is the problem. I decided to do some tests to see if I could further pin down what was really responsible. I've not finished converting all the routines yet - but what I did was convert the script and subfunctions to print out a massive avisynth splice statement (with all the clip segments) rather than actually execute/process them - then pasted that mega-splice (each individual spliced clip including one, and sometimes 2, trim statements) into a clean script. Suddenly the same action that was using around 4gb of ram to process, used only 88mb. This leads me to conclude that the loop function ArrayOpFunc and arrays (avslib), the multiple user subfunctions, and/or the way they interact with avisynth's memory caching (assuming they do) are really responsible. Last edited by jack44556677; 4th January 2024 at 19:39. |
18th January 2024, 18:18 | #2 | Link |
Registered User
Join Date: Jun 2015
Posts: 3
|
What I learned / my workaround solution
Posting my findings for posterity in the hopes they will be useful to others.
General problems I was fighting:
General solutions / workarounds (respective to above list) :
The fixes above made the final avs script require around 500 mb of ram, though the avslib one that generated it still required 8+gb and was miserably slow too. Last edited by jack44556677; 28th January 2024 at 12:17. |
19th January 2024, 09:01 | #3 | Link | |
Registered User
Join Date: Dec 2005
Location: Sweden
Posts: 716
|
Quote:
Could you provide an example of a simple script such as: Code:
Loadplugin("neo_fft3d.dll") Import("qtgmc.avsi") FFvideosource("c:/videofile.mp4") ConvertToYUV444() QTGMC(Preset="Very Slow") Temporalsoften(1,255,255,255,2) Neo_fft3d(sigma=3) Convertbits(dither=1,bits=10) Last edited by anton_foy; 19th January 2024 at 09:03. |
|
20th January 2024, 21:26 | #4 | Link |
Registered User
Join Date: Jun 2015
Posts: 3
|
I think for simple avs scripts like the one you pasted, the things I mentioned likely don't apply.
The avslib avs script that generates the final avs is over 5000 lines long and is anything but simple. The most distilled I can make it is that if you are using avslib and ArrayOpFunc to generate many segments and splice them together rather than do this : Code:
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\DGDecode.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\DePan.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\DePanEstimate.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\masktools2.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\mvtools2.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\nnedi3.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\RgTools.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\fft3dfilter.dll") LoadPackage("avslib", "array") LoadPackage("avslib", "string") # val is used so that either ints or floats can be used Function myfilter(string framerangedelimited, clip source) { ArrayDelimiterSet(":") framenumRangeStart = ArrayGet(framerangedelimited, 0) framenumRangeStop = ArrayGet(framerangedelimited, 1) rangetype_val = ArrayGet(framerangedelimited, 3) ArrayDelimiterReset() (rangetype_val == 0) ? Eval(""" #0 - QTGMC range, doubles input framerate d = source.Trim(framenumRangeStart,framenumRangeStop).QTGMC(Preset="Very Slow").AssumeFPS(24000,1001) """) : \ (rangetype_val == 1) ? Eval(""" #1 IVTC 2:3 d = source.Trim(framenumRangeStart,framenumRangeStop).SeperateFields().DoubleWeave().Pulldown(0,3).AssumeFPS(24000,1001) """) : "" return d } #Then assign to this variable your clip; note that the "global" is necessary. global testclip = mpeg2source("1.d2v").ShowFrameNumber(true).Showtime() #create an array of values (replace start, step, end with the actual values you want), #pmvalues = ArrayRange(start, end, step) pmvalues = """\ "0:705:24:1",\ "706:914:60:0",\ "915:1021:24:1",\ "1022:1023:24:1",\ "1024:1222:60:0"\ """ #apply your filter(s) wrapper function to testclip and store the results into an array, pmclips = pmvalues.ArrayOpFunc("myfilter", "testclip") #sum all array elements (with simple UnallignedSplice) to get a single clip. return pmclips.ArraySum() Code:
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\DGDecode.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\DePan.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\DePanEstimate.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\masktools2.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\mvtools2.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\nnedi3.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\RgTools.dll") LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC\fft3dfilter.dll") LoadPackage("avslib", "array") LoadPackage("avslib", "string") #Create global variable for the splice statement to be used to create a seperate avs script to generate the final video. global commandBuilder=TripleQuotes() + "return " # val is used so that either ints or floats can be used Function myfilter(string framerangedelimited) { #Then assign to this variable your clip; declare the local source variable here and avoid global variables for memory overhead reasons. source = mpeg2source("1.d2v").ShowFrameNumber(true).Showtime() ArrayDelimiterSet(":") framenumRangeStart = ArrayGet(framerangedelimited, 0) framenumRangeStop = ArrayGet(framerangedelimited, 1) rangetype_val = ArrayGet(framerangedelimited, 3) ArrayDelimiterReset() (rangetype_val == 0) ? Eval(""" #0 - QTGMC range, doubles input framerate clipSegmentString = "c.Trim(" + String((framenumRangeStart*2)) + "," + String((framenumRangeStop*2)+1) + ").AssumeFPS(24000,1001)" global commandBuilder = commandBuilder + clipSegmentString + " + " """) : \ (rangetype_val == 1) ? Eval(""" #1 IVTC 2:3 clipSegmentString = "sourcevideo.Trim(" + String(framenumRangeStart) + "," + String(framenumRangeStop) + ").SeparateFields().DoubleWeave().Pulldown(0,3).AssumeFPS(24000,1001)" global commandBuilder = commandBuilder + clipSegmentString + " + " """) : "" return d } # This function is needed to add triplequotes around the commandBuilder string. Without them, you can't write the final string to the text file. function TripleQuotes() { result = MidStr(""" " """,2,1) + MidStr(""" " """,2,1) + MidStr(""" " """,2,1) return result } #create an array of values (replace start, step, end with the actual values you want), #pmvalues = ArrayRange(start, end, step) pmvalues = """\ "0:705:24:1",\ "706:914:60:0",\ "915:1021:24:1",\ "1022:1023:24:1",\ "1024:1222:60:0"\ """ #iterate through the delimited array ranges and populate the commandBuilder string pmclips = pmvalues.ArrayOpFunc("myfilter") #set output filename filename = "commandBuilder.txt" #remove the trailing " + ", add triple quotes, and save the text file with the splice statement for use in another avs script which doesn't have avslib in it. That other avs script must have "c" defined as your source clip with qtgmc applied and "sourcevideo" as your avs video source. WriteFileEnd(BlankClip(length=1), filename, LeftStr(commandBuilder, (StrLen(commandBuilder)-3))+TripleQuotes(), append=false) #return a blank clip (optional) return BlankClip(length=1) Last edited by jack44556677; 3rd February 2024 at 18:32. |
Tags |
avisynth, avslib array, memory management, segment linking, vfr |
Thread Tools | Search this Thread |
Display Modes | |
|
|