avisynth memory issues in large scripts

[pelle412]

Hi, I've run into a memory problem with avisynth that I am at loss to resolve. Searching the forums have not yielded anything conclusive that I can use. So where do I start? I've been working on a way to make my encodes of hybrid DVD content better and I came across a script written by sharfis_brain called convert60ito24p which when I tried it on a small scale yielded very promising results. So, I made a source code change to DVD2AVI to spit out some extra information about each frame (progressive or not...) which I pass through another homegrown tool to generate a avisynth script (which you can see below). For each sequence of interlaced frames I run it through sharfis_brain's script and each sequence of progressive frames I just do a plain decomb IVTC on. I tried this on a small clip and I am very happy with the results, definitely improved over decomb mode1 or mode3.

Now to my main question. Can I format the script below in a different way to not use so much memory? Even when I bound internal frame buffers to 128MB, the VirtualDub process (which drives avisynth) is reported to use 800 - 900 MB of ram. After about 20 - 30 minutes of chewing on this script it crashes with an avisynth access violation.

Any advice?



SetMemoryMax(128)

SetWorkingDir("C:\PROGRA~1\GORDIA~1\")
LoadPlugin("mpeg2dec3.dll")
LoadPlugin("undot.dll")
#LoadPlugin("Unfilter.dll")
LoadPlugin("Convolution3D.dll")
LoadPlugin("decomb.dll")
#LoadPlugin("TomsMoComp.dll")
LoadPlugin("dgbob.dll")
#LoadPlugin("avisynth_c.dll")
#LoadCPlugin("SmartDecimate.dll")
#Import("E:\convert60ito24p.avs")

#function FOrder(clip c) {
# fo= GetParity(c)? 1:0
# return fo
#}

# sharfis_brain's filter, moded to be YV12 transparent for mode=3
function convert60ito24p (clip video, int mode, int offset)
{
work = assumefieldbased(video).ConvertToYUY2()

out =(mode==3) ? interleave(
\selectevery(
\layer(trim(work, 1, 0),
\layer(work, trim(work, 2, 0), "fast"),
\"fast"), 5, 0 + offset),
\selectevery(
\layer(
\layer(work, trim(work, 3, 0),"fast"),
\layer(trim(work, 2, 0), trim(work, 1, 0),"fast"),
\level = 170), 5, 2 + offset)).ConvertToYV12() :

\ (mode==2) ? interleave(
\selectevery(
\layer(trim(work, 1, 0),
\layer(work, trim(work, 2, 0), "fast"),
\"fast"), 5, 0 + offset),
\selectevery(
\layer(work, trim(work, 1, 0), "fast"), 5, 3 + offset)) :

\ (mode==1) ? interleave(
\selectevery(trim(work, 1, 0), 5, 0 + offset),
\selectevery(layer(work, trim(work, 1, 0), "fast"), 5, 3 + offset)) :

\ (mode==0) ? selectevery(work, 5, 1 + offset, 4 + offset) : work

assumeframebased(out)
}

mpeg2source("E:\Movie\STNGD6.d2v")

# IVTC + deinterlace

aaa = trim(1,804).DGBob(order=1,mode=1).convert60ito24p(3,0)
aab = trim(805,1619).Telecide(order=1, guide=1).Decimate()
aac = trim(1620,1622).DGBob(order=1,mode=1).convert60ito24p(3,0)
aad = trim(1623,2014).Telecide(order=1, guide=1).Decimate()
aae = trim(2015,2016).DGBob(order=1,mode=1).convert60ito24p(3,0)
aaf = trim(2017,3130).Telecide(order=1, guide=1).Decimate()
aag = trim(3131,3132).DGBob(order=1,mode=1).convert60ito24p(3,0)
aah = trim(3133,4460).Telecide(order=1, guide=1).Decimate()
aai = trim(4461,4690).DGBob(order=1,mode=1).convert60ito24p(3,0)
aaj = trim(4691,4801).Telecide(order=1, guide=1).Decimate()
aak = trim(4802,4932).DGBob(order=1,mode=1).convert60ito24p(3,0)
aal = trim(4933,5103).Telecide(order=1, guide=1).Decimate()
aam = trim(5104,5203).DGBob(order=1,mode=1).convert60ito24p(3,0)
aan = trim(5204,5347).Telecide(order=1, guide=1).Decimate()
aao = trim(5348,5497).DGBob(order=1,mode=1).convert60ito24p(3,0)
aap = trim(5498,5585).Telecide(order=1, guide=1).Decimate()
aaq = trim(5586,5735).DGBob(order=1,mode=1).convert60ito24p(3,0)
aar = trim(5736,5941).Telecide(order=1, guide=1).Decimate()
aas = trim(5942,6040).DGBob(order=1,mode=1).convert60ito24p(3,0)
aat = trim(6041,6183).Telecide(order=1, guide=1).Decimate()
aau = trim(6184,6269).DGBob(order=1,mode=1).convert60ito24p(3,0)
aav = trim(6270,7368).Telecide(order=1, guide=1).Decimate()
****** <much much more of the same> ******
ajy = trim(77526,77655).DGBob(order=1,mode=1).convert60ito24p(3,0)
ajz = trim(77656,78386).Telecide(order=1, guide=1).Decimate()
aka = trim(78387,78538).DGBob(order=1,mode=1).convert60ito24p(3,0)
akb = trim(78539,79645).Telecide(order=1, guide=1).Decimate()
akc = trim(79646,81907).DGBob(order=1,mode=1).convert60ito24p(3,0)
aaa+aab+aac+aad+aae+aaf+aag+aah+aai+aaj+aak+aal+aam+aan+aao+aap+aaq+aar+aas+aat+aau+aav+aaw+aax+aay+aaz+aba+abb+abc+abd+abe+abf+abg+abh+abi+abj+abk+abl+abm+abn+abo+abp+abq+abr+abs+abt+abu+abv+abw+abx+aby+abz+aca+acb+acc+acd+ace+acf+acg+ach+aci+acj+ack+acl+acm+acn+aco+acp+acq+acr+acs+act+acu+acv+acw+acx+acy+acz+ada+adb+adc+add+ade+adf+adg+adh+adi+adj+adk+adl+adm+adn+ado+adp+adq+adr+ads+adt+adu+adv+adw+adx+ady+adz+aea+aeb+aec+aed+aee+aef+aeg+aeh+aei+aej+aek+ael+aem+aen+aeo+aep+aeq+aer+aes+aet+aeu+aev+aew+aex+aey+aez+afa+afb+afc+afd+afe+aff+afg+afh+afi+afj+afk+afl+afm+afn+afo+afp+afq+afr+afs+aft+afu+afv+afw+afx+afy+afz+aga+agb+agc+agd+age+agf+agg+agh+agi+agj+agk+agl+agm+agn+ago+agp+agq+agr+ags+agt+agu+agv+agw+agx+agy+agz+aha+ahb+ahc+ahd+ahe+ahf+ahg+ahh+ahi+ahj+ahk+ahl+ahm+ahn+aho+ahp+ahq+ahr+ahs+aht+ahu+ahv+ahw+ahx+ahy+ahz+aia+aib+aic+aid+aie+aif+aig+aih+aii+aij+aik+ail+aim+ain+aio+aip+aiq+air+ais+ait+aiu+aiv+aiw+aix+aiy+aiz+aja+ajb+ajc+ajd+aje+ajf+ajg+ajh+aji+ajj+ajk+ajl+ajm+ajn+ajo+ajp+ajq+ajr+ajs+ajt+aju+ajv+ajw+ajx+ajy+ajz+aka+akb+akc

# Cropping
crop(18,0,692,476)

# Deinterlacing
# TomsMoComp(1,5,1)

# Resizing
LumaFilter()
Undot()
Convolution3d("moviehq")
LanczosResize(640,480)

# Compressibility Check
SelectRangeEvery(140,14)

[Leak]

Originally posted by pelle412
Hi, I've run into a memory problem with avisynth that I am at loss to resolve.

Oh yeah, that script of yours is definitely going to eat RAM for breakfast... :)

What you've done is create dozens of instances of Decomb, DGBob and whatever convert60ito24p uses, when one of each would be sufficient:


# IVTC + deinterlace

aaa = trim(1,804).DGBob(order=1,mode=1).convert60ito24p(3,0)
aab = trim(805,1619).Telecide(order=1, guide=1).Decimate()
aac = trim(1620,1622).DGBob(order=1,mode=1).convert60ito24p(3,0)
aad = trim(1623,2014).Telecide(order=1, guide=1).Decimate()
...


What you really want to do is this:


# IVTC + deinterlace

convert=DGBob(order=1,mode=1).convert60ito24p(3,0)
decimate=Telecide(order=1, guide=1).Decimate()

aaa = convert.trim(1,804)
aab = decimate.trim(805,1619)
aac = convert.trim(1620,1622)
aad = decimate.trim(1623,2014)
...


That way, you'll only create one instance of each (which would in principle process the whole video on their own) and cut out the parts of each you're really interested in.

You could probably also drop the "aaa", "aab", etc. part and just output something like this:


# IVTC + deinterlace

convert=DGBob(order=1,mode=1).convert60ito24p(3,0)
decimate=Telecide(order=1, guide=1).Decimate()

convert.trim(1,804) + \
decimate.trim(805,1619) + \
convert.trim(1620,1622) + \
decimate.trim(1623,2014) + \
...


(just drop the "+ \" on the last line...)

Hope this helps. :)

np: System - Micro (System)

[pelle412]

Wow, that makes alot of sense. I didn't realize that I was creating so many instances, wasn't thinking along those lines. I'm running a test with your suggestion right now and memory usage is way down and much faster. Looks like your suggestion was what I needed :-)

Thanks!

[rebkell]

Sorry for digging up an old thread, but it seems perfect, except it didn't work for me:

I wanted to define 4 different variable that should cover the various sequences for Telecined captures:

PPPII=SeparateFields().SelectEvery(10, ....).Weave()
PIIPP=SeparateFields().SelectEvery(10, ....).Weave()
IIPPP=SeparateFields().SelectEvery(10, ....).Weave()
PPIIP=SeparateFields().SelectEvery(10, ....).Weave()


then use it like in the above example:
PPPII.trim(0,8999) + \
PPIII.trim(12450,30904) + \
PPPII.trim(35000, 39999)

when I tried that it worked ok, but then it just kept on processing the frames, instead of jumping from frame 8999 to 12450 it just proceeded to process frames 9000 through 12449, etc...

I'm a little confused on how things get processed in avisynth scripts??

[stickboy]

Post your entire script. The clip returned at the end probably isn't the trimmed + spliced version.

[rebkell]

Post your entire script. The clip returned at the end probably isn't the trimmed + spliced version.

Ok, I don't remember the exact numbers, but it was basically this:
AVCSource("MyFile.dga").AssumeTFF()
PPPII=SeparateFields().SelectEvery(10, 0,1, 2,3, 4, 5, 7,8).Weave()
PIIPP=SeparateFields().SelectEvery(10, 0,1, 3,4,6,7,8,9).Weave()
IIPPP=SeparateFields().SelectEvery(10, 1,2, 4,5,6,7,8,9).Weave()
PPIIP=SeparateFields().SelectEvery(10, 0,1,2,3,5,6,8,9).Weave()

PPPII.trim(0,8999) + \
PPIII.trim(12450,30904) + \
PPPII.trim(35000, 39999)

LanczosResize(1280,720)

[Gavino]

Do you want frames corresponding to 0 to 8999, 12450 to 30904, etc, of the original clip?

If so, you will have to adjust the frame numbers applied to PPPII, etc, to allow for them having 20% less frames, ie

PPPII.trim(0,7199) + \
PPIII.trim(9960,24723) + \
PPPII.trim(28000, 31999)

[rebkell]

Do you want frames corresponding to 0 to 8999, 12450 to 30904, etc, of the original clip?

If so, you will have to adjust the frame numbers applied to PPPII, etc, to allow for them having 20% less frames, ie

PPPII.trim(0,7199) + \
PPIII.trim(9960,24723) + \
PPPII.trim(28000, 31999)

Is that because the trim function comes after the other function? I tried the other way around trim(0,8999).PPPII, but it gave syntax errors.

[Gavino]

What happens is that the first part of your script creates four new clips (PPPII, etc) by selecting 8 frames out of every 10 original ones in four different ways. These clips therefore each have 20% less frames than the original clip.

So the first 9000 frames (0 to 8999) turn into 7200 frames (0 to 7199) in PPPII, etc. Then to select those frames for your output, you have to trim from 0 to 7199 instead of 0 to 8999 and so on.