Out of memory (in Virtualdub)

[TCmullet]

I'm not sure whether to post this problem here or in the Virtualdub area. My gut says to post it here.

I've run an Avisynth script that takes about 2.5 hours to complete on my system. Just as it is about to finish encoding, Virtualdub gives this error message: "Out of memory (unable to allocate 3923088 bytes)". I've uploaded a screen showing the error window, Vdub's status window (showing that the job is at the very end) AND the Windows Task Manager showing various things.

I am baffled as this PC has 24GB of ram, and at least 80GB of free space on each of the several drives present. In the Avisynth script, I use avisynth.dll 2.5.8.6 MT from SVP, and SetMemoryMax is set to 1024. Opening the nearly complete file in Virtualdub triggers the "eternal" "Rekeying video stream", which will take longer to happen than rerunning Virtualdub. (But Vdub has bombed so why do that?)

This is Xeon processor with Windows 7, with 4-core, 8-hyperthreads.

In the pic of the task manager, it shows the ram flatlined. I regret that I may not have seen it quickly enough after the error occurred to see if memory usage had spiked.

How could it work so well throughout, then bomb right at the end?? (More importantly, how can I prevent this?) I tried to hunt for an answer at the Vdub forum, but it recently was taken offline.

[StainlessS]

Any clues as to what might be in this mysterious avisynth script ?

[TCmullet]

Originally Posted by StainlessS

Any clues as to what might be in this mysterious avisynth script ?

I had thought that my screen print showing task mgr and the other things would be adequate. But as the jpg is taking awhile to "get approved", I'll post the script here (it may be a bit ugly.) If necessary, I'll later add function code from .avsi.

# Requires AviSynth 2.6MT (using 2.6.0.5MT)
# 9/11/2015 switched to avisynth.2.5.8.6.MT.svp.dll
SetMemoryMax(1024)
global svp_scheduler=true
global threads=4
global svp_cache_fwd=threads+10

SetMTMode(3,threads)


#function AdjSync(clip c, int fstart, float sec)
#function MuteRange(clip c, int fstart, int fend)
#function SoftenRange(clip c, int fstart, int fend, float pctAdj)

mainfile="2015-12-03.1900.wv.R3-Rnd1.NH-v-FL.sec+.rawX7.mp4"
video = LWLibavVideoSource(mainfile,cache=false)
audio = LWLibavAudioSource(mainfile)
AudioDub(video,audio)

AssumeFPS("ntsc_video")


# black out playback progress bar on bottom
Crop(0,0,0,-8) # crop from the bottom
AddBorders(0,0,0,8) # pad to the bottom

#Normalize(show=true)
Amplify(1.7956)  # whatever Amplify Factor was shown in the "Normalize(show=true)" above


# delete comm brks

# Do sound sync check and adjust right after each edit

#zapscorebar0(180,72)


delayaudio(-0.067)
loop(0,0,6)

blackover(14939,14954)
loop(0,14955,16738)
adjsync(14955,-0.067)
loop(0,39576,43178)

loop(0,54171,55971)
blackover(54171,54173)

loop(0,72366,75948)
loop(0,76707,80314)
blackover(76707,76708)

loop(0,103818,105510)
adjsync(103818,0.033)
#loop(0,,)
#loop(0,,)
#loop(0,,)

# tack on 2 sec. of black
#main=last
#MyBlank=BlankClip(main,60)
#main++MyBlank

# use when upscaling from 432p
#aWarpSharp(depth=20.0, blurlevel=2, thresh=0.5, cm=1, bm=2)

GameDropFixV4(myGPU=true,myDebug=false,myErrSize=150)

# program is 752p.  let YT downsize.

#SR(1280,720)


SetMTMode(2)

#InterFrame(Cores=4,Tuning="Film",NewNum=60000,NewDen=1001,GPU=true)

InterFrame(Cores=threads,Tuning="Film",GPU=true)

#
# send VirtualDub output to x264v-q20(superfast)

[MysteryX]

It doesn't matter how much RAM you use. 32-bit processes can only access 2GB; and it can be patched to use 3GB or 4GB, but no more. When you look in Task Manager, how much memory does it use? One way to reduce memory usage is to reduce the amount of threads.

[TCmullet]

Originally Posted by MysteryX

It doesn't matter how much RAM you use. 32-bit processes can only access 2GB; and it can be patched to use 3GB or 4GB, but no more. When you look in Task Manager, how much memory does it use? One way to reduce memory usage is to reduce the amount of threads.

My attached graphic has been approved. Please seek the answer there, esp. as I don't know how to answer you.

Regardless, how do we "patch" it to 3GB or 4GB?

As to threads, I want to use as many as possible, but I'll limit it to 4 even tho this PC has 8, as I want scripts to be compatible with my other PC which has only 4 (4 actual, no hyperthreads).

[TCmullet]

You can see from the Vdub status window, which was unchanging when I took the pic, that Time Elapsed EQUALED Total time (estimated), which meant it was essentially finished. This makes me suspect that the out-of-mem had to do with Vdub process of closing output files. I wish it had allocated this at the beginning rather than at the end, blowing up after all that time. But maybe that's a windows function, not Vdub's.

[raffriff42]

1. Undefined "GameDropFixV4", could you post it? Also "AdjSync" and "blackover."

2.. Has this script ever run successfully?
2.a. What has changed recently?
2.b. Can you run an older script that worked before?

3. Does the error occur if you select a short segment in the VirtualDub timeline for encoding? (10 seconds is a good length)
3.a. If so, you can proceed to make changes to the script and test them much more quickly.
3.b. If not, the problem may lie in one of your Avisynth timeline editing statements (splice, loop, trim, ...)

4. Are you using an External encoder like x264.exe? (External encoders have a tendency, if they fail, to fail at the very end like you describe. Not crash or run out of memory though, AFAIK)

5. Any VirtualDub filters being used?

[TCmullet]

Originally Posted by raffriff42

1. Undefined "GameDropFixV4", could you post it? Also "AdjSync" and "blackover."

# GameDropFixFuncs.avsi

function GameDropFixV4(clip c, bool "myGPU", bool "myDebug", int "myErrSize")
{
myGPU = default( myGPU, true )
myDebug = default( myDebug, false )
myErrSize = default( myErrSize, 12 )

last=c

super_params_mini = (myGPU==true) ? "{gpu:1}" : "{gpu:0}"

analyse_params_mini="{block:{w:8,h:8}}"
miniW = int(width/64)*32
miniW = (miniW<320) ? 320 : miniW
miniH = int(height/64)*32
miniH = (miniH<160) ? 160 : miniH
global mini=BicubicResize(miniW,miniH).TemporalSoften(1, 1, 0, scenechange=1, mode=2)
super_mini=mini.SVSuper(super_params_mini)
vectors_mini=SVAnalyse(super_mini, analyse_params_mini)
vectors_Forward=SVConvert(vectors_mini, false)
HorizontalSpeed_luma=mini.MMask(vectors_Forward, kind=3).convertToRGB32().PointResize(miniW/8, miniH/8).PointResize(miniW/4, miniH/4).ConvertToYV12().mt_lut(y=2, u=128, v=128)
VerticalSpeed_luma=mini.MMask(vectors_Forward, kind=4).convertToRGB32().PointResize(miniW/8, miniH/8).PointResize(miniW/4, miniH/4).ConvertToYV12().mt_lut(y=2, u=128, v=128)

size=16
global luma_lft=HorizontalSpeed_luma.crop(0,0,miniW/16,0).BicubicResize(size,size)
global luma_rgh=HorizontalSpeed_luma.crop(miniW/16+miniW/8,0,0,0).BicubicResize(size,size)
global luma_top=VerticalSpeed_luma.crop(0,0,0,miniH/16).BicubicResize(size,size)
global luma_btm=VerticalSpeed_luma.crop(0,miniH/16+miniH/8,0,0).BicubicResize(size,size)
global luma_drop=luma_lft.mt_lut(y=-1, u=128, v=128)

global move_idx=5
global y_idx=3
global min_move=0.1
global max_stop=0.5
global drop_clip=luma_drop.ScriptClip("
    AvgLuma_lft=128-luma_lft.AverageLuma
    AvgLuma_rgh=128-luma_rgh.AverageLuma
    AvgLuma_lft_prev=128-(luma_lft.trim(1,1)+luma_lft).AverageLuma
    AvgLuma_rgh_prev=128-(luma_rgh.trim(1,1)+luma_rgh).AverageLuma
    AvgLuma_lft_next=128-luma_lft.trim(1,0).AverageLuma
    AvgLuma_rgh_next=128-luma_rgh.trim(1,0).AverageLuma

    AvgLuma_top=128-luma_top.AverageLuma
    AvgLuma_btm=128-luma_btm.AverageLuma
    AvgLuma_top_prev=128-(luma_top.trim(1,1)+luma_top).AverageLuma
    AvgLuma_btm_prev=128-(luma_btm.trim(1,1)+luma_btm).AverageLuma
    AvgLuma_top_next=128-luma_top.trim(1,0).AverageLuma
    AvgLuma_btm_next=128-luma_btm.trim(1,0).AverageLuma

    Max_lft=max(abs(AvgLuma_lft_prev),abs(AvgLuma_lft),abs(AvgLuma_lft_next))
    Max_rgh=max(abs(AvgLuma_rgh_prev),abs(AvgLuma_rgh),abs(AvgLuma_rgh_next))
    Max_top=max(abs(AvgLuma_top_prev),abs(AvgLuma_top),abs(AvgLuma_top_next))
    Max_btm=max(abs(AvgLuma_btm_prev),abs(AvgLuma_btm),abs(AvgLuma_btm_next))
    Max_all=max(Max_lft,Max_rgh,Max_top,Max_btm)
    
    MaxLuma=max(abs(AvgLuma_lft),abs(AvgLuma_rgh),abs(AvgLuma_top),abs(AvgLuma_btm))
    MaxLuma_prev=max(abs(AvgLuma_lft_prev),abs(AvgLuma_rgh_prev),abs(AvgLuma_top_prev),abs(AvgLuma_btm_prev))
    MaxLuma_next=max(abs(AvgLuma_lft_next),abs(AvgLuma_rgh_next),abs(AvgLuma_top_next),abs(AvgLuma_btm_next))
    dif=mini.YDifferenceFromPrevious
    dif_next=mini.trim(1,0).YDifferenceFromPrevious
    dif_prev=(mini.trim(1,1)+mini).YDifferenceFromPrevious
    
    GoodContrast = (max(mini.YPlaneMinMaxDifference,mini.trim(1,0).YPlaneMinMaxDifference)>20) ? 1 : 0

    drop = (MaxLuma<0.1 && MaxLuma*2<max(MaxLuma_prev,MaxLuma_next) && dif*y_idx<max(dif_prev,dif_next)) ? 1 : 0

    drop_lft = (AvgLuma_lft_prev*AvgLuma_lft_next>0 && abs(AvgLuma_lft*move_idx)<Max_lft && abs(AvgLuma_lft)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 2 : 0
    drop_rgh = (AvgLuma_rgh_prev*AvgLuma_rgh_next>0 && abs(AvgLuma_rgh*move_idx)<Max_rgh && abs(AvgLuma_rgh)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 3 : 0
    drop_top = (AvgLuma_top_prev*AvgLuma_top_next>0 && abs(AvgLuma_top*move_idx)<Max_top && abs(AvgLuma_top)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 4 : 0
    drop_btm = (AvgLuma_btm_prev*AvgLuma_btm_next>0 && abs(AvgLuma_btm*move_idx)<Max_btm && abs(AvgLuma_btm)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 5 : 0
   
    drop = (drop==0 && MaxLuma<Max_stop && Max_lft==Max_all) ? drop_lft : drop
    drop = (drop==0 && MaxLuma<Max_stop && Max_rgh==Max_all) ? drop_rgh : drop
    drop = (drop==0 && MaxLuma<Max_stop && Max_top==Max_all) ? drop_top : drop
    drop = (drop==0 && MaxLuma<Max_stop && Max_btm==Max_all) ? drop_btm : drop
    drop_dif = (dif<0.4 && MaxLuma<min_move) ? 6 : 0
    drop = (drop==0 && drop_dif>0) ? drop_dif : drop
    drop = (drop>0 && Max_all<0.1) ? 0 : drop
    drop = (drop>0 && GoodContrast==0) ? 0 : drop

    luma=mini.AverageLuma
    luma_next=mini.trim(1,0).AverageLuma
    max_luma=max(luma,luma_next)

    drop = (drop>0 && max_luma<16.5) ? 0 : drop
    (drop>0) ? luma_drop : \
     (Max_lft==Max_all) ? luma_lft : \
      (Max_rgh==Max_all) ? luma_rgh : \
       (Max_top==Max_all) ? luma_top : luma_btm
")

global fix_r50=luma_lft.mt_lut(y=-1, u=128, v=128)
global fix_lr33=luma_lft.mt_lut(y=-2, u=128, v=128)
global fix_l50=luma_lft.mt_lut(y=-3, u=128, v=128)
global fix_r133=luma_lft.mt_lut(y=-4, u=128, v=128)
global fix_r66=luma_lft.mt_lut(y=-5, u=128, v=128)

global AvgDrop=127
global MaxMulty=10
global dblMulti=1.3
global fix_clip=luma_drop.ScriptClip("
    drop=drop_clip.AverageLuma
    AvgLuma=abs(128-drop_clip.AverageLuma)
    AvgLuma_prev=abs(128-(drop_clip.trim(1,1)+drop_clip).AverageLuma)
    AvgLuma_prev_= (AvgLuma_prev==AvgDrop) ? 0 : AvgLuma_prev
    AvgLuma_prev2=abs(128-(drop_clip.trim(1,2)+drop_clip).AverageLuma)
    AvgLuma_next=abs(128-drop_clip.trim(1,0).AverageLuma)
    AvgLuma_next_= (AvgLuma_next==AvgDrop) ? 0 : AvgLuma_next
    AvgLuma_next2=abs(128-drop_clip.trim(2,0).AverageLuma)

    RightZero = (AvgLuma_prev_*MaxMulty<AvgLuma_next_) ? (AvgLuma_prev_>max_stop || AvgLuma_next_>max_stop) ? 1 : 0 : 0
    LeftZero = (AvgLuma_next_*MaxMulty<AvgLuma_prev_) ? (AvgLuma_prev_>max_stop || AvgLuma_next_>max_stop) ? 1 : 0 : 0

    drop = (AvgLuma==AvgDrop && AvgLuma_next==AvgDrop) ? -50 : 0
    drop = (AvgLuma==AvgDrop && AvgLuma_prev==AvgDrop) ? 50 : drop

    drop = (drop==0 && AvgLuma==AvgDrop && LeftZero==0 && RightZero==0) ? \
     AvgLuma_prev_>AvgLuma_next_*dblMulti ? -50 : \
      AvgLuma_prev_*dblMulti>=AvgLuma_next_ ? AvgLuma_next2==AvgDrop ? 50 : 33 : 50 : drop

    drop = (drop==0 && AvgLuma==AvgDrop && RightZero==0 && AvgLuma_next2==AvgDrop) ? -50 : drop
    drop = (drop==0 && AvgLuma==AvgDrop && LeftZero==0 && AvgLuma_prev2==AvgDrop) ? 66 : drop

    (drop==50) ? fix_r50 : \
     (drop==33) ? fix_lr33 : \
      (drop==-50) ? fix_l50 : \
       (drop==133) ? fix_r133 : \
        (drop==66) ? fix_r66 : drop_clip
")

global r50=1
global lr33=2
global l50=3
global r133=4
global r66=5

super_params = (myGPU==true) ? "{gpu:1}" : "{gpu:0}"

analyse_params="{main:{search:{coarse:{distance:4,bad:{sad:2000}},type:2,distance:4},penalty:{lambda:1,pglobal:10000}},refine:[{thsad:65000}]}"
smoothfps_params="{rate:{num:12,den:1},algo:13,scene:{blend:true}}"

super=SVSuper(super_params)
vectors=SVAnalyse(super, analyse_params)
fix_all = SVSmoothFps(super, vectors, smoothfps_params, mt=threads, url="www.svp-team.com")

dummy = myDebug ? Eval("""
# These are for visual debugging (the "Subtitle" clauses)
global fix50 = fix_all.SelectEvery(12,6).Subtitle("fix50", align=3, size=myErrSize)
global fix33 = fix_all.SelectEvery(12,4).Subtitle("fix33", align=3, size=myErrSize)
global fix66 = fix_all.SelectEvery(12,8).Subtitle("-fix33", align=3, size=myErrSize)
global fix66p = fix_all.SelectEvery(12,8).Subtitle("fix66", align=3, size=myErrSize)
global fix75 = fix_all.SelectEvery(12,9).Subtitle("-fix25", align=3, size=myErrSize)
global fix50n = fix_all.SelectEvery(12,6).Subtitle("-fix50", align=3, size=myErrSize)
global fix133 = fix_all.SelectEvery(12,4).Subtitle("fix133", align=3, size=myErrSize)
global fix125 = fix_all.SelectEvery(12,3).Subtitle("fix125", align=3, size=myErrSize)
""") : Eval("""
global fix50 = fix_all.SelectEvery(12,6)
global fix33 = fix_all.SelectEvery(12,4)
global fix66 = fix_all.SelectEvery(12,8)
global fix66p = fix_all.SelectEvery(12,8)
global fix75 = fix_all.SelectEvery(12,9)
global fix50n = fix_all.SelectEvery(12,6)
global fix133 = fix_all.SelectEvery(12,4)
global fix125 = fix_all.SelectEvery(12,3)
""")


SetMTMode(5)


ScriptClip("
    AvgFix=fix_clip.AverageLuma
    AvgFix_next=fix_clip.trim(1,0).AverageLuma
    AvgFix_next2=fix_clip.trim(2,0).AverageLuma
    AvgFix_next3=fix_clip.trim(3,0).AverageLuma
    AvgFix_prev=(fix_clip.trim(1,1)+fix_clip).AverageLuma

    (AvgFix==r50) \
      ? (AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix33 : fix50 : \
    (AvgFix==lr33) ? fix33 : \
    (AvgFix_next==lr33) \
     ? (AvgFix_prev==r66) ? fix75.trim(1,1)+fix75 : fix66.trim(1,1)+fix66 : \
    (AvgFix_next==l50) \
      ? (AvgFix_prev==lr33 || AvgFix_prev==r50) ? fix66.trim(1,1)+fix66 \
      : (AvgFix_prev==r66) ? fix75.trim(1,1)+fix75 : fix50n.trim(1,1)+fix50n : \
    (AvgFix==r66) \
      ? (AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix50 : fix66p : \
    (AvgFix_next==r66) \
      ? (AvgFix_next3==lr33 || AvgFix_next3==l50) ? fix125.trim(1,0) : fix133.trim(1,0) : \
    last
")

trim(0,framecount-threads)

}
# end of GameDropFixFuncs.avsi

function AdjSync(clip c, int fstart, float sec)
{ 	# Apply an audio delay starting at a point in the clip
	#
	# c = entire video
	# fstart = frame nbr of start of trailing section whose audio is to be changed
	# sec = nbr of seconds to delay (positive) or advance (negative) the audio (ex: -0.067)
	c
	trim(0,fstart-1)
	clip01=last
	c
	trim(fstart,0)
	clip02=last
	delayaudio(sec)
	clip02=last
	clip01++clip02
	c=last
	return c
}
function BlackOver(clip c, int fstart, int fend)
{
	before=c.Trim(0, -fstart)
	current=c.trim(fstart, fend)
	after=c.trim(fend + 1, 0)
	video=before++blankclip(current,framecount(current))++after
	return audiodub(video, c)
}

2.. Has this script ever run successfully?
2.a. What has changed recently?
2.b. Can you run an older script that worked before?

That's difficult to answer. I have perhaps hundreds of scripts, all intended to run once, or possibly a repeat someday. I test each script as I build it, making sure it runs in the Virtualdub window, both as I manually move from frame to frame, and sometimes pressing Play. So the script has been thoroughly tested in the Vdub GUI. Most of my scripts have many of the same elements. I always clone from a recently used script that was for a video file of similar nature. You can see I have some lines commented out, as illustration of this. This script does use GameDropFix which has been the cause (or trigger) of lots of headaches, usually some form of not allowing an encode to finish. This is one such instance. This script ran fine for the entire length of it's run EXCEPT at the end, probably as files were being closed.

3. Does the error occur if you select a short segment in the VirtualDub timeline for encoding? (10 seconds is a good length)
3.a. If so, you can proceed to make changes to the script and test them much more quickly.
3.b. If not, the problem may lie in one of your Avisynth timeline editing statements (splice, loop, trim, ...)

As I can play several seconds on the timeline via Virtualdub Play button, I guess the answer is no, the error does not occur. As I said, the encode runs the full 2.5 hours but gave the error at the very end, as shown by the image I included. For 3a, not sure what you mean. I make all changes and test IMMEDIATELY after each change by opening (re-opening - F2) the script on the Virtualdub timeline. 3b. If timeline editing statements were a problem, they'd fail immediately upon re-opening the file. (Of course, THAT's a pain, as LSMASH takes over 10 agonizing seconds to re-open the video file. And I re-open it every time I add a script line.

4. Are you using an External encoder like x264.exe? (External encoders have a tendency, if they fail, to fail at the very end like you describe. Not crash or run out of memory though, AFAIK)

I'm not sure. I encode with X264 that has been installed as a VFW codec. So that's X264vfw. Sounds "internal" to me, or maybe "external" depending on what you mean.

5. Any VirtualDub filters being used?

No. I'm using no filters and am using Fast Recompress, seeing as I have no filters.

I've made encoding runs of X264vfw hundreds of times. But I do have problems when it is longer than 20-30 min. of video AND I have GameDropFix in it. I've had a few interactions with the author, with only workarounds. My ultimate workaround was breaking up the 2 hour video into <30 min. chunks, a REAL PAIN. I thought I was on the verge of solving that, making this current file's run in one piece, when this stupid memory error happens at the very end. The earlier blowups were never at the end, but were somewhere between 30 min. in and the end, and a few different symptoms. So actually, this is the farthest I've gotten in a GameDropFix script. (But "an inch to short" is "too short", ha ha.)

[raffriff42]

>As I can play several seconds on the timeline via Virtualdub Play button, I guess the answer is no, the error does not occurs. As I said, the encode runs the full 2.5 hours but gave the error at the very end, as shown by the image I included.

Select 10 seconds on the timeline with Edit, set selection start & Set selection end. Make a test encode. If it fails, you just saved yourself 2.5 hours. You know something is wrong and needs fixing. Try commenting out different parts of your script. This will narrow down the problem. Comment out GameFix4 and see what happens. Try disabling MT, etc.

[TCmullet]

Originally Posted by raffriff42

Select 10 seconds on the timeline with Edit, set selection start & Set selection end. Make a test encode. If it fails, you just saved yourself 2.5 hours. You know something is wrong and needs fixing. Try commenting out different parts of your script. This will narrow down the problem. Comment out GameFix4 and see what happens. Try disabling MT, etc.

I selected 15 sec. Encoded to x264vfw just fine as I expected. I already said I could encode 20-30 min. fine, so it would make sense that 10 sec. would be fine also. But I did it anyway, as you asked me too.

[TCmullet]

Originally Posted by MysteryX

It doesn't matter how much RAM you use. 32-bit processes can only access 2GB; and it can be patched to use 3GB or 4GB, but no more. When you look in Task Manager, how much memory does it use? One way to reduce memory usage is to reduce the amount of threads.

MysteryX, I re-read this a few more times and I think now that you wanted me to click the task mgr TAB that says "Processes". When I do, it shows my task (which uses CPU like nothing else there) is "VirtualDub.exe *32". I remember noticing months ago that it didn't say "Avisynth" which I would have expected. But as Avisynth is a frontend to Vdub, then perhaps it makes sense the all of my stuff is inside of Vdub.

It started around 970,000 K and seems to be creeping upward. After 5 min. of rerunning, it's up to 992,000 K. That's almost a gigabyte. Is this in anyway connected to "SetMemoryMax(1024)" which appears to refer to 1GB?

I will await your answer to my question about "how to patch 'it' to 3GB. (There are so many different resource entities in this environment that it's confusing to say the least.) I saw so many numbers on the main window of Task Manager that I (wrongly) figured all relevant info must be somewhere there. It puzzles me that there could be 20GB free, yet I run out of a "certain kind" of memory way before that.

Okay, my "VirtualDub *32" process is now up to 1,012,000 K after 10 minutes. I guess I'll leave that tab open while I let the thing run again. Phys. mem. usage on the Performance tab is still only 3.x GB.

I have reason to believe that if I segment the video into 2 pieces, both will finish okay. It's doing the full video that triggers the bombing.

UPDATE: As of 1 hour into the run, the process is taking 1,252,000 K.

[StainlessS]

Vdub 3GB, maybe this lot will help:- https://www.google.co.uk/?gfe_rd=cr&...SAWARE+editbin

[TCmullet]

Originally Posted by StainlessS

Vdub 3GB, maybe this lot will help:- https://www.google.co.uk/?gfe_rd=cr&...SAWARE+editbin

I went to bed, setting an alarm to get me up 5 min. before it would end. However, it had already blown up from a different error. "KERNAL BASE" or something. It was a true Vdub crash. I failed to save the crashdump. But I did see that the process was currently up to 1,616,568 K. And your google hits DO say stuff about a 1.6GB limit. (And Vdub was over 4/5 done, not extremely close to the end as before, but certainly way beyond my normal break-up size. Oh, the file outputted was 6.72GB vs the prior one of 8.0.)

Okay, I will work on the 3GB "solution", but don't you see that this likely is not the real solution? The real question is why does the process keep taking more and more memory as it progresses? Conceptually, it's not doing anything that should cause that, that is, it's reading video date and generation new data *out to a file*. I would think the only thing it could be is a "MEMORY LEAK", that is, a failure of something that allocates data items regularly to *DEALLOCATE* the memory for such items when their temporary use is over (on a given iteration through the main loop). So it keeps taking and taking and taking til the limit is hit. Doesn't that make sense?? My guess would be it is Avisynth related, but I don't know enough about it's memory management to assess. Perhaps your 3GB trick(s) will get around it now. But my video files are only 2-3 hours long now. What about the day (which I'm planning on hopefully next year) where I have some unusual videos that are 6 to ***8*** hours long??? (No I don't want to break them up.) If there's a memory leak, we need to somehow find it. Is this leak an error that an Avisynth script writer can commit? I would have thought it could only be within the internals, like with any high-level programming language.

[creaothceann]

Originally Posted by TCmullet

Avisynth is a frontend to Vdub

VirtualDub is the frontend.

[raffriff42]

Ran your script with some complications removed
* substituted ColorBarsHD for your source
* disabled SetMTMode
* instead of encoding anything, ran VirtualDub's "run video analysis pass"

Memory leak in evidence after only a few minutes:



Next I commented out GameDropFixV4 and ran it again; memory leak not in evidence so far:


Must run now, more later maybe.


EDIT
In GameDropFixV4, commented out or bypassed the three ScriptClip calls - leak is gone.
The memory stops growing after the first few minutes, as in the 2nd image above.

[TCmullet]

Originally Posted by StainlessS

Vdub 3GB, maybe this lot will help:- https://www.google.co.uk/?gfe_rd=cr&...SAWARE+editbin

I read up on this as much as I thought necessary. (Thank you. This was insightful.) Some of the things appear to be needed only on XP. I'm running only on Win 7-64, so I obtained and ran the LargeAddressAware.exe. I changed the 32-bit VirtualDub.exe that I'm using. Yet, I've run the job a couple more times and each time it still blows up. Last time the Vdub task was 1,644,748 K. The "reasons" for blowup vary, but I would expect that. With this last one, I managed to save a Vdub crashdump, which I'll post if anyone thinks necessary (but I doubt it).

My question, StanlessS, is what else do I have to do besides modify VirtualDub.exe via the LAA program? I've run it on both of my 4-core PCs, and both still blow up around the 1.6 gb task size.

[TCmullet]

Originally Posted by raffriff42

Ran your script with some complications removed
* substituted ColorBarsHD for your source
* disabled SetMTMode
* instead of encoding anything, ran VirtualDub's "run video analysis pass"

Memory leak in evidence after only a few minutes:
...

Next I commented out GameDropFixV4 and ran it again; memory leak not in evidence so far:

EDIT
In GameDropFixV4, commented out or bypassed the three ScriptClip calls - leak is gone.
The memory stops growing after the first few minutes, as in the 2nd image above.

Thanks for these attempts, Raffriff. While I don't know what "scriptclip" is, I can see in the code that it is huge, as well as convoluted. Maybe I can pass this on to the author of GameDropFix. My experience has suggested that this is one of the biggest memory leaks that I've been having, but you helped isolate it more than I could have. Thanks.

I'm working on two fronts: 1. Get rid of memory leaks that eat up memory, no matter how much memory is available, and 2. increase available memory to beyond the 2GB boundary. #1 is the ultimate solution, but #2 would get me some success in the interim.

[foxyshadis]

Filtering HD video sometimes just needs lots of memory, like it or not, though what you're doing isn't too heavy. The 3GB fix is actually 4GB on 64-bit. The way I do it is to use CFF Explorer to apply the following change to the header, then save it back -- see the attachment.

Alternately, there's 64-bit VDub & Avisynth, although everything you need might not be available. Then you can use all the memory you need.

Unfortunately, ScriptClip is not made to be a replacement for writing a plugin, even if it's often abused that way, so it sucks up memory like a sieve and it's terribly slow. StainlessS could probably rewrite it in GScript better. You might have some success lowering SetMemoryMax to 512, to reduce unnecessary cached frames, since the Avisynth cache alone is half your available memory with a 2GB.

[Gavino]

Originally Posted by foxyshadis

Unfortunately, ScriptClip is not made to be a replacement for writing a plugin, even if it's often abused that way, so it sucks up memory like a sieve and it's terribly slow.

ScriptClip uses up memory in Avisynth's string table for each frame rendered.
Normally this is insignificant, but if it is called with a very long run-time script coupled with lots of source frames, it could eventually eat up a lot of memory.

See this post for the solution.

[StainlessS]

Here mod of two of your functions, (untested)

Function AdjSync(clip c, int fstart, float sec) {
    # Apply an audio delay starting at a point in the clip
    #
    # c = entire video
    # fstart = frame nbr of start of trailing section whose audio is to be changed
    # sec = nbr of seconds to delay (positive) or advance (negative) the audio (ex: -0.067)
    clip01 = (fstart>0) ? c.trim(0,fstart-1) : c.BlankClip(Length=0)
    clip02 = c.trim(fstart,0).delayaudio(sec).trim(0,0)  # trim 1st, pad delayed audio with silence. v & a same length.
#   clip02 = c.delayaudio(sec).trim(fstart,0)            # shift audio 1st, ie allow shift in from end of clip01 : v & a same length
    return clip01++clip02
}

Function BlackOver(clip c, int fstart, int fend) {
    # Accepts fend=0, to last frame
    fend = (fend==0) ? c.FrameCount-1 : fend
    before = (fstart>0) ? c.Trim(0, -fstart) : c.BlankClip(length=0)
    current= c.BlankClip(Length=fend-fstart+1)
    after=(fend+1<c.Framecount) ? c.trim(fend + 1, 0) : c.Blankclip(Length=0)
    return AudiodubEx(before+current+after, c)    # No error if no audio
}

I'll try to do something with the other game thing.

EDIT: Only brief testing (dont crash)