View Full Version : Out of memory (in Virtualdub)
TCmullet
30th December 2015, 00:31
I'm not sure whether to post this problem here or in the Virtualdub area. My gut says to post it here.
I've run an Avisynth script that takes about 2.5 hours to complete on my system. Just as it is about to finish encoding, Virtualdub gives this error message: "Out of memory (unable to allocate 3923088 bytes)". I've uploaded a screen showing the error window, Vdub's status window (showing that the job is at the very end) AND the Windows Task Manager showing various things.
I am baffled as this PC has 24GB of ram, and at least 80GB of free space on each of the several drives present. In the Avisynth script, I use avisynth.dll 2.5.8.6 MT from SVP, and SetMemoryMax is set to 1024. Opening the nearly complete file in Virtualdub triggers the "eternal" "Rekeying video stream", which will take longer to happen than rerunning Virtualdub. (But Vdub has bombed so why do that?)
This is Xeon processor with Windows 7, with 4-core, 8-hyperthreads.
In the pic of the task manager, it shows the ram flatlined. I regret that I may not have seen it quickly enough after the error occurred to see if memory usage had spiked.
How could it work so well throughout, then bomb right at the end?? (More importantly, how can I prevent this?) I tried to hunt for an answer at the Vdub forum, but it recently was taken offline.
StainlessS
30th December 2015, 00:58
Any clues as to what might be in this mysterious avisynth script ?
TCmullet
30th December 2015, 01:13
Any clues as to what might be in this mysterious avisynth script ?
I had thought that my screen print showing task mgr and the other things would be adequate. But as the jpg is taking awhile to "get approved", I'll post the script here (it may be a bit ugly.) If necessary, I'll later add function code from .avsi.
# Requires AviSynth 2.6MT (using 2.6.0.5MT)
# 9/11/2015 switched to avisynth.2.5.8.6.MT.svp.dll
SetMemoryMax(1024)
global svp_scheduler=true
global threads=4
global svp_cache_fwd=threads+10
SetMTMode(3,threads)
#function AdjSync(clip c, int fstart, float sec)
#function MuteRange(clip c, int fstart, int fend)
#function SoftenRange(clip c, int fstart, int fend, float pctAdj)
mainfile="2015-12-03.1900.wv.R3-Rnd1.NH-v-FL.sec+.rawX7.mp4"
video = LWLibavVideoSource(mainfile,cache=false)
audio = LWLibavAudioSource(mainfile)
AudioDub(video,audio)
AssumeFPS("ntsc_video")
# black out playback progress bar on bottom
Crop(0,0,0,-8) # crop from the bottom
AddBorders(0,0,0,8) # pad to the bottom
#Normalize(show=true)
Amplify(1.7956) # whatever Amplify Factor was shown in the "Normalize(show=true)" above
# delete comm brks
# Do sound sync check and adjust right after each edit
#zapscorebar0(180,72)
delayaudio(-0.067)
loop(0,0,6)
blackover(14939,14954)
loop(0,14955,16738)
adjsync(14955,-0.067)
loop(0,39576,43178)
loop(0,54171,55971)
blackover(54171,54173)
loop(0,72366,75948)
loop(0,76707,80314)
blackover(76707,76708)
loop(0,103818,105510)
adjsync(103818,0.033)
#loop(0,,)
#loop(0,,)
#loop(0,,)
# tack on 2 sec. of black
#main=last
#MyBlank=BlankClip(main,60)
#main++MyBlank
# use when upscaling from 432p
#aWarpSharp(depth=20.0, blurlevel=2, thresh=0.5, cm=1, bm=2)
GameDropFixV4(myGPU=true,myDebug=false,myErrSize=150)
# program is 752p. let YT downsize.
#SR(1280,720)
SetMTMode(2)
#InterFrame(Cores=4,Tuning="Film",NewNum=60000,NewDen=1001,GPU=true)
InterFrame(Cores=threads,Tuning="Film",GPU=true)
#
# send VirtualDub output to x264v-q20(superfast)
MysteryX
30th December 2015, 01:39
It doesn't matter how much RAM you use. 32-bit processes can only access 2GB; and it can be patched to use 3GB or 4GB, but no more. When you look in Task Manager, how much memory does it use? One way to reduce memory usage is to reduce the amount of threads.
TCmullet
30th December 2015, 01:43
It doesn't matter how much RAM you use. 32-bit processes can only access 2GB; and it can be patched to use 3GB or 4GB, but no more. When you look in Task Manager, how much memory does it use? One way to reduce memory usage is to reduce the amount of threads.My attached graphic has been approved. Please seek the answer there, esp. as I don't know how to answer you.
Regardless, how do we "patch" it to 3GB or 4GB?
As to threads, I want to use as many as possible, but I'll limit it to 4 even tho this PC has 8, as I want scripts to be compatible with my other PC which has only 4 (4 actual, no hyperthreads).
TCmullet
30th December 2015, 01:47
You can see from the Vdub status window, which was unchanging when I took the pic, that Time Elapsed EQUALED Total time (estimated), which meant it was essentially finished. This makes me suspect that the out-of-mem had to do with Vdub process of closing output files. I wish it had allocated this at the beginning rather than at the end, blowing up after all that time. But maybe that's a windows function, not Vdub's.
raffriff42
30th December 2015, 02:42
1. Undefined "GameDropFixV4", could you post it? Also "AdjSync" and "blackover."
2.. Has this script ever run successfully?
2.a. What has changed recently?
2.b. Can you run an older script that worked before?
3. Does the error occur if you select a short segment in the VirtualDub timeline for encoding? (10 seconds is a good length)
3.a. If so, you can proceed to make changes to the script and test them much more quickly.
3.b. If not, the problem may lie in one of your Avisynth timeline editing statements (splice, loop, trim, ...)
4. Are you using an External encoder (http://forum.doom9.org/showthread.php?t=171141) like x264.exe? (External encoders have a tendency, if they fail, to fail at the very end like you describe. Not crash or run out of memory though, AFAIK)
5. Any VirtualDub filters being used?
TCmullet
30th December 2015, 03:57
1. Undefined "GameDropFixV4", could you post it? Also "AdjSync" and "blackover."
# GameDropFixFuncs.avsi
function GameDropFixV4(clip c, bool "myGPU", bool "myDebug", int "myErrSize")
{
myGPU = default( myGPU, true )
myDebug = default( myDebug, false )
myErrSize = default( myErrSize, 12 )
last=c
super_params_mini = (myGPU==true) ? "{gpu:1}" : "{gpu:0}"
analyse_params_mini="{block:{w:8,h:8}}"
miniW = int(width/64)*32
miniW = (miniW<320) ? 320 : miniW
miniH = int(height/64)*32
miniH = (miniH<160) ? 160 : miniH
global mini=BicubicResize(miniW,miniH).TemporalSoften(1, 1, 0, scenechange=1, mode=2)
super_mini=mini.SVSuper(super_params_mini)
vectors_mini=SVAnalyse(super_mini, analyse_params_mini)
vectors_Forward=SVConvert(vectors_mini, false)
HorizontalSpeed_luma=mini.MMask(vectors_Forward, kind=3).convertToRGB32().PointResize(miniW/8, miniH/8).PointResize(miniW/4, miniH/4).ConvertToYV12().mt_lut(y=2, u=128, v=128)
VerticalSpeed_luma=mini.MMask(vectors_Forward, kind=4).convertToRGB32().PointResize(miniW/8, miniH/8).PointResize(miniW/4, miniH/4).ConvertToYV12().mt_lut(y=2, u=128, v=128)
size=16
global luma_lft=HorizontalSpeed_luma.crop(0,0,miniW/16,0).BicubicResize(size,size)
global luma_rgh=HorizontalSpeed_luma.crop(miniW/16+miniW/8,0,0,0).BicubicResize(size,size)
global luma_top=VerticalSpeed_luma.crop(0,0,0,miniH/16).BicubicResize(size,size)
global luma_btm=VerticalSpeed_luma.crop(0,miniH/16+miniH/8,0,0).BicubicResize(size,size)
global luma_drop=luma_lft.mt_lut(y=-1, u=128, v=128)
global move_idx=5
global y_idx=3
global min_move=0.1
global max_stop=0.5
global drop_clip=luma_drop.ScriptClip("
AvgLuma_lft=128-luma_lft.AverageLuma
AvgLuma_rgh=128-luma_rgh.AverageLuma
AvgLuma_lft_prev=128-(luma_lft.trim(1,1)+luma_lft).AverageLuma
AvgLuma_rgh_prev=128-(luma_rgh.trim(1,1)+luma_rgh).AverageLuma
AvgLuma_lft_next=128-luma_lft.trim(1,0).AverageLuma
AvgLuma_rgh_next=128-luma_rgh.trim(1,0).AverageLuma
AvgLuma_top=128-luma_top.AverageLuma
AvgLuma_btm=128-luma_btm.AverageLuma
AvgLuma_top_prev=128-(luma_top.trim(1,1)+luma_top).AverageLuma
AvgLuma_btm_prev=128-(luma_btm.trim(1,1)+luma_btm).AverageLuma
AvgLuma_top_next=128-luma_top.trim(1,0).AverageLuma
AvgLuma_btm_next=128-luma_btm.trim(1,0).AverageLuma
Max_lft=max(abs(AvgLuma_lft_prev),abs(AvgLuma_lft),abs(AvgLuma_lft_next))
Max_rgh=max(abs(AvgLuma_rgh_prev),abs(AvgLuma_rgh),abs(AvgLuma_rgh_next))
Max_top=max(abs(AvgLuma_top_prev),abs(AvgLuma_top),abs(AvgLuma_top_next))
Max_btm=max(abs(AvgLuma_btm_prev),abs(AvgLuma_btm),abs(AvgLuma_btm_next))
Max_all=max(Max_lft,Max_rgh,Max_top,Max_btm)
MaxLuma=max(abs(AvgLuma_lft),abs(AvgLuma_rgh),abs(AvgLuma_top),abs(AvgLuma_btm))
MaxLuma_prev=max(abs(AvgLuma_lft_prev),abs(AvgLuma_rgh_prev),abs(AvgLuma_top_prev),abs(AvgLuma_btm_prev))
MaxLuma_next=max(abs(AvgLuma_lft_next),abs(AvgLuma_rgh_next),abs(AvgLuma_top_next),abs(AvgLuma_btm_next))
dif=mini.YDifferenceFromPrevious
dif_next=mini.trim(1,0).YDifferenceFromPrevious
dif_prev=(mini.trim(1,1)+mini).YDifferenceFromPrevious
GoodContrast = (max(mini.YPlaneMinMaxDifference,mini.trim(1,0).YPlaneMinMaxDifference)>20) ? 1 : 0
drop = (MaxLuma<0.1 && MaxLuma*2<max(MaxLuma_prev,MaxLuma_next) && dif*y_idx<max(dif_prev,dif_next)) ? 1 : 0
drop_lft = (AvgLuma_lft_prev*AvgLuma_lft_next>0 && abs(AvgLuma_lft*move_idx)<Max_lft && abs(AvgLuma_lft)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 2 : 0
drop_rgh = (AvgLuma_rgh_prev*AvgLuma_rgh_next>0 && abs(AvgLuma_rgh*move_idx)<Max_rgh && abs(AvgLuma_rgh)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 3 : 0
drop_top = (AvgLuma_top_prev*AvgLuma_top_next>0 && abs(AvgLuma_top*move_idx)<Max_top && abs(AvgLuma_top)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 4 : 0
drop_btm = (AvgLuma_btm_prev*AvgLuma_btm_next>0 && abs(AvgLuma_btm*move_idx)<Max_btm && abs(AvgLuma_btm)<=min_move && dif*y_idx<max(dif_prev,dif_next)) ? 5 : 0
drop = (drop==0 && MaxLuma<Max_stop && Max_lft==Max_all) ? drop_lft : drop
drop = (drop==0 && MaxLuma<Max_stop && Max_rgh==Max_all) ? drop_rgh : drop
drop = (drop==0 && MaxLuma<Max_stop && Max_top==Max_all) ? drop_top : drop
drop = (drop==0 && MaxLuma<Max_stop && Max_btm==Max_all) ? drop_btm : drop
drop_dif = (dif<0.4 && MaxLuma<min_move) ? 6 : 0
drop = (drop==0 && drop_dif>0) ? drop_dif : drop
drop = (drop>0 && Max_all<0.1) ? 0 : drop
drop = (drop>0 && GoodContrast==0) ? 0 : drop
luma=mini.AverageLuma
luma_next=mini.trim(1,0).AverageLuma
max_luma=max(luma,luma_next)
drop = (drop>0 && max_luma<16.5) ? 0 : drop
(drop>0) ? luma_drop : \
(Max_lft==Max_all) ? luma_lft : \
(Max_rgh==Max_all) ? luma_rgh : \
(Max_top==Max_all) ? luma_top : luma_btm
")
global fix_r50=luma_lft.mt_lut(y=-1, u=128, v=128)
global fix_lr33=luma_lft.mt_lut(y=-2, u=128, v=128)
global fix_l50=luma_lft.mt_lut(y=-3, u=128, v=128)
global fix_r133=luma_lft.mt_lut(y=-4, u=128, v=128)
global fix_r66=luma_lft.mt_lut(y=-5, u=128, v=128)
global AvgDrop=127
global MaxMulty=10
global dblMulti=1.3
global fix_clip=luma_drop.ScriptClip("
drop=drop_clip.AverageLuma
AvgLuma=abs(128-drop_clip.AverageLuma)
AvgLuma_prev=abs(128-(drop_clip.trim(1,1)+drop_clip).AverageLuma)
AvgLuma_prev_= (AvgLuma_prev==AvgDrop) ? 0 : AvgLuma_prev
AvgLuma_prev2=abs(128-(drop_clip.trim(1,2)+drop_clip).AverageLuma)
AvgLuma_next=abs(128-drop_clip.trim(1,0).AverageLuma)
AvgLuma_next_= (AvgLuma_next==AvgDrop) ? 0 : AvgLuma_next
AvgLuma_next2=abs(128-drop_clip.trim(2,0).AverageLuma)
RightZero = (AvgLuma_prev_*MaxMulty<AvgLuma_next_) ? (AvgLuma_prev_>max_stop || AvgLuma_next_>max_stop) ? 1 : 0 : 0
LeftZero = (AvgLuma_next_*MaxMulty<AvgLuma_prev_) ? (AvgLuma_prev_>max_stop || AvgLuma_next_>max_stop) ? 1 : 0 : 0
drop = (AvgLuma==AvgDrop && AvgLuma_next==AvgDrop) ? -50 : 0
drop = (AvgLuma==AvgDrop && AvgLuma_prev==AvgDrop) ? 50 : drop
drop = (drop==0 && AvgLuma==AvgDrop && LeftZero==0 && RightZero==0) ? \
AvgLuma_prev_>AvgLuma_next_*dblMulti ? -50 : \
AvgLuma_prev_*dblMulti>=AvgLuma_next_ ? AvgLuma_next2==AvgDrop ? 50 : 33 : 50 : drop
drop = (drop==0 && AvgLuma==AvgDrop && RightZero==0 && AvgLuma_next2==AvgDrop) ? -50 : drop
drop = (drop==0 && AvgLuma==AvgDrop && LeftZero==0 && AvgLuma_prev2==AvgDrop) ? 66 : drop
(drop==50) ? fix_r50 : \
(drop==33) ? fix_lr33 : \
(drop==-50) ? fix_l50 : \
(drop==133) ? fix_r133 : \
(drop==66) ? fix_r66 : drop_clip
")
global r50=1
global lr33=2
global l50=3
global r133=4
global r66=5
super_params = (myGPU==true) ? "{gpu:1}" : "{gpu:0}"
analyse_params="{main:{search:{coarse:{distance:4,bad:{sad:2000}},type:2,distance:4},penalty:{lambda:1,pglobal:10000}},refine:[{thsad:65000}]}"
smoothfps_params="{rate:{num:12,den:1},algo:13,scene:{blend:true}}"
super=SVSuper(super_params)
vectors=SVAnalyse(super, analyse_params)
fix_all = SVSmoothFps(super, vectors, smoothfps_params, mt=threads, url="www.svp-team.com")
dummy = myDebug ? Eval("""
# These are for visual debugging (the "Subtitle" clauses)
global fix50 = fix_all.SelectEvery(12,6).Subtitle("fix50", align=3, size=myErrSize)
global fix33 = fix_all.SelectEvery(12,4).Subtitle("fix33", align=3, size=myErrSize)
global fix66 = fix_all.SelectEvery(12,8).Subtitle("-fix33", align=3, size=myErrSize)
global fix66p = fix_all.SelectEvery(12,8).Subtitle("fix66", align=3, size=myErrSize)
global fix75 = fix_all.SelectEvery(12,9).Subtitle("-fix25", align=3, size=myErrSize)
global fix50n = fix_all.SelectEvery(12,6).Subtitle("-fix50", align=3, size=myErrSize)
global fix133 = fix_all.SelectEvery(12,4).Subtitle("fix133", align=3, size=myErrSize)
global fix125 = fix_all.SelectEvery(12,3).Subtitle("fix125", align=3, size=myErrSize)
""") : Eval("""
global fix50 = fix_all.SelectEvery(12,6)
global fix33 = fix_all.SelectEvery(12,4)
global fix66 = fix_all.SelectEvery(12,8)
global fix66p = fix_all.SelectEvery(12,8)
global fix75 = fix_all.SelectEvery(12,9)
global fix50n = fix_all.SelectEvery(12,6)
global fix133 = fix_all.SelectEvery(12,4)
global fix125 = fix_all.SelectEvery(12,3)
""")
SetMTMode(5)
ScriptClip("
AvgFix=fix_clip.AverageLuma
AvgFix_next=fix_clip.trim(1,0).AverageLuma
AvgFix_next2=fix_clip.trim(2,0).AverageLuma
AvgFix_next3=fix_clip.trim(3,0).AverageLuma
AvgFix_prev=(fix_clip.trim(1,1)+fix_clip).AverageLuma
(AvgFix==r50) \
? (AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix33 : fix50 : \
(AvgFix==lr33) ? fix33 : \
(AvgFix_next==lr33) \
? (AvgFix_prev==r66) ? fix75.trim(1,1)+fix75 : fix66.trim(1,1)+fix66 : \
(AvgFix_next==l50) \
? (AvgFix_prev==lr33 || AvgFix_prev==r50) ? fix66.trim(1,1)+fix66 \
: (AvgFix_prev==r66) ? fix75.trim(1,1)+fix75 : fix50n.trim(1,1)+fix50n : \
(AvgFix==r66) \
? (AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix50 : fix66p : \
(AvgFix_next==r66) \
? (AvgFix_next3==lr33 || AvgFix_next3==l50) ? fix125.trim(1,0) : fix133.trim(1,0) : \
last
")
trim(0,framecount-threads)
}
# end of GameDropFixFuncs.avsi
function AdjSync(clip c, int fstart, float sec)
{ # Apply an audio delay starting at a point in the clip
#
# c = entire video
# fstart = frame nbr of start of trailing section whose audio is to be changed
# sec = nbr of seconds to delay (positive) or advance (negative) the audio (ex: -0.067)
c
trim(0,fstart-1)
clip01=last
c
trim(fstart,0)
clip02=last
delayaudio(sec)
clip02=last
clip01++clip02
c=last
return c
}
function BlackOver(clip c, int fstart, int fend)
{
before=c.Trim(0, -fstart)
current=c.trim(fstart, fend)
after=c.trim(fend + 1, 0)
video=before++blankclip(current,framecount(current))++after
return audiodub(video, c)
}
2.. Has this script ever run successfully?
2.a. What has changed recently?
2.b. Can you run an older script that worked before?
That's difficult to answer. I have perhaps hundreds of scripts, all intended to run once, or possibly a repeat someday. I test each script as I build it, making sure it runs in the Virtualdub window, both as I manually move from frame to frame, and sometimes pressing Play. So the script has been thoroughly tested in the Vdub GUI. Most of my scripts have many of the same elements. I always clone from a recently used script that was for a video file of similar nature. You can see I have some lines commented out, as illustration of this. This script does use GameDropFix which has been the cause (or trigger) of lots of headaches, usually some form of not allowing an encode to finish. This is one such instance. This script ran fine for the entire length of it's run EXCEPT at the end, probably as files were being closed.
3. Does the error occur if you select a short segment in the VirtualDub timeline for encoding? (10 seconds is a good length)
3.a. If so, you can proceed to make changes to the script and test them much more quickly.
3.b. If not, the problem may lie in one of your Avisynth timeline editing statements (splice, loop, trim, ...)
As I can play several seconds on the timeline via Virtualdub Play button, I guess the answer is no, the error does not occur. As I said, the encode runs the full 2.5 hours but gave the error at the very end, as shown by the image I included. For 3a, not sure what you mean. I make all changes and test IMMEDIATELY after each change by opening (re-opening - F2) the script on the Virtualdub timeline. 3b. If timeline editing statements were a problem, they'd fail immediately upon re-opening the file. (Of course, THAT's a pain, as LSMASH takes over 10 agonizing seconds to re-open the video file. And I re-open it every time I add a script line.
4. Are you using an External encoder (http://forum.doom9.org/showthread.php?t=171141) like x264.exe? (External encoders have a tendency, if they fail, to fail at the very end like you describe. Not crash or run out of memory though, AFAIK)
I'm not sure. I encode with X264 that has been installed as a VFW codec. So that's X264vfw. Sounds "internal" to me, or maybe "external" depending on what you mean.
5. Any VirtualDub filters being used?
No. I'm using no filters and am using Fast Recompress, seeing as I have no filters.
I've made encoding runs of X264vfw hundreds of times. But I do have problems when it is longer than 20-30 min. of video AND I have GameDropFix in it. I've had a few interactions with the author, with only workarounds. My ultimate workaround was breaking up the 2 hour video into <30 min. chunks, a REAL PAIN. I thought I was on the verge of solving that, making this current file's run in one piece, when this stupid memory error happens at the very end. The earlier blowups were never at the end, but were somewhere between 30 min. in and the end, and a few different symptoms. So actually, this is the farthest I've gotten in a GameDropFix script. (But "an inch to short" is "too short", ha ha.)
raffriff42
30th December 2015, 04:40
>As I can play several seconds on the timeline via Virtualdub Play button, I guess the answer is no, the error does not occurs. As I said, the encode runs the full 2.5 hours but gave the error at the very end, as shown by the image I included.
Select 10 seconds on the timeline with Edit, set selection start & Set selection end. Make a test encode. If it fails, you just saved yourself 2.5 hours. You know something is wrong and needs fixing. Try commenting out different parts of your script. This will narrow down the problem. Comment out GameFix4 and see what happens. Try disabling MT, etc.
TCmullet
30th December 2015, 05:10
Select 10 seconds on the timeline with Edit, set selection start & Set selection end. Make a test encode. If it fails, you just saved yourself 2.5 hours. You know something is wrong and needs fixing. Try commenting out different parts of your script. This will narrow down the problem. Comment out GameFix4 and see what happens. Try disabling MT, etc.
I selected 15 sec. Encoded to x264vfw just fine as I expected. I already said I could encode 20-30 min. fine, so it would make sense that 10 sec. would be fine also. But I did it anyway, as you asked me too.
TCmullet
30th December 2015, 05:27
It doesn't matter how much RAM you use. 32-bit processes can only access 2GB; and it can be patched to use 3GB or 4GB, but no more. When you look in Task Manager, how much memory does it use? One way to reduce memory usage is to reduce the amount of threads.
MysteryX, I re-read this a few more times and I think now that you wanted me to click the task mgr TAB that says "Processes". When I do, it shows my task (which uses CPU like nothing else there) is "VirtualDub.exe *32". I remember noticing months ago that it didn't say "Avisynth" which I would have expected. But as Avisynth is a frontend to Vdub, then perhaps it makes sense the all of my stuff is inside of Vdub.
It started around 970,000 K and seems to be creeping upward. After 5 min. of rerunning, it's up to 992,000 K. That's almost a gigabyte. Is this in anyway connected to "SetMemoryMax(1024)" which appears to refer to 1GB?
I will await your answer to my question about "how to patch 'it' to 3GB. (There are so many different resource entities in this environment that it's confusing to say the least.) I saw so many numbers on the main window of Task Manager that I (wrongly) figured all relevant info must be somewhere there. It puzzles me that there could be 20GB free, yet I run out of a "certain kind" of memory way before that.
Okay, my "VirtualDub *32" process is now up to 1,012,000 K after 10 minutes. I guess I'll leave that tab open while I let the thing run again. Phys. mem. usage on the Performance tab is still only 3.x GB.
I have reason to believe that if I segment the video into 2 pieces, both will finish okay. It's doing the full video that triggers the bombing.
UPDATE: As of 1 hour into the run, the process is taking 1,252,000 K.
StainlessS
30th December 2015, 07:02
Vdub 3GB, maybe this lot will help:- https://www.google.co.uk/?gfe_rd=cr&ei=tAFqVuTzKdLS8AfknKK4BQ&gws_rd=ssl#q=virtualdub+3GB+LARGEADDRESSAWARE+editbin
TCmullet
30th December 2015, 08:11
Vdub 3GB, maybe this lot will help:- https://www.google.co.uk/?gfe_rd=cr&ei=tAFqVuTzKdLS8AfknKK4BQ&gws_rd=ssl#q=virtualdub+3GB+LARGEADDRESSAWARE+editbin
I went to bed, setting an alarm to get me up 5 min. before it would end. However, it had already blown up from a different error. "KERNAL BASE" or something. It was a true Vdub crash. I failed to save the crashdump. But I did see that the process was currently up to 1,616,568 K. And your google hits DO say stuff about a 1.6GB limit. (And Vdub was over 4/5 done, not extremely close to the end as before, but certainly way beyond my normal break-up size. Oh, the file outputted was 6.72GB vs the prior one of 8.0.)
Okay, I will work on the 3GB "solution", but don't you see that this likely is not the real solution? The real question is why does the process keep taking more and more memory as it progresses? Conceptually, it's not doing anything that should cause that, that is, it's reading video date and generation new data *out to a file*. I would think the only thing it could be is a "MEMORY LEAK", that is, a failure of something that allocates data items regularly to *DEALLOCATE* the memory for such items when their temporary use is over (on a given iteration through the main loop). So it keeps taking and taking and taking til the limit is hit. Doesn't that make sense?? My guess would be it is Avisynth related, but I don't know enough about it's memory management to assess. Perhaps your 3GB trick(s) will get around it now. But my video files are only 2-3 hours long now. What about the day (which I'm planning on hopefully next year) where I have some unusual videos that are 6 to ***8*** hours long??? (No I don't want to break them up.) If there's a memory leak, we need to somehow find it. Is this leak an error that an Avisynth script writer can commit? I would have thought it could only be within the internals, like with any high-level programming language.
creaothceann
30th December 2015, 09:12
Avisynth is a frontend to Vdub
VirtualDub is the frontend.
raffriff42
30th December 2015, 16:12
Ran your script with some complications removed
* substituted ColorBarsHD for your source
* disabled SetMTMode
* instead of encoding anything, ran VirtualDub's "run video analysis pass"
Memory leak in evidence after only a few minutes:
https://www.dropbox.com/s/0nqgo0fruc063mu/memleak-test-1-00%3B06%3B13.png?raw=1
Next I commented out GameDropFixV4 and ran it again; memory leak not in evidence so far:
https://www.dropbox.com/s/zmmlblqq7rpxr4o/memleak-test-2-00%3B07%3B46.png?raw=1
Must run now, more later maybe.
EDIT
In GameDropFixV4, commented out or bypassed the three ScriptClip calls - leak is gone.
The memory stops growing after the first few minutes, as in the 2nd image above.
TCmullet
31st December 2015, 19:14
Vdub 3GB, maybe this lot will help:- https://www.google.co.uk/?gfe_rd=cr&ei=tAFqVuTzKdLS8AfknKK4BQ&gws_rd=ssl#q=virtualdub+3GB+LARGEADDRESSAWARE+editbin
I read up on this as much as I thought necessary. (Thank you. This was insightful.) Some of the things appear to be needed only on XP. I'm running only on Win 7-64, so I obtained and ran the LargeAddressAware.exe. I changed the 32-bit VirtualDub.exe that I'm using. Yet, I've run the job a couple more times and each time it still blows up. Last time the Vdub task was 1,644,748 K. The "reasons" for blowup vary, but I would expect that. With this last one, I managed to save a Vdub crashdump, which I'll post if anyone thinks necessary (but I doubt it).
My question, StanlessS, is what else do I have to do besides modify VirtualDub.exe via the LAA program? I've run it on both of my 4-core PCs, and both still blow up around the 1.6 gb task size.
TCmullet
31st December 2015, 19:21
Ran your script with some complications removed
* substituted ColorBarsHD for your source
* disabled SetMTMode
* instead of encoding anything, ran VirtualDub's "run video analysis pass"
Memory leak in evidence after only a few minutes:
...
Next I commented out GameDropFixV4 and ran it again; memory leak not in evidence so far:
EDIT
In GameDropFixV4, commented out or bypassed the three ScriptClip calls - leak is gone.
The memory stops growing after the first few minutes, as in the 2nd image above.
Thanks for these attempts, Raffriff. While I don't know what "scriptclip" is, I can see in the code that it is huge, as well as convoluted. Maybe I can pass this on to the author of GameDropFix. My experience has suggested that this is one of the biggest memory leaks that I've been having, but you helped isolate it more than I could have. Thanks.
I'm working on two fronts: 1. Get rid of memory leaks that eat up memory, no matter how much memory is available, and 2. increase available memory to beyond the 2GB boundary. #1 is the ultimate solution, but #2 would get me some success in the interim.
foxyshadis
1st January 2016, 08:21
Filtering HD video sometimes just needs lots of memory, like it or not, though what you're doing isn't too heavy. The 3GB fix is actually 4GB on 64-bit. The way I do it is to use CFF Explorer (http://explorer-suite-iv.software.informer.com/download/) to apply the following change to the header, then save it back -- see the attachment.
Alternately, there's 64-bit VDub & Avisynth, although everything you need might not be available. Then you can use all the memory you need.
Unfortunately, ScriptClip is not made to be a replacement for writing a plugin, even if it's often abused that way, so it sucks up memory like a sieve and it's terribly slow. StainlessS could probably rewrite it in GScript better. You might have some success lowering SetMemoryMax to 512, to reduce unnecessary cached frames, since the Avisynth cache alone is half your available memory with a 2GB.
Gavino
1st January 2016, 11:04
Unfortunately, ScriptClip is not made to be a replacement for writing a plugin, even if it's often abused that way, so it sucks up memory like a sieve and it's terribly slow.
ScriptClip uses up memory in Avisynth's string table for each frame rendered.
Normally this is insignificant, but if it is called with a very long run-time script coupled with lots of source frames, it could eventually eat up a lot of memory.
See this post for the solution.
StainlessS
1st January 2016, 13:18
Here mod of two of your functions, (untested)
Function AdjSync(clip c, int fstart, float sec) {
# Apply an audio delay starting at a point in the clip
#
# c = entire video
# fstart = frame nbr of start of trailing section whose audio is to be changed
# sec = nbr of seconds to delay (positive) or advance (negative) the audio (ex: -0.067)
clip01 = (fstart>0) ? c.trim(0,fstart-1) : c.BlankClip(Length=0)
clip02 = c.trim(fstart,0).delayaudio(sec).trim(0,0) # trim 1st, pad delayed audio with silence. v & a same length.
# clip02 = c.delayaudio(sec).trim(fstart,0) # shift audio 1st, ie allow shift in from end of clip01 : v & a same length
return clip01++clip02
}
Function BlackOver(clip c, int fstart, int fend) {
# Accepts fend=0, to last frame
fend = (fend==0) ? c.FrameCount-1 : fend
before = (fstart>0) ? c.Trim(0, -fstart) : c.BlankClip(length=0)
current= c.BlankClip(Length=fend-fstart+1)
after=(fend+1<c.Framecount) ? c.trim(fend + 1, 0) : c.Blankclip(Length=0)
return AudiodubEx(before+current+after, c) # No error if no audio
}
I'll try to do something with the other game thing.
EDIT: Only brief testing (dont crash)
StainlessS
3rd January 2016, 21:41
Gee TC (is that 'Top Cat' ?)
Can you try this out, no idea if it does as it is supposed to, dont do anything on the two clips (1 anime) that I tried it on, (but then the original
script did not seem to do anything either).
Function GameDropFixV4_MI(clip c, bool "myGPU", bool "myDebug", int "myErrSize",int "threads",int "MtMode") {
c
myGPU = default( myGPU, true )
myDebug = default( myDebug, false )
myErrSize = default( myErrSize, 12 )
threads = default( threads, 1 )
MtMode = default(MtMode,0) # default 0, dont call SetMtMode
###
Func_S=("""
super_params_mini = (myGPU==true) ? "{gpu:1}" : "{gpu:0}"
analyse_params_mini="{block:{w:8,h:8}}"
miniW = max(int(width/64)*32,320)
miniH = max(int(height/64)*32,160)
mini = BicubicResize(miniW,miniH).TemporalSoften(1, 1, 0, scenechange=1, mode=2)
super_mini=mini.SVSuper(super_params_mini)
vectors_mini=SVAnalyse(super_mini, analyse_params_mini)
vectors_Forward=SVConvert(vectors_mini, false)
HorizontalSpeed_luma=mini.MMask(vectors_Forward, kind=3).convertToRGB32().PointResize(miniW/8, miniH/8)
\ .PointResize(miniW/4, miniH/4).ConvertToYV12().mt_lut(y=2, u=128, v=128)
VerticalSpeed_luma=mini.MMask(vectors_Forward, kind=4).convertToRGB32().PointResize(miniW/8, miniH/8)
\ .PointResize(miniW/4, miniH/4).ConvertToYV12().mt_lut(y=2, u=128, v=128)
###
size=16
luma_lft = HorizontalSpeed_luma.crop(0,0,miniW/16,0).BicubicResize(size,size)
luma_rgh = HorizontalSpeed_luma.crop(miniW/16+miniW/8,0,0,0).BicubicResize(size,size)
luma_top = VerticalSpeed_luma.crop(0,0,0,miniH/16).BicubicResize(size,size)
luma_btm = VerticalSpeed_luma.crop(0,miniH/16+miniH/8,0,0).BicubicResize(size,size)
luma_drop = luma_lft.mt_lut(y=-1, u=128, v=128)
###
super_params = (myGPU==true) ? "{gpu:1}" : "{gpu:0}"
analyse_params="{main:{search:{coarse:{distance:4,bad:{sad:2000}},type:2,distance:4},penalty:{lambda:1,pglobal:10000}},refine:[{thsad:65000}]}"
smoothfps_params="{rate:{num:12,den:1},algo:13,scene:{blend:true}}"
super = SVSuper(super_params)
vectors = SVAnalyse(super, analyse_params)
fix_all = SVSmoothFps(super, vectors, smoothfps_params, mt=threads, url="www.svp-team.com")
###
Function FnA@@@(clip luma_drop,clip mini,clip luma_lft,clip luma_rgh,clip luma_top,clip luma_btm,Int move_idx,Int y_idx,
\ Float min_move,Float max_stop) {
luma_drop n = current_frame
AvgLuma_lft = 128.0-luma_lft.RT_AverageLuma(n=n)
AvgLuma_rgh = 128.0-luma_rgh.RT_AverageLuma(n=n)
AvgLuma_lft_prev = 128.0-luma_lft.RT_AverageLuma(n=n,delta=-1)
AvgLuma_rgh_prev = 128.0-luma_rgh.RT_AverageLuma(n=n,delta=-1)
AvgLuma_lft_next = 128.0-luma_lft.RT_AverageLuma(n=n,delta= 1)
AvgLuma_rgh_next = 128.0-luma_rgh.RT_AverageLuma(n=n,delta= 1)
AvgLuma_top = 128.0-luma_top.RT_AverageLuma(n=n)
AvgLuma_btm = 128.0-luma_btm.RT_AverageLuma(n=n)
AvgLuma_top_prev = 128.0-luma_top.RT_AverageLuma(n=n,delta=-1)
AvgLuma_btm_prev = 128.0-luma_btm.RT_AverageLuma(n=n,delta=-1)
AvgLuma_top_next = 128.0-luma_top.RT_AverageLuma(n=n,delta= 1)
AvgLuma_btm_next = 128.0-luma_btm.RT_AverageLuma(n=n,delta= 1)
Max_lft = max(abs(AvgLuma_lft_prev),abs(AvgLuma_lft),abs(AvgLuma_lft_next))
Max_rgh = max(abs(AvgLuma_rgh_prev),abs(AvgLuma_rgh),abs(AvgLuma_rgh_next))
Max_top = max(abs(AvgLuma_top_prev),abs(AvgLuma_top),abs(AvgLuma_top_next))
Max_btm = max(abs(AvgLuma_btm_prev),abs(AvgLuma_btm),abs(AvgLuma_btm_next))
Max_all = max(Max_lft,Max_rgh,Max_top,Max_btm)
MaxLuma=max(abs(AvgLuma_lft),abs(AvgLuma_rgh),abs(AvgLuma_top),abs(AvgLuma_btm))
MaxLuma_prev=max(abs(AvgLuma_lft_prev),abs(AvgLuma_rgh_prev),abs(AvgLuma_top_prev),abs(AvgLuma_btm_prev))
MaxLuma_next=max(abs(AvgLuma_lft_next),abs(AvgLuma_rgh_next),abs(AvgLuma_top_next),abs(AvgLuma_btm_next))
dif = mini.RT_YDifference(n=n,delta=-1) # from previous
dif_next = mini.RT_YDifference(n=n,delta= 1) # to next
dif_prev = mini.RT_YDifference(n=n-1,delta=-1) # n-1 -> n-2
GoodContrast = (max(mini.RT_YPlaneMinMaxDifference(n=n),mini.RT_YPlaneMinMaxDifference(n=n+1))>20)
drop = (MaxLuma<0.1 && MaxLuma*2<max(MaxLuma_prev,MaxLuma_next) && dif*y_idx<max(dif_prev,dif_next)) ? 1 : 0
if(dif*y_idx<max(dif_prev,dif_next)) {
drop_lft = (AvgLuma_lft_prev*AvgLuma_lft_next>0 && abs(AvgLuma_lft*move_idx)<Max_lft && abs(AvgLuma_lft)<=min_move) ? 2 : 0
drop_rgh = (AvgLuma_rgh_prev*AvgLuma_rgh_next>0 && abs(AvgLuma_rgh*move_idx)<Max_rgh && abs(AvgLuma_rgh)<=min_move) ? 3 : 0
drop_top = (AvgLuma_top_prev*AvgLuma_top_next>0 && abs(AvgLuma_top*move_idx)<Max_top && abs(AvgLuma_top)<=min_move) ? 4 : 0
drop_btm = (AvgLuma_btm_prev*AvgLuma_btm_next>0 && abs(AvgLuma_btm*move_idx)<Max_btm && abs(AvgLuma_btm)<=min_move) ? 5 : 0
if(drop == 0 && MaxLuma<Max_stop) {
drop = ( Max_lft==Max_all) ? drop_lft : drop
drop = (drop==0 && Max_rgh==Max_all) ? drop_rgh : drop
drop = (drop==0 && Max_top==Max_all) ? drop_top : drop
drop = (drop==0 && Max_btm==Max_all) ? drop_btm : drop
}
} else {drop_lft=0 drop_rgh=0 drop_top=0 drop_bot=0}
drop_dif = (dif<0.4 && MaxLuma<min_move) ? 6 : 0
drop = (drop==0 && drop_dif>0) ? drop_dif : drop
drop = (drop>0 && Max_all<0.1) ? 0 : drop
drop = (drop>0 && !GoodContrast) ? 0 : drop
luma = mini.RT_AverageLuma(n=n)
luma_next = mini.RT_AverageLuma(n=n,delta= 1)
max_luma = max(luma,luma_next)
drop = (drop>0 && max_luma<16.5) ? 0 : drop
(drop>0) ? last : \
(Max_lft==Max_all) ? luma_lft : \
(Max_rgh==Max_all) ? luma_rgh : \
(Max_top==Max_all) ? luma_top : luma_btm
Return Last
}
# Local arg names for constants import
move_idx = 5
y_idx = 3
min_move = 0.1
max_stop = 0.5
ARGS1 = "mini,luma_lft,luma_rgh,luma_top,luma_btm,move_idx,y_idx,min_move,max_stop"
drop_clip = luma_drop.GScriptClip("FnA@@@(last, "+ARGS1+")", local=true, args=ARGS1,after_frame=true)
################
Function FnB@@@(clip drop_clip,clip fix_r50,clip fix_lr33,clip fix_l50,clip fix_r133,clip fix_r66,
\ Float AvgDrop,Float MaxMulty,float dblMulti,Float max_stop) {
drop_clip n = current_frame
AvgLuma_prev2 = abs(128.0-Last.RT_averageLuma(n=n,delta=-2))
AvgLuma_prev = abs(128.0-Last.RT_averageLuma(n=n,delta=-1))
AvgLuma = abs(128.0-Last.RT_AverageLuma(n=n))
AvgLuma_next = abs(128.0-Last.RT_averageLuma(n=n,delta= 1))
AvgLuma_next2 = abs(128.0-Last.RT_averageLuma(n=n,delta= 2))
AvgLuma_prev_ = (AvgLuma_prev==AvgDrop) ? 0 : AvgLuma_prev
AvgLuma_next_ = (AvgLuma_next==AvgDrop) ? 0 : AvgLuma_next
RightZero = (AvgLuma_prev_*MaxMulty<AvgLuma_next_ && (AvgLuma_prev_>max_stop || AvgLuma_next_>max_stop)) ? 1 : 0
LeftZero = (AvgLuma_next_*MaxMulty<AvgLuma_prev_ && (AvgLuma_prev_>max_stop || AvgLuma_next_>max_stop)) ? 1 : 0
drop = (AvgLuma==AvgDrop && AvgLuma_next==AvgDrop) ? -50 : 0
drop = (AvgLuma==AvgDrop && AvgLuma_prev==AvgDrop) ? 50 : drop
drop = (drop==0 && AvgLuma==AvgDrop && LeftZero==0 && RightZero==0) ? \
AvgLuma_prev_>AvgLuma_next_*dblMulti ? -50 : \
AvgLuma_prev_*dblMulti>=AvgLuma_next_ ? AvgLuma_next2==AvgDrop ? 50 : 33 : 50 : drop
drop = (drop==0 && AvgLuma==AvgDrop && RightZero==0 && AvgLuma_next2==AvgDrop) ? -50 : drop
drop = (drop==0 && AvgLuma==AvgDrop && LeftZero==0 && AvgLuma_prev2==AvgDrop) ? 66 : drop
(drop==50) ? fix_r50 : \
(drop==33) ? fix_lr33 : \
(drop==-50) ? fix_l50 : \
(drop==133) ? fix_r133 : \
(drop==66) ? fix_r66 : drop_clip
Return Last
}
# Local arg names for constants import
fix_r50 = luma_lft.mt_lut(y=-1, u=128, v=128)
fix_lr33 = luma_lft.mt_lut(y=-2, u=128, v=128)
fix_l50 = luma_lft.mt_lut(y=-3, u=128, v=128)
fix_r133 = luma_lft.mt_lut(y=-4, u=128, v=128)
fix_r66 = luma_lft.mt_lut(y=-5, u=128, v=128)
AvgDrop = 127.0
MaxMulty = 10.0
dblMulti = 1.3
max_stop = 0.5
ARGS2 = "fix_r50,fix_lr33,fix_l50,fix_r133,fix_r66,AvgDrop,MaxMulty,dblMulti,max_stop"
fix_clip = drop_clip.GScriptClip("FnB@@@(last, "+ARGS2+")", local=true, args=ARGS2,after_frame=true)
################
Function FnC@@@(clip clp,clip fix_clip,clip fix50,clip fix33,clip fix66,clip fix66p,clip fix75,clip fix50n,clip fix133,clip fix125,
\ Int r50,Int lr33,Int l50,Int r133,Int r66, fix75_prv,fix66_prv,fix50n_prv,fix125_nxt,fix133_nxt) {
clp n = current_frame
AvgFix_prev = fix_clip.RT_AverageLuma(n=n,delta = -1)
AvgFix = fix_clip.RT_AverageLuma(n=n,delta = 0)
AvgFix_next = fix_clip.RT_AverageLuma(n=n,delta = 1)
AvgFix_next2 = fix_clip.RT_AverageLuma(n=n,delta = 2)
AvgFix_next3 = fix_clip.RT_AverageLuma(n=n,delta = 3)
/*
(AvgFix==r50) \
? (AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix33 : fix50 : \
(AvgFix==lr33) ? fix33 : \
(AvgFix_next==lr33) \
? (AvgFix_prev==r66) ? fix75.trim(1,1)+fix75 : fix66.trim(1,1)+fix66 : \
(AvgFix_next==l50) \
? (AvgFix_prev==lr33 || AvgFix_prev==r50) ? fix66.trim(1,1)+fix66 : \
(AvgFix_prev==r66) ? fix75.trim(1,1)+fix75 : fix50n.trim(1,1)+fix50n : \
(AvgFix==r66) \
? (AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix50 : fix66p : \
(AvgFix_next==r66) \
? (AvgFix_next3==lr33 || AvgFix_next3==l50) ? fix125.trim(1,0) : fix133.trim(1,0) : \
last
*/
if(AvgFix==r50) {
(AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix33 : fix50
} else if(AvgFix==lr33) {
fix33
} else if(AvgFix_next==lr33) {
(AvgFix_prev==r66) ? fix75_prv : fix66_prv
} else if(AvgFix_next==l50) {
if(AvgFix_prev==lr33 || AvgFix_prev==r50) {
fix66_prv
} else if(AvgFix_prev==r66) {
fix75_prv
} else {
fix50n_prv
}
}else if(AvgFix==r66) {
(AvgFix_next2==lr33 || AvgFix_next2==l50) ? fix50 : fix66p
} else if(AvgFix_next==r66) {
(AvgFix_next3==lr33 || AvgFix_next3==l50) ? fix125_nxt : fix133_nxt
} else {
Last
}
}
# Local arg names for constants import
fix50 = fix_all.SelectEvery(12,6)
fix33 = fix_all.SelectEvery(12,4)
fix66 = fix_all.SelectEvery(12,8)
fix66p = fix_all.SelectEvery(12,8)
fix75 = fix_all.SelectEvery(12,9)
fix50n = fix_all.SelectEvery(12,6)
fix133 = fix_all.SelectEvery(12,4)
fix125 = fix_all.SelectEvery(12,3)
if(myDebug) { # These are for visual debugging (the "Subtitle" clauses)
fix50 = fix50.Subtitle("fix50", align=3, size=myErrSize)
fix33 = fix33.Subtitle("fix33", align=3, size=myErrSize)
fix66 = fix66.Subtitle("-fix33", align=3, size=myErrSize)
fix66p = fix66p.Subtitle("fix66", align=3, size=myErrSize)
fix75 = fix75.Subtitle("-fix25", align=3, size=myErrSize)
fix50n = fix50n.Subtitle("-fix50", align=3, size=myErrSize)
fix133 = fix133.Subtitle("fix133", align=3, size=myErrSize)
fix125 = fix125.Subtitle("fix125", align=3, size=myErrSize)
}
fix75_prv = fix75.Trim(0,-1)++fix75
fix66_prv = fix66.Trim(0,-1)++fix66
fix50n_prv = fix50n.Trim(0,-1)++fix50n
fix125_nxt = fix125.Trim(1,0)
fix133_nxt = fix133.Trim(1,0)
r50 = 1
lr33 = 2
l50 = 3
r133 = 4
r66 = 5
ARGS3 = "fix_clip,fix50,fix33,fix66,fix66p,fix75,fix50n,fix133,fix125,r50,lr33,l50,r133,r66, fix75_prv,fix66_prv,fix50n_prv,fix125_nxt,fix133_nxt"
Last.GScriptClip("FnC@@@(last, "+ARGS3+")", local=false, args=ARGS3,after_frame=true)
""") ### End of Func_S
#######################################
# Unique Identifier Definition
#######################################
GIFunc ="GDF" # Function Name, Supply unique name for your multi-instance function.
GIName =GIFunc+"_InstanceNumber" # Name of the Instance number Global
RT_IncrGlobal(GIName) # Increment Instance Global (init to 1 if not already exists)
GID = GIFunc + "_" + String(Eval(GIName))
Func_S = RT_StrReplace(Func_S,"@@@","_"+GID)
# RT_WriteFile("DEBUG"+GID+".TXT","%s",Func_S) # Uncomment to output DEBUG text to file (with same line numbers as GSCript errors)
Last = GScript(Func_S)
(MtMode>0) ? SetMTMode(MtMode) : NOP
trim(0,framecount-threads)
}
The stuff in BLUE is original code, not sure if I did conversion to GScript correctly (I have my own way of using ternary conditional [(condition)?:], and
the way it was coded was back-to-front to how I would have done it)
Ive done as Multi-Instance script, real pig to debug the 1st 2 scriptclips as results are so tiny (and cannot read error message).
Anyway, now that its GSCript capable, author can possibly improve logic, I did not want to change too much as I had no idea
what the intent was, (not even sure what its supposed to do).
Anyways, needs GScript, Grunt and RT_Stats.
Let us know how it goes, if it works OK.
StainlessS
3rd January 2016, 21:51
This would not fit in previous post, also can add two functions from post before previous one to MI script.
Original script
AVSMeter 2.0.7 (x86)
AviSynth 2.60 (ICL10) (2.6.0.6)
Number of frames: 5000
Length (hh:mm:ss.ms): 00:03:28.542
Frame width: 720
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YV12
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 2.687 | 243.1 | 20.14
Memory usage (phys | virt): 283 | 298 MB
Thread count: 3
CPU usage (average): 51%
Time (elapsed): 00:04:08.270
MI script
AVSMeter 2.0.7 (x86)
AviSynth 2.60 (ICL10) (2.6.0.6)
Number of frames: 5000
Length (hh:mm:ss.ms): 00:03:28.542
Frame width: 720
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YV12
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 4.558 | 863.1 | 27.41
Memory usage (phys | virt): 212 | 228 MB
Thread count: 3
CPU usage (average): 52%
Time (elapsed): 00:03:02.435
EDIT: I guess that the MI function did not really need to be multi-instance, but I'de already done it by the time I realized it was not required.
All the original global vars were not necessary when using argument name importing into GScriptclip() [courtesy of Grunt].
EDIT: These are the only two google hits I get for GameDropFix (apart from this thread)
1) http://www.digital-video-processing.com/forum/topic-35237-page-23.html
2) via Google Translate (Russian original)
http://translate.google.com/translate?tl=en-us&u=http%3A%2F%2Fwww.svp-team.com%2Fforum%2Fviewtopic.php%3Fid%3D914%26p%3D6
EDIT: It apparently comes with XviD4PSP5:- http://www.google.co.uk/?gfe_rd=cr&ei=KRd1VKG6N5HCVND3gGA&gws_rd=ssl#q=XviD4PSP5
StainlessS
4th January 2016, 01:54
Post #21 update.
Added n=n type args to RT_ function calls (instead of implicit current_frame use). Small speed improvement.
Added Explicit Last to some RT_ function calls (avoid try with explicit, then retry with implicit Last). Speed improvement.
Added pre-prepared Previous and Next clips for Function 3 (small speed reduction), might be faster on clips where the function actually does something.
EDIT: I gave the numeric constants names for import into Functions via Grunt's GScriptclip(), it would be faster to change to
numeric constants within the functions (due to being handled by the parser instead of getting from Local Variables NameTable),
but decided that it might be useful to be able to change all instances of the constants from outside the functions, also sort of
self documenting.
Revised timing on update
AVSMeter 2.0.7 (x86)
AviSynth 2.60 (ICL10) (2.6.0.6)
Number of frames: 5000
Length (hh:mm:ss.ms): 00:03:28.542
Frame width: 720
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YV12
Frames processed: 5000 (0 - 4999)
FPS (min | max | average): 5.186 | 1114 | 36.13
Memory usage (phys | virt): 212 | 228 MB
Thread count: 3
CPU usage (average): 52%
Time (elapsed): 00:02:18.384
EDIT: Here timing for different clip (live video, Cabaret from DVD, RoboCropp'ed, to Lossless UT_Video)
AVSMeter 2.0.7 (x86)
AviSynth 2.60 (ICL10) (2.6.0.6)
Number of frames: 177640
Length (hh:mm:ss.ms): 01:58:25.600
Frame width: 668
Frame height: 396
Framerate: 25.000 (25/1)
Colorspace: YV12
Frames processed: 100981 (0 - 100980)
FPS (min | max | average): 3.145 | 42.59 | 33.48
Memory usage (phys | virt): 338 | 353 MB
Thread count: 3
CPU usage (average): 50%
Time (elapsed): 00:50:15.786
Got bored after 100,000 frames, Virt Memory increases by 1MB, about every 73 seconds (~2,500 frames, about 400 bytes per frame),
speed reduction of ~0.01 FPS every 18 or so seconds.
Why do speed reductions over time, seem to be in-vogue of late ?
ScriptClip uses up memory in Avisynth's string table for each frame rendered.
Oh why oh why, Darn it :( [perhaps also a contributing factor to slowdown ?]
Gavino
4th January 2016, 10:22
Oh why oh why, Darn it :(
Because the run-time script is parsed again for each frame and the strings from the previous frame are not freed. (Avisynth has no interface to clear the string table.)
A workaround is to reduce the run-time script size by using functions (see post #19).
StainlessS
4th January 2016, 19:53
A workaround is to reduce the run-time script size by using functions.
Thank you big G, if you see post #21, that is exactly what I am doing using the Multi-Instance script function model knocked up by Martin53 and your good self. (also tends to up speed a little bit). Thanks so very much to both of you, alas, we dont see enough of either of you of late :(
TCmullet
15th January 2016, 03:11
These are the only two google hits I get for GameDropFix (apart from this thread)
Thanks, guys, for all of y'alls efforts to help with this. Actually, GameDropFix originated near the beginning of this huge thread on a different forum:
http://www.svp-team.com/forum/viewtopic.php?pid=56182#p56182
I piggybacked onto the old thread months ago, then added to it very recently concerning this memory problem. (I doubt anyone will want to read the WHOLE thread there unless it's desired to learn the history of GameDropFix.) I gave the expert over there a link to this thread. I haven't been able to test the things that you all have done. (Sorry.) I'm hoping he will be able to modify GameDropFix as per the ScriptClip info embedded here. (Particularly the 2011 thread explaining about keeping the ScriptClip argument VERY small by making the passed code into a function that is outside of ScriptClip.) It appears that Avisynth has no way to reclaim memory used by ScriptClip, so minimizing the garbage it leaves behind appears to be the thing we need to do. (As well as modifying Virtuadub 32-bit to use more than 2GB of ram as a task.)
StainlessS
15th January 2016, 15:13
Gee TC,
The MI (Multi-Instance) script in post #21, already saves about as much memory as is possible, but your expert over there can probably
improve logic using GScript 'If' etc.
~400 bytes per frame, for 200,000 frames is about 80MB (would be same for HD) so memory should no longer be a problem.
EDIT:
Frames processed: 100981 (0 - 100980)
FPS (min | max | average): 3.145 | 42.59 | 33.48
Memory usage (phys | virt): 338 | 353 MB
Thread count: 3
CPU usage (average): 50%
As can be seen above, 353 MB after 100,000 frames, but that includes other Avisynth memory usage, not just string memory used.
After another 100,00 frames (ie ~200,000), would be about 393MB virtual memory used.
vBulletin® v3.8.11, Copyright ©2000-2026, vBulletin Solutions Inc.