jriker1
18th December 2014, 23:53
I've been testing some scripts and came to an odd realization and wondering why this is. Couple things. One the timing discrepancy and second the file sizes. Tried this on five identially lengthed clips and same deal. This script is to cleanup a VHS transfer. If there is any value to it perhaps someone can use it also:
AVISource("Clips1.avi")
AssumeTFF()
QTGMC(Preset="Slower", SourceMatch=3, Lossless=2, MatchEnhance=0.75, TR2=1)
mdata= DePanEstimate(trust=1.0,dxmax=4,dymax=20,range=1,pixaspect=1)
DePanStabilize(data=mdata,cutoff=0.2,dxmax=4,dymax=12,method=1,mirror=0,pixaspect=1)
Cnr2("xoo",4,2,64)
DePulse(h=70, l=1, D=15, debug=false)
DePulse(h=10, l=800, d=5, debug=false)
ConverttoYV12()
# remove flicker by Didée
rest=last
calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)#.repair(rest,1).removegrain(11)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = rest.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)
crop(10,4,-2,-10) #left,top,right,bottom
Devcr(30)
ConverttoYUY2()
function DetectVCRLines(clip c,int threshold)
{
spacial_data = GeneralConvolution(ConvertToRGB(c),0,"0 -1 0 0 2 0 0 -1 0")
bar_data = ConvertToRGB(BilinearResize(spacial_data,16,c.height))
st_data = Overlay(bar_data,Trim(bar_data,1,0),mode = "subtract")
st_data2 = Greyscale(Levels(st_data,threshold,10.0,threshold+1,0,255,coring = false))
st_data3 = Greyscale(Levels(st_data2,127,10.0,128,0,255,coring = false))
st_data4 = Overlay(st_data3,st_data3,y = -1, mode = "add")
return PointResize(st_data4,c.width,c.height)
}
function deVCR(clip c,int threshold)
{
mybars = DetectVCRLines(c,threshold)
return Overlay(c,Trim(c,1,0), mask = mybars,greymask = true)
}
I also add in NeatVideo within VDub so am running in VirtualDub in full processing mode. So the challenge here, if you run this as is, it will run at 0.04fps. I've tried this on two systems, one a lot more powerful, and identical render speed. For a 45 minute clip or so it would take weeks to finish. OK I said this ain't gonna work. Well let's try something. Let's do this in pieces and see where the slowdown is.
Start with the below and NeatVideo running in VDub full processing mode:
AVISource("Clips1.avi")
AssumeTFF()
QTGMC(Preset="Slower", SourceMatch=3, Lossless=2, MatchEnhance=0.75, TR2=1)
OK that finished in 3 hours, not to bad. Definitely not the bad process. Clip is about 10.5GB. All doing in Lagarith.
Well what about the next part. Take the resulting AVI from the last one and load it into the next script. Do that fast recompress in VDub:
AVISource("Clip1sb.avi")
mdata= DePanEstimate(trust=1.0,dxmax=4,dymax=20,range=1,pixaspect=1)
DePanStabilize(data=mdata,cutoff=0.2,dxmax=4,dymax=12,method=1,mirror=0,pixaspect=1)
OK that also took a bit under a few hours. Not the bad process. Clip is slightly smaller for some reason at 10.3GB.
Next part also fast recompress taking the resulting AVI from the last one and load it into the next script:
AVISource("Clip1sc.avi")
Cnr2("xoo",4,2,64)
DePulse(h=70, l=1, D=15, debug=false)
DePulse(h=10, l=800, d=5, debug=false)
Ok that took about 35 minutes. Not the problem. File slightly bigger again for some reason. Back to 10.5GB
Next part taking the resulting AVI from the last one and load it into the next script.
AVISource("Clip1sd.avi")
ConverttoYV12()
# remove flicker by Didée
rest=last
calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)#.repair(rest,1).removegrain(11)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = rest.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)
ConverttoYUY2()
OK that took about 35 minutes also. Not the problem. File is a lot smaller for some reason. 7.9GB. Huh. I assume it's doing cleanup that neatvideo already did though not sure why for a well known deflicker script, not a denoise script per sea or perhaps it is.
Well this part has to be it as nothing is adding up to those durations. Taking the resulting AVI from the last one and load it into the next script:
AVISource("Clip1se.avi")
crop(10,4,-2,-10) #left,top,right,bottom
Devcr(30)
ConverttoYUY2()
function DetectVCRLines(clip c,int threshold)
{
spacial_data = GeneralConvolution(ConvertToRGB(c),0,"0 -1 0 0 2 0 0 -1 0")
bar_data = ConvertToRGB(BilinearResize(spacial_data,16,c.height))
st_data = Overlay(bar_data,Trim(bar_data,1,0),mode = "subtract")
st_data2 = Greyscale(Levels(st_data,threshold,10.0,threshold+1,0,255,coring = false))
st_data3 = Greyscale(Levels(st_data2,127,10.0,128,0,255,coring = false))
st_data4 = Overlay(st_data3,st_data3,y = -1, mode = "add")
return PointResize(st_data4,c.width,c.height)
}
function deVCR(clip c,int threshold)
{
mybars = DetectVCRLines(c,threshold)
return Overlay(c,Trim(c,1,0), mask = mybars,greymask = true)
}
Darn, also like 35 minutes or so. OK none of that adds up to weeks and weeks or processing. File again slightly smaller than the last 7.5GB.
Does anyone know why if I process this script in it's entirety why it takes weeks and if I break it up into separate segments it takes part of a day? Also any idea why the file size changes during each step?
Thanks.
JR
AVISource("Clips1.avi")
AssumeTFF()
QTGMC(Preset="Slower", SourceMatch=3, Lossless=2, MatchEnhance=0.75, TR2=1)
mdata= DePanEstimate(trust=1.0,dxmax=4,dymax=20,range=1,pixaspect=1)
DePanStabilize(data=mdata,cutoff=0.2,dxmax=4,dymax=12,method=1,mirror=0,pixaspect=1)
Cnr2("xoo",4,2,64)
DePulse(h=70, l=1, D=15, debug=false)
DePulse(h=10, l=800, d=5, debug=false)
ConverttoYV12()
# remove flicker by Didée
rest=last
calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)#.repair(rest,1).removegrain(11)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = rest.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)
crop(10,4,-2,-10) #left,top,right,bottom
Devcr(30)
ConverttoYUY2()
function DetectVCRLines(clip c,int threshold)
{
spacial_data = GeneralConvolution(ConvertToRGB(c),0,"0 -1 0 0 2 0 0 -1 0")
bar_data = ConvertToRGB(BilinearResize(spacial_data,16,c.height))
st_data = Overlay(bar_data,Trim(bar_data,1,0),mode = "subtract")
st_data2 = Greyscale(Levels(st_data,threshold,10.0,threshold+1,0,255,coring = false))
st_data3 = Greyscale(Levels(st_data2,127,10.0,128,0,255,coring = false))
st_data4 = Overlay(st_data3,st_data3,y = -1, mode = "add")
return PointResize(st_data4,c.width,c.height)
}
function deVCR(clip c,int threshold)
{
mybars = DetectVCRLines(c,threshold)
return Overlay(c,Trim(c,1,0), mask = mybars,greymask = true)
}
I also add in NeatVideo within VDub so am running in VirtualDub in full processing mode. So the challenge here, if you run this as is, it will run at 0.04fps. I've tried this on two systems, one a lot more powerful, and identical render speed. For a 45 minute clip or so it would take weeks to finish. OK I said this ain't gonna work. Well let's try something. Let's do this in pieces and see where the slowdown is.
Start with the below and NeatVideo running in VDub full processing mode:
AVISource("Clips1.avi")
AssumeTFF()
QTGMC(Preset="Slower", SourceMatch=3, Lossless=2, MatchEnhance=0.75, TR2=1)
OK that finished in 3 hours, not to bad. Definitely not the bad process. Clip is about 10.5GB. All doing in Lagarith.
Well what about the next part. Take the resulting AVI from the last one and load it into the next script. Do that fast recompress in VDub:
AVISource("Clip1sb.avi")
mdata= DePanEstimate(trust=1.0,dxmax=4,dymax=20,range=1,pixaspect=1)
DePanStabilize(data=mdata,cutoff=0.2,dxmax=4,dymax=12,method=1,mirror=0,pixaspect=1)
OK that also took a bit under a few hours. Not the bad process. Clip is slightly smaller for some reason at 10.3GB.
Next part also fast recompress taking the resulting AVI from the last one and load it into the next script:
AVISource("Clip1sc.avi")
Cnr2("xoo",4,2,64)
DePulse(h=70, l=1, D=15, debug=false)
DePulse(h=10, l=800, d=5, debug=false)
Ok that took about 35 minutes. Not the problem. File slightly bigger again for some reason. Back to 10.5GB
Next part taking the resulting AVI from the last one and load it into the next script.
AVISource("Clip1sd.avi")
ConverttoYV12()
# remove flicker by Didée
rest=last
calm = rest.temporalsoften(1,255,255,32,2).merge(rest,0.25)#.repair(rest,1).removegrain(11)
calm = calm.temporalsoften(1,255,255,16,2).merge(calm,0.25)
LOP = calm.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = rest.bicubicresize(180,120).bicubicresize(640,480,1,0)
HIP = mt_makediff(rest,HIP)
mix = LOP.mt_adddiff(HIP,U=2,V=2)
sup1=rest.msuper(levels=1)
sup2=mix.removegrain(11).msuper()
bv2=sup2.manalyse(isb=true, delta=2,blksize=16,overlap=8,DCT=5)
bv1=sup2.manalyse(isb=true, delta=1,blksize=16,overlap=8,DCT=5)
fv1=sup2.manalyse(isb=false,delta=1,blksize=16,overlap=8,DCT=5)
fv2=sup2.manalyse(isb=false,delta=2,blksize=16,overlap=8,DCT=5)
rest.mdegrain2(sup1,bv1,fv1,bv2,fv2,thSAD=640)
ConverttoYUY2()
OK that took about 35 minutes also. Not the problem. File is a lot smaller for some reason. 7.9GB. Huh. I assume it's doing cleanup that neatvideo already did though not sure why for a well known deflicker script, not a denoise script per sea or perhaps it is.
Well this part has to be it as nothing is adding up to those durations. Taking the resulting AVI from the last one and load it into the next script:
AVISource("Clip1se.avi")
crop(10,4,-2,-10) #left,top,right,bottom
Devcr(30)
ConverttoYUY2()
function DetectVCRLines(clip c,int threshold)
{
spacial_data = GeneralConvolution(ConvertToRGB(c),0,"0 -1 0 0 2 0 0 -1 0")
bar_data = ConvertToRGB(BilinearResize(spacial_data,16,c.height))
st_data = Overlay(bar_data,Trim(bar_data,1,0),mode = "subtract")
st_data2 = Greyscale(Levels(st_data,threshold,10.0,threshold+1,0,255,coring = false))
st_data3 = Greyscale(Levels(st_data2,127,10.0,128,0,255,coring = false))
st_data4 = Overlay(st_data3,st_data3,y = -1, mode = "add")
return PointResize(st_data4,c.width,c.height)
}
function deVCR(clip c,int threshold)
{
mybars = DetectVCRLines(c,threshold)
return Overlay(c,Trim(c,1,0), mask = mybars,greymask = true)
}
Darn, also like 35 minutes or so. OK none of that adds up to weeks and weeks or processing. File again slightly smaller than the last 7.5GB.
Does anyone know why if I process this script in it's entirety why it takes weeks and if I break it up into separate segments it takes part of a day? Also any idea why the file size changes during each step?
Thanks.
JR