The power of Avisynth: restoring old 8mm films.

[StainlessS]

Also how should I add the GamMac script to VideoFred’s script? I am not sure how to use the FredAverage and without it, using VideoFred and GamMac’s script is resulting in noticeable color banding as mentioned elsewhere.


I'll try implement dithering.

[raffriff42]

A script like this (with many chained filters) would be an ideal candidate for porting to AVS+ at 10-bits or greater. I might be able to help, using whatever knowledge I picked up doing the (unfinished) documentation.

[johnmeyer]

I finally got around to adding GamMac to my version of VideoFred's script. I also got around to doing a lot of deferred housecleaning. As one example, I never use any of the manual controls because I do all those corrections in my NLE. So, I took them out.

I started to think about the order of the restoration steps and decided I could get a better result if I did degraining first, and then used the motion-compensated version of RemovedDirt second. This seems to produce a more pleasing result because with RemoveDirt first, I was getting too much correlation in the remaining grain, i.e., the image movement sometimes looked "gooey" as things moved around.

GamMac introduces a lot of flicker and I wasn't able to find any settings that would eliminate that. So, I moved Deflicker to the end of the restoration chain and, fortunately, it removes the flicker that GamMac introduces, as well as most flicker in the film itself. Hopefully I'll be able to get a better handle on GamMac over the coming weeks and perhaps eliminate that problem.

I thought about posting this in the thread that branched out from this one. That is where I posted my previous update. However, that thread has gone dormant, while this one is still active, so I'm posting here.

I am also including the "filldrops()" function. I find this incredibly useful. In my NLE (Vegas) I wrote a script that lets me, with the push of one button, replace any bad frame with a copy of the previous frame. This is useful for burned frames; huge dirt spots; bad registration errors during transfer (e.g., a frame that "jumps"); a photographer's flash; and anything else that is distracting or annoying that only lasts for one single frame. Since each duplicate frame I create in Vegas is a 100% perfect duplicate, it is simple for AVISynth to detect that dup, and it is easy to create a new frame by motion estimating from the surrounding frames. Any frame that is not a duplicate is passed through (which is 99.99% of all frames) so the function has almost no speed penalty because most frames are not altered. Therefore, most of the new interpolated frames are absolutely perfect and cannot be detected. Quite magic, actually.

So, I've put the script into the next two posts. Hopefully people will find it to be a slight improvement over what I've posted in the past.

[johnmeyer]

# Film restoration script by videoFred.
# Denoising, resizing, stabilising, sharpening, auto-levels and auto-white balance.
#
# Modified by John Meyer, October 7, 2012
# Modified by John Meyer, July 7, 2015 (remove errors)
# Modified by John Meyer, September 13, 2017
#
# Changes made by Meyer (June 8, 2010):
# 1. Replaced the MVDegrainMulti function, which was part of the original MVTools
# with the equivalent function using the newer and faster MVTools2.
# 2. Added RemoveDirt function prior to Degrain in order to eliminate large dust spots. This
# substantially improves the dirt removal capabilities of this script. The user will need to
# download this plugin at:
# http://www.removedirt.de.tf/
# 3. Eliminated a great deal of sharpening. The original script did sharpening in at least four places.
# With grainy film stock, this sometimes created objectionable grain. Also, the limitedSharpenFaster
# function, while excellent for video, is uncessarily slow, and isn't really needed for this low
# resolution source. Too much sharpening can make the film look too much like video.
# 4. Added multithreading. I was able to roughly triple the script performance. It may be possible to
# substantially increase this, perhaps as much as 12x instead of 3x. However, the autolevels function
# would have to be replaced.
# 5. Fixed several things I found. The result3 option didn't have the manual color correction code, so I
# added that back in. All the numbered "stab" variables (stab1, stab2, etc.) aren't needed except
# for providing a test function for stabilization. I eliminated all of this to streamline the script.
# 6. Deflicker seemed redundant, given all the averaging that takes place with MDegrain, and also the averaging
# that is done when the autolevels outputs are selected.
# 7. I reduced the number of frames used for averaging autolevels from the default (which is 5) to 2.
# I probably should add a variable in the header so the user can change this. Something else to do
# in the future ...
# 8. I added yet another set of crop parameters. I did this because both my capture and my output are
# done using NTSC DV AVI which is 720x480. However, 8mm film is almost exactly square, so the captured
# 720x480 video has black bars on the side. These need to be cropped off prior to doing motion
# stabilization, but then added back prior to the final output, which must still be 720x480 with the
# black bars on the side.
# 9. I reduced the default depan settings to 20, which is what was recommended in the original script. I
# also reduced the post-depan cropping. I did this so I could keep as much of the original frame
# as possible.
# 10. I removed the second denoising and sharpening function. It just seemed to be too much, and made the
# result too artificial
# 11. I removed the MVFLowFPS interpolation. I did this during one of dozens and dozens of attempts
# to improve the speed of the script. I should probably add this back, but if I do so, I also
# need to make it work correctly for interlaced output. If the goal is to show this on an NTSC or PAL
# television set, then it is not correct to convert from the film fps to 25 fps progressive (PAL) or
# 29.97 fps progressive (NTSC). Instead, this should be done as follows (example given is NTSC):
#
# MFlowFPS(source,super,backward_vec, forward_vec, num=60000, den=1001,ml=200)
# SeparateFields()
# SelectEvery(4, 0, 3)
# Weave()
#
# This yields interlaced 29.97, which has twice the temporal resolution as 29.97 progressive, and will
# therefore look correct on a TV set. I have done a lot of this as part of my Kinescope to video
# conversion scripts. For those scripts, the goal is to make the filmed version of a TV show look
# like it was actually videotaped. However, for something that originated on film, this "does violence"
# to the original feel of the media. It is true that it can make horizontal pans less "juddery," but
# it won't feel like film anymore. Also, this technique does break down, espcially with fast motion
# in the foreground.
# 12. Removed the unecessary "coloryuv(off_U=blue,off_V=red)" statement from the denoising section.
# 13. Added killaudio() statement to prevent lockups when using SetMTMode().
#
# Changes made on October 7, 2012
#
# 1. Modified RemoveDirtMC function to do estimation in two steps. This improves performance and quality.
# 2. Modified MDegrain to do estimation in two steps. This provides roughly the same quality with blocksize=8
# as what I was getting with blocksize=4, but with virtually no performance penalty
# 3. Updated to use special build of MVTools2 that works with SVP.
# 4. Moved Autolevels inside special multi-threading section in order to improve performance with autolevels.
# 5. Removed all trim commands because I do all editing outside the script (in Vegas).
# 6. Included two alternative ways to do frame interpolation: SVP and InterFrame. The InterFrame script is probably better
# but I included the SVP code because it may provide a starting point for more customized approaches. Neither is enabled by default.
#
# Changes made on September 13, 2017
#
# 1. Replaced old useless color correction with GamMac.
# 2. Removed never-used manual controls.
# 3. Moved deflicker to end of chain because GamMac introduces flicker.
# 4. Moved RemoveDirtMC to happen AFTER denoising.

[johnmeyer]

#====================================================================================
#7/12/2021 Fixed Mrecalculate errors
#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------
#Change the following line to point to your video file
film="e:\fs.avi"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="resultS2" #specify the wanted output here
play_speed=16 #play speed (8mm=16; Super8=18; 16mm sound=24)


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation = 1.0 #for all outputs
gamma = 1.0 #for all outputs

#GamMac Parameters
LockChan = 0 #(0=red channel)
Scale = 1
RedMul = 1.0
GrnMul = 1.0
BluMul = 1.0
LockVal = 128.0
Th = 0.1


#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=8 CTop=8 CRight=8 CBottom=8 #crop values after Depan and before final resizing
W=720 H=480 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=10 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=10
est_left=40 est_top=40 est_right=40 est_bottom=40 #crop and contast values for special Estimate clip
est_cont=1.6


#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
denoising_strength=600 #denoising level of first denoiser: MVDegrain()
block_size= 16 #block size of MVDegrain
block_size_v= 16
block_over= 8 #block overlapping of MVDegrainMulti()

#Alternate denoising using smaller block size
#denoising_strength= 600 #denoising level of first denoiser: MDegrain()
#block_size= 8 #block size of MVDegrain
#block_size_v= 8
#block_over= 4 #block overlapping of MVDegrainMulti()

#Alternate denoising using smalles block size
#denoising_strength= 600 #denoising level of first denoiser: MDegrain()
#block_size= 4 #block size of MVDegrain
#block_size_v= 4
#block_over= 2 #block overlapping of MVDegrainMulti()

dirt_strength=18 #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
#120 is pretty aggressive; 85 seems more subtle
#PRE_sharp_ness= 120 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
PRE_sharp_ness= 85 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising

Sharp_Strength = 0.1 #0.05 is often better (avoid too much sharpening)


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4 #X parameter for reducing the autolevels effect on the whites
X2=4 #X2 parameter for reducing the autolevels effect on the blacks


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=4


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

SetMemoryMax(768)

LoadPlugin ("mvtools2.dll") #Version 2.5.11.9 2/24/2012
LoadPlugin("autolevels.dll") #Version 0.6.0.0 1/09/2011
LoadPlugin("Deflicker.dll") #Version 0.4.0.0 8/16/2004
Loadplugin("Depan.dll") #Version 1.10.0.0 4/09/2007
LoadPlugin("DepanEstimate.dll") #Version 1.9.2.0 3/25/2007
LoadPlugin("fft3dfilter.dll") #Version 2.1.1.0 2/20/2007
Loadplugin("mt_masktools.dll") #Version 2.0.23.0 3/14/2008
loadplugin("RemoveDirtSSE2.dll") #Version 0.9 5/05/2005
Loadplugin("RemoveGrainSSE2.dll") #Version 0.9 5/01/2005
#Loadplugin("removegrain.dll") #Version 0.9 5/01/2005
Loadplugin("warpsharp.dll") # 4/05/2010

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll") #Version 1.2.1.0 5/29/2012
#loadplugin("svpflow2.dll") #Version 1.3.1.0 6/02/2012
#Import("InterFrame2.avsi") #Version 2.1.0 6/27/2012


#Remove all setmtmode statements (there are three in this script) if not using multi-threaded (MT) AVISynth
setmtmode(5,threads)
source1= Avisource(film).killaudio().assumefps(play_speed).converttoYV12()
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #remove any black borders on input video
setmtmode(2)

#cropped_source=filldrops(cropped_source) #Use this when removing bad frames that have been removed by duplicating previous frame


#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)
stabcrop=stab.crop(CLeft,CTop,-CRight,-CBottom)


#DENOISING
#...................................................................................................................................................................
prefiltered = RemoveGrain(stabcrop,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(superfilt, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(superfilt, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(superfilt, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

# Use these additional vectors if you enable three level denoising (below)
# bvec3 = MAnalyse(superfilt, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
# bvec3 = MRecalculate(super, bvec3, blksize=halfblksize, overlap=halfoverlap,thSAD=100)
# fvec3 = MAnalyse(superfilt, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
# fvec3 = MRecalculate(super, fvec3, blksize=halfblksize, overlap=halfoverlap,thSAD=100)


#Use the second line below, and un-comment the four lines above if you want to average more frames. Doesn't seem necessary for most work IMHO.
denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)

#Three level denoising
#denoised=stabcrop.MDegrain3(super, bvec1,fvec1,bvec2,fvec2,bvec3,fvec3,thSAD=denoising_strength).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)

#Remove Dirt
denoised = RemoveDirtMC(denoised,dirt_strength,false)

#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)

PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)

#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................
#Everything in this section is commented out because I usually choose NOT to do frame interpolation because
#it causes too many unexpected artifacts on certain frames. It also makes the film look like video which
#aesthetically is not always pleasing.

/*
prefiltered = RemoveGrain(PreBorderFrame,2)
super = MSuper(PreBorderFrame,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=8,search=3,dct=0)
forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=8,search=3,dct=0)
forward_re = MRecalculate(super, forward, blksize=8, thSAD=100)
backward_re = MRecalculate(super, backward, blksize=8, thSAD=100)
PreBorderFrame = MFlowFps(PreBorderFrame,super, backward_re, forward_re, num=30000, den=1001,ml=200,mask=2)
*/

#Alternative interpolation
/*
superfps= MSuper(PreBorderFrame,pel=2)
backward_vec2 = MAnalyse(superfps, isb = true,blksize=16)
forward_vec2 = MAnalyse(superfps, isb = false,blksize=16)
PreBorderFrame = MFlowFps(PreBorderFrame,superfps, backward_vec2, forward_vec2, num=60000, den=1001, ml=200).SeparateFields().SelectEvery(4, 0, 3).Weave()
*/

#Yet another way to do interpolation
#The following alternative can provide synthesized frames for hi-def material that causes MSuper to choke
#////// Begin alternative to MFlowFPS ////
#SmoothFPS2(PreBorderFrame,threads)
#////// End alternative to MFlowFPS ////

#A final way to do interpolation, although Interframe is just a front-end to SmoothFPS, so I don't think the results are any different.
#The following is another alternative that can provide synthesized frames for hi-def material that causes MSuper to choke. Probably better than the one above.
#////// Begin alternative to MFlowFPS ////
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)
#////// End alternative to MFlowFPS ////

#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
result1= PreBorderFrame.ConvertToRGB24.GamMac(Show=False, LockChan=LockChan, Th=Th, LockVal=LockVal, Scale=Scale, RedMul=RedMul, GrnMul=GrnMul, BluMul=BluMul).converttoYV12().deflicker().addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT2: AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.ConvertToRGB24.GamMac(Show=False, LockChan=LockChan, Th=Th, LockVal=LockVal, Scale=Scale, RedMul=RedMul, GrnMul=GrnMul, BluMul=BluMul).converttoYV12().deflicker().addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS
#.....................................................................................................................................................................
result3= PreBorderFrame.addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: NO CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"no level or color corrections",size=28,align=2))

Eval(result)


# END SCRIPT, BEGIN FUNCTIONS

[johnmeyer]

#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int "limit", bool "_grey")
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

# Alternative way to do pre-filtering.
# prefiltered = fft3dfilter(clip,sigma=1,sigma2=2,sigma3=3,sigma4=5,bw=64,bh=64)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)

super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

# Increase thSAD if moving objects are being removed
bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

#This function is the previous way to do motion estimation. It is slower and not as accurate.
function RemoveDirtMC_old(clip,int limit, bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)
i=MSuper(clip,pel=2)
fvec = MAnalyse(i,isb=false, blksize=16, delta=1, truemotion=true)
bvec = MAnalyse(i,isb=true, blksize=16, delta=1, truemotion=true)
backw = MFlow(clip,i,bvec)
forw = MFlow(clip,i,fvec)
clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

# This function will remove near duplicates ("0.1") or exact duplicates (change to "0.0"). It replaces these
# duplicates with a motion estimated frame.
# Use: In your video editor, replace any single bad frame (burned frame, jump, missing frame from splice, etc.) with a duplicate of the
# previous frame. Then, include a call to this function

function filldrops (clip c)
{
super=MSuper(c,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
filldrops = mflowinter(c,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
return fixed
}


function SmoothFPS2(clip source, threads) {
super_params="{pel:2,gpu:1}"
analyse_params="""{
block:{w:16,h:16},
main:{search:{coarse:{distance:-10}}},
refine:[{thsad:200}]
}"""
smoothfps_params="{rate:{num:30,den:16,abs:false},scene:{mode:0,limits:{scene:8500}},algo:21,cubic:1}"
threads = 4

super = SVSuper(source,super_params)
vectors = SVAnalyse(super, analyse_params)
SVSmoothFps(source,super, vectors, smoothfps_params, url="www.svp-team.com", mt=threads)

# Alternative for interlaced output
# SVSmoothFps(source,super, vectors, smoothfps_params, url="www.svp-team.com", mt=threads).SeparateFields().SelectEvery(4, 0, 3).Weave().assumefps(29.97)
}

[StainlessS]

John, scale = 2 (EDIT with Th=0) , in GamMac WILL see all pixels (even a single) as affecting min/max, avoid noise pixels by eg denoise on DC clip,
with a perfect source that may be what you want, with less than prefect, needs more effort.

@ Lem, GamMac never was intended to be any kind of one suit fits all, merely a pre tweak easy get in sync type of thing.
If we all had Lem at our sides, GamMac would be of little worth.
John, seek Lem's counsil, he seems to be the man :)

[bassquake]

I have a question regarding settings frame rate, as I did a quick search through this thread but couldnt find the info.

I've built a telecine to capture standard 8 and super 8 into separate images. I assume they were shot at 18fps and would like to create 24fps videos (true 24p). The comments in the script file confuses me when it mention 18.75fps. So do I set it like so (need to change denumerator?):

play_speed= 18.75
numerator= 24
denumerator= 1

or

play_speed= 18
numerator= 24
denumerator= 1

Help appreciated, thanks.

[johnmeyer]

"Play speed" is the speed that you think the original was played (e.g., 18 fps for Super 8).

I recommend doing standard pulldown rather than any sort of motion estimation. Despite heroic efforts to improve motion estimation, it fails way too many times and will give you a mountain of weird-looking frames. I am in the process now of doing four separate restorations of the same material. Two of the 2x2 matrix variations involve using, or not using GamMac, because it fails to do the right thing quite often. The other two variations involve generating an alternate to motion estimation which I used, not to change frame rate, but to fix hundreds of screwed-up frames. In 90% of the cases, the fix was near-perfect but, as is always true with ME, when it fails, it fails spectacularly and calls attention to itself. So, I am doing another set of GamMac and no GamMac with an alternative to ME that doesn't look as good, but never fails.

You can adapt this code to create pulldown for whatever you are doing:

#18p -> 60i

avisource("video.avi")
assumefps(18)
changefps(59.94)
separatefields()
selectevery(4,0,3) #depends on wished field-order
#selectevery(4,1,2) #alternate field order
weave()

[manono]

...and would like to create 24fps videos (true 24p).
Mind if I ask 'why'? Is it for DVD perhaps? Offhand I can't think of any other reason it has to be 24fps.

[johnmeyer]

After seeing manono's comment, I re-read his post, and he emphasized "true" 24p. I guess I need clarification of what that means. Forgetting for a moment about the 23.976 vs. 24.000 thing, if he really wants 24 unique frames each second, then motion interpolation is the only choice. That is built into both VideoFred's script, and in my adaptation of his script. I don't recommend it, however, because even if you use the masked version of Interframe that was recently developed, you'll still end up with a LOT of bad frames. The lower the frame rate (and 16 or 18 fps is pretty low), the bigger the temporal gap between frames, and the larger and more frequent these errors become. Motion estimation works really well on 60p material (for when you want to get super-slo-mo) because it doesn't have to move the pixels very far to make the tween frames, but it doesn't work that well on amateur silent film. I've posted the link to my Flint, MI parade many times which I converted from 16 to 30p before I uploaded, and it is a textbook case of what can go wrong.

So, if you want true 24p, ME is the only tool that I know of that will give you what you want, and the code is already posted in this thread.

[bassquake]

I want to output to bluray but I suppose I can use 23.976 if that's easier?

So would the code then be?:

play_speed= 18.75
numerator= 23.976
denumerator= 1001

Thanks for responses so far.

[johnmeyer]

I want to output to bluray but I suppose I can use 23.976 if that's easier?

So would the code then be?:

play_speed= 18.75
numerator= 23.976
denumerator= 1001

Thanks for responses so far.No, that is wrong. 23.976 = 24.000 / 1001.0 (make sure to use the decimal to force float)

But, as I said in my previous post, that is a relatively minor issue compared to the bigger point I was trying to make, and which I won't repeat again here.

[edit]Groucho2004 points out my error (a few posts below this one). I put the decimal in wrong place. I was only off by a factor of 1,000.
The correct equations are: 24.000 / 1.001 or 24000.0 / 1001.0

[StainlessS]

John dont overlook the DC clip, even a simple Blur() can avoid some of the flickering effect (so long as only spurious odd pixels are cause of flicker, shimmering water though would probably not benefit).

[manono]

Yes, interpolation to go from 18->24fps, but if that doesn't work out so well for the reasons johnmeyer outlined, there is another way that creates blended frames to fill in the gaps. The Motion.dll (http://web.archive.org/web/20060103143553/http://people.pwf.cam.ac.uk/mg262/posts/Motion/motion.html#MotionFPS), can do this to, for example, create 6 new frames every second (a new one after every 3) to go from 18->24fps. Each new frame will be a blend of the ones before and after. Ordinarily I don't like creating blended frames on purpose since I spend my life unblending frames but it might do here.

The reason for my question earlier was, if making a DVD is okay, you only have to bring the framerate up to 19.98fps and you can apply 3:3 pulldown afterwards to output the required 29.97fps. That way you'd have to create only a new frame after every 9, rather than after every 3. Only 10% of the frames would be interpolated. I've never worked with Blu-Ray and have no idea if soft pulldown is allowed.

[Groucho2004]

No, that is wrong. 23.976 = 24.000 / 1001.0 (make sure to use the decimal to force float)
That's also wrong. 23.976 = 24000 / 1001 (or 24000.0 / 1001.0).

ChangeFPS() takes either a float (23.976) or two integers (24000, 1001), see here (http://avisynth.nl/index.php/ChangeFPS). In order to avoid rounding errors it's recommended to use the (num, denom) format.

[johnmeyer]

That's also wrong. 23.976 = 24000 / 1001 (or 24000.0 / 1001.0).

ChangeFPS() takes either a float (23.976) or two integers (24000, 1001), see here (http://avisynth.nl/index.php/ChangeFPS). In order to avoid rounding errors it's recommended to use the (num, denum) format.Oops. You got me! That'll teach me to post too quickly. I forgot a decimal point.

[bassquake]

Hi. Is it possible to edit the avisynth script to use an image sequence rather than an avi file?

Thanks.

Edit: I ask because I'm not using a machine camera to output to an avi but am using a camera thats taking separate images. Rendering it to a huffy avi takes 2 hrs to do for a 2 minute film!

[manono]

Yes, you'd use ImageSource (http://avisynth.nl/index.php/ImageSource). I haven't read the script, but you might also have to change the colorspace. Others will know better than I.

[johnmeyer]

... and, of course, you can always assemble the image sequence into an AVI prior to feeding it to the script. Pretty much any NLE can do that, as can Virtualdub.

[Yanak]

... and, of course, you can always assemble the image sequence into an AVI prior to feeding it to the script. Pretty much any NLE can do that, as can Virtualdub.

Using the built in frame-server option of Virtualdub to feed avisynth directly and skip the need to write an .avi is a good option too yes ^^

[bassquake]

... and, of course, you can always assemble the image sequence into an AVI prior to feeding it to the script. Pretty much any NLE can do that, as can Virtualdub.

That's how I was doing it beforehand but it takes too long to render!

[bassquake]

Yes, you'd use ImageSource (http://avisynth.nl/index.php/ImageSource). I haven't read the script, but you might also have to change the colorspace. Others will know better than I.

Coolio, I got it working. Here's how to use an image sequence instead of avi:

Replace:

film= "C:\Users\You\Documents\Yourfile.avi" # source clip, you must specify the full path here

with:

film= "C:\Users\You\Documents\Images\SAM_%04d.jpg" #source clip, you must specify the full path here
start= 1 #Start frame
end= 3574 #Total number of frames

The %04d in the filename is the numbering padding used in the images. Mine is SAM_0001.jpg and never goes higher than 4 digits.

You need to tell it what frame it starts and ends with. Eg: Sequence is SAM_0003.jpg to SAM_0499.jpg then the start is 3 and end is 0499.

Also, replace the line:

source1= AviSource(film).assumefps(play_speed).trim(trim_begin,0).converttoYV12()

with:

source1= ImageSource(film, start, end).assumefps(play_speed).trim(trim_begin,0).converttoYV12().flipvertical()

Thats it. Can now load image sequences directly without having to render a lossless avi or similar to edit.

Note: I added flip vertical to the source1 line as my images are all upside down.

Hope that helps someone.

[camelopardis]

Hello all.

I'm very new to AVISynth, having just started trying to digitize my families old 8mm/Super8 films dating back to the 1950s.

I've bought a Reflecta Film Scanner super8/Normal8 (similar to the Somikon and Wolverine) and currently just experimenting, trying to learn the best way to scan and the best way to post-process.

Even though I realise the fantastic Videofred / John Meyer scripts on here are for the output of far superior scanning methods, I thought I'd give them a try.

I have a little test clip here (https://vimeo.com/255183448) showing a side-by-side result.

Input= Super 8 Reflecta scan at -1.0 exposure (scans as 1440x1080 30fps H.264 MP4)

converted to 18fps AVI using AVC2AVI

Output= latest 2012 VideoFred Script (default parameters)

I'm completely new to this but can see the output doesnt look right. I suspect the original scan is too heavily compressed?

Does anyone have any tips or suggestions? Should I use different parameters in the script - or a different method altogether?

Thank you - all or any advice most welcome :)

[johnmeyer]

It was tough for me to tell much from the Vimeo post. It actually looked pretty good.

I am not familiar with the Reflecta units, but a quick trip to their site makes it look like they must be frame accurate, meaning you get precisely one from of film on each frame of video. That is a requirement for using VideoFred's script.

I see you are storing the original scans using H.264. That is obviously a compressed format. You didn't provide any information about what settings you used. This is important because any video format that uses compression can use only a little compression (big files) or a LOT of compression (small files). The more the compression, the worse artifacts you will get. So, if artifacts are the problem, then look for a setting in your capture software that lets you use less compression. You can always compress the files when you are finished, in order that your final result doesn't require a huge array of disks to hold the final result. So, do the compression as the final step, if you need it, and not when you do the original capture.

You mention you are using a script dated 2012, but I thought VideoFred had posted something more recently than that. He has made some amazing advances over the past few years, some of which I still need to steal, er, incorporate into my own work, especially StainlessS' GamMac filter, which I still can't get to work reliably, but which shows great promise.

If you can upload a 10-second clip of one of your original transfers, without re-encoding, that will make it a lot easier to see what is going on.

[camelopardis]

It was tough for me to tell much from the Vimeo post. It actually looked pretty good.

I am not familiar with the Reflecta units, but a quick trip to their site makes it look like they must be frame accurate, meaning you get precisely one from of film on each frame of video. That is a requirement for using VideoFred's script.

I see you are storing the original scans using H.264. That is obviously a compressed format. You didn't provide any information about what settings you used. This is important because any video format that uses compression can use only a little compression (big files) or a LOT of compression (small files). The more the compression, the worse artifacts you will get. So, if artifacts are the problem, then look for a setting in your capture software that lets you use less compression. You can always compress the files when you are finished, in order that your final result doesn't require a huge array of disks to hold the final result. So, do the compression as the final step, if you need it, and not when you do the original capture.

You mention you are using a script dated 2012, but I thought VideoFred had posted something more recently than that. He has made some amazing advances over the past few years, some of which I still need to steal, er, incorporate into my own work, especially StainlessS' GamMac filter, which I still can't get to work reliably, but which shows great promise.

If you can upload a 10-second clip of one of your original transfers, without re-encoding, that will make it a lot easier to see what is going on.

Hi John - thank you very much for your reply!

yes - the Reflecta does scan frame-by-frame but unfortunately it can only save the result as a highly compressed MP4 file (this is the main complaint of most users).

Here is a longer clip of the raw 1440x1080 output from the machine (you can't change the compression ratio, you can only adjust exposure, sharpness, and frame adjustment).
I order to use with AVISynth, the above clip was changed from MP4 to AVI by first extracting the raw h.264 using "My MP4Box" and then converting to AVI using AVC2AVI. If any of this is silly or wrong please let me know. I'm learning as I go along and am more than happy to be corrected!

Raw Clip (https://vimeo.com/255255086)

Here is the same clip showing the output from Videofred's script. I used the latest 01_A script (posted on the first page of this thread (dated 20/06/2012). Is there a later one I should be using?

VideoFred output (https://vimeo.com/255263444)

Here is the actual script used:
# 8mm film restoration script by videoFred.
# www.super-8.be
# info@super-8.be

# version 01.A with frame interpolation
# release date: june 20, 2012
#============================================================================================

# august 2010: added removerdirtMC() as suggested by John Meyer
# october 2010: auto sharpening parameters

# march 2011: new autolevels.dll by Jim Battle
# www.thebattles.net/video/autolevels.html

# june 2012: improved stabilisation


#=============================================================================================

# cleaning, degraining, resizing, stabilizing, sharpening, auto-levels and auto-white balance.
#=============================================================================================


film= "C:\Users\You\Documents\Yourfile.avi" # source clip, you must specify the full path here






#PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="resultS1" # specify the wanted output here

trim_begin=2 trim_end=10 play_speed=18 #trim frames and play speed (PAL: 16.6666 or 18.75)

numerator= 25 #numerator for the interpolator (final frame rate)
denumerator= 1 #denumerator example: 60000/1001= 59.94fps


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=1.2 #for all outputs

gamma= 1.2 # for all outputs

blue= 0 red= 0 #manual color adjustment, when returning result3 or result4. Values can be positive or negative


black_level=0 white_level=255 output_black=0 output_white=255 # manual levels, when returning result4


#AUTO LEVELS PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------

autolev_low= 6 # limit of autolevels low output
autolev_high= 235 # limit of autolevels high output



#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=32 CTop=32 CRight=32 CBottom=32 #crop values after Depan and before final resizing

W=720 H=576 #final size after cropping

bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150


#STABILISING PARAMETERS, YOU REALY MUST USE RESULTS7 TO CHECK STABILISATION!
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20
maxstabV=20 #maximum values for the stabiliser (in pixels) 20 is a good start value

est_left=20 est_top=60 est_right=60 est_bottom=60 #crop values for special Estimate clip

trust_value= 1.0 # scene change detection, higher= more sensitive
cutoff_value= 0.5 # no need to change this, but you can play with it and see what you get





#CLEANING PARAMETERS
#--------------------------------------------------------------------------------------------------------------

dirt_strenght=30 # set this lower for clean films.


#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------


denoising_strenght= 300 #denoising level of second denoiser: MVDegrainMulti()
denoising_frames= 3 #number of frames for averaging (forwards and backwards) 3 is a good start value
block_size= 16 #block size of MVDegrainMulti()
block_size_v= 16
block_over= 8 #block overlapping of MVDegrainMulti()




# SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------

USM_sharp_ness= 40 USM_radi_us=3 #this is the start value for the unsharpmask sharpening
#do not set radius less then 3
#the script will automatically add two other steps with lower radius



last_sharp= 0.1 #final sharpening step after interpolation

last_blur= 0.2 #this smooths out the heavy sharpening effects





# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================


SetMemoryMax(800) #set this to 1/3 of the available memory




LoadPlugin("plugins/Deflicker.dll")
Loadplugin("plugins/Depan.dll")
LoadPlugin("plugins/DepanEstimate.dll")
Loadplugin("plugins/removegrain.dll")
LoadPlugin("plugins/removedirt.dll")
LoadPlugin("plugins/MVTools.dll")
LoadPlugin("plugins/MVTools2.dll")
Loadplugin("plugins/warpsharp.dll")
LoadPlugin("plugins/autolevels_06.dll")
Import("plugins/03_RemoveDirtMC.avs")






source= AviSource(film).assumefps(play_speed).trim(trim_begin,0).converttoYV12()
trimming= framecount(source)-trim_end
source1= trim(source,0,trimming)






#STABILIZING/CROPPING
#...........................................................................................................................................

stab_reference= source1.crop(20,20,-20,-20).colorYUV(autogain=true).crop(est_left,est_top,-est_right,-est_bottom)

mdata=DePanEstimate(stab_reference,trust=trust_value,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(source1,data=mdata,cutoff=cutoff_value,dxmax=maxstabH,dymax=maxstabV,method=0,mirror=15).deflicker()
stab2= stab.crop(CLeft,CTop,-CRight,-CBottom)
stab3=DePanStabilize(source1,data=mdata,cutoff=cutoff_value,dxmax=maxstabH,dymax=maxstabV,method=0,info=true)


WS= width(stab)
HS= height(stab)
stab4= stab3.addborders(10,10,10,10,$B1B1B1).Lanczos4Resize(WS,HS)
stab5= Lanczos4Resize(stab2,W,H).sharpen(0.5)


#UNSHARPMASK AUTO_PARAMETERS
#-------------------------------------------------------------------------------------------------------------------------------------------

USM_sharp_ness1 = USM_sharp_ness
USM_sharp_ness2 = USM_sharp_ness+(USM_sharp_ness/2)
USM_sharp_ness3 = USM_sharp_ness*2

USM_radi_us1 = USM_radi_us
USM_radi_us2 = USM_radi_us-1
USM_radi_us3 = USM_radi_us2-1


#CLEANING/PRESHARPENING/RESIZING
#..........................................................................................................................................


noise_baseclip= stab2.levels(0,gamma,255,0,255).tweak(sat=saturation)




cleaned= RemoveDirtMC(noise_baseclip,dirt_strenght).unsharpmask(USM_sharp_ness1,USM_radi_us1,0)\
.unsharpmask(USM_sharp_ness2,USM_radi_us2,0).Lanczos4Resize(W,H)



#DEGRAINING/SHARPENING
#...................................................................................................................................................................


vectors= cleaned.MVAnalyseMulti(refframes=denoising_frames, pel=2, blksize=block_size, blksizev= block_size_v, overlap=block_over, idx=1)
denoised= cleaned.MVDegrainMulti(vectors, thSAD=denoising_strenght, SadMode=1, idx=2).unsharpmask(USM_sharp_ness3,USM_radi_us3,0)






#CHANGING FRAME RATE WITH INTERPOLATION/FINALSHARPENING
#............................................................................................................................................................

super= denoised.MSuper()
backward_vec= MAnalyse(super, blksize=block_size, blksizev= block_size_v, overlap=block_over, isb=true)
forward_vec= MAnalyse(super,blksize=block_size, blksizev= block_size_v, overlap=block_over, isb= false)

interpolated= denoised.MFlowFps(super, backward_vec, forward_vec, num=numerator, den= denumerator, ml=100)\
.sharpen(last_sharp,mmx=false).sharpen(last_sharp,mmx=false).blur(last_blur,mmx=false)


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
result1= interpolated.converttoRGB24().autolevels(output_low= autolev_low, output_high= autolev_high)\
.converttoYV12().coloryuv(autowhite=true).addborders(bord_left, bord_top, bord_right, bord_bot)

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= interpolated.levels(black_level,1.0,white_level,0,255).coloryuv(autowhite=true)\
.addborders(bord_left, bord_top, bord_right, bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= interpolated.coloryuv(off_U=blue,off_V=red).converttoRGB24().autolevels(output_low= autolev_low, output_high= autolev_high)\
.converttoYV12().addborders(bord_left, bord_top, bord_right, bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= interpolated.coloryuv(off_U=blue,off_V=red).levels(black_level,1.0,white_level,0,255)\
.addborders(bord_left, bord_top, bord_right, bord_bot)

#RESULT5: SPECIAL SERVICE CLIP FOR RESULT S5
#.....................................................................................................................................................................
result5= overlay(source1,greyscale(stab_reference),x=est_left,y=est_top).addborders(2,2,2,2,$FFFFFF).Lanczos4Resize(WS,HS)




#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot




final_framerate= numerator/denumerator
source4=Lanczos4Resize(source1,W2,H2).changeFPS(final_framerate)



#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"result1: autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"result2: autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"result3: autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"result4: manual colors and levels correction",size=28,align=2))
resultS5= stackhorizontal(subtitle(result3,"result3: auto levels, manual color correction",size=28,align=2),subtitle(result4,"result4: manual colors and levels correction",size=28,align=2))
resultS6= stackhorizontal(subtitle(result1,"result1: autolevels, autowhite",size=28,align=2),subtitle(result2,"result2: manual levels, autowhite",size=28,align=2))

#SPECIAL COMPARISON CLIP FOR TESTING THE STABILIZER
#.........................................................................................................................................................................
resultS7= stackhorizontal(subtitle(result5,"baseclip for stabiliser -only the B/W clip is used",size=32,align=2),\
subtitle(stab4,"test stabiliser: dx=horizontal, dy=vertical",size=32,align=5)).converttoYUY2()




Eval(result)#.converttoRGB24()







The only I thing I changed in the above script (apart from sourcefile) was W=720 H=576 which I changed to W=720 H=540. This was because the aspect ratio looked wrong (people fatter) with 576, so I changed it to be exactly half of the original 1440x1080 ratio. I'll admit I am quite confused here as to what I should be doing regards output size!

I notice the output files from VirtualDub are quite huge (uncompressed RGB) - should I be changing to x264vfw-h.264? Also, if I do change output compression to x264vfw, am I right in thinking that I should also then change from "full processing mode" to "fast recompress" in VirtualDub?

I also still need to crop the borders properly!

[johnmeyer]

It sounds like you are doing everything right. I don't have any help to offer, other than to say that you need to find some way to store the original capture in something other than h.264, or at least find a way to change the compression setting used. It would be pretty strange if they completely hard-wired this setting. There must be a hack, .ini file, registry setting, or some other way to make these changes.

There are cropping options in VideoFred's script, and you can change those. I haven't looked at his script much since I branched off in my own direction, but I'm pretty sure his script crops and then re-sizes to whatever final size you have specified. Thus, you should be able to increase the crop at the top (that's where I saw the overhang from the previous frame). BTW, a better thing to do is adjust the framing control on the capture device so that the top and bottom frame border is roughly equal. Hopefully your transfer machine has such an adjustment.

[camelopardis]

thanks very much John. It is a shame about a compression but at least I'm getting them done after so many years. I think its better the film subjects that are still alive get to see them even if sub-par rather than not at all. I'll be keeping all the film so I can always get them re-done with better equipment at a later date.

[poisondeathray]

I order to use with AVISynth, the above clip was changed from MP4 to AVI by first extracting the raw h.264 using "My MP4Box" and then converting to AVI using AVC2AVI. If any of this is silly or wrong please let me know. I'm learning as I go along and am more than happy to be corrected!


You can use lsmash or ffms2 to open the MP4 directly . Both are external plugins separate from the main avisynth install.
e.g

LSmashVideoSource("video.mp4")

[JoeSuper8]

Hi Joe,

Welcome here :)

First of all, please try to capture as good as possible. Later, we can talk about Gammac etc....

About the Jpeg image sequence: you have not captured the full 0-255 range. It's more like 0-125. You have set red auto and blue auto off. Why? One of the great benefits of C2D is the exellent auto exposure feature. If you set the limit to 245 you will almost never get blown out whites.

Sharpness can be better also. Have you tried to find the 'sweet spot' from your lens? Every lens has a sweet spot. Easy to check: in C2D, use internal flashing and play with the lens aperture. At a certain point the image will look sharpest. This is your sweet spot. Leave it there forever. :) Then you can fine focus with your camera slide.

Also, there is no need to save as image sequence. I always save straight to AVI, at 15fps! I'm using the Canopus HQX codec for this but Lagarith etc.. will work fine too. Uncompressed is also an option but it creates huge files and why would we do it.. You will see no visual difference.

About the speed from the script: downscaling the source will speed up the script a lot. You can upscale again at the very end of the script.
https://forum.doom9.org/showthread.php?p=1810955#post1810955

many greetings,
Fred.

Thanks Fred and johnmeyer for your ideas - sorry I haven't been able to get to testing sooner. I noticed no significant change in Cine2Digits when I turned the Camera Settings - Analog - White Balance Blue and Red to Continuous instead of manual and left the other settings as described in my 8/22/17 post (the Cine2Digits manual suggested manual values for the camera white balance because the C2D program itself auto-adjusts the colors through its LED control). In both my attempt over the summer and my try this week, the C2D program LED control is set to auto.

I tried turning down the RGB dials from 1.47 to 1 but that did not affect overexposure. I also tried changing the pixel format (BayerRG8,16, YCBCR, RGB8) but that did not change the highlights.

I tried adjusting RGB gamma below 1 in C2D but the highlights did not seem to recover. Could the highlights be overexposed on the original film? Or another setting that's not right?

Fred as you suggested I adjusted my white level targets for Red, Green, and Blue to 245 in the "Exposure Options" C2D menu. I set auto exposure sensitiveity "Over" to 350 and Under to 650 (I did not notice a difference changing these values much lower and much higher).

I am still not sure how the range is closer to 0-125 in my capture.

johnmeyer and Fred I will look into the script issues and downsizing after fixing the highlights issue to achieve something closer to 0-255. Any settings I could be incorrectly setting somewhere?

Here is a video showing the C2D settings and gamma adjustments - please let me know if I might want to change anything in addition to AVI capture instead of TIFF:
https://player.vimeo.com/video/255482660

[videoFred]

Hi Joe,

It's a matter of settings, both camera settings and C2D settings.
Also, be sure that the ROI for the picture analysis is inside the black borders. Otherwise you wil get false results.

Fred.

[videoFred]

Here is a longer clip of the raw 1440x1080 output from the machine (you can't change the compression ratio, you can only adjust exposure, sharpness, and frame adjustment).

Hello Camelopardis,

We can not download this Vimeo clip. We need a "downloadable" clip :)

Also, do not use any camera sharpness (or other) settings. It is much better to do sharpening in post.

Fred.

[camelopardis]

Hello Camelopardis,

We can not download this Vimeo clip. We need a "downloadable" clip :)

Also, do not use any camera sharpness (or other) settings. It is much better to do sharpening in post.

Fred.

Hi Fred,

I've uploaded a raw output clip to my onedrive:
https://1drv.ms/v/s!Arqwxcq4sXstsA7drZ7CuhCNAQYD

The machine has 3 sharpness settings: low, medium, and high. You cant turn it completely off. I've set it to low.

[poisondeathray]

@camelopardis - why is it 23.976 fps AVI with blends ? Is it a 1:1 scanner ? I thought it recorded MP4 ? You mentioned 18FPS and 30FPS, but this clip 23.976 . Or does it scan at a fixed 30FPS rate without adjustments or variable speed ? I'm wondering at what stage are the blends being introduced

Are there options to bump up the bitrate on the machine or in software ? If that's 1st generation, there is significant quality loss from the compression. 7Mb/s for 1440x1080 isn't very much, even for final delivery formats let alone 1st gen transfer.

[johnmeyer]

Thanks for the upload.

Is this the unmodified, original video as it came directly from the Reflecta unit, or have you already done something to it, either with an AVISynth script, or something else? I ask this because what you posted is an awful mess and cannot possibly be used inside of VideoFred's script (or any other script designed to improve the transfer).

The problem is that you have blended fields. This is something you always see with a cheap transfer, where the person just points the video camera at the movie screen. However, I wouldn't think you would get it with a transfer unit that is frame accurate, which I thought was the case with the Reflecta.

So, unless you can upload something better, the short answer is that you cannot use VideoFred's script, or any other AVISynth script, because your movie frames are made up of blended fields.

[JoeSuper8]

Hi Joe,

It's a matter of settings, both camera settings and C2D settings.
Also, be sure that the ROI for the picture analysis is inside the black borders. Otherwise you wil get false results.

Fred.

Yep Fred I have set the ROI inside the Super 8 frame with room to spare, like in part 2 of your tutorial video.

As you saw in the link from my previous post, my histogram is very weak during the test capture (not much red, green, and blue area) whereas your tutorial shows significantly more R, G, and B in the C2D live histogram.

I have attempted to adjust the C2D and camera settings but I am still getting the same results.

Would you mind taking a look at the screen capture video link from my last post to see where I might want to change any settings to improve the histogram and range of capture?

[videoFred]

Would you mind taking a look at the screen capture video link from my last post to see where I might want to change any settings to improve the histogram and range of capture?

Every machine vision camera type has a different settings menu. In your case: I would try "black level". But even better: try them all and see what you get.

In C2D your gamma settings are to low.

And please try it on different films and/or other scenes too.

Fred.

[camelopardis]

Hi all.

My apologies! I realise now that wasnt the unmodified output from the Reflecta unit becuase I cut it into a short clip for upload in Sony Vegas (where it was then rendered as mpeg4 output.)

This time I uploaded a short clip DIRECT from the machine. This is the MP4 file it writes directly to the SDcard. I've done nothing to it and left it as the original 1440x1080 30fps MP4 file that the machine writes. (you cant change this output) I havent corrected the frame rate.

https://1drv.ms/v/s!Arqwxcq4sXstsA9RBsfWbucoH0By

[clivesay]

Hey everyone. I joined this forum just for this thread!

Long story short, at the first of the year I learned I have a sister that no one (including my father) knew anything about. My dad has been gone for over 25yrs. Some slides that I never seen before that were tucked away in a closet helped to unlock some of the mystery.

This now has me on a mission. I bought a good scanner to get good copies of my dad's slides and then was informed my grandparent had some 8mm reels that needed to be digitized. So, I jumped in and bought a wolverine moviemaker-pro and have jumped into 8mm film restoration. I have the scripts from here up and running and have been tinkering with them.

First question: Some of you people here have an incredible amount of experience in film restoration. How does someone like myself gain enough knowledge to do some basic but effective editing? The scripts make a difference on some of my test clips but I have no idea what parameters to tweak to refine some of the results. Is there a good tutorial somewhere? Do I post clips here and some of you can give me some pointers?

I am all about learning and not afraid to get my hands dirty.

Would appreciate any guidance.

Oh, if anyone is interested in my experience with my sister you can google "Crawfordsville DNA" and see our story that was broadcast on the news.

Thank you.

[`Orum]

First question: Some of you people here have an incredible amount of experience in film restoration. How does someone like myself gain enough knowledge to do some basic but effective editing?
I have never had the (dis)pleasure of having to restore analog footage, but I can tell you that there's no "one-size-fits-all" solution. That said his scripts are probably a good place to start.
The scripts make a difference on some of my test clips but I have no idea what parameters to tweak to refine some of the results. Is there a good tutorial somewhere? Do I post clips here and some of you can give me some pointers?
Posting clips will never hurt, and while we can give recommendations things are always best hand-tuned by the end user, as you know exactly what you want and we can only tune to what we think is best.
I am all about learning and not afraid to get my hands dirty.
Good! Sometimes the best way to learn what parameters to tune is by playing with them yourself. That said, when there's a lot of parameters to tune practiced hands can help, as they sometimes interact in important ways. There's also a lot to know about video (and a lot you probably don't need to know unless you are writing your own filters), but a good place to start is the wiki (http://avisynth.nl/index.php/Main_Page). There's a section right on the home page for getting started, but it sounds like you've already got the initial setup done.

Software like AvsPmod (https://forum.doom9.org/showthread.php?p=1801766#post1801766) can make things friendlier for those new to AviSynth and faster for those who are used to it, though it can be frustrating as it's far from "stable" and may crash on you (save your script frequently).

[johnmeyer]

First, the slides. The best way to scan them is a Nikon Coolscan scanner. They don't make them anymore, but you can get them on eBay. You can sell it when you are finished in order to recoup the $1,000+ cost.

However, slide scanning is slow. It took me 3+ years to get through my dad's 10,000 slides.

I recently needed to scan 3,000 slides for a client whose dad is dying and wanted to see the slides one last time. So, I converted a Carousel projector into a very high-quality, extremely fast scanner. The results, when using my old Nikon D-70 as the capture device are probably 85-90% as good as the quality of my Nikon scanner, albeit without any automatic dirt or dust removal, although that only works well on Ektachrome and doesn't really do much with Kodachrome. The Nikon Coolscan takes about 1.5-2.0 minutes per scan. My Carousel/Digital Camera lashup can capture about 15-20 slides per minute. So, it is almost 40 times faster.

I can provide a few construction details on how to do this conversion in a way that gives you good results. You cannot simply point the projector at a screen and then snap the result with a digital camera. The result is absolutely awful. Instead, you point the camera directly at the Carousel lens, at point blank range. You have to use the correct lenses in both the projector and camera; dim the bulb; and add an additional diffuser in he projector's light path. It is actually quite easy to do, but figuring out which parts to use took me quite a bit of time and effort.

As for your other question, VideoFred put all the control variables for his script at the beginning. You just simply change them and then look at the result.

I created my own version of his script, which I optimized for speed, but with far less sharpening. My work involves mostly really old film and I find that my version's lower levels of sharpening and grain reduction give results which I find more pleasing when dealing with typical amateur film.

Here are the variables from my script. The comments are intended to tell you what they do. So, just get the script working and then start changing these variables. VideoFred's scripts work the same way.
#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="resultS1" #specify the wanted output here
trim_begin=0 play_speed=15 #trim frames and play speed (PAL: 16.6666 or 18.75)

#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=1.0 #for all outputs
gamma=1.0 #for all outputs
blue= -0 red=-0 #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0 white_level=255 output_black=0 output_white=255 #manual levels, when returning result2 & result4


#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=16 CTop=16 CRight=16 CBottom=16 #crop values after Depan and before final resizing
W=720 H=480 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=20
est_left=40 est_top=40 est_right=40 est_bottom=40 est_cont=1.6 #crop and contast values for special Estimate clip


#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
denoising_strength=600 #denoising level of first denoiser: MVDegrain()
block_size= 16 #block size of MVDegrain
block_size_v= 16
block_over= 8 #block overlapping of MVDegrainMulti()
dirt_strength=23 #sets amount of dirt removal (big spots)
#dirt_strength=13 #sets amount of dirt removal (big spots)

#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 120 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
LSF_sharp_ness=160 LSF_radi_us=2 LSF_sub=1.5 #second sharpening parameters (LimitedSharpenFaster) sub=subsampling

#Alternative lower sharpening values
#PRE_sharp_ness= 120 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
#LSF_sharp_ness=120 LSF_radi_us=2 LSF_sub=1.5 #second sharpening parameters (LimitedSharpenFaster) sub=subsampling


Sharp_Strength = 0.4


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4 #X is a special parameter for reducing the autolevels effect on the whites
X2=2 #X2 is a special parameter for reducing the autolevels effect on the blacks


# END VARIABLES, BEGIN SCRIPT

[goodiesguy]

What parts of VideoFred's script do I remove to stop the colour correction and auto gain/brightness?

I prefer to do this manually and my few attempts to stop the script from doing this have broken it.

[johnmeyer]

What parts of VideoFred's script do I remove to stop the colour correction and auto gain/brightness?

I prefer to do this manually and my few attempts to stop the script from doing this have broken it.You simply change the "result" parameter.

[sys32768]

Was curious whether anyone had already done the work to modify Fred's original script here to use GAMMAC?

As an AVIsynth newbie, I find myself tripping up quite often and then starting over so I thought I'd ask.

[johnmeyer]

I created my own branch of Fred's script and posted it many years ago. However, like Fred, I haven't tried to post all my changes and updates because I don't have time to support it when people ask questions.

With that as preface, here is what I am using. It is not really fit for publication (i.e., not many comments on the new stuff), and there are probably bugs all over the place. However, it works for me.

It will take two posts to get it all.

/*
Film restoration script by videoFred.
Denoising, resizing, stabilising, sharpening, auto-levels and auto-white balance.

Modified by John Meyer, October 7, 2012
Modified by John Meyer, July 7, 2015 (remove errors)
Modified by John Meyer, September 13, 2017

Changes made by Meyer (June 8, 2010):
1. Replaced the MVDegrainMulti function, which was part of the original MVTools
with the equivalent function using the newer and faster MVTools2.
2. Added RemoveDirt function prior to Degrain in order to eliminate large dust spots. This
substantially improves the dirt removal capabilities of this script. The user will need to
download this plugin at:
http://www.removedirt.de.tf/
3. Eliminated a great deal of sharpening. The original script did sharpening in at least four places.
With grainy film stock, this sometimes created objectionable grain. Also, the limitedSharpenFaster
function, while excellent for video, is uncessarily slow, and isn't really needed for this low
resolution source. Too much sharpening can make the film look too much like video.
4. Added multithreading. I was able to roughly triple the script performance. It may be possible to
substantially increase this, perhaps as much as 12x instead of 3x. However, the autolevels function
would have to be replaced.
5. Fixed several things I found. The result3 option didn't have the manual color correction code, so I
added that back in. All the numbered "stab" variables (stab1, stab2, etc.) aren't needed except
for providing a test function for stabilization. I eliminated all of this to streamline the script.
6. Deflicker seemed redundant, given all the averaging that takes place with MDegrain, and also the averaging
that is done when the autolevels outputs are selected.
7. I reduced the number of frames used for averaging autolevels from the default (which is 5) to 2.
I probably should add a variable in the header so the user can change this. Something else to do
in the future ...
8. I added yet another set of crop parameters. I did this because both my capture and my output are
done using NTSC DV AVI which is 720x480. However, 8mm film is almost exactly square, so the captured
720x480 video has black bars on the side. These need to be cropped off prior to doing motion
stabilization, but then added back prior to the final output, which must still be 720x480 with the
black bars on the side.
9. I reduced the default depan settings to 20, which is what was recommended in the original script. I
also reduced the post-depan cropping. I did this so I could keep as much of the original frame
as possible.
10. I removed the second denoising and sharpening function. It just seemed to be too much, and made the
result too artificial
11. I removed the MVFLowFPS interpolation. I did this during one of dozens and dozens of attempts
to improve the speed of the script. I should probably add this back, but if I do so, I also
need to make it work correctly for interlaced output. If the goal is to show this on an NTSC or PAL
television set, then it is not correct to convert from the film fps to 25 fps progressive (PAL) or
29.97 fps progressive (NTSC). Instead, this should be done as follows (example given is NTSC):

MFlowFPS(source,super,backward_vec, forward_vec, num=60000, den=1001,ml=200)
SeparateFields()
SelectEvery(4, 0, 3)
Weave()

This yields interlaced 29.97, which has twice the temporal resolution as 29.97 progressive, and will
therefore look correct on a TV set. I have done a lot of this as part of my Kinescope to video
conversion scripts. For those scripts, the goal is to make the filmed version of a TV show look
like it was actually videotaped. However, for something that originated on film, this "does violence"
to the original feel of the media. It is true that it can make horizontal pans less "juddery," but
it won't feel like film anymore. Also, this technique does break down, espcially with fast motion
in the foreground.
12. Removed the unecessary "coloryuv(off_U=blue,off_V=red)" statement from the denoising section.
13. Added killaudio() statement to prevent lockups when using SetMTMode().

Changes made on October 7, 2012

1. Modified RemoveDirtMC function to do estimation in two steps. This improves performance and quality.
2. Modified MDegrain to do estimation in two steps. This provides roughly the same quality with blocksize=8
as what I was getting with blocksize=4, but with virtually no performance penalty
3. Updated to use special build of MVTools2 that works with SVP.
4. Moved Autolevels inside special multi-threading section in order to improve performance with autolevels.
5. Removed all trim commands because I do all editing outside the script (in Vegas).
6. Included two alternative ways to do frame interpolation: SVP and InterFrame. The InterFrame script is probably better
but I included the SVP code because it may provide a starting point for more customized approaches. Neither is enabled by default.

Changes made on September 13, 2017

1. Replaced old useless color correction with GamMac.
2. Removed never-used manual controls.
3. Moved deflicker to end of chain because GamMac introduces flicker.
4. Moved RemoveDirtMC to happen AFTER denoising.
*/
#====================================================================================


#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------
#Change the following line to point to your video file
film="e:\fs.avi"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result2" #specify the wanted output here
play_speed=17.98201798 #play speed (8mm=16; Super8=18; 16mm sound=24)


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation = 1.0 #for all outputs
gamma = 1.0 #for all outputs

#GamMac Parameters
LockChan = 1 #(0=red channel)
LockVal = 128.0 #default 128 -- Used when LockChan = -1 (for flicker)
Scale = 2 #Fred recommended 2 instead of 1
RedMul = 1.0
GrnMul = 1.0
BluMul = 1.0
Th = 0.1
GMx = 0
GMy = 0
GMw = 0
GMh = 0
LOTH = 0.2
HITH = 0.2
OMIN = 0 #limiting the output a little bit makes it a little 'softer' to look at
OMAX = 255
Al2 = 20
autolev_bord1 = 50
borderV=10 borderH=10

#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=8 CTop=8 CRight=8 CBottom=8 #crop values after Depan and before final resizing
W=720 H=480 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed

#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=10 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=10
est_left=40 est_top=40 est_right=40 est_bottom=40 #crop and contast values for special Estimate clip
est_cont=1.1 #Too large a value defeats stabilization


#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#denoising_strength=600 #denoising level of first denoiser: MVDegrain()
#block_size= 16 #block size of MVDegrain
#block_size_v= 16
#block_over= 8 #block overlapping of MVDegrainMulti()

#Alternate denoising using smaller block size
#denoising_strength= 600 #denoising level of first denoiser: MDegrain()
#block_size= 8 #block size of MVDegrain
#block_size_v= 8
#block_over= 4 #block overlapping of MVDegrainMulti()

#Alternate denoising using smalles block size
denoising_strength= 300 #denoising level of first denoiser: MDegrain()
block_size= 4 #block size of MVDegrain
block_size_v= 4
block_over= 2 #block overlapping of MVDegrainMulti()

dirt_strength=18 #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
#120 is pretty aggressive; 85 seems more subtle
#PRE_sharp_ness= 120 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
PRE_sharp_ness= 85 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising

Sharp_Strength = 0.05 #0.05 is often better (avoid too much sharpening)


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4 #X parameter for reducing the autolevels effect on the whites
X2=4 #X2 parameter for reducing the autolevels effect on the blacks


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=3


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

[johnmeyer]

[edit]Fixed errors in Mrecalculate lines
Script continued ...

SetMemoryMax(768)

AddAutoloadDir("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\Film Restoration\Script_and_Plugins",toFront=true)

/*
Here is a list of plugins used
Loadplugin("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\MVTools\mvtools2.dll")
LoadPlugin ("mvtools2.dll") #Version 2.5.11.9 2/24/2012
LoadPlugin("autolevels.dll") #Version 0.6.0.0 1/09/2011
LoadPlugin("Deflicker.dll") #Version 0.4.0.0 8/16/2004
Loadplugin("Depan.dll") #Version 1.10.0.0 4/09/2007
LoadPlugin("DepanEstimate.dll") #Version 1.9.2.0 3/25/2007
LoadPlugin("fft3dfilter.dll") #Version 2.1.1.0 2/20/2007
Loadplugin("mt_masktools.dll") #Version 2.0.23.0 3/14/2008
loadplugin("RemoveDirtSSE2.dll") #Version 0.9 5/05/2005
Loadplugin("RemoveGrainSSE2.dll") #Version 0.9 5/01/2005
Loadplugin("removegrain.dll") #Version 0.9 5/01/2005
Loadplugin("warpsharp.dll") # 4/05/2010
*/

source1= Avisource(film).killaudio().assumefps(play_speed).converttoYV12()
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #remove any black borders on input video
#cropped_source=filldrops(cropped_source) #Use this when removing bad frames that have been removed by duplicating previous frame


#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#DENOISING
#...................................................................................................................................................................
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

prefiltered = RemoveGrain(stabcrop,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(superfilt, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(superfilt, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(superfilt, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)

#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................
#Everything in this section is commented out because I usually choose NOT to do frame interpolation because
#it causes too many unexpected artifacts on certain frames. It also makes the film look like video which
#aesthetically is not always pleasing.

/*
prefiltered = RemoveGrain(PreBorderFrame,2)
super = MSuper(PreBorderFrame,hpad=16, vpad=16, levels=1) # one level is enough for MRecalculate
superfilt = MSuper(prefiltered, hpad=16, vpad=16) # all levels for MAnalyse
backward = MAnalyse(superfilt, isb = true, blksize=16,overlap=8,search=3,dct=0)
forward = MAnalyse(superfilt, isb = false, blksize=16,overlap=8,search=3,dct=0)
forward_re = MRecalculate(super, forward, blksize=8, thSAD=100)
backward_re = MRecalculate(super, backward, blksize=8, thSAD=100)
PreBorderFrame = MFlowFps(PreBorderFrame,super, backward_re, forward_re, num=30000, den=1001,ml=200,mask=2)
*/

#Alternative interpolation
/*
superfps= MSuper(PreBorderFrame,pel=2)
backward_vec2 = MAnalyse(superfps, isb = true,blksize=16)
forward_vec2 = MAnalyse(superfps, isb = false,blksize=16)
PreBorderFrame = MFlowFps(PreBorderFrame,superfps, backward_vec2, forward_vec2, num=60000, den=1001, ml=200).SeparateFields().SelectEvery(4, 0, 3).Weave()
*/

#Yet another way to do interpolation
#The following alternative can provide synthesized frames for hi-def material that causes MSuper to choke
#////// Begin alternative to MFlowFPS ////
#SmoothFPS2(PreBorderFrame,threads)
#////// End alternative to MFlowFPS ////

#A final way to do interpolation, although Interframe is just a front-end to SmoothFPS, so I don't think the results are any different.
#The following is another alternative that can provide synthesized frames for hi-def material that causes MSuper to choke. Probably better than the one above.
#////// Begin alternative to MFlowFPS ////
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)
#////// End alternative to MFlowFPS ////

#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................

Baseclip = PreBorderFrame.crop(borderV,borderH,-borderV,-borderH,align=true).bicubicresize(W,H)
blank_black = Blankclip(baseclip, width=autolev_bord1,height=autolev_bord1)
blank_white= Blankclip(baseclip, width=autolev_bord1,height=autolev_bord1, color=$FFFFFF)
Average= baseclip.FredAverage().invert()
over1 = overlay(baseclip,blank_black, x=40,y=300)
over2 = overlay (over1,blank_white, x=160, y=300) \
.bicubicresize(width(baseclip)-(al2)*2,height(baseclip)-(al2)*2)
Detect = (al2 >1) ? overlay (Average, over2,x=al2,y=al2) \
.converttoRGB24(matrix="rec709") : over2.converttoRGB24(matrix="rec709")

result1= PreBorderFrame.ConvertToRGB24.GamMac(verbosity=4,DC=Detect,Show=False, \
LockChan=LockChan, Th=Th, LockVal=LockVal, Scale=Scale, RedMul=RedMul,\
GrnMul=GrnMul, BluMul=BluMul, loTh=LOTH,hiTh=HITH,oMin=OMIN,oMax=OMAX,\
x=GMx,y=GMy,w=GMw,h=GMh).converttoYV12().deflicker().addborders(X,0,0,0,$FFFFFF) \
.addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0) \
.addborders(bord_left+in_bord_left, bord_top+in_bord_top, \
bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT2: AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.ConvertToRGB24.GamMac(DC=Detect,Show=False, Dither=TRUE,verbosity=5,LockChan=LockChan, \
Th=Th, LockVal=LockVal, Scale=Scale, RedMul=RedMul, GrnMul=GrnMul, BluMul=BluMul, \
loTh=LOTH,hiTh=HITH,oMin=OMIN,oMax=OMAX,x=GMx,y=GMy,w=GMw,h=GMh).converttoYV12() \
.deflicker().addborders(bord_left+in_bord_left, bord_top+in_bord_top, \
bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS
#.....................................................................................................................................................................
result3= PreBorderFrame.addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: NO CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, GamMac",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"GamMac",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"no level or color corrections",size=28,align=2))

Eval(result)
#return Detect
# Enable MT!
Prefetch(threads)


# END SCRIPT, BEGIN FUNCTIONS

#=================================================================================================================================
#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int "limit", bool "_grey")
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

# Alternative way to do pre-filtering.
# prefiltered = fft3dfilter(clip,sigma=1,sigma2=2,sigma3=3,sigma4=5,bw=64,bh=64)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)

super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

# Increase thSAD if moving objects are being removed
bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

[sys32768]

Thanks, John. I finally got yours working on my ProRes .MOV files that are 1920 x 1080 but am unclear on size, crop and borders.

It appears that my .MOV files have a 150px left border and 150px right border. I assume for every filter to work correctly, those must be cropped.

If I set those to 150 on in_bord_, the script crashes with an out-of-bounds memory access error on RemoveDirtSSE2. If I set them to 120 it doesn't crash.

I'm also unclear on what my final W= and H= should be in light of the other cropping/border options.

[videoFred]

If I set those to 150 on in_bord_, the script crashes with an out-of-bounds memory access error on RemoveDirtSSE2. If I set them to 120 it doesn't crash.

I think this is a mod8 issue. 1920-150-150= 1620/8= 202.5.
I you crop 152 on both sides, it"s 1920-152-152= 1616/8= 202.



I'm also unclear on what my final W= and H= should be in light of the other cropping/border options.

It depends on the aspect ratio and the final result you want.
Do you want HD again? Then you must add borders again.

And what is your source? Transfered 8mm film or something else?
Because 8mm should be 4:3 and 1616x1080 is not 4:3.

1440x1080 is 4:3.


Fred.

[johnmeyer]

Fred,

Isn't 8mm (without the material between sprocket holes) closer to 1:1? Super 8 is most definitely 4:3, but I think 8mm is pretty close to square.

Also, to add to what Fred said about cropping: he is right that it is a mod issue, but I can't remember if it is mod4 or mod8. However, you'll be safe either way if you follow his suggestion and make the crop a multiple of 8. Also, I've never had good luck using a different crop size for the left and right so I always make the black borders equal on either side in my NLE before feeding it to the script so I can use the same crop size for both. Since you'll want it centered on the screen, this actually makes sense to do even if there wasn't a problem when the sizes are different on the left and the right.

[sys32768]

I think this is a mod8 issue. 1920-150-150= 1620/8= 202.5.
I you crop 152 on both sides, it"s 1920-152-152= 1616/8= 202.

It depends on the aspect ratio and the final result you want.
Do you want HD again? Then you must add borders again.

And what is your source? Transfered 8mm film or something else?
Because 8mm should be 4:3 and 1616x1080 is not 4:3.

1440x1080 is 4:3.
You are correct about 152...math is my weakest subject. Thank you!

My sources are HD 1920 x 1080 16FPS ProRes .MOV files from 8mm reels, transferred by a FlashScan system. They all have the black borders which if cropped makes the full frame about 1560 x 1080. I should look more closely to ensure each file has the same border width.

I was hoping to do everything with these scripts but some post-production editing in Vegas Pro is necessary. I also tend to have more luck with filters like unsharp mask in Vegas Pro because the feedback is so quick, allowing me to fine tune much easier.

I've attached an unprocessed HD frame capture roughly cropped.

John: I put trim beginning back into your script, and am going to attempt to add trim end back. It helps me a lot because several of my reels are more than 30 minutes and often contain half footage I don't want, so it speeds up my processing a great deal to have things split out from the get-go.

[johnmeyer]

I was hoping to do everything with these scripts but some post-production editing in Vegas Pro is necessary. I also tend to have more luck with filters like unsharp mask in Vegas Pro because the feedback is so quick, allowing me to fine tune much easier.I totally agree: do as much as you can in Vegas.

If you need any help with Vegas workflow, let me know. For instance, I mentioned above that you have to be careful when rendering to MPEG-2 for DVD because you don't want to screw up the pulldown (the fields that are added to go from 15, 16, or 24 fps to 29.97 fps). The key setting in Vegas which will avoid problems is to "disable resample" for every event. I wrote a Vegas script that does this which I'd be happy to post. I have dozens of other Vegas scripts which automate many other aspects of editing film. For instance, one of my first Vegas scripts, from fifteen years ago, is one that will trim an event from the current cursor position to the beginning of the frame. It really helps speed up editing out all those "flash frames" that you get at the beginning of each scene, when the camera's shutter was getting up to speed and therefore overexposing the first 1-2 frames.

I also have several AVISynth scene detection scripts, some of which just look for the flash frames.

[sys32768]

Thanks! I would be interested in scripts/techniques for rendering 16fps 8mm at "true speed". My 16fps 8mm sources seem to be somewhat dreamy slow motion. I am unsure if this is the case with all 8mm or something my family did when filming.

Interpolated to 30fps they look much smoother but they are still slow. Being able to see my departed family members moving at natural speed would be a delight.

[videoFred]

Thanks! I would be interested in scripts/techniques for rendering 16fps 8mm at "true speed". My 16fps 8mm sources seem to be somewhat dreamy slow motion. I am unsure if this is the case with all 8mm or something my family did when filming.

What is your source? Super-8 or Regular-8? Super-8 was standard 18fps, Regular-8 was standard 16fps. But some cameras could be set to higher frame rates like 24 or even 32fps.

Anyhow, if your digital transfer is frame accurate (one film frame = one digital frame, no duplicates) then you can easily change the play speed with assumefps().

Fred.

[videoFred]

Fred,

Isn't 8mm (without the material between sprocket holes) closer to 1:1? Super 8 is most definitely 4:3, but I think 8mm is pretty close to square.

Hi John ;)

The film frame of standard (regular) 8mm is 4,50 x 3,30mm. This is almost 4:3. But camera and projector gates are not always the same of cource.

Fred.

[sys32768]

What is your source? Super-8 or Regular-8? Super-8 was standard 18fps, Regular-8 was standard 16fps. But some cameras could be set to higher frame rates like 24 or even 32fps.

Anyhow, if your digital transfer is frame accurate (one film frame = one digital frame, no duplicates) then you can easily change the play speed with assumefps().

Fred.Hi Fred, these would all be 16fps scans of Regular-8. I was hoping to interpolate them and "normalize" their speed.

[johnmeyer]

Thanks! I would be interested in scripts/techniques for rendering 16fps 8mm at "true speed". My 16fps 8mm sources seem to be somewhat dreamy slow motion. I am unsure if this is the case with all 8mm or something my family did when filming.That is odd. This can be caused by several things.

1. The original scanning process duplicated frames. This seems unlikely because you are editing them in Vegas and you would notice duplicate frames.

2. The speed header is wrong. I assume you have looked at the media in Vegas' Project Media view where you can see what fps the video is set for. Also, when viewing in Vegas, make sure to match the project properties to the source (there is a little "match" icon in the upper right corner of the Project Properties menu).

3. The people who operated the camera depressed the button too far and filmed at a high rate of speed.

If the problem is #3, then you are actually in luck because if you do things right and simply change the playback speed (don't interpolate frames), you will end up with a beautifully smooth result that looks more like video than film.

For fixing #3, you do NOT want to be doing anything with interpolation because you already have enough frames. The problem, in fact, is just the opposite for which interpolation is usually used. All you want to do is change the fps from 16 to 32, or whatever number gives you playback that seems like the right speed. If you want the playback speed to match a standard, then you change the playback rate to 29.97. In either case, all it takes is a single AssumeFPS(29.97) at the end of the script. What you want is something that does the exact same thing as changing the speed control on a variable speed film projector: you crank up the speed until the motion looks normal.

Remember that these days TV sets can play any speed, so there is no reason to conform the video to the old 24, 25, or 29.95 North American or European standards.

In Vegas, you can experiment to find the correct playback speed by using the J-K-L keys which let you scrub back and forth. Most Vegas users learn how to use these fairly early on, but what most users don't know is that there is a second mode to these keys. If you press and hold the "K" key and then press either the "L" (forward) or "J" (backward) keys, the playback will start at 0.25 of normal speed and then, each time you press the "L" key again (assuming you are playing in the forward direction), the speed will increase by another 0.25. You can release holding the "K" key and Vegas will stay in this playback mode until you press "K" again, at which point it will revert to its regular playback mode. Once in this special mode, if you press the "J" key, instead of immediately playing in the reverse direction, it merely slows down the speed to the next slowest setting.

So use this mode and observe the motion until you feel you have achieved the right speed. I have found that in these situations, if you have to err, make the speed a little too slow, rather than too fast. Also, keep an eye out for a clock with a second hand somewhere in the scene. I've been lucky enough to find a clock many times (I do a lot of sporting events, and old stadium clocks had a sweep second hand). You simply count the number of frames as the second hand crosses adjacent marks on the face, and that will give you a very accurate reading of how many frames per second the camera was taking when the scene was filmed. You can also use physics: if someone drops something from a kitchen counter, those are standard 36 inches height in this country. At 16 fps, that object will take almost exactly 7 frames to hit the floor.

BTW you scan look very good, although there is some red in the upper left corner that may just be a developing artifact in the film, but also may be chromatic aberration in the transfer setup. Also, you might want to use the Color Curves feature in Vegas. I've posted about this many times before. Here is what my preset looks like:

Vegas Color Curves For Film (https://forum.doom9.org/showthread.php?p=1773320#post1773320)

[sys32768]

Thanks that's a great help. I think 16fps looks unnatural, period, so I may be fooled by that alone.

The curves are definitely helping. I thought GamMac would cure everything.

Can I turn off sharpening on your script (and Fred's) by zeroing out the parameters? I'm preferring to do the sharpening in Vegas post. Also, my footage isn't very noisy so I would likely want to turn that down too.

[johnmeyer]

Fred uses a LOT more sharpening than I do. That was one of the changes I made when I adapted his work a decade ago. Just set the two remaining sharpening parameters to zero in my script.

[sys32768]

I was going to invest in https://www.svp-team.com/wiki/Manual:SVPflow to double my 8mm 16fps to 32fps with interpolation. Is there any reason I'd want that to be 30fps instead of 32fps?

[wonkey_monkey]

Is there any reason I'd want that to be 30fps instead of 32fps?

It would look smoother on displays. 32fps would result in a 2/2/2/2/2/2/1 pattern (or something like that) when viewed on a 60fps display. Maybe noticeable, maybe not. If it was me, I'd go to 32fps by interpolation so every other frame is a "real" frame, then I'd assumefps(30) to avoid that little judder. I wouldn't interpolate directly to 30fps because then the majority of frames would be interpolations.

[johnmeyer]

If you want to put this on DVD, which only accepts 30 fps (29.97), what you can do is duplicate (or interpolate) frames to 32 fps, and then use AssumeFPS(29.97) to change the speed of playback. All this does is change the playback speed, like slowing down the film projector speed. I doubt you will notice that change in speed and indeed, most PAL films are sped up from 24 to 25 fps without any adjustment, and people don't notice or complain.

FWIW, my advice is to not use interpolation to go from 16 to 30 or 32 fps. I do film transfers and have spent a LOT of time with motion estimation, tweaking all the settings to get the best possible results. The problem: motion estimation always fails at some point. You can count on it. And, when it fails, it does so spectacularly.

Having said that, I did use interpolation for some 1940s parade footage I posted on YouTube. You'll probably recognize the footage because StainlessS and others have used it a lot for testing various scripts. The problem is that at 16 fps, the temporal gap between frames is huge and therefore the motion estimation has a really hard time predicting where all those little blocks should be moved in order to synthesize the new frame. Even though I knew I'd create some problems, I used it on this clip because there was so much horizontal movement that the judder effect you get from slow frame rates was overwhelming (we were taught to never do horizontal pans with a movie camera).

Things to look for when watching this clip:


Artifacts when objects get revealed as the parade float moves down the street.

Artifacts from people legs moving back-and forth

Artifacts from waving flags.

The moose antlers (by far the worst artifact)

Rotating spokes on wheels

Object moving across the frame, close to the camera


I can go on, but you get the idea.

Here's the clip:

1940 Flint Michigan Parade (https://www.youtube.com/watch?v=t8HjRN0rw5M)

[FranceBB]

I quote everything.
Unfortunately, linear interpolation is prone to fail and it will fail in a way or another.
Overall it looks smooth, but - as john said - motion estimation introduces artifacts,
from the most subtle ones:

https://i.imgur.com/EXBMhgw.jpg

https://i.imgur.com/XMhZgfE.png

to the more evident ones:

https://i.imgur.com/pOMNSeP.jpg

https://i.imgur.com/YzHJay0.png

'till the most spectacular ones:

https://i.imgur.com/omtEORV.jpg


Truth is: there's nothing we can do about it, no fine-tuning of MVTools, not magical alternative filters that use magical algorithms, not hybrid blend-interpolation scripts based on mythological error estimation, nothing...
If you have to conform it to DVD, I would either blend or duplicate frames, but since blending would make it look even worse, my suggestion is to duplicate frames.
By the way, @sys32768 I'm sorry I didn't help you out any further when you uploaded the sample on the other topic, but I'm not that good with despot, descratch, image stabilization, denoise and color correction of old sources as I rarely do it, so I left it to other people.
Here on Doom9 there are more competent people when it's up to dealing with these kind of sources; for instance Fred and John are way more expert than me on this subject. ;)

[sys32768]

Here's an early 1960s parade I interpolated from 16fps to ~30 FPS: https://vimeo.com/316628412 . No stabilization added. Basically just GamMac and interpolation. I've basically given up on sharpening because I cannot seem to find a "sweet spot".

For me personally, I prefer the 30fps because it feels more natural and lets me focus in on the faces. I definitely see the artifacts but I find 16fps makes the entire picture distracting with flickering/choppy frame rates. That being said, I'm storing all my 8mm with the 16fps originals along with the 30fps post-produced ones out of respect for the material and future generations who will see it.

[johnmeyer]

That's a really nice looking transfer.

The playback was not smooth for me, but hopefully that is Vimeo, and not your transfer or the process you are using to interpolate to 30 fps. I am seeing persistent jerkiness, as though a frame is being dropped. If no one else sees it, then it is probably just Vimeo which often gives me problems, no matter what computer or browser I use.

[wonkey_monkey]

Makes me think of The Wonder Years.

I see some judders too. Probably a Vimeo issue. I'd seen YouTube be similarly slightly jerky on certain combinations of encoding settings.

[algia]

Hi,
I'm trying to use the original cleaning script from 2006 (first post) made by VideoFred, but it uses MVAnalyseMulti and MVDegrainMulti which are from the 'Josey Wells' multi-threaded version of MVtools, which is deprecated now.
I tried to change them to MAnalyse and MDegrain, but the parameters that Fred passes on the script are not recognised, so I think that - perhaps - the syntax has changed over time.


vectors= cleaned.MAnalyse(refframes=denoising_frames, pel=2, blksize=block_size, blksizev= block_size_v, overlap=block_over, idx=1)
denoised= cleaned.MDegrain(vectors, thSAD=denoising_strenght, SadMode=1, idx=2).unsharpmask(USM_sharp_ness3,USM_radi_us3,0).sharpen(last_sharp)


Do you have an updated version of the script?
Full script: https://pastebin.com/bQh7kEqZ

Thank you in advance,
Livio

[johnmeyer]

Hi,
I'm trying to use the original cleaning script from 2006 (first post) made by VideoFred, but it uses MVAnalyseMulti and MVDegrainMulti which are from the 'Josey Wells' multi-threaded version of MVtools, which is deprecated now.
I tried to change them to MAnalyse and MDegrain, but the parameters that Fred passes on the script are not recognised, so I think that - perhaps - the syntax has changed over time.You can use any of my versions of his scripts. I posted them fairly early on in this long thread. One of the first things I did was to upgrade to the newer versions of MVTools (i.e., MVTools2).

Here is the latest version, although it may have some bugs in it. Search on my user name within this thread to find the earlier versions which are a little simpler, and better tested.

https://forum.doom9.org/showthread.php?p=1861734#post1861734

[StainlessS]

John, If you have a script that you use often and (like me) dont like to make copies of the script, & have to change source file name [its so irksome], then Avisynthesizer_Mod is real handy.
(https://forum.doom9.org/showthread.php?t=166820&highlight=Avisynthesizer_Mod)

Just add this into your script, eg

#ASYNTHER Simple-AVISource reader
[AVISource("___FILE___")]
/*
... etc
*/


The #ASYNTHER line must be first in script, and the text following [in this case Simple AVISource reader] is shown in menu so that you can select the above script template,
(after Right click on clip, and SendTo Avisynthesizer).
The [AVISource("___FILE___")] line can live anywhere and ___FILE___ [3 underscore] is where source filename will be inserted.
Source filter should be enclosed in [], where multiple selected source files will be concatenated using same source filter.

if you use eg below instead you can just provide a file name, and do whatever you will with it (No concatenation attempted, as no [] brackets).


#ASYNTHER BATCH_Filename_Processor
SourceFile = "___FILE___"

SomeSourceFilter(SourceFile)
# ...


Above will allow creation of multiple [or just one] scripts for each filename that you group selected before SendTo.
Another similar example

#ASYNTHER BATCH_MySource
MySource("___FILE___")

Return Last

Function MySource(String fn) {
RT_DebugF("We Are Loading file '%s'",fn)
Return AviSource(fn)
}


Same but Not Batch, will concatenate multiple selections, sending each filename to Debug output

#ASYNTHER Simple_MySource
[MySource("___FILE___")]

Return Last

Function MySource(String fn) {
RT_DebugF("We Are Loading file '%s'",fn)
Return AviSource(fn)
}



And another

#ASYNTHER BATCH_simple_avisource_43
AVISource("___FILE___")
RT_SignalDar2(4,3) # Auto set Aspect Ratio for MeGUI
ConvertToYV12 # EDIT: Keep MeGUI happy, if loads as eg YUY2 or RGB
Return Last

Creates script for each file in a group selection, ie BATCH MODE where missing [] brackets.


I've lost scripts before now, where I cut them to somewhere else, instead of copy [and later deleted], so above can be also safer.

[johnmeyer]

StainlessS,

Thanks for that. I missed your post seven years ago. I'm happy with my current versioning approach, but I see some other things in Avisynthesizer_Mod that I can use, especially batch creating AVS files.

So, thanks for the extensive description and for creating another great piece of software.

[videoFred]

Avisynthesizer_Mod is real handy.
(https://forum.doom9.org/showthread.php?t=166820&highlight=Avisynthesizer_Mod)

Another fine StainlessS tool, thank you! :)

Fred.

[videoFred]

Here's an early 1960s parade I interpolated from 16fps to ~30 FPS: No stabilization added. Basically just GamMac and interpolation. I've basically given up on sharpening because I cannot seem to find a "sweet spot".

I have done lots of sharpening tests on digitized 8mm film files. For me, the best way is to sharpen in steps with unsharpmask. Big radius first, then smaller radius. And some blur between the sharpening steps will help to avoid sharpening artefacts.

For example:
yourfile.unsharpmask(30,6,0).blur(0.8).unsharpmask(30,4,0).blur(0.8).unsharpmask(30,2,0).blur(0.8)


First parameter is strenght, second parameter is radius, third parameter... I have forgotten but it must be set to 0 :D

Anyhow, play with strenght and radius from all steps and see what you get. Of cource you can play with the amount of blur between the sharpening steps too.

Now, about this Flashscan HD transfer: it's not very sharp, is it? I can barely see any film grain. The Flashscan works with continue film transport. This means that the continue moving frames are 'frozen' with a very short Led light flash. If this flash is to long, there will be some motion blur. I also have the impression there are dropped frames now and then.

Fred.

[sys32768]

I have done lots of sharpening tests on digitized 8mm film files. For me, the best way is to sharpen in steps with unsharpmask. Big radius first, then smaller radius. And some blur between the sharpening steps will help to avoid sharpening artefacts.Thank you, Fred! That simple line is giving me the same quality sharpen as I was getting testing PhotoShop CC unsharp. On this particular clip (which does seem blurry as you stated), I did the first unsharp as 130 and the 2nd unsharp as 60 and left everything else like you shared. Fantastic!

The issue now is doing this post-interpolation sharpens the odd artifacts excessively. I will see how it looks if I sharpen before interpolation.

[johnmeyer]

Since multiple people have now commented on what appear to be dropped frames, you need to make sure that your actual transfer (as opposed to the Vimeo version) doesn't have these. Perhaps you could upload 10-15 seconds of original footage for people to look at. If you have dropped frames, everything else needs to be put aside until you get that fixed, not only because it looks bad, but also because all the algorithms in Fred's amazing scripts rely on having consistent temporal cadence from one frame to the next.

[poisondeathray]

You can download the original upload from vimeo with youtube-dl . It looks like he used handbrake to encode (vimeo doesn't, so you know it's the original) . VFR, 29.52 fps, and indeed there are dropped frames. Probably handbrake's fault, use of wrong settings; but possibly a procedural error in the workflow too

[johnmeyer]

You can download the original upload from vimeo with youtube-dl . It looks like he used handbrake to encode (vimeo doesn't, so you know it's the original) . VFR, 29.52 fps, and indeed there are dropped frames. Probably handbrake's fault, use of wrong settings; but possibly a procedural error in the workflow tooThanks for doing that!

It confirms what I saw, and definitely provides backing for my statement that the OP should not spend any time on these other somewhat ancillary issues, until he figures out and fixes this very fundamental problem.

I've downloaded from YouTube many times, but AFAIK, that is always re-encoded video. Vimeo used to keep the original video for download for a period of 1-3 months, as I remember. It sounds like they still do that.

[sys32768]

Thanks for doing that!

It confirms what I saw, and definitely provides backing for my statement that the OP should not spend any time on these other somewhat ancillary issues, until he figures out and fixes this very fundamental problem.Are you guys assuming the original transfer (16fps .MOV ProRes) is missing frames? I do see the occasional skipping and jumping in that particular parade clip on the source transfer too, but if that's the case what can I even do?

Unfortunately the parade section is inside a 30-minute reel so I'd have to split it out to AVI.

In the Vimeo uploaded parade clip, I first ran the 16FPS version through this for interpolation (but no stabilization):

super=SVSuper("{gpu:1, pel:2}")
vectors=SVAnalyse(super, "{block: {w:16, overlap:3}}")
SVSmoothFps(super, vectors, "{}",mt=1)

That outputs a 32FPS. So I think in Handbrake I set it to 30FPS and VFR.

Now that I'm sharpening, however, the interpolation is starting to look more wonky than ever so maybe it's time to just stick to stabilization and live with 16fps.

[johnmeyer]

I'm assuming nothing. The only way to diagnose this is to start with a 10-15 second clip from the original transfer (cut without re-encoding it), preferably from the same footage you've already posted.

I have seen "frame accurate" systems that do end up dropping frames, usually from the software used to run them. This software is often created by people who are not seasoned programmers, so they make lots of mistakes.

[sys32768]

I'm assuming nothing. The only way to diagnose this is to start with a 10-15 second clip from the original transfer (cut without re-encoding it), preferably from the same footage you've already posted.OK I dropped a 15s AVI here of the parade. There's also a .MOV from the same transfer company of another short reel:

https://1drv.ms/f/s!AkeySxbVM_V_lJs69EQAB9p_3huXjQ

[videoFred]

The issue now is doing this post-interpolation sharpens the odd artifacts excessively. I will see how it looks if I sharpen before interpolation.

Yes, I do the sharpening myself before interpolation.

Now, while talking about interpolation... John is right of cource. There will always be unwanted artefacts. But if you are using svpflow with the Interframe.avs script, there is a parameter called 'OverrideArea'. If you set this to -say- 150 and above then the result will be less interpolation and more frame blending.

No time anymore now... will come back with some more information about aWarpsharp.

Fred.

[videoFred]

For example:
yourfile.unsharpmask(30,6,0).blur(0.8).unsharpmask(30,4,0).blur(0.8).unsharpmask(30,2,0).blur(0.8)


First parameter is strenght, second parameter is radius, third parameter... I have forgotten but it must be set to 0 :D

Got it! It's threshold:
http://avisynth.nl/index.php/WarpSharp/UnsharpMask

Setting it to 2 or 3 (not more) will reduce artefacts.

Fred.

[videoFred]

OK I dropped a 15s AVI here of the parade. There's also a .MOV from the same transfer company of another short reel:

https://1drv.ms/f/s!AkeySxbVM_V_lJs69EQAB9p_3huXjQ

On the parade file, it looks like there is a dropped frame between 24 and 25 and another one between 92 and 93.

On the MOV file, dropped frames between 624-625 and between 1142-1143.

Fred.

[sys32768]

On the parade file, it looks like there is a dropped frame between 24 and 25 and another one between 92 and 93.

On the MOV file, dropped frames between 624-625 and between 1142-1143.

Fred.
Thanks for checking. Sadly I cannot afford another transfer of these reels right now.

What processes do the dropped frames interfere with most? I assume blending and interpolation?

[johnmeyer]

You can fix the drops, almost perfectly in most cases, using the same motion estimation technology that you want to use to go from 16 to 32 fps. Here's a video of some film I transferred where the film was "losing the loop", but inside the camera when it was filmed, and not in the projector. Thus, the problem was in the film itself.

Restoration of Jumpy Film (https://www.youtube.com/watch?v=uzMFodrGHDs)

To fix it, I created a duplicate of the previous frame at each jump, replacing the jump with the dup. I have a Vegas script that does this with one button push. I then used the Filldrops() script that you can easily find on this forum. It looks for duplicates and replaces one of the dups with a motion estimated frame created from surrounding frames. As you can see from my video example, it worked perfectly. I always wish I'd used the film tens seconds later when the truck drives away, because it too came out perfectly and it is much more impressive to see the jumps removed when there is motion in the frame.

If you don't do this fix before you apply Fred's script, it won't make a huge difference, because you don't have anything like the number of bad (missing, in your case) frames as in my sample above. However, the noise reduction and deshaking will be slightly degraded at the points where there is a bigger temporal gap than there should be. The biggest problem will be when you try to interpolate frames to go from 16 to 32 fps.

As you know, you can get occasional horrible artifacts using this technology. You get these far more often with low frame rates because the algorithms have a harder and harder time figuring out how to plug those big gaps in time. If you think about it, if your video was 1 fps, and you wanted to go to 30 fps, and it was a video of someone bouncing on a trampoline, how can the algorithms know what to do? In my experience, it isn't until you get to about 24 fps (sound film) that you start to get pretty good results with interpolating between frames using motion estimation.

For an example of what happens with really low frame rate material, here is a unique video, only available (AFAIK) on a 1992 CD-ROM disc, which was released just as digital video was first starting. The original was only 160x120 pixels, 12 frames per second, so I had spatial as well as temporal challenges. If you search this forum for "celestial," and use my user name, you'll find the post where I asked for help, and see examples of what I started with.

Celestial Navigations: Horses (Storytelling At Its Best) (https://www.youtube.com/watch?v=kI-IcPUV6Gw)

[StainlessS]

Did not find "Celestial" by "JohnMeyer" in D9 search.

Google nails it, 3 links (but two are the same thread)

"Celestial" NEAR "JohnMeyer" site:forum.doom9.org

https://www.google.co.uk/search?source=hp&ei=1fNnXL6WGqmDjLsPt4uS-AY&q=%22Celestial%22+NEAR+%22JohnMeyer%22+site%3Aforum.doom9.org&btnK=Google+Search&oq=%22Celestial%22+NEAR+%22JohnMeyer%22+site%3Aforum.doom9.org&gs_l=psy-ab.3...6811.24098..25309...0.0..0.197.5941.17j32......0....1..gws-wiz.....0..0j0i131j0i10j0i13j0i13i30j33i160j33i21.AHNGvFGtHaQ



https://forum.doom9.org/showthread.php?t=173407

https://forum.doom9.org/showthread.php?t=163014

[johnmeyer]

The doom9.org search just worked for me. I entered "johnmeyer" under "search by user name" and "celestial" for the keyword, and "search entire posts". I got both of those links you posted.

[StainlessS]

Oops, damn you're right, I think I started with search titles only, and forgot to change when failed.

[JackBenRiach]

How to Restoring a Movie If You Have a Picture in It:

https://thumbs2.imgbox.com/aa/d2/ExcafaX1_t.jpg (http://imgbox.com/ExcafaX1)

Thanks for the help.

[manono]

How to Restoring a Movie If You Have a Picture in It:

It's a little unclear what you're asking. However, when I have a bunch of static images (credits for a movie, for example), I'll extract one of the frames, fix it in a photo editor, followed by replacing all of the frames with the one I fixed. Or, if out of the whole range there's a good frame already, then use the FreezeFrame (http://avisynth.nl/index.php/FreezeFrame) command to replace them all with the good one.

The command is written like so:

FreezeFrame(FirstFrame,LastFrame,GoodFrame)

And the entire range gets replaced with the good frame.

[videoFred]

I agree with Manono, I often do the same. Not with Avisynth but with my NLE. But the principle is the same.

Fred.

[johnmeyer]

I got nothing but porn at that photo hosting site when I clicked on the link. Was this a legit question from this first-time poster, or was it spam?

[videoFred]

I got nothing but porn at that photo hosting site when I clicked on the link. Was this a legit question from this first-time poster, or was it spam?

He's OK. :)
I see the film frame, no porn :confused:

Fred.

[StainlessS]

He's possibly not aware of it, I just went there (with great expectation) and saw nothing but a bigger version of the thumbnail, I gots my ad-block on.
EDIT: And so does Fred it seems.

[videoFred]

with great expectation

Haha you can always set your ad-block off :D

Fred

[SnillorZ]

I created my own branch of Fred's script and posted it many years ago. However, like Fred, I haven't tried to post all my changes and updates because I don't have time to support it when people ask questions.

With that as preface, here is what I am using. It is not really fit for publication (i.e., not many comments on the new stuff), and there are probably bugs all over the place. However, it works for me.

It will take two posts to get it all.

:thanks:

Firstly want to send many thanks to videoFred, johnmeyer, StainlessS, Jim Battle (amongst others) for some inspirational work here and many other places across T’Internet.

Apologies upfront that this likely to end up a rambling 1st post, sorry for that and so with that said, please can the last one out turn out the lights and shut the door behind them. :)



I’ve been lurking (stalking people) around this thread and Doom9 for a while now after coming across Jim Battle’s blog posts on 8mm Telecine project, which eventually lead to VideoFred’s work and here. I must have read most of the #1000+ posts in this thread and have enjoyed reading the backstories to people’s projects as much as I’ve merrily ripped off ideas for my own ends.

Why? Well, I’d finally managed to collect together old family 8mm film (standard & super8) distributed around various members of the family, motivated by mum passing away last year. Most of the footage was shot by dear old dad, some by my brothers and even my much younger self on some of the later stuff. Dad was an enthusiastic (family) cine film maker and reels cover from late 50’s through the 1980’s. Some of the early stuff isn’t great quality but got better over time and as better equipment allowed. About 20+ years ago I had a few reels transferred to VHS, which wasn’t great. It was the usual video camera recorded footage of a screen based projection, from the local camera/film shop, but all I could afford at the time. Some years later I got into DV and did some simple editing, back when (expensive) add-on firewire capture cards were required. Having now seen what can be achieved with current software tools, the 100’s of hours of DV footage shot needs to be revisited, but that’s for another day and thread.

Anyway, I decided that it would be great to get all of the worthwhile 8mm film digitally transferred and post edit myself, create a “feature length” out of it and gather all the family together and hire a local digital cinema for a screening. At this point I have to admit that after a couple too many glasses this all seemed a brilliant idea (in my head at least). So I ended up with around 3000’ of 8/S8 and after looking through all the various posts to gather ideas on what to look for and ask for, got some quotes from a few companies providing the service. Ended up getting a 1:1 frame accurate scans in uncompressed AVI’s from a company using MWA Digital Data Scanners. I did get some samples first to check the results and test with my ideas on workflow. I was happy with the results, but with some hindsight of reviewing all the scanned footage, some would have benefited from a wetgate but I’ll live with the decision not to opt for it.

So after many hours of researching AviSynth, AVSpMod, VirtualDub, AVS plugins, etc, I’ve finally got to a point of a working setup after lots of trial & error, hunting down fixes for:- software error’s / my own mistakes / lack of knowledge. I’ve ended up with VideoFred’s 2012 script / JohnMayers 2012 MT script and John’s 2017 Avisynth+. Whilst both 2012 version scripts work well and are giving good first rough cut results, I really like the enhancements that JohnM made in the newer 2017 version.

This is where I would like to ask for some pointers if I may, as despite trying various thread / maxmemory combinations have hit a bit of a performance wall i.e. I’ve not been able to get it above 0.5 – 1 fps! using VirtualDub2 (or 1).

I’ve replicated the results on a couple of different machines (Win7 64 bit AviSynth+ x86, x86 plugins, Quad i7 CPU 32GB RAM. Win 10 Quad i5 3770 CPU 16GB RAM same software)

Probably missing something obvious, but any help would be most appreciated. Also, typically do you run all of the script enhancements in a single pass or multiple runs (are there any penalties for multi)? I realise that this will vary from scene to scene and some will require different/special handling, but in general what have you found is a typical workflow?
e.g. Source>NLE review/cuts>AviSynth processing >NLE?

Apologies again for the long post (for those still awake at this point).:thanks:

[johnmeyer]

The first and most important determinant of performance is the size of your scans. What resolution did your transfer service provide? You might do a test where you compare the quality of what you get from your 1 fps current result with what you get when you scale the video down to something closer to SD resolution. You get very little, if any, benefit from HD resolution with 8mm film, and only a little with Super 8.

Second, you have to be using multi-threading to get more than 1 fps. How this is done depends on the version of AVISynth you are using.

Third, you should turn do your initial work with GamMac turned off (if you are using my latest script) and only do the basic dirt/motion/grain enhancements by specifying option "4". GamMac will slow things down a bit.

Finally, make sure you haven't change DCT in the MVTools2 calls. If it is set to anything other than DCT=0, things can get slow. DCT=1 can do wonders for removing flicker, but I generally don't use it.

There are probably a few other things you can do, but I don't have the time now to write more.

[SnillorZ]

John, appreciate you taking the time to reply :)

So:

1) scans supplied full HD 1920x1080, I'm cropping off black borders (208px L&R) for the depan / stabilise etc and was reducing to 720.

2) Using AVIsynth+ x86 v0.1 r2772 MT, with these from your 2017 script:
#NUMBER OF THREADS
threads=2

#Enable MT! with #Prefetch(No of threads)
Prefetch(2)

4) Yes all the DCT parameters in the script for Denoise are unchanged at 0

3) So reducing to 576 and running Result4 gets me a marginal gain (top 1.3fps) its still around 40+minutes for 50'. Which while not show stopping, I want to check expectations?

P.S. I've also tried Threads 3 & 4 - 3 increases fps again marginally but in the few test runs I've tried hangs VirtualDub2 a few dozen frames in. 4 threads is unstable in my setup and produces memory leaks and crash.

Again really appreciate the time taken to give pointers.

[johnmeyer]

Gosh, with SD resolution I get over 10 fps on my 12-year old 3 GHz i7 computer. Not sure what else to suggest.

[Boulder]

Why not set the number of threads to 4 (or 8 if your CPU supports hyperthreading)? I don't know if the necessary plugins have 64-bit builds, but using an x64 version of Avisynth+ should help as well.

[johnmeyer]

I'm pretty sure quite a few of the plugins will not be happy with AVISynth x64. Also, I doubt that x64 will provide any speed improvement although, since some of the stability problems are related to memory issues, x64 might provide a little more room.

I think the OP said he had stability problems when he increased threads beyond two.

[pinterf]

Regarding plugins that can be used for film restoring there have been two updates lately

RemoveDirt update (https://github.com/pinterf/RemoveDirt/releases), now we have an x64 version of RestoreMotionBlocks. (That was the last missing chain for me for a full x64 workflow)

And made DepanStabilize (https://github.com/pinterf/mvtools/releases) to work with RGB clips (RGB means: planar RGB format of Avisynth+). Motion vectors can be generated from a converted YUV clip, and one can apply stabilization on the original RGB clip. Conversion between planar RGB and RGB24, RGB32 and RGB64 is lossless in Avs+, is is not drawback of using planar RGB.

[SnillorZ]

Why not set the number of threads to 4 (or 8 if your CPU supports hyperthreading)? I don't know if the necessary plugins have 64-bit builds, but using an x64 version of Avisynth+ should help as well.

As per OP, anything 4 or above causes out of memory or VirtualDub to hang.

I'm pretty sure quite a few of the plugins will not be happy with AVISynth x64. Also, I doubt that x64 will provide any speed improvement although, since some of the stability problems are related to memory issues, x64 might provide a little more room.

I think the OP said he had stability problems when he increased threads beyond two.

As John says, the plugins for his script wont load in Avisynth+ x64.

From my own experience having a 64bit application and everything in the chain x64 hasn't provided significant performance gains over running its equivalent x86 app. Admittedly, I've not been attempting something so resource intensive as this though.

@pinterf

Thank you, will take a look at these and see if I can find x64 verisons of the other plugins John's script uses and see if I can get this running and compare to x86 performance.



I've checked that my source files aren't root cause by using the parade example clip from port #1078 to test run John's script and I'm getting same performance. So I must be missing something or have something fundamental in play here, as I get almost the same low fps performance across 2 different machines that have different hardware and OS using the same Avisynth + VirtualDub setups.

Will keep looking and also explore the x64 plugins.

[pinterf]

As per OP, anything 4 or above causes out of memory or VirtualDub to hang.
Yep, more threads - more memory is needed, at least in AVS+. Internal frame caches within Avisynth+ and multiple working buffers for external plugins.

You can use the Avsmeter utility to see the increase in memory consumption for Prefetch 2, 3, ... values. Or set SetMemoryMax manually though it cannot count with the extra memory need for external plugins, still worth playing with it.

[StainlessS]

Also (clutching at straws) suggest use DebugView, see if message from dll requiring Avstp is issued (Google "DebugView", from MicroSoft), execute DebugView and run script, view debugview window.
(I cant say what the error message might be, I'm running encode at the moment and so cannot remove dll to test.
[or just check your plugins directory for dll]
I'm guessin' that some of your filters use avstp (for multithreading), and if absent then can cause all kinds of crashes and such.

Same suggestion to stormy1777 if he sees this.

EDIT: I think message issued by some plugin (rgtools maybe) is something like "Avstp.dll missing, dropping down to single threaded".

EDIT: Could also post result of "AvsMeter avsinfo".



From the Avisynth user point of view, an AVSTP-enabled plug-in requires the avstp.dll file to be installed. Put it in the usual AviSynth 2.5\plugins\ directory, or load it manually with LoadPlugin("path\avstp.dll") in your script. The dll is shared between all plug-ins using AVSTP, so keep only one avstp.dll file in your plug-in set. If you're updating from a previous version, make sure that Avisynth will get access only to the latest one.

If a plug-in requiring AVSTP cannot find the dll file, it could crash, emit an error, or fall back gracefully on a single-threaded mode, depending on its design and implementation. There is no mandatory or pre-defined behaviour for such a case.

The number of threads is automatically set to the number of available logical processors. The thread count can also be controlled via an Avisynth function, so multi-threading can be disabled globally if not desired.

Avstp: https://forum.doom9.org/showthread.php?t=164407
Latest version is v1.0.3

[SnillorZ]

@pinterf @StainlessS Thanks chaps

You can use the Avsmeter utility to see the increase in memory consumption for Prefetch 2, 3, ... values. Or set SetMemoryMax manually though it cannot count with the extra memory need for external plugins, still worth playing with it.

Thank you for the direction to Groucho's tools, this is very useful and significantly speeds up trail & error testing scripts with the different thread, mem & prefetch options.



Same suggestion to stormy1777 if he sees this.

EDIT: I think message issued by some plugin (rgtools maybe) is something like "Avstp.dll missing, dropping down to single threaded".

EDIT: Could also post result of "AvsMeter avsinfo".

Ok making some progress here, StainlessS was right, without AVSTP plugin, Sysinternals DebugView reports:

RemoveDirt 0.9
AvstpWrapper: cannot find avstp.dll.Usage restricted to single threading.

AVSMeter AVSinfo (with -c)


AVSMeter 2.9.1 (x86) - Copyright (c) 2012-2019, Groucho2004

VersionString: AviSynth+ 0.1 (r2772, MT, i386)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 5
Multi-threading support: Yes
Avisynth.dll location: C:\windows\SysWOW64\avisynth.dll
Avisynth.dll time stamp: 2018-12-20, 12:06:28 (UTC)
PluginDir2_5 (HKLM, x86): D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins

PluginDir+ (HKLM, x86): D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins



[CPP 2.6 Plugins (32 Bit)]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\ConvertStacked.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\DirectShowSource.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\ImageSeq.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\Shibatch.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\TimeStretch.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\VDubFilter.dll [2018-12-20]


Custom plugin directory : D:\VideoTools\AVSplugins

[CPP 2.5 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\autolevels.dll [0.6.0.0]
D:\VideoTools\AVSplugins\avstp.dll [1.0.3.0]
D:\VideoTools\AVSplugins\deflicker.dll [0.4.0.0]
D:\VideoTools\AVSplugins\FFT3DFilter.dll [2.1.1.0]
D:\VideoTools\AVSplugins\Motion_06Dec05B.dll [2005-12-06]
D:\VideoTools\AVSplugins\mt_masktools.dll [2.0.32.0]
D:\VideoTools\AVSplugins\MVTools.dll [2008-09-11]
D:\VideoTools\AVSplugins\RemoveDirt.dll [2008-04-01]
D:\VideoTools\AVSplugins\RemoveDirtSSE2.dll [0.9.0.0]
D:\VideoTools\AVSplugins\RemoveGrain.dll [2005-05-01]
D:\VideoTools\AVSplugins\RemoveGrainS.dll [2005-05-01]
D:\VideoTools\AVSplugins\RemoveGrainSSE2.dll [2005-05-01]
D:\VideoTools\AVSplugins\warpsharp.dll [2008-04-06]

[CPP 2.6 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\DePan.dll [2.13.1.3]
D:\VideoTools\AVSplugins\DePanEstimate.dll [2.10.0.2]
D:\VideoTools\AVSplugins\ffms2.dll [2016-12-29]
D:\VideoTools\AVSplugins\FredAverage_x86.dll [0.3.0.0]
D:\VideoTools\AVSplugins\GamMac_x86.dll [1.10.0.0]
D:\VideoTools\AVSplugins\mvtools2.dll [2.7.39.0]

[Scripts (AVSI)]
D:\VideoTools\AVSplugins\FFMS2.avsi [2015-05-22]

[Uncategorized files]
D:\VideoTools\AVSplugins\03_RemoveDirtMC.avs [2010-06-29]
D:\VideoTools\AVSplugins\ffms2.lib [2016-12-29]

So this is getting me up to 2 - 2.3fps avg :sly: woo

I'm still progressing to try and test x64 Avisynth / VDub / plugins:

I've found x64 versions or alternatives that can be used with a modified JohnM script. Just left with 3 to find/swap out:

autolevels.dll

Deflicker.dll

warpsharp.dll - for which I've found AWarpSharp2 but this doesn't contain the unsharpmask function from John's script and I'm not sure what its equivalent is.

Anyway must go and do some paid work for a bit :devil:

[StainlessS]

Firstly, I have not found x64 version of these either.

autolevels.dll
Deflicker.dll
warpsharp.dll

I'll post this first.


[CPP 2.5 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\autolevels.dll [0.6.0.0] # I've got source for this, maybe I'll give it a go for x64.
D:\VideoTools\AVSplugins\avstp.dll [1.0.3.0]
D:\VideoTools\AVSplugins\deflicker.dll [0.4.0.0]
D:\VideoTools\AVSplugins\FFT3DFilter.dll [2.1.1.0] # I have v2.6 dll, fft3dfilter_x86_2_4_0_0.dll (maybe check latest fftw3.dll and libfftw3f-3.dll too, for SysWOW64)
# Does not seem to be used by Fred script
D:\VideoTools\AVSplugins\Motion_06Dec05B.dll [2005-12-06]
D:\VideoTools\AVSplugins\mt_masktools.dll [2.0.32.0]
D:\VideoTools\AVSplugins\MVTools.dll [2008-09-11]

# Remove one of these
D:\VideoTools\AVSplugins\RemoveDirt.dll [2008-04-01] # This is the one in Fred zip (but is slower) [EDIT: Requires MSVCR71.dll runtime in SysWOW64]
D:\VideoTools\AVSplugins\RemoveDirtSSE2.dll [0.9.0.0] # This is faster SSE2 [EDIT: Requires MSVCR71.dll runtime in SysWOW64]
# Missing RemoveDirtS.dll from fred sourcecode/removedirt/removedirt does NOT req (VS2003) MSVCR71.dll runtime in SysWOW64
# But is from 2005.

# Remove Two of these ( *** WARNING Will Robinson ***, Version from Fred script is 1st Oct 2006 - and requires MSVCR71.dll runtime in SysWOW64)
D:\VideoTools\AVSplugins\RemoveGrain.dll [2005-05-01] # Standard compile [EDIT: Probably requires either (Vs2002)MSVCR70.dll or MSVCR71.dll]
D:\VideoTools\AVSplugins\RemoveGrainS.dll [2005-05-01] # S version static linked, dont need runtime
D:\VideoTools\AVSplugins\RemoveGrainSSE2.dll [2005-05-01] # SSE2, faster

D:\VideoTools\AVSplugins\warpsharp.dll [2008-04-06] # Oops, mine seem to be from 2003


MsVCR70.dll and MsVCR71.dll:- https://support.microsoft.com/en-gb/help/326922/redistribution-of-the-shared-c-runtime-component-in-visual-c
[Groucho2004 will probably get a kick out of reading above link again :) ]

Here overrides anything above.

Suggest remove all from Plugins directory, EXCEPT


[CPP 2.6 Plugins (32 Bit)]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\ConvertStacked.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\DirectShowSource.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\ImageSeq.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\Shibatch.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\TimeStretch.dll [2018-12-20]
D:\VideoTools\AvisynthRepository\AVSPLUS_x86\plugins\VDubFilter.dll [2018-12-20]

[CPP 2.5 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\avstp.dll [1.0.3.0]

[CPP 2.6 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\ffms2.dll [2016-12-29]

[Scripts (AVSI)]
D:\VideoTools\AVSplugins\FFMS2.avsi [2015-05-22] # Optional


D:\VideoTools\AVSplugins\ is your general (non fred) plugins directory.

Follow Freds instructions to install (Into suggested directories), eg

D:\Fred_Restore\
helpfiles\
scripts\
sourcecode\

Something like above.

Might need to install install MSVCR71.dll to SysWOW64.

If MSVCR71.dll NOT in SysWOW64, then can find it here(also MSVCR70.dll):- http://www.mediafire.com/file/1220u8g1drmubi8/MSVCR71_70_Runtimes.rar/file

Dependency Walker 2.2(to check if dll needs runtimes):- http://www.dependencywalker.com/

Also can copy/move fftw3.dll to SysWOW64 (for Depan use). [should not then need to be in the script directory, thats where most people put it].


Then suggest go though ALL scripts and where any current relative directory mentioned, prepend with ".\", and Also maybe change path separator slash "/" to "\" (for no real reason)
eg, Loadplugin("plugins/Depan.dll") to

Loadplugin(".\plugins\Depan.dll")

# and

Import(".\plugins\03_RemoveDirtMC.avs")


I've found that Windows 7 seems to have screwed up relative paths somehow and so now avoid by prepending literal current directory relative ".\" thingy.

If you dont keep Fred Plugs (which are old versions) separate from daily use plugins, you will just have problems some other time.
(would be best if scripts fixed up to be working with current plugs, probably be good to get rid of old mvtools).

Gorra get some shuteye, let us know what happens

ta-ta.

[pinterf]

1.) Probably these RemoveDirt dlls can be deleted as well

D:\VideoTools\AVSplugins\RemoveDirt.dll [2008-04-01]
D:\VideoTools\AVSplugins\RemoveDirtSSE2.dll [0.9.0.0]

RemoveDirt_MC uses only RestoreMotionBlocks from this plugin.
I replaced them with RemoveDirt 0.9.2 version (https://github.com/pinterf/RemoveDirt/releases)

2.) RemoveGrain.DLL, RemoveGrainS.DLL, RemoveGrainSSE.DLL:
All these can possibly be replaced by v0.97 RgTools (https://github.com/pinterf/RgTools/releases)

3.) Then I'm using masktools2 (https://github.com/pinterf/masktools/releases/) instead of 2.0.0.32 mt_masktools

4.) MvTools2 (https://github.com/pinterf/mvtools/releases/) instead of 2008 version of mvtools

Note:
do not keep DLLs containing similarly named plugin functions in one autoloaded directory neither load them all manually. Collision happens. E.g. when multiple DLLs containing "clense" are loaded, referencing "Clense" would call either the 15 year old version or the one from the latest SSE2 or AVX2 capable version.

Some of these newer DLLs are using not only SSE2 but SSE4.1 and AVX2 instructions which can give you further speed up when your processor is capable for that. When you bought your machine in the last 5 years, probably that is the case.

[Groucho2004]

AVSMeter AVSinfo (with -c)
@StainlessS
Please note that SnillorZ's "D:\VideoTools\AVSplugins" is not an auto-load directory so none of the functions from DLLs in that directory are added to the function pool during auto-load.

Apart from that - everything that pinterf wrote in the previous post is good advice.

[SnillorZ]

@StainlessS @pinterf @Groucho2004 thank you all.

Yes this:

@StainlessS
Please note that SnillorZ's "D:\VideoTools\AVSplugins" is not an auto-load directory so none of the functions from DLLs in that directory are added to the function pool during auto-load.


I was going to ask StainlessS about his advise:- to check from his experience was Avisynth that sensitive to exact locations of plugins that it could effect performance? Understand from Groucho's comment that SS was referring to auto loading plugins.

Sorry, so to clarify, for all 3 scripts I'm testing they all load the plugins required explicitly - hence using a common "avsplugins" directory with plugins with different versions and runtime requirements etc. I just started working this way as I found it easier to work out which script required what, which plugins caused the scripts to bomb, dll dependencies (thanks Stainless, I'd picked up a recommendation for DepWalker, I think from one of your much earlier posts).

So for VideoFred2012 Script, JohnMeyer2012 (MT) Script and John Meyer2017 (Avisynth+ Gamac) script am loading each plugin explicitly and using the corresponding version of Avisynth (using Groucho's SetAVS)

Also to clarify, the VideoFred2012 script works ok and at a speed that as I understand it could not be improved upon without MT which some of the plugins it uses are not supported.

However I'm preferring the results given by the modifications made in John Meyers scripts - Just checking I have everything set in my environment to get the best performance I can before forging ahead into 3000+ foots worth of footage and finding at a later point in time I could have saved myself a ton of time with a few simple changes. :(


1.)RemoveDirt_MC uses only RestoreMotionBlocks from this plugin.
I replaced them with RemoveDirt 0.9.2 version (https://github.com/pinterf/RemoveDirt/releases)
2.) RemoveGrain.DLL, RemoveGrainS.DLL, RemoveGrainSSE.DLL:
All these can possibly be replaced by v0.97 RgTools (https://github.com/pinterf/RgTools/releases)
3.) Then I'm using masktools2 (https://github.com/pinterf/masktools/releases/) instead of 2.0.0.32 mt_masktools
4.) MvTools2 (https://github.com/pinterf/mvtools/releases/) instead of 2008 version of mvtools


Yes, thank you @pinterf I had already delved into your GitHub to pick up all these:- to get x64 bit replacements/swap outs to those x86 plugins used in johnmeyer 2017 script to test a x64 plugins/Avisynth+ version. Still missing replacements for autolevels.dll & deflicker.dll and what function for AWarpSharp2 would replace the unsharpmask function of warpsharp.dll.

:rolleyes: Doh! I'd not thought of trying swapping out for some of the newer x86 plugins to test for performance increases in the original JM2017 script - I'd focused in on trying to get running in 64bit.
So will try those also and report back.

Hope you managed to get some beauty sleep StainlessS.

Toodle-Pip.

[ChaosKing]

Maybe someone could just port Unsharpmask from warpsharp as a script? Myrsloik did a port for VS https://github.com/myrsloik/WarpSharpSupport/blob/master/wss.py#L28

[pinterf]

And probably I can take care of deflicker and autolevels if they are really used so worth doing it.

[SnillorZ]

:rolleyes: Doh! I'd not thought of trying swapping out for some of the newer x86 plugins to test for performance increases in the original JM2017 script - I'd focused in on trying to get running in 64bit.

So will try those also and report back.

........so modifying the script to use the following newer or replacement plugins with the JM2017 script and Avisynth+ x86 decreases performance further (although not by much, though they perhaps improve on quality?)

AVSMeter 2.9.1 (x86) - Copyright (c) 2012-2019, Groucho2004

VersionString: AviSynth+ 0.1 (r2772, MT, i386)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 5
Multi-threading support: Yes
Avisynth.dll location: C:\windows\SysWOW64\avisynth.dll
Avisynth.dll time stamp: 2018-12-20, 12:06:28 (UTC)
Custom plugin directory : D:\VideoTools\AVSplugins\x86


[CPP 2.5 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\x86\autolevels.dll [0.6.0.0]
D:\VideoTools\AVSplugins\x86\avstp.dll [1.0.3.0]
D:\VideoTools\AVSplugins\x86\deflicker.dll [0.4.0.0]
D:\VideoTools\AVSplugins\x86\warpsharp.dll [2008-04-06]

[CPP 2.6 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\x86\DePan.dll [2.13.1.4]
D:\VideoTools\AVSplugins\x86\DePanEstimate.dll [2.10.0.3]
D:\VideoTools\AVSplugins\x86\fft3dfilter.dll [2.6.0.0]
D:\VideoTools\AVSplugins\x86\FredAverage_x86.dll [0.3.0.0]
D:\VideoTools\AVSplugins\x86\GamMac_x86.dll [1.10.0.0]
D:\VideoTools\AVSplugins\x86\masktools2.dll [2.2.18.0]
D:\VideoTools\AVSplugins\x86\mvtools2.dll [2.7.41.0]
D:\VideoTools\AVSplugins\x86\RemoveDirt.dll [0.9.2.0]
D:\VideoTools\AVSplugins\x86\RgTools.dll [0.97.0.0]

[StainlessS]

@StainlessS
Please note that SnillorZ's "D:\VideoTools\AVSplugins" is not an auto-load directory so none of the functions from DLLs in that directory are added to the function pool during auto-load.

OK, thanks, I thought both were autoload, I dont much bother reading manuals or instructions :(
(I usually point both at same dir, and I dont like putting Avisynth supplied dll's in same dir as others, could cross pollinate dll's when duplicating dll sets to another ver$ avs plugins).

I thought that old version dll's would prompt for futher advice on replace with recent, I expected further posts on that (but thought might be reason for fred/john using old versions).

These are the avs specific dll's that I copy to system32/SysWOW64

AvsRecursion.dll
fftw3.dll # same as below file, name edited only
libfftw3f-3.dll
libsndfile-1.dll


Note also, that the fftw3.dll has probably been updated MANY times since fred supplied, see:- https://forum.doom9.org/showthread.php?t=174470

Pinterf, let me take a crack at AutoLevels x64, it uses DDigit for metrics which has been updated a few times since AutoLevels source supplied version,
If I get into trouble, I'll holla.

Oodles and Oodles of Toodles :)

EDIT: I would probably not be ambitious enouigh yet to attempt avs+ colorspaces for AutoLevels, but let me do standard v2.6 version, and if you wish to add avs+ colorspace stuff then
you can do mod on that.

[pinterf]

Pinterf, let me take a crack at AutoLevels x64, it uses DDigit for metrics which has been updated a few times since AutoLevels source supplied version,
If I get into trouble, I'll holla.

Oodles and Oodles of Toodles :)

EDIT: I would probably not be ambitious enouigh yet to attempt avs+ colorspaces for AutoLevels, but let me do standard v2.6 version, and if you wish to add avs+ colorspace stuff then
you can do mod on that.
Good news, you won the project :)

[pinterf]

........so modifying the script to use the following newer or replacement plugins with the JM2017 script and Avisynth+ x86 decreases performance further (although not by much, though they perhaps improve on quality?)

I'm really interested in which replacement is causing you the bottleneck. And what script options are you using (stabilization, autolevels, etc..), degrain on/off?

[SnillorZ]

@pinterf

Sorry, trying to follow multiple suggestions as methodically as I can without ending up with retrograde steps,

so please hold the line caller. :D

To clarify though all tests as per @johnmeyer suggestion Result4/Comparison4 - So no Autolevels or Colour Corrections. Just DirtRemoval, RemoveGrain, Stabalize, etc



These are the avs specific dll's that I copy to system32/SysWOW64

AvsRecursion.dll
fftw3.dll # same as below file, name edited only
libfftw3f-3.dll
libsndfile-1.dll

Note also, that the fftw3.dll has probably been updated MANY times since fred supplied

I'd overlooked the AVS sys32/wow64 required .dll's - I've updated fftw3/libfftw3f-3 to 02/05/2018 version linked by Goucho in the thread you show.

No performance improvement though.


However, in light of this I've gone back to testing some amendments to JohnMeyer2012 script (using AVS2.6MT) and managed to get 4+fps. Using same "script relevant" plugins as post #1111 - with the exception of RgTools.dll [0.97.0.0] which seems to cause a crash with 2.6MT? Encouraging uptick though.

[SnillorZ]

@videoFred, @johnmeyer, @StainlessS, @pinterf, @Groucho2004

Just wanted to say that in 20+ years of using forums (for both profession & personally) I've rarely come across a more welcoming and helpful bunch of members.

In fact, sadly more often than not, the experience is the complete opposite and once you admit to being a noob and lurking around a thread and then dain to ask a question, find yourself be chased down the road by the flaming torches & pitchforks wielding mob.

Anyway, just wanted to make that point. If any of you found yourself in my part of the world (English Lake District (https://www.lakedistrict.gov.uk/wow)) would happily buy you a beer

Cheers

P.S. If you see my boss, you've not seen me. If you see my wife, tell her cant come away from the computer just yet as "I've found someone who's wrong on the Internet" :D

[StainlessS]

chased down the road by the flaming torches & pitchforks wielding mob.
Still plenty of time for that, dont rush us.

Bout 35 years ago, was half way between between Newcastle and Carlisle (English Lake District), stopped at pub in middle of nowhere and went in for a beer.
Manager told me that I should have been there the day before.
Why, I asked.
Because, they heard a terrlible loud noise in back garden (pretty much nothing [EDIT: except Kielder Forest of course] for 40/50/60 miles up to Scotland, big garden),
and looking out of window saw a helicopter landing there. Out jumped Prince Charles and a handfull of bodyguards/lackies.
Somebody wanted a piss, so they stopped for a rest period, and a pint.
Nice to know that the other half have to piss as well, I thought that they just 'glistened', rather a lot.

EDIT: Lovely day here in The Big Smoke, wish I could escape from this pub and go start on AutoLevels x64 mod.

[SnillorZ]

:D @StainlessS

I wouldn't have been surprised if you'd said Charlie and T'other off-comers in their big metal bird had been chased off.

Some places up here still like "Deliverance" country and you have to be careful where one puts one's banjo.

[SnillorZ]

@johnmeyer

............Third, you should turn do your initial work with GamMac turned off (if you are using my latest script) and only do the basic dirt/motion/grain enhancements by specifying option "4". GamMac will slow things down a bit.

Please can I revisit your advise above, whilst still testing performance tweaks?

Can I clarify, after doing a R4 processing with your script, are you saying then run the result (VirtualDub>Save As>Uncompressed avi) through a script process using R1/2/3 (as preferred)?

Using same script i.e. letting Stabilise/Denoise/Sharpening process again?

or use a modified version to comment out some of these (I note your comment GamMac introduces some flicker)

:thanks:

[pinterf]

...with the exception of RgTools.dll [0.97.0.0] which seems to cause a crash with 2.6MT?
Could you please try it with 0.97 again, this time specifying ", align = true" additional parameter for the Crop line?

[SnillorZ]

Could you please try it with 0.97 again, this time specifying ", align = true" additional parameter for the Crop line?

Sorry not certain which crop line you mean?

You mean like this:

cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot,align = true) #temporarily remove any black borders on input video

If so the script still bombs the program(s) (e.g. AVSMeter)

[StainlessS]

Align = true,

its lovely that avs+ made things a bit faster/better,
but woulda been even nicer if mod whatever were not an additional issue.


Lots more work for Devs, but who the hell cares bout them !

[pinterf]

Sorry not certain which crop line you mean?

You mean like this:

cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot,align = true) #temporarily remove any black borders on input video

If so the script still bombs the program(s) (e.g. AVSMeter)

Yep, that's it. I'm not sure if there are other Crop commands around.

[SnillorZ]

Anyway, last post for today.

Quick example of a rough cut taken from the script tests:

Using @johnmeyer (VideoFred) script 2017 (Avisynth+ GamMac, etc)

https://youtu.be/UD0tTJLIrK8

[pinterf]

Yep, that's it. I'm not sure if there are other Crop commands around.
So, there were two other Crops in the script.
Using AVS 2.6 and Crop without align=true I got the message from RgTools 0.97:
"RemoveGrain: Invalid memory alignment. Unaligned crop?"
Put the missing ", align=true" lines and the script was working for Avisynth 2.6 as well.

Regarding the slowish avs+ 32-bit run:
Even for a single thread Avisynth+ needs 1300M memory for a basic out-of-box side-by-side comparison script (if not limited). My sources are 1600x1200 scans.

In Avisynth+ when the size of internal frame caches have to be shrunk due to low memory, it has much bigger penalty on speed than it happens in classic Avisynth 2.6.

Quick AvsMeter tests with Avisynth+ 2772 x86, using the formerly attached "JM2017" script:

Single thread SetMemoryMax(1024): 0.26 fps
Single thread SetMemoryMax(2048): 0.46 fps (actually 1350M used)
Prefetch(2) SetMemoryMax(1024): 0.43 fps (actually used 1384M)
Prefetch(2) SetmemoryMax(2700): 0.50 fps (actually used 1770M)

In this case an x64 workflow and thus the virtually limitless memory can help a lot for Avs+. Such videos with full-HD resolution and a complex filter chain require huge amount of memory. Too bad there are still filters without x64 version, but it's just a question of time to have them.

(Did not run x64 tests because of the missing unsharpmask function)

[SnillorZ]

So, there were two other Crops in the script.


Thank you, found those now.

So now testing John's 2012 script using AVS2.6 MT and the newer plugins that are available, gives the following with 4 Threads and 2048M. On the Parade Example AVI (1920x1080 16fps) with Result S4

AVSMeter 2.9.1 (x86) - Copyright (c) 2012-2019, Groucho2004
AviSynth 2.60, build:Feb 20 2015 [03:16:45] (2.6.0.5)

Number of frames: 241
Length (hh:mm:ss.ms): 00:00:15.063
Frame width: 2560
Frame height: 720
Framerate: 16.000 (16/1)
Colorspace: YV12
Active MT Mode: 2

Frames processed: 241 (0 - 240)
FPS (min | max | average): 1.073 | 122698 | 3.818
Process memory usage (max): 2507 MiB
Thread count: 17
CPU usage (average): 77.3%

Time (elapsed): 00:01:03.122

Thank you for the detail regarding the differences in memory handling between Avisynth+ and earlier versions. This has helped me understand why the significant difference in performance between the latest version of the cine film restore script and earlier ones.

So, in a better position now with everyone's input, I am hoping you marvelous code mechanics can come up with the x64 ports of the few remaining external plugins required to give a full 64bit test a run.

Understand that you will all have other priorities (days jobs to do), so no rush, I'll just keep a watching brief here. :cool:

[pinterf]

And here is Deflicker 0.5 for x64. Not optimized yet for x64 but at least exists.
https://github.com/pinterf/Deflicker/releases

[StainlessS]

Ooooooo Lovely, thanx P.

[SnillorZ]

Ooooooo Lovely, thanx P.
What he said +1 :thanks:

[videoFred]

Anyway, last post for today.

Quick example of a rough cut taken from the script tests:

Using @johnmeyer (VideoFred) script 2017 (Avisynth+ GamMac, etc)

https://youtu.be/UD0tTJLIrK8

Picture quality is very good! What transfer system have you used?

But aspect ratio is not good: it looks to narrow.
Also, if there is no flicker in the original scan, you can disable deflicker() in the script.

Fred.

[SnillorZ]

Picture quality is very good! What transfer system have you used?

But aspect ratio is not good: it looks to narrow.

Also, if there is no flicker in the original scan, you can disable deflicker() in the script.

Fred.

Hi Fred and thank you. Used a "transfer" company here in UK, they said the equipment they use is from the German company MWA-Nova, using MWA "Vario" Flashscan Digital Data Scanners.

I agree on the aspect ratio. This was from an initial test, based on recommendations from Johnmeyer Post #1095 (https://forum.doom9.org/showthread.php?p=1874708#post1874708) for improving fps processing of the scripts and Avisynth, he was making the point that 8mm would see little benefit from anything above SD resolutions, so this example was done at 1024x576, but yes I think with the borders added back it makes the frame too "square". Have you found something that works best for standard 8 and is it the same for Super 8? (most of the footage I have is standard 8). Scans have been supplied full HD 1920x1080.

Its funny, I have some footage from early 60's (like this example) that the original film quality is still pretty good, yet some much later stuff (80's) that has aged nowhere near as good, down to handling and equipment I guess. So some footage will require a lot more attention than others.

Sadly I've come across a couple of bits of footage that have been double exposed and I think will have to be consigned to the "too difficult" pile as they will be way beyond my skill set.

Good to hear from you, it was seeing examples of your work that provided the inspiration for doing this.

[johnmeyer]

Your fps, at around 3.5, is starting to look reasonable if your input is really:

Frame width: 2560
Frame height: 720That 2560 width is what you reported above. Is that really what is being fed to the script?

You want to do two things if you are going to somewhat reduce the resolution prior to feeding the result to your script. The first is to crop off any black bars on the side. Even if you are going to keep everything at HD resolution, it is pointless to process all those black/blank pixels. Which brings me to a key point:

You never want to feed black borders into any of these scripts (see my last comment below).

If you still want to process in HD, crop first to remove all borders (including any on top and bottom); then run the script; and then at the end, pad out the result by adding pixels on the side until you get to a standard 1920x1080 resolution. Getting rid of borders should make a huge difference in performance.

So, if you want to process in a lower resolution, resize down to the lower resolution after doing the crop recommended above. Don't include any borders in the resize operation.

If you do re-size, you'll want to pay attention to the aspect ratio of the film compared to the aspect ratio of standard PAL or NTSC SD video. I actually added some parameters for this, which I don't think Fred ever added to his script. I did this primarily for handling 8mm film which is almost perfectly square, unlike NTSC SD video which at 656x480 (square pixel) or 720x480 (non-square pixel) is 4:3. So when you deliver 8mm scans using the SD DVD format, you will always end up with black bars on the side. It's just the way the math works. However, these inevitable borders must be temporarily stripped out before feeding the film transfer to the script, and then added back. I did this with the "in_bord_left" etc. border variables. It lets you easily remove borders and then, as the very last step, adds back black borders so you end up with the same video size you started with, but without ever processing the black pixels.

This is needed not only to reduce processing time, but also to avoid having funky borders from the motion stabilization. If you leave the black borders, the black from the borders will wander into the frame of the final video.

P.S. Your scans look good, although it looks like you may be losing some highlights (that is the fault of the scan) and it also looks like you might need to do more work in post on gaining the shadows, as exhibited by the walk in the square starting at the 0:13 second mark. I do scene-by-scene gamma adjustments in Vegas, using a custom histogram, but others have used a combination of HDRAGC and GamMac in AVISynth to do this automatically. I posted my GamMac results, but I don't have it working well enough to rely on, so I run the script twice: once with, and once without GamMac and then cut between them just before I deliver the final result.

I need to get back to trying HDRAGC. It has been upgraded a lot since I first tried it a decade ago. It produce some really good results, but was prone to go AWOL on some scenes and really mess things up. When it was working, this is what the results looked like:

HDRAGC good results (https://forum.doom9.org/showpost.php?p=868799&postcount=138)

[StainlessS]

Also, vast black borders will screw with any auto color correction stuff.

[SnillorZ]

That 2560 width is what you reported above. Is that really what is being fed to the script?


John that was just a test run based off a @pinterf advised change to the Crop function. Script will have been pointing to a S4 result as the source i.e. 2 x 1280 side by side.

Your comments on borders sort of brings me back to my question up thread which think you've perhaps missed?

@johnmeyer
Please can I revisit your advise above, whilst still testing performance tweaks?

Can I clarify, after doing a R4 processing with your script, are you saying then run the result (VirtualDub>Save As>Uncompressed avi) through a script process using R1/2/3 (as preferred)?

Using same script i.e. letting Stabilise/Denoise/Sharpening process again?

or use a modified version to comment out some of these (noting your comment GamMac introduces some flicker)

to confirm source scans are 1920x1080 with 208px border left & right on the standard 8 scans and 172px Super8 (I've adjusted these values for Mod8 etc).

To confirm with testing the scripts, have the border crop functions:

in_bord_left=208 in_bord_top=0 in_bord_right=208 in_bord_bot=0 #Borders around input that must be removed

cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #temporarily remove any black borders on input video

... as I discovered without you end up with side to side "frame weave" or funky borders as you describe.

However are you saying here If you still want to process in HD, crop first to remove all borders (including any on top and bottom); then run the script; and then at the end, pad out the result by adding pixels on the side until you get to a standard 1920x1080 resolution. Getting rid of borders should make a huge difference in performance. that the borders should be removed first outside of your script and not using the crop function as shown above? Sorry I'm a little confused.

Yes I anticipate that when I get into the details of the post edit I'll have scene by scene correction to do. I've never used Vegas and likely will use PPro as I have some experience with it, although some time ago but at least GUI is familiar. So anticipating having to cut some scenes for further AVS processing or something similar in PPro to adjust.

Interested about your comment that some have managed this automatically though. As I know I have a lot of scenes to deal with like the example you picked out - Bless dear old Dad, he did used to like to get his moneys worth by shooting with the same speed film in camera indoors & out, guess film was a good chunk of a working mans wage back then. :)

[StainlessS]

John probably just means that there is no included pre-cropping in given scripts, so add your own after source filter and before rest of script, you
can just insert a Return Last before the rest of script whilst getting coords correct.

EDIT: Or Return Source, or whatever the cropped clip is called.

[pinterf]

And here is Deflicker 0.5 for x64. Not optimized yet for x64 but at least exists.
https://github.com/pinterf/Deflicker/releases
v0.6 version is ready, SSE2 and AVX2 support in both x86 and x64 version.

[SnillorZ]

Sorry that's why I'm confused, as I'm reading them as John's scripts do include pre-processing crop: as all his scripts have these relevant parts (have only changed the relevant bord pixel parameters):

#VIDEO FILE
#Change the following line to point to your video file
film="D:\VideoSource\**SOURCEFILE**.avi"

...........>

#SIZE, CROP AND BORDERS PARAMETERS
CLeft=4 CTop=4 CRight=4 CBottom=4 #crop values after Depan and before final resizing
W=1280 H=720 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=208 in_bord_top=0 in_bord_right=208 in_bord_bot=0 #Borders around input that must be removed

...........>

source= Avisource(film).killaudio().assumefps(play_speed).trim(trim_begin,0).converttoYV12()
trimming= framecount(source)-trim_end
source1= trim(source,0,trimming)

cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot,align = true)

or am I misunderstanding and just outed myself as a noob? is it pitchforks and flaming torches time now StainlessS?

[StainlessS]

is it pitchforks and flaming torches time now StainlessS?

No not yet, busy at present, maybe tomorrow.

I muchly doubt that john expects you to do an entire pass just for cropping, do as you deem fit.

[johnmeyer]

The inborder = 208 should do the trick. 208 is mod 4 which, I have found, makes the script "happier."

[StainlessS]

John, Fred,

Anybody ever use AutoLevels(autoGamma=true), if so any problems crashes at all ?

Kaboom!

K=BlankClip.Trim(0,-100)
W=K.Invert
K++W++K++W++K++W++K++W
#AutoLevels(autolevel=true,debug=true)
AutoLevels(autogamma=true,debug=true)
return last




Autolevels::adjust() called with gamma=0.0, and so 1/gamma produces Divide by Zero..

I probably will not get time to look further today, but maybe get back to it tomorrow.
Got it compiling x64 OK, thought that I had borked it, but bug in original dll too.

EDIT: Seems to only be RGB24 and RGB32.

[SnillorZ]

But aspect ratio is not good: it looks to narrow.
Fred.

The inborder = 208 should do the trick. 208 is mod 4 which, I have found, makes the script "happier."

I've realised my schoolboy error - In doing 2 processing steps (for performance - as per up thread) I have the original source borders cropped Result 4 processing but I'd not set what the output borders should be based on the resizing and not realised that if you don't set these you end up with the same original border size added back to the (now) resized output, which is obviously wrong. Then in processing step 2 (levels, colours or whatever) using the same border crop values as in step 1 reduces the actual frame size further. So this was my mistake.

If someone can please check my math(s):

Orginal scan source = 1920x1080 (Standard 8mm) with Left & Right borders gives an actual video frame size width 1504.

Step 1 process script Result 4 (denoise, stabalise, etc) Crop borders 208 left & right. Resize to 1280x720 and set resize borders to give 140px borders and 1000 frame width (being 2/3ds of original 1920 & values adjusted to Mod4)

W=1280 H=720 #final size after cropping
bord_left=-68 bord_top=0 bord_right=-68 bord_bot=0 #720p adjusted borders
in_bord_left=208 in_bord_top=0 in_bord_right=208 in_bord_bot=0 #Borders around input that must be removed

Step 2 process script Result 1 (Levels, Colours) output of Step 1. Crop borders 140 left & right. No further resize or borders resize.

W=1280 H=720 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0
in_bord_left=140 in_bord_top=0 in_bord_right=140 in_bord_bot=0 #Borders around input that must be removed

I've redone the test samples and posted them here:

Using johnmeyer script 2012 vs 2017 (GamMac) script for comparison. Both do now use latest external plugins available. :thanks: @pinterf

Result1 AVS2012 Script (https://vimeo.com/338231281)

Result1 AVS2017 Script (https://vimeo.com/338233017)

Just to note I've updated both of the above scripts to use v0.6 Deflicker:

v0.6 version is ready, SSE2 and AVX2 support in both x86 and x64 version.

[SnillorZ]

Just for completeness the AVS2017 (Avisynth+) clip vs the original

Original Source vs AVS Processed (https://vimeo.com/338253419)

Yes scene 2 (0:0:13:0+) needs further work. :p

[johnmeyer]

Anybody ever use AutoLevels(autoGamma=true), if so any problems crashes at all ?Yes. In fact, if you go to page one of this 58-page thread, you'll see that VideoFred's original script used it. I never liked the results, however.

[videoFred]

Hi Fred and thank you. Used a "transfer" company here in UK, they said the equipment they use is from the German company MWA-Nova, using MWA "Vario" Flashscan Digital Data Scanners.

Yes, the Flashscan. It gives good overall results but a bit over saturated and not much detail in the shadows.


Its funny, I have some footage from early 60's (like this example) that the original film quality is still pretty good, yet some much later stuff (80's) that has aged nowhere near as good, down to handling and equipment I guess. So some footage will require a lot more attention than others.

Good old regular-8 was top quality. All metal gear, very good film stock and the pressure plate was all metal, in the camera (like single-8) . Super-8 was more plastic gear and less quality filmstock. But more affordable for the common man.

Fred.

[videoFred]

John, Fred,

Anybody ever use AutoLevels(autoGamma=true), if so any problems crashes at all ?

I'm not using it any more because Gammac, when good used, gives much better results.

Fred.

[videoFred]

Orginal scan source = 1920x1080 (Standard 8mm) with Left & Right borders gives an actual video frame size width 1504.

Is it possible to upload a short fragment from the original scan? Just like it was when you got it from the transfer company with black borders and not manipulated.

Fred.

[SnillorZ]

Is it possible to upload a short fragment from the original scan? Just like it was when you got it from the transfer company with black borders and not manipulated.
Fred.

Yes sure, can I upload to Vimeo as source without them re-processing? Just signed up with a basic account and I'm unfamiliar with it yet and don't seem to see an option for this with a basic account?

If not I can host it elsewhere, and will upload later if Vimeo isn't an option.

I've used VirtualDub and saved as avi with no conversion options (uncompressed/unchanged codec or colourspace). Looking at the result file details looks the same, so I'm assuming this is left the same (apart from length)?

[Cary Knoop]

Yes sure, can I upload to Vimeo as source without them re-processing? Just signed up with a basic account and I'm unfamiliar with it yet and don't seem to see an option for this with a basic account?

Vimeo basic does not support upload.

[SnillorZ]

Is it possible to upload a short fragment from the original scan? Just like it was when you got it from the transfer company with black borders and not manipulated.
Fred.

Here's a copy on Google Drive

Standard8_Scan_16FPS.avi (https://drive.google.com/file/d/1z9ofd7X2sncI0Lsso_sss_S6T2S6SaTL/view?usp=sharing)

[StainlessS]

SnillorZ,
Thats a nice bit of footage you posted, guessin' that Fred's script will work wonders when all plugins available in hit bit.
I've been having one or two problems doing 64 bit AutoLevels, fixed at least 1/2 dozen bugs in source, but currently got a bug that is
proving a little difficult to track down (maybe I caused in a bug fix), trouble is, the original plugin crashes with test clips to try locate
problem, so doing comparisons is not so easy. I'll try get it functional soon.

[SnillorZ]

StainlessS,

Thank you really appreciate the effort, with 200k+ frames to process and using the better resulting version of script @ avg 2fps is a little slow going. Got a machine on night shifts.

So hoping that being able to run things without the same address space restrictions of whats currently available will see good improvements.

SnillorZ,
Thats a nice bit of footage you posted, guessin' that Fred's script will work wonders when all plugins available in hit bit.


Sadly not all the film produced scans of this quality, so some will need more TLC.

Apologies for another noob question. The 3rd "non- 64bit" available plugin is warpsharp and I think the following suggestion got lost along the way:
Maybe someone could just port Unsharpmask from warpsharp as a script? Myrsloik did a port for VS https://github.com/myrsloik/WarpSharpSupport/blob/master/wss.py#L28

So if I'm understanding correctly, the suggestion is to just get the unsharpmask function from the warpsharp.dll and create a native AVS function in the script to do the same instead, rather than compile a whole new 64bit warpsharp?

Does the code snippet from the Myrsloik link give all the pointers to what would be needed to do this?

[ChaosKing]

So if I'm understanding correctly, the suggestion is to just get the unsharpmask function from the warpsharp.dll and create a native AVS function in the script to do the same instead, rather than compile a whole new 64bit warpsharp?

Does the code snippet from the Myrsloik link give all the pointers to what would be needed to do this?

yes, because many scripts often only need these two functions. (I don't know what would be easier)
Oh and I also noticed that older warpsharp dlls are super slow. This one should be the fastest I guess http://xhmikosr.1f0.de/_old/avisynth/plugins/warpsharp-v1.0.zip

[StainlessS]

Yes. In fact, if you go to page one of this 58-page thread, you'll see that VideoFred's original script used it. I never liked the results, however.

Pinterf (or anyone else),

this dont seem right to me, from AutoLevels source


double inv_coring_func(double y) { return y * (255.0/219.0) + 16.0; } // ssS: This Dont seem right, scaled UP and then ADD 16
//double inv_coring_func(double y) { return y * (219.0/255.0) + 16.0; } // ssS: Should it be ???

Any opinion ?

EDIT: I'm converting all float's in source to double, and accumulators to int 64, else suspect problems on big frames.

EDIT: A bit more.


// use user specified input high/low points, if specified.
// if coring is specified but input high/low aren't specified,
// we must do the inverse mapping to counteract the mapping that
// gets done inside adjust().
double in_low = (force_input_low) ? double(input_low)
: (do_coring) ? inv_coring_func(ymin_avg)
: ymin_avg;
double in_high = (force_input_high) ? double(input_high)
: (do_coring) ? inv_coring_func(ymax_avg)
: ymax_avg;

DPRINTF("GETFRAME: Calling adjust")

// use levels() code to do remapping
adjust(frame, int(in_low+0.5), do_gamma, int(in_high+0.5),
int(output_low+0.5), int(output_high+0.5), do_coring);


EDIT: And near beginning of adjust()

if (coring)
p = ((i-16)*(255.0/219.0) - in_min) / divisor; // ssS: subtract and scale up, so maybe inv coring should be scale down and add


EDIT: I'de like some verification as I'm a little timid about assuming that it is so wrong-ish. :confused: [ EDIT: And I'de like someone else to blame :) ]

[SnillorZ]

Oh and I also noticed that older warpsharp dlls are super slow. This one should be the fastest I guess http://xhmikosr.1f0.de/_old/avisynth/plugins/warpsharp-v1.0.zip

Chaos, thank you tried this in earlier tests with Fred's/Johns scripts but that version seems to have the Unsharpmask function missing from it.

[pinterf]

Pinterf (or anyone else),

this dont seem right to me, from AutoLevels source


double inv_coring_func(double y) { return y * (255.0/219.0) + 16.0; } // ssS: This Dont seem right, scaled UP and then ADD 16
//double inv_coring_func(double y) { return y * (219.0/255.0) + 16.0; } // ssS: Should it be ???

Any opinion ?

You are right, if PC->TV was the intended action, that should limit 0-255 to 16-235.

EDIT:
See coring, option is not recommended
http://avisynth.nl/index.php/Levels

[pinterf]

There are other sharpeners (as other deflickers as well), someone with plenty of time :) could try them, e.g. http://avisynth.nl/index.php/AWarpSharp2

[StainlessS]

Thanks P, had already been looking at that wiki post earlier. In that Wiki post, this line with link [EDIT: Pinterf link repeated, Levels() on Wiki:- http://avisynth.nl/index.php/Levels ]

In the opinion of some, you should always use coring=false if you are working directly with luma values (whether or not your
https://forum.doom9.org/showthread.php?p=1722885#post1722885

Produces a "Invalid Thread specified. If you followed a valid link, please notify the administrator".

I suspect that the bad link post might have been a quote of mine, I dislike default coring=True intensely, and have posted such on occasion.

EDIT: ChaosKing linked Warsharp zip has only WarpSharp + XSharpen in it.

extern "C" __declspec(dllexport) const char* __stdcall AvisynthPluginInit2(IScriptEnvironment* env)
{
env->AddFunction("WarpSharp", "c[depth]i[threshold]i", Create_WarpSharp, 0);
env->AddFunction("XSharpen", "c[strength]i[threshold]i", Create_XSharpen, 0);
return 0;
}

[ChaosKing]

EDIT: ChaosKing linked Warsharp zip has only WarpSharp + XSharpen in it.

extern "C" __declspec(dllexport) const char* __stdcall AvisynthPluginInit2(IScriptEnvironment* env)
{
env->AddFunction("WarpSharp", "c[depth]i[threshold]i", Create_WarpSharp, 0);
env->AddFunction("XSharpen", "c[strength]i[threshold]i", Create_XSharpen, 0);
return 0;
}

Whoops maybe be this is why it was fast, I tested only warpsharp() back then :p

[ChaosKing]

Oh I forgot that a x64 warpsharp version exists :D http://chaosking.de/repo/avsfilters/Adjustment_Filters/Sharpeners/
(including unsharpmask)

[pinterf]

Anyway, a quick and dirty UnsharpMask replacement for Avs+

function UnsharpMask(clip clip, int "strength", int "radius", int "threshold")
{
strength = !defined(strength) ? 64 : strength
radius = !defined(radius) ? 3 : radius
threshold = !defined(threshold) ? 8 : threshold

# replace this with your blurring function
blurclip = clip
for (i=1, radius) {
blurclip = blurclip.Blur(1.0)
}

e = "x y - abs " + String(threshold) + " scaleb > x y - " + String(strength/128.0)+ " * x + x ?"
if (clip.IsRGB()) { # planar RGB only
return Expr(clip, blurclip, e) # Apply expr to all RGB planes
} else {
return Expr(clip, blurclip, e, "", "") # U and V are copied
}
}

[ChaosKing]

I got some unexpected results!

1080p source:
UnsharpMask(64, 3, 8) # 64bit - 82fps (prefetch makes it slower) [script version]
UnsharpMask(64, 3, 8) # 32bit - 124fps (prefetch makes 32bit also slower) [script version]
UnsharpMask(64, 3, 8) # warpsharp.dll 64bit - 57fps

Why is 32bit so much faster?

32/64 bit latest 2772 version. Tested with vdub2. Ryzen 2600 cpu 6 cores 12 threads.

EDIT
UnsharpMask(64, 3, 8) # warpsharp.dll 32bit - 57fps, with prefetch(2) 110fps, freezes with prefetch(4)
EDIT2
Now it seems to work with prefetch(4) -> 165fps

Edit3 for completeness
Vapoursynth version ~194fps

[SnillorZ]

Anyway, a quick and dirty UnsharpMask replacement for Avs+

@pinterf As previously, many thanks :goodpost:

Tested with AVS+ and my updated version of the VideoFred / JohnMeyer script and works fine in place of the removed warpsharp.dll.

For anyone else watching, no performance gain with this in itself, but that's wasn't the intention, its to allow a move of workflow to full x64.

..on that note by my reckoning just need the autolevels x64 now to give this a test. Looking at you StainlessS :sly: C'mon get your finger out :D

If it helps any with testing or debugging let me know. Not sure what use I would be mind as I'm absolute Rank Amateur as you can tell.


X-Post with Chaos: I got some unexpected results! Why is 32bit so much faster?

This was discussed upthread and @pinterf gave detail in #1102 (https://forum.doom9.org/showthread.php?p=1874904#post1874904)

[pinterf]

Edit3 for completeness
Vapoursynth version ~194fps
Actually the difference is in the blurring which is a slow N pass one in the posted example, plus this quick and dirty version by no means brings you the same results as either aWarpSharp.UnsharpMask or vs.UnsharpMask, just a skeleton concept.

In either case, fine tuning has to be done using our eyes.

[pinterf]

I got some unexpected results!

1080p source:
UnsharpMask(64, 3, 8) # 64bit - 82fps (prefetch makes it slower) [script version]
UnsharpMask(64, 3, 8) # 32bit - 124fps (prefetch makes 32bit also slower) [script version]
UnsharpMask(64, 3, 8) # warpsharp.dll 64bit - 57fps

Why is 32bit so much faster?

I have no such difference, x64 is faster in general.
To rule out the differently built source filter, could you try with ColorBars(1920, 1080, pixel_type = "YV12")

EDIT: We don't have memory issues here. Prefetch may cause slowdown on the source filter side, ruining the nice linear access of frames. (?)

[ChaosKing]

I have no such difference, x64 is faster in general.
To rule out the differently built source filter, could you try with ColorBars(1920, 1080, pixel_type = "YV12")

EDIT: We don't have memory issues here. Prefetch may cause slowdown on the source filter side, ruining the nice linear access of frames. (?)

Nope with Colorbars it is also 152fps vs 83 fps

EDIT
I think it is Vdub2, Avsmeter 32/64 are showing about the same speed


Frames processed: 2001 (0 - 2000)
FPS (min | max | average): 100.3 | 194.3 | 189.0
Process memory usage (max): 74 MiB
Thread count: 18
CPU usage (average): 8.0%

Time (elapsed): 00:00:10.589


AVSMeter 2.8.8 (x64) - Copyright (c) 2012-2018, Groucho2004
AviSynth+ 0.1 (r2772, MT, x86_64) (0.1.0.0)


Frames processed: 2001 (0 - 2000)
FPS (min | max | average): 115.8 | 195.3 | 191.6
Process memory usage (max): 68 MiB
Thread count: 18
CPU usage (average): 8.0%

Time (elapsed): 00:00:10.443

with source filter 13.9sec vs 14.6 sec. So all good now :-) And thx for the port!

[Reel.Deel]

Actually the difference is in the blurring which is slow N pass one in the posted example, plus this quick and dirty version by no means brings you the same results as either aWarpSharp.UnsharpMask or vs.UnsharpMask, just a skeleton concept.


Any possibility of porting BoxBlur to AVS+? Then we could use it like the VS version (https://forum.doom9.org/showthread.php?p=1806816#post1806816). :devil:



Sidenote:
Anyone using Warpsharp x64 filter from the wiki: the output of Warpsharp() x64 is different then the x32 version (https://forum.doom9.org/showthread.php?p=1794821#post1794821). I only know of that one problem so far, but I know that plugin has a lot of filters.

[StainlessS]

AutoLevels v0.7 update (avs v2.58, avs/+ v2.6 x86 & x64)

See HOSTED directory @ MediaFire in sig below this post.

Req VS 2008 Runtimes.

https://forum.doom9.org/showthread.php?p=1875869#post1875869

8 bit only.

EDIT: Gone t' pub.

[StainlessS]

RD,
There has been a script BoxBlur posted, no idea if any good (and no idea what boxblur does):- https://forum.doom9.org/showthread.php?t=100731&highlight=boxblur

[bassquake]

Hi Fred (or anyone else who might know the answer)

I have an issue with the autowhite:

* I'm getting very yellow whites. But if I don't crop the image, it looks better if a bit too blueish. If I put the ".coloryuv(autowhite=true)" in directly on the source import line so its:

source1= ImageSource(file=film, start=trim_begin, end=trim_end, fps=play_speed).FlipVertical().converttoYV12().coloryuv(autowhite=true)

it works so it seems to be something to do with the cropping.

Have attached a screenshot with the yellowing result on the right. The original is left.

Is it working better on uncropped because of the light through the sprocket holes?

[videoFred]

Hi Bassquake,

We do not see your picture.
Please upload it somewhere.

But, in general, black borders and white sprocket holes will mess up autowhite.
If in your case autowhite is better with white sprocket holes, then I think you have a scene with lots of water and blue sky or something?

Fred.

[StainlessS]

bassquake,

Can post image on PostImage.org without an account,
https://postimages.org/

When uploaded,
Click on Thumbnail For Forum, or, Hotlink For Forum, (copies link) then paste in your post.

[bassquake]

The original post says "Attachments Awaiting Approval". I guess a mod never approved it. Anyhoo, here it is again:

https://i.postimg.cc/bdsHNXNN/avspmod-compare.jpg (https://postimg.cc/bdsHNXNN)

[StainlessS]

From here:- http://forum.doom9.org/showthread.php?p=1876311#post1876311

Oops again, Ver$ AutoLevels v0.9 update (avs v2.58, avs/+ v2.6 x86 & x64)
8 Bit only.
Req VS 2008 Runtimes.
Includes 3 dll's, + source + Full VS2008 project files for easy rebuild.
See HOSTED directory @ MediaFire in sig below this post.

Left debugging switch on, below should have been commented out for release, fixed. Probably was a bit slower than necessary.

#define BUG // Uncomment to enable DPRINTF() output


EDIT: CPP comment is NOT '#', is '//', so needed be

//#define BUG

[bassquake]

@StainlessS: Was that aimed at me? Do I need to replace the autoloevels_06.dll I currently use in Johns script with your 0.9 version?

[StainlessS]

It was aimed at anyone who is using AutoLevels at all. (64 bit version was requested by someone in this thread, I updated it [update for all 3 versions, v2.58, avs/+ x86 and x64]).

Here is result from your source, with cropping.


imagesource(".\1.bmp")
crop(0,0,Width/2*2,height/2*2) # EDIT: additional cropping as VUDB2 crop was not mod 2
ConvertToYV12
ColorYUV(AutoWhite=true,autogain=true)


https://i.postimg.cc/9wTbrVVV/1src.jpg (https://postimg.cc/9wTbrVVV)

https://i.postimg.cc/Th9XPX7f/1.jpg (https://postimg.cc/Th9XPX7f)

EDIT:
Do I need to replace the autoloevels_06.dll
Yes, good idea.

EDIT: With Gammac at defaults

imagesource(".\1.bmp")
crop(0,0,Width/2*2,height/2*2).ConvertToRGB24
ORG=Last
Gam=GamMac(Show=false)
COL=ConvertToYV12.ColorYUV(AutoWhite=true,autogain=true).ConvertToRGB24
StackHorizontal(ORG,COL,GAM) # Original, ColorYUV, Gammac


https://i.postimg.cc/N0RtLVNy/3.jpg (https://postimages.org/)

[SnillorZ]

From here:- http://forum.doom9.org/showthread.php?p=1876311#post1876311


@StainlessS Muchas Gracias :cool:

Appreciate the efforts you put in to getting this port done, thank you sir your a LegEnd [sic]

Done a quick & dirty all x64 test with johnmeyer v2017.avs and seeing approx 25% performance increase when like for like with x86 version of plugins - 8192MB 8x Threads/Prefetch (x64) vs 4096 4x Threads/PF (x86)

A little underwhelming, but need to test more setting combinations.

Will post AVSMeter results in a bit

[SnillorZ]

Will post AVSMeter results in a bit

...........

Testing using the earlier posted "Parade" example clip (https://forum.doom9.org/showthread.php?p=1865604#post1865604) with a version of the JohnMeyer 2017 .avs script with newest versions of plugins

x86 4086M 4 x Threads (Prefetch)

AVSMeter 2.9.1 (x86) - Copyright (c) 2012-2019, Groucho2004

VersionString: AviSynth+ 0.1 (r2772, MT, i386)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 5
Multi-threading support: Yes
Avisynth.dll location: C:\windows\SysWOW64\avisynth.dll
Avisynth.dll time stamp: 2018-12-20, 12:06:28 (UTC)
Custom plugin directory : D:\VideoTools\AVSplugins\x86

[CPP 2.5 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\x86\avstp.dll [1.0.3.0]
D:\VideoTools\AVSplugins\x86\warpsharp.dll [2008-04-06]

[CPP 2.6 Plugins (32 Bit)]
D:\VideoTools\AVSplugins\x86\AutoLevels_x86.dll [0.9.0.0]
D:\VideoTools\AVSplugins\x86\deflicker.dll [0.6.0.0]
D:\VideoTools\AVSplugins\x86\DePan.dll [2.13.1.4]
D:\VideoTools\AVSplugins\x86\DePanEstimate.dll [2.10.0.3]
D:\VideoTools\AVSplugins\x86\fft3dfilter.dll [2.6.0.0]
D:\VideoTools\AVSplugins\x86\FredAverage_x86.dll [0.3.0.0]
D:\VideoTools\AVSplugins\x86\GamMac_x86.dll [1.10.0.0]
D:\VideoTools\AVSplugins\x86\masktools2.dll [2.2.18.0]
D:\VideoTools\AVSplugins\x86\mvtools2.dll [2.7.41.0]
D:\VideoTools\AVSplugins\x86\RemoveDirt.dll [0.9.2.0]
D:\VideoTools\AVSplugins\x86\RgTools.dll [0.97.0.0]

AVSMeter 2.9.1 (x86) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2772, MT, i386) (0.1.0.0)
Number of frames: 241
Length (hh:mm:ss.ms): 00:00:15.063
Frame width: 2424
Frame height: 720
Framerate: 16.000 (16/1)
Colorspace: YV12
Frames processed: 241 (0 - 240)
FPS (min | max | average): 0.688 | 12.63 | 2.076
Process memory usage (max): 1437 MiB
Thread count: 20
CPU usage (average): 81.8%
Time (elapsed): 00:01:56.077

x64 8192M 8 x Threads (Prefetch)

AVSMeter 2.9.1 (x64) - Copyright (c) 2012-2019, Groucho2004

VersionString: AviSynth+ 0.1 (r2772, MT, x86_64)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 5
Multi-threading support: Yes
Avisynth.dll location: C:\windows\system32\avisynth.dll
Avisynth.dll time stamp: 2018-12-20, 11:55:18 (UTC)
Custom plugin directory : D:\VideoTools\AVSplugins\x64

[CPP 2.5 Plugins (64 Bit)]
D:\VideoTools\AVSplugins\x64\avstp.dll [1.0.3.0]
D:\VideoTools\AVSplugins\x64\warpsharp.dll [2011-06-15]

[CPP 2.6 Plugins (64 Bit)]
D:\VideoTools\AVSplugins\x64\AutoLevels_x64.dll [0.9.0.0]
D:\VideoTools\AVSplugins\x64\Deflicker.dll [0.6.0.0]
D:\VideoTools\AVSplugins\x64\DePan.dll [2.13.1.4]
D:\VideoTools\AVSplugins\x64\DePanEstimate.dll [2.10.0.3]
D:\VideoTools\AVSplugins\x64\fft3dfilter.dll [2.6.0.0]
D:\VideoTools\AVSplugins\x64\FredAverage_x64.dll [0.3.0.0]
D:\VideoTools\AVSplugins\x64\GamMac_x64.dll [1.10.0.0]
D:\VideoTools\AVSplugins\x64\masktools2.dll [2.2.18.0]
D:\VideoTools\AVSplugins\x64\mvtools2.dll [2.7.41.0]
D:\VideoTools\AVSplugins\x64\RemoveDirt.dll [0.9.2.0]
D:\VideoTools\AVSplugins\x64\RgTools.dll [0.97.0.0]


AVSMeter 2.9.1 (x64) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2772, MT, x86_64) (0.1.0.0)
Number of frames: 241
Length (hh:mm:ss.ms): 00:00:15.063
Frame width: 2424
Frame height: 720
Framerate: 16.000 (16/1)
Colorspace: YV12
Frames processed: 241 (0 - 240)
FPS (min | max | average): 0.210 | 78877 | 2.606
Process memory usage (max): 3351 MiB
Thread count: 25
CPU usage (average): 98.0%
Time (elapsed): 00:01:32.490

[Meloware]

My brain is exploding! (Sorry)
Is there a repository that provides or describes the state-of-the-art scripts and components for using the film restoration scripts? I have a Windows 10 32core Threadripper system with 64GB of ram. I would dearly like to update my first generation VideoFred script, 32 bit VirtualDub and old 32bit Avisynth install. The original script and discussion has been ongoing for years, and I could really use help in getting up to date.

Could someone please suggest or point to a complete list of components needed for installing a state of the art film restorer for 2019? It would be great to use 64 bit operations and more than one thread. A comprehensive list of sources, for 2019, may be helpful to many new and future users! Thanks.

[SnillorZ]

My brain is exploding! (Sorry)
Yes feel your pain, welcome to the wonderful world of open source, you can check out when you like but you can never leave. :devil:

It might not cost you much in hard cash but you will need to invest your time to get the best out of it and find what your looking for - but you've already figured that.

Is there a repository that provides or describes the state-of-the-art scripts and components for using the film restoration scripts?
Yes lots, but not all in one place that your asking for.

Could someone please suggest or point to a complete list of components needed for installing a state of the art film restorer for 2019?

My own personal interest and workflow is still a work in progress to find the best balance of processing performance vs result quality.

However I had intended posting all that I have gathered together over the last several weeks in case it can be of help to someone like yourself, as I know it would have been a help to someone like me.

Just keep in mind that some of the tools / Avisynth plugins do and will get superseded by the brilliantly helpful dev's here in Doom9 and elsewhere. So you will need to keep an eye out for something that could be bigger and better, so to speak.

So if you can wait a little, stick around and I'll find somewhere to host the collection of programs/tools/plugins/example avs and bit of documentation to provide some pointers. In next week or 2.

[StainlessS]

MediaFire is reasonably good and free filehost (lots of adverts apparently, if you don't not got no adblock).
The longer you have an account, the more your allowed storage quota grows. Last time I looked, (bout 12 months ago),
mine was up to about 52GB, only use a few hundred MB, so I've still got plently left..
Can have public download folder/s, Private folders for you alone, and apparently public upload folders if you wish (I don't not havn't tried that yet).
And big bonus for you, they do not mind if you are a leg end, so a result there I think you can agree.

[SnillorZ]

@StainlessS :D Hehe

Thank you sir, yes I've got a couple of options I can use, just need to pick one that will work best for other people (no hoop jumping/spamming/tracking your every move), will give MediaFire a look.

[SnillorZ]

Please may I ask for any opinion from experience on the following x64 bit testing:



[CPP 2.5 Plugins (64 Bit)]
D:\VideoTools\AVSplugins\x64\avstp.dll
D:\VideoTools\AVSplugins\x64\warpsharp.dll [2011-06-15]

AVSMeter 2.9.1 (x64) - Copyright (c) 2012-2019, Groucho2004
AviSynth+ 0.1 (r2772, MT, x86_64) (0.1.0.0)
Number of frames: 241
Length (hh:mm:ss.ms): 00:00:15.063
Frame width: 2424
Frame height: 720
Framerate: 16.000 (16/1)
Colorspace: YV12
Frames processed: 241 (0 - 240)
FPS (min | max | average): 0.210 | 78877 | 2.606
Process memory usage (max): 3351 MiB
Thread count: 25
CPU usage (average): 98.0%
Time (elapsed): 00:01:32.490

That the warpsharp.dll is a 2.5 "branch" CPP x64 rather a CPP 2.6 plugin (like all the others used by this avs) will be having much impact on performance?

Realise this maybe a bit of a "how longs a piece of string" question.

By the way the warpsharp.dll x64 (6-2011) is the most up to date available I believe, having trawled all over for anything newer/better.

Also since the original post I've removed the avstp.dll that @StainlessS pointed me to, found the script runs without it in its latest all x64 form and gives a little more 0.fps without it being loaded.

[pinterf]

Please may I ask for any opinion from experience on the following x64 bit testing:



That the warpsharp.dll is a 2.5 "branch" CPP x64 rather a CPP 2.6 plugin (like all the others used by this avs) will be having much impact on performance?

performance impact: no.
Maybe you have to specify the proper MT mode with SetFilterMTMode for the function you are using from that DLL. (MT_SERIALIZED, MT_MULTI_INSTANCE, MT_NICE_FILTER)

Realise this maybe a bit of a "how longs a piece of string" question.

By the way the warpsharp.dll x64 (6-2011) is the most up to date available I believe, having trawled all over for anything newer/better.

But this dll was only needed because of UnsharpMask, wasn't it? Which can be replaced with that custom user script function. It won't be bit-wise identical but after examining the impact of its parameters it should be usable.


Also since the original post I've removed the avstp.dll that @StainlessS pointed me to, found the script runs without it in its latest all x64 form and gives a little more 0.fps without it being loaded.
avstp.dll is used to help multithreading for specific filters even if they were used in a single-thread Avisynth environment. Not all filters, only for those that were written to use it.

E.g. in MSuper, MAnalyze, MCompensate, MDegrainN in mvtools2 2.7.xx and the old 2.6.0.5 version.
See bool mt (default true) parameters in the html docs of mvtools2 2.7.41 (bundled with the dll files)

Which other filters are using that from which plugins, I don't know.

[SnillorZ]

performance impact: no.
Maybe you have to specify the proper MT mode with SetFilterMTMode for the function you are using from that DLL. (MT_SERIALIZED, MT_MULTI_INSTANCE, MT_NICE_FILTER)

Yes coincidentally reviewing all this now, as I realised in the copy of john's script I've nicked & amended it doesn't have the SetFilterMTMode, but he maybe handling this with a a call to an auto-loaded separate avs he's not given details of.


But this dll was only needed because of UnsharpMask, wasn't it? Which can be replaced with that custom user script function. It won't be bit-wise identical but after examining the impact of its parameters it should be usable.

Yes that's right and testing both finding Unsharpmask with warpsharp is marginally faster than your (kindly provided) script based function. Remembering I'm only concentrating on processing speed and not quality between the 2, so no decision made here yet.


avstp.dll is used to help multithreading for specific filters even if they were used in a single-thread Avisynth environment. Not all filters, only for those that were written to use it.

Thank you, yes I didn't go back and properly check the details of what this did. Just removed to see if script ran without and any fps changes.

[Meloware]

The skills and experience here are far more than anything I am capable of contributing, technically. I am, however considering publishing my 16mm film digital conservation methods. They largely depend on open source software and hand-modified vintage gear. The modern physical stuff I use was assembled with a Chinese stepping motor controller (from eBay), an Arduino Due and LED projection lighting. My server at Meloware.com is largely unused, but I would be happy to maintain a repository of tested links and setup procedures, if the skilled members wished to work with me. In any case, I appreciate the group's sympathy and await further discussion! BTW, I notice most of these scripts work with video files. I use high resolution frame sequence images. It would be great to see some discussion on best methods to use frame sequences as the source for the film restoration scripting.

[SnillorZ]

I notice most of these scripts work with video files. I use high resolution frame sequence images. It would be great to see some discussion on best methods to use frame sequences as the source for the film restoration scripting.

That I can help you with now, this was discussed a little earlier in this thread and can be used with these scripts

Using Image Sequence as Source (https://forum.doom9.org/showthread.php?p=1832992#post1832992)

[GillesH]

Hello,
I had asked the question a few years ago, but no answer.

The Getdups Plugin is a very important tool for capturing S8, 8mm movies, etc.
https://avisynth.org.ru/getdups/getdups.html
But it does not work properly in Multi-Threading.

With Avisynth-MT, it's not too penalizing. Just specify the SetMTMode just AFTER the Getdups command.

With Avisynth+, you can not use GetDups with other Filters and Prefetch.
Because, in this case, the notion of Multi-Thread is valid for the whole Script and GetDups does not work properly.

GetDups is really useful for Movie Captures made with the method specified by Fizick.
Unfortunately, we no longer see this super-developer on Doom9.

My new request is for all our developers to find a way to modify GetDups and use it in Multi-Thread.
Or, do you have another solution for this method of capture ?

Thank you.

[pinterf]

Perhaps:
SetFilterMTMode("GetDups", MT_SERIALIZED)

Does not work properly = wrong result?

[SnillorZ]

avstp.dll is used to help multithreading for specific filters even if they were used in a single-thread Avisynth environment. Not all filters, only for those that were written to use it.

Also since the original post I've removed the avstp.dll that @StainlessS pointed me to, found the script runs without it in its latest all x64 form and gives a little more 0.fps without it being loaded.

So turns out I was wrong, back to testing things in vDub, processing hangs (memory leaks/race condition?) and checking as StainlessS told me to do in the first place shows me:


RemoveDirt 0.9.2
AvstpWrapper: cannot find avstp.dll.Usage restricted to single threading.


:thanks:

[pinterf]

The statement applies to the internal multithreading, avisynth-level multithreading is still available. The Removedirt debug output is irrelevant there it just outputs it on its own at each dll load

[GillesH]

Perhaps:
SetFilterMTMode("GetDups", MT_SERIALIZED)

Does not work properly = wrong result?

Thanks pinterf.
I just tested and it works with this line MT and Prefetch(4).
And it also works with other associated filters.
Great !
Thank you very much and congratulations for your research.
:thanks:

[SnillorZ]

Popping in to x-post from this discussion (https://forum.doom9.org/showthread.php?p=1876852#post1876852)

and from I would prefer if somebody else did it, you game ?

to follow up All good though, I can get on with some proper work.

to say What a Tool @StainlessS (https://forum.doom9.org/showthread.php?p=1874068#post1874068)

has provided in the modified Avisynthesizer.

Really helps speed up work, with a little preparation. If like me you end up with a whole heap of scene by scene clips, because a Telecined conversion calls for a more granular approach. (Again :thanks: to a johnmeyer script ideas on Scene Detection)

Bloody marvelous time saver.

Now looking to see if there is a way to batch resulting avs into a VirtualDub job list automatically (rather than manually one by one). :sly:

[StainlessS]

Snillorz,
What is the intended output format (why VDub2, AVI) ?
What bit depth, colourspace, codec ?

EDIT:
MeGUI AutoEncode Batcher:-
https://forum.doom9.org/showthread.php?t=163343&highlight=Batcher

TWriteAVI (NOT if using MT : EDIT: x86 Only):-
https://forum.doom9.org/showthread.php?t=172837&highlight=TWriteAVI

[SnillorZ]

Just what I took the time to work out how to use, what works with what, etc, being a noob to this scripted workflow for the 8mm film restoration work.

I'm supposed to be working to a deadline of sorts (self imposed), I'm getting to the point where I figured I need to stop poncing about picking which trainers are slightly better than which and get into the race, so to speak.

I'm not married to VDub though and did have a quick look at MeGUI as a possible solution for one of the challenges (cant remember what now) but decided to swerve it.

Anyway to answer your Q,s - Yes output to avi while using script based tools. Working on the basis of as fewer handling steps and conversions are better until I get to editing at NLE and final content delivery(basing this on my own, probably outdated thinking). I have found that some conversion has been necessary anyway to keep all things happy - mostly the NLE,s tested have complained (something else I've not made a final decision on - used PP before but testing Davincii and Vegas). So ended up YV12 using UT Code with YUV420 .709 which everything seems happy with.

[johnmeyer]

This thread has sure drifted a long way away from restoring film ...

[StainlessS]

SnillorZ,

Select your first post related to your specific problems, and ask moderator to move all posts related, to a new thread,
click the little red triangle below the selected 1st post with your name/avatar, and send report to a moderator, ask for new thread, and tell him which is start post, and any posts to omit,
and also new name for your thread.
EDIT: When new thread created, add some kind of "Moved from this thread post #nnnn": Link_to_Post.

Below only for non MT and x86 only I'm afraid, I dont speak assembler. [EDIT: Although no MT allowed, nothing to stop you processing several similar scripts at once, except maybe x86 mem constraints]

#ASYNTHER BATCH-MY_THINGY
# Batch Mode, multiple AVS files generated.
#FN="___FILE___" # AvisynthesizerMod BATCH mode usage
####################
FN="D:\Parade.avi" # use ".\" for current path relative
###
OUTPATH = "D:\OUTPUT\" # Must exist, include Trailing backslash
FOURCC = "ULY0" # Ut_Video, YUV 420 601 TV Levels I think
###
FN=RT_GetFullPathName(FN)
OUTFN = OUTPATH + RT_FilenameSplit(FN,4) + ".AVI" # 4 = Name node only (no drive/path/extension)
###
#GetSeq(FN) # Avisynthesizer source
Avisource(FN)
###
########## DO PROCESSING Of SOME SORT ###
# Import("D:\MYSTUFF\GENERIC_PROC_1.avs") # Generic pre-process script (Affect all template expanded scripts [even already existing ones] with single GENERIC_PROC script edit)

Sharpen(0.3) # Whatever

# Import("D:\MYSTUFF\GENERIC_PROC_2.avs") # generic post-process script
########## END OF PROCESSING ############
#return last # Comment OUT for final
#########################################
TWriteAVI(OUTFN,Overwrite=true,fourcc=FOURCC) # Write Sharp AVI whilst clip is played
#return Last # Write File as you play
ForceProcessAVI() # Force Write above TWriteAVI file without playing clip.
Return Colorbars.Subtitle("ALL DONE\nWritten:- "+RT_FilenameSplit(OUTFN,12),size=48,align=5,lsp=0) # Complete with nice TONE as user attention getter. (12=Name + extension)
##################################
### TWRITEAVI - NOT FOR MT USE ###
### x86 ONLY ###
### x64 Requires Asm progger ###
##################################

[SnillorZ]

This thread has sure drifted a long way away from restoring film ...

Noted.

@StainlessS. Will do, not sure one thread will cover all but ok. Also might end up with a number of orphaned answers from others.

Done

In the mean time @mods feel free to move / remove any posts of mine not considered on topic, wont be offended. Including this one.

Cheers.

[SnillorZ]

......So back on topic ;)

an updated example using the latest variant of the VideoFred / JohnMeyer script from the new x64bit workflow.

Still a work in progress but getting pleasing results from just scripted automation and no by-hand adjustments (yet)

New x64bit plugins example (https://vimeo.com/342316863)

[bassquake]

If anyones interested Ive posted a comparison to original capture as well using John Meyers script which is a variation of Freds script:

https://youtu.be/JMeztmugk08

(Don't know how to embed on here.)

Its been downrezzed to 720p. The final output will be 1440 x 1080.

Shame YouTubes compression is so high but you get the idea.

Was slow to render though. About 0.2fps with a i7-9800K!

Check out the channel on how telecine was done!

@SnillorZ: How was your capturing done?

[SnillorZ]

(Don't know how to embed on here.)


Was slow to render though. About 0.2fps with a i7-9800K!


@SnillorZ: How was your capturing done?

@bassquake Forum automatically embeds Vimeo hosted only I think.

I had the same issue with processing, managed to get this up around 4-5fps by migrating the scripts to newer versions of the filters and x64bit workflow throughout. Check the last half dozen or so pages for details (Note these have be requested to be moved to a separate thread, so mods may move soon)

Details of the Telecine here (https://forum.doom9.org/showthread.php?p=1875142)

[bassquake]

@SnillorZ: That 0.2fps was with 64bit plugins, 64bit ffmpeg 4.1.3 as well as x64 Avisnyth+ MT r2772! Maybe I don't have the latest plugins but they were all grabbed from newest sources I could find.

The versions I have are:


mvtools2.dll - 2.7.41.0
autolevels_x64.dll - 0.9
Deflicker.dll - 0.6
Depan.dll - 2.13.1.4
DepanEstimate.dll - 2.10.0.3
masktools2.dll - 2.2.18
RemoveDirtSSE2.dll - 0.9.2
RgTools.dll - 0.97
warpsharp.dll - no version number in file


My input images are 2976 x 1984 if that makes a difference, which are cropped to 1440 x 1080 in the avs script.

[johnmeyer]

If you are feeding 1440x1080 into the script (i.e., you are cropping BEFORE restoring), then 0.2 fps is about 10% (or less) of what you should be getting. Even without MT, you should be getting at least 1-2 fps.

[bassquake]

I haven't majorly altered the script. Mainly changing the input to accept an image sequence, flip the image, tweak the crop and stabilise values and paths to 64 bit plugins.

See below script I use:

# John Meyers Script
# Use AVISynth version 2.60
#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------

SetFilterMTMode("DEFAULT_MT_MODE", 2)

#Change the following line to point to your video file
film="E:\Videos\imgseq\SAM_%05d.jpg"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result1" #specify the wanted output here; valid results are "result1" through "result4" and "resultS1" through "resultS4" for before/after

trim_begin=1 trim_end= 28160

play_speed=18 #play speed (8mm=16; Super8=18; 16mm sound=24)


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=0.9 #for all outputs
gamma=1.0 #for all outputs
blue= -0 red=-0 #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0 white_level=255 output_black=0 output_white=255 #manual levels, when returning result2 & result4


#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=796 CTop=376 CRight=550 CBottom=388 #crop values after Depan and before final resizing
W=1440 H=1080 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=40
est_left=760 est_top=380 est_right=500 est_bottom=350 #crop and contast values for special Estimate clip
est_cont=1.6

#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#Fastest parameters
#denoising_strength=600 #denoising level of first denoiser: MVDegrain()
#denoising_strength=300 #denoising level of first denoiser: MVDegrain()
#block_size= 16 #block size of MVDegrain
#block_size_v= 16
#block_over= 8 #block overlapping of MVDegrainMulti()

#Best compromise between speed and quality
denoising_strength= 0 #denoising level of first denoiser: MDegrain()
block_size= 8 #block size of MVDegrain
block_size_v= 8
block_over= 4 #block overlapping of MVDegrainMulti()

dirt_strength=20 #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 30 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
Sharp_Strength= 0.1


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4 #X is a special parameter for reducing the autolevels effect on the whites
X2=16 #X2 is a special parameter for reducing the autolevels effect on the blacks


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=5


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

# Using AVISynth version 2.60, build August 28, 2012 (Ben-Rudiak-Gould, etc.)
# Using VirtualDub 1.8.6

#Change the following (lower or higher) if you have stability problems with multi-threading
SetMemoryMax(1600)

#Load plugins explicitly
LoadPlugin("plugins_x64/mvtools2.dll")
LoadPlugin("plugins_x64/autolevels_x64.dll")
LoadPlugin("plugins_x64/Deflicker.dll")
Loadplugin("plugins_x64/Depan.dll")
LoadPlugin("plugins_x64/DepanEstimate.dll")
Loadplugin("plugins_x64/masktools2.dll")
Loadplugin("plugins_x64/RemoveDirt.dll")
Loadplugin("plugins_x64/RgTools.dll")
Loadplugin("plugins_x64/warpsharp.dll")

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll") #Version 1.2.1.0 5/29/2012
#loadplugin("svpflow2.dll") #Version 1.3.1.0 6/02/2012
#Import("InterFrame2.avsi") #Version 2.1.0 6/27/2012

#Open and crop the video
#Remove all setmtmode statements (there are four in this script) if not using multi-threaded (MT) AVISynth
#setmtmode(5,threads)
source1= ImageSource(file=film, start=trim_begin, end=trim_end, fps=play_speed).FlipVertical().converttoYV12() #killaudio() improves stability (in my experience -- others disagree)
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #temporarily remove any black borders on input video
#setmtmode(2)

#cropped_source=filldrops(cropped_source) #Use this when removing bad frames that have been removed by duplicating previous frame
#(see notes in function at end of script)

#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#Alternative line that includes deflicker. I find that this usually is not needed.
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()


#DENOISING
#...................................................................................................................................................................

#Remove dirt
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

#Reduce grain
prefiltered = RemoveGrain(stabcrop,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(super, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)


#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................

#Interpolate frames
#Use this for progressive output -- example shows 29.97 progressive
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)

#Use this for interlaced output -- example shows 29.97 interlaced
#PreBorderFrame=InterFrame(NewNum=60000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads).SeparateFields().SelectEvery(4, 0, 3).Weave()


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
#SetMTMode(5) #Turn off SetMTMode for Autolevels
result1= PreBorderFrame.coloryuv(autowhite=true).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= PreBorderFrame.coloryuv(off_U=blue,off_V=red).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)
#SetMTMode(2) #Re-enable SetMTMode after Autolevels

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.coloryuv(off_U=blue,off_V=red).levels(black_level,gamma,white_level,0,255).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)


#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"manual colors and levels correction",size=28,align=2))

#Finish and go home
Eval(result)


# END SCRIPT, BEGIN FUNCTIONS
#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int limit, bool _grey)
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

# The following function will remove near duplicates ("0.1") or exact duplicates (change to "0.0").
# It replaces these duplicates with a motion estimated frame. It works really well (IMHO).
#
# Use: In your video editor, replace any single bad frame (burned frame, jump, missing frame from splice, etc.) with a duplicate of the
# previous frame. Then, include a call to this function (see script above, where it is commented out).
#
# Here's an example of how this function can be use:
#
# http://www.youtube.com/watch?v=uzMFodrGHDs
#

function filldrops (clip c)
{
super=MSuper(c,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
filldrops = mflowinter(c,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
return fixed
}

Prefetch(4)

#return last

And the command for ffmpeg is:

ffmpeg.exe -i "test.avs" -c:v h264 -preset slow "test.mp4"

Would appreciate some insight, thanks!

[StainlessS]

autolevels_x64.dll - 0.9



Latest is v0.10

[SnillorZ]

Would appreciate some insight, thanks!

@bassquake Just to check the script you posted doesn't show the same versions as you listed in your previous post?

As StainlessS advised v0.10 Autolevels is latest, other differences are:

avstp_x64.dll #x64 Version 1.0.3.0 30/12/2015
AutoLevels_x64.dll #x64 Version 0.10.0.0 11/06/2019
fft3dfilter.dll #x64 Version 2.6.0.0 31/01/2019
FredAverage_x64.dll #x64 Version 0.3.0.0 16/01/2019
GamMac_x64.dll #x64 Version 1.10.0.0 15/06/2018
RemoveDirt.dll #x64 Version 0.9.2 24/03/2019
warpsharp.dll #x64 15/06/2011

As I've adapted a version of the later 2017 johnmeyer script to x64 plugins.

Also just to be clear, as per previous posts I'm processing in 2 passes i.e. Clean & Stabilise in 1st pass and Colour & White correction in 2nd. Edit otherwise all processing in 1 pass produces an average of 2fps (max) with my source.

Heading off course again here though and as suggested a request has been made to the @Mods to get a separate dedicated thread to discuss performance of the VideoFred/JohnMeyer scripts and such posts moved there.

[bassquake]

@Snillorz: Yes, I left the versions in the comments, you can ignore those. The versions I have are the ones I bulleted earlier. Ill edit the avs code and remove them to avoid confusion!

Edit: I already have those versions you highlighted Snillorz. (avstp_x64, fft3dfilter.dll, FredAverage_x64.dll,GamMac_x64.dll).

[Meloware]

Are there any significant changes suggested in the current 8mm script, when processing frames of varying size and film formats?

My 16mm frame sequence captures are processed at a resolution of 2604x1738px. What operations (denoise, stabilize, sharpen, etc) require special consideration when taking into account frame size?

[Groucho2004]

Was slow to render though. About 0.2fps with a i7-9800K!You should go through your script again. For example, this line:
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #temporarily remove any black borders on input video
does nothing because all crop parameters are zero.

[Meloware]

I am still trying to get a working installation of AVISynth+, for film restoration..

Where am I able to find?
FredAverage_x64.dll #x64 Version 0.3.0.0 16/01/2019
GamMac_x64.dll #x64 Version 1.10.0.0 15/06/2018

The link, for "FredAverage", at http://avisynth.nl/index.php/AviSynth%2B_x64_plugins is dead.
The link for GamMac, on the same page, points to another post on this forum, and there is no clear download link anywhere I can find.

[Groucho2004]

I am still trying to get a working installation of AVISynth+, for film restoration..

Where am I able to find?
FredAverage_x64.dll #x64 Version 0.3.0.0 16/01/2019
GamMac_x64.dll #x64 Version 1.10.0.0 15/06/2018

The link, for "FredAverage", at http://avisynth.nl/index.php/AviSynth%2B_x64_plugins is dead.
The link for GamMac, on the same page, points to another post on this forum, and there is no clear download link anywhere I can find.You will find both of these in StainlessS' mediafire directory. Check his sig a few posts above.

[Meloware]

You will find both of these in StainlessS' mediafire directory. Check his sig a few posts above.
Thank you - I figured it out. I am not overly familiar with forums. I saw the above posts and it was not clear to me that the slogans at the bottom of the posts might contain repositories

[bassquake]

You should go through your script again. For example, this line:
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #temporarily remove any black borders on input video
does nothing because all crop parameters are zero.

Removed the code for the black crop and it didnt really make a difference.

I decided to recreate the script a step at a time to see what frames I get when adding each feature:


Crop + Resize: ~15fps
Crop + Resize + Stabilise: ~7fps
Crop + Resize + Stabilise + Dirt Removal: ~6.8fps
Crop + Resize + Stabilise + Dirt Removal + Denoise & Sharpen: ~6.5fps
Crop + Resize + Stabilise + Dirt Removal + Denoise & Sharpen + Gammac: ~2.8fps
Crop + Resize + Stabilise + Dirt Removal + Denoise & Sharpen + Gammac + Autolevels: ~0.4fps


I could probably live with 2fps but Autolevels seems to cause it to choke.

[Meloware]

I haven't majorly altered the script.

I am doing something very similar. I would like to benchmark your script on my 32 core Threadripper, if someone could please provide me a link to a _x64 version of warpsharp. I think I have all the other 64 bit dlls.
What program are you running the 64 bit AviSynth+ in? How would you modify this script to write out a frame sequence? I have had some trouble with that.

It may have been already mentioned, but with 8mm, you might get away with a lower source resolution and enjoy a higher speed. My old AviSynth script was using 2604x1738 px jpeg frames, and that was for 16mm films.

My goal in AviSynth+ would be to export a high resolution frame sequence, for a final processing in Blender's fine compositor. Blender 3D would give me the final control of the levels, center box cropping and even stabilizing some inappropriate shakes from AviSynth's stabilizer. Blender's compositor functions may all be animated, so temporal adjustments are very practical. It is all also Open Source!

I built my telecine from a Bell&Howell (filmosound) 1550, a eBay stepping motor controller, and Arduino Due, and bright LED lighting. My computer is a Threadripper 2950x (Windows 10) with 64GB memory. I sure would like to use all those threads! Who has a good link to a 64 bit warpsharp.dll ?

[Groucho2004]

Removed the code for the black crop and it didnt really make a difference.Since it didn't do anything I would have expected that. :rolleyes:

[bassquake]

@Meloware: Attached the warpsharp Im using.

[poisondeathray]

if someone could please provide me a link to a _x64 version of warpsharp.

http://chaosking.de/repo/avsfilters/Adjustment_Filters/Sharpeners/


What program are you running the 64 bit AviSynth+ in? How would you modify this script to write out a frame sequence? I have had some trouble with that.


An "easy" way for 8bit formats with a GUI would be vdub2 (VirtualDub64.exe). Open the avs (just drop and drag or File=> open video file; do not use file=>run script - that is not for avs scripts) , then File=> export image sequence

[Meloware]

http://chaosking.de/repo/avsfilters/Adjustment_Filters/Sharpeners/


Great! Thanks to both poisondeathray and bassquake for the help.
I had been using Virtualdub but I am now attempting to use 32 and 64 bit AvsPmod. Bassquake's script is now running. My input sequence images are 2604x1738 and his script is not modified. The script now runs in 64 bit AvsPmod at about 2 fps. I tried to use the "Save Image Sequence" macro, but it throws me an error "There is not bookmarks". An explanation would be helpful, but I do have to learn stuff on my own, as well. Getting Imagewriter() to work here will be priority this evening. It is not clear to me how you save rendered output in AvsPmod (such as in bassquake's version).

[poisondeathray]

I tried to use the "Save Image Sequence" macro, but it throws me an error "There is not bookmarks".

When you use the macro, there should be a configuration box. Under "select frames", choose "all frames" instead of "Bookmarks" from the drop down list

[pinterf]

The link, for "FredAverage", at http://avisynth.nl/index.php/AviSynth%2B_x64_plugins is dead.
The link for GamMac, on the same page, points to another post on this forum, and there is no clear download link anywhere I can find.
Links fixed

[bassquake]

Great! Thanks to both poisondeathray and bassquake for the help... It is not clear to me how you save rendered output in AvsPmod (such as in bassquake's version).

I would load the avs script in ffmpeg (64 bit version) and export as image sequence. Eg:

ffmpeg -i "yourfile.avs" -r 1 "output_%04d.png"

Change %04d to how many digits your images take up. So %5d would be for up to 10,000 images. -r 1 switch is the frame rate, I think can leave it at 1.

Or use jpgs to reduce disk size, setting the quality -q:

ffmpeg -i "yourfile.avs" -q:v 1 "%05d.jpg"

[videoFred]

Are there any significant changes suggested in the current 8mm script, when processing frames of varying size and film formats?

My 16mm frame sequence captures are processed at a resolution of 2604x1738px. What operations (denoise, stabilize, sharpen, etc) require special consideration when taking into account frame size?

There's no need for such a high resolution. I have wonderful results with 16mm film, captured at 1388x1032px. The quality of the capturing is more important than the size.

To speed up the script, I even downscale the original capturing to 1024x768px, then run the script, then upscale it to full HD. (Pillarbox to keep 4:3) It runs at 5fps in Avisynth+ with MT enabled.

PS: you are doing a great job on your website. ;)

Fred.

[videoFred]

Crop + Resize + Stabilise + Dirt Removal + Denoise & Sharpen + Gammac + Autolevels: ~0.4fps
[/LIST]

I could probably live with 2fps but Autolevels seems to cause it to choke.

Why Autolevels if you are already use Gammac?

Fred.

[bassquake]

Why Autolevels if you are already use Gammac?

Fred.

Because it was in John Meyers original script. I assumed was needed.

[videoFred]

Because it was in John Meyers original script. I assumed was needed.

I see.... you can try it without Autolevels and see what you get.


Fred.

[videoFred]

Never tried this myself but this might help to speed up:
https://forum.doom9.org/showthread.php?p=1846757#post1846757

Fred.

[bassquake]

I see.... you can try it without Autolevels and see what you get.
Fred.

As mentioned earlier, I get about 2.8fps with Autolevels off. If it's not needed then Ill permanently remove it.

Never tried this myself but this might help to speed up:
https://forum.doom9.org/showthread.p...57#post1846757
Fred.

Thanks, will have a look.

[johnmeyer]

I forgot to take Autolevels out once I added GamMac. Autolevels never worked for me and I never, ever used it.

Also, I second Fred's recommendation to use a lower resolution.

To support Fred's advice about using lower resolution, I should post a comparison to my transfer of some 1930s German B&W film which I transferred for a friend a dozen years ago. Since the film involves Nazis, I have never posted it publicly. However, the following link shows some non-controversial film from the same source (over 200 400' reels of film were smuggled out):

https://www.youtube.com/watch?v=qpM9jpmYMt4

Because of the historical significance of some of the reels, when the Smithsonian Channel found out about it, they wanted to use it as part of a documentary. Long story made short, they insisted on having it transferred by a professional lab in L.A. that uses a Cintel/Spirit scanner. I had them send the scans to me before I sent the results to the Smithsonian, and I did a comparison of my transfer vs. theirs.

The punch line to the story is that yes, they did do a better job than my camcorder capture off my modified, shutterless Eiki projector, but the differences in resolution between their 4K scan and my 720x480 (down res'd in my Sony FX1 HD camcorder) were extremely small, even though this was 16mm film and even though it was taken by a man who owned a German movie camera factory (i.e., he knew what he was doing). The place where they were better, although by less than I expected, was in the shadow details.

If the film had been professional film, shot on high-end cameras with a $50,000 lens, and developed in a pro lab, I think the resolution differences would have been obvious and substantial. However, when you are dealing with amateur film, shot on a $100 camera and developed at the local drug store, there just isn't a lot there for the extra HD resolution to preserve.

Do some tests yourself to make sure this is the case with your material, but I suspect that you won't see any difference whatsoever if you follow Fred's advice, and you'll end up saving yourself a huge amount of time by using a lower resolution.

[Meloware]

@johnmeyer and @videoFred's comments on resolution are well taken and worthy of my own study. Thanks..
Everyone's help has contributed real and rapid progress in my effort! There are obvious concepts, in Avisynth, which have alluded me for years, that I am now gaining a grasp of! I plan on standardizing my script and source files, and then benchmark them on a number of applications (AvsPmod, VirtualDub, and ffmpeg64). Then, there will be many questions!

I mentioned my system as having a 32 core Threadripper, with 64GB ram. My source and destination files are to and from SSDs. I supposedly have 64 threads of processing available, yet any stable run of my script uses only 3% of my CPU, at best. The script's “threads” variable actually slows down processing, as I increase the number, and “SetMemoryMax” has very little beneficial effect. I have never seen more than 18% of my fast RAM ever used. I'd really like to light up my system, but I will have to ask the meaningful questions later.

I would like to mention a trick, which occurred to me when I was using 32 bit AviSynth 2.6 with Vdub.

There is an important advantage in rendering your films to frame sequences. I have yet to run any script, which alone is able to fully utilize the processing power of my Threadripper system, or even my 10 year old i7 machine. I got around that problem by launching multiple instances of my host program (VirtualDub) and then pointing each instance to it's own range of frames to process. Five instances of the program can process 5 times faster, than a single instance! This saved me from waiting a week to process a 25,000 frame film on my i7! My Threadripper is happy to assign each instance it's own memory and processing thread resources. The only problem is, that it takes time to setup and manage 50 or 60 instances of VirtualDub. Still, it is worth the effort if there is no other option. Once all the individual frames are processed, it is a simple matter to pass it to Blender (for the final touches) or transcode it into any playable video format you wish.

This trick will work with other, off topic applications. It is even possible to to run multiple instances of these bone-head commercial programs, which prohibit multiple instances. In that case, just create additional users on your machine, and then launch each instance as a separate user.

I will try and report back some solid benchmarks, later. Thanks again, to all.

[Groucho2004]

The script's “threads” variable actually slows down processingIf you're referring to bassquake's script above, the 'thread' variable does nothing. Post your complete script, the Avisynth version you're using and the DLL versions.
If you want to do all that in one go, use AVSMeter with this command line:
"AVSMeter script.avs -info -log". This will produce a log file in the script's directory. Post that file.

[Meloware]

If you're referring to bassquake's script above, the 'thread' variable does nothing. Post your complete script, the Avisynth version you're using and the DLL versions.
If you want to do all that in one go, use AVSMeter with this command line:
"AVSMeter script.avs -info -log". This will produce a log file in the script's directory. Post that file.
I will do that. Yes, this was with @bassquake's version, posted a few days ago. First, l need some time to get my 'day job' out of the way and then produce a version I can benchmark. I will be certain to include the source info and AVSMeter report.

[johnmeyer]

any stable run of my script uses only 3% of my CPUSomething definitely is not right. Even without MT enabled, you should be getting better utilization than that.

[Meloware]

Here is a copy of my AVSMeter log. The script is too long to include in this post, but could be re-posted if asked.

Source files were jpg frames 2604 x 1738 resolution and the output frames are written as 1640 x 1280 and are intended to be exported to Blender 3D for final tweaking and resizing.

Performance with these source images are a miserable .1 fps. Strangely, another run I made with source frames cropped and resized to 2260 x 1738 ran at .8fps.

CPU performance was around 3%. Total Memory consumption is below 12%. I did experiment with using ffmpeg64 and 14 CMD prompt instances. The CPU utilization was then 35% and memory at 74%. My fans spun up the system then produced close to 700 frames per minute. Is something obviously wrong? What is keeping the 64 threads from kicking in?

C:\software\AVSMeter292\AVSMeter292>AVSMeter64 avsinfo

AVSMeter 2.9.2 (x64) - Copyright (c) 2012-2019, Groucho2004

VersionString: AviSynth+ 0.1 (r2772, MT, x86_64)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 6
Multi-threading support: Yes
Avisynth.dll location: C:\WINDOWS\SYSTEM32\avisynth.dll
Avisynth.dll time stamp: 2018-12-20, 17:55:16 (UTC)
PluginDir2_5 (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64
PluginDir+ (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64+


[CPP 2.5 Plugins (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\warpsharp.dll [2011-06-15]

[CPP 2.6 Plugins (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\AutoLevels_x64.dll [0.10.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\ConvertStacked.dll [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\Deflicker.dll [0.6.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\DePan.dll [2.13.1.4]
C:\Program Files (x86)\AviSynth+\plugins64+\DePanEstimate.dll [2.10.0.3]
C:\Program Files (x86)\AviSynth+\plugins64+\DirectShowSource.dll [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\ffms2.dll [2016-12-29]
C:\Program Files (x86)\AviSynth+\plugins64+\FredAverage_x64.dll [0.3.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\GamMac_x64.dll [1.10.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\ImageSeq.dll [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\masktools2.dll [2.2.18.0]
C:\Program Files (x86)\AviSynth+\plugins64+\mvtools2.dll [2.7.41.0]
C:\Program Files (x86)\AviSynth+\plugins64+\nnedi3.dll [0.9.4.53]
C:\Program Files (x86)\AviSynth+\plugins64+\RemoveDirt.dll [0.9.2.0]
C:\Program Files (x86)\AviSynth+\plugins64+\RgTools.dll [0.97.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\Shibatch.dll [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\TimeStretch.dll [2018-12-20]
C:\Program Files (x86)\AviSynth+\plugins64+\VDubFilter.dll [2018-12-20]

[Scripts (AVSI)]
C:\Program Files (x86)\AviSynth+\plugins64+\colors_rgb.avsi [2016-07-05]
C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC.avsi [2019-06-17]
C:\Program Files (x86)\AviSynth+\plugins64+\smdegrain.avsi [2019-06-17]

[Uncategorized DLLs (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\libfftw3f-3.dll [2019-06-17]

[Uncategorized files]
C:\Program Files (x86)\AviSynth+\plugins64+\colors_rgb.txt [2016-07-05]



[Plugin errors/warnings]
______________________________________________________________________________________________________

Cannot load file 'C:/Program Files (x86)/AviSynth+/plugins64+/nnedi3.dll'. Platform returned code 126:
The specified module could not be found.

Dependencies that could not be loaded:
libmmd.dll
______________________________________________________________________________________________________


Log file created with: AVSMeter 2.9.2 (x64)
Script file: C:\Program Files (x86)\AviSynth+\scripts\Meloware.avs

[OS/Hardware info]
Operating system: Windows 10 (x64) (Build 17763)

CPU: AMD Ryzen Threadripper 2990WX 32-Core Processor / Threadripper (Colfax)
MMX, MMXEXT, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, SSE4A, FMA3, AVX, AVX2
32 physical cores / 64 logical cores


[Avisynth info]
VersionString: AviSynth+ 0.1 (r2772, MT, x86_64)
VersionNumber: 2.60
File / Product version: 0.1.0.0 / 0.1.0.0
Interface Version: 6
Multi-threading support: Yes
Avisynth.dll location: C:\WINDOWS\SYSTEM32\avisynth.dll
Avisynth.dll time stamp: 2018-12-20, 17:55:16 (UTC)
PluginDir2_5 (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64
PluginDir+ (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64+


[Clip info]
Number of frames: 1586
Length (hh:mm:ss.ms): 00:01:06.083
Frame width: 1640
Frame height: 1280
Framerate: 24.000 (24/1)
Colorspace: YV12
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a

[Groucho2004]

Is something obviously wrong? What is keeping the 64 threads from kicking in?Need to see the script. You can post it on pastebin.com.

[Meloware]

Here is the script used in my last post:

# John Meyers Script
# Use AVISynth version 2.60
#
# This script version is being evaluated by Meloware (June 19, 2019 ver.1) for optimal performance
# on a AMD 32 core Threadripper 2990WX system with 64GB ram - Windows 10
# The source files for this experiment are 2604 x 1738 jpg images captured and resized from a Sony A6300 mirrorless camera.
# The output frames produced by this script would normally then be exported to Blender 3D's compositor, for final tweaks and resizing
# into 1440 x 1080 pixel frames for final transcoding into a usable video.
# Many examples of my restored 16mm films are at http://industrialhistory.org/education/videos/
#
#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------

SetFilterMTMode("DEFAULT_MT_MODE", 2)

#Change the following line to point to your video file
film="J:\film\Prestone\source\p-%04d.jpg"

##### Destination files (if option "C1" is used and ImageWriter writes individual frames to a defined directory
output= "J:\film\Prestone\stab\p-%04d.jpg"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result1" #specify the wanted output here; valid results are "result1" through "result4" and "resultS1" through "resultS4" for before/after

trim_begin=1 trim_end= 1586

play_speed=18 #play speed (8mm=16; Super8=18; 16mm sound=24)

fps = 24







#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=0.9 #for all outputs
gamma=1.0 #for all outputs
blue= -0 red=-0 #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0 white_level=255 output_black=0 output_white=255 #manual levels, when returning result2 & result4


#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=250 CTop=100 CRight=300 CBottom=10 #crop values after Depan and before final resizing
W=1640 H=1280 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=40
est_left=760 est_top=380 est_right=500 est_bottom=350 #crop and contast values for special Estimate clip
est_cont=1.6

#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#Fastest parameters
#denoising_strength=600 #denoising level of first denoiser: MVDegrain()
#denoising_strength=300 #denoising level of first denoiser: MVDegrain()
#block_size= 16 #block size of MVDegrain
#block_size_v= 16
#block_over= 8 #block overlapping of MVDegrainMulti()

#Best compromise between speed and quality
denoising_strength= 0 #denoising level of first denoiser: MDegrain()
block_size= 8 #block size of MVDegrain
block_size_v= 8
block_over= 4 #block overlapping of MVDegrainMulti()

dirt_strength=20 #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 30 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
Sharp_Strength= 0.1


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4 #X is a special parameter for reducing the autolevels effect on the whites
X2=16 #X2 is a special parameter for reducing the autolevels effect on the blacks


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=5 #####What does this do?


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

# Using AVISynth version 2.60, build August 28, 2012 (Ben-Rudiak-Gould, etc.)
# Using VirtualDub 1.8.6

#Change the following (lower or higher) if you have stability problems with multi-threading
SetMemoryMax(800) ## What does this do

#Load plugins explicitly
LoadPlugin("../plugins64+/mvtools2.dll")
LoadPlugin("../plugins64+/autolevels_x64.dll")
LoadPlugin("../plugins64+/Deflicker.dll")
Loadplugin("../plugins64+/Depan.dll")
LoadPlugin("../plugins64+/DepanEstimate.dll")
Loadplugin("../plugins64+/masktools2.dll")
Loadplugin("../plugins64+/RemoveDirt.dll")
Loadplugin("../plugins64+/RgTools.dll")
Loadplugin("../plugins64+/warpsharp.dll")

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll") #Version 1.2.1.0 5/29/2012
#loadplugin("svpflow2.dll") #Version 1.3.1.0 6/02/2012
#Import("InterFrame2.avsi") #Version 2.1.0 6/27/2012

#Open and crop the video
#Remove all setmtmode statements (there are four in this script) if not using multi-threaded (MT) AVISynth
#setmtmode(5,threads)
source1= ImageSource(file=film, trim_begin, trim_end).converttoYV12() #killaudio() improves stability (in my experience -- others disagree)
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #temporarily remove any black borders on input video
#setmtmode(2)

#cropped_source=filldrops(cropped_source) #Use this when removing bad frames that have been removed by duplicating previous frame
#(see notes in function at end of script)

#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#Alternative line that includes deflicker. I find that this usually is not needed.
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()


#DENOISING
#...................................................................................................................................................................

#Remove dirt
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

#Reduce grain
prefiltered = RemoveGrain(stabcrop,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(super, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)


#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................

#Interpolate frames
#Use this for progressive output -- example shows 29.97 progressive
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)

#Use this for interlaced output -- example shows 29.97 interlaced
#PreBorderFrame=InterFrame(NewNum=60000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads).SeparateFields().SelectEvery(4, 0, 3).Weave()


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
#SetMTMode(5) #Turn off SetMTMode for Autolevels
result1= PreBorderFrame.coloryuv(autowhite=true).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= PreBorderFrame.coloryuv(off_U=blue,off_V=red).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)
#SetMTMode(2) #Re-enable SetMTMode after Autolevels

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.coloryuv(off_U=blue,off_V=red).levels(black_level,gamma,white_level,0,255).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)


#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"manual colors and levels correction",size=28,align=2))

resultC1= result1.ConverttoRGB24().ImageWriter(file= output,start = 1, type = "jpg", info = true)

#Finish and go home
Eval(result)


# END SCRIPT, BEGIN FUNCTIONS
#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int limit, bool _grey)
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

# The following function will remove near duplicates ("0.1") or exact duplicates (change to "0.0").
# It replaces these duplicates with a motion estimated frame. It works really well (IMHO).
#
# Use: In your video editor, replace any single bad frame (burned frame, jump, missing frame from splice, etc.) with a duplicate of the
# previous frame. Then, include a call to this function (see script above, where it is commented out).
#
# Here's an example of how this function can be use:
#
# http://www.youtube.com/watch?v=uzMFodrGHDs
#

function filldrops (clip c)
{
super=MSuper(c,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
filldrops = mflowinter(c,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
return fixed
}

Prefetch(4)

#return last

[Groucho2004]

Here is the script used in my last post:

# John Meyers Script

SetFilterMTMode("DEFAULT_MT_MODE", 2)

#Change the following line to point to your video file
film="J:\film\Prestone\source\p-%04d.jpg"

##### Destination files (if option "C1" is used and ImageWriter writes individual frames to a defined directory
output= "J:\film\Prestone\stab\p-%04d.jpg"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result1" #specify the wanted output here; valid results are "result1" through "result4" and "resultS1" through "resultS4" for before/after

trim_begin=1 trim_end= 1586

play_speed=18 #play speed (8mm=16; Super8=18; 16mm sound=24)

fps = 24


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=0.9 #for all outputs
gamma=1.0 #for all outputs
blue= -0 red=-0 #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0 white_level=255 output_black=0 output_white=255 #manual levels, when returning result2 & result4


#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=250 CTop=100 CRight=300 CBottom=10 #crop values after Depan and before final resizing
W=1640 H=1280 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=20 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=40
est_left=760 est_top=380 est_right=500 est_bottom=350 #crop and contast values for special Estimate clip
est_cont=1.6

#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#Fastest parameters
#denoising_strength=600 #denoising level of first denoiser: MVDegrain()
#denoising_strength=300 #denoising level of first denoiser: MVDegrain()
#block_size= 16 #block size of MVDegrain
#block_size_v= 16
#block_over= 8 #block overlapping of MVDegrainMulti()

#Best compromise between speed and quality
denoising_strength= 0 #denoising level of first denoiser: MDegrain()
block_size= 8 #block size of MVDegrain
block_size_v= 8
block_over= 4 #block overlapping of MVDegrainMulti()

dirt_strength=20 #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 30 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
Sharp_Strength= 0.1


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4 #X is a special parameter for reducing the autolevels effect on the whites
X2=16 #X2 is a special parameter for reducing the autolevels effect on the blacks


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=5 #####What does this do?


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

# Using AVISynth version 2.60, build August 28, 2012 (Ben-Rudiak-Gould, etc.)
# Using VirtualDub 1.8.6

#Change the following (lower or higher) if you have stability problems with multi-threading
SetMemoryMax(800) ## What does this do

#Load plugins explicitly
LoadPlugin("../plugins64+/mvtools2.dll")
LoadPlugin("../plugins64+/autolevels_x64.dll")
LoadPlugin("../plugins64+/Deflicker.dll")
Loadplugin("../plugins64+/Depan.dll")
LoadPlugin("../plugins64+/DepanEstimate.dll")
Loadplugin("../plugins64+/masktools2.dll")
Loadplugin("../plugins64+/RemoveDirt.dll")
Loadplugin("../plugins64+/RgTools.dll")
Loadplugin("../plugins64+/warpsharp.dll")

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll") #Version 1.2.1.0 5/29/2012
#loadplugin("svpflow2.dll") #Version 1.3.1.0 6/02/2012
#Import("InterFrame2.avsi") #Version 2.1.0 6/27/2012

#Open and crop the video
#Remove all setmtmode statements (there are four in this script) if not using multi-threaded (MT) AVISynth
#setmtmode(5,threads)
source1= ImageSource(file=film, trim_begin, trim_end).converttoYV12() #killaudio() improves stability (in my experience -- others disagree)
cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #temporarily remove any black borders on input video
#setmtmode(2)

#cropped_source=filldrops(cropped_source) #Use this when removing bad frames that have been removed by duplicating previous frame
#(see notes in function at end of script)

#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#Alternative line that includes deflicker. I find that this usually is not needed.
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()


#DENOISING
#...................................................................................................................................................................

#Remove dirt
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

#Reduce grain
prefiltered = RemoveGrain(stabcrop,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(super, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)


#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................

#Interpolate frames
#Use this for progressive output -- example shows 29.97 progressive
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)

#Use this for interlaced output -- example shows 29.97 interlaced
#PreBorderFrame=InterFrame(NewNum=60000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads).SeparateFields().SelectEvery(4, 0, 3).Weave()


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
#SetMTMode(5) #Turn off SetMTMode for Autolevels
result1= PreBorderFrame.coloryuv(autowhite=true).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= PreBorderFrame.coloryuv(off_U=blue,off_V=red).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)
#SetMTMode(2) #Re-enable SetMTMode after Autolevels

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.coloryuv(off_U=blue,off_V=red).levels(black_level,gamma,white_level,0,255).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)


#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"manual colors and levels correction",size=28,align=2))

resultC1= result1.ConverttoRGB24().ImageWriter(file= output,start = 1, type = "jpg", info = true)

#Finish and go home
Eval(result)


# END SCRIPT, BEGIN FUNCTIONS
#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int limit, bool _grey)
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

function filldrops (clip c)
{
super=MSuper(c,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
filldrops = mflowinter(c,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
return fixed
}

Prefetch(4)

#return last

Add these lines to the very top of your script:
SetMemoryMax(16000) #increase to 32000 or even 40000 if necessary.
threads = 16 #increase gradually up to 32 or 40. Any higher than that, the memory consumption will get close to your 64G (I think)
### Check speed, memory consumption and CPU usage with AVSMeter64 ###


Change the last line in your script ("Prefetch(4)") to this:
Prefetch(threads)

Also, put avstp.dll (https://forum.doom9.org/showthread.php?t=164407) into your plugins64+ directory. It appears to be quite beneficial for this particular script.

[videoFred]

Source files were jpg frames 2604 x 1738 resolution and the output frames are written as 1640 x 1280 and are intended to be exported to Blender 3D for final tweaking and resizing.

Why Blender 3D? It can all be done in one Avisynth script. If you want sliders you can tweak the script for AvsPmod.

Can you upload an original untouched image sequence somewhere? 50 frames or so...

Fred.

[Meloware]

Why Blender 3D? It can all be done in one Avisynth script. If you want sliders you can tweak the script for AvsPmod.

Can you upload an original untouched image sequence somewhere? 50 frames or so...

Fred.

100 frames of the source for this script are temporarily at: http://meloware.com/source/

Why Blender 3D? Blender is my Open Source "Swiss Army Knife" for graphics. I have been using it for 20 years and still haven't explored everything possible with it.
Blender has great potential in movie film restoration. The Compositor feature has a large bag of tools. There is the usual contrast/brightness and scale, flip, etc., but the color correction, gamma adjustments, and center box imaging positioning (translate) are very powerful. Perhaps the most important feature is that any of the Compositor settings may be animated. This means any adjustments in frame position, gamma, white/black levels and color corrections may be applied temporally, to compensate for momentary exposure or position. Professional film sources are well made for exposure, lighting, etc. Moment to moment adjustment for exposure and gamma are infrequent. Amateur films, however, usually have horrible lighting and can be greatly improved if you make the effort to make adjustment for each scene. Blender also features a tracker/stabilizer node. I have improved my work, even more, by using that feature in addition to AviSynth's stabilizer.

Blender is free and preparing for a major new release. I have included (in the source directory, link above) an example Blender file I used in one of my films. The source files are missing, but you can see the compositor setup and temporal edits I made.

AviSynth is amazing, and I am grateful for your creation of these scripts and the support I am getting. As I learn, I may find better alternatives to problems I now address in Blender. AviSynth appears to apply it's filters throughout a video according to the static script settings. I don't see any feature which allows temporal adjustment to change settings from moment to moment. You also need fast manual interaction to be able to edit these momentary overrides. Blender gives me the platform to make the final tweaks before rendering a final frame sequence.

[StainlessS]

Just a little question, is this script known to work ok (fast) where source are image files rather than video clip ?

EDIT: Perhaps try timing with same set up on video source, any difference ?

[Meloware]

Add these lines to the very top of your script:
SetMemoryMax(16000) #increase to 32000 or even 40000 if necessary.
threads = 16 #increase gradually up to 32 or 40. Any higher than that, the memory consumption will get close to your 64G (I think)
### Check speed, memory consumption and CPU usage with AVSMeter64 ###


Change the last line in your script ("Prefetch(4)") to this:
Prefetch(threads)

Also, put avstp.dll (https://forum.doom9.org/showthread.php?t=164407) into your plugins64+ directory. It appears to be quite beneficial for this particular script.

Wow -what a change! I have never seen this CPU so active. The same script is now rendering at 2.5 frames per second. The script has threads set at 16, yet the CPU is at 96% utilization with all 64 threads fully active. Memory consumption is at 20% of the available 64GB. I don't believe I have any more computer available to improve this performance more.
How much of this change is due to the new avstp.dll ? I noticed the zip containing a 32 bit version, and I installed that as well, in the appropriate 32 bit plugin directory.
I haven't tried AVSMeter64, as yet, but CPUID is the source for my system performance data, and ffmpeg reports the 2.5 fps frame render rate in the CMD box.
You guys are going to get tired of me saying "Thank you"!

[johnmeyer]

I assume the breakthrough increase in speed was from the avstp.dll addition since it didn't appear that he was getting any MT even with the hard-wired threads(4) statement. I also assume that the SetMemoryMax increase had no impact on speed since, in my experience, memory allocation is entirely a setting for stability, not performance.

Feel free to correct me, if these assumptions are incorrect.

BTW, the next thing to try is to change the threads setting to find the "sweet spot." This can sometimes be counter-intuitive, given the strange way in which multi-threading works in various flavors of AVISynth. The only way I've been able to tune this is to run the script for 30-60 seconds, and then note where the fps results end up after that time (the initial fps can be quite misleading because of caching from a previous run). Then, I change threads up or down, and repeat.

I only have eight cores, and with my system, with most scripts (including this film restoration script), threads=8 is almost always really slow and unstable. Based on experience, I usually start my tuning with threads=4 (i.e., half of my cores), and work up or down from there. For many of my scripts, I get the best performance, with stability, with threads=5, but I've had to go as low as 3 and sometimes have gotten as high as 7, so I have learned that I must always do these brief tests.

[Groucho2004]

Just a little question, is this script known to work ok (fast) where source are image files rather than video clip ?

EDIT: Perhaps try timing with same set up on video source, any difference ?I played with the script quite a bit, the type of source does not make a difference. It's mostly the high resolution that slows it down and consumes lots of memory/CPU.

[Groucho2004]

I assume the breakthrough increase in speed was from the avstp.dll addition since it didn't appear that he was getting any MT even with the hard-wired threads(4) statement. I also assume that the SetMemoryMax increase had no impact on speed since, in my experience, memory allocation is entirely a setting for stability, not performance.

Feel free to correct me, if these assumptions are incorrect.They are partially incorrect. The number of threads you pass to prefetch and the resolution make it necessary to increase the value for SetMemoryMax(). With 64 logical cores you need a pretty high prefetch value to get that CPU working. Avstp.dll helps but does not perform miracles.

[Groucho2004]

Wow -what a change! I have never seen this CPU so active. The same script is now rendering at 2.5 frames per second. The script has threads set at 16, yet the CPU is at 96% utilization with all 64 threads fully active. Memory consumption is at 20% of the available 64GB. I don't believe I have any more computer available to improve this performance more.Well, you just widened the AVS script bottleneck a bit. In order to optimise the script you have to play with the variables I mentioned and test without ffmpeg obfuscating the script performance.

[Meloware]

The following is the results from a series of tests I just performed with the script I posted earlier, plus the changes suggested by Groucho2004. Each test runs the same script and conditions, except for the indicated thread count settings.

The system is based on a Threadripper 2990WX, (32 cores/ 64 threads) with 64GB available ECC ram. The frame source and destination reads and writes are using SSD and M.2 drives. The video source is an image sequence of JPGs, at a resolution of 2604 x 1738 pixels. The output is another frame sequence, cropped and resized to 1640 x 1280. The chosen script processing option is “result1” (Autolevels, Autowhite).

The idle system is using 9% of memory, as reported by CPUID. The script is run via command line, using ffmpeg, and is allowed to run 500 frames, before the frame rendering rate is recorded from ffmpeg and CPU and memory utilization is read out of CPUID.

I hope this summary is useful for the developers and those who are considering a serious investment in a new machine.

Threads CPU Memory FPS
8 65% 17% 1.5
16 97% 20% 2.4
24 99% 30% 2.6
32 99% 34% 2.6

[Groucho2004]

This is the best result I get with that script on my 9 year old i5 2500K (4 cores/threads @4GHz):


SetMemoryMax(16000)
Using prefetch(10)

[Runtime info]
Frames processed: 100 (1000 - 1099)
FPS (min | max | average): 0.085 | 155632 | 1.679
Process memory usage (max): 6426 MiB
Thread count: 25
CPU usage (average): 98.9%


Command line was "avsmeter64 test.avs -range=1000,1099 -log". Even though the CPU only has 4 cores, the best performance was with 10 threads passed to prefetch().

[poisondeathray]

If a 4C/4T older CPU can get 1.6fps , that suggests that there are still some inefficiencies and his numbers still are low

Could it be that many plugins used Intel optimized compilers in the past ? IIRC there were some investigations into this in the past about unexpectedly poor AMD performance

[Groucho2004]

If a 4C/4T older CPU can get 1.6fps , that suggests that there are still some inefficiencies and his numbers still are low

Could it be that many plugins used Intel optimized compilers in the past ? IIRC there were some investigations into this in the past about unexpectedly poor AMD performanceHis numbers are with ffmpeg encoding. He has not used AVSMeter.

[videoFred]

Why Blender 3D? Blender is my Open Source "Swiss Army Knife" for graphics.

Ok I understand. But then why Avisynth? :D

Fred.

[Meloware]

Ok I understand. But then why Avisynth? :D

Fred.

It is a different tool, and does 95% of everything I need, very well, without a lot of hand editing. The occasional bump from the stabilizer gets corrected, in Blender, the minutia of levels/gamma differences, and center box cropping, from scene to scene, but your scripts and AVISynth is an excellent "wide brush" tool which, in my opinion, is absolutely needed in the work flow! This horrible pink fade we see, in old Eastman Color, I have yet to correct well in either Blender or Photoshop, but this script probably makes the best color corrections I could expect from a red faded film!

[Meloware]

His numbers are with ffmpeg encoding. He has not used AVSMeter.

Correct. And thank you for the polite nudge into using your amazing tool. Your posted example of the command was very tactful in helping someone not so familiar with AVSMeter's features.

So, below are the same tests, as monitored by AVSMeter64. It would seem my system's sweet spot is in setting the script to about 20 threads...
[Clip info]
Number of frames: 1586
Length (hh:mm:ss.ms): 00:01:06.083
Frame width: 1640
Frame height: 1280
Framerate: 24.000 (24/1)
Colorspace: YV12
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a

[Runtime info] Setting in script for 8 Threads
Frames processed: 500 (1 - 500)
FPS (min | max | average): 0.226 | 196079 | 2.078
Process memory usage (max): 4668 MiB
Thread count: 138
CPU usage (average): 57.5%

Time (elapsed): 00:04:00.601

[Runtime info] Setting in script for 16 Threads
Frames processed: 500 (1 - 500)
FPS (min | max | average): 0.181 | 104167 | 2.378
Process memory usage (max): 7818 MiB
Thread count: 146
CPU usage (average): 95.8%

Time (elapsed): 00:03:30.231

[Runtime info] Setting in script for 20 Threads
Frames processed: 500 (1 - 500)
FPS (min | max | average): 0.220 | 105263 | 3.223
Process memory usage (max): 8662 MiB
Thread count: 150
CPU usage (average): 89.4%

Time (elapsed): 00:02:35.153

Time (elapsed): 00:03:23.059

[Runtime info] Setting in script for 24 Threads
Frames processed: 500 (1 - 500)
FPS (min | max | average): 0.205 | 96154 | 3.231
Process memory usage (max): 9305 MiB
Thread count: 153
CPU usage (average): 89.2%

Time (elapsed): 00:02:34.740

[Runtime info] Setting in script for 32 Threads
Frames processed: 500 (1 - 500)
FPS (min | max | average): 0.098 | 69444 | 3.147
Process memory usage (max): 13708 MiB
Thread count: 162
CPU usage (average): 87.6%

Time (elapsed): 00:02:38.894

[poisondeathray]

Maybe 2 instances of 6-8 threads would yield higher numbers overall

[Groucho2004]

Maybe 2 instances of 6-8 threads would yield higher numbers overallQuite possible. Maybe Avisynth becomes less and less efficient with an increasing number of threads, particularly with scripts that run various temporal filters.

[Meloware]

Maybe 2 instances of 6-8 threads would yield higher numbers overall
I ran 2 instances of AvsPmod, and pointed one instance to frame 1, and the other to frame 750. I then hit "play" on each and allowed the ImageWriter function to write out frames as set in my result option "C1". Each instance of the script was set to operate with 12 threads.
CPUID reported the CPU was 95%+ occupied, with 35% memory use. I then reviewed the time stamps over a complete 6 minute period of rendering. There were 1050 frames total rendered frames, which suggests a total system performance of about 2.9 frames per second. This doesn't appear to be an exciting option, compared to a single instance running at 20 threads.

Groucho2004 - I noticed AVSMeter will allow me to run the script and the Imagewriter() function will write frames. Is this an option for rendering out of the script? The images are low quality. Do you know if there is a setting option to improve jpg quality? Lastly, AVSMeter will only allow a single instance to be run, at a time. Was this intentional? Maybe the option to run multiple instances might be good.

Since getting this system, it has been rare to find any software that shows any interest in using 64 threads. Blender tries, hard, but it seems like many of the legacy libraries and compile options have no concept of what to do with 32 cores.

The help I have been offered, here, has pegged all 64 threads to the limit. This is the first time I have seen a single instance of an app even come close. Perhaps the future processing algorithms may become more efficient, for my AMD chip, but at least I am seeing the resources in my investment finally being used. Life has already improved, dramatically. I certainly hope further development will improve performance on multi-core AMD chips, but I already appreciate the current improvement!

[Groucho2004]

Groucho2004 - I noticed AVSMeter will allow me to run the script and the Imagewriter() function will write frames. Is this an option for rendering out of the script? The images are low quality. Do you know if there is a setting option to improve jpg quality? Lastly, AVSMeter will only allow a single instance to be run, at a time. Was this intentional? Maybe the option to run multiple instances might be good.
There is an option in the AVSMeter.ini file ("AllowOnlyOneInstance" ) which, if set to '0', allows multiple instances of the program. Check the included documentation. However, if you just want to write an image sequence from a script I recommend using avsr (https://forum.doom9.org/showthread.php?t=173259).

As for the quality of the images I can imagine that png or tif will produce better results than jpeg. Check the documentation (http://avisynth.nl/index.php/ImageWriter).

[bassquake]

I forgot to take Autolevels out once I added GamMac. Autolevels never worked for me and I never, ever used it.

Also, I second Fred's recommendation to use a lower resolution.


I'd rather keep the high resolution capture for future proofing!

Anyhoo, took out Autolevels and the Denoise stuff as I prefer to maintain the film grain. Results are now at around 7fps in ffmpeg which will do for now.

Edit: Cranking threads variable to 64 when changing the Prefetch(4) to Prefetch(threads) gives me up 12fps! Looks like it forces the cpu to render at a much higher percentage. Nice :)

[pinterf]

Quite possible. Maybe Avisynth becomes less and less efficient with an increasing number of threads, particularly with scripts that run various temporal filters.
I suppose that memory bandwidth is also a significant limiting factor if not the biggest one.

[minusthebear]

Hey guys

Over the past year I have modified and updated this script to all 64 bit versions and equivalents as well as versions that use the GPU over CPU.

I will be posting my script here in the next month for everyone. Im still having a bit of trouble getting Mtools2 to utilize the GPU version of SVP, but im hoping to fix it soon.

The last plugin I cannot find a good 64 bit equivalent for is the unsharpen mask. So if anyone knows a good version for that let me know please.

Works great with 4k, 8k etc content. Much faster with GPU utilization.

[felicialee]

Thanks for a wonderful script and the results are great for colour 8mm film. :):)
I've also got some black and white 8mm films as well.

[SnillorZ]

The last plugin I cannot find a good 64 bit equivalent for is the unsharpen mask. So if anyone knows a good version for that let me know please.


@Bear did you see the earlier posts?

AVS+ Script Unsharpmask function (https://forum.doom9.org/showthread.php?p=1875711)

Warpsharp x64 DLL (https://forum.doom9.org/showthread.php?p=1876526)

Interested to see what your numbers are like on GPU vs CPU processing (based on these scripts) :script:

[bassquake]

If anyones interested, I posted a video showing the different stages of post-processing. I used VirtualDub2 mostly as have better control, but use AviSynth+ for some mild dirt removal and sharpening using settings posted on this thread.

Youtube Video (https://www.youtube.com/watch?v=lWE4eOX740w)

Appreciate comments.

[johnmeyer]

If anyones interested, I posted a video showing the different stages of post-processing.Very nice!

What software did you use for dirt removal? If you used RemoveDirtMC, what settings did you use?

All your other steps look really, really good, but I think you might be able to get better dirt removal.

[edit]Here is one of my "before/after" comparisons, using the scripts I've posted. I have it queued up to a section where there is a lot of dirt removal. I'm not doing this to get into a "who's script is better" discussion, but merely to point out that I think you can get better dirt removal:

https://www.youtube.com/watch?v=tBAHzO7rJS0&t=1m6s

and here is that same clip queued up to the point where the amount of dirt removal hits "ludicrous speed:

https://www.youtube.com/watch?v=tBAHzO7rJS0&t=3m51s

[bassquake]

Thanks johnmeyer!

Cheers for a look at dirt removal comparisons.

Yes, was RemoveDirtMC for dirt removal and a smidge of sharpening for the final touch.

I prefer to keep film grain which is why I don't use DeNoise, and kept dirt removal to a setting of 20 for the variable dirt_strength. I guess I could go higher but I like a bit of "character" on the film :D

[johnmeyer]

Yes, was RemoveDirtMC for dirt removal and a smidge of sharpening for the final touch.

I prefer to keep film grain which is why I don't use DeNoise, and kept dirt removal to a setting of 20 for the variable dirt_strength. I guess I could go higher but I like a bit of "character" on the film :DHmm ... If you are using the same version of RemoveDirtMC that I use, 20 should have given you a little more dirt removal (it is almost exactly what I use). However, I think the numerical setting interacts with the resolution you use. I do most of my film work at the NTSC SD resolution of 720x480 (non-square pixels). If you are doing your work at 720p or higher resolution, you might need to use a larger value in order to get the same level of dirt removal.

The two big downsides to using larger values of RemoveDirt are the disappearance of objects you want to keep (a thrown ball will almost always disappear, for instance), and the introduction of "gooey" artifacts where objects begin to morph instead of move.

[bassquake]

However, I think the numerical setting interacts with the resolution you use.

Probably. I create 1440 x 1080 videos! Looks great on a 55" tv :D

I might play around with upping settings more, but I do like some of the charm of flawed film!

[johnmeyer]

... I do like some of the charm of flawed film!Then why restore it at all? Along these same lines, many people prefer leaving the gate weave (the residual bounce) because that is an inherent quality of movie film, but you've removed that. What's the difference between removing all the gate weave, but only some of the dirt?

I got my start in media restoration by restoring 78 rpm recordings. I heard many of the same arguments back then, that removing the residual hiss and pops took away much of the "life" and "authenticity" of the original medium.

OK, that's a fair criticism, but I do have this to say: listening to restored shellac or vinyl recordings, or watching completely restored film sure is a lot easier on the ears and eyes, and lets you concentrate on the content of the media.

[bassquake]

Some dirt removal is better than none but I don't want to go overboard due to the issues you listed.

[minusthebear]

@Bear did you see the earlier posts?

AVS+ Script Unsharpmask function (https://forum.doom9.org/showthread.php?p=1875711)

Warpsharp x64 DLL (https://forum.doom9.org/showthread.php?p=1876526)

Interested to see what your numbers are like on GPU vs CPU processing (based on these scripts) :script:

Thank you! will incorporate this.

[minusthebear]

Ok, so here is my mod.

This is a 64-bit GPU enabled version of Fred's filter.

This requires AviSynth+ btw, if you want to use the prefetch enhancements,

If you don't want the prefetch, just comment out that line and you can use AviSynth.

Obviously needs virtualdub64 or another 64 bit program to capture.

You will most likely have to put the FFT3dGPU.dll and fft3dfilter.dll plugins in your system32 folder. If you have 32 bit versions there you will have to rename them with .bak and replace. Can't have both as far as I can tell.

There are 2 versions I created;

1. Fred 64 with MDegrain - This uses a GPU enabled SVPflow motion flow. The rest is 64-bit but CPU.
2. Fred 64 with GPU MDegrain - This utilizes the SPVConvert function to utilize the CPU with the MDegrain denoising filter. However it has artifacting and doesnt work properly. Maybe someone can fix it.

I will be honest, the speed enhancements are marginal at best, unfortunately there are many functions that the GPU is waiting for while the CPU handles them. Especially the AutoAdjust plugin.

However this is 64-bit now, meaning you will not have any issues with Ram overflow etc. The skys the limit, 4k, 8k, whatever you want.

Enjoy.

https://drive.google.com/file/d/1O-ACG2V-ptp5_LTv3Vpo6AxNtMf37YC_/view?usp=sharing

[Groucho2004]

You will most likely have to put the FFT3dGPU.dll and fft3dfilter.dll plugins in your system32 folder. If you have 32 bit versions there you will have to rename them with .bak and replace. Can't have both as far as I can tell.Firstly, these DLLs are plugins and should therefore be in the appropriate plugin directories. Secondly, on 64 bit Windows there are dedicated system directories for DLLs that need to be within the reach of the standard Windows search hierarchy (System32 for 64 bit, SysWoW64 for 32 bit). No renaming necessary/recommended.

[Natty]

1. Fred 64 with MDegrain - This uses a GPU enabled SVPflow motion flow. The rest is 64-bit but CPU.

can u pls give a basic example? :thanks:

[videoh]

@minusthebear

You are distributing one of my DLLs without my permission (DGDecodeNV.dll). It is not in the public domain. I demand that you remove it from your distribution. If you do not, I will ask the mods to take appropriate action.

[minusthebear]

@minusthebear

You are distributing one of my DLLs without my permission (DGDecodeNV.dll). It is not in the public domain. I demand that you remove it from your distribution. If you do not, I will ask the mods to take appropriate action.

Oh shoot, I didnt realize.

Its gone.

[minusthebear]

can u pls give a basic example? :thanks:

I'm not sure what you mean?

Do you want like a before and after type clip?

The end result is almost identical to the 32-bit FRED version.

This is just using 64-bit and GPU

[videoh]

Oh shoot, I didnt realize.

Its gone. Thank you, Sir.

[SnillorZ]

I'm not sure what you mean?

Do you want like a before and after type clip?

The end result is almost identical to the 32-bit FRED version.

This is just using 64-bit and GPU

Looking to see what your performance numbers are?

e.g. from @Groucho2004's AVSMeter to compare the CPU based script & plugins vs GPU ones

:thanks:

[SnillorZ]

Here is one of my "before/after" comparisons, using the scripts I've posted. I have it queued up to a section where there is a lot of dirt removal. I'm not doing this to get into a "who's script is better" discussion, but merely to point out that I think you can get better dirt removal:

https://www.youtube.com/watch?v=tBAHzO7rJS0&t=1m6s


John in this example of yours, there are a lot of vertical lines along with the dirt spots that you've managed to remove in the "after" clip.

Interested to know, was this all achieved with RemoveDirt?

I have some footage with vertical lines (some static, some with frame weave) that I was thinking would need another method for removal, including possibly frame by frame by hand for the "static lines".

[johnmeyer]

John in this example of yours, there are a lot of vertical lines along with the dirt spots that you've managed to remove in the "after" clip.

Interested to know, was this all achieved with RemoveDirt?Amazingly, RemoveDirt can sometimes remove gigantic spots (you'll easily see that in the beach scene) and even a few scratches, but only if the scratches move around a lot. Unfortunately, normal vertical scratches really cannot be removed by any automatic methods because they persist from frame to frame and don't move around much. As a result, automated algorithms really can't tell the difference between a scratch and a normal thin vertical line (fence, wallpaper, etc.). RemoveDirt, and other similar plugins like Despot, rely on the artifact lasting for only a single frame.

About the only way I know of to truly remove film scratches is to do it during transfer using a "wet gate," which involves putting a lubricant on the film, prior to transfer, which fills in the scratch and causes them to almost perfectly disappear. Your transfer equipment has to be able to accommodate wet film, which mine does not, in order to be able to take advantage of this "trick." Also, it involves extra steps, thus making the transfer take longer.

The man who started this thread, "VideoFred," uses a wet gate and has posted about it several times, both in this thread and elsewhere. Do a forum search on "wet gate" and his user name to learn more.

[bassquake]

Maybe try the DeScratch (https://avisynth.org.ru/descratch/descratch.html) tool for AVISynth to remove vertical scratches. Haven't tried it myself yet.

[SnillorZ]

The man who started this thread, "VideoFred," uses a wet gate and has posted about it several times, both in this thread and elsewhere. Do a forum search on "wet gate" and his user name to learn more.

Thanks John, yes I had researched wetgate transfers for emulsion damage. I'll revisit the footage I have to see if the amount that would benefit justify the cost of a rescan with wetgate.

Maybe try the DeScratch (https://avisynth.org.ru/descratch/descratch.html) tool for AVISynth to remove vertical scratches. Haven't tried it myself yet.

Think I had previously stumbled across Fizicks DeScratch plugin, but bypassed it in the pursuit of x64 bit plugins for Fred's/John's scripts. Will take another look and do some (x32 script) testing. :sly:

[johnmeyer]

I was never able to get ANY positive results from Descratch, but you should try it, just to see if you have better luck.

[manono]

I was never able to get ANY positive results from Descratch...
Me neither.

[bassquake]

Attached 64bit version of Descratch in case want to try it in your 64bit workflow.

[videoFred]

I had researched wetgate transfers for emulsion damage.

Only layer scrathes can be made (temporal) invisible with wetgate.
Unfortunately emulsion damage can not be fixed with wetgate.
(See also my turorial on Youtube)

However, if the emulsion damage change for frame to frame, then Removedirt() might remove it.

Otherwise you will have to look for other solutions.

Fred.

[minusthebear]

Looking to see what your performance numbers are?

e.g. from @Groucho2004's AVSMeter to compare the CPU based script & plugins vs GPU ones

:thanks:

Ok ill try that, but really, the biggest issue is that with 4k video the 32-bit version literally crashes, it fills up the 2gb of ram VirtualDub can access and creates a memory access violation.

Which isnt an issue with 64-bit.

The GPU helps a bit with speed but not much, ill post a comparison though.

[rjansen1]

Hi everyone,

I've only recently stumbled upon all the wonderful stuff you guys have created here when trying to restore some digitized Fuji Single 8 family movies. Unfortunately my dad threw away the original movies after he'd had them restored so I had to make do with the MPG files I extracted from the DVD. After removing duplicate frames and trimming some of the uglier splices I managed to get some pretty decent results with Fred's scripts to start out with and later on with John's scripts.

But of course I wouldn't be posting here if I didn't have any question. From what I read in the threads here GamMac is preferably used with a detect clip as this creates a more subtle / less harsh color correction. Unfortunately I've found that a lot of the movies I'm fixing have whites that are tinged pink-ish. GamMac without a detect clip seems to fix these fine:

https://i9.photobucket.com/albums/a56/dmrazor/avisynth/beach-nodc_zps9gfazc0w.png

with the detect clip the pink tinge remains although not a strong as before:

https://i9.photobucket.com/albums/a56/dmrazor/avisynth/beach-dc_zpsy6ha9adh.png

Of course the easy answer is to just go without the dc, but on some of the other scenes I find the color better with dc. Is this expected behaviour, or am I doing something wrong here? Or should I adjust my workflow and fix the movies scene by scene (not something I'm looking forward to tbh)

Cheers and thanks for creating all these great scripts!

Remco

[videoFred]

Hi everyone,

Hi Remco, welcome here :)

What DC clip have you used?

Fred.

[rjansen1]

What DC clip have you used?

I used this one from John's version of your script:

Baseclip = PreBorderFrame.crop(borderV,borderH,-borderV,-borderH,align=true).bicubicresize(W,H)
blank_black = Blankclip(baseclip, width=autolev_bord1,height=autolev_bord1)
blank_white= Blankclip(baseclip, width=autolev_bord1,height=autolev_bord1, color=$FFFFFF)
Average= baseclip.FredAverage().invert()
over1 = overlay(baseclip,blank_black, x=40,y=300)
over2 = overlay (over1,blank_white, x=160, y=300) \
.bicubicresize(width(baseclip)-(al2)*2,height(baseclip)-(al2)*2)
Detect = (al2 >1) ? overlay (Average, over2,x=al2,y=al2) \
.converttoRGB24(matrix="rec709") : over2.converttoRGB24(matrix="rec709")

I initially left all the settings at their defaults: (Al2 = 20; autolev_bord1 = 50; borderV=10; borderH=10), but I've also experimented with a smaller border. This leaves less of the pink tinge, but it doesn't dissappear.

Cheers,
Remco

[videoFred]

I used this one from John's version of your script:

Hmmmm... yes, I must take a closer look at John's version.
Can you upload an example clip?

Fred.

[rjansen1]

Hmmmm... yes, I must take a closer look at John's version.
Can you upload an example clip?

Yeah, sure. But will be on Sunday as I'm away tonight and tomorrow. What type of clip do you need: couple of seconds of the source AVI?

Cheers,
Remco

[johnmeyer]

There are many versions of "FredAverage" which is used to create the detect clip. This is the version I am using:

AVISynth FredAverage_26_dll_v0.01_20170414.zip

The DLL is dated 4/14/2017.

[rjansen1]

Ok, found time to prepare two clips:

pink clip (https://drive.google.com/open?id=1Hr-_39Lsc3hME4O_3hOt7mvhN4rjAN9W)
purple clip (https://drive.google.com/open?id=1O6tK2NUfrJhtzFr6Cf-egEquQULn8Ugn)

Both clips clean up to good white balance without using a DC, and keep the pink/purple tinge with DC.

I'm using FredAverage_25&26_x86_x64_dll_v0.03_20190115

Ciao,
Remco

[StainlessS]

There are many versions of "FredAverage"

Not that many versions really,


FredAverage_26_dll_v0.01_20170414.zip
FredAverage_25&26_x86_x64_dll_v0.02_20190115.zip
FredAverage_25&26_x86_x64_dll_v0.03_20190115.zip

/*
v0.01, First release. 2.60 only.
v0.02, 15 Jan 2019. Moved to VS2008, Added Version Resource + v2.58 + x64.
Added bool arg TV_YUV;
v0.03, 15 Jan 2019. BugFix version, fixed TV_YUV rounding. # Same day fix
*/


EDIT: The only difference between first and current version (apart from being now v2.58 and x64 versions)

TV_YUV, Default false, If True(And YUV), then photo negative invert around TV levels mid Y(125.5), rather than 127.5.

Default TV_YUV false, same as original v0.01.

TV Levels Y Mid point = (16 + 235) / 2.0 = 125.5. [ Same as (235-16) / 2.0 + 16.0 = 125.5 ]

EDIT: So, for Avs YUV TV_Levels Invert() filter is really a bit wrong, as invert is around 127.5 [0->$FF, $FF->0, $7F->$80, $80->$7F].
Maybe Invert() Filter should have something like TV_YUV arg, default false. [ignored for RGB]

EDIT:

G1=BlankClip(Width=128,height=64,Pixel_Type="YV12",Color_YUV=$7F8080)
G2=G1.Invert
G3=G2.Invert
B=G1.BlankClip(height=8)
G1=G1.RT_Subtitle("$%02X",G1.RT_YPLaneMin(n=0))
G2=G2.RT_Subtitle("$%02X",G2.RT_YPLaneMin(n=0))
G3=G3.RT_Subtitle("$%02X",G3.RT_YPLaneMin(n=0))
StackVertical(G1,B,G2,B,G3)

https://i.postimg.cc/7Y5BGwZs/Test-00.jpg (https://postimages.org/)

Fred Average same as Invert

TV_YUV=False
G1=BlankClip(Width=128,height=64,Pixel_Type="YV12",Color_YUV=$7F8080)
G2=G1.FredAverage(Invert=true,TV_YUV=TV_YUV)
G3=G2.FredAverage(Invert=true,TV_YUV=TV_YUV)
B=G1.BlankClip(height=8)
G1=G1.RT_Subtitle("$%02X",G1.RT_YPLaneMin(n=0))
G2=G2.RT_Subtitle("$%02X",G2.RT_YPLaneMin(n=0))
G3=G3.RT_Subtitle("$%02X",G3.RT_YPLaneMin(n=0))
StackVertical(G1,B,G2,B,G3)

https://i.postimg.cc/7Y5BGwZs/Test-00.jpg (https://postimages.org/)

FredAverage Y=125($7D) TV_YUV=False (same as Invert) [For TV Levels is WRONG-ish]

TV_YUV=False
G1=BlankClip(Width=128,height=64,Pixel_Type="YV12",Color_YUV=$7D8080)
G2=G1.FredAverage(Invert=true,TV_YUV=TV_YUV)
G3=G2.FredAverage(Invert=true,TV_YUV=TV_YUV)
B=G1.BlankClip(height=8)
G1=G1.RT_Subtitle("$%02X",G1.RT_YPLaneMin(n=0))
G2=G2.RT_Subtitle("$%02X",G2.RT_YPLaneMin(n=0))
G3=G3.RT_Subtitle("$%02X",G3.RT_YPLaneMin(n=0))
StackVertical(G1,B,G2,B,G3)

https://i.postimg.cc/LX2S2vjp/Test-04.jpg (https://postimages.org/)

FredAverage Y=125($7D) TV_YUV=true (TV mid Y = 125.5, 125[$7D]->126[$7E], 126[$7E]->125[$7D] )

TV_YUV=True
G1=BlankClip(Width=128,height=64,Pixel_Type="YV12",Color_YUV=$7D8080)
G2=G1.FredAverage(Invert=true,TV_YUV=TV_YUV)
G3=G2.FredAverage(Invert=true,TV_YUV=TV_YUV)
B=G1.BlankClip(height=8)
G1=G1.RT_Subtitle("$%02X",G1.RT_YPLaneMin(n=0))
G2=G2.RT_Subtitle("$%02X",G2.RT_YPLaneMin(n=0))
G3=G3.RT_Subtitle("$%02X",G3.RT_YPLaneMin(n=0))
StackVertical(G1,B,G2,B,G3)

https://i.postimg.cc/6QSHhQ51/Test-05.jpg (https://postimages.org/)

EDIT: And just for good measure, (16=$10, 235 =$EB)
FredAverage Y=16($10) TV_YUV=true (TV mid Y = 125.5, 16[$10]->235[$EB], 235[$EB]->16[$10] )

TV_YUV=True
G1=BlankClip(Width=128,height=64,Pixel_Type="YV12",Color_YUV=$108080)
G2=G1.FredAverage(Invert=true,TV_YUV=TV_YUV)
G3=G2.FredAverage(Invert=true,TV_YUV=TV_YUV)
B=G1.BlankClip(height=8)
G1=G1.RT_Subtitle("$%02X",G1.RT_YPLaneMin(n=0))
G2=G2.RT_Subtitle("$%02X",G2.RT_YPLaneMin(n=0))
G3=G3.RT_Subtitle("$%02X",G3.RT_YPLaneMin(n=0))
StackVertical(G1,B,G2,B,G3)

https://i.postimg.cc/SxM9WmRD/Test-06.jpg (https://postimages.org/)

EDIT:

if(invert) { // invert ?
if(tvy) {
ave = int(-(ave_D - 125.5) + 125.5 + 0.5); // TV_YUV Y mid = 125.5, invert, and Round
} else {
ave = int(ave_D + 0.5) ^ 0xFF; // PC_YUV Y mid = 127.5, symmetrical about 127.5
}
} else {


For YUV,
Invert() filter + FredAverage(TV_YUV=false), both mid Y = 127.5, symmetrical about 127.5.
FredAverage(TV_YUV=True), mid Y = 125.5, symmetrical about 125.5.

[Bernardd]

Hi Rjansen1,

An other way based on Stainlesss plugins.

I tried quickly my script on your videos. https://forum.doom9.org/showthread.php?p=1881328#post1881328

The first seem in accordance with default args :

https://i.postimg.cc/J4NWdYTV/frame-65.png

The secund need spimd args tune and manual spow action, thus with r_spmid = 0.2 , g_spmid = 0.5, b_spmid = 0.8 and g_spow = 0.8 :

https://i.postimg.cc/d07HgLqk/frame-45.png

[videoFred]

Both clips clean up to good white balance without using a DC, and keep the pink/purple tinge with DC.

It is not mandatory (is this good English? :o) to use FredAverage() on the DC clip. The idea of using a DC clip is to use a prepared clip before the Gammac analysis. For example, denoising/cleaning/cropping. Only in some cases, for example on scenes with water and blue sky, FredAverage() can help to avoid wrong Gammac() results.

So, in your case, I advice to use a DC clip without FredAverage().

Fred.

[videoFred]

I tried quickly my script on your videos.

Good results Bernardd! Gammac() gives very similar results.

Fred.

[rjansen1]

Only in some cases, for example on scenes with water and blue sky, FredAverage() can help to avoid wrong Gammac() results.

So, in your case, I advice to use a DC clip without FredAverage().
Thanks, that's good to know. And I think I've seen this on some scenes with water that seem to render better with FredAverage().

@Bernardd: I'll also give your script a go!

Greetings,
Remco

[StainlessS]

It is not mandatory (is this good English? ) to use FredAverage() on the DC clip.

Just like a native speaker, perfectly good, Fred ol' chap :)

[rjansen1]

@Bernardd: tried replacing Gammac with your script and found that it gives good results on some scenes, but then again worse results on others. Also it doesn't seem to work multithreaded. Is that correct?

I guess there's no magic bullit and in the end it's a matter of selecting the method that works good for 80% of the movie and fix the other bits using different methods. From what I read that's what you guys do as well :)

Cheers,
Remco

[Bernardd]

@rjansen1, thanks for you feed back.
For multithread, i do not know.

[videoFred]

Just like a native speaker, perfectly good, Fred ol' chap :)

Haha I do my best with a little help from.... my friends no I mean Google. :D

Fred.

[johnmeyer]

@Bernardd: tried replacing Gammac with your script and found that it gives good results on some scenes, but then again worse results on others. Also it doesn't seem to work multithreaded. Is that correct?

I guess there's no magic bullit and in the end it's a matter of selecting the method that works good for 80% of the movie and fix the other bits using different methods. From what I read that's what you guys do as well :)

Cheers,
Remco1. It definitely works multi-threaded, with VD+.

2. There is no automated solution to either gamma or color correction. I always create two versions of my "restored" footage, one with GamMac (or equivalent tool) and one without, and I then cut between them on my NLE timeline to create the final version.

[rjansen1]

1. It definitely works multi-threaded, with VD+.
Ok, then I'm doing something wrong. I replaced Gammac with Bernardd's script and found the overall script failing on Prefetch(2).

I'm using Avisynth+ x64, what's VD+?

Cheers,
Remco

[johnmeyer]

Ok, then I'm doing something wrong. I replaced Gammac with Bernardd's script and found the overall script failing on Prefetch(2).

I'm using Avisynth+ x64, what's VD+?

Cheers,
RemcoMy bad. I wrote that in a hurry and didn't check. VD would be VirtualDub, but I really meant to say AVISynth+.

I'm using AVISynth+ version 0.1 r2508, 32-bit, running under WinXP Pro 32-bit.

[rjansen1]

No problemo. I'll tinker with it some more over the next couple of days and try to figure it out.

Thanks,
Remco

[rjansen1]

@Bernardd @Johnmeyer

I can't seem to get Bernardd's script to run multithreaded. Did a little testscript:

AviSource("Testclip.avi")

SetFilterMTMode("DEFAULT_MT_MODE", 2)

Import("C:\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\RGBAdapt_Awb_Process function.avs")

ConvertToRGB32()
AWB_Func()
converttoYV12()

Prefetch(threads=2)
Without the Prefetch line the script runs fine, with Prefetch it gives me an error on that line saying "I don't know what 'Fmt' means". I'm using Avisynth+ x64.

As far as I can tell 'Fmt' is a variable used somewhere within Bernardd's script. I've tried different ways of importing the script, even importing "RGBAdapt_Awb_Process script call.avs" since it seems to hold a bunch of variables. But so far I haven't been able to figure out how to get it to use MT.

Any tips are welcome therefore :)

Cheers,
Remco

[johnmeyer]

I've not used AVISynth+64. I only use the 32-bit version. Many of the plugins are 32-bit and I simply don't know how they behave in a 64-bit environment.

There is no reason whatsoever to use the 64-bit version. If you want to switch between several versions of AVISynth, this is such a common need that Groucho built a simple utility that lets you select which version you want to use. You just run this when you need the 32-bit version, do your work, and then when you need the 64-bit version, you switch back.

[real.finder]

I've not used AVISynth+64. I only use the 32-bit version. Many of the plugins are 32-bit and I simply don't know how they behave in a 64-bit environment.

There is no reason whatsoever to use the 64-bit version. If you want to switch between several versions of AVISynth, this is such a common need that Groucho built a simple utility that lets you select which version you want to use. You just run this when you need the 32-bit version, do your work, and then when you need the 64-bit version, you switch back.

you can have both 32 and 64 in the same time, switch between them depends on the program/tool/encoder you use

edit: also you can run both in same time in mpp (mp_pipeline)

[real.finder]

@Bernardd @Johnmeyer

I can't seem to get Bernardd's script to run multithreaded. Did a little testscript:

AviSource("Testclip.avi")

SetFilterMTMode("DEFAULT_MT_MODE", 2)

Import("C:\VideoTools\AvisynthRepository\AVSPLUS_x64\plugins\RGBAdapt_Awb_Process function.avs")

ConvertToRGB32()
AWB_Func()
converttoYV12()

Prefetch(threads=2)
Without the Prefetch line the script runs fine, with Prefetch it gives me an error on that line saying "I don't know what 'Fmt' means". I'm using Avisynth+ x64.

As far as I can tell 'Fmt' is a variable used somewhere within Bernardd's script. I've tried different ways of importing the script, even importing "RGBAdapt_Awb_Process script call.avs" since it seems to hold a bunch of variables. But so far I haven't been able to figure out how to get it to use MT.

Any tips are welcome therefore :)

Cheers,
Remco

it's known problem with runtime (http://avisynth.nl/index.php/Runtime_environment) and mt (Prefetch) that didn't fixed for long time (maybe it will be ever)

[rjansen1]

I've not used AVISynth+64. I only use the 32-bit version. Many of the plugins are 32-bit and I simply don't know how they behave in a 64-bit environment.

There is no reason whatsoever to use the 64-bit version. If you want to switch between several versions of AVISynth, this is such a common need that Groucho built a simple utility that lets you select which version you want to use. You just run this when you need the 32-bit version, do your work, and then when you need the 64-bit version, you switch back.
I'm using Groucho's installer already. Have tried running my testscript on 32-bits AVS+ in Vdub2-32 but get the same "don't know what 'Fmt' means" error.

Could (any of) you maybe post a simple example of how you use Bernardd's script in a multithreaded setup? And tbh... if I can't get it to run multithreaded that's no biggy. That one render will just take a little longer.

Cheers,
Remco

[StainlessS]

Could (any of) you
See Real.Finder last line,
it's known problem with runtime and mt (Prefetch) that didn't fixed for long time (maybe it will be ever)

I never use MT, its just not worth the hassle (so it takes a bit longer - who cares - Maybe can do multiple encodes at once instead of MT).

[rjansen1]

See Real.Finder last line,
Yeah, I read that. Just seemed to read John saying he had it working multithreaded, so I wondered how.

But you're right; it's not an issue.

Thanks,
Remco

[StainlessS]

Maybe can do multiple encodes at once instead of MT

MakeMultiPartScripts & FFMS_MakeMultiPartScripts:- http://forum.doom9.org/showthread.php?t=176386&highlight=MakeMultiPartScripts


# FFMS_MakeMultiPartScripts.avsi

Function FFMS_MakeMultiPartScript(String FnList,String "OutPath",Int "Parts") {
/*
FFMS_MakeMultiPartScripts v0.04 ::: Requires RT_Stats, Avs+ OR GScript.

FFMS_MakeMultiPartScript(String FnList,String "OutPath"="",Int "Parts"=4)
FnList, No default. Multi-line string of avs filenames(1 or more).
Source scripts to split must use FFMS2 source filter.
Source scripts clip length must be at least (20 * Parts) frames else error abort.
This function sets a Global FFMS_MakeMultiPartScript_Glb variable, can be detected in source
avs scripts so as to determine if called by this function (for whatever reason).
OutPath, Default "", Same Directory as source file directory. Path (must exist) where output scripts written.
Where OutPath = default "", and multiple input files with different directories given in FnList, OutPath will use input file directory
for each input file (even when different directories, will not be different paths where fileSelector client script used).
Use "." for current directory.
Parts, Default 4. Number of parts that source script is split into.
*/

[johnmeyer]

Yeah, I read that. Just seemed to read John saying he had it working multithreaded, so I wondered how.I looked into this, and I have a mystery for all the AVISynth developers and gurus.

Here it is:

My film script runs multi-threaded without any problem if I set the "play_speed" to 24 fps. However (and here is where it gets weird) if I set it to 23.976, it won't run multi-threaded and simply hangs on loading.

Well, I thought, it must be an integer/float issue. So, I tried 24.0 fps instead of 24.

Worked fine.

I then tried 24 * 1 / 1

Worked fine.

I then tried 24 * 1.0 / 1.0

Nope, that hung. So did 24.0 * 1 / 1

And, as I stated above, simply entering 23.976 also made the script hang.

The play_speed variable only gets used one time in the script, and that is as the main variable to AssumeFPS. That's it. Somehow that causes MT to fail and AVISynth to hang.

I am using AVISynth+ 0.1 r2508 32-bit under XP 32-bit.

[edit]I decided to simply remove the assumefps() statement since I am already feeding the script 23.976 sound film. Once I removed that, the film speed was now 23.976 and the script ran multi-threaded without a problem.

AVISynth sure is a mystery at times. [end edit]

[Reel.Deel]

I am using AVISynth+ 0.1 r2508 32-bit under XP 32-bit.


I am not sure if it will fix the problem, but why not update to the latest AviSynth+ (r2772) (https://github.com/pinterf/AviSynthPlus/releases)? r2508 is over 2 years old and there has been lots of bugfixes since then.

[bassquake]

Does anyone know what the min and max values are for the following AVISynth plugins:

- RemoveDirtMC (dirt_strength)
- UnSharpMask (strength, radius and threshold)

Its not mentioned in their wikis, only the default numbers are shown.

I ask as I'd like to use sliders in AvsPMod. Thanks.

[videoFred]

Does anyone know what the min and max values are for the following AVISynth plugins:

- RemoveDirtMC (dirt_strength)
- UnSharpMask (strength, radius and threshold)

Its not mentioned in their wikis, only the default numbers are shown.

I ask as I'd like to use sliders in AvsPMod. Thanks.

Hi Bassquake,

I do not know the max. values but I assume you will get an error message if you set the values over the limit. (if there is a limit)

Anyhow, you will never need dirt_strength above 100 for example.

For unsharpmask I suggest the following max. values:
strength: 200 radius: 6 threshold: 10

And if you need more, it's very easy to change the max. value from the AvsPmod sliders, as you know of cource.

Fred.

[bassquake]

Hi Bassquake,

I do not know the max. values but I assume you will get an error message if you set the values over the limit. (if there is a limit)

Anyhow, you will never need dirt_strength above 100 for example.

For unsharpmask I suggest the following max. values:
strength: 200 radius: 6 threshold: 10

And if you need more, it's very easy to change the max. value from the AvsPmod sliders, as you know of cource.

Fred.

Thanks, I went with:

Dirt Strength: 100

UnSharpmask Settings (Similar to VirtualDubs):
Strength: 0 - 200
Radius: 0 - 25
Threshold: 0 - 255

Doesn't seem to error no matter how large the values are!

One last thing, is there a way to hide the variables showing as highlighted? Its a bit cluttered.

https://i.postimg.cc/tTKTry5b/hide-variables.png (https://postimg.cc/cgc0Hp2k)

[videoFred]

One last thing, is there a way to hide the variables showing as highlighted? Its a bit cluttered.


Set AVSPMod as followed:
Options > Program settings > User Sliders > Create user sliders automatically > Select "Type string(filename)" Please unselect all other options.

Fred.

[bassquake]

Set AVSPMod as followed:
Options > Program settings > User Sliders > Create user sliders automatically > Select "Type string(filename)" Please unselect all other options.

Fred.

Cool, thanks.

I kept "type string (list)" selected so I could change colorspace options on the fly.

Would be nice to be able add show/hide each of the variables separately. Oh well. It'll do for now. :thanks:

[imsrk48]

UPDATE 20/06/2012
-------------------------

Newest version of the script:
http://www.super-8.be/avisynth/Film_Restoring_vs_06_2012.zip

- improved autolevels
- improved stabilisation
- improved sharpening
- less plugins
- plugins are now in "plugins' folder.
- better helpfiles



=================================================================

UPDATE 06/06/2011
--------------------
The script discussed in this thread only works good with high quality film transfers.
The digital source must be progressive and there should be no duplicate frames.

Many people, however, are having already a digital transfer of their old 8mm films on DVD.
My script is useless here. But you can use some parts of it.
For those people, FPP has made this exellent thread:
http://forum.doom9.org/showthread.php?t=161493

===================================================================







ORIGINAL POSTING 14th january 2009:
-----------------------------------------
I have posted an example clip on Vimeo to show the people the incredible power of Avisynth.

http://www.vimeo.com/2823934

With special thanks to Fizick, Didée, Manao for making these exellent plugins available for all of us. :thanks:

Special thanks to Josey_Wells too for the multithreaded version.
Averaging more then 4 frames in MVDegrainMulti() works very well for removing film dirt spots.

The example clip is showing the result of a special film restoring script I have made.

The filmtransfer itself was done by me, but that film needs a second wetgate transfer.
There was a dirt spot on the CCD from my camera too. But that's easy enough to fix.

I just could not wait to show you all these first results. [emoji14]

Fred.Fabulous Work [emoji108][emoji119]

Sent from my LG-M700 using Tapatalk

[vidschlub]

I have a collection of about 8-15 8mm reels here my ma left me after she passed away.

Having seen the examples at the start of this thread, it feels like it would be an injustice to just hand the reels to any schmo for $10 a reel and having them convert them.

I did some reading this evening on a product called a Wolverine Pro, however it sounds like it rips at 20fps, not the original 16 or 18fps.
So I guess my question is, what's the best format to rip these films, so ensure I can clean them up in some capacity? Is 1080p enough (Wolverine Pro) - should I rip at some kind of 'literal' frame rate (actually 16 / 18 fps, or even 60fps somehow?)

I'd hate to have half blurry frames or something that can't be cleaned up.
Also, it sounds like some scanning tools 'zoom in' the film frame too close, losing some of the details in the image. I see several purists mentioning ripping the films twice, once zoomed in for the actual action and once zoomed right out allowing you to even see the film holes on the side.


What are peoples thoughts and to the op, thanks for the excellent scripts, I'm not in a position to use them yet but the results you seem to have achieved are well well impressive.

[videoFred]

I did some reading this evening on a product called a Wolverine Pro,

It's a cheap device, not bad for that price. But it will never give the quality you want.

Is 1080p enough (Wolverine Pro)

More than enough for 8mm film. Also, aspect ratio from 8mm film is 4:3. That gives 1440x1080.
I capture at 1392x1032 with my machine vision camera.
But the quality from the digital transfer is much more important than the size.


- should I rip at some kind of 'literal' frame rate (actually 16 / 18 fps, or even 60fps somehow?)

What we need is one film frame = one digital frame. No duplicate frames.
Then play speed does not matter and can be easily changed afterwards.

Fred.

[johnmeyer]

To add and clarify to two important things VideoFred said:

1. The device must be "frame accurate" which means you get exactly one frame of film on each frame of video, without blends of adjacent frames (like you get if you just point a video camera at a movie screen), and without any flickering from a projector shutter being closed. I quickly skimmed the Woverine literature, and it does look like it is frame accurate.

2. As Fred said, you can set the speed of the playback after the capture. This is like speeding up or slowing down a movie projector: the same exact frames are shown, but they are just projected at a faster or slower speed. However, there are target speeds that should give you a result which gives you motion that is the same speed as the real-life event that was filmed. Super 8 film is always 18 fps. Sound 16mm is always 24 fps. Really old film can be any frame rate because the camera was either hand-cranked, or it used a wind-up mechanism that was often not very accurate. Having said that, most 16mm and 8mm silent film was shot at a speed around 16 fps, so that is always a good starting point.

You will find that some amateur movie cameras had a "film saver" speed which would slow down the film to 10 fps or slower. You just have to look at the result and decide whether you need to speed it up or slow it down.

BTW, your Wolverine apparently stores the captured movie in h.264 MP4 format. This is OK if you are just going to capture and view, but if you plan to edit the film, this "delivery format" is most definitely not the ideal. It usually doesn't play as smoothly on your NLE timeline as a "capture format," and it also will tend to introduce additional digital artifacts by the time you are finished with your project. This isn't a huge deal, because the artifacts will probably be fairly small (depending on how much compression the Wolverine uses), but it is most definitely not professional and not ideal.

I also agree with VideoFred that you don't need to capture or edit in full HD for 8mm film, and probably not Super 8 either. Even 16mm film can be captured at less than full HD with very little degradation. I had a chance to test this when some 1934 16mm film I had transferred was licensed to the Smithsonian Channel. They wanted to have the transfer re-done by a "Hollywood" transfer house, using a professional Spirit transfer system. I had the transfer house send the transfer to me before I sent it to the Smithsonian Channel. As I expected, their transfer, which was delivered in 1920x1080 ProRes, did a little better job on shadows than my projector/camcorder system (it was very contrasty German film stock), but the sharpness and increase in detail, while most definitely evident, was not strikingly better. It was a matter of 10-20% better (I'm trying to put my evaluation into numbers), not twice as good, or four times as good, which is how much more resolution 1920x1080 is compared to my 720x480 (non-square pixel) transfers.

[bruce356]

Hi, I have VirtualDub2_43803 and AviSynth v2.6 Official installed.

I am attempting to apply Video Fred's Script (01_A_Film_Restoring_Frame_Interpolation) and save the result.

I am new to this process of using VD & AviSth and to apply a scripts.

Not sure how to go about saving only the output frame as "Save Video" in VirtualDub saves both the original and result1.

I have read a large portion of this forum and only found one reference to saving by Patrick "Quote - Where is the modified clip stored?" but the reply by Video Fred is not detailed except partly to say "you must save the output somewhere" and am not sure if Patrick had the problem of it saving both windows.

Example image attached.
https://www.dropbox.com/s/xznvu9oyyu5wv0a/VirtualDub%20Sample%20image.jpg?dl=0
Hoping you can help, or point me to an explanation of how to.

Thanks and regards - Bruce

[kedautinh12]

These are the avs specific dll's that I copy to system32/SysWOW64

AvsRecursion.dll
fftw3.dll # same as below file, name edited only
libfftw3f-3.dll
libsndfile-1.dll



please share me link download AvsRecursion.dll and libsndfile-1.dll, can't fint 64 bit ver

[Groucho2004]

please share me link download AvsRecursion.dll and libsndfile-1.dll, can't fint 64 bit verThere is no 64 bit AvsRecursion.dll.

[StainlessS]

There dont seem to be an x64 libsndfile-1.dll either.

[tebasuna51]

...I am attempting to apply Video Fred's Script (01_A_Film_Restoring_Frame_Interpolation) and save the result.

Not sure how to go about saving only the output frame as "Save Video" in VirtualDub saves both the original and result1.

The 01_A_Film_Restoring_Frame_Interpolation.avs is a script to compare different effects. You can select the effect to compare at the beginning of the script, by default:

#PARAMETERS
#------------------
result="resultS1" # specify the wanted output here

At the end of the script there are some options:
#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"result1: autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"result2: autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"result3: autolevels, manual color correction",size=28,align=2))
...

Like you see resultS1 is the output you obtain in your image:
Both the original and result1

If you want only the result1 (autolevels, autowhite) you must change at the begining:


#PARAMETERS
#------------------
result="result1" # specify the wanted output here

[StvG]

There dont seem to be an x64 libsndfile-1.dll either.
Here (https://ln2.sync.com/dl/1cf8de120/2txzzw2t-asx7mbxj-gxwquync-dxeuea7d) if someone wants it.

[StainlessS]

Thanks StvG :)

[bruce356]

The 01_A_Film_Restoring_Frame_Interpolation.avs is a script to compare different effects. You can select the effect to compare at the beginning of the script, by default:



At the end of the script there are some options:


Like you see resultS1 is the output you obtain in your image:
Both the original and result1

If you want only the result1 (autolevels, autowhite) you must change at the begining:

Thanks very much for the reply, I did read the help files but for some reason could not figure out how to save output screen only. With your help, information and pointing me in the right direction, am able to save only the output.
Regards - Bruce

[kedautinh12]

Here (https://ln2.sync.com/dl/1cf8de120/2txzzw2t-asx7mbxj-gxwquync-dxeuea7d) if someone wants it.

are you have AvsRecursion.dll x64?? i have src of AvsRecursion but can't compiler become 64 bit?? can anyone can do it?? thanks :)
http://web.archive.org/web/20071017115323/http://home.arcor.de/kassandro/AvsRecursion/

[Groucho2004]

are you have AvsRecursion.dll x64?? i have src of AvsRecursion but can't compiler become 64 bit?? can anyone can do it?? thanks :)
http://web.archive.org/web/20071017115323/http://home.arcor.de/kassandro/AvsRecursion/I don't know which plugin in this context even needs AvsRecursion. I know of RemoveGrain and RemoveDirt but both have been "modernised" by Tp7/pinterf and don't depend on this ancient contraption.

[StainlessS]

No idea if these still have AvsRecursion requirement, but Google search "AvsRecursion site:avisynth.nl" :- https://www.google.com/search?source=hp&ei=akK4XZe1B4GMlwTxuqLIBA&q=AvsRecursion+site%3Aavisynth.nl&oq=AvsRecursion+site%3Aavisynth.nl&gs_l=psy-ab.3...4071.13274..13734...5.0..0.302.2983.10j11j1j1....3..0....2j1..gws-wiz.......33i160.5leCYBB-RJY&ved=0ahUKEwiXkp-bzcHlAhUBxoUKHXGdCEkQ4dUDCAc&uact=5

brings up AnimeIVTC and DeFlicker as well as RemoveGrain.

EDIT: I think Kassandro dll's that require AvsRecusion will send an "AvsRecusion.dll Not found" type message to DebugView window(Google),
but still work ok (but slower) without it.

[StvG]

are you have AvsRecursion.dll x64?? i have src of AvsRecursion but can't compiler become 64 bit?? can anyone can do it?? thanks :)
http://web.archive.org/web/20071017115323/http://home.arcor.de/kassandro/AvsRecursion/
Here (https://www.jottacloud.com/s/2092849485676b2490eb148d2a082efa741) compiled from the source you gave.

[kedautinh12]

Here (https://www.jottacloud.com/s/2092849485676b2490eb148d2a082efa741) compiled from the source you gave.

wow, thanks

[dgalland]

Hello,

Not exactly the subject of this thread on the excellent videofred scripts that I use for a long time but I think that may interest those who are asking the question of building a DIY frame by frame scanner.

You can find many projects on this subject with very different solutions and all sorts of interesting ideas. I would like to introduce mine using a PI camera and a modified projector controlled by a Raspberry PI.

I specifically studied the following aspects:
- Use and calibration of the PI camera with a non-stock lens
- Comparison of PI cameras V1 and V2
- Adaptation of the projector and lighting
- Python programming of a client-server application for controlling a stepper motor and the camera.
- Improving the dynamics of the image with an HDR technique

Code and documentation (In french, Google translate is your best friend):
https://github.com/dgalland/yart

Dominique

[maks-lee]

Hello!

Dont know if it's correct thread.
I would like to upgrade my camcorder, which i use to capture 8mm films to machine vision camera
I already have a c-mount obectiv, but have no clue what camera to choose
Could someone please give some links to cameras with different cost

Thanks
Max

[johnmeyer]

Hello!

Dont know if it's correct thread.
I would like to upgrade my camcorder, which i use to capture 8mm films to machine vision camera
I already have a c-mount obectiv, but have no clue what camera to choose
Could someone please give some links to cameras with different cost

Thanks
MaxThe really good transfer systems use a custom camera because that lets you customize the transfer curve, so you can get better shadow detail without blowing out the highlights. Search the Internet for the author of this thread. His real name is Freddy van de Putte. Here is a link to his Vimeo page where he hints at what he built.

https://vimeo.com/user678523/about

I don't have time to search, but I think there is a page where he describes quite a bit about his equipment and how he configured it.

As for the camcorder, you want HD (although SD is actually enough for most 8mm and Super 8 work). Ideally you want to be able to capture uncompressed, something that most low-end camcorders don't allow.

[maks-lee]

The really good transfer systems use a custom camera because that lets you customize the transfer curve, so you can get better shadow detail without blowing out the highlights. Search the Internet for the author of this thread. His real name is Freddy van de Putte. Here is a link to his Vimeo page where he hints at what he built.

https://vimeo.com/user678523/about

I don't have time to search, but I think there is a page where he describes quite a bit about his equipment and how he configured it.

As for the camcorder, you want HD (although SD is actually enough for most 8mm and Super 8 work). Ideally you want to be able to capture uncompressed, something that most low-end camcorders don't allow.

Thanks!

[maks-lee]

Hello!

I have a modified 8mm projector, which captures one frame at a time. As i think, after capture i will need to create an avi file to feed to Fred's script first.
Is it possible to modify script somehow to import sequence right away after capture ?
I have tried ImageSource with no luck.

[manono]

"With no luck?"

You'll have to be more specific than that. Please post your script (just the ImageSource line is enough), how the pictures are numbered, and the error message VirtualDub gives you when trying to open that script.

[johnmeyer]

I have a modified 8mm projector, which captures one frame at a time. As i think, after capture i will need to create an avi file to feed to Fred's script first.
Is it possible to modify script somehow to import sequence right away after capture ?
I have tried ImageSource with no luck.I have not used ImageSource

ImageSource Wiki (http://avisynth.nl/index.php/ImageSource)

but I have used similar features in VirtualDub and also my NLE, Sony Vegas Pro. The only issues I have had with either of those is that


You have to have a numbering sequence that conforms to what the image import function expects;
You have to use an image type that the function can read (i.e., JPG, TIFF, PNG). Some only read a few types
And, with some, you cannot have any gaps in the numbering sequence.


To avoid gaps, you can re-number the sequence prior to import.

[bassquake]

Hello!

I have a modified 8mm projector, which captures one frame at a time. As i think, after capture i will need to create an avi file to feed to Fred's script first.
Is it possible to modify script somehow to import sequence right away after capture ?
I have tried ImageSource with no luck.

Hi maks-lee. I posted my script earlier in this forum which shows ImageSource in use. See: https://forum.doom9.org/showthread.php?p=1877304#post1877304

The important lines are:

film="E:\Videos\imgseq\SAM_%05d.jpg"
Set the name and number to match how many digits your images use.

trim_begin=1 trim_end= 28160
play_speed=18
Set how many frames used.

source1= ImageSource(file=film, start=trim_begin, end=trim_end, fps=play_speed).FlipVertical().converttoYV12()
This is the ImageSource line which calls the previous variables. FlipVertical flips the image as theyre upside down when captured and then converted to YV12 so the plugins like AutoLevels can be used.

Hope that helps.

[maks-lee]

@manono, @johnmeyer, thanks!
Hi maks-lee. I posted my script earlier in this forum which shows ImageSource in use. See: https://forum.doom9.org/showthread.php?p=1877304#post1877304

The important lines are:


Set the name and number to match how many digits your images use.


Set how many frames used.


This is the ImageSource line which calls the previous variables. FlipVertical flips the image as theyre upside down when captured and then converted to YV12 so the plugins like AutoLevels can be used.

Hope that helps.

Thanks! It seems, that now i can feed sequence to VD after capture, many thanks! I will continue to read forum (still so much interesting)

[GMJCZP]

Here (https://ln2.sync.com/dl/1cf8de120/2txzzw2t-asx7mbxj-gxwquync-dxeuea7d) if someone wants it.

The link for AVSRecursion x64 is down, could someone please re-upload the file?

[Groucho2004]

-deleted-

[StainlessS]

Thanks G2K4.
I note that x86 version is 5KB, x64 38KB, checked and x64 is static linked (no runtimes required).
I also note that old x86 version (that I have) requires runtimes MSVCR71.dll which is I think from VS 2003 (maybe also VS Toolkit 2003).
Just observations.

Dont know what x64 plugs might require AvsRecursion x64, but I've added to my x64 system32 dll storage folder, and also system32.
I'll shout if something goes BANG !

[Groucho2004]

Thanks G2K4.
I note that x86 version is 5KB, x64 38KB, checked and x64 is static linked (no runtimes required).
I also note that old x86 version (that I have) requires runtimes MSVCR71.dll which is I think from VS 2003 (maybe also VS Toolkit 2003).
Just observations.

Dont know what x64 plugs might require AvsRecursion x64, but I've added to my x64 system32 dll storage folder, and also system32.
I'll shout if something goes BANG !
I made a new VC10 build with 32 and 64 bit binaries. These require VS2010 runtimes as it should be. I have added it to my Avisynth Stuff.

Please discard the build I posted before.

[StainlessS]

Roger Wilco, over and out.

EDIT: AvsRecursion by Rainer Wittmann (think that might be Kassandro)
For:- AvsTimer, RemoveGrain, ReduceFlicker, RemoveDirt.

[real.finder]

IIRC, AvsRecursion is not needed in RgTools

[GMJCZP]

I made a new VC10 build with 32 and 64 bit binaries. These require VS2010 runtimes as it should be. I have added it to my Avisynth Stuff.

Please discard the build I posted before.

Thanks Groucho 2004!

[GMJCZP]

IIRC, AvsRecursion is not needed in RgTools

The filters with T letter are not included in rgtools.

[frubsen]

Hi all, I recently purchased a moviestuff Retro-8 Pro scanner and have been transferring films and would like to try out fred's script. I have a pretty good knowledge of avisynth, but am just looking for a little direction on what is the best route to take.
I have been exporting 1080p PNG image sequences of the captured film from the moviestuff software. Would it be wise to clean up the edit in Premiere and then re-export a PNG image sequence which I would then process with Fred's script?

Any advice on where to start with a script? I know there is Fred's original script, and then John has posted some modifications as well as some others.... I've been reading through a lot of this thread but it's pretty overwhelming.

Thanks

[poisondeathray]

Would it be wise to clean up the edit in Premiere and then re-export a PNG image sequence which I would then process with Fred's script?



It depends on the footage specifics and what you mean by "clean up the edit in Premiere"

What were you planning to do in PP exactly ? What are you doing in PP that cannot be done in the avs script? or what are you doing in PP that would benefit the results of the script processing ?

[frubsen]

It depends on the footage specifics and what you mean by "clean up the edit in Premiere"

What were you planning to do in PP exactly ? What are you doing in PP that cannot be done in the avs script? or what are you doing in PP that would benefit the results of the script processing ?

I guess removing junk(leader, etc) off the start and end, cleaning up any blank spots in the middle...although I guess technically that could be done in avisynth too.

[poisondeathray]

I guess removing junk(leader, etc) off the start and end, cleaning up any blank spots in the middle...although I guess technically that could be done in avisynth too.

Yes, that's a good idea

[Groucho2004]

IIRC, AvsRecursion is not needed in RgToolsCorrect.

[DiablosDemon]

I'm trying Minusthebear script on page 64. When I try to run it I get this

AVSMeter 2.9.9.2 (x64), 2012-2020, (c) Groucho2004
AviSynth+ 0.1 (r2772, MT, x86_64) (0.1.0.0)

I don't know what 'super' means.
(C:\Users\ME\Desktop\Seemtobe working\newtest.avs, line 205)
PS C:\Users\ME\Desktop\Seemtobe working>

Referencing this part of the script.

#DEGRAINING/SHARPENING
#...................................................................................................................................................................


super_mv = MSuper(cleaned, pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)

Any Ideas why?

If I comment out those lines it runs. But it's running at 30FPS when I set it to 16 and I only have trim set to 300 but it says 1315 (this changes with different frame input but I can't see a pattern). Where is it getting 30FPS why 1315 frames? I am inputting PNG image files. Thanks for any help on this.

AVSMeter 2.9.9.2 (x64), 2012-2020, (c) Groucho2004
AviSynth+ 0.1 (r2772, MT, x86_64) (0.1.0.0)

Number of frames: 1315
Length (hh:mm:ss.ms): 00:00:43.833
Frame width: 2048
Frame height: 1600
Framerate: 30.000 (30/1)
Colorspace: YV12

Frame (current | last): 461 | 1314
FPS (cur | min | max | avg): 1.729 | 0.099 | 564.0 | 2.991
Process memory usage: 1872 MiB
Thread count: 39
CPU usage (current | average): 24.7% | 23.8%

Time (elapsed | estimated): 00:02:34.127 | 00:07:19.646

[DiablosDemon]

Do I even need FPS in the script if I'm just using image files? Seems I just need one in and one out as I'm not making a video yet. Inputting 2048X1600 PNG files from a 8mm film converted from DPX files.

[DiablosDemon]

I had a earlier post that referenced the problems I was having with Minusthebear's script but I think It went to a moderator. It was automatically putting the FPS at 30 no matter what I changed. hopefully that post will show soon.

[DiablosDemon]

I found why the 30 FPS was hard wired at the very bottom of the script.

interpolated= SVSmoothFps(autolevels, super, vectors, "{rate:{abs:true, num:30, den:1}}", mt=threads).sharpen(last_sharp,mmx=false).sharpen(last_sharp,mmx=false).blur(last_blur,mmx=true)

Changed it to 16 and it was better, still something else going on with this to change how many frames I input. must be part of this line I'm still trying to figure it out.
I'm still not sure about the super in my previous post.
I was hoping this would work faster using the GPU for some tasks.

[`Orum]

Anyway, a quick and dirty UnsharpMask replacement for Avs+
<snip>
This seems to cause artifacts on the right side of the frame for me, near (but not on) the edge, from the top of the frame to the bottom, though it varies in brightness/contrast and is less visible in some areas than others.

Is there another option for UnsharpMask() with 64 bit and HBD support? I'm guessing it's due to the blur filter, so maybe just replacing that would be enough.

Edit: Looks like you can just swap out the original blur with aBlur from AWarpSharp2. Here's the modified script, which doesn't have the artifacts from the original, but also will no longer work with RGB:
function UnsharpMask(clip clip, int "strength", int "passes", bool "highq", int "threshold") {
strength = default(strength, 64)
passes = default(passes, 3)
highq = default(highq, true)
threshold = default(threshold, 8)

blurclip = aBlur(clip, passes, highq ? 1 : 0)
e = "x y - abs " + String(threshold) + " scaleb > x y - " + String(strength/128.0)+ " * x + x ?"
Expr(clip, blurclip, e, "", "") # U and V are copied
}

[dgalland]

I use avisynth to restore 8mm films scanned frame by frame with the classic steps, creating the film from jpeg images, deshake, cleaning, denoise, sharpen and color adjustment.
To automate the process I created a set of batch scripts. The processing is carried out step by step with an intermediate lossless codec, par exemple UTvideo.
So the first steps do not need to be repeated if you change for example the color adjustment script.
Obviously you have to have a lot of disk space !

Each step processes the result of the previous step, for example with these scripts to process the film 01:
do join images 01
do concat join 01
do deshake concat 01
do clean deshake 01
do adjust clean 01
do adjust_gammac clean 01
do compare adjust adjust_gammac 01
do render264 adjust 01

Note:
- Avisynth and all plugins are in 64bits MT
- For the deshake, I preferred the virtualdub deshake plugin
- You can change the avisynth script for a step and use your favorite plugins or add other steps.

You can find these scripts in my github project

https://github.com/dgalland/Restore

[spotty]

Hi, I have tried JohnMeyers 2017 script which can be found at https://forum.doom9.org/showthread.php?t=144271&page=55 about 2/3 the way down.

This worked very well with my 8mm film digitized by a wolverine.
It was the first time I had seen sharpening work in my setup without any drawbacks, and the colour correction was very good.

I'm wanting to use it a bit differently though, I don't want any change to the resolution nor any frame rate change.

So what I'm intending to do is crop and stabilise with virtualdub & deshaker (with it filling in the black gaps created by stabilisation), then use all the other parts of JohnMeyers scripts.

I've been able to separate the autolevels and gamma sections, but separating the denoise and sharpening seems a big challenge as they seem to interact with the stabilisation.

I'm hoping someone has already done this or something like it, any ideas?
I'm thinking of looking at dgalland's scripts (above) but I'm trying to get closer to a solution rather keep opening up new options.

[johnmeyer]

Hi, I have tried JohnMeyers 2017 script which can be found at https://forum.doom9.org/showthread.php?t=144271&page=55 about 2/3 the way down.

...

So what I'm intending to do is crop and stabilise with virtualdub & deshaker (with it filling in the black gaps created by stabilisation), then use all the other parts of JohnMeyers scripts.

I've been able to separate the autolevels and gamma sections, but separating the denoise and sharpening seems a big challenge as they seem to interact with the stabilisation.VideoFred's scripts were pretty modular, and I made them even more so in my version of his scripts.

I've cut out sections many times, and usually all you have to do is assign the output variable from that section to the input variable for that section. This eliminates that section. So, for instance, if you want to remove the stabilization section because you've already done that in Deshaker, take the input variable, "cropped_source", and assign that to the output variable, "stab". So, you delete the stabilization section and replace it with this:

stab = cropped_source

[MIGLIO74]

Yes feel your pain, welcome to the wonderful world of open source, you can check out when you like but you can never leave. :devil:

It might not cost you much in hard cash but you will need to invest your time to get the best out of it and find what your looking for - but you've already figured that.


Yes lots, but not all in one place that your asking for.



My own personal interest and workflow is still a work in progress to find the best balance of processing performance vs result quality.

However I had intended posting all that I have gathered together over the last several weeks in case it can be of help to someone like yourself, as I know it would have been a help to someone like me.

Just keep in mind that some of the tools / Avisynth plugins do and will get superseded by the brilliantly helpful dev's here in Doom9 and elsewhere. So you will need to keep an eye out for something that could be bigger and better, so to speak.

So if you can wait a little, stick around and I'll find somewhere to host the collection of programs/tools/plugins/example avs and bit of documentation to provide some pointers. In next week or 2.

I can't wait for it to be ready :)
I'm waiting for this, thank you

:thanks::thanks::thanks:

[frubsen]

I'm trying to piece together a stabilize only script, and I'm using the one in the original post here.

However, I can't seem to get it to add black bars to the final output and looking at the script it doesn't look like the parameters are even used.

Does anyone have a stabilize only script they can post? I'm searched through this thread but can't seem to find specifically what I'm looking for.

Thanks

[johnmeyer]

In my version of the script, the stabilization is done with these three lines:

stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)


All you do is first crop any black bars from the video (because they'll wander around the edge after you've done the stabilization) and then do these three lines.

These are my default stabalizing parameters:
#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=10 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=10
est_left=40 est_top=40 est_right=40 est_bottom=40 #crop and contast values for special Estimate clip
est_cont=1.1 #Too large a value defeats stabilization


For film shot on a tripod, you can use much smaller stabilization parameters because you are only trying to remove gate weave. For wildly shaky video, you might have to increase the parameters.

[frubsen]

Thanks John

Been playing around with it and it gives decent results. Although I've noticed what almost look like dropped frames in some cases. And it's not even in cases where there is a lot of camera movement, but it almost looks like its dropping frames. Is this normal?

[johnmeyer]

I've done a LOT of work with various motion stabilization packages (I wrote a ten-page guide to Deshaker fifteen years ago which used to be all over the video sites, but has become harder to find). I've used Deshaker a lot; I own Mercalli; I obviously use DepanStabilize; I got started using Motionperfect; and I've trialed Twixtor.

None of them drop frames, nor do they ever give the appearance of frames being dropped.

You would need to post your entire script in order to figure out what is going on. In addition, it is possible that you did some incorrect decimation further upstream, before putting the video into the stabilization script. If you put the shaky video into your NLE, and then put the stabilized version below it, so you can A/B between them, are they the same length and, when you find a frame where you think a skip has happened, does the non-stabilized version show a frame that didn't make it to your final version?

[StainlessS]

(I wrote a ten-page guide to Deshaker fifteen years ago which used to be all over the video sites, but has become harder to find).
Would that be the one that I posted a link to as French translation of your work, if so then I think I also posted (and you took) a
copy of the translation back to english, you have it somewhere [and will be on-site somewhere].

EDIT: Here tis:- https://forum.doom9.org/showthread.php?p=1854682&highlight=deshaker+french#post1854682


Perhaps I should re-post my tutorial on my own web page, for those who don't speak those other languages.


Yeh, perhaps [ maybe you get a Sticky - then easy for you to find it :) ]

[videoFred]

I'm trying to piece together a stabilize only script, and I'm using the one in the original post here.

However, I can't seem to get it to add black bars to the final output and looking at the script it doesn't look like the parameters are even used.

Does anyone have a stabilize only script they can post? I'm searched through this thread but can't seem to find specifically what I'm looking for.

Thanks

Here it is:
https://forum.doom9.org/showthread.php?t=175669

Fred.

[johnmeyer]

If frubsen wants to use Deshaker, here is a link to my old Deshaker guide:

John Meyer Guide to using Deshaker (https://drive.google.com/file/d/1hgJGIyxH1Sdbm8Cl19dyD_rUxm09ylnD/view?usp=sharing)

[frubsen]

Thanks guys, going to look through all this and find what works best

[loonvf]

Hi All,

I have been reading some of this thread and replaying some of the scripts.
First I started with Fred's package "Film_Restoring_vs_06_2012.zip" and as certain plugins did not work, I tried the "Restore" scripts from D Galland (last post a few messages up in this thread).
Also there I encountered issues with plugins.

Basic question: how can I make sure to have the right plugins available on my system?

Any help is much appreciated getting a 8mm restore flow up and running.

Thanks in advance

[chmars]

Hello Users, Masters,

This post is long, I apologize.
Jump to "THE IDEA" to get it brief.

I hope this is the good place. My goal (and I can't wait), is to admire the magic of the Fred's/John's scripts on my images. But first, I have something to solve.

THANKS
I'm new to this forum and here specially to learn and use Avisynth, Fred's and/or John's scripts. The work they have done creatind, documenting, explaining and helping users is beautyfull. As well is beautyfull the work of all of those who wrote or adapted plugins and tools.
The atmosphere on this thread is also very collaborative and constructive. All along il, more than 10 years, no disputes, gentleman arrangements and elegant solutions to disagreements are found. This is rare.
Now, it's my turn to share, if this system appears to be efficient and good.

HISTORY
I spent/enjoyed hours (days, actually) of posts and discussions reading to understand the logic of the process. My brain is producing smoke actually and I am still very confused though having some programming skills.
I am used to Linux and havn't used windows since WinXP. So, there is work until I can uses these so powerfull scripts and programs.
At the moment, with AVS, I am able to read a video file and apply some basic filters.
Installations and use achieved actually easyer on Linux than windows, for me.
But I believe all filters/plugins are not (yet?) there for these scripts to be used under Linux.
I began filming the image on a screen, dealing with the flikcering as I could with the analogic potentiometer of the unmodified projector. This gave me bad results with mixed frames. Reason for the modifications I'll submit here to happen.

MY CONSTRAINTS
If I achieve to build a good quality and workflow, I'd like to hire the service of doing 8/S8mm transfers. I need not only very good quality transfers but also a not too slow setup. This is why I have choosen to use a projector instead of a frame by frame system, beside of the accessibility of the system, which is easyer than having to buy a machine vision camera and using it through software (Windows), syncing the whole thing to the projector.

THE IDEA
As I understood, a first mandatory step to achieve, prior to use the scripts, is to have a frame accurate capture of the footage.

The main idea is to use a LED signal on the side of the filmgate as mark for software pseudo-synchronisation with the projector.
The LED is turned on while the projector changes from one frame to the next one. (the LED is captured on the side of the framegate)
This was inspired from JohnMeyer's system where the software detects the framechange to be able to keep only one video frame for each film frame.
I didn't feel confortable with the idea of relying on software to determine which image to throw away. (And was upmost too lazy, as I feared having to tweak a lot to find the right parameters. "will it work on a totally different roll or will I have to re-set all the params?" and so on.
Intuitively, this solution also seems to me faster to calculate than calculating if image are mixes. Especially while pans for example.
The sound idea to sync is also good but I know nothing about sound electronics...

THE SETUP EDIT:Detailed setup + pictures here (https://forum.doom9.org/showthread.php?t=182868)
At the moment, I am shooting with a borrowed Sony FS7 and a Canon MP-e 65mm macro.
The projector has been modified to receive a microcontroller which is driving the LED and the PWM for the AC motor speed control.
I removed all blades except approx 15% width of the one passing behind the gate while the frame is changing.
This rest of blade triggers the microcontroller via an optical endstop. This allows to control the speed +/- precisely and permits to program the moment when the LED has to be lit.

MY GRALE
So, as first step, I need to find out how to tell avisynth this:
"Each time You see the blue light, throw the frame and keep only one of the next ones, until You see the blue light again!"
I can say it in english and some other language but not yet in the way we speek in AVSland.

MY QUESTIONS
Do You think the way I took is good?
Have You got a hint, a starting point where I could begin to search for the adapted plugin?
(If You really insist, I could tolerate some lines of code fulfilling the goal) :D


By advance thank you for having read till here.
If someone is interrested, don't hesitate to ask details/pictures/code/videos/shopping list.
While this post is on the forum, I'll prepare a clip so you can see how the capture looks like.
Best Regards

[chmars]

Here are some seconds of a capture made with the LED indicating the frame change:
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/MIDa0EQO5amzqSR/download

There are still other points to solve:
-I am not satisfied with the encoding coming out of the camera but the alternative is only XAVC (from what I found for the moment) which is very heavy.
-The focus seemed ok during capture but, probably due to compression, it seems here not to be very sharp.
-The image is kind of dancing/wandering. I have to find out why. I presume that the camera is slightly vibrating. This diagonal effect might be due to rolling shutter.

[chmars]

Oh, I am sorry. My post explaining the whole thing is not yet here. Maybe because it was a bit long, has to be approved by moderators?

[johnmeyer]

I developed a 16mm capture system that uses a projector from which the shutter has been removed. I developed software which removes the pulldown frames. Since you have an LED to indicate pulldown, you should have an easier time developing the field removal software.

I used a 30 fps interlaced camera, which meant that I had to deal with fields. A better approach would be to use a 60 fps progressive camera. You projector needs to be running at 24 fps or less. Any faster and you cannot guarantee that you will be a pristine frame of video for each frame of film.

I strongly suggest that you increase the shutter speed on your camera to something above 1/50 or 1/60 of a second. This will avoid any fuzziness if the camera captures the film just as it is coming to rest in the projector gate. I use 1/1000 of a second, but with your system, probably 1/250 will be just fine.

Make sure to turn off auto-focus. Also set white balance to something fixed. I balance mine off the bare bulb, with no film in the gate.

Some bouncing of the film is normal. This is called gate weave, and even expensive projectors used in movie theaters (if there are any film projectors left) caused the film to bounce around. You would always see this on the title of the movie, if it was fixed: it would bounce around.

[chmars]

Hi John, Hi everybody,

Thank You so much for all what you gave and for this answer.
Yes, thank to all your explanations, I chosen the LED way in the hope to make computing faster to get a frame accurate capture as this is the mandatory step to use your incredible scripts.

This thread is beautiful in terms of civility and generosity. Lots of thanks to all, sharing since + than 10years.

I am very new to AVS and Windows (Linux user). Still puzzled! But studying. At the moment, with AVS, I am able to read a video file and apply some basic filters.

My post describing the whole system + idea was too long.
I'll try to make it shorter:
Borrowed Sony Fs7 + MP-E macro (manual), 50fps, 1/600th second progressive.
MPEG 2 422 (Too compressed? I don't see grain any more)
Projecting at 18 fps, LED and speed controled by a microcontroler, pwm, optical endstop. Almost bladeless.

So, as first step, I need to find out how to tell avisynth to keep one frame between every blue light.

Have You got a hint, a starting point where I could begin to search for the adapted plugin?
(If You really insist, I could tolerate some lines of code) :D

[StainlessS]

So, as first step, I need to find out how to tell avisynth to keep one frame between every blue light.
Post a sample.

[chmars]

Thanks StainlessS,
I posted one before up here but it was not very good.
Here is a new one in XAVC-L, as it comes out of the camera and in mp4 after some grading and cropping.
Sample (https://www.swisstransfer.com/d/51e54f83-95d5-4506-bb0a-92a6eac00742)

[StainlessS]

Sorry, I did not see [backtrack to your prev post] the sample, I've no idea what this "Blue Light" is, hence the request.

[chmars]

Olala, I really started the wrong way.
My presentation post was too long and will never appear.

I'll try to do it shorter and hopefully will I be clear enough this time.

I am planing to do S8/8mm transfers.
I'd like my captures to be frame accurate, guess why ;)
Because nobody shouls live without Fred's and/or John's script.

I am using a projector and shooting directly into the gate.
Film at 18 fps, video at 50 fps.
The blue LED is lit while the film frame is changing.

The idea is to use the LED signal to tell Avisynth:

"
if image contains blue on the side,
remove it and keep only one until next blue light.
"

This was inspired from JohnMeyer's system where the software detects the frame-change to be able to keep only one video frame for each film frame.

Intuitively, the LED solution seems to me faster to calculate than calculating if image are mixed.

[StainlessS]

I downed both clips, the mp4 with MD5 of "b3f520fb147bf030a84c72476512d4a6"
will not load/play in anything I've tried and just shows this in MediaInfo

General
Complete name : D:\_BLUE_LED\MBGV0403.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 41.0 MiB
IsTruncated : Yes

Does it play your end, and in what player. Also is MD5 correct.

EDIT: Not sure what this means
IsTruncated : Yes
but has non zero data all the way to the end.

[videoh]

The MP4 sample plays fine for me in DGIndexNV and MPC-HC.

[videoFred]

Borrowed Sony Fs7 + MP-E macro (manual), 50fps, 1/600th second progressive.


Overall quality is pretty good.
Can you not trigger the Sony Fs7 to take pictures? (photos)

Fred.

[chmars]

Hi, thanks for your answers and interest!

@ Stainlesss
The MD5 I get is 41A41D305177148A957BC72F6BE730D0 after re-downloading the mp4 clip.
It plays in anything like vlc or WMP but also from Avspmod.

But I may have done (only) wrong things re-encoding it as I really know nothing about formats, codecs...about video actually frankly said.

Overall quality is pretty good.
Can You not trigger the Sony Fs7 to take pictures? (photos)

Fred.
Thank You,
This so-unfinished-work took already some efforts, I admit.
I didn't find a way to trigger the FS7 for stills.

But as the cam is not mine, as it is expensive, I didn't want to plug my projector's microcontroler in any way into the camera.
The same with sound. My idea in the beginning was to use sound to pseudo-sync with AVS but I didn't want to mess with cables to the cam.

Fred, do You think the result could be significantly much better with a stopped frame while capturing like in your system?

My secret final goal is to be able to do all the job, including the use of Your/John's version script into Linux.
The use of a Machine Vision Camera adds one more layer that I'll have to "de-Windowize". It also add some costs and some software to learn.
Do You think I made a wrong choice?

[johnmeyer]

You are doing a lot of things that will take a lot of time without any improvement in the end result. Just find an old Windows XP or Windows 7 computer and use that. The Machine Vision Camera is probably a good thing, depending on the make and model. It is my understanding that you can program the gamma curve, something that could make a massive difference in the final quality, given the difference in film's gamma curve compared to normal video. Getting good shadow detail is always the problem when using cameras (as opposed to scanners) when capturing film.

[StainlessS]

The MD5 I get is 41A41D305177148A957BC72F6BE730D0
OK, I got a bad download.

EDIT: I downloaded them separately, not in the package, and also the MFX file went bad after
frame 356 [about 1/3 clip length), dont know if it was supposed to play to end.
I presumed that both were just file cuts by size [no regard for contained video].

[chmars]

You are doing a lot of things that will take a lot of time without any improvement in the end result.
...


Do You refer to my ideal to run the whole workflow under Linux or about my attempt to remove frames with the blue LEDlight mark?
About Linux, yes, You are probably right. But it is a very cosy and powerfull environment from my point of view. It's personal though, let's not start a debate :--) On the immediate productivity, You are probably very right. Also because not all the tools needed for the script to run are ported there.
At the moment, I'll be very happy if ever I reach some goal in windows already.

About camera adjustable gamma curve, I'll document myself. It's very interesting. I could consider using a Machine Vision Camera in a second time however.


...
also the MFX file went bad after
frame 356
...
I presumed that both were just file cuts by size [no regard for contained video].

I an very sorry for these inconveniences. They are not cut by size. The MXF file is as it came from the camera and has 850 frames. The mp4 was edited quick balanced in Resolve and saved to mp4.
Here is a new transfer (https://we.tl/t-Z5o2yzQmpA) via another service.

Mp4 md5:6314E25461B323B2309B59B85731C27C
MXF md5:0820A24F2D0AB47483FA5E3BF5DADB69

[wonkey_monkey]

"
if image contains blue on the side,
remove it and keep only one until next blue light.
"

Perhaps I've misunderstood, but I counted a couple of occasions where there seemed to be two separate good frames between blue lights, e.g. frames 6 and 7, frames 86 and 87,

[chmars]

You understood correctly. I made a mistake with the mp4 file frame rate in Resolve.
New mp4 file + md5 + link updated in my previous post.

[johnmeyer]

Perhaps I've misunderstood, but I counted a couple of occasions where there seemed to be two separate good frames between blue lights, e.g. frames 6 and 7, frames 86 and 87,Yes, that is an inevitable outcome when capturing this way. It gets really tricky to write the decimation software. No matter what you do there will be duplicates that must be removed. It took me a couple of years to perfect the decimation software for the setup I described in the link I provided above. It is totally unique to my own contraption and the OP will have to write his own.

[wonkey_monkey]

The LED idea is nice but from the updated sample I can still imagine problems. There are either one or two "LED" frames, and most of the time it's obvious which has more blue and should be removed, but sometimes it isn't, e.g. frame 38 and 39.

Frames 314 and 315 are another problem. The one with most blue is actually the good frame.

I notice the new sample is only 25fps rather than 50fps as stated in the description post. At 50fps things might be easier... if you could get it to the point where you could guarantee capturing the frame change LED every time, you could then pick the frame equidistant between the two "peak" LED frames, but I suspect that by that point you could just use an offset and a fixed "distance" to pick out good frames (that's assuming the projector keeps to a certain tolerance, of course).

[poisondeathray]

T
I notice the new sample is only 25fps rather than 50fps as stated in the description post.

In the old sample , the MXF is 50p, the MP4 is 25p

In the new sample, the MXF is 50p, the MP4 is still 25p

The MXF is the same for both, and appears to be flipped vertically (No problem to "fix" with flipvertical)

[chmars]

Forget about that mp4 file!
Thank You for your answers ans suggestions.


...
it isn't, e.g. frame 38 and 39.

Frames 314 and 315 are another problem. The one with most blue is actually the good frame.
...


Yes, please look at the MXF one instead, wich is the original.
I apologize about my poor ability to reencode a file for now, and for the resultant loss of time for some of You.
The mp4 seems to have blended/mixes frames...I have to clear that.
About playspeed fps, 50 or 25, it seemed to me not being an issus as long as we have every frame to work on. But I might miss something...
In the old sample , the MXF is 50p, the MP4 is 25p
In the new sample, the MXF is 50p, the MP4 is still 25p
The MXF is the same for both, and appears to be flipped vertically (No problem to "fix" with flipvertical)

Yes, again, sorry for that 50-25 error, forget about the mp4 one. (or directly upload the knowledge on "how to generate a correct mp4 with avisynth" to my brain. I am alas not yet there ATM.)
Vertical inversion is normal as the file comes directly from the camera. Not a pleasure to look at but still allows to work on the G/B sorting question.


At 50fps things might be easier... if you could get it to the point where you could guarantee capturing the frame change LED every time, you could then pick the frame equidistant between the two "peak" LED frames, but I suspect that by that point you could just use an offset and a fixed "distance" to pick out good frames (that's assuming the projector keeps to a certain tolerance, of course).

Yes, was shot at 50 fps - 18 fps projected, all the film frames are present at least once.
Constructing an offset or a pattern looks a bit risky:
Although the projector speed is microcontroler+pwm controlled, it is not timecode controlled. In the time, there can be a shift with the camera cadence.

I would rather go this way:

B = Bad frame (thresholded blue light presence)
G = good frame (no blue light)

Example starting from frame 314 iccluded, to 335:
G B GG B GG B G BB G BB G B GG B GG B

We sometimes have GG, sometimes BB.
But we always have at least one G between one or some B.
So if we say:


"keep only one G between each B/each group of B", it should work, don't You think?

Step by step, more or less human spoken, it would give:
If B, throw away and flag "last was B"
else if G, and flag is "last was B", keep and flag "last was G"
else if G, and flag is "last was G", throw away
and so on...

Shouldn't this work?

My final goal is to use these so incredible and exiting scripts from Fred and the modified version from John on the footage I'll have to capture.

But I am a bit offtopic now. Amn't I?
Should I start a new thread?

Because the question I have to face now is
"How to detect a blue (instead of black) spot into a certain portion of the image with avisynth?".

[videoFred]

Fred, do You think the result could be significantly much better with a stopped frame while capturing like in your system?

Yes, because it's so simple. I capture straight to a "growing" AVI, one filmframe=one digital AVI frame. It can run at 15fps, but to stay at the safe side, I always capture at 12fps.
Capturing frame rate does not matter with this sytem. The created AVI file can have any frame rate, even when capturing at -say- 4 fps.
Play speed from the projector can be changed while capturing, the result is always one film frame= one AVI frame.
Then I run my script on the AVI file and.... done.


My secret final goal is to be able to do all the job, including the use of Your/John's version script into Linux.
The use of a Machine Vision Camera adds one more layer that I'll have to "de-Windowize". It also add some costs and some software to learn.


But it can be simple. For example, we do not need all the bells and whistles from the camera and the camera software.
I always set camera settings to neutral. Sometimes a litte bit of gamma correction, that's all.
All the rest can be done in post with Avisynth.

Fred.

[StainlessS]

chmars,
yes, might be best to start your own thread,
In the mean time, I'll post below, and delete & re-post in your thead when new thread opened.

EDIT: Sciprt moved to here:- https://forum.doom9.org/showthread.php?p=1942049#post1942049

[chmars]

This is incredible, great, thank You StainlessS, very much! I am going to study it next night(s!)
Ok, as said, I made a new thread, more specific to my question (https://forum.doom9.org/showthread.php?t=182809).
Should have done before, sorry for that.

Fred, thank You for your answer. I'll consider the solution of an MVC. But I also need something not too slow. i'll see in the future after having taxed my system and earned some money if I succeed :--).

John, about Gamma, the camera I am using is given for 14 stops range and, from what I understand, is not so poor in shadows level resolution if used with S-Log3. I'll check the manual to see if and how to reduce the range to fit the projected film one, which fills about two third of the histogram. Do you think this is the right approach concerning gamma (out of MVC, I mean)?

[chmars]

Thanks to Avisynth, and the script and help from users, my capture contraption is now working very well.
If interested, you can see detailed description, sample, code and pictures here (https://forum.doom9.org/showthread.php?t=182868).
And another sample here (https://forum.doom9.org/showthread.php?p=1943094#post1943094).
The most difficult part now comes: understand how deshaking and dedirting work !

[Wamberto00]

Dear all,

I own a Reflecta Super8+ scanner (https://www.filmscanner.info/en/ReflectaSuper8Scanner.html) which creates 1 jpg per scanned frame (it takes nearly 1 hour to scan 1 minute of footage ☹) and saves them all in a folder. When it’s done scanning the software automatically merges all the JPG’s into an AVI file (codec ID MJPG, compression mode Lossy, color space YUV, Chroma subsampling 4:2:0, 8 bits) of about 300mb for 3 mins of footage.

It does not delete the JPG files so there’s also an option to generate an AVI with your own settings. In order to keep as much detail as possible I used VirtualDub2 to generate an AVI file (uncompressed RGB/ YCbCr; RGB24), which is about 10gb.
In order to use this file in Fred script I first did a batch resize in IrfranView (size of each JPG is 1175 * 863) and in order to run the script it needs to be dividable by 2.
When I use the large uncompressed AVI file as an input for Fred’s Script everything looks fine, except that it looks like a broken mirror (see attached photo). The issue seem to go away when I change the Cleft, Ctop, Cright and Cbottom values from 32 to 0, but later on -when I edit my clips in DaVinci Resolve- it looks like the left and right border of the clip is doing funky stuff (‘jumping’).

Is there anybody who can advise what would best settings to convert the JPG’s into an AVI file without losing details?
Also, I do my editing in DaVinci Resolve and for some reason this software doesn’t accept AVI files as an input (funny enough the only AVI file it seems to accept is the AVI file generated by the scanner) so I guess I need to convert the output of Fred’s script to an MP4 file. What would be your suggestion to do this, again without loosing too much detail?

Hopefully someone can help me. A few pages earlier I read about a similair issue, but I didn’t quite get it.
Thanks and keep up the great work. It’s great that so many people have interest in preserving this great old footage.

Here's the photo of the 'broken mirror'

https://www.dropbox.com/s/ydjt2u9srq189cm/broken%20mirror.JPG?dl=0

[chmars]

Maybe you could just trim to remove 1 pixel to make the dimension even, instead of resizing the whole image.
I believe it's better not to resize, as long as possible.
Then, try to see what happens in Fred's script and resolve.

Resolve accepts well AVI: I use v210: Vdub -> Video -> Compression (uncomp. avi)-> Pixel Format -> V210.
Files are big though, might be overkill, but works.

EDIT:
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
Iirc, these crop values are important to be kept, as removedirt does funky stuff if borders are not clean. Not sure... I believe this was discussed in this thread previously.
Also, I wander if images dimensions must be divisible by 2 only, not by 4, 8 or 16? Are you sure? I think it was also discussed previously but don't remember.

BTW, I personally don't like systematic deshake as it crops the image, sometimes for nothing.
If you don't need deshake, autolevels and autocolor, there is also this solution to remove dirt (and evtl keep grain): https://forum.doom9.org/showthread.php?t=182831

[wonkey_monkey]

When it’s done scanning the software automatically merges all the JPG’s into an AVI file (codec ID MJPG, compression mode Lossy, color space YUV, Chroma subsampling 4:2:0, 8 bits) of about 300mb for 3 mins of footage.

It does not delete the JPG files so there’s also an option to generate an AVI with your own settings. In order to keep as much detail as possible I used VirtualDub2 to generate an AVI file (uncompressed RGB/ YCbCr; RGB24), which is about 10gb.

If it's producing intermediate JPEGs, which are then combined into an MJPEG AVI, you may not be losing anything at that stage. Have you totalled up the size of the JPEGs and compared the total to the size of the AVI?

You may even be losing something by convertting to uncompressed because of chroma upsampling and colourspace conversion.

[Wamberto00]

Thanks for coming back to me. Really appreciate the help of the experts. Sorry for not coming back to you yet, it takes a lot of time to experiment and compare results.

Removing just 1 pixel from each of my 1175 * 863 JPG does not seem to help. This will result in the 'broken mirror effect.
However, when I do a batch resize in Irfan View from 1175 * 863 to 784 * 576 pixels and then generate an AVI file (Uncompressed RGB/YcbCr, 18 fps) through Virtual Dub the scripts are able to process the file.
When I then save the result as a AVI V210 indeed Davinci accepts the file.

However, I'm still in doubt whether I'm using the right dimensions when I'm resizing the JPG's. I would assume that shrinking the dimensions would decrease the quality, but is that a true assumption? What would be the ideal file dimensions to use the script?

[AVIL]

Hi Wamberto00:

1175 * 863 is a strange format for a JPEG image. Better than crop 1 pixel is to add 1 pixel to each dimension (1176 * 864, divisibilty by 8) . Could you post an untouched jpeg file that not contains personal info? So we could make any tests.

[Wamberto00]

Hi Avil,

Good suggestion! Adding 1 pixel does the trick and I now no longer see the 'broken mirror effect' when running Fred's script.

As you requested I've attached 1 untouched file to this post. You can find the JPG here: https://www.dropbox.com/s/duw99ovdvqzv7cc/4000000016.jpg?dl=0

So, now my next challenge is convert the AVI I generate from the JPG files into a file that I can burn to DVD (I'm scanning film for a elderly couple who like to have the film on DVD rather than a memory stick).

If possible I would like to continue to use Davinci Resolve since this is free software and has good color correction options.

Can anyone advise the right settings to preserve the best possible quality?

Currently my workflow is
1) Convert all JPG's to 1176 * 864 (initial file 1175 * 863)
2) Open first JPG in Virtual Dub2
3) Set Frame rate to 18FPS
4) Save video (Magic YUV Lossless v1.2rev2 32bit), Pixel format24
5) Use AVI output as source for Fred's Script (script 01_A_Film_Restoring_Frame_Interpolation.avs, using result 4)
6) Run script and save output as AVI Uncompressed RGB/YcBcR, pixelformat 4:2:2 Y CbCr 10-bit(v210) --> this seems to be the only AVI Davinci Resolve accepts
7) Import file in Davinci Resolve
8) change project settings to 25fps
9) Edit movie (manual or auto color correction)
10) Output to MP4, codec H.264, resolution 1920 x 1080HD, frame rate 25, Quality restricted to 8192 kb/s

[mogsta]

Hi people, I hope someone can quickly see what I've done wrong.

I have processed over 100 reels with the script successfully over the past couple of years, but I've just upgraded my computer and can't get it all working again.

I have installed AviSynth, VirtualDub (old and new versions), put all scripts and plugins in the same directories they were in on the old PC... but I still get an error.

LoadPlugin: unable to find "....dll" etc for all of these:

LoadPlugin("Deflicker.dll")
Loadplugin("Depan.dll")
LoadPlugin("DepanEstimate.dll")
Loadplugin("removegrain.dll")
LoadPlugin("removedirt.dll")
LoadPlugin("MVTools.dll")
LoadPlugin("MVTools2.dll")
Loadplugin("warpsharp.dll")
LoadPlugin("autolevels_06.dll")

I have tried placing these in the same dir as the script, but still no luck.

What could I be missing? What might I have not installed? Do I need to register the DLLs somehow?

I'm attaching a screenshot of the part of the script in question, along with the error message.

Thanks heaps!
Brett

[StainlessS]

You dont say what version Avisynth you're using. [Latest Avs+]:- https://github.com/AviSynth/AviSynthPlus/releases

Also, some versions of Windows seems to break local paths, try adding '.\' before plugin name, eg

LoadPlugin(".\Deflicker.dll")


EDIT: Or put them in the plugins directory and maybe rename any script in plugins from *.AVS to *.AVSI

EDIT:
Also might be that you need CPP runtimes [ie Module Not Found].
Here an All-In-One setup for CPP runtimes, smaller than M$ installers:-
abbodi1406 Visual C++ 2022 Redistributable v0.52.0 - 14.30.30423.0 [1st August 2021] [x86 & x64]:-
https://github.com/abbodi1406/vcredist/releases

[mogsta]

Also might be that you need CPP runtimes [ie Module Not Found].


Nailed it! You're my Hero of the Day!
Installing that Cpp runtimes package solved it immediately.

Thanks so much

:thanks:

[rjansen1]

Hi guys,

I've been working on the restoration of some Super 8 movies of which unfortunately I only have access to a DVD version as source. Still, extracting the MPEG-2, using Srestore to deinterlace and remove duplicate frames and move on from there yields pretty good results.

However, I noticed some weird effects in one of the movies I'm currently working on. Some white areas seem to be moving/shifting/shimmering around in a weird way. I think this may be caused by jumps in brightness in the source material but I'm not sure.

Using John Meyer's script I've tried different settings (like no autolevels, with/without deflicker, no corrections) but I can't seem to get rid of the effect.

I've added a short clip where the effect is visible in the area top-left of the boat.

Clip: https://drive.google.com/file/d/1H77Fw7Y3qXliPC5MCb1WZPLaqwNAS-IU/view?usp=sharing

Do you guys have any idea what might be causing it and how to prevent it from happening?

Cheers,
Remco

[Bernardd]

hello, i think
This is result of movment compensation during removedirt process.

[Selur]

@rajansen1: You can try "MDegrain2i2(last,32,16,400,1)"
function MDegrain2i2(clip source, int "blksize", int "overlap", int "denoising_strength", int "dct")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker

fields=source.SeparateFields()
fixed_fields=fields.deflicker(debug=false,scene=200,percent=60,lag=25,lmax=225,lmin=20,noise=10,border=0)

super = fixed_fields.MSuper(pel=2, sharp=1)
backward_vec2 = super.MAnalyse(isb = true, delta = 2, blksize=blksize, overlap=overlap, dct=dct)
forward_vec2 = super.MAnalyse(isb = false, delta = 2, blksize=blksize, overlap=overlap, dct=dct)
backward_vec4 = super.MAnalyse(isb = true, delta = 4, blksize=blksize, overlap=overlap, dct=dct)
forward_vec4 = super.MAnalyse(isb = false, delta = 4, blksize=blksize, overlap=overlap, dct=dct)

MDegrain2(fixed_fields,super, backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=denoising_strength)

Weave()
}
from: https://forum.videohelp.com/threads/402603-Flickering-Luma-and-Unstable-Frame-at-certain-points#post2627220
it seems to help with the flickering quite well

Cu Selur

[johnmeyer]

How are you "deinterlacing?" I ask because you do NOT want to deinterlace your MPEG-2 sources. Instead, you want to inverse telecine, using TFM/TDecimate. The settings for those two will depend on the actual pulldown pattern that was applied. If you don't do that correctly, then you cannot apply either VideoFred's script or my variation of his script.

As for the movement of the mist in the upper left corner, that is definitely caused by RemoveDirtMC. You can test this theory by setting the dirt removal to zero.

To fix it but still reduce dirt, you could try removing RemoveDirt and then doing the dirt removal using Spotless (http://forum.doom9.org/showthread.php?t=181777), something that is now a viable alternative.

[rjansen1]

How are you "deinterlacing?" I ask because you do NOT want to deinterlace your MPEG-2 sources. Instead, you want to inverse telecine, using TFM/TDecimate. The settings for those two will depend on the actual pulldown pattern that was applied. If you don't do that correctly, then you cannot apply either VideoFred's script or my variation of his script.

As for the movement of the mist in the upper left corner, that is definitely caused by RemoveDirtMC. You can test this theory by setting the dirt removal to zero.

To fix it but still reduce dirt, you could try removing RemoveDirt and then doing the dirt removal using Spotless (http://forum.doom9.org/showthread.php?t=181777), something that is now a viable alternative.

I'm using a script that uses Bob() to do the deinterlacing and then Srestore to remove duplicates. This seems to do the job pretty well. I've looked into inverse telecine but the source I have doesn't seem to have any regular pattern to reverse. That's why I ended up with Srestore.

Thanks for the tips on dirt / noise removal everyone. I'll give that a go!

Cheers,
Remco

[johnmeyer]

Srestore is the wrong thing to use, but without access to a unprocessed clip from one of your original MPEG-2 files, I can't provide any more guidance, and without seeing a clip you have processed using SRestore, I can't tell you if that result looks clean.

But, if it's good enough for you, then it's good enough for me!

[rjansen1]

Srestore is the wrong thing to use, but without access to a unprocessed clip from one of your original MPEG-2 files, I can't provide any more guidance, and without seeing a clip you have processed using SRestore, I can't tell you if that result looks clean!

I'm always open to learning new stuff, so all tips are welcome. This is a clip of the MPEG-2 I extracted from the DVD: https://drive.google.com/file/d/1JXQX4KbFbiCpY1VJA9hF8vcFjMT_yz6n/view?usp=sharing

As for the motion compensation effects, this seems to be related to noise/grain removal more than to dirtremoval. Using Spotless() - which is brilliant - I still got the smearing effect. Using the MDegrain2i2(32,16,400,1) as suggested above gave better results. See this clip: https://drive.google.com/file/d/1h8GKkYjnnvAqJWeumH43AKl4Hd7cmuEG/view?usp=sharing

The only drawback is that this method is VERY slow (< 1 fps) on my system.

Cheers,
Remco

[DanoKrnac]

Hi guys,

first of all, I would like to thank the people of this forum for sharing so much knowledge and helping other people convert their 8mm videos. This woudln´t be possible without your help, or definitely not at this quality!

I would like to add my 2 cents and ask a question at the end.

In short - I have a frame by frame 8mm film capture machine consisting of modified PWM powered, variable speed, LED lit 8 mm film projector and a Canon 600D DSLR with magic lantern firmware using a 70m Sigma lens and 3 macro rings.

The machine creates frame everytime a neodymium magnet stuck to a modified shutter plate (with blades cut of) goes near a hall sensor and that sends a signal through arduino (not nescesary for the machine to work but I can implement more functionality later) and optocoupler directly to external shutter input on the 600D. I shoot at 1280x960 without the gate so there is perforation and some free space next to the film visible in order to make later stabilisation work better.

The 600D is set via magic lantern to record a frame (using electronic / silent shutter) everytime the external shutter is triggered and to put it in a video file (MLV video file). This results in a playable video right out of the camera while all of the frames are frame perfect, RAW (!) and can (must) be opened in MLV app on a computer in order to be played and converted and downsized to H264, mov containter. File is then fed to the fantastic Fred / John Meyer script in avisynth and after the processing is done, I edit the output video in Premiere pro (scale, cropping, some color correction etc.) and then render it. So far I am very pleased with the results. The script runs at about 7 fps on my new-ish 10700KF i7 which I think is very quick considering others posting their speeds.


Now here lies the question - I´ve been using Freds / John Meyers script with great results but sometimes there are very noticeable artifatcs and I would like to pin point the module / modules responsible. This is what the artifact looks like - a pixel "stairs" in bootom right corner:

https://i.ibb.co/wy7p2KR/04-00-10-48-20-Still001.png (https://ibb.co/fM9rTC2)

They are always at the bottom and always when there is movement from the side into the frame. In this case, it´s a shot from a train ride.

I want to make my workflow simple and fast and if for example my on-size-fits-all script has problems with certain videos, I just create a script with certain module off and either ignore it or try to fix it in Premiere. But experimenting with half hour videos in avisynth is very time consuming and I have dozens of films waiting to be transfered so speed is a must.

So if someone could at least point me to where to look for the casue of the artifact, I´d be very thankful :)

Notes:

Why cut the blades of a shutter?

My projectors film advance mechanism has a cam directly on the shutter so I cannot be simply removed. But to give the camera enough time to expose properly, the blades needed to go. Currently I have about 3 frost filters behind the film to make backliting as homogenous as possible which leads to shutter speed of 1/100 to 1/200. Lowe shutter speeds are possible with less frost filters but the backliting may be less homogenous The f stop is set around 7.


Why Canon 600D?

Many reasons - I already had one, the magic lantern implementation is among the best of any Canon cameras, it´s cheap and (!) it has a 3x sensor crop zoom mode which gets you real sensor pixels (if I´m correct) and fixes my problem with being unable to get the camera close enough to the film (flaw by this projector design). It can record safely at about 7-13 fps RAW with the resolution of about 1280x960 pixels. More speed can be achieved with less pixels and vice versa, but I found this to be the sweet spot. I also have a 70D but the magic lantern on that is nowhere near as stable rendering it useless even though it could convert 2x faster than 600D.


If you like the setup and you want more details or help with your, please PM me.

[johnmeyer]

That is an artifact of using Fred's motion estimation method of changing speed. In my version of his script, I initially made significant changes to the MVTools2 motion estimation settings, but I then commented it out because, even with improved settings, you get WAY too many of these ugly break ups.

My recommendation is to almost never use it or, if you do, create a second version that uses standard pulldown to get to the same framerate. Then, in your NLE, cut to that backup for 2-4 frames whenever you get an artifact like this. That's a lot of work, but is the only way around it.

Having said that this is the only way around it, there has been an attempt, in the Framerate Converter (https://forum.doom9.org/showthread.php?t=174793)thread, to mask out these artifacts. I haven't spent much time with that, but the samples posted made me believe that it doesn't work well enough to be useful.

My final recommendation is to just keep the final result at 15, 16, or 18 fps and play it as is. All modern "smart TVs" (that accept USB thumb drives) will happily play video at those rates. Yes, you will get that visual jumpiness when the camera pans too fast horizontally, but that is how the original film looked.

BTW, there are some issues when using MVTools2 on HD resolution. You didn't mention the resolution at which you are capturing. Refer to this old thread where I described the problem:

MVTools2 artifacts, but only at high resolution (https://forum.doom9.org/showthread.php?t=165075)

I just now replaced the pictures deleted by the horrible Photobucket site with the original pictures, so you can still visually see the problems I was describing. You can try temporarily downsizing your video, if it is indeed HD, and see if the problems go away or are diminished. FWIW, even Fred, who created his own capture rig, just like you and I have done, recommends NOT capturing at super-high resolutions. I capture 8mm, Super 8, and even 16mm using an HD camera, but then down-res in the camera to SD. I have compared my captures to the same 16mm film later re-captured in HD by a "Hollywood" transfer facility and found that they did not provide any real increase in detail (although they did do a better job bringing out detail in shadows, but that is not a resolution thing).

Unfortunately, a LOT of old TV shows that were shot on film are now using motion estimation to change from 24 fps to 60 fps interlaced, and I absolutely hate the result, mostly because it completely changes the "feel" of the show (24 fps has a "once removed" quality), but also because, once you are tuned in to these artifacts, you'll easily spot them all over the place, even in the results that have been created by a commercial company using whatever "ultimate" software is available to them.

[DanoKrnac]

Thanks for the info!

I think I'll just make a script without the module and stretch the video in premiere for 25 fps render (I guess it will do a standard pulldown).

I'm not sure rendering nonstandard frame-rates would make for total compatibility. Some customers still want DVDs so I assume they don't have the newest TVs or even a PC so it could cause problems.

[johnmeyer]

Thanks for the info!

I think I'll just make a script without the module and stretch the video in premiere for 25 fps render (I guess it will do a standard pulldown).

I'm not sure rendering nonstandard frame-rates would make for total compatibility. Some customers still want DVDs so I assume they don't have the newest TVs or even a PC so it could cause problems.I don't know Premiere at all (I use Vegas Pro), but you absolutely do need to make sure that it is configured so it will simply duplicate fields (for interlaced video) or frames (for progressive video). In Vegas, the default is to NOT do that, but instead to blend adjacent frames. This avoids the judder you get with pulldown, but unfortunately also makes the result less sharp, because of the blended frames.

AVISynth can easily do pulldown for any frame rate using this code (you can substitute this for the motion estimation code). This is for 29.97 interlaced output. AssumeFPS(30000.0 / 1001.0, true) might be a little more accurate than using 29.97, but I don't think it matters.

ChangeFPS(60000,1001)
AssumeBFF() # or TFF, as required
SeparateFields()
SelectEvery(4, 0, 3)
weave()
AssumeFPS(29.97, true)

[DanoKrnac]

Thanks for the code, I'll try to integrate it and compare it with what premiere can do. Since I'm in Europe, I use 25p output, same as (frame by frame perfect) input.

[johnmeyer]

Ah, Slovakia. Since you want progressive, all you really need is ChangeFPS(25).

[DanoKrnac]

Sorry if I caused confusion.

I would like to slow down the footage to 18 progressive fps playback speed from 25 progressive fps while keeping the output video as 25 fps. Is this possible with the code above?

That should mean either a pulldown or interpolating new frames. As I understand this correctly, my script at the moment is interpolating the frames (ie. creating new ones calculating those from previous and next frame closest frame) which is what sometimes causes the artifacts.

Now that I'm thinking about it, the premiere should be better at this, because it's all automatic and in 15 years of editing I never had any problems with those conversions.

For example, I never new one of my cameras was 30i since the conversion to 25p timeline (both deinterlacing and framerate change) was impossible to notice.

So if I calculate correctly, when I set the video speed on a 25p timeline in Premiere to 72%, it will play the video at 18 fps of the original video and either use a pulldown conversion or interpolate the frames. Either way, I'll try it soon and report back :)

Thanks again for the info!

[johnmeyer]

You can do what you want this way:

AssumeFPS(18)
ChangeFPS(25)

However, as I stated in my earlier post, I recommend forgetting about pulldown unless you are making a DVD, in which case you'll need to do that. Even then, keep the original 18 fps vidoe, with no pulldown or motion estimated frames added. It is as pristine as it can get, and will provide the most authentic viewing experience when you play it directly on your big screen TV set.

P.S. I'm not sure what you meant about your 30i camera. If you captured with that, since you are using it to take a single frame at a time, it actually will not be interlaced, even though that is how the camera is designed. The reason for this is that if you take video of a perfectly static object (in this case, that is one frame of film, held frozen in the gate), then since both fields of the video have zero spatial difference, the video can -- and should -- be treated as though it were progressive. This is actually how I've been capturing movie film on my Workprinter and on my own transfer system, and I've been doing it for twenty years, so I'm quite certain of this.

You also mentioned deinterlacing. You should NOT be deinterlacing as part of your frame accurate film workflow, for the reasons given above.

[DanoKrnac]

Thanks, I am keeping the original material that came out of the camera, only cnoverted to H264 since the MLV format (video of RAW images) can only be viewed with the proprietary "MLV app".

I'm not using deinterlacing at all - workflow is completely progressive. The 30i camera (which I think should be called 60i camera really, but they stated this for some reason) is something from completely different project (SD POV camera from like 10 years ago) and ut was just as an example of premiere doing stuff :)

[DanoKrnac]

BTW I wish this historical framerate nonsense would be gone. It seems like PAL makes more sense in this regard - 25 fps is as near as makes no difference to the film 24fps and for fluid motion, 50i/p is plenty enough I think. The 23. and 29. is there just for historical reasons, which are no longer relevant (CRT TVs are long gone). But I guess, until we have the same electrical grid frequency, it's not going to change anytime soon...

[DanoKrnac]

I´ve just played with the script and actually, the interpolation was commented out so it couldn´t be that. So I tried to remove modules one by one and I found nothing...

If there was no stabilising, same artifacts.
No fps / play speed change. same artifacts.
No denoising, same artifacts.
No sharpening, same artifacts.
No effects at all (decoded source = result), no artifacts :-/

At this point I have no clue what could it be. This is the code I´m using. Its basis is your code but with the stuff I´m not using removed.

/*
Film restoration script by videoFred.
Modified by John Meyer,
Minimised by Dano Krnac

*/
#====================================================================================


#VIDEO FILE
#---------------------------------------------------------------------------------------------------------------------------
#Change the following line to point to your video file
film="E:\8mm\0temp\AVSinput.mov"


#GENERAL PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
result="result4" #specify the wanted output here
play_speed=17.98201798 #play speed (8mm=16; Super8=18; 16mm sound=24)


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation = 1.0 #for all outputs
gamma = 1.0 #for all outputs


#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=8 CTop=8 CRight=8 CBottom=8 #crop values after Depan and before final resizing
W=980 H=736 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed

#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=10 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=10
est_left=40 est_top=40 est_right=40 est_bottom=40 #crop and contast values for special Estimate clip
est_cont=1.1 #Too large a value defeats stabilization


#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------

#Alternate denoising using smalles block size
denoising_strength= 300 #denoising level of first denoiser: MDegrain()
block_size= 4 #block size of MVDegrain
block_size_v= 4
block_over= 2 #block overlapping of MVDegrainMulti()

dirt_strength=30 #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
#120 is pretty aggressive; 85 seems more subtle
#PRE_sharp_ness= 120 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
PRE_sharp_ness= 85 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising

Sharp_Strength = 0.05 #0.05 is often better (avoid too much sharpening)

#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=4


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

/*
normally I go with 128, 256, 384, 512, 768 or 1024. you need to lower the value when using more cores - rule of thumb (threads)x(max memory) = 2GB , maximum memory for 32 bit process which this is */
SetMemoryMax(256)
SetCacheMode(CACHE_OPTIMAL_SIZE)
/*
Here is a list of plugins used
*/

LoadPlugin("plugins2\ffms2.dll") # 2020
LoadPlugin ("plugins2\mvtools2.dll") #Version 2.5.11.9 2/24/2012
LoadPlugin("plugins2\Deflicker.dll") #Version 0.4.0.0 8/16/2004
Loadplugin("plugins2\Depan.dll") #Version 1.10.0.0 4/09/2007
LoadPlugin("plugins2\DepanEstimate.dll") #Version 1.9.2.0 3/25/2007
Loadplugin("plugins2\mt_masktools.dll") #Version 2.0.23.0 3/14/2008
loadplugin("plugins2\RemoveDirtSSE2.dll") #Version 0.9 5/05/2005
Loadplugin("plugins2\RemoveGrainSSE2.dll") #Version 0.9 5/01/2005
Loadplugin("plugins2\warpsharp.dll") # 4/05/2010


#imports h264, converts colorspace, flips horizontaly
source= FFmpegSource2(film).assumefps(play_speed).ConvertToYV12().FlipHorizontal()


#no cropping is done
cropped_source=source


#STABILIZING
#.........................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)


#DENOISING
#...........................................................................................................................................................
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

prefiltered = RemoveGrain(stabcrop,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(super, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


/*
#SHARPENING
#..........................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)
*/


#RESULT4: DENOISED, DEGRAINED, STABILISED, SHRAPENED
#...........................................................................................................................
result4 = denoised.addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#...........................................................................................................................

Eval(result)
#return Detect
# Enable MT!
Prefetch(threads)


# END SCRIPT, BEGIN FUNCTIONS

#=================================================================================================================================
#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int "limit", bool "_grey")
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

# Alternative way to do pre-filtering.
# prefiltered = fft3dfilter(clip,sigma=1,sigma2=2,sigma3=3,sigma4=5,bw=64,bh=64)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)

super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

# Increase thSAD if moving objects are being removed
bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

This is the video I am using as a test (shot of motorway on a trainride):
https://drive.google.com/file/d/1s5QVpieuUzS63waegvE8ieKM_EVQSZCd/view?usp=sharing

Thanks for any advice in advance.

[johnmeyer]

Try taking RemoveDirt down to 15. Your setting is quite high and is probably the cause of the problem.

[DanoKrnac]

Thanks - spot on! (pun intended :D ).
Even though the 15 was still too much for this quick of a movement, 5 is completely artifact free. Although it somewhat stops working as a spot remover :D
Never mind, I´ll fine-tune it for videos with not much movement and a lot of movement.
Thanks again!

[johnmeyer]

Thanks - spot on! (pun intended :D ).
Even though the 15 was still too much for this quick of a movement, 5 is completely artifact free. Although it somewhat stops working as a spot remover :D
Never mind, I´ll fine-tune it for videos with not much movement and a lot of movement.
Thanks again!
You could try substituting Spotless (http://forum.doom9.org/showthread.php?t=181777)for RemoveDirt. StainlessS keeps making improvements. However, it uses the same motion estimation technology as RemoveDirt, so it may be prone to the same artifacts. However, it is more easily tuned and therefore you might be able to avoid some of the artifacts, but still remove dirt.

Do remember that both RemoveDirt and Spotless can remove LOTS of things that you don't want removed. People's feet often get amputated, and any thrown ball will disappear, something I deal with all the time because I do restoration work for a sports collector with ties to NFL Films.

As a result of that football restoration work, I always do two restorations, one with RemoveDirt set to something like 20, and another with it set to 4 or less. I then put both on my NLE (Vegas) timeline, with the aggressive dirt removal on top. Then, when I see an artifact I don't like (missing ball, etc.) I cut to the other version for a few frames. Since the rest of the restoration is identical, you don't notice anything, and if you are lucky enough to not have much dirt for those few frames, the fix is perfect.

It does, however, take some extra work.

[BabaG]

Question about johnmeyer's mod of VideoFred's script.

I've gotten this script to install and run but have a few questions. I'll start with just one, can I mod this script to accept mov files, probably mostly in ProResHQ format? Around line 90 there is this: film="e:\fs.avi"

I've been able to alter that to point to another path that holds my files.

There is also this, somehere around line 210:source1= Avisource(film).killaudio().assumefps(play_speed).converttoYV12()

I'm thinking that needs to be changed as well? To what?

I have a couple of other questions but, if I can load non-avi formats into this script it will give me a chance to investigate further before pestering anyone here again.

thanks,
babag

[poisondeathray]

Question about johnmeyer's mod of VideoFred's script.

I've gotten this script to install and run but have a few questions. I'll start with just one, can I mod this script to accept mov files, probably mostly in ProResHQ format? Around line 90 there is this: film="e:\fs.avi"

I've been able to alter that to point to another path that holds my files.

There is also this, somehere around line 210:source1= Avisource(film).killaudio().assumefps(play_speed).converttoYV12()

I'm thinking that needs to be changed as well? To what?

I have a couple of other questions but, if I can load non-avi formats into this script it will give me a chance to investigate further before pestering anyone here again.

thanks,
babag


You can use LSmash to open ProRes without indexing, using LSmashVideoSource("video.mov") . So you can modify those lines


film="path\prores.mov"

source1= LSmashVideoSource(film).killaudio().assumefps(play_speed).converttoYV12()


I would look at modifying it farther, because ConvertToYV12 makes it 8bit 4:2:0 . ProresHQ will be 10bit422 . Those scripts were probably before other pixel formats and bit depths were added to avisynth classic . I think most functions have been modified to have additional pixel format support by now for avs+

[BabaG]

Thanks, poisondeathray. I see what you mean. Just got an error on 'converttoYV12' in the line you've suggested. Said only 8bit sources were allowed. No idea where to go from here. I'm not much of a coder. I worry that the script uses a lot of things that are older and only 8 bit. I'd have no idea how to figure that out if that's the case.

thanks again,
babag

[kedautinh12]

Are you using latest ver avs+??
https://gitlab.com/uvz/AviSynthPlus-Builds

[BabaG]

I installed from this file:

AviSynthPlus_3.7.0_20210111_vcredist.exe

I think I used the same installer as I have on another system so they'd be in sync with each other.

thanks,
babag

[StainlessS]

First things first, DONT PANIC,

Check that you are actually using AVS+ current,

Return Version


then

film="path\prores.mov"
LSmashVideoSource(film)
Info
#killaudio()
#assumefps(play_speed)
#converttoYV12()
Return last

Does that work, what does it (info) say.
Remove comments one at a time, trying until it has problem.

[BabaG]

Thanks StainlessS. Panic on hold for the moment. ;)

Return Version shows:

AviSynth+ 3.7.0 (r3382, 3.7, x86_64)

The second bit of code works until I uncomment the line:

converttoYV12()

That line throws an error that only 8bit sources are allowed. I think the relevant info from the 'Info' line is:

ColorSpace: YUV422P10, BitsPerComponent: 10

thanks again,
babag

[StainlessS]

You can convert any colorspace to RGB24 and RGB32, but apparently not to YV12 (I had forgotten whether you could or not).
See ConvertBits(8):- http://avisynth.nl/index.php/ConvertBits

EDIT: Maybe others can advise on other args, I dont often go high bit depth, and I've just got a new machine (a couple of hours ago) and am playing with it :)

[Boulder]

Avsresize could be helpful: http://avisynth.nl/index.php/Avsresize (not just resizes but also converts between various formats).

[BabaG]

hey! thanks for that, Boulder! that actually seems to have made some progress for me. while i don't really know how to use this, i replaced converttoYV12()
with
z_ConvertFormat(pixel_type="YV12")
and the error stops and the clip loads. i'll continue to investigate with the real script now that i seem to be able to load 10bit proreshq.

one thing i'm wondering about is the bit depth going forward in the script. am i not taking my 10bit file down to 8bit by using this to convert to YV12? i'm guessing that this will possibly allow the script to run as before, with 8bit source material, but wouldn't it be better to try to keep the 10bit throughout? i'm also thinking that that might be dependent on the filters and their ability to handle 10bit.

thanks again, Boulder, and all, for the help so far,
babag

[Boulder]

hey! thanks for that, Boulder! that actually seems to have made some progress for me. while i don't really know how to use this, i replaced converttoYV12()
with
z_ConvertFormat(pixel_type="YV12")
and the error stops and the clip loads. i'll continue to investigate with the real script now that i seem to be able to load 10bit proreshq.

one thing i'm wondering about is the bit depth going forward in the script. am i not taking my 10bit file down to 8bit by using this to convert to YV12? i'm guessing that this will possibly allow the script to run as before, with 8bit source material, but wouldn't it be better to try to keep the 10bit throughout? i'm also thinking that that might be dependent on the filters and their ability to handle 10bit.

thanks again, Boulder, and all, for the help so far,
babag
I think YV12 means 8 bits. YUV420P10 would be the 10-bit "equivalent". You can use Info() to see all the necessary information.

I'd probably go for YUV420P16 for the processing part myself, but it always depends on what you do next ;)

[BabaG]

Thanks again, Boulder.

Tried playing with your suggestions and was able to get my files to load. when I tried applying things in a real-world sense, though, I've started running into problems, probably related to the bit depth but I'm way too much of a novice and casual user to be able to make that call.

What I found was that, when my script calls
UnsharpMask(strength=50, radius=2, threshold=0)
I get a program crash in both AVSPmod and VirtualDub2. A message said it was related to warpsharp.

Also, when I try to use the Levels filter, such as
Levels(20, 1.0, 240, 0, 255)
I get unexpected results, such as, very gray or black image whereas, before switching to 10 or 16 bit, the image had looked normal. Adjusting the levels also became very erratic. I found some info someplace in the wiki that indicated either the Tweak or ColorYUV filters would be better for higher bit depth material. I tried both, preferring ColorYUV but find it very cumbersome and unintuitive as I have a lot of experience using levels filtering but little with ColorYUV. I've found it difficult to find sweet spots with it where, with levels, I find it quite easy.

Not sure what to do with all of this.

Thanks again,
babag

[poisondeathray]

8 bit goes from 0-255. 10 bit goes from 0-1023. You can get a rough approximation for 8 to 10bit by multiplying by 4. Use coring=false, unless you want to clip to legal range (Y=64 to 940 in 10bit)

Levels(80, 1, 960, 0, 1023, false)

Histogram (Y waveform) works at different bit depths in avs+

Histogram(bits=10)

[BabaG]

Thanks, poisondeathray.

The range tip for levels was very helpful. This came out looking fairly good:
Levels(210, 0.85, 800, 0, 1023, false)

I haven't gotten Boulder's tip to work with YUV420P16, though, the script does run with YUV420P10:
z_ConvertFormat(pixel_type="YUV420P10")

I have a line for unsharpmask that has been crashing VirtualDub and AVSPmod both when it is left uncommented:
#UnsharpMask(strength=50, radius=2, threshold=0)

Commenting the above does allow the rest of the script to run but I don't get the benefit of that line, which does seem to make a difference. Things look better when that line is working. I guess the warpsharp dll it is associated with doesn't like 10bit.

An additional issue has cropped up with audio, though, that may have to do with my source. Source is 2048x1536p 24fps ProResHQ with PCM audio. I haven't gotten the audio to carry through the script. If I try to process the audio down to aac, I get horrible static like white noise and can hear the dialog faintly behind it playing at about half speed. Transcoding the audio in any way seems to do the same thing. If, however, I 'direct stream copy' the audio, it comes through fine. I can also use full processing mode if I choose PCM as the target of the processing but any other target turns to white noise and slowed playback. Odd.

thanks again,
babag

[Boulder]

I have a line for unsharpmask that has been crashing VirtualDub and AVSPmod both when it is left uncommented

You could try Dogway's ex_unsharp, it should work without any crashing issues.

https://github.com/Dogway/Avisynth-Scripts/blob/767e90f7cbf02976d112fe85716fa0d39174f462/MIX%20mods/SharpenersPack.avsi

[videoFred]

I have a line for unsharpmask that has been crashing VirtualDub


Dependencies! Please check if both MSVCP71.dll and MSVCR71.dll are in Windows/SysWOW64.

About your 10 bits source: do you realy need 10 bits? Is your source digitized 8mm film? Can we see an example?

Fred.

[gispos]

Same quality as the original, also for 10bit and higher: UnsharpMask_avsi
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/UnsharpMask_avsi.avsi

[BabaG]

wow. thanks for all of the suggestions!

i looked at the Dogway option and it looks quite thorough. comes with dependencies listed, though, so i continued on and tried realfinder's UnsharpMask_avsi as suggested by gispos. seems to work well and i didn't have to chase down dependencies. no crashing in virtualdub. no error messages. problem solved.

@VideoFred
thanks for your interest. i have a variety of sources and much of it is, indeed, 10bit proreshq or dnx of some flavor. i see everything from regular8 to 35mm and even stuff from the internet archive that's way worse than regular 8.

right now i'm just trying a bunch of things to see what they do and if i can even figure out how to use them. i'm starting with a simple script of my own that uses some of the filters i found in your script as well as johnmeyer's mod of it. i figure if i can get that working with 10bit material then i'll look at adapting your script for that. i figure it would be handy to have one script template that works for both 8 and 10bit.

i ran a few simple tests with your script on some insanely grainy 16mm material. at that time i was still converting to YV12, so 8bit. that test showed me that there are a variety of combinations of restoration routines in the script that i'd like to be able to select and turn on or off. maybe there's a simple way to do that now but, to my non-scripter, non-programmer's eye, it looks like all of the routines, such as dirt removal, auto levels, stabilization, etc are pretty well woven together. one test i did actually introduced movement, probably from the stabilization routine, and that got me thinking that i might be able to select which routines get applied to the file in question. also, i often see files that have been pretty well balanced for color and levels so turning that off might be handy. i took one quick stab at isolating a routine but it ran into issues later in the script so i backed away and started small, with my own, single purpose script as a starting point.

sorry for the excess verbiage and thanks again, all,
babag

[johnnytaco]

Hi Everyone, Ive been lurking for a while as I play around with my workflow for my new wolverine hawkeye - scanning super8 frames as 2560x1920 TIFF then pumping AVI through Avisynth for stabilization, dust, scratches etc.. My days with Avisynth started with DVD and ivtc, then converting interlaced DV, then inverse pulldown from my Canon HV20 to get 24p from 30i. Now Im converting my dad's super 8 dating back to the 60s. So far so good, Ive been playing with both Fred and John version of script but have a question that might save me hours of reading and looking. Is there a recent multi-threaded version of Fred's script. More to the point, can anyone tell me which plugins support MT and which don't? Im using the 32 bit versions it for everything except interpolation and color correction. I pump TIFFs back through Davinci for that. That said, any pointers on how I can speed up the script that seems to be limited to 1 core by adding MT statements for specific plugins? This will save me time from investigating each plugin separately. Im on latest Avisynth package and the scripts are working fine so far, just slow - and Id like to use the 16 thread potential of my i7 CPU. Thanks!

[johnmeyer]

My version supports MT. It was one of the main reasons I developed my branch of Fred's script: speed.

BTW 2560x1920 is way more resolution than you need for Super 8. Try scanning and/or saving at 1/4 that resolution (1280x960) and see if you spot any difference at all. Scanning and using more resolution than is in the media just slows down your workflow, as well as eating up more disk space than you need to. I find that SD resolution is more than enough and therefore I suspect that 1280x960 will be 100% the same as 2560x1920.

[johnnytaco]

I want to test my rig to the full potential (modified wolverine hawkeye with 12MP camera) to see if I can tell the different on my 65" 4k OLED. I have used your 2017 script and like the results. Curious that I dont see the SetFilterMTMode statements, but my i7 10700 did hit around 80-90% with threads=4. I liked the output from Fred's script a little bit better, but not sure what the differences are in the settings - it looks a little softer and less grainy. I use option4 mainly for stabilizing, cropping and cleanup then do color correction, interpolation and rendering to 1080p 24p MP4 in Resolve.

[StainlessS]

Wiki :- http://avisynth.nl/index.php/AviSynth+#Help_filling_MT_modes

Help filling MT modes

The following script contains MT modes for various plugins, save it as mtmodes.avsi and place in your auto-load folder. The script is a work-in-progess, there's still lots of plugins that need to be tested and validated. When the script is finalized, the only thing the user will have to write in his script is the Prefetch call, all SetFilterMtMode calls will be hidden in a single .avsi script.

You can find the latest revision here: AviSynth+ MT modes, if you like to contribute please do so here. Report any issues here.

Note for filter writers: filters can report their mt modes in their SetCacheHint, in this case the manual setting is not necessary.
BEST GET IT FROM ABOVE WIKI LINK as might change, below as it is at time of this post.

AviSynth+ MT modes:- https://publishwith.me/ep/pad/view/ro.rDkwcdWn4k9/latest

I have in my plugins as "SetMTMode.avsi", {many others do too I think} but others as MtModes.avsi.

Header of {at time of posting} file,

###########################################################################
# DO NOT OPEN THIS DOCUMENT IN GOOGLE TRANSLATE OR BAD THINGS WILL HAPPEN #
# Reverted back to last English version 898 and cleaned up text. #
# Please, let's stick to English text to document the script. #
###########################################################################
# Latest version of this script can always be found here: http://avisynth.nl/index.php/AviSynth+#Help_filling_MT_modes
###########################################################################
# General information:
# 1. Do not include core filters in the list.
# 2. Specify the lowest mode that produces correct output with any settings.
# For example, if something works with MT_MULTI_INSTANCE on some parameters and MT_SERIALIZED with others, MT_SERIALIZED should be used.
# You can, of course, specify this in comments so people who do want to get MAXIMUM PERFORMANCE can modify the script themselves.
# 3. Do not specify MT modes for runtime filters. Behavior of runtime filtering with mt is not very well known yet.
# 4. In case it isn't obvious, MT_NICE_FILTER - mt mode 1, MT_MULTI_INSTANCE - mt mode 2, MT_SERIALIZED - mt mode 3. Variables are defined by avs+ core.
# 5. Starting from AviSynth+ r2069, AviSynth+ will now automatically recognize source filters. If it sees a source filter which has no MT-mode specified at all,
# it will automatically use mode 3 instead of the default MT mode. This is great for source filters as it rarely hurts performance and they often need mode 3 anyway,
# and it will help yet unclassified source filters perform correctly without any user action. One step closer to the "it just works" target.
##################################################################################
# Other Lists:
# https://gist.github.com/tp7/8899021
# https://github.com/realfinder/UniversalPluginsFolders/blob/master/plugins64%2B/MtModes.avsi
##################################################################################

[johnnytaco]

Fred, Ive seen different metrics that line up with your view that 720p is sufficient relative to lines per mm for Super 8, but here is an interesting article that makes me feel better about capturing at 2k. https://forums.kinograph.cc/t/8mm-super-8-16mm-scanning-resolution/2297

[johnmeyer]

As StainlessS says, the SetMTMode is usually done inside of a single file that your script references. That is why you don't see it explicitly in my scripts.

As for resolution, that link you provided has some really great discussion, but even though I have no problems with what was posted there, you ALWAYS have to do your own tests. Do those tests with your own film and your own transfer setup.

Even 16mm film can be captured at less than full HD with very little degradation. I had a chance to test this when some 1934 16mm film of Nazi rallies in Dresden Germany that I had transferred was licensed to the Smithsonian Channel. They wanted to have the transfer re-done by a "Hollywood" transfer house, using a professional Spirit transfer system. I had the transfer house send the transfer to me before I sent it to the Smithsonian Channel. As I expected, their transfer, which was delivered in 1920x1080 ProRes, did a little better job on shadows than my projector/camcorder system (the original was very contrasty German film stock), but the sharpness and increase in detail, while evident, was not strikingly better. It was a matter of 10-20% better (I'm trying to put my evaluation into numbers), not twice as good, or four times as good, which is how much more resolution 1920x1080 is compared to my 720x480 (non-square pixel) transfers.

It is from this test that I come to the conclusion that Super 8 film, which has only about 1/3 the picture area of 16mm, will not benefit from scanning/capturing at HD, much less 4K resolution.

Do your own tests, and then make your decision.

[johnnytaco]

Import("plugins/SetMTMode.avsi") makes it much simpler to deal with SetMTMode.

I understand where youre coming from on the resolution bit, and largely agree I have a reel that was scanned by Cinepost at 480p with their wetgate system years back. Im going to compare that to a few of my own versions.

Has anyone successfully used ImageSource to pump TIFFs directly into Avisynth? Im seeing if I can skip the step using Virtualdub2 making an AVI from RGB32 TIFFs.

I tried this in the JM script but I get alot of dropped (black) frames in the AVI I save from virtualdub2 regardless of settings.

source1=ImageSource("H:\veta eng\Image%04d.tiff", 1, 3589, fps=18, use_DevIL = true).converttoYV12().FlipVertical()

[poisondeathray]

Has anyone successfully used ImageSource to pump TIFFs directly into Avisynth? Im seeing if I can skip the step using Virtualdub2 making an AVI from RGB32 TIFFs.

I tried this in the JM script but I get alot of dropped (black) frames in the AVI I save from virtualdub2 regardless of settings.

source1=ImageSource("H:\veta eng\Image%04d.tiff", 1, 3589, fps=18, use_DevIL = true).converttoYV12().FlipVertical()


ImageSource works ok

Do you have gaps in your numbering ?

Black frame are inserted when there are gaps

eg.
Image0001.tiff, Image0002.tiff, Image0004.tiff... there would be a black frame inserted where Image0003.tiff should have been

Do you see black frames when you preview in vdub with the avs script loaded ?

What format are you saving it as (what video compression), and how are you viewing it (what software) ? If you re-import the export into vdub2 , are there any black frames ?

[StainlessS]

@johnnytaco
for future projects suggest use 6 digit index on image files, just more flexible and avoids probs if you have more than 9999 images.
ie,

"H:\veta eng\Image%04d.tiff"


"H:\veta eng\Image%06d.tiff"

[johnnytaco]

ImageSource works ok

Do you have gaps in your numbering ?

Black frame are inserted when there are gaps

eg.
Image0001.tiff, Image0002.tiff, Image0004.tiff... there would be a black frame inserted where Image0003.tiff should have been

Do you see black frames when you preview in vdub with the avs script loaded ?

What format are you saving it as (what video compression), and how are you viewing it (what software) ? If you re-import the export into vdub2 , are there any black frames ?

I dont see any gaps in the sequence. These are RGB32 2560x1920 TIFFs from IC Capture. The black frames show up both in the preview and in the final AVI. I am not compressing at the moment. I use virtualdub2 to normally make an AVI and then run a script (no missing frames this way - works fine). Nothing changes except trying to skip the AVI creation from TIFFs before I pump through Virtualdub/Avisynth script. Im on Win10x64 with an i7 10700 and 64GB RAM, I like using Media Player Classic for checking avisynth outputs, but same black frames show up in VLC when viewing the AVI.

I did some troubleshooting and its not an issue with ImageSource. The standalone function works well. As soon as I change result="stab" (to isolate the first function in JM's script after source) the black frames show up. I suspect DePanEstimate or DePanStabilize might be looking back/ahead at frames and results in the black frames showing up.

[poisondeathray]

I did some troubleshooting and its not an issue with ImageSource. The standalone function works well. As soon as I change result="stab" (to isolate the first function in JM's script after source) the black frames show up. I suspect DePanEstimate or DePanStabilize might be looking back/ahead at frames and results in the black frames showing up.

What version of JM script, can you post a link or post the script ?

Depan/DepanStablilize works ok for me. What version of Depan are you using ? It's supposed to look at other frames, otherwise it couldn't establish the relationship between adjacent frames and stabilize

[videoFred]

BTW 2560x1920 is way more resolution than you need for Super 8. Try scanning and/or saving at 1/4 that resolution (1280x960) and see if you spot any difference at all. Scanning and using more resolution than is in the media just slows down your workflow, as well as eating up more disk space than you need to. I find that SD resolution is more than enough and therefore I suspect that 1280x960 will be 100% the same as 2560x1920.

I agree! It's not the size that counts, but the quality from the scanning device.

Fred.

[videoFred]

Depan/DepanStablilize works ok for me.

You might want to try this: much better stabilisation and scene detection can be set with "error"
SetFilterMTMode("DEFAULT_MT_MODE", 2)

clip= AVISource("Z:\VDP\VdP_Sp2.avi").converttoYV12()

vectors= clip.MSuper().MAnalyse(isb=false)
mdata= MDepan(clip,vectors,rot=false, zoom=false, error= 65)

stab= clip.DePanStabilize(data=mdata,dxmax=40,dymax=40,zoommax=0, rotmax=0,method=0, mirror=0, info=true)

stab

Prefetch(4)

SetFilterMTMode and Prefetch is for use with Avisynth+ of cource.

Forgot to mention:
Requires depan.dll vs. 2.13.1.3 or higher modified versions by Printerf
mvtools2.dll vs. 2.7.23.0 or higher
https://github.com/pinterf/mvtools/releases

Fred.

[BabaG]

getting the following error on trying to load a proreshq clip:

Script error: can't have a named (quoted) parameter followed by an ordinary parameter.

the line it highlights is this function line:
function RemoveDirt(clip input, int "limit", bool _grey)

i'm guessing we're referring to the "limit" parameter. how do i get around this? is it as simple as removing the quotes? will try that while waiting for response.

thanks,
babag

[VoodooFX]

Try:

function RemoveDirt(clip input, int "limit", bool "_grey")

[BabaG]

thanks, voodoofx. that got me on to the next error.

now getting:

DePanEstimate: Can not load fftw3.dll or libfftw3f-3.dll!

i do have fft3dfilter.dll in my plugins64+ folder. maybe that doesn't cover it?

thanks again,
BabaG

[VoodooFX]

Can not load fftw3.dll or libfftw3f-3.dll!

FFTW 3.3.5 (http://www.fftw.org/install/windows.html) (fftw-3.3.5-dll32.zip or fftw-3.3.5-dll64.zip)

*** 32-bit libfftw3f-3.dll needs to be in the search path (C:\Windows\SysWOW64 64-bit OS or C:\windows\system32 32-bit OS)
*** 64-bit libfftw3f-3.dll needs to be in the search path (C:\windows\system32 64-bit OS)

[BabaG]

hmm. i've got the lib files in the proper places but getting crashes in avspmod on trying to preview the image. tried copy/renaming the lib files to fftw3.dll as per posts here and elsewhere and placed them in every conceivable combination of location. no luck. also tried simply copying the fft3.dll that VideoFred includes with his download to, again, every conceivable location. same thing. crash on preview.

out of ideas.
babag

[Arx1meD]

BabaG, try LoadDLL (http://avisynth.nl/index.php/LoadDLL). Perhaps you first need to delete fftw3.dll from C:\Windows\...

[StainlessS]

posted by BabaG:
tried copy/renaming the lib files to fftw3.dll as per posts here and elsewhere and placed them in every conceivable combination of location.

You need the zip for the bittage of avisynth that you are using [32 bit Avisynth, then 32 bit zip, else 64 bit zip].

Make sure that you use the file with EXACT source name libfftw3f-3.dll
and make a copy named as fftw3.dll, and copy both of these files to ONLY ONE location as below,

32bit_OS_32bit_Avisynth: C:\Windows\system32
64bit_OS_64bit_Avisynth: C:\Windows\system32
64bit_OS_32bit_Avisynth: C:\Windows\SysWOW64

as outlined by VoodooFX, and delete any others that you mistakenly copied to the wrong directory [ie, "every conceivable combination of location"].

Some plugins use dll with filename libfftw3f-3.dll, and some with filename fftw3.dll, and some plugins may use either name.

[BabaG]

thanks all.

avisynth version shows:AviSynth+ 3.7.3 (r4003, 3.7, x86_64)

placed libfftw3f-3.dll and fftw3.dll in:

C:\Windows\system32

neither in SysWOW64. system is 64 bit.

same crash. other scripts load and run fine.

fwiw, my process has been to try the files in various locations and, after trying in that location, to remove them before trying in the next location. i do this to avoid having duplicates or wrong versions cluttered around the system.

thanks again,
babag

[poisondeathray]

thanks all.

avisynth version shows:AviSynth+ 3.7.3 (r4003, 3.7, x86_64)

placed libfftw3f-3.dll and fftw3.dll in:

C:\Windows\system32

neither in SysWOW64. system is 64 bit.

same crash. other scripts load and run fine.


Try vdub2 to preview , any crash ?

How do you know the crash now is related to libfftw3f-3.dll ? That was for the previous error with DepanEstimate ? Do you have a different error message with the new crash ?

What other types of scripts run ok ? Simple ones ?

I'm thinking a bad (other) plugin .


fwiw, my process has been to try the files in various locations and, after trying in that location, to remove them before trying in the next location. i do this to avoid having duplicates or wrong versions cluttered around the system.


You can try avsinfo tool for information and to identify versions of plugins

[BabaG]

thanks, pdr.

i was actually just doing that. vdub2 also crashed but did give this as its best guess as to the cause:An exception occurred in module 'warpsharp'.

thanks again,
babag

[BabaG]

ran avsinfo tool and the 32 bit reported no problems but the 64 bit reported this:[Plugin errors/warnings]
________________________________________________________________________________

Function duplicates:

"m4" : "C:\Program Files (x86)\AviSynth+\plugins64+\GrainFactory3.avsi"
"m4" : "C:\Program Files (x86)\AviSynth+\plugins64+\HistogramRGBParade.avsi"
"undefined" : "[InternalFunction]"
"Undefined" : "C:\Program Files (x86)\AviSynth+\plugins64+\Zs_RF_Shared.avsi"

don't really know what to do about that.

thanks,
babag

[poisondeathray]

Function duplicates shouldn't cause a crash


Where is the modified JM script ?


An exception occurred in module 'warpsharp'.

Check warpsharp version and is it actually being used? Move it into a different directory and/or comment out the LoadPlugin call


Check the dates of the other .dll's in avsinfo . Older ones that haven't been updated are more likely the culprit

If you still have problem - clean out the plugins directory , and one by one add them back according to the error message, checking the script each time . Since other scripts work ok without crashing, it's likely a bad/old plugin

[johnmeyer]

Where is the modified JM script ?I've skimmed these new posts and had the same exact question. He's obviously made a lot of changes. As one example, the modification VoodooFX made by putting the quotes in the correct place corrected an error that was NOT in any version of my script I ever posted. That syntax error, as BabaG has found, causes the script to fail, and my scripts did not have any obvious syntax errors like that.

Therefore, without seeing the script he is actually using, we cannot assume that it is actually the same as anything that I wrote.

[BabaG]

here's the copy of the script, minus the head notes for space.

part 1 of 2.#GENERAL PARAMETERS #moved before file load by BabaG to define play_speed before it is first called (line 90)
#----------------------------------------------------------------------------------------------------------------------------
result="result4" #specify the wanted output here; valid results are "result1" through "result4" and "resultS1" through "resultS4" for before/after
play_speed=18 #play speed (8mm=16; Super8=18; 16mm sound=24)


#VIDEO FILE
#----------------------------------------------------------------------------------------------------------------------------
#Change the following line to point to your video file
#film="e:\fs.avi" #original johnmeyer line
#film="path\to\prores_file.mov"
film="Appointment_Room_512_2048x1536p_GV-HQX_18fps____CLEANUP_MASTERING_SCRIPT_Lvl_4400x65535_TESTCLIP.mov"
#source1= LSmashVideoSource(film).killaudio().assumefps(play_speed).converttoYV12() #supports proreshq import; suggested by poisondeathray
source1= LSmashVideoSource(film).killaudio().assumefps(play_speed).z_ConvertFormat(pixel_type="YUV420P10") #supports proreshq import; suggested by poisondeathray
#z_ConvertFormat(pixel_type="YUV420P10") suggested by Boulder
#z_ConvertFormat(pixel_type="YUV420P16") suggested by Boulder also


#COLOR AND LEVELS PARAMATERS
#----------------------------------------------------------------------------------------------------------------------------
saturation=1.0 #for all outputs
gamma=1.0 #for all outputs
blue= -0 red=-0 #manual color adjustment, when returning result3 & result4. Values can be positive or negative
black_level=0 white_level=255 output_black=0 output_white=255 #manual levels, when returning result2 & result4


#SIZE, CROP AND BORDERS PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
CLeft=16 CTop=16 CRight=16 CBottom=16 #crop values after Depan and before final resizing
W=720 H=480 #final size after cropping
bord_left=0 bord_top=0 bord_right=0 bord_bot=0 #720p= borders 150
in_bord_left=0 in_bord_top=0 in_bord_right=0 in_bord_bot=0 #Borders around input that must be removed


#STABILISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
maxstabH=05 #maximum values for the stabiliser (in pixels) 20 is a good start value
maxstabV=20
est_left=40 est_top=40 est_right=40 est_bottom=40 #crop and contast values for special Estimate clip
est_cont=1.6

#DENOISING PARAMETERS
#----------------------------------------------------------------------------------------------------------------------------
#Fastest parameters
#denoising_strength=600 #denoising level of first denoiser: MVDegrain()
#denoising_strength=300 #denoising level of first denoiser: MVDegrain()
#block_size= 16 #block size of MVDegrain
#block_size_v= 16
#block_over= 8 #block overlapping of MVDegrainMulti()

#Best compromise between speed and quality
denoising_strength= 600 #denoising level of first denoiser: MDegrain()
block_size= 8 #block size of MVDegrain
block_size_v= 8
block_over= 4 #block overlapping of MVDegrainMulti()

dirt_strength=23 #sets amount of dirt removal (big spots)


#FOUR STEP SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------------------------------------------
PRE_sharp_ness= 120 PRE_radi_us= 3 #presharpening (UnsharpMask) just after first denoising
Sharp_Strength= 0.1


#AUTO LEVELS PARAMETER
#--------------------------------------------------------------------------------------------------------------------------------
X=4 #X is a special parameter for reducing the autolevels effect on the whites
X2=4 #X2 is a special parameter for reducing the autolevels effect on the blacks


#NUMBER OF THREADS
#--------------------------------------------------------------------------------------------------------------------------------
threads=5


# END VARIABLES, BEGIN SCRIPT
#=================================================================================================================================

# Using AVISynth version 2.60, build August 28, 2012 (Ben-Rudiak-Gould, etc.)
# Using VirtualDub 1.8.6

#Change the following (lower or higher) if you have stability problems with multi-threading
SetMemoryMax(800)

#Load plugins explicitly all commented by BabaG to avoid error that they are not found; make sure they are all in the proper avisynth plugin folders so as to auto load
#LoadPlugin ("mvtools2.dll") #Version 2.5.11.9 2/24/2012
#LoadPlugin("autolevels.dll") #Version 0.6.0.0 1/09/2011
#LoadPlugin("Deflicker.dll") #Version 0.4.0.0 8/16/2004
#Loadplugin("Depan.dll") #Version 1.10.0.0 4/09/2007
#LoadPlugin("DepanEstimate.dll") #Version 1.9.2.0 3/25/2007
#Loadplugin("mt_masktools.dll") #Version 2.0.23.0 3/14/2008
#loadplugin("RemoveDirtSSE2.dll") #Version 0.9 5/05/2005
#Loadplugin("RemoveGrainSSE2.dll") #Version 0.9 5/01/2005
#Loadplugin("warpsharp.dll") # 4/05/2010

#Use the following for alternative frame interpolation
#loadplugin("svpflow1.dll") #Version 1.2.1.0 5/29/2012
#loadplugin("svpflow2.dll") #Version 1.3.1.0 6/02/2012
#Import("InterFrame2.avsi") #Version 2.1.0 6/27/2012

#Open and crop the video
#Remove all setmtmode statements (there are four in this script) if not using multi-threaded (MT) AVISynth
#setmtmode(5,threads) #commented by BabaG
#source1= Avisource(film).killaudio().assumefps(play_speed).converttoYV12() #killaudio() improves stability (in my experience -- others disagree)
source1= LSmashVideoSource(film).killaudio().assumefps(play_speed).z_ConvertFormat(pixel_type="YUV420P10") #supports proreshq import; suggested by poisondeathray
#z_ConvertFormat(pixel_type="YUV420P10") suggested by Boulder
#z_ConvertFormat(pixel_type="YUV420P16") suggested by Boulder also

cropped_source=source1.crop(in_bord_left,in_bord_top,-in_bord_right,-in_bord_bot) #temporarily remove any black borders on input video
#setmtmode(2) #commented by BabaG

#cropped_source=filldrops(cropped_source) #Use this when removing bad frames that have been removed by duplicating previous frame
#(see notes in function at end of script)

#STABILIZING
#....................................................................................................................................................................
stab_reference= cropped_source.crop(est_left,est_top,-est_right,-est_bottom).tweak(cont=est_cont).MT_binarize(threshold=80).greyscale().invert()
mdata=DePanEstimate(stab_reference,trust=1.0,dxmax=maxstabH,dymax=maxstabV)
stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15)

#Alternative line that includes deflicker. I find that this usually is not needed.
#stab=DePanStabilize(cropped_source,data=mdata,cutoff=0.5,dxmax=maxstabH,dymax=maxstabV,method=1,mirror=15).deflicker()


#DENOISING
#...................................................................................................................................................................

#Remove dirt
input_to_removedirt=stab.crop(CLeft,CTop,-CRight,-CBottom)
stabcrop=RemoveDirtMC(input_to_removedirt,dirt_strength,false)

#Reduce grain
prefiltered = RemoveGrain(stabcrop,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(stabcrop, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(superfilt, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(superfilt, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(superfilt, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


#SHARPENING
#...................................................................................................................................................................
sharp1=denoised.sharpen(Sharp_Strength)
PreBorderFrame = sharp1.Lanczos4Resize(W - bord_left - in_bord_left - bord_right - in_bord_right, H - bord_top - in_bord_top - bord_bot - in_bord_bot)

[BabaG]

paret 2 of 2.#FRAME INTERPOLATION (optional)
#...................................................................................................................................................................

#Interpolate frames
#Use this for progressive output -- example shows 29.97 progressive
#PreBorderFrame=InterFrame(NewNum=30000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads)

#Use this for interlaced output -- example shows 29.97 interlaced
#PreBorderFrame=InterFrame(NewNum=60000,NewDen=1001,PreBorderFrame,GPU=true,Cores=threads).SeparateFields().SelectEvery(4, 0, 3).Weave()


#RESULT1: AUTOLEVELS,AUTOWHITE
#......................................................................................................................................................................
#SetMTMode(5) #Turn off SetMTMode for Autolevels #commented by BabaG
result1= PreBorderFrame.coloryuv(autowhite=true).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT3: AUTOLEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result3= PreBorderFrame.coloryuv(off_U=blue,off_V=red).addborders(X,0,0,0,$FFFFFF).addborders(0,0,X2,0,$000000).autolevels(filterRadius=2).crop(X,0,-X2,-0).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)
#SetMTMode(2) #Re-enable SetMTMode after Autolevels #commented by BabaG

#RESULT2: MANUAL LEVELS, AUTOWHITE
#......................................................................................................................................................................
result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

#RESULT4: MANUAL LEVELS, MANUAL COLOR CORRECTIONS
#.....................................................................................................................................................................
result4= PreBorderFrame.coloryuv(off_U=blue,off_V=red).levels(black_level,gamma,white_level,0,255).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)


#PARAMETERS FOR THE COMPARISONS
#.....................................................................................................................................................................
W2= W+bord_left+bord_right
H2= H+bord_top+bord_bot
source4=Lanczos4Resize(source1,W2,H2)


#COMPARISONS: ORIGINAL VS RESULTS
#......................................................................................................................................................................
resultS1= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result1,"autolevels, autowhite",size=28,align=2))
resultS2= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result2,"autowhite, manual levels correction",size=28,align=2))
resultS3= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result3,"autolevels, manual color correction",size=28,align=2))
resultS4= stackhorizontal(subtitle(source4,"original",size=28,align=2),subtitle(result4,"manual colors and levels correction",size=28,align=2))

#Finish and go home
Eval(result)


# END SCRIPT, BEGIN FUNCTIONS
#=================================================================================================================================


#REMOVE DIRT FUNCTION
#......................................................................................................................................................................
function RemoveDirt(clip input, int "limit", bool "_grey") #quotes added around _grey per forum suggestion for error
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.RemoveGrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,debug=false,noise=limit,noisy=4, grey=_grey)

# Alternative settings
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=true)
# return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance= 12,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey,show=false)
}

#This function provides motion compensation for the remove dirt function, thus improving quality.
function RemoveDirtMC(clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=16, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=16, overlap=2,delta=1, truemotion=true)

bvec_re = Mrecalculate(super,bvec,blksize=8, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=8, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(forw,clip,backw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

# The following function will remove near duplicates ("0.1") or exact duplicates (change to "0.0").
# It replaces these duplicates with a motion estimated frame. It works really well (IMHO).
#
# Use: In your video editor, replace any single bad frame (burned frame, jump, missing frame from splice, etc.) with a duplicate of the
# previous frame. Then, include a call to this function (see script above, where it is commented out).
#
# Here's an example of how this function can be use:
#
# http://www.youtube.com/watch?v=uzMFodrGHDs
#

function filldrops (clip c)
{
super=MSuper(c,pel=2)
vfe=manalyse(super,truemotion=true,isb=false,delta=1)
vbe=manalyse(super,truemotion=true,isb=true,delta=1)
filldrops = mflowinter(c,super,vbe,vfe,time=50)
fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
return fixed
}


i moved the 'play_speed' variable definition ahead of everything else as i was getting an error that avisynth didn't know what it meant. i assume that meant it hadn't been defined before it was called. moving it allowed the script to continue.

i commented out the 'loadplugins' commands and put the plugins in the main avisynth plugin folders.

setmtmode was causing errors so i commented those for the time being.

thanks,
babag

[poisondeathray]

i commented out the 'loadplugins' commands and put the plugins in the main avisynth plugin folders.


I meant don't even move them to autoload directory. Move them somewhere else. They are all older versions and some of them are known to cause problems with modern avs+ versions .

Or stick with old avs classic with old plugins.

If you're going to use avs+ x64:

Don't use the plugin warpsharp.dll. It's probably the old avs 2.5 plugin.
#Loadplugin("warpsharp.dll") # 4/05/2010
There is a newer awarpsharp2 , that has a compatibility function for old scripts and it works for avs+ x64
But it looks like warpsharp isn't even being called. Sharpen is using the internal sharpen function in that script

Another common problem is mt_masktools.dll - this version is known to cause various problems with modern avs+
#Loadplugin("mt_masktools.dll") #Version 2.0.23.0 3/14/2008
#LoadPlugin ("mvtools2.dll") #Version 2.5.11.9 2/24/2012

Use the pinterf masktools and mvtools2
https://github.com/pinterf/mvtools/releases
https://github.com/pinterf/masktools/releases

In avsinfo tool , look at the dates of the plugins - that should be a big clue. Move any old versions to some other folder and update them to the updated versions


setmtmode was causing errors so i commented those for the time being.


That is for avs classic . For avs+ you would use SetFilterMTMode and Prefetch
http://avisynth.nl/index.php/SetFilterMTMode

[BabaG]

here's the avsinfo file for 64:[OS/Hardware info]
Operating system: Windows 10 (x64) (Build 18362)

CPU: Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz / Coffee Lake (Core i7)
MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX
6 physical cores / 12 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.3 (r4003, 3.7, x86_64)
VersionNumber: 2.60
File / Product version: 3.7.3.0 / 3.7.3.0
Interface Version: 10
Multi-threading support: Yes
Avisynth.dll location: C:\windows\system32\avisynth.DLL
Avisynth.dll time stamp: 2023-07-16, 07:48:08 (UTC)
PluginDir2_5 (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64
PluginDir+ (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64+


[C++ 2.5 Plugins (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\Decomb.dll [2013-12-02]
C:\Program Files (x86)\AviSynth+\plugins64+\warpsharp.dll [2011-06-15]

[C++ 2.6 Plugins (64 Bit)]
C:\Program Files (x86)\AviSynth+\plugins64+\AutoLevels_x64.dll [0.12.3.0]
C:\Program Files (x86)\AviSynth+\plugins64+\avsresize.dll [0.0.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\aWarpSharp.dll [2016-06-24]
C:\Program Files (x86)\AviSynth+\plugins64+\ConvertStacked.dll [2023-07-16]
C:\Program Files (x86)\AviSynth+\plugins64+\Deflicker.dll [0.6.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\DePan.dll [2.13.1.6]
C:\Program Files (x86)\AviSynth+\plugins64+\DePanEstimate.dll [2.10.0.4]
C:\Program Files (x86)\AviSynth+\plugins64+\DirectShowSource.dll [2023-07-16]
C:\Program Files (x86)\AviSynth+\plugins64+\ffms2.dll [2020-08-22]
C:\Program Files (x86)\AviSynth+\plugins64+\fft3dfilter.dll [2.10.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\FluxSmooth.dll [1.4.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\hqdn3d.dll [1.1.1.0]
C:\Program Files (x86)\AviSynth+\plugins64+\ImageSeq.dll [2023-07-16]
C:\Program Files (x86)\AviSynth+\plugins64+\KNLMeansCL.dll [2020-11-12]
C:\Program Files (x86)\AviSynth+\plugins64+\LSMASHSource.dll [2022-05-05]
C:\Program Files (x86)\AviSynth+\plugins64+\masktools2.dll [2.2.30.0]
C:\Program Files (x86)\AviSynth+\plugins64+\mvtools2.dll [2.7.45.0]
C:\Program Files (x86)\AviSynth+\plugins64+\neo-fft3d.dll [1.0.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\nnedi3.dll [0.9.4.60]
C:\Program Files (x86)\AviSynth+\plugins64+\RemoveDirt.dll [0.9.3.0]
C:\Program Files (x86)\AviSynth+\plugins64+\RgTools.dll [1.2.0.0]
C:\Program Files (x86)\AviSynth+\plugins64+\Shibatch.dll [2023-07-16]
C:\Program Files (x86)\AviSynth+\plugins64+\TimeStretch.dll [2023-07-16]
C:\Program Files (x86)\AviSynth+\plugins64+\TIVTC.dll [1.0.26.0]
C:\Program Files (x86)\AviSynth+\plugins64+\VDubFilter.dll [2023-07-16]
C:\Program Files (x86)\AviSynth+\plugins64\AddGrainC.dll [1.8.4.0]

[Scripts (AVSI)]
C:\Program Files (x86)\AviSynth+\plugins64+\colors_rgb.avsi [2022-10-07]
C:\Program Files (x86)\AviSynth+\plugins64+\GrainFactory3.avsi [2022-10-23]
C:\Program Files (x86)\AviSynth+\plugins64+\HistogramRGBParade.avsi [2024-01-18]
C:\Program Files (x86)\AviSynth+\plugins64+\QTGMC.avsi [2022-10-24]
C:\Program Files (x86)\AviSynth+\plugins64+\SMDegrain.avsi [2023-12-16]
C:\Program Files (x86)\AviSynth+\plugins64+\TemporalDegrain.avsi [2023-03-01]
C:\Program Files (x86)\AviSynth+\plugins64+\TransformsPack - Main.avsi [2023-12-16]
C:\Program Files (x86)\AviSynth+\plugins64+\TransformsPack - Models.avsi [2023-12-16]
C:\Program Files (x86)\AviSynth+\plugins64+\TransformsPack - Transfers.avsi [2023-12-16]
C:\Program Files (x86)\AviSynth+\plugins64+\Zs_RF_Shared.avsi [2022-10-24]

[Uncategorized files]
C:\Program Files (x86)\AviSynth+\plugins64+\colors_rgb.txt [2022-10-07]

omitted the duplicate functions section as i already posted that. i think things seem to be pretty current. don't really know what i'm doing as far as checking though.

thanks,
babag

[StainlessS]

Avisynth Wiki:- http://avisynth.nl/index.php/Main_Page
External AviSynth+ x64 plugins:- http://avisynth.nl/index.php/AviSynth%2B_x64_plugins
External AviSynth+ x86 plugins:- http://avisynth.nl/index.php/External_filters
Internal filters:- http://avisynth.nl/index.php/Internal_filters

NOTE, Warpsharp is deprecated, use Warpsharp2.

EDIT:
[Uncategorized files]
C:\Program Files (x86)\AviSynth+\plugins64+\colors_rgb.txt [2022-10-07]
Delete that.

[BabaG]

ok. so, i'm in the middle of, one-by-one, bringing filters and functions into my plugins64 folder and, sure enough, have hit a snag with this line (230):denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)
the error is:Script error: There is no function named 'unsharpmask'.
not sure how to get past this as pdr has recommended to use another filter for this. do i have to change something in the line (230) to match the syntax of the substitute filter?

as i look into this, it's looking to me as if the warpsharp2 suggestion is what would address this? how woul the 230 line be changed?

edit:
thanks, stainlesss. i deleted the file you pointed out.

thanks,
babag

[poisondeathray]

You can replace unsharpmask with updated functions that support high bit depth, so you don't have to downgrade your prores. The original unsharpmask in warpsharp is a 2.5 plugin and does not support other avs+ pixel types like YUV422P10 . You can change the function name to UnsharpMask or change the script UnsharpMask calls to UnsharpMask_avsi
https://github.com/realfinder/AVS-Stuff/blob/master/avs%202.5%20and%20up/UnsharpMask_avsi.avsi

Or dogway's sharpener's pack has a variation
https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/SharpenersPack.avsi

[BabaG]

hey! that's awesome! it finally opens. very dark, though. will look into that later. thanks so much for all the help! (unfortunately, i'll probably be back soon.)

thanks again,
babag

[poisondeathray]

hey! that's awesome! it finally opens. very dark, though.

Levels filter. 10bit goes 0 to 1023. Edit all the lines with levels . There might be a few more


black_level=0 white_level=255 output_black=0 output_white=255 #manual levels, when returning result2 & result4


denoised=stabcrop.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength).levels(0,gamma,255,0,255).tweak(sat=saturation).unsharpmask(PRE_sharp_ness,PRE_radi_us,0)


result2= PreBorderFrame.levels(black_level,gamma,white_level,0,255).coloryuv(autowhite=true).addborders(bord_left+in_bord_left, bord_top+in_bord_top, bord_right+in_bord_right, bord_bot+in_bord_bot)

[BabaG]

that's got it! thanks!

[johnmeyer]

hey! that's awesome! it finally opens. very dark, though. will look into that later. If it is darker than the original, try one of the two settings that doesn't use autolevels. I usually see no change in gamma, but I almost always use the number 4 setting which doesn't touch either levels or color.

[BabaG]

thanks, johnmeyer! great tip. much appreciated. changing 255 to 1023 seems to have worked here.

i think the last thing i need to try in order to really test this on my materials is to get the change from setmtmode to setfiltermtmode working. for example, with this line:setmtmode(5, threads)
what should the replacement setfiltermtmode line be? if i just say:SetFilterMTMode(5, threads) it throws an error. wrong syntax, i suppose. not sure what to put into it.

edit:
this seems to allow the clip to load:SetFilterMTMode("source1", 2)
i only replaced the first instance of setmtmode with this and the clip does load. don't understand if i should replace the remaining instances of setmtmode with it. they are currently commented.

thanks again, all!
babag

[johnmeyer]

Here is a link to my current script:

John Meyer Film Restoration Script 2024 (https://drive.google.com/file/d/1eLUhz-LL3v_tSLlaH1spnwCNUVWMJn1_/view?usp=sharing)

It is designed for the more modern versions of AVISynth which no longer use the SetMTMode version of multithreading.

This version used GamMac for color, and provides the option of using SpotLess instead of RemoveDirt for getting rid of film dirt.

(Note: I added SpotLess only a few years ago but forgot to update my revision log at the beginning of the script.)

[BabaG]

wow! thanks johnmeyer. just downloaded it. would like to get the version i've been currently working with fully functional before i jump into something new. guess i should study it while waiting for info on these last few questions about the version i've had up to now.

thanks again!
babag

[johnmeyer]

The main reason for making the script available was to answer your question about getting multithreading to work in more modern versions of AVISynth. These versions use the more modern version of multithreading so you should be able adapt whatever you almost have working. The "Prefetch(threads)" line is the main thing. It needs to be the very last statement in the main body of the script (the function calls, of course, can be put anywhere, so "last statement" doesn't apply to them).

I assume you have read this:

AVISynth+ Multithreading (http://avisynth.nl/index.php/AviSynth+#Multithreading)

[BabaG]

thanks again, johnmeyer.

sadly, it's all getting to be a bit more than i can handle. the differences between setmtmode and setfiltermtmode and avisynth 32 and 64. i'm terrible at anything like coding but keeping this all in line is just too much for me. i just can't figure it out. i don't know how you do it.

thanks,
babag

[johnmeyer]

You seem to have gotten pretty far, so I think you can do it. A few suggestions.

1. If you still have a computer using Windows 7, use that.

2. Don't use 64-bit. I still have yet to see any compelling reason why that would be better. You will find that the 32-bit version will have fewer compatibility problems.

3. Turn off all multi-threading options until you have it working. This means commenting out the "Prefetch" line (for the current versions of AVISynth).

[manolito]

Please forgive me if it looks like I want to highjack this discussion. I am asking about this film restauration script because a few days ago a good friend asked me if I had any ideas if this video from 1979 could be improved.

It is an amateur live recording from a band called "Jump Street" shot at the Catalyst in Santa Cruz, the quality is awful, but it does have a sentimental value for my friend (and for me).

Here is the link:
https://www.youtube.com/watch?v=NBxBFOI3rSQ&pp=ygUUanVtcCBzdHJlZXQgY2F0YWx5c3Q%3D

I am not too optimistic if this is even worth the effort, but if you think that this clip could be improved using this script, please let me know...


Cheers
manolito

[BabaG]

1. If you still have a computer using Windows 7, use that.

unfortunately, no windows 7 here.

2. Don't use 64-bit. I still have yet to see any compelling reason why that would be better. You will find that the 32-bit version will have fewer compatibility problems.

this is one of the areas with which i have difficulty: distinguishing between things intended for either version. can they be run alongside each other? how would one make clear which is being called? pretty much everything in my system is 64 bit so i went with 64 bit avisynth+.

3. Turn off all multi-threading options until you have it working. This means commenting out the "Prefetch" line (for the current versions of AVISynth).

i thought i'd pretty much gotten to that point. that's why i was trying to get the multithreading up now. setting the max memory made a huge difference, though. one thing i noted between your two scripts was that, i think it was for autolevels, the earlier version that used setmtmode had a comment about turning autolevels off and then back on. the newer version didn't seem to have any use of the replacement, setfiltermtmode, at all so i couldn't figure out how to update setmtmode in the earlier script.

@manolito, i think this script, and variants, are intended for captured film, in the case of the videofred version, film captured at good quality and, in the johnmeyer version, maybe more tolerant of source quality. obviously, i'm not someone to pay too much attention to but, in this case, i think i have it right.

thanks again,
babag

[StainlessS]

this is one of the areas with which i have difficulty: distinguishing between things intended for either version. can they be run alongside each other? how would one make clear which is being called? pretty much everything in my system is 64 bit so i went with 64 bit avisynth+.
If you load avs file with 64 bit MPC-HC/VirtualDub2/AvsPMod/etc, then it uses Avisynth 64 bit, which in turn uses (AND REQUIRES) 64 plugins in 64 bit plugins directory,
similarly, 32 bit players require 32 bit everything.
EDIT: Also, 64 bit MegUI (or other encoder), uses 64 bit Avisynth etc, and 32 bit ecoder uses 32 bit avisynth etc.
Above bittage requirements apply to all Windows executables and dll's, not just avisynth, n-bit executables require n-bit dll's.

Wiki:- http://avisynth.nl/index.php/Main_Page

[BabaG]

thanks stainlesss. that's helpful.

[rod93]

hi
try to restore a 8mm with "John Meyer Film Restoration Script 2024" but it's a mp4 file so i put "source1= DirectShowSource(film, fps=25, convertfps=true)" don't know if it is ok, then ive got a script error " there is no function named "prefetch" thk for your help

[Emulgator]

Full script, please. And, for starters:
3. Turn off all multi-threading options until you have it working. This means commenting out the "Prefetch" line (for the current versions of AVISynth).
And you will want to use LSmashVideoSource(), even better LWLibavVideoSource() instead of DirectShowSource().
and: converting any film source to 25fps within the sourcefilter, even later based on MVTools2 might do more harm than good.

[rod93]

thk i turn off "prefetch" line the video loading into virtualdub but nothing happen...
try to run the script with virtualdul in x86 but got an another error "mu_binarize:masktools:unsupported color space" thk

[Emulgator]

Press F7.

[rod93]

thk i turn off "prefetch" line the video loading into virtualdub but nothing happen...
try to run the script with virtualdul in x86 but got an another error "mu_binarize:masktools:unsupported color space" thk

thk i change for LSmashVideoSource() no more error but did not work virtualdub is opening can't see the output result

[rod93]

Press F7.

i try to make f7 (save) but nothing is apply the output video is in same quality

[szabi]

A good transfer ;)

Fred.

Hi

It is a long time since you replied to me.
I am interested whether do you still work on best method of transfer?

Regards
szabi

[videoFred]

Hi

It is a long time since you replied to me.
I am interested whether do you still work on best method of transfer?

Regards
szabi

Hello Szabi,

My film project is finished, no need to work on it anymore.

many greetings,
Fred.

[szabi]

Hi Fred,

The reason for asking because I am near to Netherlands at this time.
I remembered what you told about good transfer needed several years ago.
I hoped we might make an arrangement on that.

Regards
Szabi

[Jenyok]

Hello Szabi,
My film project is finished, no need to work on it anymore.
many greetings, Fred.
.
Question to you.
.
How do you improve 8mm old films, especially
big black spots,
small compact spots,
small dots and dust,
thin white lines,
thin black lines,
thick hair lines (avoid real hair),
thick black lines,
lines contacting motion, avoid motion areas,
long horizontal scratches,
long vertical scratches ?
.
Thanks.

[johnmeyer]

How do you improve 8mm old films ...
,Thanks.The answer to that question is to go to post #1 in this thread and then start reading. VideoFred provided his script and has described how it works to fix most (not all) of the defects you listed.

One of the defects which are not removed by his scripts are lines which persist from one frame to the next, like scratches created by running the film through a dirty projector gate. They cannot be removed with any of the software techniques discussed in this thread. The best way to remove them -- and VideoFred has discussed this -- is to transfer the film using a "wet gate" transfer where you put lubricant on the film prior to transfer. That lubricant physically fills the scratches and makes them almost invisible.

[VideoMilk78]

The best way to remove them -- and VideoFred has discussed this -- is to transfer the film using a "wet gate" transfer where you put lubricant on the film prior to transfer. That lubricant physically fills the scratches and makes them almost invisible.

To add on to John, Fred provides his wet gate solution on one of his YouTube videos.

[szabi]

Hi

I run out of my timeframe slowly .
My alternate plan is to check transfer nearby.
I found mediafix (https://mediafix.de/super-8-digitalisieren/)and film-retter (https://www.film-retter.de/super-8-digitalisieren.html) in vicinity.
Does anyone have experience with them?

[johnmeyer]

I asked ChatGPT and they confirm that Mediafix does use a frame-by-frame transfer. You absolutely MUST have that. So, they pass that test.

Film-retter also uses frame-by-frame scanning. There is a picture of their equipment on their site. It looks similar to the equipment VideoFred built for his own use, and similar to the units sold in this country by MovieStuff. They should provide reasonably good results. The "ultimate" is to have the transfers done by a company which has a Spirit scanner, or something similar.

Since I don't live in Germany, I cannot provide any help on whether either company has good people or whether they know how to use their equipment.

[StainlessS]

@szabi, from another thread [you got chastised for posting in Avisynth Q & A thread],
I have multiple video in one avs and lots of trim.
I would like to identify which scene belongs to which video.
Is tere any syntax to display source's filmename?

See Prune() v1.14 - Trim & Splice plug:- https://forum.doom9.org/showthread.php?t=162446
maybe this post for example:- https://forum.doom9.org/showthread.php?p=1715260#post1715260
Set Show=True, to show clip index,
or can use NickNames to show an actual name and trim frame numbers.

#Avisource("D:\avs\test.avi")
Colorbars.ShowFrameNumber


ConvertToYV24 # v2.6

V0 = Last
V1 = FlipVertical
V2 = FlipHorizontal
V3 = TurnRight.BilinearResize(Width,Height)
V4 = TurnLeft.BilinearResize(Width,Height)
V5 = Invert

NickNames =""" # Psuedonyms for clips (clip index number)
ButlinsBytheSea_76 = 0
PontinsRainedOut_77 = 1
BultinsBognorRegis_78 = 2
BultinsSkegness_79 = 3
PontinsCamberSands_80 = 4
PontinsLowestoft_81 = 5
"""

SCMD=""" # Clip editing commands in string, can also use commands in file
ButlinsBytheSea_76 500,600
PontinsRainedOut_77 900,1000
BultinsBognorRegis_78 1400,1600
BultinsSkegness_79 2000,2400
PontinsCamberSands_80 2640,3000
PontinsLowestoft_81 4400,4800
PontinsRainedOut_77 5000,5400
5 6000,6500 # clip index instead of nickname
1 6501,7000
"""

SHOW=True

Prune(V0,V1,V2,V3,V4,V5,scmd=SCMD,nickname=NickNames,show=SHOW,FadeIn=True,FadeSplice=True,FadeOut=True)

return ConvertToRGB32 # For YV24 viewing

https://i.postimg.cc/K1Z3zKLX/test-00.jpg (https://postimg.cc/K1Z3zKLX)
so for image, shows

201] Using Clip
PontinsRainedOut_77{R1}[1000]

ie result frame 201, which is clip R1 [EDIT: I've actually called it V1 in the script, R1 is a generic name originally meaning "Replacement clip 1"]
frame 1000, (with alternate clip NickName of "PontinsRainedOut_77").
EDIT: Of course the clips can be different sources, I've just used FlipHorizontal etc for demo.
All clips must of course be of similar dimension and colorspace.

Sorry Fred for cluttering this thread,
I suspected szabi would not create new thread to re-ask question for which he/she was chastised.

EDIT: What the Fade stuff does for audio {FadeIn=True,FadeSplice=True,FadeOut=True}, avoids clicks and cracks.
https://s20.postimg.cc/ly9e2tqpp/Prune_Fade_Splice_zpsd3quvejt.jpg (https://postimg.cc/image/e5iqaukqh/)

[szabi]

I asked ChatGPT and they confirm that Mediafix does use a frame-by-frame transfer. You absolutely MUST have that. So, they pass that test.

Film-retter also uses frame-by-frame scanning. There is a picture of their equipment on their site. It looks similar to the equipment VideoFred built for his own use, and similar to the units sold in this country by MovieStuff. They should provide reasonably good results. The "ultimate" is to have the transfers done by a company which has a Spirit scanner, or something similar.

Since I don't live in Germany, I cannot provide any help on whether either company has good people or whether they know how to use their equipment.

Hi

I am more or less ok when the video is on my harddrive but steps before that. I am without knowledge.
My first thought was the post by VideoFred 11 years ago, I never imagined I will be in vicinity but seems this option is not available.
Thanks for checking this I go for one of them.

@szabi, from another thread [you got chastised for posting in Avisynth Q & A thread],

See Prune() v1.14 - Trim & Splice plug:................

I have to realise I forget a lot about the usage of avisynth.
I started using video editor software more I used avisynth less and now several years later it is complicate for me. :)
I still have that eight super 8 family video I would like to have a better transfer now until I am here.
I try to figure out what I edited 11 years ago.

Thanks for both of your help. :)

[Emulgator]

Film scanning: berola.de comes into mind as well (Forchheim, but you will ship your footage anyway.)

[szabi]

Film scanning: berola.de comes into mind as well (Forchheim, but you will ship your footage anyway.)

Thanks, I check it.

[virgi]

Hi, I'm converting some 8mm film to digital. I used a frame-by-frame technique, PNG frame capture, and a UVC camera.
I cleaned the film with a cloth and Avio solvent. But the frames have some stains and streaks. I wanted to try
the popular Videofred scripts, but I can't find them. I tried clicking the link in the first post, but nothing happens.
Can someone help me? Thanks (automatic translation)

[StainlessS]

I tried clicking the link in the first post, but nothing happens.

Downloads fine for me in Downloads folder.

uploaded to my MediaFire account,
https://www.mediafire.com/file/0devqxlglbyivpq/Film_Restoring_vs_06_2012.zip/file

also see stuff in VideoFred's sig, below this post.
https://forum.doom9.org/showthread.php?p=2014968#post2014968

EDIT:
To find all threads (or posts) by a particular user (eg VideoFred or JohnMeyer), go to Search, AdvancedSearch,
Enter username eg JohnMeyer, Click Show Results As POSTS,
Will show all posts by the user.
Now click on any of the users posts, and click on the users name (block left of post) above his/her avatar (if any) and click View Public Profile,
click on Statistics, and click eg "Find all threads started by johnmeyer".

[virgi]

Thanks for the file and the information. Unfortunately, the "Newest version of the script:" link doesn't work for me. It produces a flash that opens the page and stops, but the download doesn't start. Have a good evening!

[StainlessS]

link doesn't work for me.
My link does not work either ???

Try a different browser. (or maybe even your phone)

[videoFred]

Thank you Stainless for explaining everything. :thanks:

The link works for me, I have tested it on different computers.
If you click on it, it seems that nothing happens.
But on the background, it is downloaded the 'downloads' folder.

I am using Firefox but you can also copy the link and paste it in your favorite browser. That should work too.

Fred.

[virgi]

I installed Firefox and the download works fine. I probably have some settings in Chrome that are blocking the download. Sorry and thanks.

[szabi]

Hi

I visited the digitalise office.
I have 5 roll, 122 minutes.

I asked for 1080p digitalisation, 25fps, Avid DNxHR HQX, mp4 format, frame to frame progressive scan.
Customer service front desk clerk said they can convert only 24fps, 25fps is not available.
I was aslo told it is better to get 60fps digitalisation because 24fps has problem by continous play, 60fps musch better.
Unfortunately she did not know the meaning of DNxHR HQX and progressive frame.
I told her please ask a tech guy about my question but it was not an option.
Than I told ok but mp4 is a container I would like to know the codec and comression.
This point I was asked I want avi or what?
After I was told I should not worry because mp4 has a good quality and they will reduce the dust and scratches and optimize the colors and sharpness, of course they use AI for that but it costs extra payment.

Than the price calculation was done by roll number so calculated 5*30min.
Overall the price was 3 times more expensive than I have seen on their website and did not get any answer to my question I skipped the digitalisation.
I am bit disappointed now.

Regards
szabi

[Emulgator]

I visited the digitalise office.
Which one ?

[szabi]

Which one ?

I decided to wrote an email to both shop.
Only mediafix replied me see below:

Our Premium Full HD digitisation is delivered in MP4 format (H.264 codec), which is a widely compatible format suitable for playback, sharing, and long-term storage.
At the moment, ProRes 422 HQ or DNxHR HQX are unfortunately not available as output codecs in our standard workflow.

It would be nice to know which H264 is used and few info about settings too.
Answering your question I had visited them.

Film Retter simple did not reply me.

[Emulgator]

I was suggesting Berola. What about them ?

[szabi]

These two shop are in vicinity, I can drive there, Berola is far away.
I have 5 roll of film overall 122 minutes.
I calculated the 3 option and Berola is way much more expensive.
Berola with all extra option Premium TLS HD is ~840€ while Film-Scan Profiline ~2000€.
An other hand best option from mediafix ~300€ and film-retter ~200€.

Link for comparion: click (https://copilot.microsoft.com/shares/yWGTPjvH6Xb93cw5BYLGB)

[johnmeyer]

Film transfer IS expensive, which is why some of us designed and built our own transfer systems. Here in the states a decent transfer will cost at least $0.40/foot. I don't know how that cost converts to Spain and Euros. Your 133 minutes of film is about 1,600 feet, so at $0.50/foot (an above-average transfer house) that would be $800.

[Emulgator]

And I can only concur.
Letting the lowest bidder win gets no steak, but the sizzle...

[szabi]

And I can only concur.
Letting the lowest bidder win gets no steak, but the sizzle...

I would pay 20-30% more for better quality but here the difference is 10x more.
Anytime I search by different search engine, A.I search, whatever, the result is same in this region here mediafix or retter are my 2 option.

Film transfer IS expensive, which is why some of us designed and built our own transfer systems. Here in the states a decent transfer will cost at least $0.40/foot. I don't know how that cost converts to Spain and Euros. Your 133 minutes of film is about 1,600 feet, so at $0.50/foot (an above-average transfer house) that would be $800.

That is why I hoped I can make a deal with Fred until I am in vicinity.
Within a very few months I am going to leave Westfalen and that option will be gone.

[JoeSuper8]

Once again a big thanks to videoFred and Frank from Cine2Digits. Below are some of my latest regular 8mm Kodachrome transfers. Please excuse YouTube destroying the original quality of these films (altering the colors and saturation, crushing the shadows, eliminating detail and grain). The originals look noticeably better! No AI image or video processing was used.

Here are the scripts and files I use (https://modelfkeyboards.com/wp-content/uploads/2026/05/8mmTransfer.zip), along with notes on my entire step-by-step capture and processing procedure, which may help you do your own transfers. The excellent videofred script that mine is modified from has been updated to get the best results from my specific machine vision capture system (details below). All credit for the script and transfer process goes to videoFred. This new script may not make sense without consulting my notes in the above attachment. The zip file also contains newer plugin versions and Avisynth addons, which required updating some of the code. The new master script takes image sequences or avi files and generates individual AVS files for each unique name, which is good for bulk transfers. You'll likely need to modify the script (provided AS IS without warranty) to use in your own setup.

If you wanted to watch just one film, my favorite one is the first one below, the 1951 Memorial Day Parade.

Regular 8mm Kodachrome still capture (https://www.modelfkeyboards.com/wp-content/uploads/2016/08/1953-Florida.avi_snapshot_04.30.229.jpg)

1951 Memorial Day Parade Somerville Massachusetts https://www.youtube.com/watch?v=XW8XHkRqicY
1957 May Crowning of Mary https://www.youtube.com/watch?v=wZO-XzL7GRg
1954 Statue of Liberty and New York City Skyline https://www.youtube.com/watch?v=SDO32lvk-ew
1958 May Crowning of Mary Procession https://www.youtube.com/watch?v=eVZL6LdG3CA
1953 Florida Boats, Sailing, Orange Groves, Dolphins, Church Grounds, Palm Trees, Dirigible Blimp https://www.youtube.com/watch?v=YZJnaX2EFKA
1958 Storytown USA and Ghost Town amusement parks Lake George, New York https://www.youtube.com/watch?v=a6wtES_RBOE
1957 Bank Robbery Show Dodge City Storytown USA Lake George New York theme park https://www.youtube.com/watch?v=J2DT3-B6Jek

Setup:
1. Eumig Mark 610D, projector blades removed, gate widened, set to 9fps capture. Hall effect sensor with magnet attached to the cam (?) inside
2. Point Grey Machine Vision camera CM3-U3-31S4C-CS Chameleon3 3.2 MP Color USB3 Vision (Sony IMX265) + Point Grey 9 pin 0.15m GPIO Cable with JST Connector
3. Schneider Componon 50mm f/2.8 lens, set to f/5.6
4. Kulicke Soffa Multi-axis linear positioning xyz stage for focus and framing adjustment of the camera
5. Flim cleaned with Filmrenew; no wetgate transfer
6. Cine2Digits Lighting System and Capture Program, capturing to 16 bit TIFF, highlights and shadows restored with lightroom and saved to 16 bit TIFF, processed with an updated version of videofred's script


import os
import subprocess
import glob
import time
import re
from concurrent.futures import ProcessPoolExecutor

# ============================================================================================
# USER VARIABLES & PROCESSING NOTES
# 8mm film restoration script by videoFred, 2026 modification uploaded by JoeSuper8
# www.super-8.be
# info@super-8.be
# ============================================================================================
# Path for TIFF sequences (Lightroom processed)
TIFF_SOURCE_DIR = r"C:\Users\Joe\Desktop\8mm Film Transfers\3 Lightroom processed"

# Path for AVI files (Unprocessed or 16-bit uncompressed)
# LWLibavVideoSource is used for these as it is superior for uncompressed 16-bit AVI.
AVI_SOURCE_DIR = r"C:\Users\Joe\Desktop\8mm Film Transfers\5 unprocessed AVI test"

# Destination for the rendered AVI files
OUTPUT_DIR = r"C:\Users\Joe\Desktop\8mm Film Transfers\4 AviSynth Processed"

# Secondary directory to check for existing files (Archived/Secondary storage)
# The script will skip rendering if the file exists in either OUTPUT_DIR or ARCHIVE_DIR.
ARCHIVE_DIR = r"L:\8mm Film Transfers\4 AviSynth Processed 16 bit YUV 444"

# --- BIT DEPTH SETTINGS ---
# 8 = Fast processing (FFV1 8-bit YUV 4:4:4) - Ideal for daily viewing/sharing.
# 16 = High Precision (FFV1 16-bit YUV 4:4:4) - Best for archival and further grading.
USE_BIT_DEPTH = 16

# --- TRIM SETTINGS ---
# Set to None to process the full video.
# To trim, use [start_frame, end_frame]. Example: [0, 1000] processes first 1001 frames.
TRIM_FRAMES = None

# --- ACTION MODES ---
# "create" = Generates .avs files in current folder based on found sources.
# "run" = Scans current folder for .avs files and starts FFmpeg rendering.
# "both" = Creates the scripts and then immediately starts the render queue.
ACTION = "create"

# --- PERFORMANCE ---
# MAX_SIMULTANEOUS: How many FFmpeg windows to open at once.
# USE_PREFETCH: Set to True to enable AviSynth+ multi-threading (e.g., Prefetch(12)).
MAX_SIMULTANEOUS = 10
USE_PREFETCH = False

# --- TEMPORAL CLEANING & DEFLICKER ---
# These parameters control the initial pre-cleaning and flicker reduction.
DEFLICKER_TR = 5 # Temporal radius (Default option 5 looks at 11 frames total)
DEFLICKER_THSAD = 100 # SAD threshold for initial cleaning (Default 100)

# PARAMETERS
#--------------------------------------------------------------------------------------------
play_speed = 16
saturation = 1.60
gamma = 1.0
black_level = 0
white_level = 255

# STABILISING PARAMETERS
#--------------------------------------------------------------------------------------------
maxstabH = 40
maxstabV = 40
est_left = 40
est_top = 40
est_right = 40
est_bottom = 40
trust_value = 1.0
cutoff_value = 0.5

# CLEANING/DENOISING PARAMETERS
#--------------------------------------------------------------------------------------------
dirt_strenght = 40 # Default value 35
denoising_strenght = 100
denoising_frames = 5

# SHARPENING PARAMETERS
#--------------------------------------------------------------------------------------------
USM_sharp_ness = 50
USM_radi_us = 4
last_sharp = 0.8
last_blur = 0.8

# ============================================================================================
# SCRIPT ENGINE
# ============================================================================================

def get_last_frame_number(first_file_path):
directory = os.path.dirname(first_file_path)
base_name = os.path.basename(first_file_path).replace(" (1).tif", "")
pattern = re.compile(rf"{re.escape(base_name)} \((\d+)\)\.tif", re.IGNORECASE)

frame_numbers = []
if os.path.exists(directory):
for f in os.listdir(directory):
match = pattern.match(f)
if match:
frame_numbers.append(int(match.group(1)))
return max(frame_numbers) if frame_numbers else 1

def get_ffmpeg_args(bit_depth, output_path):
if bit_depth == 16:
# Use FFV1 Lossless for 16-bit YUV 4:4:4 to maintain 0-255 Full Range
return f'ffmpeg -i "IN_FILE" -c:v ffv1 -coder 1 -context 1 -g 1 -level 3 -pix_fmt yuv444p16 -color_range pc "{output_path}"'
else:
# Use FFV1 Lossless for 8-bit YUV 4:4:4 to maintain 0-255 Full Range
return f'ffmpeg -i "IN_FILE" -c:v ffv1 -coder 1 -context 1 -g 1 -level 3 -pix_fmt yuv444p -color_range pc "{output_path}"'

def create_avs_file(source_path, is_sequence):
file_no_ext = os.path.basename(source_path).replace(" (1).tif", "").replace(".avi", "")
avs_name = f"{file_no_ext}.avs"
clean_path = source_path.replace("\\", "/")

if is_sequence:
seq_path = clean_path.replace(" (1).tif", " (%d).tif")
last_frame = get_last_frame_number(source_path)
source_line = f'source = ImageSource("{seq_path}", start=1, end={last_frame}, fps={play_speed})'
else:
source_line = f'source = LWLibavVideoSource("{clean_path}").Assumefps({play_speed})'

if isinstance(TRIM_FRAMES, list) and len(TRIM_FRAMES) == 2:
trim_line = f"source = source.Trim({int(TRIM_FRAMES[0])}, {int(TRIM_FRAMES[1])})"
else:
trim_line = "# source = source.Trim(start, end)"

script_template = f"""
# 8mm film restoration script
# Filename: {file_no_ext}
# ============================================================================================
# 1. LOAD PLUGINS
# ============================================================================================
LoadPlugin("avsplugins64/SmoothAdjust.dll")
LoadPlugin("avsplugins64/MaskTools2.dll")
LoadPlugin("avsplugins64/ImageSeq.dll")
LoadPlugin("avsplugins64/LSMASHSource.dll")
LoadPlugin("avsplugins64/Depan.dll")
LoadPlugin("avsplugins64/DepanEstimate.dll")
LoadPlugin("avsplugins64/RemoveDirt.dll")
LoadPlugin("avsplugins64/MVTools2.dll")
LoadPlugin("avsplugins64/RgTools.dll")

# 2. IMPORT DEPENDENCIES
Import("avsplugins64/Zs_RF_Shared.avsi")
Import("avsplugins64/ExTools.avsi")
Import("avsplugins64/ResizersPack.avsi")
Import("avsplugins64/03_RemoveDirtMC.avs")
Import("avsplugins64/UnsharpMask_avsi.avsi")
Import("avsplugins64/SMDegrain.avsi")

# 3. LOAD SOURCE
{source_line}
{trim_line}

# 4. DIMENSION CORRECTION
w = width(source)
h = height(source)
crop_r = w % 4
crop_t = h % 2
source = (crop_r != 0 || crop_t != 0) ? source.Crop(0, crop_t, -crop_r, 0) : source

# 5. BIT DEPTH INITIALIZATION
source = ({USE_BIT_DEPTH} == 16) ? source.ConvertBits(16).ConvertToYUV444(matrix="Rec709") : source.ConvertBits(8).ConvertToYUV444(matrix="Rec709")
source = source.propset("_ColorRange", 0).propset("_Matrix", 1)

# 6. STABILIZING & DEFLICKER
source1_8bit = source.ConvertBits(8).ConvertToYV12()
stab_ref = source1_8bit.crop(20,20,-20,-20).colorYUV(autogain=true).crop({est_left},{est_top},-{est_right},-{est_bottom})
mdata = DePanEstimate(stab_ref, trust={trust_value}, dxmax={maxstabH}, dymax={maxstabV})
stab = DePanStabilize(source, data=mdata, cutoff={cutoff_value}, dxmax={maxstabH}, dymax={maxstabV}, method=0, mirror=15)
stab_clean = stab.SMDegrain(tr={DEFLICKER_TR}, thSAD={DEFLICKER_THSAD}, prefilter=2, plane=4)

# 7. CLEANING & LEVELS
noise_base = stab_clean.SmoothLevels(0, {gamma}, 255, 0, 255, LMode=0).Tweak(sat={saturation})
cleaned = RemoveDirtMC(noise_base, {dirt_strenght}).UnsharpMask_avsi({USM_sharp_ness}, {USM_radi_us}, 0).UnsharpMask_avsi({USM_sharp_ness}+25, {USM_radi_us}-1, 0)

# 8. DEGRAINING & SHARPENING
denoised = cleaned.McDegrainSharp(frames={denoising_frames}, strenght={denoising_strenght}, bsrch=true).UnsharpMask_avsi({USM_sharp_ness}*2, {USM_radi_us}-2, 0).sharpen({last_sharp})

# 9. FINAL LEVELS & BLUR
result4 = denoised.SmoothLevels({black_level}, 1.0, {white_level}, 0, 255, LMode=0).blur({last_blur})

# 10. FINAL EXPORT TAGGING
final_render = result4.propset("_ColorRange", 0).propset("_Matrix", 1)
# Both 16-bit and 8-bit output YUV444 to maintain 0-255 Full Range
final_render = ({USE_BIT_DEPTH} == 16) ? final_render.ConvertToYUV444(matrix="Rec709") : final_render.ConvertToYUV444(matrix="Rec709")

final_render
{"Prefetch(12)" if USE_PREFETCH else "#Prefetch(12)"}

# --- MCDEGRAINSHARP FUNCTION ---
function McDegrainSharp(clip c, int \"frames\", int \"strenght\", float \"bblur\", float \"csharp\", bool \"bsrch\") {{
f_radius = default(frames, 2)
s_sad = default(strenght, 400)
b_val = default(bblur, 1.0)
c_val = default(csharp, 1.0)
b_srch = default(bsrch, true)
blocks = 16
overl = 8
c2 = c.blur(b_val)
super = b_srch ? c2.MSuper(pel=2, sharp=1) : c.MSuper(pel=2, sharp=1)
super_rend = c.sharpen(c_val).MSuper(pel=2, sharp=1, levels=1)

b3v = MAnalyse(super, isb = true, delta = 3, blksize=blocks, overlap=overl)
b2v = MAnalyse(super, isb = true, delta = 2, blksize=blocks, overlap=overl)
b1v = MAnalyse(super, isb = true, delta = 1, blksize=blocks, overlap=overl)
f1v = MAnalyse(super, isb = false, delta = 1, blksize=blocks, overlap=overl)
f2v = MAnalyse(super, isb = false, delta = 2, blksize=blocks, overlap=overl)
f3v = MAnalyse(super, isb = false, delta = 3, blksize=blocks, overlap=overl)

res = (f_radius<=0) ? c : (f_radius==1) ? c2.MDegrain1(super_rend, b1v, f1v, thSAD=s_sad) : \\
(f_radius==2) ? c2.MDegrain2(super_rend, b1v, f1v, b2v, f2v, thSAD=s_sad) : \\
c2.MDegrain3(super_rend, b1v, f1v, b2v, f2v, b3v, f3v, thSAD=s_sad)
return res
}}
"""

with open(avs_name, "w") as f:
f.write(script_template)
return avs_name

def run_ffmpeg(script):
"""
Checks if output already exists in Primary or Archive directory.
Skips if found.
"""
target_filename = script.replace(".avs", ".avi")
primary_out = os.path.join(OUTPUT_DIR, target_filename)
archive_out = os.path.join(ARCHIVE_DIR, target_filename)

# Check for existence in either directory
if os.path.exists(primary_out):
print(f"SKIPPING: {target_filename} already exists in OUTPUT_DIR.")
return
if os.path.exists(archive_out):
print(f"SKIPPING: {target_filename} already exists in ARCHIVE_DIR.")
return

# Construct the ffmpeg command
cmd = get_ffmpeg_args(USE_BIT_DEPTH, primary_out).replace("IN_FILE", script)
print(f"Starting: {script}")
subprocess.run(cmd, shell=True)
print(f"Finished: {script}")

if __name__ == "__main__":
# Ensure the output directory exists
if not os.path.exists(OUTPUT_DIR):
os.makedirs(OUTPUT_DIR)

# Gather source files
tiffs = glob.glob(os.path.join(TIFF_SOURCE_DIR, "*(1).tif"))
avis = glob.glob(os.path.join(AVI_SOURCE_DIR, "*.avi"))
all_sources = [(s, True) for s in tiffs] + [(s, False) for s in avis]

if not all_sources:
print(f"!!! ERROR: No sources found. Check your directories.")

# Phase 1: Script Creation
if ACTION in ["create", "both"]:
for source_path, is_seq in all_sources:
name = create_avs_file(source_path, is_seq)
print(f"Created: {name}")
print(f"DONE: Created {len(all_sources)} AVS files.")

# Phase 2: Parallel Rendering
if ACTION in ["run", "both"]:
# Find all scripts to process
job_list = glob.glob("*.avs")

if not job_list:
print("No .avs files found to run.")
else:
print(f"--- Starting Render Queue (Max {MAX_SIMULTANEOUS} simultaneous) ---")

with ProcessPoolExecutor(max_workers=MAX_SIMULTANEOUS) as executor:
executor.map(run_ffmpeg, job_list)

print("--- ALL RENDER JOBS COMPLETE ---")

[johnmeyer]

Great-looking transfers!

[szabi]

I handed over my 8mm rolls 3 weeks ago and now I received the email to pickup my staff due to digitalisation is done.
All my story started 12 years ago (https://forum.doom9.org/showthread.php?p=1697308#post1697308).
I do not know the technology they use for digitalisation.
I selected the best digit option, it was called professional 1080p@25fps.
I asked for prores 422hq for better post processing.
I paid 225€ for the 5 roll digit.
And now I am disappointed.

Shop was not willing to use my external usb drive so I had to buy one from them.
There was 2 videofile about every roll, mp4 (avc) and mov (prores).
I did think it is a good approach I can watch mp4 instantly and later I can start work on mov.
Than I saw the file attributes, mp4 was created earlier.
I have an assumption their digitalisation method have yuv420 8bit output that is why mp4 was first and later they converted prores yuv422 10bit from that.
It makes no sense to do such a conversion.
The transfer quality does not look like a professional one as I selected. Somtimes seems it is an SD lanczos resized to HD.

I realised some video missing the first few seconds. It is bit unprofessional, careless approach.
Finally the moment I went really sad when I checked the rolls.
https://i.postimg.cc/QF4DrLRP/A6705327-re.jpg (https://postimg.cc/QF4DrLRP)
They cut two pieces.
I feel sorrow and disappointment by the shop that has review 4.9 point (https://maps.app.goo.gl/Z9Mh5KSCoFipLgmk8).

[Emulgator]

I am sorry to hear that.

[johnmeyer]

Old film often must be cut and spliced, both for repair and also to get it to run through the transfer machine.

Your criticisms seem to be entirely about file formats and other digital aspects, but you say nothing about the transfers. How do they look? Can you post an unmodified sample?

I have seen hundreds of good transfers and many bad transfers. I know what to look for and what makes a difference. YUV422 vs. YUV420 is, quite frankly, not a big issue compared to all the other aspects of how a transfer is done.

[szabi]

I uploaded 13sec sample here is the link: click (https://1drv.ms/v/c/352659f788abfc41/IQCqiM2wmbGQTL9J3lLY1NBxAbBS3AkEIHGbfIazVozC1JU?e=mWiktT)
I did use avidemux in copy mode to export an unmodified sample.
I understand for some reason it might be needed to cut the film. But only 40cm is even not possible to run through any machine.

EDIT:
Because I still have the first transfer from 12 years ago, the same scene from it for comparison: click (https://1drv.ms/v/c/352659f788abfc41/IQD_gs0tbnDyRrngO7RFjiZTAT9x3xjmOJIK6oI2ziaUsYQ?e=emC263).

[johnmeyer]

Neither scan is very good, although the new scan is worse. Was this Polavision film? I used to transfer that and it had massive chemical stains, like your film. I developed a way to remove them, somewhat. Here is an example:

Polavision Before/After (https://www.youtube.com/watch?v=ud1XNIoh1P8)

If your film is 8mm, there should be more detail, although since detail is lacking in both transfers, perhaps it was recorded on bad film stock or with a cheap camera.

[szabi]

I cannot tell you, my uncle recorded these films at late 70's early 80's.
I really do not know it is Polavision or not. I am that kid on the video and do not remember too much.
It seems eighter the cheap camera, bad film, improper film storage or any of these combination makes that result.
My conclusion I can spend a tons of money for transfer but my source is bad which makes bad outcome.

About the firm my opinion is still the same.
Especially as you say the 12 years old SD transfer is better.
Additionally you know that 40cm they cut has a 2 seconds scene with 2 of my relatives, none of them alive since. For them it just a simple cut for me it is a tragedy.
Disappointment.

Anyway I did try to use the script and files uploaded by JoeSuper8 but first attempt failed.
I was never a good sript writer but now I even lost a lot of knowledge.
I simply do not get how to modify it to my need.

[videoFred]

Hello Szabi,

There is fungus on your film. I do not know how to remove it but I can ask a friend.

You need a new transfer, without duplicates. One film frame = one digital frame. Then play speed does not matter, it's just a number telling the computer how fast to play the file. Frame rate conversion can be done afterwards with interpolation or whatever.

Frame size depends on the used camera from the transfer system.Max. res from my Flea camera is 1392x1044 (4:3) this is more than enough for 8mm film. Upscaling and adding black borders can be done afterwards.

There is no need for space consuming and slow exotic codecs, 10 bits etc.... I always use the Canopus HQX codec, it can be downloaded for free and it's very fast.

Many greetings from the Netherlands,
Fred.

[johnmeyer]

VideoFred is correct: it could be mold. I initially thought it was chemical stains because of my Polavision experience. The chemical stains on Polavision (and any other film) are impossible to remove because they are embedded in the emulsion. By contrast, mold can sometimes be removed, depending on how deeply it has penetrated.

About five years ago I did the second of two huge transfers and renovations of 78 and 33-1/3 rpm one-of-a-kind recordings done by Jerry Lewis, the late well-known comedian. These had been stored in horrible conditions on the Paramount lots since the late 1950s. Some of the records were covered in white mold. I did an initial transfer which, to my surprise, did actually produce sound (I didn't think the needle would penetrate the gunk). I then got the idea that perhaps my 9 liter ultrasonic cleaner could remove the mold without damaging the grooves. I quickly did some research and found that this is actually something which extreme audiophiles do to their records, with some of them cleaning brand-new recordings before their first play. So, I went ahead and tried it and the results were absolutely astounding.

Having said that, many years before this, when I was trying to get a decent transfer from the horribly stained Polavision, I decided to try the same thing. I put some distilled water and Photoflo in my ultrasonic cleaner and dipped an inch of unexposed leader into the operating cleaner.

It took less than two seconds to remove the entire emulsion!!

However, Polavision's emulsion is totally different than regular film emulsion and for years I've wanted to re-try the experiment with my dad's 1930s B&W negatives that got badly stained when our basement flooded.

[szabi]

Hi

Due to your experience you were in my mind firstly, I asked you about last year (https://forum.doom9.org/showthread.php?p=2014968#post2014968) because I live near Aachen and travelled several times in Netherlands.
However this part of my life goes to the end, one month and I leave Germany.
I did run out of time to find any other transfer firm nearby and sure at home the option is zero.
Even I have years here the opportunity is missed.
The question is no longer the transfer but what I can do to the movie I have now.

Regards

[szabi]

I received an email from the firm asking to fill the satisfactory survey.

[FredThompson]

I have not done any AviSynth work in at least 10 years. Am "rebuilding" my toolset (locating current "best-in-class" filters, etc.) I see the AviSynth (yeah, I'm old school...still use the waiter/platter logo...) .nl site is out of bandwidth until ?

videoFred, is your 8mm script set updated from 20/06/2012? If so, where is it now?
JoeSuper8, just grabbed your script set from a few posts prior.

Are there any fundamental differences, as far as those filters go, between 8mm and 16mm film? I don't imagine there are any, just a large frame size so a larger data set per frame.

I've got 16mm film stock. Does anyone have links to DIY telecine equipment? I'd like to build my own.

Thanks, in advance, for any helpful guidance.

[videoFred]

I

videoFred, is your 8mm script set updated from 20/06/2012? If so, where is it now?

Hello Fred, no, it is not updated, this is the latest version online.


Are there any fundamental differences, as far as those filters go, between 8mm and 16mm film? I don't imagine there are any, just a large frame size so a larger data set per frame.

There is no difference. I advice not to focus on digital frame size, but on quality.

Many greetings,
Fred.

[johnmeyer]

There is no difference. I advice not to focus on digital frame size, but on quality.I could be wrong, but I think VideoFred (two Freds here:)) is saying that you should make sure you don't scan at more resolution than you need. If that is not what he was saying, then I apologize. However, it IS what I am saying. I have found that both individuals in this forum, as well as various film transfer services, often scan at 4K resolutions. I never thought that was needed, even for 16mm (35mm or 70mm are a different situation). If you scan at too high a resolution, you pay a huge penalty in processing time, as well as file size, without any real increase in quality.

I had a chance to test this myself. About twenty years ago I did the biggest single film transfer project I've ever done. It consisted of well over 200 reels of film, mostly 16mm, and mostly from pre-war Germany. Some of the movies showed political rallies and other historically significant events. The owner of the movies eventually licensed these to The Smithsonian Channel. The Smithsonian Channel wanted to have them scanned by their film-scanning post house. I arranged to have this done, and then asked that the digital scans be sent directly to me. After I reviewed them, I then sent them to The Smithsonian Channel. This let me compare "the ultimate" scan with what I do with my modified Eiki projector transfer system which I capture with my ancient Sony FX1 1440x1080 camera, but which is down-res'd in the camera to 720x480 DV video before it is saved to my hard drive.

Their scan is obviously better, but resolution is not the main reason. As I expected, the main difference was that their captures did a better job of preserving detail in the shadows. Also, mine had slightly more compression artifacts due to the DV compression, although I don't think you'd notice this much when actually watching the movie rather than studying a still frame. The professional scans are sharper. However, some of this was due to sharpening (I don't use much); some of this was due to better optics (I don't think they use a lens, but instead scan directly). I don't think much was due to scanning at higher resolution.

Here is a sample of my scan of 1936 16mm German film:

https://i.imgur.com/8cWqpxD.png

Here is the same frame scanned by a professional film scanning post production facility:

https://i.imgur.com/8OnMjIe.png

I should add that they adjusted their gamma (more or less the same thing as contrast) slightly differently, and you have to ignore this when doing the comparison. I could have easily changed the gamma on my captures to match their aesthetic.

You cannot read any signs or resolve any detail in the professional 1920x1080 scan that can't be resolved (via adding more sharpening) in my 720x480 home-brew transfer. Their scan is better, but not because they scanned at higher resolution.

The reason I think most of their better quality is due to sharpening and optics is something that I posted in this thread almost two decades ago. These two images are from my own scan. The second is the result of applying sharpening. Look at the fence railing and you will swear that I must have re-scanned at a higher resolution. The extra "detail" is entirely due to sharpening.
https://i.imgur.com/yhceYyZ.jpeg

[Wilbert]

I have not done any AviSynth work in at least 10 years. Am "rebuilding" my toolset (locating current "best-in-class" filters, etc.) I see the AviSynth (yeah, I'm old school...still use the waiter/platter logo...) .nl site is out of bandwidth until ?
(...)

The hosting company increased some resource limits and it should be working again.

[videoFred]

I could be wrong, but I think VideoFred (two Freds here:)) is saying that you should make sure you don't scan at more resolution than you need.

Hi John, you are not wrong, this is what I was saying. :)

[FredThompson]

Thanks guys.
johnmeyer, looks like the pro scan was digitally cleaned, too. There's a small scratch and some flecks that have been filled in.
I remember many discussions here about digitizing resolution, schmidt triggers, multiple passes, grain size, etc. - similar to audio curve fitting...

[johnmeyer]

They may have done a wet gate scan, which would have filled in the scratches. I did do dirt removal in the frame I posted, but their dirt removal software is better. Having just compared theirs with mine again, I think most of the difference in "detail" is simply due to them using more aggressive (and probably better) sharpening algorithms, as I did with the before/after sample I posted above where the porch railing magically appear after applying sharpening.