[film transfer] Remove dirt but keep grain?

[chmars]

I Forgot the code used for the test:
Can't exclude I missed something about SpotRemover...

source1= AviSource("D:\somepath\459_s2_002-XAVCI-4K-BLD_5_0-trim_crop-for_tests-graded.avi")
source1= KillAudio(source1)
source1 = crop(source1,700,0,-500,0)

source1_8bit = ConvertBits(source1, bits=8) #I dont'know much about formats, maybe this is not ideal...
source1YV12 = ConvertToYV12 (source1_8bit) #I dont'know much about formats, maybe this is not ideal...

txt_sz = 36

BlkSz =32
OLap = (BlkSz/2) #important not too low (eg 4 for blksz 12, because creates stairs)
Pel = 1
Tm=false
Bblur = 0.6
ThSAD = 10000
RadT = 1

Usharp_strength= 80
Usharp_radius= 5
Usharp_th = 1 #too high = too many dark pixel packs (ex trees)

#Remove spots-----------------------------------------------------------------
spotless_107=SpotLess107(source1,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
SpotRemoved = SpotRemover(source1YV12,Spot=128, RadT=1, Pel=Pel, BlkSz=32, DeGrain=false, Chroma=true, PreFilter=true, mask=false, Tm = true,thsad=thsad, Glob = true)
#----------------------------------------------------------------------------------

SpotRemoved = ConvertBits(SpotRemoved, bits=10) #I dont'know much about formats, maybe this is not ideal...
SpotRemoved = ConvertToYUV422(SpotRemoved) #I dont'know much about formats, maybe this is not ideal...

#Sharpen----------------------------------------------------------------------
spotless_107_shp=spotless_107.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
SpotRemoved_shp=SpotRemoved.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#----------------------------------------------------------------------------------

# Restore grain -------------------------------------------------------------
spotless_107_shp_Rg = mRD_RestoreGrain(spotless_107_shp, source1)
SpotRemoved_shp_Rg = mRD_RestoreGrain(SpotRemoved_shp, source1)
#----------------------------------------------------------------------------------

group = StackHorizontal(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(spotless_107_shp_Rg,"spotless_107_shp_Rg",size = txt_sz,align=2),\
subtitle(SpotRemoved,"SpotRemover",size = txt_sz,align=2))

return(group)#.AddBorders(1000, 0, 500, 0, color=$000000)



#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------


Function mRD_RestoreGrain (clip rd, clip o, float "str1", float "str2") {

sisavs26 = !(VersionNumber() < 2.60)

str1 = string(default(str1, 10.0))
str2 = string(default(str2, 5.0))

expr = sisavs26 ? " x range_half - abs "+str1+" scalef min "+str2+" scalef x range_half - abs - min 0 max x range_half - x range_half - abs 1 max / * range_half + " : " x 128 - abs "+str1+" min "+str2+" x 128 - abs - min 0 max x 128 - x 128 - abs 1 max / * 128 + "

Return sisavs26 ? mt_makediff(o, rd).mt_lut(expr, use_expr=2).mt_adddiff(rd, chroma="copy second") : mt_makediff(o, rd).mt_lut(expr).mt_adddiff(rd, chroma="copy second")
}


Function SpotLess107(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess107: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

Function SpotRemover(clip c, int "Spot", bool "DeGrain", int "RadT", int "ThSAD", int "ThSAD2", int "Pel", bool "Chroma",
\ int "BlkSz", Int "Olap", bool "Tm", bool "Glob", bool "PreFilter", bool "Mask", string "Denoiser") {
Spot = Default(Spot, 32) # Spot size
DeGrain = Default(DeGrain, false) # MDeGrain in MvTools
RadT = Default(RadT, 1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD, 1000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2, 130) # SAD threshold at radius RadT.
Pel = Default(pel, 1) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma, true) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz, 8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, (Blksz/2)) # Default half of BlkSz. #################################################################
Tm = Default(tm, true) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob, true) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
PreFilter = Default(PreFilter, true)
Mask = Default(Mask, false)
Denoiser = Default(Denoiser, Undefined)

pad = Max(BlkSz, 8)

sup = c.mt_lut(expr="x 20 1 x 127.5 / - 21 ^ pi x * 255 / cos 21 ^ - * 1.05 * -", Y=3, U=2, V=2)
\ .RemoveGrain(mode=17).MSuper(hpad=pad, vpad=pad, pel=pel, sharp=2, chroma=Chroma)
sup_rend = MSuper(c, hpad=pad, vpad=pad, pel=pel, sharp=2, levels=1, chroma=Chroma) # Only 1 Level required where not MAnalyse-ing.

c1 = PreFilter ? Eval("""
MultiVec = MAnalyse(sup, multi=true, delta=RadT, blksize=BlkSz*2, overlap=OLap*2, search=5, dct=0, chroma=Chroma, truemotion=Tm, global=Glob)
c0 = MCompensate(c, sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
c0 = RadT == 1 ? Clense(c0) : MedianBlurTemporal(c0, radiusY=0, radiusU=0, radiusV=0, temporalradius=RadT)
c0 = SelectEvery(c0, RadT*2+1, RadT) # Return middle frame
return Repair(c0, c.RemoveDust().Sharpen(0.1), mode=17)
""") : c

bvec1 = MAnalyse(sup, isb=true, blksize=BlkSz, overlap=OLap, divide=0, delta=1, search=5, dct=7, plevel=0,
\ chroma=Chroma, truemotion=Tm, global=Glob) # backward vectors
fvec1 = MAnalyse(sup, isb=false, blksize=BlkSz, overlap=OLap, divide=0, delta=1, search=5, dct=7, plevel=0,
\ chroma=Chroma, truemotion=Tm, global=Glob) # forward vectors
backw1 = MFlow(c1, sup_rend, bvec1)
forw1 = MFlow(c1, sup_rend, fvec1)

c2 = Defined(Denoiser) ? Interleave(backw1, c1, forw1).Eval(Denoiser) : Interleave(backw1, c1, forw1)

msk = Mask ? Eval("""
sadb = MMask(c, bvec1, ml=50, kind=1, gamma=1, thscd1=ThSAD)#.mt_binarize(15, upper=true) # mask SAD backward
sadf = MMask(c, fvec1, ml=50, kind=1, gamma=1, thscd1=ThSAD)#.mt_binarize(15, upper=true) # mask SAD forward
msad = mt_logic(sadf, sadb, "and", chroma="-127.5")#.mt_expand() # combined mask SAD
return Interleave(msad, msad, msad)
""") : Undefined

filtered = c2.DeSpot(p1=6, p2=6, pwidth=Spot, pheight=Spot, mthres=16, mwidth=Spot/2, mheight=Spot/2,
\ blur=1, motpn=true, seg=2, color=true, ranked=true, show=0, extmask=msk)

restore = c2.RemoveDust().Sharpen(0.1)

c3 = RestoreMotionBlocks(filtered, restore, alternative=restore, pthreshold=10, cthreshold=10, gmthreshold=160,
\ dist=3, dmode=2, noise=3, noisy=9, grey=false, debug=false, show=false)

SelectEvery(c3, 3, 1)

DeGrain ? MDeGrain1(last, sup_rend, bvec1, fvec1, plane=4, limit=255.0, thSad=400) : last
}


function RemoveDust(clip input, int "mode") {
mode = Default(mode, 17)

clensed = Clense(input)
rep = Repair(clensed, input, mode=mode)

RemoveGrain(rep)
}

[zorr]

I liked your idea of restricting the filtering to darkish spots. So here's a modded script that does that, based on Spotless (and this same trick will work with any other degrain filter as well):

source1= AviSource("459_s2_002-XAVCI-4K-BLD_5_0-trim_crop-for_tests.avi")
source1= KillAudio(source1)
source1 = crop(source1,700,0,-500,0)

source1_8bit = ConvertBits(source1, bits=8) #I dont'know much about formats, maybe this is not ideal...
source1YV12 = ConvertToYV12 (source1_8bit) #I dont'know much about formats, maybe this is not ideal...

txt_sz = 36

BlkSz =32
OLap = (BlkSz/2) #important not too low (eg 4 for blksz 12, because creates stairs)
Pel = 1
Tm=false
Bblur = 0.6
ThSAD = 10000
RadT = 1

Usharp_strength= 80
Usharp_radius= 5
Usharp_th = 1 #too high = too many dark pixel packs (ex trees)

#Remove spots-----------------------------------------------------------------
spotless_107=SpotLess107(source1,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
SpotRemoved = SpotRemover(source1YV12,Spot=128, RadT=1, Pel=Pel, BlkSz=32, DeGrain=false, Chroma=true, PreFilter=true, mask=false, Tm = true,thsad=thsad, Glob = true)
#----------------------------------------------------------------------------------

SpotRemoved = ConvertBits(SpotRemoved, bits=10) #I dont'know much about formats, maybe this is not ideal...
SpotRemoved = ConvertToYUV422(SpotRemoved) #I dont'know much about formats, maybe this is not ideal...

#Sharpen----------------------------------------------------------------------
spotless_107_shp=spotless_107.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
SpotRemoved_shp=SpotRemoved.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#----------------------------------------------------------------------------------

# Restore grain -------------------------------------------------------------
spotless_107_shp_Rg = mRD_RestoreGrain(spotless_107_shp, source1)
SpotRemoved_shp_Rg = mRD_RestoreGrain(SpotRemoved_shp, source1)
#----------------------------------------------------------------------------------


# diff ----------------------------------------------------------------------

diff = Grayscale(mt_makediff(spotless_107_shp, source1))

#----------------------------------------------------------------------------


# white mask ----------------------------------------------------------------

threshold = 0.9 # adjust threshold to change how bright objects can get removed
whitemask = mt_lut(diff, expr="x range_half "+String(threshold)+" * > 255 scalef 0 ?")
whitemask_orig = whitemask

# fill small mask holes -----------------------------------------------------

filltimes = 1 # increase filltimes to remove larger holes from the mask
whitemask = whitemask.mt_invert()
whitemask_inverted = whitemask
for (i = 1, filltimes) {
whitemask = whitemask.mt_inpand()
}
whitemask = mt_hysteresis(whitemask, whitemask_inverted)
whitemask = whitemask.mt_invert(chroma="-127")

# enlarge mask (optional) ---------------------------------------------------

# experiment with mt_inpand and/or mt_deflate to restore more of the removed object
#whitemask = whitemask.mt_inpand(chroma="-127")
#whitemask = whitemask.mt_deflate(chroma="-127")

#----------------------------------------------------------------------------

# whitemask overlay ---------------------------------------------------------

overlay = Overlay(source1, spotless_107_shp, mask=whitemask)

#----------------------------------------------------------------------------

group = StackHorizontal(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(spotless_107_shp,"spotless_107_shp",size = txt_sz,align=2),\
subtitle(overlay,"overlay",size = txt_sz,align=2),\
subtitle(whitemask,"filled whitemask",size = txt_sz,align=2),\
subtitle(whitemask_orig,"orig. whitemask",size = txt_sz,align=2),\
subtitle(diff,"diff",size = txt_sz,align=2))
#subtitle(SpotRemoved,"SpotRemover",size = txt_sz,align=2))
#subtitle(spotless_107_shp_Rg,"spotless_107_shp_Rg",size = txt_sz,align=2),\

return(group)#.AddBorders(1000, 0, 500, 0, color=$000000)



#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------


Function mRD_RestoreGrain (clip rd, clip o, float "str1", float "str2") {

sisavs26 = !(VersionNumber() < 2.60)

str1 = string(default(str1, 10.0))
str2 = string(default(str2, 5.0))

expr = sisavs26 ? " x range_half - abs "+str1+" scalef min "+str2+" scalef x range_half - abs - min 0 max x range_half - x range_half - abs 1 max / * range_half + " : " x 128 - abs "+str1+" min "+str2+" x 128 - abs - min 0 max x 128 - x 128 - abs 1 max / * 128 + "

Return sisavs26 ? mt_makediff(o, rd).mt_lut(expr, use_expr=2).mt_adddiff(rd, chroma="copy second") : mt_makediff(o, rd).mt_lut(expr).mt_adddiff(rd, chroma="copy second")
}


Function SpotLess107(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess107: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

Function SpotRemover(clip c, int "Spot", bool "DeGrain", int "RadT", int "ThSAD", int "ThSAD2", int "Pel", bool "Chroma",
\ int "BlkSz", Int "Olap", bool "Tm", bool "Glob", bool "PreFilter", bool "Mask", string "Denoiser") {
Spot = Default(Spot, 32) # Spot size
DeGrain = Default(DeGrain, false) # MDeGrain in MvTools
RadT = Default(RadT, 1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD, 1000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2, 130) # SAD threshold at radius RadT.
Pel = Default(pel, 1) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma, true) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz, 8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, (Blksz/2)) # Default half of BlkSz. #################################################################
Tm = Default(tm, true) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob, true) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
PreFilter = Default(PreFilter, true)
Mask = Default(Mask, false)
Denoiser = Default(Denoiser, Undefined)

pad = Max(BlkSz, 8)

sup = c.mt_lut(expr="x 20 1 x 127.5 / - 21 ^ pi x * 255 / cos 21 ^ - * 1.05 * -", Y=3, U=2, V=2)
\ .RemoveGrain(mode=17).MSuper(hpad=pad, vpad=pad, pel=pel, sharp=2, chroma=Chroma)
sup_rend = MSuper(c, hpad=pad, vpad=pad, pel=pel, sharp=2, levels=1, chroma=Chroma) # Only 1 Level required where not MAnalyse-ing.

c1 = PreFilter ? Eval("""
MultiVec = MAnalyse(sup, multi=true, delta=RadT, blksize=BlkSz*2, overlap=OLap*2, search=5, dct=0, chroma=Chroma, truemotion=Tm, global=Glob)
c0 = MCompensate(c, sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
c0 = RadT == 1 ? Clense(c0) : MedianBlurTemporal(c0, radiusY=0, radiusU=0, radiusV=0, temporalradius=RadT)
c0 = SelectEvery(c0, RadT*2+1, RadT) # Return middle frame
return Repair(c0, c.RemoveDust().Sharpen(0.1), mode=17)
""") : c

bvec1 = MAnalyse(sup, isb=true, blksize=BlkSz, overlap=OLap, divide=0, delta=1, search=5, dct=7, plevel=0,
\ chroma=Chroma, truemotion=Tm, global=Glob) # backward vectors
fvec1 = MAnalyse(sup, isb=false, blksize=BlkSz, overlap=OLap, divide=0, delta=1, search=5, dct=7, plevel=0,
\ chroma=Chroma, truemotion=Tm, global=Glob) # forward vectors
backw1 = MFlow(c1, sup_rend, bvec1)
forw1 = MFlow(c1, sup_rend, fvec1)

c2 = Defined(Denoiser) ? Interleave(backw1, c1, forw1).Eval(Denoiser) : Interleave(backw1, c1, forw1)

msk = Mask ? Eval("""
sadb = MMask(c, bvec1, ml=50, kind=1, gamma=1, thscd1=ThSAD)#.mt_binarize(15, upper=true) # mask SAD backward
sadf = MMask(c, fvec1, ml=50, kind=1, gamma=1, thscd1=ThSAD)#.mt_binarize(15, upper=true) # mask SAD forward
msad = mt_logic(sadf, sadb, "and", chroma="-127.5")#.mt_expand() # combined mask SAD
return Interleave(msad, msad, msad)
""") : Undefined

filtered = c2.DeSpot(p1=6, p2=6, pwidth=Spot, pheight=Spot, mthres=16, mwidth=Spot/2, mheight=Spot/2,
\ blur=1, motpn=true, seg=2, color=true, ranked=true, show=0, extmask=msk)

restore = c2.RemoveDust().Sharpen(0.1)

c3 = RestoreMotionBlocks(filtered, restore, alternative=restore, pthreshold=10, cthreshold=10, gmthreshold=160,
\ dist=3, dmode=2, noise=3, noisy=9, grey=false, debug=false, show=false)

SelectEvery(c3, 3, 1)

DeGrain ? MDeGrain1(last, sup_rend, bvec1, fvec1, plane=4, limit=255.0, thSad=400) : last
}


function RemoveDust(clip input, int "mode") {
mode = Default(mode, 17)

clensed = Clense(input)
rep = Repair(clensed, input, mode=mode)

RemoveGrain(rep)
}

Basically we just compare the filtered image to the original and if something too white (and large enough) was removed we take that part from the original clip.

You can adjust how bright objects need to be in order to be restored and some other minor adjustments, there are comments in the code.
The script displays the mask that is used to restore missing features, it's easy to spot the differences from that (there's a lot of stuff in the train scene).

Here's the train frame where Spotless ate a logo.
https://i.postimg.cc/pXfjBX2f/chmars-00.png

It also restores the missing balls at the beach scene.
https://i.postimg.cc/Gm9ynPFQ/chmars-01.png

[kedautinh12]

Thanks

[johnmeyer]

Zorr, that looks really good.

[chmars]

Whaaa, excellent, Thanks!
What a knowlage you all have folks!

EDIT: This time I really can mark this thread as [solved], isn't it? I wouldn't have dreamt better.

[zorr]

I was a bit surprised myself how well that turned out. :)

About that mask, I would probably go with filltimes = 2 and then add this code for the optional mask enlargement part:

inpandtimes = 6
for (i = 1, inpandtimes) {
whitemask = whitemask.mt_inpand(chroma="-127")
}
maskblur = 3
if (maskblur > 0) {
whitemask = whitemask.FastBlur(maskblur)
}

That makes the balls fit in better, also the balls have a small shadow which is darker than what is being restored so by enlarging the mask we can restore at least some of that shadow as well. The blur makes sure that we don't get any harsh edges. Here's a gif comparison new / old mask:

https://i.postimg.cc/x83FPwJQ/chmars-mask.gif

I like to compare before/after frames using Interleave, in this case it was

return StackHorizontal(Interleave(source1, overlay), Interleave(whitemask, whitemask))

There could be situations where Spotless removes a moving object which has both bright and dark areas, in that case this script will return (mostly) just the bright areas which could look funny. If you find such a problem I could take a look at how to improve the masking to handle such cases as well.

[kedautinh12]

Thank zorr, you're great

[Arx1meD]

zorr, I experimented with masks just like you. The fast moving objects restored, but the spots also returned.

[GMJCZP]

zorr, great job!

I'd like to test my Killerspots script with this video to see what improvements could be made to it.

[chmars]

It's really really cool, Zorr!

Spotless + your script is a perfect conjunction.
I tested it for 1-2 hours on footage (not yet with last blur improvement though). Impressive!

Spotless removes the dirt (middle up left), the stick and a part of arm and finger.
Zorr Script brings the stick back but not the arm+finger.
EDIT: changed pic with anim. Gif, graded for the eyes, no sharpening at all, no grain restoration.
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/2XJ5OQ8DmDmwQoG/download

EDIT: here the values used

threshold = 1.0
filltimes = 2
No inpandtimes

I tweaked the threshold until I could recover the arm without bringing back the spot...
Like every Chrismas, wonderment feeds new dreams... :o

Doing this, I noticed light grains will come back too, on the whole image. Normal. EDIT: But super nice details eg. water surface too.
EDIT: Not sure about that phenomenon. Forget this next paragraph!
What could be done though (easy to say, I couldn't) would be to define a minimum size of compensated spots. This would avoid the return of grain/noise.
Yes yes, I said "keep grain" earlyer, "I like grain"...
I do for sure. But I wander if it is good to mix the grain return from two processes. Specially if discriminated (only light grain) on one output. This makes steps of the process unclear and hard to dose/manage.

I wrote "ZorrBack" on the picture as I had to describe what was illustrated. :D
If you dislike, I won't use that name any more.


There could be situations where Spotless removes a moving object which has both bright and dark areas, in that case this script will return (mostly) just the bright areas which could look funny. If you find such a problem I could take a look at how to improve the masking to handle such cases as well.

Actually, this is also the case of this arm, a bit. Or?
Btw, thanks too for the interleave trick! Tried to make an animated gif from Vdub but...well, you know...
.

[kedautinh12]

If you find out the scripts tweaked the threshold until I could recover the arm without bringing back the spot... you can share me, thanks

[chmars]

Here a detail of the same frame as post #60.
Graded, no sharpening at all, no grain restoration.


Original frame alternating with Spotless only (left part in post #60):
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/9D74XqZr8RpbQrl/download

Spotless+Zorr alternating with Spotless only (middle part in post #60):
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/iFWFlPgWDOy0wpb/download

Original frame alternating with Spotless+Zorr (right part in post #60):
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/EnkQqYfiLdIsX1z/download


Note how details are nicely coming back into the water.
.

[chmars]

If you find out the scripts tweaked the threshold until I could recover the arm without bringing back the spot... you can share me, thanks

Not sure it's about threshold. Tried every value possible.
I didn't measure the arm's brightness but seems not as dark as the spot.
However, I couldn't separate both.

Here is the short avi file (https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/XxkOwIytdJvJDI1/download) if you wish to test.
The script above in this thread, no changes.

[zorr]

zorr, I experimented with masks just like you. The fast moving objects restored, but the spots also returned.

What kind of spots returned? If you mean bright (or brighter than average) spots then that's what the script does, it restores those. And you should not go overboard with the mask, it should only contain the moving objects and not all little spots of white, those are just mostly noise which Spotless graciously removed. :)

[zorr]

It's really really cool, Zorr!

Spotless + your script is a perfect conjunction.
I tested it for 1-2 hours on footage (not yet with last blur improvement though). Impressive!

Thanks for testing it, your animations are really helpful!

Zorr Script brings the stick back but not the arm+finger.

The arm+finger are mostly the same brightness or a little bit darker than the background. You can bring back that arm completely with threshold 1.4 but then your mask is all black and you have arrived at the original source with no noise removed. :)

I think I can fix this by looking at the color channels as well, the difference is much clearer there. I'll do some experimentation and report back. Thanks for the test video.


EDIT: here the values used

threshold = 1.0
filltimes = 2
No inpandtimes

threshold 1.0 is already pretty hardcore, take a look at the mask. :)

https://i.postimg.cc/9XTDgDRf/threshold1-0.png

What could be done though (easy to say, I couldn't) would be to define a minimum size of compensated spots. This would avoid the return of grain/noise.

The script already has that feature but I didn't explain it well. The filltimes is what determines how large spots will remain in the mask. Crank it up and you will very quickly lose the "grain". But it doesn't work with threshold 1.0 because then you pretty much have only one "spot" filling the whole frame. See the example below, this one is using threshold 0.9.

https://i.postimg.cc/28zK9Bcd/filltimes.gif

[chmars]

...
I think I can fix this by looking at the color channels as well, the difference is much clearer there. I'll do some experimentation and report back. Thanks for the test video.
...

Cool. But keep in mind I am nitpicking. Dont wanna make people loose time.
These scripts do a really great job and fulfill 98% of the task IMO. (Regrettably), no client has a chance to see that a blurred arm is missing on one frame, I believe.


threshold 1.0 is already pretty hardcore, take a look at the mask. :)

I know, yes, seen it during my tests (btw the grayscale diff is really excellent to detect/identify removed objects vs removed spots ).
I finally consider that th=1 was the good value for this scene.
It really brings back nice details, like into the water. I said "white grain", it's just more life, actually (in my taste, of course).
EDIT: example: middle zoomed pic: we can see It brings back onto the water, the reflection of the clear lines from the stone. Isn't that brilliant?


... The filltimes is what determines how large spots will remain in the mask.
... But it doesn't work with threshold 1.0 because then you pretty much have only one "spot" filling the whole frame. See the example below, this one is using threshold 0.9.

Ok, [Hit my forehead] of course, thank you for explaining more.

EDIT: Side question: Missing something: how can the variable whitemask be different than the other variable same-named whitemask?
.

[Arx1meD]

What kind of spots returned? If you mean bright (or brighter than average) spots then that's what the script does, it restores those. And you should not go overboard with the mask, it should only contain the moving objects and not all little spots of white, those are just mostly noise which Spotless graciously removed. :)

Yes. Light spots have returned.
I could not test your masks on moving dark objects. I don't have clips with moving dark objects.
Will dark moving objects be repaired?

[chmars]

Yes. Light spots have returned.
I could not test your masks on moving dark objects. I don't have clips with moving dark objects.
Will dark moving objects be repaired?

(Zorr, please correct me if I'm wrong.)
The principle of Zorr's script is to search for whatSpotless has removed. Then It can bring back everything, or only a part of it.
The criteria of decision is lightness.
The threshold sets the limit in terms of brightness, from this limit up to white, pixels will be retained to be brought back.

In addition, you can adjust the variable "filltimes" to set the size of the pixel groups you want to keep.
Then, there is "inpandtimes" which allows you to blur the brought back groups reintegrate them more discretely.

So, depending on how you set the threshold, dark objects will return. However, if they are lighter than the spots you want to remove, you have a chance.

[zorr]

It took some tinkering but the arm has been saved! :D

New script to play with below:

#source1= AviSource("459_s2_002-XAVCI-4K-BLD_5_0-trim_crop-for_tests.avi")
#source1 = crop(source1,700,0,-500,0)

source1= FFVideoSource("ZorrBack_test.avi")
source1 = crop(source1,450,50,1352,-40)

source1= KillAudio(source1)


source1_8bit = ConvertBits(source1, bits=8) #I dont'know much about formats, maybe this is not ideal...
source1YV12 = ConvertToYV12 (source1_8bit) #I dont'know much about formats, maybe this is not ideal...

txt_sz = 36

BlkSz =32
OLap = (BlkSz/2) #important not too low (eg 4 for blksz 12, because creates stairs)
Pel = 1
Tm=false
Bblur = 0.6
ThSAD = 10000
RadT = 1

Usharp_strength= 80
Usharp_radius= 5
Usharp_th = 1 #too high = too many dark pixel packs (ex trees)

#Remove spots-----------------------------------------------------------------
spotless_107=SpotLess107(source1,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
#SpotRemoved = SpotRemover(source1YV12,Spot=128, RadT=1, Pel=Pel, BlkSz=32, DeGrain=false, Chroma=true, PreFilter=true, mask=false, Tm = true,thsad=thsad, Glob = true)
#----------------------------------------------------------------------------------

#SpotRemoved = ConvertBits(SpotRemoved, bits=10) #I dont'know much about formats, maybe this is not ideal...
#SpotRemoved = ConvertToYUV422(SpotRemoved) #I dont'know much about formats, maybe this is not ideal...

#Sharpen----------------------------------------------------------------------
spotless_107_shp=spotless_107.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#SpotRemoved_shp=SpotRemoved.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#----------------------------------------------------------------------------------

# Restore grain -------------------------------------------------------------
spotless_107_shp_Rg = mRD_RestoreGrain(spotless_107_shp, source1)
#SpotRemoved_shp_Rg = mRD_RestoreGrain(SpotRemoved_shp, source1)
#----------------------------------------------------------------------------------

# Delta Restore -------------------------------------------------------------

src = source1
filtered = spotless_107_shp

# extra clipping (if needed)
#crop_left = 150
#crop_right = 250
#crop_left = 750
#crop_right = 250
#src = src.Crop(crop_left, 0, -crop_right, 0)
#filtered = filtered.Crop(crop_left, 0, -crop_right, 0)

# diff with exaggerated chroma difference
diff1 = mt_makediff(src, filtered, y=3, u=3, v=3)
diff2 = diff1.mt_adddiff(diff1, y=0, u=3, v=3)
diff2 = diff2.mt_adddiff(diff1, y=0, u=3, v=3)
diff = CombinePlanes(diff1, diff2, diff2, planes="YUV", source_planes="YUV", sample_clip=diff1)

# prepare masks --------------------------------------------------------------------

# luma mask - selects where source brightness > filtered brightness
luma_mask = LumaDeltaMask(src, filtered, ">", brightness=1.1, spotSize=3)

# chroma mask - selects where source and filtered chromas have large enough difference (formula: sqrt(u*u + v*v))
chroma_mask = ChromaDeltaMask(src, filtered, delta=3, spotSize=3)

# suppression mask - selects where source brightness < filtered brightness
suppression_mask = LumaDeltaMask(src, filtered, "<", brightness=0.9, spotSize=2)

# suppression mask removes too dark areas from chroma mask
suppressed_chroma_mask = mt_lutxy(chroma_mask, suppression_mask, expr="x y -")

# chroma override mask - selects larger color deltas and spot sizes than plain chroma mask and is not suppressed
chroma_override = ChromaDeltaMask(src, filtered, delta=5, spotSize=6)
chroma_override = ExpandMask(chroma_override, 3, blur=4)

# combine luma, chroma and chroma override masks
combined_mask = mt_logic(luma_mask, suppressed_chroma_mask, "or")
combined_mask = mt_logic(combined_mask, chroma_override, "or")

# TEST - luma and chroma override only
#combined_mask = mt_logic(luma_mask, chroma_override, "or")

# combined mask without suppression and chroma override - for testing
combined_no_suppr = mt_logic(luma_mask, chroma_mask, "or")


# restore ---------------------------------------------------------------------------
delta_restore = Overlay(filtered, src, mask=combined_mask)
restore_nosuppression = Overlay(filtered, src, mask=combined_no_suppr)


# create mask visualization (for debugging) -----------------------------------------
black = BlankClip(src)
yellow = BlankClip(src, color=$d1cc2e) # #d1cc2e f7f140
green = BlankClip(src, color=$40f75b)
gray = BlankClip(src, color=$c0c0c0)
red = BlankClip(src, color=$9e2424)
maskviz = Overlay(black, yellow, mask=chroma_mask) # chroma mask - yellow
maskviz = Overlay(maskviz, red, mask=suppression_mask, opacity=0.7) # suppression mask - red
maskviz = Overlay(maskviz, green, mask=chroma_override) # chroma override - green
maskviz = Overlay(maskviz, gray, mask=luma_mask) # luma mask - gray

comp = Overlay(delta_restore, maskviz, opacity=0.4)


stack = StackHorizontal(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(diff, "diff", txt_sz)\
)

# scaling (if needed)
stack = stack.BilinearResize(stack.width/2, stack.height/2)

return stack

inter = Interleave(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(diff, "diff", txt_sz)\
)

return inter

function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}


video = Interleave(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(overlay,"delta restore",size = txt_sz,align=2),\
subtitle(restore_nosuppression,"no suppression",size = txt_sz,align=2),\
subtitle(spotless_107_shp,"spotless_107_shp",size = txt_sz,align=2)\
)

masks = Interleave(\
subtitle(BlankClip(source1),"",size = txt_sz,align=2),\
subtitle(combined_mask,"combined_mask",size = txt_sz,align=2),\
subtitle(combined_no_suppr,"combined_no_suppr",size = txt_sz,align=2),\
subtitle(BlankClip(spotless_107_shp),"",size = txt_sz,align=2)\
)

return StackHorizontal(video, masks)

group = StackHorizontal(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(spotless_107_shp,"spotless_107_shp",size = txt_sz,align=2),\
subtitle(overlay,"delta restore",size = txt_sz,align=2),\
subtitle(combined_mask,"combined mask",size = txt_sz,align=2),\
subtitle(luma_mask,"luma delta mask",size = txt_sz,align=2),\
subtitle(suppression_mask,"suppression mask",size = txt_sz,align=2),\
subtitle(suppressed_chroma_mask,"suppressed chroma mask",size = txt_sz,align=2),\
subtitle(chroma_mask,"chroma delta mask",size = txt_sz,align=2),\
subtitle(diff,"diff",size = txt_sz,align=2))
#subtitle(chroma_mask,"chroma mask",size = txt_sz,align=2),\
#subtitle(whitemask,"whitemask",size = txt_sz,align=2),\
#subtitle(whitemask,"filled whitemask",size = txt_sz,align=2),\
#subtitle(uv,"UV",size = txt_sz,align=2))
#subtitle(whitemask_orig,"orig. whitemask",size = txt_sz,align=2),\
#subtitle(SpotRemoved,"SpotRemover",size = txt_sz,align=2))
#subtitle(spotless_107_shp_Rg,"spotless_107_shp_Rg",size = txt_sz,align=2),\

return(group)#.AddBorders(1000, 0, 500, 0, color=$000000)



#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------

function RemoveSmallSpots(clip mask, int spotSize) {
mask_orig = mask

# remove too small spots ----------------------------------------------------
for (i = 1, spotSize) {
mask = mask.mt_inpand()
}
mask = mt_hysteresis(mask, mask_orig, chroma="-127")
return mask
}

function LumaDeltaMask(clip source, clip filtered, string direction, float "brightness", int "spotSize") {
# direction is ">" if you want mask to contain areas where source is brighter than filtered by "brightness", direction "<" selects where source is darker
# adjust brightness to change how bright objects are selected, typical values between 1.0 and 1.1, larger value = brighter object
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

brightness = Default(brightness, 0.9)
spotSize = Default(spotSize, 2)

# diff
diff = Grayscale(mt_makediff(source, filtered))

# mask
mask = mt_lut(diff, expr="x range_half "+String(brightness)+" * " + direction + " 255 scalef 0 ?")

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

return mask
}

function ChromaDeltaMask(clip source, clip filtered, int "delta", int "spotSize") {

# adjust delta to change how different colors are selected
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

delta = Default(delta, 3)
spotSize = Default(spotSize, 2)

# diff ----------------------------------------------------------------------
diff = mt_makediff(source, filtered, y=3, u=3, v=3)

# extract chroma channels
u = ExtractU(diff)
v = ExtractV(diff)

# calculate deltas
u_delta = mt_lut(u, expr="x range_half - abs")
v_delta = mt_lut(v, expr="x range_half - abs")

# combine deltas
# uv_delta = mt_lutxy(u_delta, v_delta, expr="x y +")
uv_delta = mt_lutxy(u_delta, v_delta, expr="x x * y y * + 0.5 ^")

# rescale (if needed)
if (uv_delta.width < source.width || uv_delta.height < source.height) {
uv_delta = uv_delta.BilinearResize(source.width, source.height)
}

# create mask
mask = uv_delta.mt_binarize(delta)

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

# restore original format
blank = BlankClip(source)
mask = CombinePlanes(mask, blank, blank, planes="YUV", source_planes="YUV", sample_clip=source)

return mask
}

function ExpandMask(clip mask, int amount, float "blur") {
blur = Default(blur, 0)

for (i = 1, amount) {
mask = mask.mt_expand(chroma="-127")
}
if (blur > 0) {
mask = mask.FastBlur(blur)
}

return mask
}



Function mRD_RestoreGrain (clip rd, clip o, float "str1", float "str2") {

sisavs26 = !(VersionNumber() < 2.60)

str1 = string(default(str1, 10.0))
str2 = string(default(str2, 5.0))

expr = sisavs26 ? " x range_half - abs "+str1+" scalef min "+str2+" scalef x range_half - abs - min 0 max x range_half - x range_half - abs 1 max / * range_half + " : " x 128 - abs "+str1+" min "+str2+" x 128 - abs - min 0 max x 128 - x 128 - abs 1 max / * 128 + "

Return sisavs26 ? mt_makediff(o, rd).mt_lut(expr, use_expr=2).mt_adddiff(rd, chroma="copy second") : mt_makediff(o, rd).mt_lut(expr).mt_adddiff(rd, chroma="copy second")
}


Function SpotLess107(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess107: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


Explanations in the next post because of character limitations...

[zorr]

I segmented to code into a couple of functions and decided to give the whole machinery a name "Delta Restore".


LumaDeltaMask() is basically what the previous script did - compares two clips and selects mask where either clip is lighter or darker. It doesn't do the optional mask expansion and/or blur, that part has been moved to a separate function ExpandMask(). You can choose the direction of the comparison, "<" for src brightness < filtered brightness and ">" for the opposite direction. Parameter threshold is now called brightness and its value has been switched around - now larger brightness value means brighter and smaller brightness means darker (makes sense right). For example threshold 0.9 is now brightness 1.1. (conversion formula: 2.0 - threshold).
ChromaDeltaMask() has a similar idea to LumaDeltaMask but this time we look at differences in the chroma channels. There's no clear way to say "darker" or "brighter" so it just measures the difference in the two channels and combines them into one value using euclidean distance sqrt(u*u + v*v). Selection is made where distance is large enough, controlled by parameter delta.
ExpandMask() does the mask expansion and/or blur, can be called if needed after mask has been created with LumaDeltaMask() or ChromaDeltaMask().
RemoveSmallSpots() removes too small spots from the mask. It's called by both LumaDeltaMask() and ChromaDeltaMask(). The parameter filltimes has been replaced with more understandable "spotSize", the filter removes all spots which are smaller than that.


What needs to be done to save the arm is to create two masks - luma mask like before and a new chroma mask. The arm in particular is only detectable in the chroma channels. We combine these two masks with "or" operation (additive) to select pixels from the original clip where either mask has selections.

All good? It's not bad but we can do better. The chroma delta and spotSize has to be set quite high (delta=5, spotSize=6) in order not to bring back spots which also have chroma differences. We can improve things if we use a couple more masks. We create another LumaDeltaMask called "suppression mask" where source is a lot darker than filtered and remove those areas from the chroma mask. The chroma mask can now use more sensitive values (delta=3, spotSize=3) and will no longer select the dark spots. But turns out that our little suppression mask will also remove parts of the arm - it has areas where source is a lot darker. To fix that we create another ChromaDeltaMask "chroma override mask" with large delta and spotSize (just like the original chroma mask) and this one is not suppressed. Now we get smaller chroma deltas (which are not too dark) and make sure large chroma deltas are always selected even when they are darker.

A couple of examples: this one shows the arm and the different masks used to salvage it. The masks are color coded:

gray - luma mask
yellow - chroma mask (sensitive)
red - suppression mask
green - chroma override mask


https://i.postimg.cc/DyRLTR1g/delta-restore-hand.png

We can see that the arm is mostly restored by the chroma override mask. I used luma mask brightness 1.1 here but it's ok to use 1.0 which will bring even more details back.

Here's another example showing the sensitive chroma mask (yellow) in action. It has restored a small hand that was undetected by luma mask.

https://i.postimg.cc/4ymHC1dq/delta-restore-hand2.png

And finally this shows why we need a suppression mask. There's a dark spot which was detected by the sensitive chroma mask. Without the suppression mask (red overlay) the spot would have been restored.

https://i.postimg.cc/NM05Tx1F/delta-restore-suppression.png

[StainlessS]

Nifty :)

[kedautinh12]

Wow, thanks

[Arx1meD]

zorr, really great job!
Will you make your own script like post-processing after cleaning spots?

[chmars]

What a big work!
Gonna test that on a terrible new footage that made me sweat yesterday.

[chmars]

Found some hours to test on relevant footage parts, I can say: Brilliant

I tested "out of the box" as provided, in exception of:
-Spotless BlkSz = 24 (48 was too much, not useful, eating too much)
-Unsharpmask applied before spotless (less pixelisation/artefacts in eaten parts, same cleaning efficiency (different but same success average)).
As Spotless softens/degrains the image, the DeltaRestored parts are more grainy since coming from source. With unsharpmask applied, this difference tends to disapear.


Remarks:
Spotless cleans very well.
DeltaR restores most of the eaten parts:
-heads
-arms
-birds
-glass reflections...

Sometimes doesn't work on
-birds dark parts
-some rare spots (too chromatic?)

I repeat, I didn't tweak params: result is very good already. Dunno if some more hours tweaking would help, time runs...

Maybe, for the birds wings + other unrestored parts:
There is one difference between wanted parts and dirt:
Dirt is always sharp.
Unrestored parts of the images had unsharp edges.
Could this difference be used to determine if dirt/not dirt?

Here are extracts.
Sorry for those who dislike grain, lots of 1950-60 8mm to do...
I reordered the Interleave to faster locate spots on the picture.

Dirt away, leg back
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/E8EAeX9hnH5KIiE/download


Leg, hand, retored and remaining dirt...or bird?
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/3TLsI3nFO3bVgu3/download


Even birds while flying.
This seems nothing but don't miss it:
An Italian 1955 square is not itself any more without all the birds taking off at the same time.
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/yEZLqgaWh4UA8P2/download

https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/WNM9VpYGnEkud3E/download


Same for this tramway. Some reflections, you say. But these details make the image alive, bring some kind of beauty. We can see it well in the original movie.
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/rrsf38O5VAlpgjq/download


Here is a partially restored fronthead. Only the bright part is back, not the hair.
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/04qLjGPNefrqsX1/download


Here, a spot is restored.
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/VsNZMrS4ahSGy17/download


Links:
source (https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/4VMlbuB16GpZVko/download),
full scenes comparison result (https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/zJgYDe92nbYwwjG/download).

(realizing I mixed up the order of the frames. Give me a simple trick, (Linux), to make webm from tiff, avi, whatever and I'll do better next time ;))

[johnmeyer]

I have a small film transfer project coming up next week and am eager to try this new approach to dirt removal. I'll report on my results when I'm finished.

If it works, it will save me a huge amount of time when cleaning the old football films that a collector sends to me. Because of RemoveDirtMC's tendency to remove footballs while in flight, I've had to do a second restoration, but without dirt removal, and then cut to that during pass, punts, and kicks, while the ball was in the air. This could therefore save me perhaps as much as an hour for each of those projects, and also result in a much better final result since all the frames will be cleaned.

[chmars]

Howevwer, I noticed it is still good to run two scripts to prepare the grading in the NLE.
To put one underneath the other in the timeline.
Both with same crop, trim, everything, but one with [sharpening+Spotless+DeltaRestoreOne] and one without.

In the case of the man with missing hair (pic6), it becomes easy to just delete a part of the head to let appear the original from the clip laying underneath.

But when DeltaRestore will discriminate between sharp and unsharp dark removed "spots", this will become unusefull .:D
Anyways, with or without, thanks again for these scripts, you are AviSaints!

[zorr]

-Spotless BlkSz = 24 (48 was too much, not useful, eating too much)
-Unsharpmask applied before spotless


I applied these changes to the latest script.

There is one difference between wanted parts and dirt:
Dirt is always sharp.
Unrestored parts of the images had unsharp edges.
Could this difference be used to determine if dirt/not dirt?

Challenge accepted. :D

#source1= AviSource("459_s2_002-XAVCI-4K-BLD_5_0-trim_crop-for_tests.avi")
#source1 = crop(source1,700,0,-500,0)

#source1= FFVideoSource("ZorrBack_test.avi")
#source1 = crop(source1,450,50,1352,-40)

source1= FFVideoSource("464_-trim_crop_grd_cut.avi")


source1= KillAudio(source1)


source1_8bit = ConvertBits(source1, bits=8) #I dont'know much about formats, maybe this is not ideal...
source1YV12 = ConvertToYV12 (source1_8bit) #I dont'know much about formats, maybe this is not ideal...

txt_sz = 36

BlkSz = 24
OLap = (BlkSz/2) #important not too low (eg 4 for blksz 12, because creates stairs)
Pel = 1
Tm=false
Bblur = 0.6
ThSAD = 10000
RadT = 1

Usharp_strength= 80
Usharp_radius= 5
Usharp_th = 1 #too high = too many dark pixel packs (ex trees)


#Sharpen----------------------------------------------------------------------
spotless_107_shp=source1.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#SpotRemoved_shp=SpotRemoved.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#----------------------------------------------------------------------------------

#Remove spots-----------------------------------------------------------------
spotless_107=SpotLess107(spotless_107_shp,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
#SpotRemoved = SpotRemover(source1YV12,Spot=128, RadT=1, Pel=Pel, BlkSz=32, DeGrain=false, Chroma=true, PreFilter=true, mask=false, Tm = true,thsad=thsad, Glob = true)
#----------------------------------------------------------------------------------


#SpotRemoved = ConvertBits(SpotRemoved, bits=10) #I dont'know much about formats, maybe this is not ideal...
#SpotRemoved = ConvertToYUV422(SpotRemoved) #I dont'know much about formats, maybe this is not ideal...


# Restore grain -------------------------------------------------------------
#spotless_107_shp_Rg = mRD_RestoreGrain(spotless_107_shp, source1)
#SpotRemoved_shp_Rg = mRD_RestoreGrain(SpotRemoved_shp, source1)
#----------------------------------------------------------------------------------

# Delta Restore -------------------------------------------------------------

src = source1
filtered = spotless_107

# extra clipping (if needed)
#crop_left = 150
#crop_right = 250
#crop_left = 750
#crop_right = 250
#crop_left = 30
#crop_right = 1070
#src = src.Crop(crop_left, 0, -crop_right, 0)
#filtered = filtered.Crop(crop_left, 0, -crop_right, 0)

# diff with exaggerated chroma difference
diff1 = mt_makediff(src, filtered, y=3, u=3, v=3)
diff2 = diff1.mt_adddiff(diff1, y=0, u=3, v=3)
diff2 = diff2.mt_adddiff(diff1, y=0, u=3, v=3)
diff = CombinePlanes(diff1, diff2, diff2, planes="YUV", source_planes="YUV", sample_clip=diff1)

# prepare masks --------------------------------------------------------------------

# luma mask - selects where source brightness > filtered brightness
luma_mask = LumaDeltaMask(src, filtered, ">", brightness=1.1, spotSize=3)

# chroma mask - selects where source and filtered chromas have large enough difference (formula: sqrt(u*u + v*v))
chroma_mask = ChromaDeltaMask(src, filtered, delta=3, spotSize=3)

# suppression mask - selects where source brightness < filtered brightness
suppression_mask = LumaDeltaMask(src, filtered, "<", brightness=0.9, spotSize=2)

# suppression mask removes too dark areas from chroma mask
suppressed_chroma_mask = mt_lutxy(chroma_mask, suppression_mask, expr="x y -")

# edge mask - detects sharp edges
edgeMask = scharr(src).mt_binarize(52)
#edgeMask = src.vsTEdgeMask(threshY=51, type=5, link=0, y=3, u=2, v=2)
#return edgeMask


# TODO: don't apply edge mask at the edges of the frame (to fix cases like frame 276)


# chroma override mask - selects larger color deltas and spot sizes than plain chroma mask and is not suppressed
chroma_override = ChromaDeltaMask(src, filtered, delta=5, spotSize=6) # orig. spotSize 6

# NOTE: using blur causes weird bug (example: frame 147)
#chroma_override = ExpandMask(chroma_override, 3, blur=4)


# combine luma, chroma and chroma override masks
combined_mask = mt_logic(luma_mask, suppressed_chroma_mask, "or")
combined_mask = mt_logic(combined_mask, chroma_override, "or")

# remove mask areas where sharp edges were found (those are most likely dirt)
edgeAreaMask = mt_hysteresis(edgeMask, combined_mask, chroma="-127")
combined_mask = mt_lutxy(combined_mask, edgeAreaMask, expr="x y -")

# TEST - luma and chroma override only
#combined_mask = mt_logic(luma_mask, chroma_override, "or")

# combined mask without suppression and chroma override - for testing
combined_no_suppr = mt_logic(luma_mask, chroma_mask, "or")


# restore ---------------------------------------------------------------------------
delta_restore = Overlay(filtered, src, mask=combined_mask)
restore_nosuppression = Overlay(filtered, src, mask=combined_no_suppr)



# create mask visualization (for debugging) -----------------------------------------
black = BlankClip(src)
yellow = BlankClip(src, color=$d1cc2e) # #d1cc2e f7f140
green = BlankClip(src, color=$40f75b)
gray = BlankClip(src, color=$c0c0c0)
red = BlankClip(src, color=$9e2424)
blue = BlankClip(src, color=$0d3b7a) #24579e 243c9e
maskviz = Overlay(black, yellow, mask=chroma_mask) # chroma mask - yellow
maskviz = Overlay(maskviz, red, mask=suppression_mask, opacity=0.7) # suppression mask - red
maskviz = Overlay(maskviz, green, mask=chroma_override) # chroma override - green
maskviz = Overlay(maskviz, gray, mask=luma_mask) # luma mask - gray
maskviz = Overlay(maskviz, blue, mask=edgeAreaMask, opacity=0.8) # sharpness mask - blue

comp = Overlay(delta_restore, maskviz, opacity=0.4)


#return StackHorizontal(src, filtered, delta_restore, edgeMask, maskviz)


stack = StackHorizontal(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(edgeMask, "edge mask", txt_sz),\
sub(diff, "diff", txt_sz)\
)

# scaling (if needed)
stack = stack.BilinearResize(stack.width/2 + 1, stack.height/2)

return stack

inter = Interleave(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(diff, "diff", txt_sz)\
)

return inter

function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}


video = Interleave(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(overlay,"delta restore",size = txt_sz,align=2),\
subtitle(restore_nosuppression,"no suppression",size = txt_sz,align=2),\
subtitle(spotless_107_shp,"spotless_107_shp",size = txt_sz,align=2)\
)

masks = Interleave(\
subtitle(BlankClip(source1),"",size = txt_sz,align=2),\
subtitle(combined_mask,"combined_mask",size = txt_sz,align=2),\
subtitle(combined_no_suppr,"combined_no_suppr",size = txt_sz,align=2),\
subtitle(BlankClip(spotless_107_shp),"",size = txt_sz,align=2)\
)

return StackHorizontal(video, masks)

group = StackHorizontal(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(spotless_107_shp,"spotless_107_shp",size = txt_sz,align=2),\
subtitle(overlay,"delta restore",size = txt_sz,align=2),\
subtitle(combined_mask,"combined mask",size = txt_sz,align=2),\
subtitle(luma_mask,"luma delta mask",size = txt_sz,align=2),\
subtitle(suppression_mask,"suppression mask",size = txt_sz,align=2),\
subtitle(suppressed_chroma_mask,"suppressed chroma mask",size = txt_sz,align=2),\
subtitle(chroma_mask,"chroma delta mask",size = txt_sz,align=2),\
subtitle(diff,"diff",size = txt_sz,align=2))
#subtitle(chroma_mask,"chroma mask",size = txt_sz,align=2),\
#subtitle(whitemask,"whitemask",size = txt_sz,align=2),\
#subtitle(whitemask,"filled whitemask",size = txt_sz,align=2),\
#subtitle(uv,"UV",size = txt_sz,align=2))
#subtitle(whitemask_orig,"orig. whitemask",size = txt_sz,align=2),\
#subtitle(SpotRemoved,"SpotRemover",size = txt_sz,align=2))
#subtitle(spotless_107_shp_Rg,"spotless_107_shp_Rg",size = txt_sz,align=2),\

return(group)#.AddBorders(1000, 0, 500, 0, color=$000000)



#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------

function scharr(clip c) {
scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), chroma="-127")
return scharr
}

function RemoveSmallSpots(clip mask, int spotSize) {
mask_orig = mask

# remove too small spots ----------------------------------------------------
for (i = 1, spotSize) {
mask = mask.mt_inpand()
}
mask = mt_hysteresis(mask, mask_orig, chroma="-127")
return mask
}

function LumaDeltaMask(clip source, clip filtered, string direction, float "brightness", int "spotSize") {
# direction is ">" if you want mask to contain areas where source is brighter than filtered by "brightness", direction "<" selects where source is darker
# adjust brightness to change how bright objects are selected, typical values between 1.0 and 1.1, larger value = brighter object
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

brightness = Default(brightness, 0.9)
spotSize = Default(spotSize, 2)

# diff
diff = Grayscale(mt_makediff(source, filtered))

# mask
mask = mt_lut(diff, expr="x range_half "+String(brightness)+" * " + direction + " 255 scalef 0 ?")

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

return mask
}

function ChromaDeltaMask(clip source, clip filtered, int "delta", int "spotSize") {

# adjust delta to change how different colors are selected
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

delta = Default(delta, 3)
spotSize = Default(spotSize, 2)

# diff ----------------------------------------------------------------------
diff = mt_makediff(source, filtered, y=3, u=3, v=3)

# extract chroma channels
u = ExtractU(diff)
v = ExtractV(diff)

# calculate deltas
u_delta = mt_lut(u, expr="x range_half - abs")
v_delta = mt_lut(v, expr="x range_half - abs")

# combine deltas
# uv_delta = mt_lutxy(u_delta, v_delta, expr="x y +")
uv_delta = mt_lutxy(u_delta, v_delta, expr="x x * y y * + 0.5 ^")

# rescale (if needed)
if (uv_delta.width < source.width || uv_delta.height < source.height) {
uv_delta = uv_delta.BilinearResize(source.width, source.height)
}

# create mask
mask = uv_delta.mt_binarize(delta)

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

# restore original format
blank = BlankClip(source)
mask = CombinePlanes(mask, blank, blank, planes="YUV", source_planes="YUV", sample_clip=source)

return mask
}

function ExpandMask(clip mask, int amount, float "blur") {
blur = Default(blur, 0)

for (i = 1, amount) {
mask = mask.mt_expand(chroma="-127")
}
if (blur > 0) {
mask = mask.FastBlur(blur)
}

return mask
}



Function mRD_RestoreGrain (clip rd, clip o, float "str1", float "str2") {

sisavs26 = !(VersionNumber() < 2.60)

str1 = string(default(str1, 10.0))
str2 = string(default(str2, 5.0))

expr = sisavs26 ? " x range_half - abs "+str1+" scalef min "+str2+" scalef x range_half - abs - min 0 max x range_half - x range_half - abs 1 max / * range_half + " : " x 128 - abs "+str1+" min "+str2+" x 128 - abs - min 0 max x 128 - x 128 - abs 1 max / * 128 + "

Return sisavs26 ? mt_makediff(o, rd).mt_lut(expr, use_expr=2).mt_adddiff(rd, chroma="copy second") : mt_makediff(o, rd).mt_lut(expr).mt_adddiff(rd, chroma="copy second")
}


Function SpotLess107(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess107: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

Details in the next message.

[zorr]

This version has a new mask type - edge mask. It's a standard edge detection combined with my favourite MaskTools filter mt_hysteresis. Basically if there are sharp edges anywhere within an isolated mask area then those areas will be cut out so that part will be taken from the filtered version.

I tested a couple of different edge detection filters, two different modes of a recent vsTEdgeMask and my own Scharr edge detector. I think the former is faster but I didn't make measurements.

What I did measure was the proper threshold to use in the edge mask, how hard edges should be considered to be a spot and not some real feature. It's a surprisingly specific value and for the latest test video only Scharr managed to handle all the test cases. The most problematic frames are 9 (shoe should be restored), 155 (highlight should be restored), 190 (large spot should be removed) and 204 (bird should be restored). There's also frame 276 which cannot be fixed with the current version but is fixable. Scharr handles all cases when binarized with threshold 52. vsTEdgeMask type 5 was close, the bird needed threshY > 51 but the spot at frame 190 needs threshY <= 51. Type 3 was also close with threshY 29.

Some test notes below:

# using vsTEdgeMask type 2: threshY min 30 - max 29
# frame 9 - shoe with threshY <= 27 -> threshY > 27
# frame 20 - small dot with threshY <= 30
# frame 90 - medium spot with threshY <= 31
# frame 121 - large spot with threshY <= 48
# frame 132 - small spot with threshY <= 46
# frame 147 - large spot with threshY <= 49
# frame 155 - a highlight with threshY <= 25 -> threshY > 25
# frame 183 - a large spot with threshY <= 37
# frame 190 - a large spot with threshY <= 29
# frame 204 - bird with threshY <= 29 -> threshY > 29
# frame 229 - a spot with threshY <= 62
# frame 241 - small spot with threshY <= 56
# frame 260 - small spot with threshY <= 42
# frame 265 - medium dark spot with threshY <= 55
# frame 276 - large chroma delta at the bottom with threshY <= 42 -> threshY > 42

# using vsTEdgeMask type 5: threshY min 52 - max 51
# frame 9 - shoe with threshY <= 48 -> threshY > 48
# frame 155 - a highlight with threshY <= 46 -> threshY > 46
# frame 190 - a large spot with threshY <= 51
# frame 204 - bird with threshY <= 51 -> threshY > 51
# frame 276 - large chroma delta at the bottom with threshY <= 71 -> threshY > 71

# using scharr: binarize min 52 - max 52
# frame 9 - shoe with binarize <= 48 -> threshY > 48
# frame 155 - a highlight with binarize <= 46 -> threshY > 46
# frame 190 - a large spot with binarize <= 52
# frame 204 - bird with binarize <= 51 -> threshY > 51
# frame 276 - large chroma delta at the bottom with binarize <= 72 -> threshY > 72

Let's look at some example frames:

Here's a large spot which was restored earlier. The mask visualization now includes the edge mask (or rather the hysterized area grown from the edge mask) as blue (luckily there were still some colors left). Whenever that blue color appears it means edge mask was used to cut out a feature which would have been restored from the original clip without the edge mask.

https://i.postimg.cc/FKM4nKt3/delta-restore-edge-mask-1.png

Here's another one. In this one we can also see that some part of a wing was not restored.

https://i.postimg.cc/zBTrm15M/delta-restore-edge-mask-2.png

I didn't yet try to restore the dark areas which would fix this wing and the guy's missing hair. Perhaps we don't need as many masks if we can rely on the edge mask.

There are many blue areas around the frame edges, that's because those contain sharp... edges. :) The problem with frame 276 is caused by that. The edge mask should not be applied near the edges. I'll try that in the next version.

I took out the blur (and expansion) of the chroma override mask, it looks like there's a bug in FastBlur which causes a weird glitch in the edge mask (try to enable it and look at frame 147).

[patul]

Hi zorr,

Great work there. I'm experimenting with your script, while I don't have old material to begin with, hence I used this (https://www.youtube.com/watch?v=eB91_OHB5Rg) video as a sample, which probably is not an ideal sample because it might have been through some heavy processing prior upload, but I use it anyway.

I noticed some dirt is not removed by SpotLess, and some dirt reappear on Delta Restore. Is there any hints that you can provide please? Thanks in advance

[chmars]

Seems challenge brightly fullfilled ;)
Just copied it, testing.
EDIT: Your explanations are vital, thanks for this effort too.

Side note, my error:
Minor importance but introduces confusion.
Names are wrong, it reflected as if a not sharpened version was used.

spotless_107_shp=source1.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
Should be
source1_shp=source1.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
.

And then,
spotless_107=SpotLess107(spotless_107_shp,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
should bespotless_107=SpotLess107(source1_shp,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)

[chmars]

Can confirm: returning spots are away!
Lots of steps... Added thread summary in post #1 to recap.


I didn't yet try to restore the dark areas which would fix this wing and the guy's missing hair. Perhaps we don't need as many masks if we can rely on the edge mask.

I was looking... maybe by combining suppression_mask and edgeMask?

There are many blue areas around the frame edges, that's because those contain sharp... edges. :) The problem with frame 276 is caused by that. The edge mask should not be applied near the edges. I'll try that in the next version.

I plan to fill the perforations white holes with black anyway.

Going to test it all on a new 15meter footage, before grading with Resolve (Slog3, very bright-uncontrasted). Will see...
I prefer to do it before grading. So I can verify all at once, avoiding having to look a second time to the whole movie.
Maybe, I'll add some functions at script end to simulate grading, so I can check if it works that way.

EDIT: As expected, Spotless seems ok but Data Restore works less good on ungraded Slog3.
So I will kind of grade the source with Autolevels to build the masks, then apply the masks on spotlessed original + original.

[zorr]

zorr, really great job!
Will you make your own script like post-processing after cleaning spots?

No, unless I come up with something worth contributing. I haven't thought about post-processing much (and that's quite a wide definition). Did you have something specific in mind?

[zorr]

I have a small film transfer project coming up next week and am eager to try this new approach to dirt removal. I'll report on my results when I'm finished.

That would be interesting, I'd love hear how it goes. Perhaps it needs some adjustment or even a new approach. The method is quite general in principle so I think you have better than 50% chance of having useful results. And if not then we can learn something new.

[zorr]

I used this (https://www.youtube.com/watch?v=eB91_OHB5Rg) video
...
I noticed some dirt is not removed by SpotLess, and some dirt reappear on Delta Restore. Is there any hints that you can provide please? Thanks in advance

I took a quick look at the video. It has many duplicate/repeating frames and noise removal algorithms don't work well with those. So the first step should be getting rid of the duplicate frames, I believe the correct term for that is decimation. There are people on this forum who are much more knowledgeable than me on that subject so I can't give you detailed instructions.

I didn't try Delta Restore on your clip but it seems to have softer look, even the dirt spots are much softer than the material chmars has shown so the edge mask is probably not going to work very well on that. You may have to adjust the suppression mask and/or the chroma override mask to prevent dark spots from reappearing.

[zorr]

Names are wrong, it reflected as if a not sharpened version was used.

Thanks, I corrected the error.

I've worked on the "dark side" and it's almost finished but it's getting late so I will write a comprehensive report tomorrow.

Meanwhile have you considered removing the vertical streaks? Spotless can't touch them because they last many consecutive frames but there are other tools. I fiddled a bit with DeScratch (http://avisynth.nl/index.php/DeScratch) and got some promising results.

https://i.postimg.cc/85brv8Rv/descratch.gif

Here it was applied after Spotless but I think it might work even better before it. If DeScratch leaves behind a lone streak but manages to remove it from the previous and next frame then Spotless can remove the remaining one.

[kedautinh12]

Thanks, Waiting for result

[chmars]

...
I think you have better than 50% chance of having useful results. And if not then we can learn something new.
...

I'say 90%- > Used it yesterday on see/boat/coast 5mn 8mm footage.
Very efficient, even on Slog washed out source (pseudo-graded to feed Sl (Spotless) +Dr (DeltaRestore)).
Wave splashing water and hands removed by Sl were restored well by Dr.

...
I've worked on the "dark side" and it's almost finished but it's getting late so I will write a comprehensive report tomorrow.

Cool. What do you think of writing Dr as a function with version number as like Sl?
So we could put the whole cleaned script in post #1. It would also be easier in cases of updates, I guess.
I'll add the pseudo-grading thing for washed out footages too.


Meanwhile have you considered removing the vertical streaks? Spotless can't touch them because they last many consecutive frames but there are other tools. I fiddled a bit with DeScratch (http://avisynth.nl/index.php/DeScratch) and got some promising results.

Yes, was thinking of it. Just discovered Avisynth - which is beautiful but hurts a bit sometimes - short ago. My head is not very good at multutasking :). I'll try it then could add it to the general script in #1, btw.

[Arx1meD]

No, unless I come up with something worth contributing. I haven't thought about post-processing much (and that's quite a wide definition). Did you have something specific in mind?I mean to combine all your work (RemoveSmallSpots, LumaDeltaMask, ChromaDeltaMask, ExpandMask) into one script. For example, something similar to the Repair (http://avisynth.nl/index.php/RgTools/Repair) filter in RgTools.

[chmars]

Here is the updated - cleaned script.
The script is not functional yet - Don't try it.
Zorr, here is the idea of using a levelled clip for Spotless and Delta Restore and then apply masks to the original clip when necessary (Slog).

I'll pm you the DR I turned into a function.
This Avoids confusions between versions and mostly, it's yours, dunno if it suits you, I don't wanna "walk on your grass".
But as it was some work to transpose variables, check the whole thing..., it could help perhaps.

Part 1
/*
SpoDel (Spotless+Delta Restore)

Script intended to clean and sharpen digitalized film.
Can optionally restore grain after cleaning.
Sharpening and denoising could be hardly improved. Please see VideoFred's and JohnMeyer's scripts and posts fore more knowlege. (VideoFred's sharp-blur-sharp-blur-sharp trick). Please share emprovements!

Functions: StainlessS, Zorr, real.finder
Script: chmars - 06/06/2021 - some parts copied here and there, thank to solutions given by doom9 forum users.
Thread: https://forum.doom9.org/showthread.php?t=182831
Spotless thread, last versions: https://forum.doom9.org/showthread.php?t=181777
Lots of informaions on film capture/restoration: https://forum.doom9.org/showthread.php?t=144271

Script tested on HD 4:2:2 10bits sources with Avisynth+370_x64

How it works:
Spotless removes dirt spots that are not present in neibourg frames.
But sometimes, fast moving objects are removed too (balloons, birds, hands, feet)
Hence is Delta Restore needed. It brings back details (but not spots) that are part of the image.
As Spotless tends to soften/degrain the image, parts brought back from the original image by Delta Restore are more grainy
Solution for a discreete replacement of these parts are either to sharpen the Spotless input, or degrain Delta Restore output, before to apply it to Spotless output.
Washed out, Slog rushes:
Spotless seems to work as well with washed out images (old footage, Slog rushes), but Delta Restore needs colors and contrast to funtion properly.
In this case, a more contrasted clip is generated to feed Delta Restore and generate masks only. But final result will not be graded.

*/



SetMemoryMax(8000) # Dunno if optimal...
threads = 16 # Dunno if optimal...
### Check speed, memory consumption and CPU usage with AVSMeter64 ###
SetFilterMTMode("DEFAULT_MT_MODE", 2) # Dunno if optimal...

# VARIABLES #################################################################################################
########################################################################################################################
########################################################################################################################
# Here, define what you want

# Path to file
video_file = "X:\some_path\some_file"

play_speed = 25 # This will not change your frames/fields, neither their count
txt_sz = 36 #Test size if infos displayed
Sharpen_it = True # Default True. Sharpen clip or not (recomanded, see intro)
Slog = True # Default Falde. If True, Generate a leveled clip if or wahed-out or Slog rushes

#Grainitude, cancels sharpen
Rgr = False # Set True Restores grain after passed through Spotless. Might restore some dust too. 5,10 = good compromise grain/dirt. Dunno how that works.
val1 = 10
val2 = 20 # 5,10 = good compromise grain/dirt. Dunno how that works exactly...

# Sharpen variables ----------------------------------------------------------------------------------------------------------------------------------------------------------
Usharp_strength= 80
Usharp_radius= 5
Usharp_th = 1 #too high = too many dark pixel packs (ex trees)


# Spotless variables, to remove dirt and spots. -------------------------------------------------------------------------------------------------------------------------
# See Function script for more advanced settings
BlkSz =24 #Spot Size, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]
Pel = 1 # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)
Tm=false #TrueMotion Default True. Some folk swear truemotion=false is better.
Bblur = 0.6 #Default 0.6. If used, Suggest about 0.6, where MAnalyse create vectors is performed on denoised (blurred) super clip for better motion analysis.
ThSAD = 10000 # [SAD, Sum of Absolute (pixelwise) Differences] Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1)
RadT = 1 #Removes Spots on up to RadT [Temporal Radius] consecutive frames. RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
#Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

#Delta Restore variables, restore removed parts
LuDeMaDir = ">" # ">" Removed parts to restore are: "<"=brighter or ">"=darker
LuDeMaBr = 1.1 # 1.1 LumaDeltaMask Brightness
LuDeMaSz = 3 # 3 LumaDeltaMask SpotSize
ChroDeMaDe = 3 # 3 ChromaDeltaMask Delta
ChroDeMaSz = 3 # 3 ChromaDeltaMask SpotSize
SLuDeMaDir = "<" # "<"
SupLuDeMaBr = 0.9 # 0.9 Suppression LumaDeltaMask brightness threshold
SupLuDeMaSz = 2 #2 Suppression LumaDeltaMask SpotSize
Ov_ChroDeMaDe = 5 # 5 Override ChromaDeltaMask Delta # chroma override mask - selects larger color deltas and spot sizes than plain chroma mask and is not suppressed
Ov_ChroDeMaSz = 6 # 6 Override ChromaDeltaMask SpotSize
EdMaShp = 52 #52 bigger = sharper


# Levels correction -------------------------------------------------------------------------------------------------------------------------------------------------------------
#In case of washed-out footage or Slog captures
# Needed to generate dark and saturated enough clips, to feed the filters whith what they need.
# Will NOT apply to final result
gamma = 1.0
input_low = 80 # What level is considered as black in input
input_high = 240 # Same for white
output_low = 0 # Minimum level wanted for output black pixels
output_high = 255 # Same for white

#Denoise----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# If needed, removes grain
#Not deeply tested, please test and report
#~ denoising_strength= 200 #denoising level of first denoiser: MDegrain() # high reduces colors
#~ block_size= 4 #block size of MVDegrain
#~ block_size_v= 4
#~ block_over= 2 #block overlapping of MVDegrainMulti()
#~ dirt_strength=10 #sets amount of dirt removal (big spots)


# / END OF VARIABLES ######################################################################################################
########################################################################################################################
########################################################################################################################

[chmars]

Part 2 of 2


source1 = FFVideoSource(vid_file)
# Set play speed (if needed)
source1 = assumefps(source1,play_speed)


# Optional
## ------------------------------------------------------------------------------------
# Keep only from frame n to frame m
# source1 = trim(source1, 2,0)
# Resize (speeds-up for tests?)
# source1 = source1.BilinearResize(source.width/2 , source.height/2)

# Leveled clip
## ------------------------------------------------------------------------------------
# If Slog = True,
# Make a decently contrasted clip to feed the filters.
# Else, do nothing

#EDIT:
#The next line was crap.
#lvl_src = Slog ? Autolevels(source1, sceneChgThresh=255, gamma = gamma, midpoint = 0.5, input_low = input_low, input_high = input_high, output_low = output_low, output_high = output_high) : source1
#Has to be replaced by, at least:
lvl_src = Slog ? source1.convertbits(8).Autolevels(filterradius = 3, sceneChgThresh = 20, autogamma = true, midpoint = 0.5, autolevel = true, border=500, input_high = 230 ).convertbits(16) : source1
#/EDIT END

# If needed, verify levels
h = Histogram(source1,"classic")
h_lvl = Slog ? Histogram(lvl_src,"classic") : NOP
#~ return(StackHorizontal(h, h_lvl)) # Un comment to see clips with histograms

#Sharpening
# -----------------------------------------------------------------------------------
# Best results found by sharpening before (not after) cleaning.
source1_shp = Sharpen_it ? source1.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th) : source1
# If Using Slog rushes, also sharpen the leveled clip
source1_lvl_shp = Sharpen_it ? lvl_src.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th) : source1_shp

#Remove spots, Spotless, version 1.07
# ------------------------------------------------------------------------------------
filtered_orig = SpotLess(source1_shp,blksz=BlkSz,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
lvl_filtered = Slog ? SpotLess(source1_lvl_shp,blksz=BlkSz,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT) : filtered_orig
#~ return(filtered_orig)
# Restore removed objects seen as dirt
# ------------------------------------------------------------------------------------
restored = DeltaRestore(


#To see if Delta Restore becomes a function or not


# Restore grain - cancels sharpen
#------------------------------------------------------------------------------------
restored_2 = Rgr ? mRD_RestoreGrain(restored, source1,val1, val2) : restored #5,10 = good compromise grain/dirt

#~ return(Interleave(source1, restored_2, restored))
return(restored_2)

###############################################################################################################################
###############################################################################################################################
###############################################################################################################################
###############################################################################################################################
# Delta Restore -------------------------------------------------------------

...

# Delta Restore sub-functions
###############################################################################################################################
###############################################################################################################################

function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}

function scharr(clip c) {
scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), chroma="-127")
return scharr
}

function RemoveSmallSpots(clip mask, int spotSize) {
mask_orig = mask

# remove too small spots ----------------------------------------------------
for (i = 1, spotSize) {
mask = mask.mt_inpand()
}
mask = mt_hysteresis(mask, mask_orig, chroma="-127")
return mask
}

function LumaDeltaMask(clip source, clip filtered, string direction, float "brightness", int "spotSize") {
# direction is ">" if you want mask to contain areas where source is brighter than filtered by "brightness", direction "<" selects where source is darker
# adjust brightness to change how bright objects are selected, typical values between 1.0 and 1.1, larger value = brighter object
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

brightness = Default(brightness, 0.9)
spotSize = Default(spotSize, 2)

# diff
diff = Grayscale(mt_makediff(source, filtered))

# mask
mask = mt_lut(diff, expr="x range_half "+String(brightness)+" * " + direction + " 255 scalef 0 ?")

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

return mask
}

function ChromaDeltaMask(clip source, clip filtered, int "delta", int "spotSize") {

# adjust delta to change how different colors are selected
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

delta = Default(delta, 3)
spotSize = Default(spotSize, 2)

# diff ----------------------------------------------------------------------
diff = mt_makediff(source, filtered, y=3, u=3, v=3)

# extract chroma channels
u = ExtractU(diff)
v = ExtractV(diff)

# calculate deltas
u_delta = mt_lut(u, expr="x range_half - abs")
v_delta = mt_lut(v, expr="x range_half - abs")

# combine deltas
# uv_delta = mt_lutxy(u_delta, v_delta, expr="x y +")
uv_delta = mt_lutxy(u_delta, v_delta, expr="x x * y y * + 0.5 ^")

# rescale (if needed)
if (uv_delta.width < source.width || uv_delta.height < source.height) {
uv_delta = uv_delta.BilinearResize(source.width, source.height)
}

# create mask
mask = uv_delta.mt_binarize(delta)

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

# restore original format
blank = BlankClip(source)
mask = CombinePlanes(mask, blank, blank, planes="YUV", source_planes="YUV", sample_clip=source)

return mask
}

function ExpandMask(clip mask, int amount, float "blur") {
blur = Default(blur, 0)

for (i = 1, amount) {
mask = mask.mt_expand(chroma="-127")
}
if (blur > 0) {
mask = mask.FastBlur(blur)
}

return mask
}


# RestoreGrain Function
###############################################################################################################################
###############################################################################################################################

Function mRD_RestoreGrain (clip rd, clip o, float "str1", float "str2") { #https://forum.doom9.org/showthread.php?p=1943371#post1943371

sisavs26 = !(VersionNumber() < 2.60)

str1 = string(default(str1, 10.0))
str2 = string(default(str2, 20.0))

expr = sisavs26 ? " x range_half - abs "+str1+" scalef min "+str2+" scalef x range_half - abs - min 0 max x range_half - x range_half - abs 1 max / * range_half + " : " x 128 - abs "+str1+" min "+str2+" x 128 - abs - min 0 max x 128 - x 128 - abs 1 max / * 128 + "

Return sisavs26 ? mt_makediff(o, rd).mt_lut(expr, use_expr=2).mt_adddiff(rd, chroma="copy second") : mt_makediff(o, rd).mt_lut(expr).mt_adddiff(rd, chroma="copy second")
}


# Spotless Function
###############################################################################################################################
###############################################################################################################################
# Version 1.07
Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess107: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

Prefetch(threads)

[chmars]

Little off topic: I didn't find a way to automate light levels. Is that so hard to do? So, I used autolevels in "manual mode"...

EDIT: I might have been tired... and just didn't RTFM
Autolevels line use should be, at least:
lvl_src = Slog ? source1.convertbits(8).Autolevels(filterradius = 3, sceneChgThresh = 20, autogamma = true, midpoint = 0.5, autolevel = true, border=500, input_high = 230 ).convertbits(16) : source1
As it needs 8bits and needs to be told to use autogamma and autolevels.
On my source, borders contain always darker and lighter parts than the image. Therefore border=500 has to be used to not calculate these areas.

[zorr]

What do you think of writing Dr as a function with version number as like Sl?

I mean to combine all your work (RemoveSmallSpots, LumaDeltaMask, ChromaDeltaMask, ExpandMask) into one script.

Yes that's a good idea and I will reform the script into one or several functions. The sub-functions can be useful as is but the "main" code needs to be wrapped into something as well. I think it may be more than one function (maybe several variations, each a bit more extensive) to keep the number of parameters sensible.

I'll add the pseudo-grading thing for washed out footages too.

Easiest way is to have one function return the final mask and then other to apply it, then you can use a different clip to create the masks.

[zorr]

WORK IN PROGRESS - dark mask

I have implemented another mask variation - the dark mask (spooky!). It's not working very well at this point as it seems to be impossible to detect the correct dark areas to restore without also restoring the spots here and there. At least with the current approach which uses edge sharpness, spot size and brightness criterias.

I thought it would help to include a limit to the brightness (parameter limitBrightness in LumaDeltaMask), for example to select all areas where brightness is < 0.97 but only if most of the brightness is > 0.66. It works for some of the spots but in order to restore the guy's hair the limit has to be set too low to be effective. I might remove this parameter later.

The blurring functionality is back - I switched to vcmohan's GBlur.

Also edge detection can now be restricted to areas not close the the frame borders, there's function Padding() for that. I had to set the left padding quite high, perhaps a better implementation is needed (or some preprocessing to remove sharp features near the borders).

Latest script here (didn't implement as a function yet or include chmars' latest changes, sorry)

#source1= AviSource("459_s2_002-XAVCI-4K-BLD_5_0-trim_crop-for_tests.avi")
#source1 = crop(source1,700,0,-500,0)

#source1= FFVideoSource("ZorrBack_test.avi")
#source1 = crop(source1,450,50,1352,-40)

source1= FFVideoSource("464_-trim_crop_grd_cut.avi")

source1= KillAudio(source1)

source1_8bit = ConvertBits(source1, bits=8) #I dont'know much about formats, maybe this is not ideal...
source1YV12 = ConvertToYV12 (source1_8bit) #I dont'know much about formats, maybe this is not ideal...

txt_sz = 36

BlkSz = 24
OLap = (BlkSz/2) #important not too low (eg 4 for blksz 12, because creates stairs)
Pel = 1
Tm=false
Bblur = 0.6
ThSAD = 10000
RadT = 1

Usharp_strength= 80
Usharp_radius= 5
Usharp_th = 1 #too high = too many dark pixel packs (ex trees)


#Sharpen----------------------------------------------------------------------
source1_shp=source1.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#SpotRemoved_shp=SpotRemoved.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#----------------------------------------------------------------------------------

#Remove spots-----------------------------------------------------------------
spotless_107=SpotLess107(source1_shp,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
#SpotRemoved = SpotRemover(source1YV12,Spot=128, RadT=1, Pel=Pel, BlkSz=32, DeGrain=false, Chroma=true, PreFilter=true, mask=false, Tm = true,thsad=thsad, Glob = true)
#----------------------------------------------------------------------------------

#SpotRemoved = ConvertBits(SpotRemoved, bits=10) #I dont'know much about formats, maybe this is not ideal...
#SpotRemoved = ConvertToYUV422(SpotRemoved) #I dont'know much about formats, maybe this is not ideal...


# Restore grain -------------------------------------------------------------
#spotless_107_shp_Rg = mRD_RestoreGrain(spotless_107_shp, source1)
#SpotRemoved_shp_Rg = mRD_RestoreGrain(SpotRemoved_shp, source1)
#----------------------------------------------------------------------------------

# Delta Restore -------------------------------------------------------------

src = source1
filtered = spotless_107

# extra clipping (if needed)
#crop_left = 800
#crop_right = 280
#src = src.Crop(crop_left, 0, -crop_right, 0)
#filtered = filtered.Crop(crop_left, 0, -crop_right, 0)

# diff with exaggerated chroma difference
diff1 = mt_makediff(src, filtered, y=3, u=3, v=3)
diff2 = diff1.mt_adddiff(diff1, y=0, u=3, v=3)
diff2 = diff2.mt_adddiff(diff1, y=0, u=3, v=3)
diff = CombinePlanes(diff1, diff2, diff2, planes="YUV", source_planes="YUV", sample_clip=diff1)

# prepare masks --------------------------------------------------------------------

# luma mask - selects where source brightness > filtered brightness
luma_mask = LumaDeltaMask(src, filtered, ">", brightness=1.1, spotSize=3)
luma_mask = ExpandMask(luma_mask, 3, blur=4)

# chroma mask - selects where source and filtered chromas have large enough difference (formula: sqrt(u*u + v*v))
chroma_mask = ChromaDeltaMask(src, filtered, delta=3, spotSize=3)

# suppression mask - selects where source brightness < filtered brightness
suppression_mask = LumaDeltaMask(src, filtered, "<", brightness=0.9, spotSize=2)

# suppression mask removes too dark areas from chroma mask
suppressed_chroma_mask = mt_lutxy(chroma_mask, suppression_mask, expr="x y -")

# edge mask - detects sharp edges
edges = scharr(src)

# don't detect edges near the frame edges
edges = edges.Padding(60, 14, 14, 10)

edgeMask = edges.mt_binarize(52)


# dark mask - selects where source brightness < filtered brightness but not below limitBrightness
dark_mask = LumaDeltaMask(src, filtered, "<", brightness=0.97, limitBrightness=0.66, spotSize=5)
darkEdgeMask = edges.mt_binarize(44) # 40
darkEdgeAreaMask = mt_hysteresis(darkEdgeMask, dark_mask, chroma="-127")
dark_mask = mt_lutxy(dark_mask, darkEdgeAreaMask, expr="x y -")
dark_mask = ExpandMask(dark_mask, 3, blur=4)

# chroma override mask - selects larger color deltas and spot sizes than plain chroma mask and is not suppressed
chroma_override = ChromaDeltaMask(src, filtered, delta=5, spotSize=6) # orig. spotSize 6
chroma_override = ExpandMask(chroma_override, 3, blur=2)

# combine luma, chroma and chroma override masks
combined_mask = mt_logic(luma_mask, suppressed_chroma_mask, "or")
combined_mask = mt_logic(combined_mask, chroma_override, "or")

# remove mask areas where sharp edges were found (those are most likely dirt)
edgeAreaMask = mt_hysteresis(edgeMask, combined_mask, chroma="-127")
combined_mask = mt_lutxy(combined_mask, edgeAreaMask, expr="x y -")

# add dark areas (these are already checked for edges)
combined_mask = mt_logic(combined_mask, dark_mask, "or")

# TEST - luma and chroma override only
#combined_mask = mt_logic(luma_mask, chroma_override, "or")

# combined mask without suppression and chroma override - for testing
combined_no_suppr = mt_logic(luma_mask, chroma_mask, "or")


# restore ---------------------------------------------------------------------------
delta_restore = Overlay(filtered, src, mask=combined_mask)
restore_nosuppression = Overlay(filtered, src, mask=combined_no_suppr)


edgeAreaMask = mt_logic(edgeAreaMask, darkEdgeAreaMask, "or")
#edgeAreaMask = darkEdgeAreaMask

# create mask visualization (for debugging) -----------------------------------------
black = BlankClip(src)
yellow = BlankClip(src, color=$d1cc2e) # #d1cc2e f7f140
green = BlankClip(src, color=$40f75b)
gray = BlankClip(src, color=$d0d0d0)
darkgray = BlankClip(src, color=$909090)
orange = BlankClip(src, color=$a36b02)
purple = BlankClip(src, color=$4e0091)
red = BlankClip(src, color=$9e2424)
blue = BlankClip(src, color=$0d3b7a) #24579e 243c9e
maskviz = Overlay(black, yellow, mask=chroma_mask) # chroma mask - yellow
maskviz = Overlay(maskviz, red, mask=suppression_mask, opacity=0.7) # suppression mask - red
maskviz = Overlay(maskviz, green, mask=chroma_override) # chroma override - green
maskviz = Overlay(maskviz, gray, mask=luma_mask) # luma mask - gray
maskviz = Overlay(maskviz, purple, mask=dark_mask) # dark mask - dark gray
maskviz = Overlay(maskviz, blue, mask=edgeAreaMask, opacity=0.8) # sharpness mask - blue

comp = Overlay(delta_restore, maskviz, opacity=0.45)

#return Interleave(StackHorizontal(src, comp), StackHorizontal(delta_restore, maskviz))

#return StackHorizontal(src, filtered, delta_restore, edgeMask, maskviz)

stack = StackHorizontal(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(edgeMask, "edge mask", txt_sz),\
sub(diff, "diff", txt_sz)\
)

# scaling (if needed)
stack = stack.BilinearResize(stack.width/2 + 1, stack.height/2)

return stack

inter = Interleave(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(diff, "diff", txt_sz)\
)

return inter

function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}


#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------

function Padding(clip c, int left, int top, int right, int bottom) {
return c.Crop(left, top, -right, -bottom).AddBorders(left, top, right, bottom, color_yuv=$008080)
}

function scharr(clip c) {
scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), chroma="-127")
return scharr
}

function RemoveSmallSpots(clip mask, int spotSize) {
mask_orig = mask

# remove too small spots ----------------------------------------------------
for (i = 1, spotSize) {
mask = mask.mt_inpand()
}
mask = mt_hysteresis(mask, mask_orig, chroma="-127")
return mask
}

function LumaDeltaMask(clip source, clip filtered, string direction, float "brightness", float "limitBrightness", int "spotSize") {
# direction is ">" if you want mask to contain areas where source is brighter than filtered by "brightness", direction "<" selects where source is darker
# adjust brightness to change how bright objects are selected, typical values between 1.0 and 1.1, larger value = brighter object
# remove too bright or dark areas using limitBrightness
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

brightness = Default(brightness, 0.9)
spotSize = Default(spotSize, 2)
limitBrightness = Default(limitBrightness, brightness)

# diff
diff = Grayscale(mt_makediff(source, filtered))

# mask
mask = mt_lut(diff, expr="x range_half "+String(brightness)+" * " + direction + " 255 scalef 0 ?")
mask_orig = mask

# limitBrightness suppression
if (limitBrightness != brightness) {
mask_suppression = mt_lut(diff, expr="x range_half "+String(limitBrightness)+" * " + direction + " 255 scalef 0 ?")
mask = mt_lutxy(mask, mask_suppression, expr="x y -")
}

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

# limitBrightness suppression (part 2)
if (limitBrightness != brightness) {
# if suppression didn't kill the whole mask then return it without suppression
mask = mt_hysteresis(mask, mask_orig, chroma="-127")
}

return mask
}

function ChromaDeltaMask(clip source, clip filtered, int "delta", int "spotSize") {

# adjust delta to change how different colors are selected
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

delta = Default(delta, 3)
spotSize = Default(spotSize, 2)

# diff ----------------------------------------------------------------------
diff = mt_makediff(source, filtered, y=3, u=3, v=3)

# extract chroma channels
u = ExtractU(diff)
v = ExtractV(diff)

# calculate deltas
u_delta = mt_lut(u, expr="x range_half - abs")
v_delta = mt_lut(v, expr="x range_half - abs")

# combine deltas
# uv_delta = mt_lutxy(u_delta, v_delta, expr="x y +")
uv_delta = mt_lutxy(u_delta, v_delta, expr="x x * y y * + 0.5 ^")

# rescale (if needed)
if (uv_delta.width < source.width || uv_delta.height < source.height) {
uv_delta = uv_delta.BilinearResize(source.width, source.height)
}

# create mask
mask = uv_delta.mt_binarize(delta)

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

# restore original format
blank = BlankClip(source)
mask = CombinePlanes(mask, blank, blank, planes="YUV", source_planes="YUV", sample_clip=source)

return mask
}

function ExpandMask(clip mask, int amount, float "blur") {
blur = Default(blur, 0)

for (i = 1, amount) {
mask = mask.mt_expand(chroma="-127")
}
if (blur > 0) {
#mask = mask.FastBlur(blur)
#mask = mask.binomialBlur(varY=blur)
# mask = mask.ConvertBits(16)
mask = mask.GBlur(rad=6, sd=blur, u=true)
# mask = mask.ConvertBits(10)
}

return mask
}



Function mRD_RestoreGrain (clip rd, clip o, float "str1", float "str2") {

sisavs26 = !(VersionNumber() < 2.60)

str1 = string(default(str1, 10.0))
str2 = string(default(str2, 5.0))

expr = sisavs26 ? " x range_half - abs "+str1+" scalef min "+str2+" scalef x range_half - abs - min 0 max x range_half - x range_half - abs 1 max / * range_half + " : " x 128 - abs "+str1+" min "+str2+" x 128 - abs - min 0 max x 128 - x 128 - abs 1 max / * 128 + "

Return sisavs26 ? mt_makediff(o, rd).mt_lut(expr, use_expr=2).mt_adddiff(rd, chroma="copy second") : mt_makediff(o, rd).mt_lut(expr).mt_adddiff(rd, chroma="copy second")
}


Function SpotLess107(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess107: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

[zorr]

Some examples of dark mask in action:

Let's look at the happy cases first. The main goal was to restore the guy's hair and yes, we can do that! The dark mask is shown as purple in the mask visualization.

https://i.postimg.cc/MZ5vN8P4/dark-mask-4-frame-219.png

Here's another, the dark mask restores the missing features of a bird.

https://i.postimg.cc/SN4Jtxfc/dark-mask-1.png

But the dark mask also restores spots, here's a typical example:

https://i.postimg.cc/VsqNMcPv/dark-mask-3-frame-79.png

There's also at least one frame where the edge mask suppresses the dark mask because a sharp-edged spot is located over the dark mask:

https://i.postimg.cc/9fsfPmfR/dark-mask-2.png


I have another approach for the dark mask that I'm going to try next. Most of the time when it's a real feature the diff has both darker and lighter deltas. So if we only select those dark areas which have bright areas in close proximity we might have better success in detecting only what we want.

There also could be a better way to detect the sharpness of a feature. Now it's considered sharp if there is an edge anywhere within the area. Perhaps it works better if we only look at the sharpness of the edge pixels of the area.

[chmars]

(or some preprocessing to remove sharp features near the borders).

Yes, I just had dark scenes where these perforations disturb sight.
Don't want to crop the image but let's fill them.
I borrowed some code of yours. If we fill them with something like this? No sharp edges any more:

source1=FFVideoSource("464-13_63.mov").KillAudio()
perfo = LumaDeltaMask(source1, ">", brightness=1.0, spotSize=100)
fill_color = BlankClip(source1,color=$26282d)
# enlarge mask (optional) ---------------------------------------------------
expandtimes = 6
for (i = 1, expandtimes) {
perfo = perfo.mt_expand( chroma="-127")
}
maskblur = 6
if (maskblur > 0) {
perfo = perfo.FastBlur(maskblur)
}

filled = Overlay(source1,fill_color,mask=perfo)
return(filled)

#########################################################################################################################

function RemoveSmallSpots(clip mask, int spotSize) {
mask_orig = mask

# remove too small spots ----------------------------------------------------
for (i = 1, spotSize) {
mask = mask.mt_inpand()
}
mask = mt_hysteresis(mask, mask_orig, chroma="-127")
return mask
}

function LumaDeltaMask(clip source, string direction, float "brightness", int "spotSize") {
# direction is ">" if you want mask to contain areas where source is brighter than filtered by "brightness", direction "<" selects where source is darker
# adjust brightness to change how bright objects are selected, typical values between 1.0 and 1.1, larger value = brighter object
# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

brightness = Default(brightness, 0.9)
spotSize = Default(spotSize, 2)

gray = BlankClip(source, color=$c0c0c0)
# diff
diff = Grayscale(mt_makediff(source, gray ))

# mask
mask = mt_lut(diff, expr="x range_half "+String(brightness)+" * " + direction + " 255 scalef 0 ?")

# remove too small spots
mask = RemoveSmallSpots(mask, spotSize)

return mask
}


Have TODO:
-find the way to copy the color which is between the holes, as it sometimes shortly changes like in the example below.
-maybe limit horizontal blur.
It will never be 100% but I think could be discreete enough.
https://nmldqjct.preview.infomaniak.website/shared_files/index.php/s/KZs5b4MCjsHb8Et/download

EDIT: Made new thread not to mix all:
https://forum.doom9.org/showthread.php?t=182930


...Didn't implement as a function yet or include chmars' latest changes, sorry)
No problem, later.

It's great, you are getting closer with the dark mask.

I wander if we shouldn't look earlier in the process.
If we could talk to Spotless in terms of surface instead of block side size (n*n), I believe this would help.
I mean: we specify a spot size to remove. But the big spots are rather long but have a small surface (liker hairs).
A dirt spot as big as the birds or guy's hair is rare.
So if we could specify [max length] AND [max surface] for spots to be removed, wouldn't this help a lot?

[kedautinh12]

I think Padding() is a normal name, maybe same function names in the future. I think you need rename for your personal. Ex: zorr_Padding() or z_Padding()

[zorr]

Yes, I just had dark scenes where these perforations disturb sight.
Don't want to crop the image but let's fill them.

I noticed you made another thread for this subject, so let's keep that discussion there.

It's great, you are getting closer with the dark mask.

Yes I am actually. The idea I described in my previous post seems to work fairly well. :D And I also added a limit based on the absolute brightness. No results to show yet, have to get some sleep...

I mean: we specify a spot size to remove. But the big spots are rather long but have a small surface (liker hairs).
A dirt spot as big as the birds or guy's hair is rare.
So if we could specify [max length] AND [max surface] for spots to be removed, wouldn't this help a lot?

Spotless mostly doesn't remove those long hair-like things so Delta Restore cannot help there. It would be simple to specify the spot in terms of width and height instead of size but I'm not sure how many cases there are where that would make an improvement. Surface area is trickier... it's possible to calculate the total area of the mask but not per spot, at least without writing an actual plugin (I shouldn't say it's not possible because it probably is but it would be *very* slow to execute such code).

I think Padding() is a normal name, maybe same function names in the future. I think you need rename for your personal. Ex: zorr_Padding() or z_Padding()

Thanks for the hint, I'll rename it or perhaps it's going to be an internal function not visible outside Delta Restore.

[chmars]

I noticed you made another thread for this subject, so let's keep that discussion there.
Yes, thought it was better to sort the questions. But still, it will help with borders, I hope.

Spotless mostly doesn't remove those long hair-like things

I ment, for the ones it removes (almost all "long hairs" on my footages are away) thanks to it.

Surface area is trickier...I shouldn't say it's not possible because it probably is but it would be *very* slow to execute such code).

Hm, I understand, ok. Anyway very far from what I could achieve in present knowlage...

No results to show yet, have to get some sleep...

Hey, don't forget Spotelss + Delta Restore are already brilliant and working in a huge majority of cases!
I am verifying another footage made with them, I found no fault. Footage with so many birds, guys and kids like the previous one seems rare.

[Sharc]

Spotless mostly doesn't remove those long hair-like things ....

Perhaps this may help:
turnleft() #some angle
spotless()
turnright()