[film transfer] Remove dirt but keep grain?

[poisondeathray]

Originally Posted by coolgit

What did you mean by "stack enabled"?

The issue I had no longer exist. I can't seem to replicate it either. Must be something to do with the variables I kept messing around with to find the optimum performance.

"stack enabled" - I mean the stackhorizontal/stackvertical view is the current output in the script you provided . i.e The view comparing 4 versions at the same time in a 2x2 grid

[zorr]

Originally Posted by coolgit

Spotless removes the following defects but delta restore restored them. Commonly happens when whitish to light yellowish and black defects surrounded by light colour background. Blackish defect surrounded by black background. These defects have none to little motion between frames and are background defects. Why would delta restore restore them?

I need the original video or at least shots of the mask visualization to analyze those. Most likely the reason these defects are restored is that the default Delta Restore parameter values which you're using were tuned for other clip and need some changes. And even then the process is not perfect.

[johnmeyer]

I finally got around to trying Zorr's "Delta Restore (Dark Mask).avs" as shown in Post #101 in this thread.

I initially tried it on some American football games from 1941 that I'm restoring, but then tried it on the "torture" test clip I posted earlier in this thread.

Bottom line: it did not work well at all. The thrown balls disappeared just like they do with "RemoveDirtMC". I saw no evidence of any masking in the two far-right windows in the debug display.

The ONLY change I made to the script from what was posted is that I removed the UnsharpMask call because it kept complaining that there is no such thing as "sspasses". I looked all over for that, and then decided that I really didn't need sharpening.

I've copied the script that I used into the next post.

This is too bad because I was really looking forward to not having to turn dirt removal on/off around each thrown ball in a football game.

[johnmeyer]

Here is the copy/pasted version of the script I'm using. It should be identical to Post #101, except for the removal of the UnsharpMask_avsi call.

loadplugin("E:\Documents\My Videos\AVISynth\AVISynth Plugins\plugins\MVTools2 (latest)\mvtools2.dll")

source1= AVISource("e:\fs.avi")
source1= KillAudio(source1).ConvertToYV12

source1_8bit = ConvertBits(source1, bits=8) #I dont'know much about formats, maybe this is not ideal...
source1YV12 = ConvertToYV12 (source1_8bit) #I dont'know much about formats, maybe this is not ideal...
     
txt_sz = 36

BlkSz = 24
OLap = (BlkSz/2) #important not too low (eg 4 for blksz 12, because creates stairs)
Pel = 1
Tm=false
Bblur = 0.6
ThSAD = 10000
RadT = 1
               
Usharp_strength= 80
Usharp_radius= 5
Usharp_th = 1      #too high = too many dark pixel packs (ex trees)


#Sharpen----------------------------------------------------------------------
#source1_shp=source1.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
source1_shp=source1
#SpotRemoved_shp=SpotRemoved.UnsharpMask_avsi(Usharp_strength,Usharp_radius,Usharp_th)
#----------------------------------------------------------------------------------

#Remove spots-----------------------------------------------------------------
spotless_107=SpotLess107(source1_shp,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)
#SpotRemoved = SpotRemover(source1YV12,Spot=128, RadT=1, Pel=Pel, BlkSz=32, DeGrain=false, Chroma=true, PreFilter=true, mask=false, Tm = true,thsad=thsad, Glob = true)
#----------------------------------------------------------------------------------

#SpotRemoved = ConvertBits(SpotRemoved, bits=10) #I dont'know much about formats, maybe this is not ideal...
#SpotRemoved = ConvertToYUV422(SpotRemoved) #I dont'know much about formats, maybe this is not ideal...


# Restore grain -------------------------------------------------------------
#spotless_107_shp_Rg = mRD_RestoreGrain(spotless_107_shp, source1)
#SpotRemoved_shp_Rg = mRD_RestoreGrain(SpotRemoved_shp, source1)
#----------------------------------------------------------------------------------

# Delta Restore -------------------------------------------------------------

src = source1
filtered = spotless_107

# extra clipping (if needed)
#crop_left = 800
#crop_right = 280
#src = src.Crop(crop_left, 0, -crop_right, 0)
#filtered = filtered.Crop(crop_left, 0, -crop_right, 0)

# diff with exaggerated chroma difference
diff1 = mt_makediff(src, filtered, y=3, u=3, v=3)
diff2 = diff1.mt_adddiff(diff1, y=0, u=3, v=3)
diff2 = diff2.mt_adddiff(diff1, y=0, u=3, v=3)
diff = CombinePlanes(diff1, diff2, diff2, planes="YUV", source_planes="YUV", sample_clip=diff1)

# prepare masks --------------------------------------------------------------------

# luma mask - selects where source brightness > filtered brightness 
luma_mask = LumaDeltaMask(src, filtered, ">", brightness=1.1, spotSize=3)
luma_mask = ExpandMask(luma_mask, 3, blur=4)

# chroma mask - selects where source and filtered chromas have large enough difference (formula: sqrt(u*u + v*v))
chroma_mask = ChromaDeltaMask(src, filtered, delta=3, spotSize=3)

# suppression mask - selects where source brightness < filtered brightness
suppression_mask = LumaDeltaMask(src, filtered, "<", brightness=0.9, spotSize=2)

# suppression mask removes too dark areas from chroma mask 
suppressed_chroma_mask = mt_lutxy(chroma_mask, suppression_mask, expr="x y -")

# edge mask - detects sharp edges
edges = scharr(src)

# don't detect edges near the frame edges
edges = edges.ZPadding(60, 14, 14, 10)


# chroma override mask - selects larger color deltas and spot sizes than plain chroma mask and is not suppressed
chroma_override = ChromaDeltaMask(src, filtered, delta=5, spotSize=6) # orig. spotSize 6
chroma_override = ExpandMask(chroma_override, 3, blur=2)


# new approach for dark mask:
# select both darker and lighter mask and keep dark mask if it is near lighter mask or overlaps chroma override mask
dark = LumaDeltaMask(src, filtered, "<", brightness=0.97, limitBrightness=0.66, spotSize=5)
light = LumaDeltaMask(src, filtered, ">", brightness=1.04, spotSize=5)
light = light.ZPadding(60, 14, 14, 10)

dark_near_light = isNearMask(dark, light, 28)
expanded = IsNearMask(dark, light, 28, expanded=true) # for debugging
dark_final = ExpandMask(dark_near_light, 3, blur=4)

# secondary dark mask - doesn't need to be near lighter area but has stricter limits
dark2 = LumaDeltaMask(src, filtered, "<", brightness=0.97, limitBrightness=0.69, limitAbsBrightness=0.12, spotSize=5)
dark2 = ExpandMask(dark2, 2, blur=2)
dark_final = mt_logic(dark_final, dark2, "or")


# edge test for dark areas
darkEdgeMask = edges.mt_binarize(54) # at least 54 needed for frame 222
darkEdgeAreaMask = mt_hysteresis(darkEdgeMask, dark_final, chroma="-127")
dark_final = mt_lutxy(dark_final, darkEdgeAreaMask, expr="x y -")


# combine luma, chroma and chroma override masks
combined_mask = mt_logic(luma_mask, suppressed_chroma_mask, "or")
combined_mask = mt_logic(combined_mask, chroma_override, "or")

# remove mask areas where sharp edges were found (those are most likely dirt)
edgeMask = edges.mt_binarize(52) 
edgeAreaMask = mt_hysteresis(edgeMask, combined_mask, chroma="-127")
combined_mask = mt_lutxy(combined_mask, edgeAreaMask, expr="x y -")

# add dark areas (no edge test for those)
combined_mask = mt_logic(combined_mask, dark_final, "or")


# restore ---------------------------------------------------------------------------
delta_restore = Overlay(filtered, src, mask=combined_mask)

edgeAreaMask = mt_logic(edgeAreaMask, darkEdgeAreaMask, "or")
#edgeAreaMask = darkEdgeAreaMask


# create mask visualization (for debugging) -----------------------------------------
black = BlankClip(src)
yellow = BlankClip(src, color=$d1cc2e) # #d1cc2e	f7f140
green = BlankClip(src, color=$40f75b)
gray = BlankClip(src, color=$d0d0d0)
darkgray = BlankClip(src, color=$909090)
orange = BlankClip(src, color=$a36b02)
purple = BlankClip(src, color=$4e0091)
purple_light = BlankClip(src, color=$bc6eff)
purple_dark = BlankClip(src, color=$340061)
red = BlankClip(src, color=$9e2424)
blue = BlankClip(src, color=$0d3b7a) #24579e 243c9e

# maskviz for dark mask
maskviz = Overlay(black, purple, mask=expanded) 
maskviz = Overlay(maskviz, purple_dark, mask=dark) 
maskviz = Overlay(maskviz, purple_light, mask=light) 
maskviz = Overlay(maskviz, black, mask=dark_near_light) 

# only show dark mask?
showDarkMaskOnly = false
if (!showDarkMaskOnly) {
	maskviz = Overlay(maskviz, yellow, mask=chroma_mask) 					# chroma mask - yellow
	maskviz = Overlay(maskviz, red, mask=suppression_mask, opacity=0.7)	# suppression mask - red
	maskviz = Overlay(maskviz, green, mask=chroma_override)				# chroma override - green
	maskviz = Overlay(maskviz, gray, mask=luma_mask)						# luma mask - gray
	maskviz = Overlay(maskviz, blue, mask=edgeAreaMask, opacity=0.8) 		# sharpness mask - blue
}

# mask overlay
comp = Overlay(delta_restore, maskviz, opacity=0.45)
comp = Overlay(comp, purple_light, mask=light, opacity=0.2)
comp = Overlay(comp, black, mask=dark, opacity=0.2)
comp = Overlay(comp, black, mask=dark_final, opacity=0.6)

maskviz = Overlay(maskviz, black, mask=dark_final, opacity=0.6)

#return Interleave(StackHorizontal(src, comp), StackHorizontal(delta_restore, maskviz))
#return StackHorizontal(src, filtered, delta_restore, edgeMask, maskviz)

stack = StackHorizontal(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(edgeMask, "edge mask", txt_sz),\
sub(diff, "diff", txt_sz)\
)

# scaling (if needed)
stack = stack.BilinearResize(stack.width/2 + 2, stack.height/2)

return stack

inter = Interleave(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(diff, "diff", txt_sz)\
)

return inter

function sub(c, label, txt_sz) {
	return subtitle(c, label, size = txt_sz, align=8)
}


video = Interleave(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(overlay,"delta restore",size = txt_sz,align=2),\
subtitle(restore_nosuppression,"no suppression",size = txt_sz,align=2),\
subtitle(spotless_107_shp,"spotless_107_shp",size = txt_sz,align=2)\
)

masks = Interleave(\
subtitle(BlankClip(source1),"",size = txt_sz,align=2),\
subtitle(combined_mask,"combined_mask",size = txt_sz,align=2),\
subtitle(BlankClip(spotless_107_shp),"",size = txt_sz,align=2)\
)

return StackHorizontal(video, masks)

group = StackHorizontal(\
subtitle(source1,"original",size = txt_sz,align=2),\
subtitle(spotless_107_shp,"spotless_107_shp",size = txt_sz,align=2),\
subtitle(overlay,"delta restore",size = txt_sz,align=2),\
subtitle(combined_mask,"combined mask",size = txt_sz,align=2),\
subtitle(luma_mask,"luma delta mask",size = txt_sz,align=2),\
subtitle(suppression_mask,"suppression mask",size = txt_sz,align=2),\
subtitle(suppressed_chroma_mask,"suppressed chroma mask",size = txt_sz,align=2),\
subtitle(chroma_mask,"chroma delta mask",size = txt_sz,align=2),\
subtitle(diff,"diff",size = txt_sz,align=2))
#subtitle(chroma_mask,"chroma mask",size = txt_sz,align=2),\
#subtitle(whitemask,"whitemask",size = txt_sz,align=2),\
#subtitle(whitemask,"filled whitemask",size = txt_sz,align=2),\
#subtitle(uv,"UV",size = txt_sz,align=2))
#subtitle(whitemask_orig,"orig. whitemask",size = txt_sz,align=2),\
#subtitle(SpotRemoved,"SpotRemover",size = txt_sz,align=2))
#subtitle(spotless_107_shp_Rg,"spotless_107_shp_Rg",size = txt_sz,align=2),\

return(group)#.AddBorders(1000, 0, 500, 0, color=$000000)



#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------
#~ ------------------------------------------------------------------------------------------------------------------

function OverlapsMask(clip mask, clip otherMask) {
	return mt_logic(mask, mt_hysteresis(otherMask, mask), "and")
}

function IsNearMask(clip mask, clip otherMask, int maxDistance, bool "expanded") {
	expanded = Default(expanded, false)

	expand_amount = maxDistance / 2
	mask_expanded = ExpandMask(mask, expand_amount)
	other_expanded = ExpandMask(otherMask, maxDistance - expand_amount)
	common = mt_logic(mask_expanded, other_expanded, "or")
	if (expanded) {
		return common
	}
	other_near = mt_hysteresis(otherMask, common)
	return mt_logic(mask, other_near, "and")	
}

function ZPadding(clip c, int left, int top, int right, int bottom) {
	return c.Crop(left, top, -right, -bottom).AddBorders(left, top, right, bottom, color_yuv=$008080)
}

function scharr(clip c) {
	scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
	scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
	scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), chroma="-127")
	return scharr
}

function RemoveSmallSpots(clip mask, int spotSize) {
	mask_orig = mask

	# remove too small spots ----------------------------------------------------
	for (i = 1, spotSize) {
		mask = mask.mt_inpand()
	}
	mask = mt_hysteresis(mask, mask_orig, chroma="-127") 
	return mask
}

function LumaDeltaMask(clip source, clip filtered, string direction, float "brightness", float "limitBrightness", float "limitAbsBrightness", int "spotSize") {
	# direction is ">" if you want mask to contain areas where source is brighter than filtered by "brightness", direction "<" selects where source is darker
	# adjust brightness to change how bright objects are selected, typical values between 1.0 and 1.1, larger value = brighter object
	# remove too bright or dark areas using limitBrightness
	# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

	brightness = Default(brightness, 0.9)
	spotSize   = Default(spotSize, 2)
	limitBrightness = Default(limitBrightness, brightness)
	limitAbsBrightness = Default(limitAbsBrightness, 0.0)

	# diff 
	diff = Grayscale(mt_makediff(source, filtered))

	# mask 
	mask = mt_lut(diff, expr="x range_half "+String(brightness)+" * " + direction + " 255 scalef 0 ?")
	mask_orig = mask

	# limitBrightness suppression
	if (limitBrightness != brightness) {
		mask_suppression = mt_lut(diff, expr="x range_half "+String(limitBrightness)+" * " + direction + " 255 scalef 0 ?")
		mask = mt_lutxy(mask, mask_suppression, expr="x y -")
	}

	# limitAbsBrightness suppression
	if (limitAbsBrightness > 0.0) {
		mask = mt_lutxy(mask, source, expr="y range_max "+String(limitAbsBrightness)+" * < 0 x ?")
	}

	# remove too small spots
	mask = RemoveSmallSpots(mask, spotSize)

	# limitBrightness or limitAbsBrightness suppression (part 2)
	if (limitBrightness != brightness || limitAbsBrightness > 0.0) {
		# if suppression didn't kill the whole mask then return it without suppression
		mask = mt_hysteresis(mask, mask_orig, chroma="-127")
	}
	
	return mask
}

function ChromaDeltaMask(clip source, clip filtered, int "delta", int "spotSize") {

	# adjust delta to change how different colors are selected
	# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

	delta = Default(delta, 3)
	spotSize   = Default(spotSize, 2)

	# diff ----------------------------------------------------------------------
	diff = mt_makediff(source, filtered, y=3, u=3, v=3)

	# extract chroma channels
	u = ExtractU(diff)
	v = ExtractV(diff)

	# calculate deltas
	u_delta = mt_lut(u, expr="x range_half - abs")
	v_delta = mt_lut(v, expr="x range_half - abs")

	# combine deltas
#	uv_delta = mt_lutxy(u_delta, v_delta, expr="x y +")
	uv_delta = mt_lutxy(u_delta, v_delta, expr="x x * y y * + 0.5 ^")

	# rescale (if needed)
	if (uv_delta.width < source.width || uv_delta.height < source.height) {
		uv_delta = uv_delta.BilinearResize(source.width, source.height)
	}

	# create mask
	mask = uv_delta.mt_binarize(delta)

	# remove too small spots 
	mask = RemoveSmallSpots(mask, spotSize)	

	# restore original format
	blank = BlankClip(source)
	mask = CombinePlanes(mask, blank, blank, planes="YUV", source_planes="YUV", sample_clip=source)

	return mask
}

function ExpandMask(clip mask, int amount, float "blur") {
	blur = Default(blur, 0)

	for (i = 1, amount) {
		mask = mask.mt_expand(chroma="-127")
	}
	if (blur > 0) {
		#mask = mask.FastBlur(blur)
		#mask = mask.binomialBlur(varY=blur)
#		mask = mask.ConvertBits(16)
		mask = mask.GBlur(rad=6, sd=blur, u=true)
#		mask = mask.ConvertBits(10)
	}

	return mask
}


Function mRD_RestoreGrain (clip rd, clip o, float "str1", float "str2") {

sisavs26   = !(VersionNumber() < 2.60)

str1 = string(default(str1, 10.0))
str2 = string(default(str2, 5.0))

expr = sisavs26 ? " x range_half - abs "+str1+" scalef min "+str2+" scalef x range_half - abs - min 0 max x range_half - x range_half - abs 1 max / * range_half + " : " x 128 - abs "+str1+" min "+str2+" x 128 - abs - min 0 max x 128 - x 128 - abs 1 max / * 128 + "

Return sisavs26 ? mt_makediff(o, rd).mt_lut(expr, use_expr=2).mt_adddiff(rd, chroma="copy second") : mt_makediff(o, rd).mt_lut(expr).mt_adddiff(rd, chroma="copy second")
}

[zorr]

Originally Posted by johnmeyer

I finally got around to trying Zorr's "Delta Restore (Dark Mask).avs"
...
tried it on the "torture" test clip I posted earlier in this thread.
Bottom line: it did not work well at all. The thrown balls disappeared just like they do with "RemoveDirtMC". I saw no evidence of any masking in the two far-right windows in the debug display.

There might be something weird going on as you should see some masking even with those "default" settings which were tuned for other video.

But you're right that the script without any changes doesn't do a very good job with the torture clip's ball. I never meant the script to work as-is for every video, it was just an example use case. You really need to tune the values for each case.

When you start you should only use the basic features starting with luma mask, adjust that and if it's not enough you can try the chroma mask and/or the dark mask. The dark mask is quite tricky to get right so I advise to use the luma and chroma masks if possible.

I did just that for the torture clip. Some minor adjustments were needed for the luma mask and already most of the missing balls were restored. The remaining problem was when ball moves across the sky and it's darker than the background. That's where you usually need the dark mask but in this case there's also big difference in chroma (easy to see in the diff debug view) so I enabled the chroma mask and it did a pretty good job. There's still some frames where ball is missing (I didn't aim for perfection) which might be salvageable with more tuning and/or using the more advanced mask features.

The tuning of parameters may not be very intuitive and I had an idea how to "learn" the correct parameters using Zopti and user provided examples, but I'm not working on that currently so the manual tinkering is unfortunately needed for now.

I'll post the script in the next message.

[zorr]

Here's the modified script:

source1= FFVideoSource("Dirt Removal Test Clip - Thrown Ball.mp4")
source1= KillAudio(source1)

source1_8bit = ConvertBits(source1, bits=8) #I dont'know much about formats, maybe this is not ideal...
source1YV12 = ConvertToYV12 (source1_8bit) #I dont'know much about formats, maybe this is not ideal...
     
txt_sz = 36

BlkSz = 24
OLap = (BlkSz/2) #important not too low (eg 4 for blksz 12, because creates stairs)
Pel = 1
Tm=false
Bblur = 0.6
ThSAD = 10000
RadT = 1
               


#Remove spots-----------------------------------------------------------------
spotless_107=SpotLess107(source1,blksz=BlkSz,OLap=Olap,pel=pel,Tm=Tm,Bblur=Bblur,thsad=thsad,RadT=RadT)


# Delta Restore -------------------------------------------------------------

src = source1
filtered = spotless_107

# extra clipping (if needed)
crop_left = 20
crop_right = 1120
#src = src.Crop(crop_left, 0, -crop_right, 0)
#filtered = filtered.Crop(crop_left, 0, -crop_right, 0)

# diff with exaggerated chroma difference
diff1 = mt_makediff(src, filtered, y=3, u=3, v=3)
diff2 = diff1.mt_adddiff(diff1, y=0, u=3, v=3)
diff2 = diff2.mt_adddiff(diff1, y=0, u=3, v=3)
diff = CombinePlanes(diff1, diff2, diff2, planes="YUV", source_planes="YUV", sample_clip=diff1)

# prepare masks --------------------------------------------------------------------

# luma mask - selects where source brightness > filtered brightness 
luma_mask = LumaDeltaMask(src, filtered, ">", brightness=1.13, spotSize=2).ExpandMask(3, blur=4)

# chroma mask - selects where source and filtered chromas have large enough difference (formula: sqrt(u*u + v*v))
chroma_mask = ChromaDeltaMask(src, filtered, delta=4, spotSize=3).ExpandMask(0, blur=2)

# suppression mask - selects where source brightness < filtered brightness
# suppression mask removes too dark areas from chroma mask 
suppression_mask = LumaDeltaMask(src, filtered, "<", brightness=0.9, spotSize=2)
suppressed_chroma_mask = chroma_mask.SuppressWith(suppression_mask)

# edge mask - detects sharp edges
# don't detect edges near the frame edges
edges = scharr(src).ZPadding(60, 14, 14, 10)
edgeMask = edges.mt_binarize(52) 

# chroma override mask - selects larger color deltas and spot sizes than plain chroma mask and is not suppressed
chroma_override = ChromaDeltaMask(src, filtered, delta=5, spotSize=6).ExpandMask(3, blur=2)

# dark mask: select both darker and lighter mask and keep dark mask if it is near lighter mask or overlaps chroma override mask
dark = LumaDeltaMask(src, filtered, "<", brightness=0.97, limitBrightness=0.66, spotSize=5)
light = LumaDeltaMask(src, filtered, ">", brightness=1.04, spotSize=5).ZPadding(60, 14, 14, 10)
dark_near_light = dark.MustBeNear(light, 28)#.ExpandMask(3, blur=4)
expanded = dark.MustBeNear(light, 28, expanded=true) # for debugging

# secondary dark mask - doesn't need to be near lighter area but has stricter limits
dark2 = LumaDeltaMask(src, filtered, "<", brightness=0.97, limitBrightness=0.69, limitAbsBrightness=0.12, spotSize=5).ExpandMask(2, blur=2)

# edge test for dark areas
darkEdgeMask = edges.mt_binarize(54) # at least 54 needed for frame 222
darkEdgeAreaMask = mt_hysteresis(darkEdgeMask, dark_near_light.AddTo(dark2), chroma="-128")	# for debug

dark_final = DarkDeltaMask(dark_near_light, secondary=dark2, edgeMask=darkEdgeMask)


combined_mask = luma_mask.AddTo(suppressed_chroma_mask).AddTo(chroma_override)

# remove mask areas where sharp edges were found (those are most likely dirt)
edgeAreaMask = mt_hysteresis(edgeMask, combined_mask, chroma="-128")	# for debug

#delta_restore = DeltaRestore(filtered, src, luma=luma_mask, chroma=suppressed_chroma_mask, chromaOverride=chroma_override, edges=edgeMask, dark=dark_final)
#delta_restore = DeltaRestore(filtered, src, luma=luma_mask)
delta_restore = DeltaRestore(filtered, src, luma=luma_mask, chroma=chroma_mask)

edgeAreaMask = mt_logic(edgeAreaMask, darkEdgeAreaMask, "or")
#edgeAreaMask = darkEdgeAreaMask

# create mask visualization (for debugging) -----------------------------------------
black = BlankClip(src)
yellow = BlankClip(src, color=$d1cc2e) # #d1cc2e	f7f140
green = BlankClip(src, color=$40f75b)
gray = BlankClip(src, color=$d0d0d0)
darkgray = BlankClip(src, color=$909090)
orange = BlankClip(src, color=$a36b02)
purple = BlankClip(src, color=$4e0091)
purple_light = BlankClip(src, color=$bc6eff)
purple_dark = BlankClip(src, color=$340061)
red = BlankClip(src, color=$9e2424)
blue = BlankClip(src, color=$0d3b7a) #24579e 243c9e

# maskviz for dark mask
maskviz = Overlay(black, purple, mask=expanded) 
maskviz = Overlay(maskviz, purple_dark, mask=dark) 
maskviz = Overlay(maskviz, purple_light, mask=light) 
maskviz = Overlay(maskviz, black, mask=dark_near_light) 

# reset masks
maskviz = black
#chroma_mask = black
suppression_mask = black
chroma_override = black
edgeAreaMask = black

# only show dark mask?
showDarkMaskOnly = false
if (!showDarkMaskOnly) {
	maskviz = Overlay(maskviz, yellow, mask=chroma_mask) 					# chroma mask - yellow
	maskviz = Overlay(maskviz, red, mask=suppression_mask, opacity=0.7)	# suppression mask - red
	maskviz = Overlay(maskviz, green, mask=chroma_override)				# chroma override - green
	maskviz = Overlay(maskviz, gray, mask=luma_mask)						# luma mask - gray
	maskviz = Overlay(maskviz, blue, mask=edgeAreaMask, opacity=0.8) 		# sharpness mask - blue
}

# mask overlay
comp = Overlay(delta_restore, maskviz, opacity=0.45)
comp = Overlay(comp, purple_light, mask=light, opacity=0.2)
comp = Overlay(comp, black, mask=dark, opacity=0.2)
comp = Overlay(comp, black, mask=dark_final, opacity=0.6)

maskviz = Overlay(maskviz, black, mask=dark_final, opacity=0.6)

stack = StackHorizontal(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(diff, "diff", txt_sz)\
)
#sub(edgeMask, "edge mask", txt_sz),\

# scaling (if needed)
#stack = stack.BilinearResize(stack.width/2 + 1, stack.height/2)

return stack

inter = Interleave(\
sub(src, "source", txt_sz), \
sub(filtered, "Spotless", txt_sz), \
sub(delta_restore, "Delta Restore", txt_sz), \
sub(comp, "restore + mask", txt_sz), \
sub(maskviz, "masks used", txt_sz),\
sub(diff, "diff", txt_sz)\
)

return inter

function sub(c, label, txt_sz) {
	return subtitle(c, label, size = txt_sz, align=8)
}


#~ ------------------------------------------------------------------------------------------------------------------

function DeltaRestore(clip filtered, clip source, clip "luma", clip "chroma", clip "chromaOverride", clip "edges", clip "dark") {
	Assert(luma.Defined || chroma.Defined || chromaOverride.Defined || dark.Defined, "DeltaRestore:must define at least one of luma, chroma, chromaOverride, dark")

	mask = Default(luma, BlankClip(filtered))
	if (chroma.Defined) {
		mask = mask.AddTo(chroma)
	}
	if (chromaOverride.Defined) {
		mask = mask.AddTo(chromaOverride)
	}
	if (edges.Defined) {
		mask = mask.MustNotOverlap(edges)
	}
	if (dark.Defined) {
		mask = mask.AddTo(dark)
	}
	return Overlay(filtered, source, mask=mask)
}

function DarkDeltaMask(clip mask, clip "secondary", clip "edgeMask") {	
	if (secondary.Defined) {
		mask = mask.AddTo(secondary)
	}
	if (edgeMask.Defined) {
		mask = mask.MustNotOverlap(edgeMask)
	}
	return mask
}

function AddTo(clip mask, clip other) {
	return mt_logic(mask, other, "or")
}

function SuppressWith(clip mask, clip suppressionMask) {
	return mt_lutxy(mask, suppressionMask, expr="x y -")
}

function MustNotOverlap(clip mask, clip other) {
	other = mt_hysteresis(other, mask, chroma="-128")
	return mt_lutxy(mask, other, expr="x y -")
}

function DeltaMaskChroma(clip src, clip filtered, int threshold, int "spotSize", int "expand", float "blur") {
	spotSize = Default(spotSize, 3)
	expand = Default(expand, 0)
	blur = Default(blur, 0)

	chroma_mask = ChromaDeltaMask(src, filtered, delta=threshold, spotSize=spotSize)	
	chroma_mask = ExpandMask(chroma_mask, expand, blur=blur)
	return chroma_mask
}

function MustOverlap(clip mask, clip otherMask) {
	return mt_logic(mask, mt_hysteresis(otherMask, mask), "and")
}

function storeClip(clip c, clip stored, string name) {
	return propSetClip(c, name, function[stored](clip c) { return stored } )
}

function restoreClip(clip c, string name) {
	return ScriptClip(c, function[name]() { return propGetClip(name) })
}

function MustBeNear(clip mask, clip otherMask, int maxDistance, bool "expanded") {
	expanded = Default(expanded, false)

	expand_amount = maxDistance / 2
	mask_expanded = ExpandMask(mask, expand_amount)
	other_expanded = ExpandMask(otherMask, maxDistance - expand_amount)
	common = mt_logic(mask_expanded, other_expanded, "or")


	if (expanded) {
		return common
	}
	other_near = mt_hysteresis(otherMask, common)
	final = mt_logic(mask, other_near, "and")	

	return final
}

function ZPadding(clip c, int left, int top, int right, int bottom) {
	return c.Crop(left, top, -right, -bottom).AddBorders(left, top, right, bottom, color_yuv=$008080)
}

function scharr(clip c) {
	scharr_x = c.mt_edge("3 0 -3 10 0 -10 3 0 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
	scharr_y = c.mt_edge("3 10 3 0 0 0 -3 -10 -3", thY1 = 0, thY2 = 255, y=3, u=1, v=1)
	scharr = mt_lutxy(scharr_x, scharr_y, yexpr=mt_polish("((x*x)+(y*y))^0.5"), chroma="-127")
	return scharr
}

function RemoveSmallSpots(clip mask, int spotSize) {
	mask_orig = mask

	# remove too small spots
	for (i = 1, spotSize) {
		mask = mask.mt_inpand()
	}
	mask = mt_hysteresis(mask, mask_orig, chroma="-127") 
	return mask
}

function LumaDeltaMask(clip source, clip filtered, string direction, float "brightness", float "limitBrightness", float "limitAbsBrightness", int "spotSize") {
	# direction is ">" if you want mask to contain areas where source is brighter than filtered by "brightness", direction "<" selects where source is darker
	# adjust brightness to change how bright objects are selected, typical values between 1.0 and 1.1, larger value = brighter object
	# remove too bright or dark areas using limitBrightness
	# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

	brightness = Default(brightness, 0.9)
	spotSize   = Default(spotSize, 2)
	limitBrightness = Default(limitBrightness, brightness)
	limitAbsBrightness = Default(limitAbsBrightness, 0.0)

	# diff 
	diff = Grayscale(mt_makediff(source, filtered))

	# mask 
	mask = mt_lut(diff, expr="x range_half "+String(brightness)+" * " + direction + " 255 scalef 0 ?")
	mask_orig = mask

	# limitBrightness suppression
	if (limitBrightness != brightness) {
		mask_suppression = mt_lut(diff, expr="x range_half "+String(limitBrightness)+" * " + direction + " 255 scalef 0 ?")
		mask = mt_lutxy(mask, mask_suppression, expr="x y -")
	}

	# limitAbsBrightness suppression
	if (limitAbsBrightness > 0.0) {
		mask = mt_lutxy(mask, source, expr="y range_max "+String(limitAbsBrightness)+" * < 0 x ?")
	}

	# remove too small spots
	mask = RemoveSmallSpots(mask, spotSize)

	# limitBrightness or limitAbsBrightness suppression (part 2)
	if (limitBrightness != brightness || limitAbsBrightness > 0.0) {
		# if suppression didn't kill the whole mask then return it without suppression
		mask = mt_hysteresis(mask, mask_orig, chroma="-127")
	}
	
	return mask
}

function ChromaDeltaMask(clip source, clip filtered, int "delta", int "spotSize") {

	# adjust delta to change how different colors are selected
	# spotSize controls the smallest spot size allowed in the mask, typical values between 0 and 3

	delta = Default(delta, 3)
	spotSize   = Default(spotSize, 2)

	# diff 
	diff = mt_makediff(source, filtered, y=3, u=3, v=3)

	# extract chroma channels
	u = ExtractU(diff)
	v = ExtractV(diff)

	# calculate deltas
	u_delta = mt_lut(u, expr="x range_half - abs")
	v_delta = mt_lut(v, expr="x range_half - abs")

	# combine deltas
#	uv_delta = mt_lutxy(u_delta, v_delta, expr="x y +")
	uv_delta = mt_lutxy(u_delta, v_delta, expr="x x * y y * + 0.5 ^")

	# rescale (if needed)
	if (uv_delta.width < source.width || uv_delta.height < source.height) {
		uv_delta = uv_delta.BilinearResize(source.width, source.height)
	}

	# create mask
	mask = uv_delta.mt_binarize(delta)

	# remove too small spots 
	mask = RemoveSmallSpots(mask, spotSize)	

	# restore original format
	blank = BlankClip(source)
	mask = CombinePlanes(mask, blank, blank, planes="YUV", source_planes="YUV", sample_clip=source)

	return mask
}

function ExpandMask(clip mask, int amount, float "blur") {
	blur = Default(blur, 0)

	for (i = 1, amount) {
		mask = mask.mt_expand(chroma="-127")
	}
	if (blur > 0) {
		#mask = mask.FastBlur(blur)
		#mask = mask.binomialBlur(varY=blur)
		mask = mask.GBlur(rad=6, sd=blur, u=true)
	}
	return mask
}


Function SpotLess107(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
    myName   = "SpotLess107: "
    RadT     = Default(RadT,1)        # Temporal radius. (MCompensate arg)
    ThSAD    = Default(ThSAD,10000)   # SAD threshold at radius 1 (Default Nearly OFF).
    ThSAD2   = Default(ThSAD2,ThSAD)  # SAD threshold at radius RadT.
    Pel      = Default(pel,2)         # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
    Chroma   = Default(chroma,True)   # MAnalyse chroma arg. If set to true, use chroma in block matching.
    BlkSz    = Default(BlkSz,8)       # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps  better, esp for HD clips. Maybe also better where BIG noise.
    OLap     = Default(OLap, BlkSz/2) # Default half of BlkSz.
    Tm       = Default(tm,True)       # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
    Glob     = Default(glob,True)     # Default True, Allow set MAnalyse(global) independently of TrueMotion.
    bBlur    = Default(bblur,0.6)     # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
    HasDC    = dc.Defined             # bblur ignored if HasDC, ie user already provided prefiltered clip.
    Assert(1 <= RadT,myName + " 1 <= RadT")
    Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
    Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
    pad = max(BlkSz,8)
    sup = (HasDC ? dc : bBlur>0.0  ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
    # Only 1 Level required where not MAnalyse-ing.
    sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
    MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
    c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
    MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT)  # Temporal median blur only [not spatial]
    SelectEvery(RadT*2+1,RadT)                                             # Return middle frame
}

[johnmeyer]

Zorr, thanks for the updated information. I had to complete the project, so I did the old-fashioned thing of creating two restored clips: one with dirt removal, and the other identical in every way, but with the call to RemoveDirtMC removed. I then laboriously go through the entire game, and any time the ball is hiked, kicked, fumbled, or thrown, I cut over to the version without dirt removal.

I still have the project all set up, so I'll try again tomorrow, starting with the settings you provided, and then attempting to tune the script using the information you provided.

[coolgit]

Have you ever tried reducing ThSAD = Default(ThSAD,10000) to ThSAD = Default(ThSAD,6000) and Tm = Default(tm,True) to Tm = Default(tm,False)?
BlkSz,8 should be 12 or 16

I find that reducing ThSAD helps with motion analysis but too low and big spot, say size of 2 pence coin or more doesn't get wholly remove. tm,False is good for motion analysis but poorer at removing big spots.

Having used this for 2 months now I've now understood the limitations and able to detect them when scrolling through a video.

I create two files.

1 return spotless with
RadT = 1
ThSAD = 6000
Pel = 2
BlkSz = 16
OLap = (BlkSz/2)
Tm=true
Bblur = 0.3

This will remove the defects but poor at motion when close to camera. Knowing these limitations I use 2nd file.

2 return delta restore with
RadT = 1
ThSAD = 6000
Pel = 2
BlkSz = 16
OLap = (BlkSz/2)
Tm=false
Bblur = 0.3

I can run both script at the same time since vdub is piss poor at multithreading so 2 instances of vdub doesn't affect each other performance.

open 3 instances of vdub. 1 for each above files and the original file.
1 extract all spotless frames and go through spotless file for scenes that has excessive motion and close to camera motion.
2 extract those delta restore frames to replace bad spotless frames. Check no defects before extracting.
3 extract original frames to replace bad frames from the above 2. Rare. Case examples are scenes where there are white flashes etc from weapon fire or explosion. Check no defects before extracting.

Put extracted frames together and it will be 99.5% fixed. When I watch the file before encoding i detected a massive defect like over 50 pixel wide and 100 or so pixel length on a 720x576 vid.

At the beginning an episode would take 2/3 days. Spotless got that down to 12 hours. Now with the above I can do an episode in under 3 hours. That includes fixing colours, contrasts etc with photoshop.

[johnmeyer]

The ThSAD numbers are extraordinarily high, so that was one of the first things I reduced.

As for your "1, 2, 3" suggestion, that is a much more complicated way of doing what I've been doing in my NLE for many years now (i.e., cutting between two versions of the restoration). It's a LOT easier doing it in Vegas, especially because I can use Vegas' scripts to automate much of the cutting.

[StainlessS]

ThSAD may be extraordinarily high, but,
If src block is bad, then low ThSAD will likely use the bad src block when doing median

    ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
        ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
        ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
        In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
        greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
        in the result MCompensated frame].
        Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
        block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
        or some other mixture.
          A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
        block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
        after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
        threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
        Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
          It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
        no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
        middle pixel value.
        I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
            SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
        And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
        as we still had the temporal median filter to fall back on and pick the middle pixel value.

        From mvtools2 docs:
          ThSAD is SAD threshold for safe (dummy) compensation.
              If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

[coolgit]

Originally Posted by johnmeyer

The ThSAD numbers are extraordinarily high, so that was one of the first things I reduced.

As for your "1, 2, 3" suggestion, that is a much more complicated way of doing what I've been doing in my NLE for many years now (i.e., cutting between two versions of the restoration). It's a LOT easier doing it in Vegas, especially because I can use Vegas' scripts to automate much of the cutting.

I find it easier since I am good with Photoshop but zero skills using film editing software and I can't be arsed to learn it. Photoshop has scripts like 'action' command which automate the processes and image sequences (batch processing), overnight when necessary.

Each to their own I suppose.

[johnmeyer]

I tried the modified script given in post #166. The good news is that it restored the ball in flight. The bad new is that it also restored almost all of the dirt.

I'll spend a little more time tweaking parameters, but for me, I don't think this is going to be useful.

[coolgit]

Originally Posted by StainlessS

ThSAD may be extraordinarily high, but,
If src block is bad, then low ThSAD will likely use the bad src block when doing median

    ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
        ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
        ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
        In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
        greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
        in the result MCompensated frame].
        Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
        block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
        or some other mixture.
          A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
        block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
        after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
        threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
        Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
          It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
        no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
        middle pixel value.
        I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
            SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
        And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
        as we still had the temporal median filter to fall back on and pick the middle pixel value.

        From mvtools2 docs:
          ThSAD is SAD threshold for safe (dummy) compensation.
              If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

A problem with above is, if a low ThSAD and current_frame block is mostly noise

I denoise and temporal smooth my vid after deinterlace, before spotless, hence i can do 6,000 without any problem. At the beginning when I didn't denoise first, lowering ThSAD was an issue sometimes.

I did some experiments and kept lowering ThSAD to 500. It helped with motion, no missing hand for example, but removed almost zero spots of any kind. Kind of fun seeing how using different values works.

Last restoration act before encoding, I add noise/grain back in, obviously not as much as the original.

[coolgit]

Originally Posted by johnmeyer

I tried the modified script given in post #166. The good news is that it restored the ball in flight. The bad new is that it also restored almost all of the dirt.

I'll spend a little more time tweaking parameters, but for me, I don't think this is going to be useful.

Similar issue with table tennis ball (white). If a ball only exist in 1 frame and none previous/next then it would be a spot. Any sport with a ball in motion is going to be in one location then another location too far away for accurate motion analysis. I had the same when an arm moved fast in 5 frames toward the camera. Half of the arm and hand disappeared.

I do think there is a need to be a more specialise script when dealing with ball in sports.

[real.finder]

if someone don't mind manual work there are

string outfile = ""

Nname of a generated file containing the detected spot locations. The file is in the .ASS subtitle file format. Spots are located with their bounding boxes drawn as magenta rectangles (dilated by 2 pixels in each direction relative to the actual bounding boxes, for clarity). The default Outline style can be later changed to Mask in an ASS editor (like Aegisub) to produce a mask made of rectangles filled with white. The file is overwritten at each filter run, so make a copy before editing it!

in DeSpot (3.6.2.0 or better), I think it can be used to mask only the dirts

[zorr]

Originally Posted by johnmeyer

I tried the modified script given in post #166. The good news is that it restored the ball in flight. The bad new is that it also restored almost all of the dirt.

This is a different video than the one I adjusted the parameters for. The "torture clip" was in this post, right?

Thanks for posting the screenshots, it's easy to see what is wrong. The yellow chroma mask fills the whole screen (not sure why) which means it will take everything from the original source and nothing from the Spotless-filtered version. Since this looks to be a black and white clip chroma mask is going to be useless.

The ball and the dust are the same size and have pretty much the same brightness and sharpness, I think automatic masking is not going to work here.

[johnmeyer]

Originally Posted by coolgit

Similar issue with table tennis ball (white). If a ball only exist in 1 frame and none previous/next then it would be a spot. Any sport with a ball in motion is going to be in one location then another location too far away for accurate motion analysis. I had the same when an arm moved fast in 5 frames toward the camera. Half of the arm and hand disappeared.

I do think there is a need to be a more specialise script when dealing with ball in sports.

I think you are correct, and I don't think Zorr or anyone else should spend more time trying to perfect the masking technique, at least not for my situations.

Another big contributing factor to the masking failure in my example above and in my "torture clip" is the low frame rate of most amateur film. The American football game shown in my screenshot above is 24 fps, but most amateur film is 16 or 18 fps. The temporal gap between frames at these fps speeds is such that a thrown ball will have no spatial overlap with the ball in the previous frame, and thus will behave just like a dirt spot (i.e., it only appears in one location for one single frame, without any way to tie it to the previous frame).

Which brings me to the one thing that WOULD be useful: a luma parameter for dirt spots. I know this has been requested before, and I think someone did some work on it, but I don't know if it ever made its way into the algorithms.

Dirt on film prints is always near-black. Speckles on video are almost always near-white. A ball is almost never as dark as the dirt (although in my example above, the dirt is gray, not black). Thus, if there was a way to ignore any "object" that didn't contain at least "n" pixels less than a luma threshhold, that would help a lot.

[zorr]

I think there's a little misunderstanding of how Spotless + Delta Restore works. What coolgit mentioned about the ball moving too fast for motion analysis is true - but it only affects Spotless. Delta Restore does not use any motion analysis, it only cares about the differences between original and the Spotless (or any other noise removal method) processed version.

As long as there are some consistent differences in brightness, chroma, sharpness and/or the size of the spots between real features and dust then Delta restore can restore mostly features and not the dust.

And what John just requested is one of the features of Delta Restore.

[johnmeyer]

OK, I get it. Delta Restore is actually doing what I asked, and doing it in a rather advanced way, using more than just luma to decide which things should not have been removed. I'll try tweaking one more time.