How to denoise/despot old cartoon footage?

[kypec]

I've an TV capture from late 80's encoded with Xvid as AVI.
Here is a short sample:
http://www.mediafire.com/file/gtzldnmdxjg/sample.avi
I don't know how to get rid of all that spots, tried MCTemporalDenoise like this:
# Initialize variable with path to my A/V tools
avdir=GetSystemEnv("PROGRAMFILES")+"\AVTools\"
# Load all the plugins that will be used in the script
Import(avdir + "plugins\MCTemporalDenoise.v1.4.16.avsi") # MCTemporalDenoise (v1.4.16)
LoadPlugin(avdir + "plugins\mvtools2.dll") # MVTools (v2.5.10)
LoadPlugin(avdir + "plugins\mt_masktools-25.dll") # Masktools (v2.0a42)
Import(avdir + "plugins\LSFmod.v1.9.avsi") # LSFmod (v1.9)
LoadPlugin(avdir + "plugins\FFT3DFilter.dll") # FFT3Dfilter (v2.1.1)
LoadPlugin(avdir + "plugins\TTempSmooth.dll") # TTempsmooth (v0.9.4)
LoadPlugin(avdir + "plugins\RemoveGrainSSE2.dll") # Removegrain (v1.0PR)
LoadPlugin(avdir + "plugins\RepairSSE2.dll") # Repair (v1.0PR)
# Deblock (v1.2)
# Deblock_QED (18.aug.2008)
# DctFilter (v0.0.1.4)
LoadPlugin(avdir + "plugins\EEDI2.dll") # EEDI2 (v0.9.2)
LoadPlugin(avdir + "plugins\SangNom.dll") # Sangnom (v1.0beta)
Import(avdir + "plugins\GradFun2DBmod.v1.5.avsi") # GradFun2DBmod (v1.5)
LoadPlugin(avdir + "plugins\gradfun2db.dll") # Gradfun2db (v1.0)
LoadPlugin(avdir + "plugins\AddGrainC.dll") # AddgrainC (v1.4)

# Open source video clip
AVISource("D:\SOURCE\sample.avi",audio=false)
untouched=last
MCTemporalDenoise(settings="high")
StackHorizontal(untouched,last)
It removed some of the tiny dark spots and smoothed large uniformly coloured areas of picture quite nicely but...most of the bigger spots are still there! :(
Is there any filter available which would erase that kind of noise?
I'd noticed that all spots occur just in one frame so I think it could be fairly easy to somehow remove them going by temporal approach, or am I thinking wrong?
All other filters that I've tried (UnDot, DeGrainMedian, DeSpot) enable to tune mostly spatial parameters like sizes, thresholds, distances with use of motion compensation but I couldn't get satisfactory results with any of them.
Please advice :thanks:

[Guest]

[bump after thread re-opened]

[kypec]

Wow, more than 170 views and not a single reply :( Is my source really that bad that no one is able to come with at least a suggestion?

[2Bdecided]

MVDegrainMulti?

This is just a guess - not sure it's ideal as a "just remove what's different in one frame only" filter, but it's not bad.

Cheers,
David.

[Didée]

The task of spot removal has not been solved yet. The name of the game is "choose your poison" : There are lots of possibilities to try, but still, there is no solution. You can only choose between different compromises:

a) Less spot removal with only little damage to original content,
-or-
b) more spot removal with more damage to original content.

Choose your poison.

I could write you a survey about the plentitude of different problems ... would take me two hours, take up two/three/four pages of text, and at the end the conclusion still would be the same: "no solution, only compromises. Choose your poison."


Quick headlines:

- no motion compensation: weak approach, in motion areas it's either ineffective or dangerous

- with motion compensation: theoretically much better, but ... the spots will mislead the motion engine upon deciding if a compensation is "good" or "bad". The larger the spots, the bigger the problem.

- Temporal averaging: in theory it's the "wrong" approach. A spot will get averaged into clean neighbor frames. Also a spot will not be removed, just cut-down in intensity. To be effective, it needs LOTS of temporal neighbors to consider, which in turn is a problem in other aspects.

- temporal median: in theory the "method of choice" for spot removal - does clean removal of spots, no spreading into neighbor frames, needs only a small temporal neighborhood. However, IF something goes wrong (from the side of MVTools in the first place), then it will produce artifacts more worse than those of temporal averaging.

[Terka]

has anyone some experience with:
http://spotremoverfilter.com/

[Didée]

Me not. But if it has to be despite all problems, then a deeper look into kassandro's RemoveDirt is worthwile, and recommended. ;)

[Terka]

many of the frames are duplicate, could this be somehow useful?

[Didée]

Dunno ... the dup's might help to some extend, but they're not frequent enough that you could rely on them.

For what it's worth, here is a possible primitive for spot removal:

AviSource("D:\sample_spotty.avi")

o=last ox=o.width() oy=o.height()

osup = o.MSuper(pel=2,sharp=2)
bv1 = osup.MAnalyse(isb=true, delta=1,blksize=8,overlap=4,search=4)
fv1 = osup.MAnalyse(isb=false,delta=1,blksize=8,overlap=4,search=4)
bc1 = o.MCompensate(osup,bv1)
fc1 = o.MCompensate(osup,fv1)

Interleave(fc1,o,bc1)
MedianblurT(0,0,0,1) # you can also use "Clense(reduceflicker=false)" instead
SelectEvery(3,1)

StackHorizontal(o,last)

[kypec]

Thank you very much Didée, your approach did wonders to my source. I had to replace MedianBlurT with Clense (hopefully you mentioned that as well) because the encoding process kept crashing at the very end. First I thought that it's x264 latest revision to blame (r1613) but the error persisted even with r1538. Once I found the offending part in this filter chain and replaced it the encoding finished always fine.
Here is my complete script, I tried to smooth big uniformly coloured areas (sky, water surface) even more with FFT3DFilter but didn't see any effect really so in the end I left it out completely.
# Open source video clip
AVISource("D:\SOURCE\Family-Ness\Rodina Nessovcov.avi",audio=false).Trim(0,7599)
# Initialize variable with path to my A/V tools
avdir=GetSystemEnv("PROGRAMFILES")+"\AVTools\"
# Load all the plugins that will be used in the script
LoadPlugin(avdir + "plugins\mvtools2.dll") # MVTools (v2.5.10)
#~ LoadPlugin(avdir + "plugins\FFT3DFilter.dll") # FFT3Dfilter (v2.1.1)
LoadPlugin(avdir + "plugins\aWarpSharp.dll") # aWarpSharp (2009.06.19)
#~ LoadPlugin(avdir + "plugins\medianblur.dll") # MedianBlur (v0.84)
LoadPlugin(avdir + "plugins\DePan.dll") # DePan (v1.10.1)
LoadPlugin(avdir + "plugins\DePanEstimate.dll") # DePanEstimate (v1.9.2)
LoadPlugin(avdir + "plugins\RepairSSE2.dll") # Repair (v1.0PR)
LoadPlugin(avdir + "plugins\RemoveGrainSSE2.dll") # RemoveGrain (v1.0PR)
LoadPlugin(avdir + "plugins\RemoveGrainTSSE2.dll") # RemoveGrainT (v1.0PR)
Import(avdir + "plugins\Stab.avsi") # Stabilize script (v1.0)

# remove single-frame spots
o=last
osup = o.MSuper(pel=2,sharp=2)
bv1 = osup.MAnalyse(isb=true, delta=1,blksize=8,overlap=4,search=4)
fv1 = osup.MAnalyse(isb=false,delta=1,blksize=8,overlap=4,search=4)
bc1 = o.MCompensate(osup,bv1)
fc1 = o.MCompensate(osup,fv1)
Interleave(fc1,o,bc1)
#~ MedianblurT(0,0,0,1) # you can also use "Clense(reduceflicker=false)" instead
Clense()
SelectEvery(3,1)
# stabilize subtle shaking of whole image
Stab(8,10,10)
# crop and resize
Crop(8,2,-0,-0).Spline36Resize(640,480)
#~ FFT3DFilter(plane=3, bt=3, sigma=3, bw=16, bh=16, ow=8, oh=8) # denoise chroma
aWarpSharp2(thresh=160, blur=3, type=0, depth=16, chroma=4) # sharpen the image, make it crisp and less blurry
#~ StackHorizontal(o,last)


The resulting MP4 (AVC+AAC-HE) is only 41% size of original AVI (ASP+MP3) with better visual quality, isn't that simply amazing? :p

[kypec]

has anyone some experience with:
http://spotremoverfilter.com/
I've downloaded and tried that in VDub. I couldn't see anything useful in the output preview. Nothing for the first hundred frames and some debug mode later on with lots of orange and red rectangles, triangles running all over the frame. Only then I realized this is a payware with trial version being awfully crippled I guess.
Author's example screens looked promising though:mad:

[GMJCZP]

I have made Didée's idea to remove spots:

# Function KillerSpots 3.1
# For primitive videos (live action/animation)
# Function for spot removal
# Original idea by Didée (https://forum.doom9.org/showthread.php?p=1402690#post1402690)
# Developed by GMJCZP
# With adaptation of johnmeyer's revision of RemoveDirt function
# (See RemoveDirt documentation and Film restoration script by videoFred)
# Requirements: MVTools, RGTools, RemoveDirt

Function KillerSpots(clip o, int "limit", bool "advanced",bool "anim", int "repmode", bool "comp")
{
advanced = Default(advanced, true) # best performance, otherwise is the original Didée's idea (for live action is better)
comp = Default(comp, false) # for compare results

# Options for advanced = true:

limit = Default(limit, 10) # spot removal limit. Higher values, more killing spots but can hurt details
anim = Default(anim, true) # for animated content, otherwise is live action
repmode = Default(repmode, 2) # repair mode for preserve details: repmode=1 for classic Repair, repmode=2 for temporal repair
# Temporal repair is more safe (limit values can be more highs) but more spots can reappear

Assert((repmode == 1 || repmode == 2), "repmode error, values of 1 or 2 are only admitted")

# Didée's idea (if advanced=false, with direct Clense method for killing spots)
osup = o.MSuper(pel=2,sharp=2)
bv1 = osup.MAnalyse(isb=true, delta=1,blksize=8,overlap=4,search=4)
fv1 = osup.MAnalyse(isb=false,delta=1,blksize=8,overlap=4,search=4)
bc1 = o.MCompensate(osup,bv1)
fc1 = o.MCompensate(osup,fv1)

Interleave(fc1,o,bc1)

advanced ? RemoveDirtMod(limit,anim,repmode) : Clense()

SelectEvery(3,1)

comp ? StackHorizontal(o,last) : last
}

# From function RemoveDirt, original adaptation thanks to johnmeyer
Function RemoveDirtMod(clip input, int "limit", bool "anim", int "repmode")
{
d = anim ? 1 : 3
dm = anim ? 1 : 2
pt = anim ? 4 : 6
ct = anim ? 6 : 8
clensed = input.Clense()
alt = input.RemoveGrain(dm)
restore = (repmode==2) ? TemporalRepair(clensed, input, mode=4) : Repair(clensed, input, mode=16)
return RestoreMotionBlocks(clensed,restore,alternative=alt,neighbour=input,pthreshold=pt,cthreshold=ct, gmthreshold=40,
\ dist=d,dmode=dm,debug=false,noise=limit,noisy=12)
}

[manono]

I have made Didée's idea to remove spots

It seems to work well. Thanks a lot. Might I suggest a name change? When I'm looking to use a spot remover and I'm going down my list of functions. 'Primitive' just doesn't jump out as a spot remover. Since 'RemoveSpots' and 'RemoveDirt' are already taken, then how about 'SpotRemover'? Or maybe you can suggest a different but still relevant name? CrudBeGone. SpotEx. CleanUp. DirtClean. Your choice. :)

Yeah, I know. I can just change it myself. But it's your filter (and Didée's).

[StainlessS]

Just had something jump into my head, hows bout "SpotLess()" ? :)

EDIT: From my pop-up dictionary, "Completely neat and clean".

[GMJCZP]

It seems to work well. Thanks a lot. Might I suggest a name change? When I'm looking to use a spot remover and I'm going down my list of functions. 'Primitive' just doesn't jump out as a spot remover. Since 'RemoveSpots' and 'RemoveDirt' are already taken, then how about 'SpotRemover'? Or maybe you can suggest a different but still relevant name? CrudBeGone. SpotEx. CleanUp. DirtClean. Your choice. :)

Yeah, I know. I can just change it myself. But it's your filter (and Didée's).


manono, I named it because Didée used that expression and I wanted to respect it, in addition the "primitive" videos have these characteristics.

Let's do one thing, I'll modify the name for another one I had already thought of, but in the description I'll put some of this, lol.

PD: TinMan, I hope this function is always SpotLess ;-)

[manono]

SpotLess. Good one, StainlessS.

[Busty]

wow, this seems to work really great.

Although, I notice that the result is somewhat softer than the input.

Can this script be tweaked to retain the original sharpness / detail?

[johnmeyer]

Given who suggested this name, perhaps it should be "SpotLessS"?

[GMJCZP]

Given who suggested this name, perhaps it should be "SpotLessS"?

Lol*1000.

[GMJCZP]

Now to the function KillerSpots I have added a "advanced" mode, which produces better performance (at least I noticed in a test video) and also makes the function more configurable when defining a "limit" parameter, which by the way Is totally equivalent to the parameter present in the RemoveDirtMC function developed by johnmeyer.

[johnmeyer]

Just to be clear, while I'd love to take creidt for RemoveDirtMC, I did not develop it. I did, however, re-discover it and found that it was amazing for removing dirt from film transfers. I then spent quite a bit of time tweaking parameters for that particular purpose (and also developed set of hyper-aggressive settings for Polavision instant movie film).

[GMJCZP]

No problem John, the function I met thanks to you, in which you worked hard. I actually put "adapted".
:)

[StainlessS]

Didée post earlier in this thread.


For what it's worth, here is a possible primitive for spot removal:

AviSource("D:\sample_spotty.avi")

o=last ox=o.width() oy=o.height()

osup = o.MSuper(pel=2,sharp=2)
bv1 = osup.MAnalyse(isb=true, delta=1,blksize=8,overlap=4,search=4)
fv1 = osup.MAnalyse(isb=false,delta=1,blksize=8,overlap=4,search=4)
bc1 = o.MCompensate(osup,bv1)
fc1 = o.MCompensate(osup,fv1)

Interleave(fc1,o,bc1)
MedianblurT(0,0,0,1) # you can also use "Clense(reduceflicker=false)" instead
SelectEvery(3,1)

StackHorizontal(o,last)


Nifty mod, SpotLess()

Function SpotLess(clip c,int "RadT",int "ThSAD2") {
/*
SpotLess v1.0. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2, Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.

NOT FOR cartoon/anime, live video only, sorry.

RadT, Range 1 -> 6, Default 2. Removes Spots on up to RadT consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring [no problems encountered so far at max RadT=6].
ThSAD2, 400 <= ThSAD2 <= 10000, Default 10000=NEARLY OFF. Lower removes fewer spots, but less chance of blurring.
Reduce if produces blurred frames, absolute minimum of maybe about 400, probably pretty good as is.
Lower reduces effect of bad non matching blocks, reduced more so on blocks from more distant radii frames.
ThSAD for frame @ distance 1 defaults to 10,000 [and we let it default], ThSAD2 sets the SAD [Sum of Absolute Differences] threshold
for most distant frame at distance RadT, with those frames that are distances in-between being given a SAD threshold linearly spread between the two.
Wild guess that maybe about 10000 at RadT=1, and 5000 (10000/2) where RadT=6, might be a good compromise if concerned about blurring.
Suggested, based on wild guesses above :) [if you experience blurred results, otherwise leave as is]
Where below RadT is 6, and ThSAD2=5000, then blocks @ frames current_frame +- 6 gets weighted SAD threshold value of 5000 (same as ThSAD2),
@ current_frame +- 3, gets distance weighted SAD threshold value 8000, etc
Radius SAD threshold : Where RadT=1, SAD threshold=ThSAD2. Where RadT>1; SAD threshold=(RadT-Radius)*((10000-ThSAD2)/(RadT-1))+ThSAD2; (6-Radius)*((10000-5000)/(6-1))+5000.
1 10000 10000, Almost switches OFF SAD thresholding at frame distance 1. [Ignores blocks of only really high SAD differences]
2 9000
3 8000 In between frames acquire SAD threshold value linearly between 10000 @ distance 1, and ThSAD2[here 5,000] at distance RadT[here 6].
4 7000
5 6000
6 5000 Where RadT=6 and ThSAD2 = 5000, so frame @ distance 6 gets SAD threshold value of ThSAD2[5000]
And to weight similar to above where eg RadT=3, set RadT=3, ThSAD2=8000 to set as above Radius=3,SAD threshold=8000. Radii 2 and 1 will acquire 9000 and 10000 resp.
From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).
ThSAD2:
Defines the SAD soft threshold for the furthest frames.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame, EDIT: Default 10000)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
In SpotLess, we dont actually set mvtools ThSAD arg, only ThSAD2, ThSAD will default 10000, but where RadT=1 and you set ThSAD2 to eg 5000 it should (I hope)
set soft SAD threshold at radii 1 to ThSAD2 [and override the ThSAD default 10000].
If anybody can explain above any better, then please do. I hate doing docs and as you can see, I aint very good at it.
*/
RadT = Default(RadT,2) # Temporal radius
ThSAD2 = Default(ThSAD2,10000) # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks.
Assert(1 <= RadT <= 6,"SpotLess: 1 <= RadT <= 6")
sup = c.MSuper(pel=2,sharp=2)
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=8,overlap=4,search=4)
c.MCompensate(sup,MultiVec,tr=RadT,ThSAD2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


Hacked this for a clip with spots on double frames, did not know if would be much better, but as it turns out is fandabbydozy.

EDIT: MvTools2/MedianBlur2 by Pinterf, so AVS+ colorspace and x64 compatible.
EDIT: I'm not using for cartoons, ie live video [Old B&W WW2 footage].
EDIT: Chosen function name would seem to be a very good choice, not perfect, but very nearly is.
EDIT: BugFix, + Mod RadT up to 6, more than 2 probably overkill but the choice is there.

Now has its own threaad here:- https://forum.doom9.org/showthread.php?p=1921773#post1921773

EDIT:
Perhaps someone would like to test SpotLess() on 'old cartoon' which is subject of this thread.
I dont usually touch cartoon/anime at all, and even less 'old cartoon'.
I not sure that Mvtools copes well with cartoon at all.

It doesn't work on anime, it wiped out all moving objects in motion scenes.

So not much good for cartoon/anime, sorry.

[ChaosKing]

You should also try spotless as a prefilter in smdegrain. Worked very good for me with almost zero artifacts left from spotless.

[StainlessS]

I'm sure that RealFinder will give it a try.

[StainlessS]

Think may need arg for setting MCompensate(ThSad) esp where at scene change [and RadT=3, can produce bad frame).

thSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

Not sure what to set it [ThSAD] at, nor if should set separate ThSAD for 2nd vector pair and also for 3rd vector pair compensations.
Think we dont need ThSAD setting for 1st MCompensate vector pair.
Any ideas ??? [10,000 is OFF, what are usable values]

EDIT: Hang on, Just fixed bug in SpotLess, gotta test it again. Fixed in prev post #23.

[StainlessS]

OK, Spotless() update. think problem was mostly/all self-inflicted, fixed.
Post 23 update.

I'm not seeing any problems so far with this one, even at RadT=6 [and ThSAD2=10,000 ie nearly OFF].

EDIT:
From recent MvTools2, (I added ThSAD2 arg)

thSAD2

Define the SAD soft threshold for the furthest frames.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2.
Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error
and the risk of bluring when the result of MCompensate is passed to a temporal denoising filter.

I Luv this filter. https://www.cosgan.de/images/smilie/liebe/a005.gif
So much nicer than that horrid GMJCZP KillerSpots().

[real.finder]

I'm sure that RealFinder will give it a try.
yes :)

You should also try spotless as a prefilter in smdegrain. Worked very good for me with almost zero artifacts left from spotless.

do you have a video sample?

[StainlessS]

do you have a video sample?
SendSpace, below in sig. VTS_01_1.demuxed.m2v 30 secs.


Mpeg2source(".\VTS_01_1.demuxed.d2v")
ORG=Last
SpotLess(RadT=2)
StackHorizontal(ORG,Last)

[real.finder]

SendSpace, below in sig. VTS_01_1.demuxed.m2v 30 secs.


Mpeg2source(".\VTS_01_1.demuxed.d2v")
ORG=Last
SpotLess(RadT=2)
StackHorizontal(ORG,Last)


:thanks:

yes, it's do better job than any methods, I was using this

RemoveDirtMC_SE(twopass=false,dct=4,mcthSAD=2800,resrepmode=16,repmode=0,clmode=0,rstr=1.5,PreNR=FFT3DFilter(sigma=4,plane=4,bw=16,bh=16,ow=8,oh=8,bt=3,wintype=1).ColorYUV(autogain=true).TemporalDeGrain_SE(5,9))

[StainlessS]

OK, me thinks I set description of RadT wrong, was probably correct-ish the first time around. This bit.

RadT Range 1 -> 6, Default 2. RadT > 2 will usually be overkill. Setting too high could remove fast moving non spot objects.
RadT = 1: Remove spots on single frame only.
RadT = 2: Remove spots on up to 3 consecutive frames. Seems pretty reliable. [Default]
RadT = 3: Remove spots on up to 5 consecutive frames. Possibly many false spots removed. [Needs testing]
RadT = n: Remove Spots on up to 2n-1 consecutive frames.

As it is Median, and number of frames spanned is ODD and 2*RadT+1 (the 1 being current frame), so max number of frames where spot can occur is
maybe I think RadT (as I previously stated), with max bad count being RadT and good having to be at least RadT+1 to fix it.

So number of consecutive frames on which spot can occur is RadT.

Would that sound about right to anybody ? [if so I'll mod description as so]

EDIT: @ 05:30AM this morning, I thought would fix all between outer frames, which is obviously wrong for median.

EDIT: Maybe summick like this

RadT Range 1 -> 6, Default 2. Removes Spots on up to RadT consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly remove fast moving non spot objects.
400 <= ThSAD2 <= 10000, Default 10000=NEARLY OFF. Lower removes fewer spots, but less chance of removing non spot object.
Reduce if produces bad frames, absolute minimum of maybe about 400, probably pretty good as is.
Lower reduces effect of bad non matching blocks, more so on blocks from more distant MC frames.
ThSAD for frame @ distance 1 is always 10,000, ThSAD2 sets ThSAD for most distant frame at distance RadT, with those frames
that are distances in-between being given a value linearly spread between the two.
Wild guess that maybe about 10000 at RadT=1, and 5000 (10000/2) where RadT=6, might be a good compromise if concerned about it.
Suggested, based on wild guesses above :) [if you experience bad results, otherwise leave as is]
Where RadT below is 6, and ThSAD2=5000, then blocks @ frames current_frame +- 6 gets weighted ThSAD value of 5000 (same as ThSAD2),
@ current_frame +- 3, gets distance weighted ThSAD value 8000, etc
Radius ThSAD : Where RadT > 1 :: (RadT-Radius) * ((10000-ThSAD2)/(RadT-1)) + ThSAD2 :: (6-Radius) * ((10000-5000)/(6-1)) + 5000
1 10,000
2 9,000
3 8,000 In between frames get ThSAD value between 10000 @ distance 1, to ThSAD2 at distance RadT.
4 7,000
5 6,000
6 5,000 When RadT=6 and ThSAD2 = 5000, so frame @ distance 6 get ThSAD value same as ThSAD2
And to weight similar to above where eg RadT=3, then set ThSAD2=8000.

Does above make sense ? [suggestions, I'm lousy at explaining things]

[Boulder]

This is a wonderful tool. As it happens, I've been cleaning a VHS cap and the silly video has quite a lot of spots and scratches in every other field. SpotLess cleans them up nicely and also takes care of most of the chroma noise too.. I haven't checked things with longer samples but looks good anyway.

[StainlessS]

Yeh but does above make any sense[EDIT: post #31], please comment. [the ThSAD2 stuff]

I'm considering seeing if I can add sharpening, as per McDegrainSharp (optional), maybe could avoid doing 2nd pass with McDegrainSharp() func.
Might not work as required, but I'm gonna have a play, if it dont get added, you know it did not work very well.

EDIT: Also, would SmDegrain/McDegrain be required after SpotLess, it does a pretty good job of removing spots, but also
noise and temporal grain too.
EDIT: OK, maybe SmDegrain/McDegrain would be of benefit I guess, this func does a median [middle value] rather than mean
which smdegrain/mcdegrain would do.

[StainlessS]

This was the original purpose in knocking up Spotless,
EDIT: Update here:- https://forum.doom9.org/showthread.php?p=1923123#post1923123

### DGIndex Template, HyBrid_PAL.avs
#VideoFileName ="__vid__"
#AudioFileName ="__aud__"
#AudioDelay =Value("__del__")

# Filled in by DGIndex
VideoFileName ="D:\DVD\PS1\VIDEO_TS\PS1.d2v"
AudioFileName ="D:\DVD\PS1\VIDEO_TS\PS1 T80 2_0ch 192Kbps DELAY 0ms.ac3"
AudioDelay =Value("0.000")

###############
# PAL Hybrid Interlaced/Progressive mix, output to singlerate or doublerate.
###############
MPEG2Source(VideoFileName)
AudioExt=RT_GetFileExtension(AudioFileName)

Audio= (AudioExt==".ac3") ? NICAC3Source(AudioFileName,channels=2,DRC=0)
\ : (AudioExt==".mpa"||AudioExt=="mp1"||AudioExt==".mp2"||AudioExt==".mp3") ? NicMPG123Source(AudioFileName,Normalize=False)
\ : (AudioExt==".wav") ? RaWavSource(AudioFileName)
\ : (AudioExt==".dts") ? NicDTSSource(AudioFileName)
\ : 0

Assert(!isInt(Audio),"NO AUDIO")

(!isInt(Audio)) ? AudioDub(Audio).DelayAudio(AudioDelay).Trim(0,0) : NOP # Trim, chop/add audio to length

(!isInt(Audio) && AudioRate() <> 44100) ? ResampleAudio(44100) : NOP
####### CONFIG ##########

AssumeTFF # Whatever (check) : Almost always TFF for PAL DVD. Default field order is BFF.

OUTPUT = 0 # 0=SingleRate 25FPS Progressive. Progressive src frames as input, interlaced src frames are even frames from dbl rate deinterlacing.
# 1=SingleRate 25FPS Hybrid. Progressive src frames as input, interlaced src frames dbl rate deinterlaced, reinterlaced to 25FPS after processing.
# 2=DblRate 50fps. Prog=ChangeFPS. Progressive src frames doubled with ChangeFPS, interlaced src frames are dbl rate deinterlaced.
# 3=DblRate 50fps. Prog=Interpolate. Progressive src frames double rate interpolated with JohnFPS [ie MvTools], interlaced src frames are dbl rate deinterlaced.

UNSPOT_P=True # Use SplotLess() on Progressive parts to remove spots and noise.
UNSPOT_I=True # Use SplotLess() on Interlaced parts to remove spots and noise.

RADT = 2 # SpotLess temporal radius (spots on 2 consecutive frames)
ThSAD2 = 10000 # Almost OFF.


FRAMES_P=1 # Progressive McDegrainSharp(frames=FRAMES_P), (0=Dont McDegrainSharp)
CSHARP_P=0.6

FRAMES_I=2 # Interlaced McDegrainSharp(frames=FRAMES_I), (Maybe double FRAMES_P for detail migration, although max possible for McDegrainSharp=3). (0=Dont McDegrainSharp)
CSHARP_I=CSHARP_P

# IsCombedTIVTC args, all at defaults.
CTHRESH = 10 # Area combing threshold.
# Valid settings are from -1 (every pixel will be detected as combed) to 255 (no pixel will be detected as combed).
# This is basically a pixel difference value. A good range is between 8 to 12.
MI = 85 # Number of combed pixels inside any of the blocky by blockx size blocks on the frame for the frame to be detected as combed.
CHROMA = true # Whether or not chroma is considered in the combed frame decision. [We want chroma as interlaced parts in color]
BLOCKX = 16 # X-axis size of the window used during combed frame detection.
BLOCKY = 16 # Y-axis size of the window used during combed frame detection.
# So in any single block, if 85[MI] out of 256[BLOCKX x BLOCKY] pixels pass CThresh as combed, then is Interlaced.

# Deinterlace mode
BOBMOD=1 # 0=BOB, 1=YadifMod2, 2=QTGMC : QTGMC here is way memory/CPU intensive, req x64. Suggest YadifMod2

BOB_FUNC_0 = """Bob()""" # Cheap & cheerful Doublerate simple Bob.
BOB_FUNC_1 = """YadifMod2(Order=-1,Mode=1,edeint=NNedi3(Field=-2, nns=2))""" # Doublerate Yadifmod2 filter.
# # Field-2=double rate, internal parity value to start : Order-1=Internal Parity. Mode 1=double rate do spactial check (3=no spatial check).

BOB_FUNC_2 = """QTGMC(Preset="Slow",NoisePreset="Fast",SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1, tr2=0)""" # Doublerate QTGMC filter.

FUNC = Select(BOBMOD, BOB_FUNC_0, BOB_FUNC_1, BOB_FUNC_2)
FNAME = Select(BOBMOD,"BOB", "YADIF", "QTGMC")


PRE_FETCH=4 # 1 = No Prefetch, Else eg 4 = Prefetch(4) ie MT mode

### # Display stuff
SUBS = True # Show where Deinterlaced/ReInterlaced
STACK = True # Show Src and Result
TITLE = False # Show Title Bar
TITNUM = True # Show Frame Number on Title Bar (Only if TITLE)
FINAL = False # True, switch off SUBS, STACK, and Title

##### End Config ##########

Src=Last

SUBS = (!FINAL&&SUBS)
TITLE = (!FINAL&&TITLE)
STACK = (!FINAL&&STACK)

PreProc = Src.SpotLess(Radt=RADT,thsad2=ThSAD2) # Remove Spot noise even for double frame spots.

# Interlaced parts
# DeSpot, Deinterlace to doublerate, McDeGrainSharp to migrate detail between original fields[now frames].
PreProc_I = (UNSPOT_I) ? PreProc : Src
Deint=PreProc_I.Eval(FUNC)
Deint=(FRAMES_I>0) ? Deint.McDeGrainSharp(frames=FRAMES_I,csharp=CSHARP_I) : Deint

SUBTXT = (OUTPUT==0) ? FNAME+"_DEINTERLACED-EVEN" : (OUTPUT==1) ? FNAME+"_RE-INTERLACED" : FNAME+"_DblRate-DEINTERLACED"
PostProc_I = (OUTPUT==0) ? Deint.SelectEven : (OUTPUT==1) ? Deint.SeparateFields.SelectEvery(4,0,3).Weave : Deint
PostProc_I = (!SUBS) ? PostProc_I : PostProc_I.Subtitle(SUBTXT,Align=5)

# Progressive parts, Despot, Degrain sharp, still @ singlerate
PreProc_P = (UNSPOT_P) ? PreProc : Src
Prog = (FRAMES_P>0) ? PreProc_P.McDeGrainSharp(frames=FRAMES_P,csharp=CSHARP_P) : PreProc_P
# Progressive to O/P rate
PostProc_P = (OUTPUT==2) ? Prog.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator) : (OUTPUT==3) ? Prog.JohnFPS() : Prog

#################

### IsCombedTIVTC with args, as a string eg "IsCombedTIVTC(cthresh=10,mi=85,chroma=true,blockx=16,blocky=16)"
COMBDET = RT_String("IsCombedTIVTC(cthresh=%d,mi=%d,chroma=%s,blockx=%d,blocky=%d)",CTHRESH,MI,CHROMA,BLOCKX,BLOCKY)

SrcTest = (OUTPUT<=1) ? Src : Src.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)

# Select Progressive / Deinterlaced frame, based on IsCombedTIVTC detection on original source clip frame.
Result=ConditionalFilter(SrcTest, PostProc_I, PostProc_P, COMBDET,"=","true")


SrcOut = (!TITLE)?SrcTest:(OUTPUT<=1) ? Src.TSub("Source",TITNUM) : Src.TSub("Source DblRate via ChangeFPS, ie dupes",TITNUM).ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)
Result = (!TITLE)?Result :(OUTPUT<=1) ? Result.TSub(String(OUTPUT,"%.0f) Result SingleRate"),TITNUM) : (OUTPUT==2) ? Result.TSub("2) Result DblRate (Progressive frames via ChangeFPS)",TITNUM)
\ : Result.TSub("3) Result DblRate (Progressive frames via JohnFPS, ie MvTools2)",TITNUM)

(STACK) ? Stackhorizontal(SrcOut, Result) : Result

(PRE_FETCH>1) ? Prefetch(PRE_FETCH) : NOP # I had problems using x64 and Prefetch [Access Violation], culprit was old IsCombedTIVTC, fixed via Pinterf version.

Return Last

#####################
# Stack Overhead Subtitle Text, with optional FrameNumber shown.
Function TSub(clip c,string Tit,Bool "ShowFrameNo",Int "Col"){
c.BlankClip(height=20,Color=Default(Col,0))
(Default(ShowFrameNo,False))?ScriptClip("""Subtitle(String(current_frame,"%.f] """+Tit+""""))"""):Trim(0,-1).Subtitle(Tit)
Return StackVertical(c).AudioDubEx(c)
}


Hybrid PAL Progressive/Interlaced, Many double frame Spots, noise. did not want doublerate result [although with above script I now have options].
Spots mainly [but not exclusively] on progressive parts. Interlaced parts color, Progressive B&W.
EDIT: Updated a bit. Output to doublerate, or singlerate Progressive, OR hybrid; Optional Interpolation for Progressive parts, re-Interlace deinterlaced parts [after Processing DeSpot, Deinterlace, McDegrainSharp].

EDIT:
To make into a DGIndex template, remove the comment # from lines 2, 3 and 4, and delete the 3 lines preceded by "# Filled in by DGIndex".
Save the file in your DGIndex "Templates" folder, and to use set DGIndex Menu/Option/AVS Template to point to the template file.
DGIndex will create an avs file, and fill in the VideoFilename, AudioFilename, and AudioDelay with correct data, when you "Save Project".

[StainlessS]

SpotLess update post #23, awarded a version of v1.0:- https://forum.doom9.org/showthread.php?p=1921583#post1921583

Could probably do with its own thread, if only to be able to point to something in New Plugins thread.
Hope the updated docs make a bit more sense.

EDIT: Changed to v1.0
New thread here:- https://forum.doom9.org/showthread.php?p=1921773#post1921773

[StainlessS]

Perhaps someone would like to test SpotLess() on 'old cartoon' which is subject of this thread.
I dont usually touch cartoon/anime at all, and even less 'old cartoon'.
I not sure that Mvtools copes well with cartoon at all.

[lansing]

Perhaps someone would like to test SpotLess() on 'old cartoon' which is subject of this thread.
I dont usually touch cartoon/anime at all, and even less 'old cartoon'.
I not sure that Mvtools copes well with cartoon at all.

It doesn't work on anime, it wiped out all moving objects in motion scenes.

[StainlessS]

Thanks for the report lansing, too bad.

EDIT: Ill put warning in Spotless thread and next to spotless post in this thread.

[GMJCZP]

KillerSpots 3.1 (see post #12):

- Added anim (KillerSpots 3.0)
- Added repmode (KillerSpots 3.1)

Merry Xmas!

[fAy01]

KillerSpots 3.1 (see post #12):

- Added anim (KillerSpots 3.0)
- Added repmode (KillerSpots 3.1)

Merry Xmas!

Wow. This is wonderful. Thank you!