SpotLess - Temporal denoise

[coolgit]

ScSelect always had the problem of SOS sometimes not following EOS or EOS not always before SOS, my mod does not change that.
EOS frame considers only itself and those either side, SOS frame considers only itself and those either side,
so EOS knows nothing of the frame follwing SOS and SOS knows nothing of the frame before EOS.

It aint the math, its logic that fails, ie the algorithm, but IIRC, mvTools scene change detect has exact same problem. [ie mismatched EOS / SOS detects]

ScSelect_HBD is definitely better than ScSelect, but it aint perfect by any means.

EDIT: Maybe you can make some use of the stuff here:- https://forum.doom9.org/showthread.php?p=1954985#post1954985
EndOfSceneClip() and StartOfSceneClip.

Don't worry no software is perfect, otherwise Microsoft would have went bankrupt in the last century.

What I meant was something like this.

If EOS=True
Then nextframe=SOS (not relying on maths but a fact that SOS must be after EOS. Accept for start of video of course.)
Else end of video.

[StainlessS]

But if its wrong, then they're both then wrong.
And if SOS detected and not EOS, then how does it go back and make EOS true on previous frame [its already gone past it].
There is no easy solution testing only 3 frame, prev,curr,next, and testing 2 prev, curr, 2 next aint without its own problems either.

[coolgit]

But if its wrong, then they're both then wrong.
And if SOS detected and not EOS, then how does it go back and make EOS true on previous frame [its already gone past it].
There is no easy solution testing only 3 frame, prev,curr,next, and testing 2 prev, curr, 2 next aint without its own problems either.

If EOS is not detected then there can be no SOS.
If SOS is detected but not previous frame as EOS then there is no SOS.

For scene change to be true there must be EOS and SOS. It can't be either or none.

[StainlessS]

For scene change to be true there must be EOS and SOS. It can't be either or none.
Tell that to Kassandro (ScSelect) and the authors of MvTools (Manao, Fizick, Tsp, TSchniede, SEt, cretindesalpes, pinterf).

If you dont like'em, there are others to choose from:- http://avisynth.nl/index.php/SCDetect#Scene_Change_Detection

[MysteryX]

You could add the VapourSynth code version to the first post. I'll remove it from my code but don't want this to get lost.

In VapourSynth, it currently only works with the 32-bit library. The dubhater library has a bug.


# SpotLess denoising method (m1=4) EXPERIMENTAL
def SpotLess(c: vs.VideoNode, radt: int = 1, thsad: int = 10000, thsad2: Optional[int] = None, pel: int = 2, chroma: bool = True, blksize: int = 8,
overlap: Optional[int] = None, truemotion: bool = True, pglobal: bool = True, blur: float = 0.0, ref: Optional[vs.VideoNode] = None) -> vs.VideoNode:
if radt < 1 or radt > 3:
raise ValueError("Spotless: radt must be between 1 and 3")
if not pel in [1, 2, 4]:
raise ValueError("Spotless: pel must be 1, 2 or 4")

thsad2 = thsad2 or thsad
icalc = c.format.bits_per_sample < 32
S = core.mv.Super if icalc else core.mvsf.Super
A = core.mv.Analyse if icalc else core.mvsf.Analyse
C = core.mv.Compensate if icalc else core.mvsf.Compensate
pad = max(blksize, 8)

sup = ref or (c.std.Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1]) if blur > 0.0 else c)
sup = S(sup, hpad=pad, vpad=pad, pel=pel, sharp=2)
sup_rend = S(c, hpad=pad, vpad=pad, pel=pel, sharp=2, levels=1) if ref or blur > 0.0 else sup

th1 = thsad
th2 = (thsad + thsad2)/2 if radt==3 else thsad2
th3 = thsad2

bv1 = A(sup, isb=True, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv1 = A(sup, isb=False, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
if radt >= 2:
bv2 = A(sup, isb=True, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv2 = A(sup, isb=False, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
if radt >= 3:
bv3 = A(sup, isb=True, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv3 = A(sup, isb=False, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)

bc1 = C(c, sup_rend, bv1, thsad=th1)
fc1 = C(c, sup_rend, fv1, thsad=th1)
if radt >= 2:
bc2 = C(c, sup_rend, bv2, thsad=th2)
fc2 = C(c, sup_rend, fv2, thsad=th2)
if radt >= 3:
bc3 = C(c, sup_rend, bv3, thsad=th3)
fc3 = C(c, sup_rend, fv3, thsad=th3)

ic = core.std.Interleave([bc1, c, fc1]) if radt == 1 else \
core.std.Interleave([bc2, bc1, c, fc1, fc2]) if radt == 2 else \
core.std.Interleave([bc3, bc2, bc1, c, fc1, fc2, fc3])

output = core.tmedian.TemporalMedian(ic, radius=radt)
return output.std.SelectEvery(radt*2+1, radt) # Return middle frame

[StainlessS]

How bout I put link to it in 1st post.
I dont speak python/vapoursynth, so it means little to me and I dont really want to be tip-toe-ing around it, wondering why its there.

EDIT: Put link in first post.

[kedautinh12]

I think you need creat new post in Vapoursynth and add your script in that, MX

[coolgit]

Hey Stainless, s 3 ep17 war without end f 8013 to 8215 is weird. Had to do that from vob again using assume bff to get it right. Any idea why this happen in the first place? Had this in an earlier ep in s3, can't remember which now.

[StainlessS]

Is that the sequence starting with subs "Special guest star Michael O'hare".
Seems progressive in my VOB, no jumping with AssumeTFF or AssumeBFF. [EDIT: Assume??? then SeparateFields]

[coolgit]

No, sub produced by john copeland... Sinclair walking away to go indoors. I use assumetff but in some rare cases, need asssumebff.

[coolgit]

Stainless, can you check you vob filesize for season 5 ep 18. I got 38:42 length with teaser part at the beginning missing.

[StainlessS]

S05E18, Disk 5, ep Clip2, - "The Fall of Centauri Prime".
is 42:21
https://i.postimg.cc/18DMS14V/Untitled-00.jpg (https://postimg.cc/18DMS14V)

EDIT:
Stainless, can you check you vob filesize for season 5 ep 18. I got 38:42 with teaser part at the beginning missing.
I really dont know how to answer that, episodes are combined in DVD VOB so VOB size is all 4 episodes and who knows what else.
38:42 is a time, not file size and teaser is part of the clip.
OK, cutting out all of "teaser", and starting at first frame of B5 intro/credits with first frame showing very small but distinct "2258", [frame before is black]
leaves 58024 frames, 38:40.96 duration. [but it depends upon where you cut the black sequence - I cut on very first visible "2258" so you should get exactly the same if cut at that point]

EDIT: My season 5 is not boxed version as my other seasons, is single DVD case with multiple flip disk holders (6).

[coolgit]

Well it seems my dvd got a defect. Tried other software and can't read it. Tried vdub and as soon as i move to the first frame it crashes.

If you have time could you upload the missing vob clip up to the first keyframe of intro for me.

Cheers.

[StainlessS]

Check PM.

[StainlessS]

Any of you mask wizards out there improve on this ?


AviSourcE(".\somefile.avi")
ConvertToYV24
#ConvertBits(16)
Src=Last
Filtered=SpotLess(RadT=2)
YTH8 = 12
CTH8 = 12
CAMP = True
SHOW = True
SpotTest(Src,Filtered,YTH8,CTH8,CAMP,SHOW)
Return Last.ConvertToRGB32

/*
SpotTest(),
Purpose:
Spotless produces frame global changes [most pixels are changed], here we want to try alter only real spots where there is significant change in spotless Y [by YTh8]
and also that are near grey in source [by CTh8], these targeted pixels are then overlayed from Spotless result onto original src.

Src, Pre-Spotless source
Filtered, Spotless result
YTh8, threshold for Y in 8 bit range.
CTh8, threshold for Chroma in 8 bit range.
CAMP, If True amplify chroma MaskC when Show
Show, Show Masks. Mask is the final mask made from MaskY and MaskC.

*/
Function SpotTest(clip Src,clip Filtered,int "YTh8",int "Cth8",Bool "AmpC",Bool "Show") { # 8 -> 16 bit YUV444 only
YTh8 = Default(YTh8,12)
CTh8 = Default(CTh8,12)
AmpC = Default(AmpC,True)
Show = Default(Show,False)
try {bpc = Src.BitsPerComponent} catch(msg) {bpc=8}
try {Y444=Src.IsYV24||Src.Is444} catch(msg) {Y444=false}
Assert(Y444,"SpotTest: Must be YV24/YUV444")
YTh = BitLShift(YTh8,bpc-8)
CTh = BitLShift(YTh8,bpc-8)
MaskY = Mt_Lutxy(Src.ExtractY,Filtered.ExtractY,"x y - abs " + String(YTh) + " > range_max 0 ?") # "(abs(x-y) > YTh) ? range_max : 0"
MU = Mt_Lut(Src.ExtractU,"x range_half - abs") # "abs(x - range_half)"
MV = Mt_Lut(Src.ExtractV,"x range_half - abs") # "abs(x - range_half)"
MaskC = Mt_Lutxy(MU,MV,"x x * y y * + 0.5 ^") # "((x^2)+(y^2)) ^ 0.5" # 2D chroma (by Pythagoras)
Mask = Mt_Lutxy(MaskC,MaskY,"x " + string(CTh) + " < y 0 ?") # "(x < CTh) ? y : 0
Mask = Mask.mt_Expand.Blur(1.0)
MaskC = (!AmpC) ? MaskC : MaskC.mt_lut("x 8 *") # 2D chroma Amp'ed for view if AmpC
Overlay(Src,Filtered,Mask=Mask,Opacity=1.0)
CGREY = Mask.Mt_lut("range_half")
TOP=StackHorizontal(Src.Subtitle("Src"),Filtered.Subtitle("Filt"),YtoUV(CGrey,CGrey,MaskY).Subtitle("MaskY"))
BOT=StackHorizontal(Last.Subtitle("Result"),YtoUV(CGrey,CGrey,Mask).Subtitle("Mask"),YtoUV(CGrey,CGrey,MaskC).Subtitle((AmpC)?"MaskC Amp'ed":"MaskC"))
STK = StackVertical(TOP,BOT)
(Show) ? STK : Last
}


Hopefully reduces the number of changed pixels, and avoids arms and legs and balls and stuff going missing.

[Dogway]

It might be obvious but maybe looking into current algorithms (Wiki (https://en.wikipedia.org/wiki/Blob_detection#Spatio-temporal_blob_detectors)) can provide some insight. I plan to give it a stab in a few weeks.

[StainlessS]

Thanks doggie, looks like Gobble-De-Gook to me though.

Above script does pretty well with VOB where looks like original film stock had emulsion chipped off (or something),
white-ish dots every now and then.
[will undergo further processing and did not want to overdo it with spotless - RADT=2 was just to make it more difficult test (with repaired disappearing arm ),
RADT=1 is quite sufficient for my clip]

Just tried with dirty home movie where various color dirt, not so good.

Anyways, I think it does pretty much exactly as I want for current clip, and really quite simple.

EDIT: Here, two frames from very jerky InGoldie clip that you might remember [boy in purple throws ball in the air, Spotless removes it but we use original source pixels not spotless result].
[no white spots for demo here]

https://i.postimg.cc/rwX9npSB/PBDe-Spot-00.jpg (https://postimg.cc/8j4WsNX4)


https://i.postimg.cc/j5LgVFCm/PBDe-Spot-01.jpg (https://postimg.cc/KkhrBQPD)

[coolgit]

A new code....mmmm. The results looks good but shouldn't both source should have a 'spot' in it to ensure the code doesn't put it back in.

If it works it could replace spotlessF.

[StainlessS]

shouldn't both source should have a 'spot' in it to ensure the code doesn't put it back in.
No,
MaskY is white where change in pixel Y [for src->spotless] is > YTh, MaskC is amount Chroma in SOURCE pixels [*8 Amplified in images for view only],
will only overlay with Spotless pixels where MaskC is < CTh and MaskY = white.

Mask = Mt_Lutxy(MaskC,MaskY,"x " + string(CTh) + " < y 0 ?") # "(x < CTh) ? y : 0 # x is MaskC pixel and y = corresponding MaskY pixel


Its really a bit specific for white or black noise only. [and its where are white pixels in MASK that are overlaid from Spotless filtered onto source]
(we are selecting what we want rather than restoring what we dont want)
There are not usually that many white dots on final mask, probably down to all them there daisies.

[coolgit]

Inserted the code into Spotless and got it to work.

Does well in restoring hands (a common problem) but also does well in restoring all kinds of defects. It rarely removes anything.

What a pity.

[StainlessS]

Yeah, I processed the clip in question, removed a lot of emulsion chips, but not as many as I would like,
however the purpose was to do only very little denoise, and it did that as the purpose was to to avoid frame global changes.
Need to figure out at least one more conditional restriction so as to be able to relax YTh and CTh restriction a little.
I still want only a very light touch.
[The frames I was processing needed maybe up to about 3 small chips/dots removed on SOME frames only].

[coolgit]

My thinking is, fresh pot of coffee helped, do not restore if mask object(what you call the white blob) or any other means, doesn't exist in adjacent frames.

ns=no spot
s=spot
b=ball

frame 1 2 3
ns s ns
b b b

Spot in frame 2 removed and since nothing exist in either 1 or 3 then do not restore. If object like a ball exist in 1 or 3 then restore.
Spot are motionless and the ball isn't.

[StainlessS]

All you did is move the problem.
Spot are motionless and the ball isn't.
How do you tell if ball exists in frames 1,and 2, and 3, as you say, ball isnt motionless.
[And if MoComp could tell, then ball would not disappear in the first place via Spotless]
I'm less concerned with balls, more with hands and feet, and arms and legs (but balls would be nice too).
EDIT: Anyways, have not given up yet, still plugging away at it.

EDIT: What I got right now, take comments with pinch of salt, and havta figure out what its doin'.

#AviSource(".\10Bit_Standard8_Scan_16FPS.avi")
#AviSource(".\IG18.avi")
#AviSource(".\AMOLAD.demuxed.m2v.AVI")
AviSource("D:\DVD\PBDeSpot_1.avi")
ConvertToYV24
#ConvertToYUV444
#Return last.info
#ConvertBits(16)
Src=Last
Filtered=SpotLess(RadT=1)
YTH8 = 8
CTH8 = 12
YNOISETH8 = 2
SHOW = True
SpotTest(Src,Filtered,YTH8,CTH8,YNOISETH8,SHOW)
Return Last.ConvertToRGB32

/*
SpotTest(),
Purpose:
Spotless produces frame global changes [most pixels are changed], here we want to try alter only real spots where there is significant change in spotless Y [by YTh8]
and also that are near grey in source [by CTh8], these targeted pixels are then overlayed from Spotless result onto original src.

Src, Pre-Spotless source
Filtered, Spotless result
YTh8, threshold for Y in 8 bit range.
CTh8, threshold for Chroma in 8 bit range.
YNoiseTh8, ?????
Show, Show Masks.Mask is the final mask made from MaskY and MaskC.

*/
Function SpotTest(clip Src,clip Filtered,int "YTh8",int "Cth8",int "YNoiseTh8",Bool "Show") { # 8 -> 16 bit YUV444 only
YTh8 = Min(Max(Default(YTh8,12),0),255)
CTh8 = Min(Max(Default(CTh8,12),0),255)
YNoiseTh8 = Min(Max(Default(YNoiseTh8,8),0),255)
Show = Default(Show,False)
try {bpc = Src.BitsPerComponent} catch(msg) {bpc=8}
try {Y444=Src.IsYV24||Src.Is444} catch(msg) {Y444=false}
Assert(Y444,"SpotTest: Must be YV24/YUV444")
YTh = BitLShift(YTh8,bpc-8).String
CTh = BitLShift(CTh8,bpc-8).String
YNoiseTh = BitLShift(YNoiseTh8,bpc-8).String
SY = Src.ExtractY
FY = Filtered.ExtractY
PSY = SY.SelectEvery(1,-1)
NSY = SY.SelectEvery(1, 1)
MaskN = Mt_Lutxyz(PSY,SY,NSY,"x "+ YNoiseTh + " - y <= y z "+ YNoiseTh + " + <= & z "+ YNoiseTh + " - y <= y x "+ YNoiseTh + " + <= & | 0 range_max ?")
# ( ((x-YNoiseTh <= y) & (y <= z+YNoiseTh)) | ((z-YNoiseTh <= y) & (y <= x+YNoiseTh)) ) ? 0 : 255 ## Infix @ 8 bit
# ((YPrv-YNoiseTh <= YCur <= YNxt+YNoiseTh) || (YNxt-YNoiseTh <= YCur <= YPrv+YNoiseTh)) ? 0 : 255

MaskY = Mt_Lutxy(SY,FY,"x y - abs " + YTh + " > range_max 0 ?") # "(abs(x-y) > YTh) ? 255 : 0"
MU = Mt_Lut(Src.ExtractU,"x range_half - abs") # "abs(x - 128)"
MV = Mt_Lut(Src.ExtractV,"x range_half - abs") # "abs(x - 128)"
MaskC = Mt_Lutxy(MU,MV,"x 2 ^ y 2 ^ + .5 ^ round " + CTh + " <= range_max 0 ?") # "(round(((x^2)+(y^2))^0.5) <= CTh) ? 255 : 0" # 2D chroma distance from grey (by Pythagoras)
MaskN = MaskN.mt_Expand
MaskY = MaskY.mt_Expand
MaskC = MaskC.mt_Expand
MaskAnd = MaskC.Mt_Logic(MaskY,"and") #
MaskAnd = MaskAnd.Mt_Logic(MaskN,"and") #
MaskSelect = MaskAnd.mt_Expand.Blur(1.0)
#
Overlay(Src,Filtered,Mask=MaskSelect,Opacity=1.0)
CGREY = MaskSelect.Mt_lut("range_half")
SEP=$FF0000
HBAR=Src.BlankClip(length=1,Height=4,color=SEP)
LFT=StackVertical(Src.Subtitle("Src"),HBAR,Filtered.Subtitle("Filt"),HBAR,Last.Subtitle("Result"))
MID=StackVertical(YtoUV(CGrey,CGrey,MaskN).Subtitle("MaskN"),HBAR,YtoUV(CGrey,CGrey,MaskY).Subtitle("MaskY {abs(SrcY-FiltY) > YTh}"),HBAR,YtoUV(CGrey,CGrey,MaskSelect).Subtitle("MaskSelect=MaskAnd.mt_Expand.Blur(1.0) {Blk=Src used : Wht=Filt used}"))
RGT=StackVertical(YtoUV(CGrey,CGrey,MaskC).Subtitle("MaskC {Src Chroma distance from Grey <= Cth8"),HBAR,YtoUV(CGrey,CGrey,MaskAnd).Subtitle("MaskAnd {MaskN & MaskC & MaskY}"),HBAR,YtoUV(CGrey,CGrey,CGrey))
VBAR=LFT.BlankClip(length=1,width=4,color=SEP)
STK = StackHorizontal(LFT,VBAR,MID,VBAR,RGT)
(Show) ? STK : Last
}

EDIT: some cosmetic changes.
I'm gonna get some sleep soon.

MaskN (noiseMask) shows no noise = BLACK, noise = WHITE for current frame Cur yPixel where,
(PrvY - NoiseTh) <= CurY <= (NxtY + NoiseTh) OR (NxtY - NoiseTh) <= CurY <= (PrvY + NoiseTh) Then BLACK Else WHITE
If pixel lies between corresponding pixels in previous and next frames (with a little tolerance by NoiseTh), then not noise, else noise.
[ie No spot if BLACK, and MAYBE spot (or ball) if WHITE]

Currently, MaskC is the main problem [BLACK is colored ie dont use BALL/HAND/LEG from Filt clip, and WHITE if Copy from filt clip].
Detecting colored ball as BLACK, also tends to stop detect of what looks like white spots at high luma level (but with chroma distance above CTh).

[coolgit]

What I was thinking was the ball would exist in the previous or next frame, as in your example post 217, by searching a part of the frame (previous or next) for similar object, say 50 by 50 pixels search area (freedom to change the variables according to video), colour shouldn't be an issue. Where the ball disappeared would be the centre of 50 by 50. Surely this would narrow down the search and mistaken identifying of details by not searching the whole frame. If it was a spot then nothing would be found and therefore not restored. In your post 217 mask y frames made it clear where the ball was and is. Comparing the ball real colour from the source frames would be the final confirmation before restoring. Neither ball in Mask y and colour have to be 100% the same in both frames, perhaps 80% is enough. Obviously this would apply to hand and foot.

[StainlessS]

OK, then please supply script code to do that, and then we is cookin'.
[If it works as well as you say, then maybe can look forward to MvTools algo update, and then this script (or parts of it) may not be needed at all]

EDIT: At present, the function is only looking to target single pixels, [EDIT: blob-ish objects are also made up of single pixels] it is possible that it may extend to blob-ish objects,
but original requirement is for fast, simple, de-speckler, which causes little global frame changes, later other main denoiser may be also used.
Its definitely just experimental at the moment.

[StainlessS]

https://academic.microsoft.com/paper/2037299049/reference/search?q=Color%20space%20normalization%3A%20Enhancing%20the%20discriminating%20power%20of%20color%20spaces%20for%20face%20recognition&qe=Or(Id%253D2121647436%252CId%253D2163808566%252CId%253D2130660124%252CId%253D2914885528%252CId%253D2137659841%252CId%253D2106309274%252CId%253D3146003712%252CId%253D2994340921%252CId%253D2012352340%252CId%253D1591385104%252CId%253D2160835070%252CId%253D1761337995%252CId%253D2144119003%252CId%253D1974097586%252CId%253D2147933509%252CId%253D2096813265%252CId%253D2140907607%252CId%253D2112724657%252CId%253D1759219755%252CId%253D148999028%252CId%253D2156334937%252CId%253D2126687887%252CId%253D2010560725%252CId%253D2150892735%252CId%253D2043999711%252CId%253D2110249778%252CId%253D2113593239%252CId%253D2162557814%252CId%253D2143857284%252CId%253D2106887132%252CId%253D1968189504%252CId%253D2002552184%252CId%253D2511353375%252CId%253D2106822035%252CId%253D2134486056%252CId%253D1515707506%252CId%253D2122426897%252CId%253D2128293876%252CId%253D2028339207%252CId%253D1574019097%252CId%253D1997071866%252CId%253D2163626218%252CId%253D2111433705%252CId%253D2062199239%252CId%253D2136353882%252CId%253D2400676549%252CId%253D639253003)&f=&orderBy=0

Color space normalization: Enhancing the discriminating power of color spaces for face recognition
Abstract

This paper presents the concept of color space normalization (CSN) and two CSN techniques, i.e., the within-color-component normalization technique (CSN-I) and the across-color-component normalization technique (CSN-II), for enhancing the discriminating power of color spaces for face recognition. Different color spaces usually display different discriminating power, and our experiments on a large scale face recognition grand challenge (FRGC) problem reveal that the RGB and XYZ color spaces are weaker than the I"1I"2I"3, YUV, YIQ, and LSLM color spaces for face recognition. We therefore apply our CSN techniques to normalize the weak color spaces, such as the RGB and the XYZ color spaces, the three hybrid color spaces XGB, YRB and ZRG, and 10 randomly generated color spaces. Experiments using the most challenging FRGC version 2 Experiment 4 with 12,776 training images, 16,028 controlled target images, and 8,014 uncontrolled query images, show that the proposed CSN techniques can significantly and consistently improve the discriminating power of the weak color spaces. Specifically, the normalized RGB, XYZ, XGB, and ZRG color spaces are more effective than or as effective as the I"1I"2I"3, YUV, YIQ and LSLM color spaces for face recognition. The additional experiments using the AR database validate the generalization of the proposed CSN techniques. We finally explain why the CSN techniques can improve the recognition performance of color spaces from the color component correlation point of view.

Basically, Color Space Normalizing of YUV just involves forgetting all about the Y channel.
Used in Facial recognition, because skin tones have similar-ish U and V range values, no matter how light / dark the skin tone.

Some, above maybe of some use. [we are using only U and V in the chroma distance thingy].

EDIT: I did see a pdf saying similar to above for YUV but deleted it.
Maybe Color Space Normalizing of YUV or Facial recognition in YUV is specifically what we want, so above extract is maybe not the one we ideally want.

[coolgit]

Just chucking in ideas, was thinking of the top of my head. Obviously i don't know the full capabilities of avisynth.

[StainlessS]

Obviously i don't know the full capabilities of avisynth.
Obviously, Nobody knows the full capabilities of avisynth. :)

[coolgit]

Stainless - tried to message you but your inbox is full.

[StainlessS]

Coolgit, sorry, bit involved,
I is a bit busy,

[coolgit]

Remove line 306 downwards and insert this...

prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCT=src.SCSelect_HBD(next,prev,SpotLessT,show=false)

prev = SpotLessF.selectevery(1,-1)
next = SpotLessF.selectevery(1,1)
SpotLessSCF=src.SCSelect_HBD(next,prev,SpotLessF,show=false)

SpotLessSCF2P=SpotLess(SpotLessSCF,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=3000,RadT=RadT) # Useful for many small defects per frame and not cleared on first pass.
# 2nd pass should clear up remaining defects and improve quality on 1st pass.
S1 = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessT, "Spotless TM true", txt_sz),\
sub(SpotLessF, "Spotless TM false", txt_sz)\
)
S2 = StackHorizontal(\
sub(SpotLessSCT, "Spotless TM true and scene change", txt_sz),\
sub(SpotLessSCF, "Spotless TM false and scene change", txt_sz),\
sub(SpotLessSCF2P, "Spotless 2 pass TM false and scene change", txt_sz)\
)
output = StackVertical(S1,S2)

Return output # Comparison


#Return SpotLessSCT # Conversion
#Return SpotLessSCF
#Return SpotLessSCF2P
#Return SpotLessT
#Return SpotLessF

# Debugging to check scene change detection------------------------------------
prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCTDM=src.SCSelect_HBD(next,prev,SpotLessT,show=true)

SSCTD1 = StackHorizontal(\
sub(src.AddBorders(0,30,0,0), "Source", txt_sz),\
sub(SpotLessSCTDM.AddBorders(0,30,0,0), "Spotless and scene change with metrics", txt_sz)\
)
SSCTD2 = StackHorizontal(\
sub(SpotLessSCT.AddBorders(0,30,0,0), "Spotless and scene change without metrics", txt_sz),\
sub(BlankClip(src).AddBorders(0,30,0,0), "Blank", txt_sz)\
)
outputD = StackVertical(SSCTD1,SSCTD2)

#Return outputD # Comparison

Added two new choices.
1. Use SpotLessF with scene changes for clips with small defects only.
2. 2nd pass available to improve on 1st pass.

[StainlessS]

DoubleRate deinterlacer template, uses SpotLess. For DGIndex / DGIndexNV. [works with either, copy to their Templates Directory]

Need some tester to suggest optimal defaults, but pretty damn good as it is.
Uses Spotless(RadT=1) on separated Even/Odd fields, where removes spots and reduces any artifically added grain [added to hide how bad the clip is].

DespotDeinterlaceDbleRate.avs

### DgIndex / DgIndexNV, Template, DespotDeinterlaceDbleRate.avs

# Temp Pre-Processing step for Interlaced, with Spotless() on separated fields, prior to deinterlace.
# Spotless(RadT=1) nicely reduces any artificial grain as well as getting rid of single field/frame spots.
# We sort of prefer BOBMOD = 1, YadifMod2 deinterlace with little/no additional overprocessing at pre-proc stage. Although can BOBMOD=2, ie QTGMC.
# Output is YV24, save AVI from VDub2, with eg Fast Recompress, Codec UT_Video, YUV444 BT.601 VCM

VideoFileName = "__vid__"
AudioFileName = "__aud__"
AudioDelay = Value("__del__")

###############

vExt = GetFilenameExtension(VideoFileName)

If(vExt == ".dgi") { DGSource(VideoFileName) } # DGIndexNV
Else { Mpeg2Source(VideoFileName) } # DGIndex

AudioExt = GetFilenameExtension(AudioFileName)

# May prefer other audio source, but these usually work just fine.
Audio= (AudioExt==".ac3") ? NICAC3Source(AudioFileName,channels=2,DRC=0)
\ : (AudioExt==".mpa"||AudioExt=="mp1"||AudioExt==".mp2"||AudioExt==".mp3") ? NicMPG123Source(AudioFileName,Normalize=False)
\ : (AudioExt==".wav") ? RaWavSource(AudioFileName)
\ : (AudioExt==".dts") ? NicDTSSource(AudioFileName)
\ : (AudioExt==".w64") ? RaWavSource(AudioFileName,samplerate=6)
\ : 0

Assert(!isInt(Audio),"NO AUDIO")

(!isInt(Audio)) ? AudioDub(Audio).DelayAudio(AudioDelay).Trim(0,0) : NOP # Trim, chop/add audio to length

(!isInt(Audio) && AudioRate() <> 44100) ? ResampleAudio(44100) : NOP # If you want 44.1 KHz (I do)


####### CONFIG ##########

AssumeTFF # Whatever (check) : Almost always TFF for PAL DVD. Default field order is BFF.

ConvertToYV24(Interlaced=true)

FINAL = False # True, switch off SUBS, STACK, and Title

DBLRATE = true # Output rate Double input
RADT = 1 # Spotless() on separated fields only, before de-interlaced.
DEGRAIN_FIELDS=False # True, Degrain separated fields rather than deinterlaced frames (On one or other, but Not both). [done after Spotless when on separated fields].
BOBMOD=1 # 0=BOB, 1=YadifMod2, 2=QTGMC.

FRAMES = 0 # McDegrainSharp(frames=FRAMES), (0=Dont McDegrainSharp) # McDegrainSharp() on fields if DEGRAIN_FIELDS=true, else on frames [Never on both].
CSHARP = 0.03 # Very light McDegrainSharp sharpen arg



# For Spotless on separated fields.
ThSAD = 1000 # Ditto
ThSAD2 = 990 # Ditto
PEL = 2 # Ditto
CHROMA = True # Ditto
BlkSz = 16 # Ditto
TM = False # Ditto
DC = "RemoveGrain(1).Blur(0.3,0.3)" # DC = Detect Clip. Or Just "Blur(0.3)"

# Deinterlace mode

BOB_FUNC_0 = """Bob()""" # Cheap & cheerful Doublerate simple Bob.
BOB_FUNC_1 = """YadifMod2(Order=-1,Mode=1,edeint=NNedi3(Field=-2, nns=2))""" # Doublerate Yadifmod2 filter.
# # Field-2=double rate, internal parity value to start : Order-1=Internal Parity. Mode 1=double rate do spactial check (3=no spatial check).

BOB_FUNC_2 = """QTGMC(Preset="fast",NoisePreset="Fast",SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1, tr2=0)""" # Doublerate QTGMC filter.

FUNC = Select(BOBMOD, BOB_FUNC_0, BOB_FUNC_1, BOB_FUNC_2)
FNAME = Select(BOBMOD,"BOB", "YADIF", "QTGMC")


PRE_FETCH=4 # 1 = No Prefetch, Else eg 4 = Prefetch(4) ie MT mode

### # Display stuff
SUBS = True
STACK = True # Show Src and Result
TITLE = true # Show Title Bar
TITNUM = True # Show Frame Number on Title Bar (Only if TITLE)

##### End Config ##########

Src=Last


PROCESS_FIELDS = (RADT>0 || (DEGRAIN_FIELDS && FRAMES>0)) # any Field pre-processing before deinterlacing ?

# DeSpot and/or McDeGrainSharp on fields
Sep = Src.SeparateFields
SEP_Even = Sep.SelectEven
SEP_Even = (RADT>0) ? SEP_Even.SpotLess(Radt=RADT,thsad=ThSAD,thsad2=ThSAD2,Pel=PEL,chroma=CHROMA,blksz=BLKSZ,tm=TM,glob=True,dc=SEP_Even.Eval(DC)) : SEP_Even
SEP_Even = (DEGRAIN_FIELDS && FRAMES>0) ? SEP_Even.McDeGrainSharp(frames=FRAMES,csharp=CSHARP) : SEP_Even
SEP_Odd = Sep.SelectOdd
SEP_Odd = (RADT>0) ? SEP_Odd.SpotLess(Radt=RADT,thsad=ThSAD,thsad2=ThSAD2,Pel=PEL,chroma=CHROMA,blksz=BLKSZ,tm=TM,glob=True,dc=SEP_Odd.Eval(DC)) : SEP_Odd
SEP_Odd = (DEGRAIN_FIELDS && FRAMES>0) ? SEP_Odd.McDeGrainSharp(frames=FRAMES,csharp=CSHARP) : SEP_Odd
PreProc = (PROCESS_FIELDS) ? Interleave(Sep_Even,Sep_Odd).Weave : Src

Deint = PreProc.Eval(FUNC) # Double Rate deinterlace
Deint = (!DEGRAIN_FIELDS && FRAMES>0) ? Deint.McDeGrainSharp(frames=FRAMES,csharp=CSHARP) : Deint
Result = (!DBLRATE) ? Deint.SelectEven : Deint

SUBS = (!FINAL&&SUBS)
TITLE = (!FINAL&&TITLE)
STACK = (!FINAL&&STACK)

SUBTXT = (!DBLRATE) ? FNAME+"_DEINTERLACED-EVEN" : FNAME+"_DblRate-DEINTERLACED"


RESULT = (SUBS) ? Result.Subtitle(SUBTXT,Align=5) : Result

SrcTest = (!DBLRATE) ? Src : Src.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)

SrcOut = (!TITLE)?SrcTest:(!DBLRATE) ? Src.TSub("Source",TITNUM) : Src.TSub("Source DblRate via ChangeFPS, ie dupes",TITNUM).ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)

RESULT = (!TITLE) ? RESULT :(!DBLRATE) ? Result.TSub("Result SingleRate",TITNUM) : Result.TSub("Result DblRate",TITNUM)

(STACK) ? Stackhorizontal(SrcOut, Result) : Result

(PRE_FETCH>1) ? Prefetch(PRE_FETCH) : NOP # I had problems using x64 and Prefetch [Access Violation], culprit was old IsCombedTIVTC, fixed via Pinterf version.

Return Last

#####################
# Stack Overhead Subtitle Text, with optional FrameNumber shown.
Function TSub(clip c,string Tit,Bool "ShowFrameNo",Int "Col"){
c.BlankClip(height=20,Color=Default(Col,0))
(Default(ShowFrameNo,False))?ScriptClip("""Subtitle(String(current_frame,"%.f] """+Tit+""""))"""):Trim(0,-1).Subtitle(Tit)
Return StackVertical(c).AudioDubEx(c)
}

Function GetFilenameExtension(String fn) {
fn=fn.RevStr i1=fn.FindStr("\") i2=fn.FindStr("/")
i=(i1==0) ? i2 : (i2==0) ? i1 : min(i1,i2)
fn=(i>0) ? Fn.LeftStr(i-1) : fn
i=fn.FindStr(".")
Return (i>0)?fn.LeftStr(i).RevStr:""
}


EDIT: Intended where entire input clip is known to be interlaced, not mix of interlaced and progressive.

Here a [vob source] frame picked out at nearly random, shows stacked with subs. [CLICK ME]
https://i.postimg.cc/N9XhSFjC/CL-A-01.jpg (https://postimg.cc/N9XhSFjC)
I did not look for frame with spots, but shows grain reduction.

EDIT:
Changed Frames=0 [MDegrainSharp off by default], and default CSHARP=0.03 [was 0.1]
Makes it a bit faster by default {less pre-processing}, and even milder sharpen if switched on.
https://i.postimg.cc/RqQpFyqD/Cold-Lazarus-Frames-0.jpg (https://postimg.cc/RqQpFyqD)

[johnmeyer]

StainlessS,

I just used (today!) Spotless on half an hour of 1956 NFL (American football) film. It nailed spots that the old RemoveDirtMC wouldn't touch.

I'm still looking for a way to add adjustments for spot size and color. These "spots" (they were mostly circles) were all white. At other times most spots are black. Adding this capability would add one other feature: ignore real objects, most of which are not near-black or near-white. I still lose a lot of shirt buttons as people move around. In this project, the ball often disappeared and I had to create a separate video that didn't have any dirt removal and cut to that when the ball disappeared. That is tedious work.

However, even as is, I'm finding it more useful than RemoveDirtMC for many projects.

So, thanks again for the excellent work!

Here is a before/after. All those white circles on the black uniforms on the left were moving around like a snowstorm from frame to frame, and it got 90% of them.
https://i.imgur.com/13zoiHx.png

[StainlessS]

Slight alterations to previously given DgIndex/DgIndexNV script template. [McDegrainSharp(Frames=0) by default, ie off, faster]

For Spot size, BlkSz maybe at least twice as wide/tall as spots. And also, ThSAD,ThSAD2 affects result. [ThSAD >= ThSAD2].

Also this lot from Docs [I doubt that I can easily make it less opaque].


ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle [EDIT: median] pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD, 0 < ThSAD2 < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.


Several guys earlier in thread did some mods where arms/legs were recovered, there are links to other scripts.

Also, Dogways posted here:- https://forum.doom9.org/showthread.php?p=1959548#post1959548
It might be obvious but maybe looking into current algorithms (Wiki) can provide some insight. I plan to give it a stab in a few weeks.
So maybe something in the pipeline one day.

[johnmeyer]

Thanks for those hints. That helps a lot.

[StainlessS]

SpotLess v1.07, see first post.

Repeat post here, and zip @ mediafire in sig below this post.

SpotLess.avsi v1.07


/*
SpotLess v1.07. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.

NOT FOR cartoon/anime, live video only, sorry.

v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.
v1.05, Additional checks on args.
v1.06, Glob Default true, Almost always better.
v1.07, bBlur Default changed from 0.0 OFF, to 0.3, slower but better for mc analysis. Maybe try 0.6 for HD to overcome HD 'grittiness'.
ThSAD default changed from 10,000 to 255 * (8/2 * 8/2), ie 1/4 of 8x8 block at 255, ie 1/4 BlkSz spots @ max SAD. Also ThSAD2 a bit less than ThSAD.
Added dc arg, detect clip.

SpotLess(clip c,int "RadT"=1,int "ThSAD"=255*(8/2 * 8/2),int "ThSAD2"=ThSAD-(8*8),int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,
\ bool "tm"=true,Bool "glob"=True,Float "bBlur"=0.3, clip "dc"=Undefined)

RadT, 1 or more, Default 1. Removes Spots on up to RadT [Temporal Radius] consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

ThSAD, Default 255 * (8/2 * 8/2) ie 4080 <ie 1/4 of 8x8 block at 255, ie 1/4 BlkSz spots @ max SAD>.
0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD-(8*8), 0 < ThSAD2 <= ThSAD < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.

Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)

Chroma, Default True. MAnalyse chroma arg. If true, use chroma in block matching when creating vectors. Maybe use False if source B&W or color noise.

BlkSz, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]

OLap, Default half BlkSz, Block overlap.

Tm, TrueMotion Default True. Some folk swear truemotion=false is better.

Glob, Default True (True v1.06, was same as Tm, true almost always better), Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable.

bBlur, Default 0.3. If used, Suggest about 0.3, where MAnalyse create vectors is performed on denoised (blurred) super clip
for better motion analysis. Maybe try 0.6 for HD clip to counteract HD 'grittiness'. bBlur ignored if dc clip specified,
If providing bBlur, then ALWAYS specify as named argument ie bBlur=bBlur, we will likely insert any additional MvTools args
before bBlur to keep them together.

dc, Default UnDefined. If dc Specified, then must be same size and colorspace as source clip.
If detection clip specified then bBlur prefilter arg is ignored, and analysis is performed on
dc.MSuper() clip [ instead of c.Blur(bBlur).MSuper() ]. Allows to provide your own prefiltered clip.
If providing dc clip, then ALWAYS specify as named argument ie dc=DC, we will likely insert any additional MvTools args
before bBlur to keep them together.

*/

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,255*(8/2 * 8/2)) # SAD threshold at radius 1. Here is max'ed out at 255 * 1/4 of 8*8 block <ie 4080> (MvTools Default 10,000 is nearly OFF, maybe way Too High).
# Maybe above 255*8/2*8/2 still a bit high, maybe 128*(8/2 * 8/2) Better <ie 2048>. For General noise perhaps somethimg like 24 * (8 * 8) <ie 1536>.
ThSAD2 = Default(ThSAD2,ThSAD-(8*8)) # SAD threshold at radius RadT(a bit less than ThSAD).
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.3) # Default 0.3. Suggest about 0.3 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

[guest]

[QUOTE=StainlessS;1975475]SpotLess v1.07, see first post.

Repeat post here, and zip @ mediafire in sig below this post.

SpotLess.avsi v1.07

Can't find it :(

[kedautinh12]

[QUOTE=StainlessS;1975475]SpotLess v1.07, see first post.

Repeat post here, and zip @ mediafire in sig below this post.

SpotLess.avsi v1.07

Can't find it :(

Just copy script above

[guest]

[QUOTE=TDS;1975479]

Just copy script above

I did do that, but the other option is not there....that's all.

I had this working a while back, but something must have changed, as currently, I can't get it to work.

But it's a script I wouldn't use too often, anyway.

Thanks.

[StainlessS]

Sorry, and thanks guys, its on MediaFire now.

@ TDS,
maybe you missed out the final "}" at end of script, its easily done.

[Stereodude]

Sorry if this is a really dumb question, but is this mostly for removing spots like the examples posted and the name suggests, or can it also do the same sort of motion compensated general noise reduction as something like MCTD (https://forum.doom9.org/showthread.php?t=139766)?

I've started playing with SpotLess, but was curious what other's have found.

[kedautinh12]

Sorry if this is a really dumb question, but is this mostly for removing spots like the examples posted and the name suggests, or can it also do the same sort of motion compensated general noise reduction as something like MCTD (https://forum.doom9.org/showthread.php?t=139766)?

I've started playing with SpotLess, but was curious what other's have found.

Latest ver:
https://github.com/Asd-g/AviSynthPlus-Scripts/blob/master/MCTD_.avsi

[StainlessS]

At StereoDude,
Suggest try with about, RadT=1, and ThSAD = N * (8 * 8) for general denoise [where N about 6 ==>> 24].
Also, maybe something like "DC=Src.RemoveGrain(22).Blur(0.3)"

[Stereodude]

At StereoDude,
Suggest try with about, RadT=1, and ThSAD = N * (8 * 8) for general denoise [where N about 6 ==>> 24].
Also, maybe something like "DC=Src.RemoveGrain(22).Blur(0.3)"
Okay, I will give it a try. :thanks:

[guest]

Sorry, and thanks guys, its on MediaFire now.

@ TDS,
maybe you missed out the final "}" at end of script, its easily done.

Thanks SssS, but the } was there, so there must be something else missing.

LoadPlugin("%AVISYNTHPLUGINS%\MedianBlur2\MedianBlur2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
Import("%AVISYNTHPLUGINS%\SPOTLESS\Spotless.avs")
video=Spotless(video)

I know I had this working, a while ago :(

[StainlessS]

Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.


Only those bolded above maybe necessary, I think.
Some modes of MvTools use fftw3.dll, but not usually neccesary, I think [dont think we use those modes].
Also, current AVS+ a good idea, but should work on other.
NOTE, should be recent Pinterf version dll's.

Apart from above, you dont give any indication of what went wrong, no error message or anything, so little chance of any further assistance.
Suggest try with no arguments at all, ie Spotless(SomeClip) to see if works.

[kedautinh12]

Only those bolded above maybe necessary, I think.
Some modes of MvTools use fftw3.dll, but not usually necesary, I think [dont think we use those modes].
Also, current AVS+ a good idea, but should work on other.
NOTE, should be recent Pinterf version dll's.

Apart from above, you dont give any indication of what went wrong, no error message or anything, so little chance of any further assistance.
Suggest try with no arguments at all, ie Spotless(SomeClip) to see if works.

Ripbot don't give any error message and he still use it :D

[StainlessS]

Thanks K,

then just load script into VDub2.
https://forum.doom9.org/showthread.php?t=172021

[guest]

Ripbot don't give any error message and he still use it :D

Come on K, I'm right here, you knew I'd read this :rolleyes:

So, I'd like to know how many of "you guy's" have even tried RipBot264 ??

And how many of you actually do a considerable amount of movie encoding ??

RipBot is the ONLY (free) app that can use remote PC's to assist with the encoding process, it's called "Distributed Encoding", and can use up to 16 PC's.

App's like MeGUI, VD2, Handbrake, Staxrip, etc, can only be used on the PC the app runs on.

I'm aware that RipBot has quite a few limitations, and not too many filtering options, but that's the way Atak_Snajpera has written it, and is VERY reluctant to change.....and his support for his app has declined a lot recently, and is a pretty well guarded code.

Fortunately, a guy called Pauly Dunne has incorporated a plethora of Dogway's SMDegrain based (and others) scripts & filter options, for RipBot, and provides regular updates to these and other important components of the app.

So don't knock it, 'til you try it, you might be surprised, and it would also be beneficial to the app if more "developers" like yourselves, supported it. ;)

Spread the word...

[kedautinh12]

Come on K, I'm right here, you knew I'd read this :rolleyes:

So, I'd like to know how many of "you guy's" have even tried RipBot264 ??

And how many of you actually do a considerable amount of movie encoding ??

RipBot is the ONLY (free) app that can use remote PC's to assist with the encoding process, it's called "Distributed Encoding", and can use up to 16 PC's.

App's like MeGUI, VD2, Handbrake, Staxrip, etc, can only be used on the PC the app runs on.

I'm aware that RipBot has quite a few limitations, and not too many filtering options, but that's the way Atak_Snajpera has written it, and is VERY reluctant to change.....and his support for his app has declined a lot recently, and is a pretty well guarded code.

Fortunately, a guy called Pauly Dunne has incorporated a plethora of Dogway's SMDegrain based (and others) scripts & filter options, for RipBot, and provides regular updates to these and other important components of the app.

So don't knock it, 'til you try it, you might be surprised, and it would also be beneficial to the app if more "developers" like yourselves, supported it. ;)

Spread the word...

If i change from Megui to other apps, i will change to Avspmod or selur's hybrid. And yeah, i still suprise although don't use it cause i don't know what errors when use with scripts and how to fix it :D