SpotLess - Temporal denoise

[StainlessS]

EDIT: See also, related threads:::
SpotRemover - clean video from spots:- https://forum.doom9.org/showthread.php?t=182407
[film transfer] Remove dirt but keep grain? :- https://forum.doom9.org/showthread.php?t=182831

Script filter to remove spots and noise using mvTools2 and Medianblur2.
Copes with spots on multiple consecutive frames, setting RadT accordingly.
Mvtools2/MedianBlur2 by Pinterf, so AVS+ colorspace and x64 compatible.
Based on an idea by Didée.
Originally posted here:- https://forum.doom9.org/showthread.php?p=1921583#post1921583

Spotless.avsi v1.07

/*
SpotLess v1.07. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.

NOT FOR cartoon/anime, live video only, sorry.

v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.
v1.05, Additional checks on args.
v1.06, Glob Default true, Almost always better.
v1.07, bBlur Default changed from 0.0 OFF, to 0.3, slower but better for mc analysis. Maybe try 0.6 for HD to overcome HD 'grittiness'.
ThSAD default changed from 10,000 to 255 * (8/2 * 8/2), ie 1/4 of 8x8 block at 255, ie 1/4 BlkSz spots @ max SAD. Also ThSAD2 a bit less than ThSAD.
Added dc arg, detect clip.

SpotLess(clip c,int "RadT"=1,int "ThSAD"=255*(8/2 * 8/2),int "ThSAD2"=ThSAD-(8*8),int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,
\ bool "tm"=true,Bool "glob"=True,Float "bBlur"=0.3, clip "dc"=Undefined)

RadT, 1 or more, Default 1. Removes Spots on up to RadT [Temporal Radius] consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

ThSAD, Default 255 * (8/2 * 8/2) ie 4080 <ie 1/4 of 8x8 block at 255, ie 1/4 BlkSz spots @ max SAD>.
0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD-(8*8), 0 < ThSAD2 <= ThSAD < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.

Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)

Chroma, Default True. MAnalyse chroma arg. If true, use chroma in block matching when creating vectors. Maybe use False if source B&W or color noise.

BlkSz, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]

OLap, Default half BlkSz, Block overlap.

Tm, TrueMotion Default True. Some folk swear truemotion=false is better.

Glob, Default True (True v1.06, was same as Tm, true almost always better), Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable.

bBlur, Default 0.3. If used, Suggest about 0.3, where MAnalyse create vectors is performed on denoised (blurred) super clip
for better motion analysis. Maybe try 0.6 for HD clip to counteract HD 'grittiness'. bBlur ignored if dc clip specified,
If providing bBlur, then ALWAYS specify as named argument ie bBlur=bBlur, we will likely insert any additional MvTools args
before bBlur to keep them together.

dc, Default UnDefined. If dc Specified, then must be same size and colorspace as source clip.
If detection clip specified then bBlur prefilter arg is ignored, and analysis is performed on
dc.MSuper() clip [ instead of c.Blur(bBlur).MSuper() ]. Allows to provide your own prefiltered clip.
If providing dc clip, then ALWAYS specify as named argument ie dc=DC, we will likely insert any additional MvTools args
before bBlur to keep them together.

*/

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,255*(8/2 * 8/2)) # SAD threshold at radius 1. Here is max'ed out at 255 * 1/4 of 8*8 block <ie 4080> (MvTools Default 10,000 is nearly OFF, maybe way Too High).
# Maybe above 255*8/2*8/2 still a bit high, maybe 128*(8/2 * 8/2) Better <ie 2048>. For General noise perhaps somethimg like 24 * (8 * 8) <ie 1536>.
ThSAD2 = Default(ThSAD2,ThSAD-(8*8)) # SAD threshold at radius RadT(a bit less than ThSAD).
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.3) # Default 0.3. Suggest about 0.3 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


Hacked this for a clip with spots on double frames, did not know if would be much better, but as it turns out is fandabbydozy.

Chosen function name would seem to be a very good choice, not perfect, but very nearly is.

Adjective: spotless
1) Completely neat and clean

You should also try spotless as a prefilter in smdegrain. Worked very good for me with almost zero artifacts left from spotless.

yes, it's do better job than any methods ...

This is a wonderful tool. As it happens, I've been cleaning a VHS cap and the silly video has quite a lot of spots and scratches in every other field. SpotLess cleans them up nicely and also takes care of most of the chroma noise too.. I haven't checked things with longer samples but looks good anyway.

Here Vapoursynth version of Spotless, posted later in this thread by MysteryX:- https://forum.doom9.org/showthread.php?p=1955170#post1955170
In VapourSynth, it currently only works with the 32-bit library. The dubhater library has a bug.

Find also Spotless_v1.07_20220928.zip @ MediaFire in sig below this post.

[StainlessS]

Perhaps someone would like to test SpotLess() on 'old cartoon' which is subject of this thread.
I dont usually touch cartoon/anime at all, and even less 'old cartoon'.
I not sure that Mvtools copes well with cartoon at all.

It doesn't work on anime, it wiped out all moving objects in motion scenes.

So not much good for cartoon/anime, sorry.

[StainlessS]

If you've got a clip with a big dollop of black crap (spot) at frame edge which is not removed by SpotLess, may possibly
be removed by eg AddBorders(32,32,32,32,$808080) prior to SpotLess and Crop(32,32,-32,-32) afterwards.
I had one of 4 frame duration which was successfully removed with RadT=6, [maybe I should up RadT max to 8 or 10 or something, for one-offs].
Will likely not work if 1st or last spot near scene change.

EDIT: My spot would seem to be a film burn hole [dont know what they are called].
EDIT: MSuper( ..., hpad=32,vpad=32) [ EDIT: inside of Spotless] dont do it, gotta Addborders pad with grey or white to get rid of black spot).
EDIT: Weird, was working fine, now only removes 2 of the 4 spots, ???
EDIT: I may have modified prior color correction since the time it removed all 4 spots, still only removing 2 spots.

[EDIT: Frame Top Right Hand corner] Source spot, burn in near vacinity of other dark object
https://i.postimg.cc/V0mKqdXM/Play-1-B-01.jpg (https://postimg.cc/V0mKqdXM)
Spotless fixed
https://i.postimg.cc/Dmhhj6gQ/Play-1-B-02.jpg (https://postimg.cc/Dmhhj6gQ)
Next frame, Not Fixed (well modified a bit but) is too close now to dark object
https://i.postimg.cc/7bNkpzsF/Play-1-B-00.jpg (https://postimg.cc/7bNkpzsF)
The color correction that I changed somehow previously allowed fix, but not now, forgot what prev color correction was.

[Boulder]

EDIT: My spot would seem to be a film burn hole [dont know what they are called]

I think that's called "a cigarette burn" :) I wouldn't be surprised if they actually were those back in the day.

[StainlessS]

Change from

MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=8,overlap=4,search=4)

To

MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=32,overlap=16,search=4)

Allows detect and removal of above posted spot, I'll make Blocksize selectable.
Spot from above bottom image Now gone [as is the last of the 4 spots also removed, so all 4 spots now gone].
https://i.postimg.cc/jwL7GKK9/Play-1-B-Copy-00.jpg (https://postimg.cc/jwL7GKK9)

EDIT
Cigarette burn, well I thought could happen when reel stopped spinning and projector lamp still hot.

[StainlessS]

1st post updated script.

v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.


EDIT:
Adding hpad, vpad greater than 8, assists when changing framerate, avoids oscillating effects at frame edge sometimes, will not do any harm to also have it here.

[Boulder]

EDIT
Cigarette burn, well I thought could happen when reel stopped spinning and projector lamp still hot.

My thought was more nostalgic, like people cutting the film while smoking heavily and dropping ashes and sparks all over the place :)

[StainlessS]

Yeh, you're problably correct in my particular spot instance, think projector lamp burns were not always totally symmetrical.

Nostalgia, its not what it used to be. [stolen from some D9 guys sig].

Maybe I should set hpad/vpad to same as BlkSz, when above 8.

Also, ClipClop, [or ReplaceFramesSimple] could be used to apply special instance SpotLess args to awkward ranges where spots resist fixing.

[StainlessS]

Clipclop script

AviSource(".\Play_1.avi")
O=Last
SpotLess() # Apply Spotless(Radt=2,Blksz=8) to entire clip

FIX1=O.AddBorders(32,32,32,32,$808080).Spotless(RadT=4,BlkSz=32).Crop(32,32,-32,-32) # Special clip index 1
FIX2=O.AddBorders(32,32,32,32,$808080).Spotless(RadT=5,BlkSz=32).Crop(32,32,-32,-32) # Special clip index 2

SCMD= """
1 28251,28254 # Replace fix range 28251,28254 using FIX1 [clip index 1]
2 30001,30005 # Replace fix range 30001,30005 using FIX2 [clip index 2]

"""

ClipClop(Last,FIX1,FIX2,scmd=SCMD) # also has file commands. Max 255 replacement clips [unlimited replacment ranges]
StackHorizontal(O,Last)


ClipClop:- https://forum.doom9.org/showthread.php?t=162266&highlight=Clipclop

EDIT: Its like magic :) Now to go back and try fix non removed large frame edge spot that I did yesterday.

Or mod


AviSource(".\Play_1.avi")
O=Last
SpotLess(2) # Apply Spotless(Radt=2,Blksz=8) to entire clip

FIX1=O.EdgeKludge(4) # Special clip index 1
FIX2=O.EdgeKludge(5) # Special clip index 2

SCMD= """
1 28251,28254 # Replace fix range 28251,28254 using FIX1 [clip index 1]
2 30001,30005 # Replace fix range 30001,30005 using FIX2 [clip index 2]

"""

ClipClop(Last,FIX1,FIX2,scmd=SCMD) # also has file commands # BUGFIX, ClipClop(Last ... NOT ... ClipClop(O
StackHorizontal(O,Last)

Return Last

Function EdgeKludge(clip c,Int RadT,Int "BlkSz", Int "Border") {
BlkSz=Default(BlkSz,32)
Border=Default(Border,32)
Return c.AddBorders(Border,Border,Border,Border,$808080).Spotless(RadT=RadT,BlkSz=BlkSz).Crop(Border,Border,-Border,-Border)
}

[FranceBB]

I think that's called "a cigarette burn" :) I wouldn't be surprised if they actually were those back in the day.

Is it?
I don't know whether we're talking about the same thing and I might be wrong but to me it looks like a cue mark (https://en.wikipedia.org/wiki/Cue_mark), which was used to let the projectionist know that the reel was close to the end or that it reached a certain length. Even though those things were used way before I was even born, I actually found 'em in some of the very old contents I'm Remastering and in the recording-sheet they were saying like "Cue Mark at " and so on

[Boulder]

Is it?
I don't know whether we're talking about the same thing and I might be wrong but to me it looks like a cue mark (https://en.wikipedia.org/wiki/Cue_mark), which was used to let the projectionist know that the reel was close to the end or that it reached a certain length. Even though those things were used way before I was even born, I actually found 'em in some of the very old contents I'm Remastering and in the recording-sheet they were saying like "Cue Mark at " and so on

You could well be right, especially if you've run into those like that :)

[Frank62]

I actually found 'em in some of the very old contents

I find these every week in for-me-not-so-old-content. Round or sometimes square. If the content is worth it, we fix it by hand, but this way looks quite elegant. Will give it a try. Thanks @StainlessS again! :)

[StainlessS]

There were so many fag burns, all in same-ish position, and all 4 frames duration, so FBB probably correct.
The 4 frame replacements in the script are nearly all, 'fag burns'.


The 'border color' prob, imported a few pixels of the gray addborders into where the burn was, probably due to burn surround color being mid green (close to mid grey),
so I changed the kludge fliter to use YUV PC Levels white, and that fixed the grey imports.
One of the spots fixes caused a little bit of blurring on fast moving obect, but way better than the burn, so left it [maybe could do special fix to apply just to spot 1/4 of frame,
or through mask, but probably gonna leave it, so hard to notice.
At one point, had mid frame bad full frame width vhs breakup, maybe 30 or 40 pixels tall, and also full frame desaturation, kludge fixed both perfectly, hehe.

Script used

AviSource(".\Play_1.avi")
O=Last
SpotLess(2) # Apply Spotless(Radt=2,Blksz=8) to entire clip

FIX1 = O.EdgeKludge(1)
FIX2 = O.EdgeKludge(2)
FIX3 = O.EdgeKludge(3)
FIX4 = O.EdgeKludge(4)
FIX5 = O.EdgeKludge(5)
FIX6 = O.EdgeKludge(6)
FIX7 = O.EdgeKludge(7)
FIX8 = O.EdgeKludge(8)
FIX9 = O.EdgeKludge(9)

SCMD= """
3 1756,-3 # ripple [-3 means 3 frames, instead of start,end frames]
4 28251,-4 #
1 28352 # single frame
4 28426,-4
4 51911,-4
4 78032,-4
4 78208,-4
4 105201,-4 # attn, change to border color = YUV PC levels white
4 105377,-4
7 122341,-7
4 127242,-4 # attn, desat
4 127418,-4
8 130004,-8 # film stratch
4 133844,-4
"""

ClipClop(Last,FIX1,FIX2,FIX3,FIX4,FIX5,FIX6,FIX7,FIX8,FIX9,scmd=SCMD) # also has file commands : BUGFIX: NOT ClipClop(O, but ClipClop(Last,
StackHorizontal(O,Last)

Return Last

Function EdgeKludge(clip c,Int RadT,Int "BlkSz", Int "Border",int "ColYUV") {
BlkSz=Default(BlkSz,32)
Border=Default(Border,32)
ColYUV=Default(ColYUV,$FF8080) # YUV PC RANGE WHITE

Return c.AddBorders(Border,Border,Border,Border,Color_yuv=ColYUV).Spotless(RadT=RadT,BlkSz=BlkSz).Crop(Border,Border,-Border,-Border)
}


EDIT: Not sure, but I think the original SpotLess(RadT=2) also fixed a single full frame dropout, I did not notice it in the fixed version.

Didee really should have pushed his idea more, this thing is bloody marvelous.
EDIT: GMJCZP did KillerSpots() function, but only single frame fixes.

EDIT: Above I used EdgeKludge() funcs, even for non edge problems, just saved time not doing separate functions.

EDIT: BUGFIX in above script, did not matter to me as I had already called Spotless(2) creating the input AVI

[StainlessS]

SpotLess Update v1.02. See 1st post.

SpotLess v1.02. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2, Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.

NOT FOR cartoon/anime, live video only, sorry.
v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.

RadT, 1 or more, Default 2. Removes Spots on up to RadT consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring [no problems encountered so far at RadT=6].
ThSAD2, 400 <= ThSAD2 <= 10000, Default 10000=NEARLY OFF. Lower removes fewer spots, but less chance of blurring.
Reduce if produces blurred frames, absolute minimum of maybe about 400, probably pretty good as is.
Lower reduces effect of bad non matching blocks, reduced more so on blocks from more distant radii frames.
ThSAD for frame @ distance 1 defaults to 10,000 [and we let it default], ThSAD2 sets the SAD [Sum of Absolute Differences] threshold
for most distant frame at distance RadT, with those frames that are distances in-between being given a SAD threshold linearly spread between the two.
Wild guess that maybe about 10000 at RadT=1, and 5000 (10000/2) where RadT=6, might be a good compromise if concerned about blurring.
Suggested, based on wild guesses above :) [if you experience blurred results, otherwise leave as is]
Where below RadT is 6, and ThSAD2=5000, then blocks @ frames current_frame +- 6 gets weighted SAD threshold value of 5000 (same as ThSAD2),
@ current_frame +- 3, gets distance weighted SAD threshold value 8000, etc
Radius SAD threshold : Where RadT=1, SAD threshold=ThSAD2. Where RadT>1; SAD threshold=(RadT-Radius)*((10000-ThSAD2)/(RadT-1))+ThSAD2; (6-Radius)*((10000-5000)/(6-1))+5000.
1 10000 10000, Almost switches OFF SAD thresholding at frame distance 1. [Ignores blocks of only really high SAD differences]
2 9000
3 8000 In between frames acquire SAD threshold value linearly between 10000 @ distance 1, and ThSAD2[here 5,000] at distance RadT[here 6].
4 7000
5 6000
6 5000 Where RadT=6 and ThSAD2 = 5000, so frame @ distance 6 gets SAD threshold value of ThSAD2[5000]
And to weight similar to above where eg RadT=3, set RadT=3, ThSAD2=8000 to set as above Radius=3,SAD threshold=8000. Radii 2 and 1 will acquire 9000 and 10000 resp.
From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).
ThSAD2:
Defines the SAD soft threshold for the furthest frames.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame, EDIT: Default 10000)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
In SpotLess, we dont actually set mvtools ThSAD arg, only ThSAD2, ThSAD will default 10000, but where RadT=1 and you set ThSAD2 to eg 5000 it should (I hope)
set soft SAD threshold at radii 1 to ThSAD2 [and override the ThSAD default 10000].
If anybody can explain above any better, then please do. I hate doing docs and as you can see, I aint very good at it.
Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma, Default True. MAnalyse chroma arg. If set to true. Use chroma in block matching.
BlkSz, Default c.width>960 ? 16 : 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips (Maybe on occasion 24 or 32).
OLap, Default half BlkSz, Block overlap.
tm, TrueMotion, False if BlkSz>8, else true. Some folk swear truemotion=false is better, NOTE MAnalyse Global will always be true.


EDIT: From MvTools2 docs for MAnalyse,
global

Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable. Default is like truemotion.
We always set Global=true, even when Truemotion turned OFF.

[StainlessS]

SpotLess v1.03, update, see 1st post.


SpotLess v1.03. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2, Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.

NOT FOR cartoon/anime, live video only, sorry.
v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.

SpotLess(clip c,int "RadT"=2,int "ThSAD2"=10000,int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,bool "tm"=true,Bool "glob"=Tm,Float "bBlur"=0.0)

RadT, 1 or more, Default 2. Removes Spots on up to RadT consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring [no problems encountered so far at RadT=6].
ThSAD2, 400 <= ThSAD2 <= 10000, Default 10000=NEARLY OFF. Lower removes fewer spots, but less chance of blurring.
Reduce if produces blurred frames, absolute minimum of maybe about 400, probably pretty good as is.
Lower reduces effect of bad non matching blocks, reduced more so on blocks from more distant radii frames.
ThSAD for frame @ distance 1 defaults to 10,000 [and we let it default], ThSAD2 sets the SAD [Sum of Absolute Differences] threshold
for most distant frame at distance RadT, with those frames that are distances in-between being given a SAD threshold linearly spread between the two.
Wild guess that maybe about 10000 at RadT=1, and 5000 (10000/2) where RadT=6, might be a good compromise if concerned about blurring.
Suggested, based on wild guesses above :) [if you experience blurred results, otherwise leave as is]
Where below RadT is 6, and ThSAD2=5000, then blocks @ frames current_frame +- 6 gets weighted SAD threshold value of 5000 (same as ThSAD2),
@ current_frame +- 3, gets distance weighted SAD threshold value 8000, etc
Radius SAD threshold : Where RadT=1, SAD threshold=ThSAD2. Where RadT>1; SAD threshold=(RadT-Radius)*((10000-ThSAD2)/(RadT-1))+ThSAD2; (6-Radius)*((10000-5000)/(6-1))+5000.
1 10000 10000, Almost switches OFF SAD thresholding at frame distance 1. [Ignores blocks of only really high SAD differences]
2 9000
3 8000 In between frames acquire SAD threshold value linearly between 10000 @ distance 1, and ThSAD2[here 5,000] at distance RadT[here 6].
4 7000
5 6000
6 5000 Where RadT=6 and ThSAD2 = 5000, so frame @ distance 6 gets SAD threshold value of ThSAD2[5000]
And to weight similar to above where eg RadT=3, set RadT=3, ThSAD2=8000 to set as above Radius=3,SAD threshold=8000. Radii 2 and 1 will acquire 9000 and 10000 resp.
From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).
ThSAD2:
Defines the SAD soft threshold for the furthest frames.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame, EDIT: Default 10000)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
In SpotLess, we dont actually set mvtools ThSAD arg, only ThSAD2, ThSAD will default 10000, but where RadT=1 and you set ThSAD2 to eg 5000 it should (I hope)
set soft SAD threshold at radii 1 to ThSAD2 [and override the ThSAD default 10000].
If anybody can explain above any better, then please do. I hate doing docs and as you can see, I aint very good at it.
Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma, Default True. MAnalyse chroma arg. If set to true. Use chroma in block matching.
BlkSz, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips.
OLap, Default half BlkSz, Block overlap.
Tm, TrueMotion Default True. Some folk swear truemotion=false is better.
Glob, Default Tm, Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable. Default is like truemotion.
bBlur, Default 0.0 [OFF]. If used, Suggest about 0.6, where MAnalyse create vectors performed on denoised (blurred) super clip.



Reverted some args, BlkSz=8, tm(truemotion)=true.
Found instances where fast-ish moving objects got swallowed up [a swift 'heil hitler' would do it], so reverted to safer defaults.
Added Glob, and 'bBlur args.

EDIT: From MvTools2 docs for MAnalyse,
global

Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable. Default is like truemotion.

[StainlessS]

Here script that I've been using for SingleRate deinterlace of mixed Progressive / Interlaced PAL.
Removed from DBSC thread last post, as have changed it here.
I used SpotLess on pre-deinterlaced source, it seemed like a good idea at the time but causes fuzziness around eg eyes
and sort of prevents deinterlace where fuzzy.
So, moved SpotLess to after deinterlace, much better detail and acts as a sort of residual combing fixer.
I only use below for single rate deinterlacing [ie OUTPUT=0], the other modes being experimental.
Many oldish [80's] PAL TV sourced DVD's are mixes of interlaced and progressive, the one I've recently been 'doin' spotless for is interlaced video
color for indoor scenes, progressive for outdoor scenes, and progressive B&W for ancient footage.
Such PAL hybrid progressive/interlaced is quite often cursed with changing borders, I use this script prior to removing dynamically changing borders
with DBSC.

DGIndex template, DVD_PAL_Hybrid_DeintSingleRate.avs

### DGIndex Template, PAL_Hybrid_DeintDoubleRate.avs
VideoFileName ="__vid__"
AudioFileName ="__aud__"
AudioDelay =Value("__del__")

###############
# PAL Hybrid Interlaced/Progressive mix, output to singlerate or doublerate.
###############
MPEG2Source(VideoFileName)
AudioExt=RT_GetFileExtension(AudioFileName)

Audio= (AudioExt==".ac3") ? NICAC3Source(AudioFileName,channels=2,DRC=0)
\ : (AudioExt==".mpa"||AudioExt=="mp1"||AudioExt==".mp2"||AudioExt==".mp3") ? NicMPG123Source(AudioFileName,Normalize=False)
\ : (AudioExt==".wav") ? RaWavSource(AudioFileName)
\ : (AudioExt==".dts") ? NicDTSSource(AudioFileName)
\ : 0

Assert(!isInt(Audio),"NO AUDIO")

(!isInt(Audio)) ? AudioDub(Audio).DelayAudio(AudioDelay).Trim(0,0) : NOP # Trim, chop/add audio to length

(!isInt(Audio) && AudioRate() <> 44100) ? ResampleAudio(44100) : NOP
####### CONFIG ##########

AssumeTFF # Whatever (check) : Almost always TFF for PAL DVD. Default field order is BFF.

# Uncomment below if has any Perverse Telecine [eg Babylon5].
TFM(pp=0)
O=Last
# and a bit of color tweaking, PRE-MC stuff (else VERY SLOW)
#AutoAdjust(Auto_Gain=true).Merge(O,0.6) # un-comment if wished

OUTPUT = 3 # 0=SingleRate 25FPS Progressive. Progressive src frames as input, interlaced src frames are even frames from dbl rate deinterlacing.
# 1=SingleRate 25FPS Hybrid. Progressive src frames as input, interlaced src frames dbl rate deinterlaced, reinterlaced to 25FPS after processing.
# 2=DblRate 50fps. Prog=ChangeFPS. Progressive src frames doubled with ChangeFPS, interlaced src frames are dbl rate deinterlaced.
# 3=DblRate 50fps. Prog=Interpolate. Progressive src frames double rate interpolated with JohnFPS [ie MvTools], interlaced src frames are dbl rate deinterlaced.

UNSPOT_P=true # Use SplotLess() on Progressive parts to remove spots and noise.
UNSPOT_I=False # Use SplotLess() on Interlaced parts to remove spots and noise.

RADT_P = 2 # SpotLess temporal radius (spots on 2 consecutive frames)
ThSAD_P = 5000 # Almost OFF.
ThSAD2_P = 5000 # Almost OFF.
BlkSz_P = 16
Tm_P = false
Glob_P = true
bBlur_P = 0.6

FRAMES_P=1 # Progressive McDegrainSharp(frames=FRAMES_P), (0=Dont McDegrainSharp)
CSHARP_P=0.5

RADT_I = 2 # SpotLess temporal radius (spots on 2 consecutive frames)
ThSAD_I = 5000 # Almost OFF.
ThSAD2_I = 5000 # Almost OFF.
BlkSz_I = 16
Tm_I = false
Glob_I = true
bBlur_I = 0.6

FRAMES_I=2 # Interlaced McDegrainSharp(frames=FRAMES_I), (Maybe double FRAMES_P for detail migration, although max possible for McDegrainSharp=3). (0=Dont McDegrainSharp)
CSHARP_I=CSHARP_P

# IsCombedTIVTC args, Defaults, 1st after comment hash.
CTHRESH = 12 # 10), Area combing threshold.
# Valid settings are from -1 (every pixel will be detected as combed) to 255 (no pixel will be detected as combed).
# This is basically a pixel difference value. A good range is between 8 to 12.
MI = 80 # 80), Number of combed pixels inside any of the blocky by blockx size blocks on the frame for the frame to be detected as combed.
CHROMA = True # True), Whether or not chroma is considered in the combed frame decision. [We want chroma as interlaced parts in color]
BLOCKX = 16 # 16), X-axis size of the window used during combed frame detection.
BLOCKY = 16 # 16), Y-axis size of the window used during combed frame detection.
# So in any single block, if 85[MI] out of 256[BLOCKX x BLOCKY] pixels pass CThresh as combed, then is Interlaced.
METRIC = 1 # 0) Detecto method: 0=Old, 1=New

# Deinterlace mode
BOBMOD=1 # 0=BOB, 1=YadifMod2, 2=QTGMC : QTGMC here is way memory/CPU intensive, req x64. Suggest YadifMod2

BOB_FUNC_0 = """Bob()""" # Cheap & cheerful Doublerate simple Bob.
BOB_FUNC_1 = """YadifMod2(Order=-1,Mode=1,edeint=NNedi3(Field=-2, nns=2))""" # Doublerate Yadifmod2 filter.
# # Field-2=double rate, internal parity value to start : Order-1=Internal Parity. Mode 1=double rate do spactial check (3=no spatial check).

BOB_FUNC_2 = """QTGMC(Preset="fast",NoisePreset="Fast",SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1, tr2=0)""" # Doublerate QTGMC filter.

FUNC = Select(BOBMOD, BOB_FUNC_0, BOB_FUNC_1, BOB_FUNC_2)
FNAME = Select(BOBMOD,"BOB", "YADIF", "QTGMC")


PRE_FETCH=6 # 1 = No Prefetch, Else eg 4 = Prefetch(4) ie MT mode

### # Display stuff
SUBS = True # Show where Deinterlaced/ReInterlaced
STACK = True # Show Src and Result
TITLE = True # Show Title Bar
TITNUM = True # Show Frame Number on Title Bar (Only if TITLE)
FINAL = True # True, switch off SUBS, STACK, and Title

##### End Config ##########

Src=Last

SUBS = (!FINAL&&SUBS)
TITLE = (!FINAL&&TITLE)
STACK = (!FINAL&&STACK)


### IsCombedTIVTC with args, as a string eg "IsCombedTIVTC(cthresh=10,mi=85,chroma=true,blockx=16,blocky=16,metric=1)"
COMBDET = RT_String("IsCombedTIVTC(cthresh=%d,mi=%d,chroma=%s,blockx=%d,blocky=%d,metric=%d)",CTHRESH,MI,CHROMA,BLOCKX,BLOCKY,METRIC)
SUBTXT = (OUTPUT==0) ? FNAME+"_DEINTERLACED-EVEN" : (OUTPUT==1) ? FNAME+"_RE-INTERLACED" : FNAME+"_DblRate-DEINTERLACED"

# Interlaced parts
# DeSpot, Deinterlace to doublerate, McDeGrainSharp to migrate detail between original fields[now frames].
Deint = Src.Eval(FUNC) # Double Rate deinterlace
Deint = (OUTPUT==0) ? Deint.SelectEven : Deint # If (OUTPUT==0) McDegrainSharp & SpotLess done after Deinterlace.SelectEven, Else after doublerate deinterlace.
Deint = (UNSPOT_I) ? Deint.SpotLess(Radt=RADT_I,thsad=ThSAD_I,thsad2=ThSAD2_I,blksz=BLKSZ_I,tm=Tm_I,glob=Glob_I,bBlur=bBlur_I) : Deint
Deint = (FRAMES_I>0) ? Deint.McDeGrainSharp(frames=FRAMES_I,csharp=CSHARP_I) : Deint
Deint = (OUTPUT==1) ? Deint.SeparateFields.SelectEvery(4,0,3).Weave : Deint # If singlerate Re-Interlacing
Deint = (SUBS) ? Deint.Subtitle(SUBTXT,Align=5) : Deint

# Progressive parts, Despot, Degrain sharp done still @ singlerate
Prog = Src
Prog = (UNSPOT_P) ? Prog.SpotLess(Radt=RADT_P,thsad=ThSAD_P,thsad2=ThSAD2_P,blksz=BLKSZ_P,tm=Tm_P,glob=Glob_P,bBlur=bBlur_P) : Prog
Prog = (FRAMES_P>0) ? Prog.McDeGrainSharp(frames=FRAMES_P,csharp=CSHARP_P) : Prog
Prog = (OUTPUT==2) ? Prog.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator) : (OUTPUT==3) ? Prog.JohnFPS() : Prog # Progressive to O/P rate

SrcTest = (OUTPUT<=1) ? Src : Src.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)
# Select Progressive / Deinterlaced frame, based on IsCombedTIVTC detection on original source clip frame.
Result=ConditionalFilter(SrcTest, Deint, Prog, COMBDET,"=","true")

SrcOut = (!TITLE)?SrcTest:(OUTPUT<=1) ? Src.TSub("Source",TITNUM) : Src.TSub("Source DblRate via ChangeFPS, ie dupes",TITNUM).ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)

Result = (!TITLE)?Result :(OUTPUT<=1) ? Result.TSub(String(OUTPUT,"%.0f) Result SingleRate"),TITNUM) : (OUTPUT==2) ? Result.TSub("2) Result DblRate (Progressive frames via ChangeFPS)",TITNUM)
\ : Result.TSub("3) Result DblRate (Progressive frames via JohnFPS, ie MvTools2)",TITNUM)

(STACK) ? Stackhorizontal(SrcOut, Result) : Result

(PRE_FETCH>1) ? Prefetch(PRE_FETCH) : NOP # I had problems using x64 and Prefetch [Access Violation], culprit was old IsCombedTIVTC, fixed via Pinterf version.

Return Last

#####################
# Stack Overhead Subtitle Text, with optional FrameNumber shown.
Function TSub(clip c,string Tit,Bool "ShowFrameNo",Int "Col"){
c.BlankClip(height=20,Color=Default(Col,0))
(Default(ShowFrameNo,False))?ScriptClip("""Subtitle(String(current_frame,"%.f] """+Tit+""""))"""):Trim(0,-1).Subtitle(Tit)
Return StackVertical(c).AudioDubEx(c)
}

EDIT: Above Edited, now Doublerate, and uses TFM(pp=0) to remove perverse telecine as for Babylon5. Added in BLUE

EDIT: Dont ,miss previous post where Spotless update to v1.03.
EDIT: Also, with above script I found that is better with ThSAD2=5000, rather than default 10000.
EDIT: Perhaps good guess for ThSAD2 would be 10,000 / RadT, so
RadT=2 then 5000 : RadT=6 then 1666 : RadT=10 then 1000 : RadT=20 then 500

[FranceBB]

I just tried SpotLess and it's absolutely Spot On xD
No, seriously, it's coming in handy as I'm cleaning the 1948 Olympics. ;)

[StainlessS]

Dont tell anybody your secret, you'll be able to ask for a pay rise :)

[Frank62]

Also a big thank you to @StainlessS (and Didee), I am using it too, with a very dirty and scratchy source, and it works very well!
I agree to those who use it with truemotion=false.
Fast big things in the foreground give far less artefacts - I have no idea why, just tested.

[johnmeyer]

I decided to take SpotLessS (note my improved spelling) out for a spin, hoping that it might be a more modern alternative to RemoveDirt.

I am very encouraged by the initial trials and, if I can figure out how to tune it, it might just be a good replacement.

I did run into two problems with my "torture clip." First, SpotLessS is slow. I made no attempt to run it with multithreading enabled because some of the posts made it sound like that wasn't going to work. For me, getting it to work multi-threaded will be mandatory if I am to use it. I didn't want to waste time on that effort until I am first convinced that it is something I can really use.

The second issue was its tendency to remove far more "good things" than does RemoveDirt. Let me explain by referring to my torture clip.

Here is a single, fairly long play, from an early 1960s American football game, recorded on 24 fps B&W movie film. This is cut directly from the DVD that was supplied to me (i.e., it is as far up the food chain as I can go, since I do not have access to the film itself):

http://www.mediafire.com/file/fgtsy99n439mj6z/test+clip_02.mpg/file

This film was horribly damaged, and I spent a full day repairing it, mostly by hand, by masking in good video from adjacent frames to replace the huge chemical spots (they look like nitrate film deterioration, but this film seems too modern to have been nitrate). I was still left with lots of dirt to remove, and the RemoveDirt portion of my film restoration script did a really good job with that.

Here is the script I used to test SpotLessS:

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\Latest Version\mvtools2.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\MedianBlur2.dll")
source = AVISource("e:\fs.avi").ConvertToYV12(interlaced=true).killaudio()
output = SpotLess(source)
#return output

Interleave(
\ source
\ , output
\ )As I stepped through each frame using the Interleave code (which shows me the original frame followed immediately by the restored frame) I began to note that an awful lot of arms and legs were disappearing. Having the ball disappear in flight is something I have never been able to avoid with RemoveDirt, but arms and legs seldom disappear.

Then, with this particular clip, I noticed a huge number of hash marks were also disappearing. For those not familiar with American football, hash marks are the short lines on the field which are paced exactly one yards apart, using for spotting the ball for the next play. I have never seen these disappear in all the dozens of hours of football plays I've restored for a major sports film collector using my version of VideoFred's film restoration scripts to which I added RemoveDirt.

I'm hoping that both problems can be fixed by tweaking a few variables, but I am asking for just a few hints as to which variables I should focus on.

StainlessS, you may find this torture clip pretty useful because it has a LOT of things going on: motion in the frame, as well as horizontal camera motion; high contrast and low contrast dirt; massive dirt stains (some of which SpotLessS removes almost perfectly!!); and of course, "spots!!"
------------

[edit]
I'm having some success by massively reducing ThSAD2 (1000 instead of 10000). Chroma=false also helps a lot (source is B&W). BlkSz massively changes the results: 16 makes everything disappear, but 4 significantly degrades the spot removal. The sensitivity to block size is way more pronounced than I have ever seen with other MVTools2 scripts and may provide hints on what other things to tune.

[edit2]
OMG, the true motion variable is the secret: set that to false and really great things happen. Wow, wow, wow!

These are the variables that created magic:

output = SpotLess(source,ThSAD2=1000,RadT=2,Chroma=true,BlkSz=8,OLap=4,Tm=false)

Makes me wonder if I should experiment with that for other MVTools2 scripts I have.

I'll play around with this tomorrow when I'm fresher (age kills energy late in the day, I'm finding). However, as of right now, this may be the best AVISynth script or function I've used in the last year.

Way to go Mr. Lack of Stain.

[StainlessS]

SplotLess() v1.04, update. See 1st post.

I made the executive decision to make some script breaking changes, sorry.


SpotLess v1.04. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.

NOT FOR cartoon/anime, live video only, sorry.

v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.

SpotLess(clip c,int "RadT"=1,int "ThSAD"=10000,int "ThSAD2"=ThSAD,int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,bool "tm"=true,Bool "glob"=Tm,Float "bBlur"=0.0)

RadT, 1 or more, Default 1. Removes Spots on up to RadT [Temporal Radius] consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD, 0 < ThSAD2 < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.

Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)

Chroma, Default True. MAnalyse chroma arg. If true, use chroma in block matching when creating vectors. Maybe use False if source B&W or color noise.

BlkSz, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]

OLap, Default half BlkSz, Block overlap.

Tm, TrueMotion Default False. Some folk swear truemotion=false is better.

Glob, Default Tm, Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable. Default is like truemotion.

bBlur, Default 0.0 [OFF]. If used, Suggest about 0.6, where MAnalyse create vectors is performed on denoised (blurred) super clip.


EDIT: RequestLinear() following the source filter might suffice as frame accurate source filter.

[johnmeyer]

I just tried the new v1.04 on the full version of that B&W football game, but didn't see any significant differences.

I'm still getting a little too much removal of legs, etc., but I'm still trying to experiment with all the settings.

[StainlessS]

I have not as yet tried with that clip, I'm a bit knackered and probably heading for bed soon.
Maybe I should have given moniker of this filter StainLessS(), then my name would be remembered down the millennia,
me and Achilles, yes they would remember us.

EDIT: John, your torture clip is 29.976FPS with dupes, so assume 23.976 FPS.
You need to at least remove duplicates, they will mess stuff up, ie count bad frames twice and mess with median,
things look tricky enough without having same error counted multiple times.

[johnmeyer]

John, your torture clip is 29.976FPS with dupes, so assume 23.976 FPS.
You need to at least remove duplicates, they will mess stuff up, ie count bad frames twice and mess with median,
things look tricky enough without having same error counted multiple times.Let me go back and look at that. I just checked and it looks like proper 23.976 progressive without dups, but maybe something got screwed up along the way.

[edit]I just looked at the original MPEG video, as well as the clip that I losslessly extracted using Womble's MPEG Video Wizard, and both are indeed 23.976 progreesive without dups or drops. I looked at the test clip I posted in MediaInfo and it reports 23.976, with no hint of 29.97 w/ pulldown. My NLE (Vegas) also reports 23.976 and when I play the clip in Vegas, as well as when I serve from Vegas into your script, I see no evidence of mismatched frame rates.

Have another look tomorrow morning and see if something got glitched on your end.

[Boulder]

Have you tested if doing MRecalculate (or multiple recalculations) with a smaller blocksize on the vectors would help with those disappearing human parts issues?

[joka]

Spotless() is a motion compensated temporal median filter of n frames (n=2*RadT+1). In case of n=5 this is a very heavy filter that removes a lot of dirt, spots, grain, ...; but removes details and (parts of) moving objects too.
Such a filter but with n=3 (like the script proposed by Didee) is the cleaning part of RemoveDirtMC. In RemoveDirtMC the cleaning filter is followed by a motion area restoring part ("RestoreMotionBlocks") to remove the artifacts.

In my opinion spotless() is a try worth to use it as cleaning filter in RemoveDirtMC. For a stand alone usage there are too much articfacts.

However, as I understood the original intend was to remove overlapping spots on 2 consecutive frames. Spotless() can do it in no-motion areas, but the required "RestoreMotionBlocks" will restore it.

[johnmeyer]

Have you tested if doing MRecalculate (or multiple recalculations) with a smaller blocksize on the vectors would help with those disappearing human parts issues?I never thought of doing that. I've re-written quite a few MVTools2 scripts to use MRecalculate so, if time permits today I'll try that with StainlessS' code and see what happens.

I made the observation, in my earlier post, that block size had the strongest impact on the filter of any variable and this impact was far larger than I'd ever seen in other uses of MVTools2. It never occurred to me to use MRecalculate, which I normally think of as a way to get the precision of the smaller block size, but with the speed of the larger blocks, but which also, I guess, has other impacts.

[wonkey_monkey]

Spotless() is a motion compensated temporal median filter of n frames (n=2*RadT+1).

Huh. There was me skimming this thread, not really paying much attention, seeing the video and thinking "What this really needs is a motion compensated temporal median filter of, ooh, around 5 frames." Great minds think alike :cool:

[Boulder]

I never thought of doing that. I've re-written quite a few MVTools2 scripts to use MRecalculate so, if time permits today I'll try that with StainlessS' code and see what happens.

I made the observation, in my earlier post, that block size had the strongest impact on the filter of any variable and this impact was far larger than I'd ever seen in other uses of MVTools2. It never occurred to me to use MRecalculate, which I normally think of as a way to get the precision of the smaller block size, but with the speed of the larger blocks, but which also, I guess, has other impacts.
I have no idea if it does help here, but I find it stabilizes the vector field quite a bit. Maybe SSS will add an MShow-related parameter to display the vectors to help determine the correct settings.

[johnmeyer]

I was unable to get MRecalculate to work with this script. I suspect that the problem is that the "delta" variable in MAnalyze is not supported by MRecalculate. This script uses large values for Delta. Once I used MRecalculate, the resulting video was from the wrong moment in time.

[StainlessS]

John, I suspect it [Mrecalculate thingy] just needs vectors=Multivec, and tr=RadT.

MRecalculate

MRecalculate (
clip super,
clip vectors,
int thSAD (200),
int smooth (1),
int blksize,
int blksizeV,
int search,
int searchparam,
int lambda,
bool chroma,
bool truemotion,
int pnew,
int overlap,
int overlapV,
string outfile,
int dct,
int divide,
int sadx264,
bool isse,
int tr
int scaleCSAD (0)
)

Refines and recalculates motion data of previously estimated (by MAnalyse) motion vectors with different super clip or new parameters set (e.g. lesser block size), after divide, etc. The two-stage method may be also useful for more stable (robust) motion estimation. The refining is at finest hierarchical level only. Interpolated vectors of old blocks are used as predictors for new vectors, with recalculation of SAD. Only bad quality new vectors with SAD above threshold thSAD will be re-estimated by search. thSAD value is scaled to 8x8 block size. Good vectors are not changed, but their SAD will be re-calculated and updated.

You must set the tr variable if you process "multi" motion vector clips.

Parameters not described below have the same meaning as in MAnalyse, but you can use other values. Their default settings are same as in MAnalyse.

smooth

This is method for dividing coarse blocks into smaller ones.
0 Use motion of nearest block.
1 Bilinear interpolation of 4 neighbors.

tr

This is the temporal radius for motion vector clips generated by MAnalyse with multi = true. Default 0 (normal vector clip).


EDIT:
vectors is not an optional named argument for Mrecalculate, so skip the "vectors=".

MAnalyse "c[blksize]i[blksizeV]i[levels]i[search]i[searchparam]i[pelsearch]i[isb]b[lambda]i[chroma]b[delta]i[truemotion]b[lsad]i[plevel]i[global]b[pnew]i[pzero]i[pglobal]i
"[overlap]i[overlapV]i[outfile]s[dct]i[divide]i[sadx264]i[badSAD]i[badrange]i[isse]b[meander]b[temporal]b[trymany]b[multi]b[mt]b[scaleCSAD]i"
MBlockFps "cccc[num]i[den]i[mode]i[ml]i[blend]b[thSCD1]i[thSCD2]i[isse]b[planar]b[mt]b"
MCompensate "ccc[scbehavior]b[recursion]f[thSAD]i[fields]b[time]f[thSCD1]i[thSCD2]i[isse]b[planar]b[mt]b[tr]i[center]b[cclip]c[thSAD2]i"
MDegrain1 "cccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain2 "cccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain3 "cccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain4 "cccccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain5 "cccccccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain6 "cccccccccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrainN "ccci[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[thsad2]i[thsadc2]i[mt]b[out16]b"
MDepan "cc[mask]c[zoom]b[rot]b[pixaspect]f[error]f[info]b[log]s[wrong]f[zerow]f[range]i[thSCD1]i[thSCD2]i[isse]b[planar]b"
MFlow "ccc[time]f[mode]i[fields]b[thSCD1]i[thSCD2]i[isse]b[planar]b[tclip]c"
MFlowBlur "cccc[blur]f[prec]i[thSCD1]i[thSCD2]i[isse]b[planar]b"
MFlowFps "cccc[num]i[den]i[mask]i[ml]f[blend]b[thSCD1]i[thSCD2]i[isse]b[planar]b[optDebug]i"
MFlowInter "cccc[time]f[ml]f[blend]b[thSCD1]i[thSCD2]i[isse]b[planar]b[tclip]c"
MMask "cc[ml]f[gamma]f[kind]i[time]f[Ysc]i[thSCD1]i[thSCD2]i[isse]b[planar]b"

MRecalculate "cc[thsad]i[smooth]i[blksize]i[blksizeV]i[search]i[searchparam]i[lambda]i[chroma]b[truemotion]b[pnew]i[overlap]i[overlapV]i[outfile]s[dct]i[divide]i[sadx264]i[isse]b[meander]b[tr]i[mt]b[scaleCSAD]i"
^
|


MRestoreVect "c[index]i"
MScaleVect "c[scale]f[scaleV]f[mode]i[flip]b[adjustSubPel]b[bits]i"
MSCDetection "cc[Ysc]i[thSCD1]i[thSCD2]i[isse]b"
MShow "cc[scale]i[sil]i[tol]i[showsad]b[number]i[thSCD1]i[thSCD2]i[isse]b[planar]b"
MStoreVect "c+[vccs]s"
MSuper "c[hpad]i[vpad]i[pel]i[levels]i[chroma]b[sharp]i[rfilter]i[pelclip]c[isse]b[planar]b[mt]b"

[johnmeyer]

I thought that's what I did, but this code didn't work.

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
RadT = Default(RadT,2) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,10000) # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true. Use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,Tm) # Default Tm, Allow set MAnalyse(global) independently of TrueMotion.
Bblur = Default(bblur,0.0) # Default OFF
Assert(1 <= RadT,"SpotLess: 1 <= RadT")
pad = max(BlkSz,8)
sup = (bBlur<=0.0 ? c : c.blur(bblur)).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
sup_rend = (bBlur<=0.0) ? sup : c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,levels=1) # Only 1 Level required where not MAnalyse-ing.
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,search=4,chroma=Chroma,truemotion=Tm,global=Glob)

MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,search=4,chroma=Chroma,truemotion=Tm,global=Glob)
MultiVec_re = MRecalculate(sup,MultiVec, chroma=Chroma,blksize=BlkSz/2, overlap=OLap/2,truemotion=Tm,tr=1)
c.MCompensate(sup_rend,MultiVec_re,tr=RadT,thSad=ThSAD,thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

[StainlessS]

MultiVec_re = MRecalculate(sup,MultiVec, chroma=Chroma,blksize=BlkSz/2, overlap=OLap/2,truemotion=Tm,tr=RadT)

[johnmeyer]

That works. Thanks!

I'll play around with the variables and see if I can come up with something which cleans as well or better than RemoveDirt, and removes the same or fewer arms and legs.

[edit]If you ever decide to play with my torture clip, this set of variables did a pretty good job of cleaning the basic dirt (forget about the chemical stains because they are too big to be dealt with automatically). It also didn't completely amputate arms and legs, although socks and wristbands sometimes disappear. As long as that doesn't happen too often, most people won't notice.

This is getting tantalizingly close to being one of the truly great scripts.

SpotLess(source,ThSAD=5000,ThSAD2=5000,RadT=1,Chroma=false,BlkSz=8,OLap=4,Tm=false)

[StainlessS]

I will have a play with your clip, I'm just about done recovering a working linux after moving Linux / Windows onto SSHD (Solid State Hybrid Drive), it killed/crippled my setup for some days [well a week].
I had to look, I'm currently on Linux.

EDIT: This,

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
RadT = Default(RadT,2) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,10000) # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)


Should be

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)

[johnmeyer]

Yeah, changing RadT to 1 is what made it behave the way I expected. Of course RadT>1 is part of what makes the script so powerful for other applications, like when you removed that massive color cast, so I'm sure I'll be using larger numbers for other applications.

[FranceBB]

And this is where things went completely wrong.
Sadly those athletes were all black and SpotlessS clearly thought they were big giant spots and tried its best to remove them xD

https://i.imgur.com/sSBKO4g.png
https://i.imgur.com/MzpZr9S.png
https://i.imgur.com/FM0ocbw.png


StainlessS, I think your filter might be bit racist xD

[kedautinh12]

And this is where things went completely wrong.
Sadly those athletes were all black and SpotlessS clearly thought they were big giant spots and tried its best to remove them xD

https://i.imgur.com/sSBKO4g.png
https://i.imgur.com/MzpZr9S.png
https://i.imgur.com/FM0ocbw.png


StainlessS, I think your filter might be bit racist xD

I think they are ghost :) :) :)

[StainlessS]

StainlessS, I think your filter might be bit racist

Yeh well in this crazy mixed up PC (EDIT: Black Legs Matter) world, you gotta have a bit of balance somewhere.

EDIT: Although the guy in yellow at the back seems to have lost a bit more than his legs.

[johnmeyer]

And this is where things went completely wrong.What settings did you use? Without that information, your observations don't mean much.

I too noticed the same thing (limb amputation), and posted about it earlier in this thread. I then spent some time to learn and understand the filter, and found settings that completely avoid the disappearance of arms and legs that you posted.

Posting that, without providing any information about settings is exactly like complaining that a noise reduction filter eliminates all detail after you turn all the settings to "max." Such an observation does a disservice to the people reading the thread who may think that the filter/plugin has no utility and is not worth much.

SpotlessS is most definitely a very worthwhile script.

[FranceBB]

I know, I wasn't complaining, I was just pointing out that I had the very same issues you had on some scenes, that's all.
Anyway, I'll provide the script if it helps others.

[johnmeyer]

I know, I wasn't complaining, I was just pointing out that I had the very same issues you had on some scenes, that's all.
Anyway, I'll provide the script if it helps others.Got it. Thanks!

[StainlessS]

Just noticed, mistake in docs [fixed]

Tm, TrueMotion Default True. Some folk swear truemotion=false is better. # Was False

[johnmeyer]

I was able to eliminate a lot of the rainbows in the clip provided in a thread over at Videohelp. This post shows the SpotlessS script I used, along with one "before/after" snapshot:

https://forum.videohelp.com/threads/398864-Need-help-on-Avisynth?p=2595644&viewfull=1#post2595644

[kedautinh12]

I was able to eliminate a lot of the rainbows in the clip provided in a thread over at Videohelp. This post shows the SpotlessS script I used, along with one "before/after" snapshot:

https://forum.videohelp.com/threads/398864-Need-help-on-Avisynth?p=2595644&viewfull=1#post2595644

Can you try ASTDR derainbow??
https://github.com/realfinder/AVS-Stuff/blob/master/avs%202.5%20and%20up/ASTDR.avsi

[johnmeyer]

Can you try ASTDR derainbow??
https://github.com/realfinder/AVS-Stuff/blob/master/avs%202.5%20and%20up/ASTDR.avsiI'll try to remember to do that when I get back to my office computer tomorrow. I really only wanted to kill the rainbows, but not do any denoising, and SpotlessS uses a function which desnoises as well as averages. I'll let you know if it works.

[StainlessS]

John, you posted on VideoHelp, non current script,


RadT = Default(RadT,2) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,10000) # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)


should be (and for good reason)

RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)


Changing ThSAD also changes default ThSAD2.

We dont really want gazllions of versions of the function floating about, there are already more than enough official ones.

[johnmeyer]

John, you posted on VideoHelp, non current script, <snip>

We dont really want gazllions of versions of the function floating about, there are already more than enough official ones.Sorry about that. I don't think it will make any difference, since I fed my own numbers into the script so the new presets in your current version will be overridden.

[StainlessS]

Update SpotLess(), v1.06, see 1st post.


v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.
v1.05, Additional checks on args.
v1.06, Glob Default true, Almost always better.

[StainlessS]

Dont know if this is of any use at all (I usually like to get rid of interlacing).


Function SpotLessI(clip c,String "Bobber",int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
/*
SpotLess use on Interlaced material.
Input requires correct parity, use AssumeTFF/AssumeBFF before use.
RadT etc applied to Bobbed fields rather than frames, Might want to use at least RadT=2 (default is 1).
*/
c
Bobber = Default(Bobber,"Bob") # Your Favourite Bob deinterlacer whotsit
Last.Eval(Bobber)
SpotLess(RadT,ThSAD,ThSAD2,pel,chroma,BlkSz,Olap,tm,glob,bBlur)
SeparateFields.SelectEvery(4,0,3).Weave
Return Last
}


Have I done it correctly, any comments and testing welcome.
If is of value, I'll add to main script, otherwise if no response I'll give it a miss.

EDIT: Or any suggestions on single/doublerate use (maybe skip last step SeparateFields.weave).
EDIT: Or could use above as a template.
EDIT: Possibly of best use where original clip was progressive with noise before artificial conversion to Interlaced.

EDIT: Alternative.

Function SpotLess2i2(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
/*
SpotLess use on Interlaced material, fields filtered separately.
Input requires correct parity, use AssumeTFF/AssumeBFF before use.
*/
c
e=Separatefields.SelectEven.AssumeFrameBased.SpotLess(RadT,ThSAD,ThSAD2,pel,chroma,BlkSz,Olap,tm,glob,bBlur)
o=Separatefields.SelectOdd.AssumeFrameBased.SpotLess(RadT,ThSAD,ThSAD2,pel,chroma,BlkSz,Olap,tm,glob,bBlur)
Return interleave(e.AssumeFieldBased,o.AssumeFieldBased).Weave
}

[FranceBB]

That is actually correct.
You're bobbing, filtering, dividing in fields, so that's alright.
By the way, I would encourage users to do it themselves rather than relying on it 'cause it's definitely gonna work for truly interlaced sources which will be bobbed to 59.940p or 50p, however, on telecined sources it will make the temporal prediction of MVTools useless, besides, even with truly interlaced sources, with Bob, fields would jump "up-down-up-down" making it harder to analyze and filter, so it's always better to let users handle it (hoping they'll use something decent as a deinterlacer and that they're aware of their source).
As to the alternative, again, you're right, as you're filtering each field separately and then putting them back together.
The second approach is probably the safest and I prefer it more than bobbing and re-interlacing. :)


Well done, Stainly, you're really following the government advice:

https://i.imgur.com/b3KhBsF.png

[StainlessS]

Cheers FaB, tanks for feedback.
I'll not bother adding to base script then.

EDIT: By the way, Bobber default of "Bob" is only because its a builtin filter, I would not recommend it.

[Emulgator]

FranceBB, now that poster is brilliant !

[piscator]

NOT FOR cartoon/anime, live video only, sorry.

What limitations are there that it should not work so great for cartoons? I have some DVDs with old children cartoons from 1956+ (Mole TV series, children cartoon) which I am trying to clean up. I have been looking for a bit to clean the spots and scratches (black and white, some quite large) from it and SpotLess works great on it :D Thanks, StainLessS!

[StainlessS]

Thanks for the feedback piscator,
MvTools motion compensation type stuff dont usually work to well on anime.
Did you use any non default args ? [maybe large block size, which I thought might help].

[piscator]

I just used default arguments.
Perhaps it works so well because this cartoon is quite 'static'. The background is usually completely fixed with sometimes a bit of panning /zooming on it. So, it is all really quite slow / limited motion.

[StainlessS]

static background might help,
anyways glad you are happy.

[FranceBB]

That's because cartoons / anime are generally not with the real framerate.
I mean, if you go through an anime encoded at 23.976fps, you'll almost definitely find repeated frames, but it's not a mistake, they're there by purpose as whoever animated it (I mean the studio) wasn't really able to deliver so many frames per seconds in terms of each individual character moving a part of the body etc. These repeated frames don't generally work well with MVTools, hence the assumption that it might not work for anime.
In your case it worked absolutely fine, so don't get me wrong, you might be lucky and have it working perfectly fine, but generally speaking, MVTools doesn't like non-live contents. ;)

FranceBB, now that poster is brilliant !

Ehehehe thanks

[piscator]

I mean, if you go through an anime encoded at 23.976fps, you'll almost definitely find repeated frames

Thanks, I can understand why that would make it more difficult. My source was just 25fps interlaced. And the motion is so slow, that when dropping half of the frames, you barely see a difference :D

I encoded this source for fun on crf 30 (x265, Ultralow) and it looks the same as on higher bitrates. But degraining, descratching and removing the spots make it much better.

[Frank62]

An idea:
Maybe possible to
-drop the dups, log it, maybe in StainlessS' database thing
-removedirt/spotless
-take the logged dups in another clip
-removedirt these
-bring everything together again
Could be a cleaning-chance for old anime :)

[Frank62]

Sorry, bad idea. Should rather be the opposite. Doubles are good for removedirting. Motion-compensated frames are of course less good.
So forget my proposal...

[StainlessS]

Maybe just try the temporal median filter that spotless uses.

RadT = ??? [int, try bout 2]
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]

If it is so static, it might work well.

[Frank62]

What exactly makes it more static than f. e. RemoveDirt?
To me it seems somehow more stepless, but how does it do this? "Blur" implicates some temporal interpolating, but it seems to be more than this.

[StainlessS]

Dont know but,
Here it is on Wiki:- http://avisynth.nl/index.php/MedianBlur2
And link (from wiki page) to a paper on it:- http://nomis80.org/ctmf.html

[Frank62]

Thanks, but it says nothing about the temporal part of filtering.

[Arx1meD]

Another variant of the script with saving fast moving objects.


# MvTools2, RemoveDirt, RgTools, DeSpot
Function SpotLess2(clip c, int "RadT", int "Spot", bool "DeGrain", int "RadT", int "ThSAD", int "ThSAD2", int "pel", bool "chroma", int "BlkSz", Int "Olap", bool "tm", Bool "glob") {
RadT = Default(RadT, 1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD, 10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2, ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel, 2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma, true) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz, 8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, 2) # Default half of BlkSz.
Tm = Default(tm, true) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob, true) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
Spot = Default(Spot, 32) # Spot size
DeGrain = Default(DeGrain, false) # MDeGrain in MvTools

pad = max(BlkSz, 8)
sup = c.ColorYUV(levels="PC->TV").MSuper(hpad=pad, vpad=pad, pel=pel, sharp=2)
sup_rend = MSuper(c, hpad=pad, vpad=pad, pel=pel, sharp=2, levels=1) # Only 1 Level required where not MAnalyse-ing.
MultiVec = MAnalyse(sup, multi=true, delta=RadT, blksize=BlkSz*2, blksizeV=BlkSz*2, overlap=OLap*2, chroma=Chroma, truemotion=Tm, global=Glob)#, search=5, dct=7, divide=0, plevel=0)

# c0 = DeGrain ? MDeGrainN(c1, sup_rend, MultiVec, RadT, mt=true, plane=4, limit=255.0, thSad=400) : c # gave a bad result

c1 = MCompensate(c, sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=130)
c1 = c1.ForwardClense() #faster than MedianBlurTemporal(radiusY=0, radiusU=0, radiusV=0, temporalradius=RadT) # Temporal median blur only [not spatial]

c1 = Repair(SelectEvery(c1, RadT*2+1, RadT), c.RemoveDust().Sharpen(0.1), mode=17) # Return middle frame

bvec1 = MAnalyse(sup, isb=true, blksize=BlkSz, blksizeV=BlkSz, overlap=OLap, divide=0, delta=1, search=5, dct=7, plevel=0, truemotion=true) # backward vectors
fvec1 = MAnalyse(sup, isb=false, blksize=BlkSz, blksizeV=BlkSz, overlap=OLap, divide=0, delta=1, search=5, dct=7, plevel=0, truemotion=true) # forward vectors
backw1 = MFlow(c1, sup_rend, bvec1)
forw1 = MFlow(c1, sup_rend, fvec1)
clipa = interleave(backw1, c1, forw1)

filtered = DeSpot(clipa, p1=6, p2=6, pwidth=Spot, pheight=Spot, mthres=16, mwidth=Spot/2, mheight=Spot/2, blur=1, motpn=true, seg=2, color=false, ranked=true, show=0)
restore = clipa.RemoveDust().Sharpen(0.1)

clipb = RestoreMotionBlocks(filtered, restore, alternative=RemoveGrain(clipa,2), pthreshold=10, cthreshold=10, gmthreshold=160, dist=3, dmode=2, noise=3, noisy=9, grey=false, debug=false, show=false)

SelectEvery(clipb, 3, 1)

DeGrain ? MDeGrain1(last, sup_rend, bvec1, fvec1, plane=4, limit=255.0, thSad=400) : last
}


function RemoveDust(clip input, int "mode") {
mode = Default(mode, 17)

clensed = Clense(input)
rep=Repair(clensed, input, mode=mode)

RemoveGrain(rep)
}

[StainlessS]

Arx1meD,
Perhaps you should create your own thread for your script function, also maybe give it another name other than Spotless2(),
as it does not function much at all like the Spotless filter.
I only tried this to see if it threw an error or not due to you having 2 [EDIT: formal] args named RadT,


AviSourcE("D:\parade.avi")
Spotless2()


However, caused access violation on line 33, ie the DeSpot line.

Suggest you create new thread, and correct/rename your script filter,
and post an example sample clip together with a working client script.
EDIT:

Function SpotLess2(clip c, int "RadT", int "Spot", bool "DeGrain", int "RadT", int "ThSAD", int "ThSAD2",
\ int "pel", bool "chroma", int "BlkSz", Int "Olap", bool "tm", Bool "glob") {


also, it is not good practice to destroy compatibility by inserting new formal arguments before existing arguments ie your

int "Spot", bool "DeGrain"


EDIT: Version Despot I was using was v3.6.3.0, there is a v3.6.3.1 here:- http://www.avisynth.nl/index.php/DeSpot
V3.6.3.1 does not cause access violation.

Strangely, it does not throw an error due to 2 different instances of the formal parameter "RadT", maybe a bit more checking needed in Avs+.

[Arx1meD]

Thank you StainlessS. I will fix the errors and create another thread.
I am using Avs+. When testing the script, I got no errors.

[StainlessS]

New Test [1.07beta] version of SpotLess, comments ?

Changed bBlur default from 0.0 [OFF] to 0.6, slower but should be considerably better on MC analysis.
Would it be preferable to remain at 0.0 faster.

Also, added DC [prefiltered detection clip], where supplied, bBlur prefiltering ignored and uses dc clip instead.


/*
SpotLess v1.07Beta. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.

NOT FOR cartoon/anime, live video only, sorry.

v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.
v1.05, Additional checks on args.
v1.06, Glob Default true, Almost always better.
v1.07Beta, bBlur Default changed from 0.0 OFF, to 0.6, slower but better for mc analysis. Maybe try 1.0 for HD to overcome HD 'gritiness'.
Added dc arg, detect clip.

SpotLess(clip c,int "RadT"=1,int "ThSAD"=10000,int "ThSAD2"=ThSAD,int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,
\ bool "tm"=true,Bool "glob"=True,Float "bBlur"=0.6, clip "dc"=Undefined)

RadT, 1 or more, Default 1. Removes Spots on up to RadT [Temporal Radius] consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD, 0 < ThSAD2 < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.

Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)

Chroma, Default True. MAnalyse chroma arg. If true, use chroma in block matching when creating vectors. Maybe use False if source B&W or color noise.

BlkSz, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]

OLap, Default half BlkSz, Block overlap.

Tm, TrueMotion Default True. Some folk swear truemotion=false is better.

Glob, Default True (True v1.06, was same as Tm, true almost always better), Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable.

bBlur, Default 0.6. If used, Suggest about 0.6, where MAnalyse create vectors is performed on denoised (blurred) super clip
for better motion analysis. Maybe try 1.0 for HD clip to counteract HD 'grittiness'. bBlur ignored if dc clip specified,
If providing bBlur, then ALWAYS specify as named argument ie bBlur=bBlur, we will likely insert any additional MvTools args
before bBlur to keep them together.

dc, Default UnDefined. If dc Specified, then must be same size and colorspace as source clip.
If detection clip specified then bBlur prefilter arg is ignored, and analysis is performed on
dc.MSuper() clip [ instead of c.Blur(bBlur).MSuper() ]. Allows to provide your own prefiltered clip.
If providing dc clip, then ALWAYS specify as named argument ie dc=DC, we will likely insert any additional MvTools args
before bBlur to keep them together.

*/

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


I'm quite happy with it as is, but anybody anything to say before I post as new non beta update.

Here some Local de-flicker whotsit, can be used for DC clip where local flicker.

Function DeFlickLocal(clip c) {
/*
Local DeFlicker. Didee, https://forum.doom9.org/showthread.php?p=1599682#post1599682
Calms local flickering, where some small object in frame flickers, rather than whole frame flickering.
[Didee:]
A small clip is created. The small clip is temporal-softened.
The difference achieved by temporalsoften is taken from the small clip, then re-scaled, and applied to the original clip.
Effectively, this is (very similar to) a spatial lowpass on the difference of a normal temporalsoften.
In particular with MVTools-Denoising etc., then the result makes a good source for motion vector search. No need for sluggish DCT=1.
*/

c
w=Width/8*2 h=Height/8*2 # Can be altered, but ~25% of original resolution seems reasonable
sm = c.bicubicresize(w,h, b=1.0/3, c=1.0/3) # b,c as defaults, b=blurring, c=ringing
smm = sm.temporalsoften (1,32,255,24,2).merge(sm,0.25)
smm = smm.temporalsoften(2,12,255,20,2)
c.mt_makediff(mt_makediff(sm,smm,U=3,V=3).bicubicresize(c.width,c.height,b=0,c=0),U=3,V=3) # blurring=0.0, ringing=0.0
}


Also, GMJCZP posted a script function Small_Deflicker():- https://forum.doom9.org/showthread.php?p=1812060#post1812060
which is based on DeFlickLocal(), which is an implementation of a Didee script as a function.
NOTE Didee comments.

[GMJCZP]

Hi TinMan, I wish you good luck with your script. The biggest challenge you are going to have is with anime, specifically with scenes with rain, because if the spot detection method is incorrect you will notice how the raindrops disappear before your eyes, KillerSpots was precisely tested exhaustively under these conditions

[StainlessS]

Hi GMJCZP, [I think I messaged you when I implemented Spotless, to let you know I extended your/didee script.]
I'm personally not bothered about anime, and said 'not for anime' in 1st or 2nd post, as uses RadT greater than 1, there is little
chance that any anime rain would survive RadT>=2.
If you aint tried it [SpotLess], give it a go, is almost magic with RadT>1, removes double spots, and even had it remove 4 consecutive BIG BIG BIG spots.
Glad to see you back, keep safe.

[Sharc]

SpotLess is perfect for my VHS sources. Thank you!

[StainlessS]

Fixed in post #16 [not the Spotless script]
Was

Prog = (UNSPOT_I) ? Prog.SpotLess(Radt=RADT_P,thsad2=ThSAD2_P,blksz=BLKSZ_P,tm=Tm_P,bBlur=bBlur_P) : Prog


Fixed

Prog = (UNSPOT_P) ? Prog.SpotLess(Radt=RADT_P,thsad2=ThSAD2_P,blksz=BLKSZ_P,tm=Tm_P,bBlur=bBlur_P) : Prog


Doing Babylon5 again [spent huge amount of time [hand edits] first time around).
But think I'll use TFM(pp=0) beforehand to get rid of Perverse Telecine [in some episodes], before using post #16 script.

Babylon5 [PAL version] Perverse Telecine mentioned here:- https://forum.doom9.org/showthread.php?p=1785472#post1785472

EDIT:
Episode 1 season 1 has some double frame spots [on progressive parts], so I thought I'de do it all again with spotless,
via the post #16 script, script is a bit slow [with doublerate output, which I'm doing this time].
I'm gettin about 60FPS with i7-8700 [so I guess maybe not that slow, but is only a bit faster than real time 50FPS output,
and 110 episodes, so is slow/tedious].

EDIT: "with doublerate output, which I'm doing this time"
OUTPUT=3 # 3=DblRate 50fps. Prog=Interpolate.
and TFM(pp=0) after AssumeTFF.

[coolgit]

Doing Babylon5 again [spent huge amount of time [hand edits] first time around).

EDIT:
Episode 1 season 1 has some double frame spots [on progressive parts], so I thought I'de do it all again with spotless,
via the post #16 script, script is a bit slow [with doublerate output, which I'm doing this time].
I'm gettin about 60FPS with i7-8700 [so I guess maybe not that slow, but is only a bit faster than real time 50FPS output,
and 110 episodes, so is slow/tedious].

EDIT: "with doublerate output, which I'm doing this time"
OUTPUT=3 # 3=DblRate 50fps. Prog=Interpolate.
and TFM(pp=0) after AssumeTFF.

Are you still doing B5 then? I have done s1 ep1 a while back, removed all the dust, scratch, etc with dirt cleaner which was about 95% good. The rest I'd spotted during playback. Upsampled it to 1920x1048, fixed some colouring, sharpening etc.
Hell a lot better than my dvd for sure.

Onto ep2 - Soul Hunter. Can you read this link for me

https://forum.videohelp.com/threads/402493-How-do-I-remove-these-vertical-lines

And possibly come up with a solution. Tried many software and can't clean it up. If you have the dvd then look at frame 13745 to 18634 (9mins 9 secs to 12:25) and you see those 3 static lines that appear in all scenes involving the non-regular casts.

I have minimised it using photoshop to create 3 vertical fixed mask and denoise to remove the bluish lines but there is a faint white line in parts left. Those scenes have slight band from WB bad attempted removal before dumping it onto the market. Those scenes looks too dark as if WB tried to reduce the visibility of the lines. Did some colour correction and looks good, just those pesky whitish lines.

Let me know if you have any solutions. Cheers.

[StainlessS]

OK, I found the lines, (although I did not originally even notice them in your sample file - I thought it must have been cleaned result).
Just tried my used script, and still there afterwards.

I personally have reached the conclusion that life is too short to be spending eons on such things,
do yourself a favour and live with them (look the other way when they show up on screen).

Wish I could give you a better answer, but I dont got one :(

EDIT:
I think this is about same as template script I used [I stopped mid Season 3, and am busy doing other stuff, continuing later].

###############
# PAL Hybrid Interlaced/Progressive mix, output to singlerate or doublerate.
###############

### DGIndex Template, PAL_Hybrid_DeintDoubleRate.avs
VideoFileName ="__vid__"
AudioFileName ="__aud__"
AudioDelay =Value("__del__")

MPEG2Source(VideoFileName)
AudioExt=RT_GetFileExtension(AudioFileName)

Audio= (AudioExt==".ac3") ? NICAC3Source(AudioFileName,channels=2,DRC=0)
\ : (AudioExt==".mpa"||AudioExt=="mp1"||AudioExt==".mp2"||AudioExt==".mp3") ? NicMPG123Source(AudioFileName,Normalize=False)
\ : (AudioExt==".wav") ? RaWavSource(AudioFileName)
\ : (AudioExt==".dts") ? NicDTSSource(AudioFileName)
\ : 0

Assert(!isInt(Audio),"NO AUDIO")

(!isInt(Audio)) ? AudioDub(Audio).DelayAudio(AudioDelay).Trim(0,0) : NOP # Trim, chop/add audio to length

(!isInt(Audio) && AudioRate() <> 44100) ? ResampleAudio(44100) : NOP
####### CONFIG ##########

AssumeTFF # Whatever (check) : Almost always TFF for PAL DVD. Default field order is BFF.

# Get rid Of Perverse Telecine
TFM(pp=0)
O=Last
AutoAdjust(Auto_Gain=true).Merge(o,0.6)


OUTPUT = 3 # 0=SingleRate 25FPS Progressive. Progressive src frames as input, interlaced src frames are even frames from dbl rate deinterlacing.
# 1=SingleRate 25FPS Hybrid. Progressive src frames as input, interlaced src frames dbl rate deinterlaced, reinterlaced to 25FPS after processing.
# 2=DblRate 50fps. Prog=ChangeFPS. Progressive src frames doubled with ChangeFPS, interlaced src frames are dbl rate deinterlaced.
# 3=DblRate 50fps. Prog=Interpolate. Progressive src frames double rate interpolated with JohnFPS [ie MvTools], interlaced src frames are dbl rate deinterlaced.

UNSPOT_P=true # Use SplotLess() on Progressive parts to remove spots and noise.
UNSPOT_I=False # Use SplotLess() on Interlaced parts to remove spots and noise.

RADT_P = 2 # SpotLess temporal radius (spots on 2 consecutive frames)
ThSAD_P = 5000 # Almost OFF.
ThSAD2_P = 5000 # Almost OFF.
BlkSz_P = 16
Tm_P = false
Glob_P = true
bBlur_P = 0.6

FRAMES_P=1 # Progressive McDegrainSharp(frames=FRAMES_P), (0=Dont McDegrainSharp)
CSHARP_P=0.5

RADT_I = 2 # SpotLess temporal radius (spots on 2 consecutive frames)
ThSAD_I = 5000 # Almost OFF.
ThSAD2_I = 5000 # Almost OFF.
BlkSz_I = 16
Tm_I = false
Glob_I = true
bBlur_I = 0.6

FRAMES_I=1 # Interlaced McDegrainSharp(frames=FRAMES_I), (Maybe double FRAMES_P for detail migration, although max possible for McDegrainSharp=3). (0=Dont McDegrainSharp)
CSHARP_I=CSHARP_P

# IsCombedTIVTC args, Defaults, 1st after comment hash.
CTHRESH = 12 # 10), Area combing threshold.
# Valid settings are from -1 (every pixel will be detected as combed) to 255 (no pixel will be detected as combed).
# This is basically a pixel difference value. A good range is between 8 to 12.
MI = 80 # 80), Number of combed pixels inside any of the blocky by blockx size blocks on the frame for the frame to be detected as combed.
CHROMA = True # True), Whether or not chroma is considered in the combed frame decision. [We want chroma as interlaced parts in color]
BLOCKX = 16 # 16), X-axis size of the window used during combed frame detection.
BLOCKY = 16 # 16), Y-axis size of the window used during combed frame detection.
# So in any single block, if 85[MI] out of 256 pixels pass CThresh as combed, then is Interlaced.
METRIC = 1 # 0) Detecto method: 0=Old, 1=New

# Deinterlace mode
BOBMOD=1 # 0=BOB, 1=YadifMod2, 2=QTGMC : QTGMC here is way memory/CPU intensive, req x64. Suggest YadifMod2

BOB_FUNC_0 = """Bob()""" # Cheap & cheerful Doublerate simple Bob.
BOB_FUNC_1 = """YadifMod2(Order=-1,Mode=1,edeint=NNedi3(Field=-2, nns=2))""" # Doublerate Yadifmod2 filter.
# # Field-2=double rate, internal parity value to start : Order-1=Internal Parity. Mode 1=double rate do spactial check (3=no spatial check).

BOB_FUNC_2 = """QTGMC(Preset="fast",NoisePreset="Fast",SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1, tr2=0)""" # Doublerate QTGMC filter.

FUNC = Select(BOBMOD, BOB_FUNC_0, BOB_FUNC_1, BOB_FUNC_2)
FNAME = Select(BOBMOD,"BOB", "YADIF", "QTGMC")


PRE_FETCH=6 # 1 = No Prefetch, Else eg 4 = Prefetch(4) ie MT mode

### # Display stuff
SUBS = True # Show where Deinterlaced/ReInterlaced
STACK = True # Show Src and Result
TITLE = True # Show Title Bar
TITNUM = True # Show Frame Number on Title Bar (Only if TITLE)
FINAL = True # True, switch off SUBS, STACK, and Title

##### End Config ##########

Src=Last

SUBS = (!FINAL&&SUBS)
TITLE = (!FINAL&&TITLE)
STACK = (!FINAL&&STACK)


### IsCombedTIVTC with args, as a string eg "IsCombedTIVTC(cthresh=10,mi=85,chroma=true,blockx=16,blocky=16,metric=1)"
COMBDET = RT_String("IsCombedTIVTC(cthresh=%d,mi=%d,chroma=%s,blockx=%d,blocky=%d,metric=%d)",CTHRESH,MI,CHROMA,BLOCKX,BLOCKY,METRIC)
SUBTXT = (OUTPUT==0) ? FNAME+"_DEINTERLACED-EVEN" : (OUTPUT==1) ? FNAME+"_RE-INTERLACED" : FNAME+"_DblRate-DEINTERLACED"

# Interlaced parts
# DeSpot, Deinterlace to doublerate, McDeGrainSharp to migrate detail between original fields[now frames].
Deint = Src.Eval(FUNC) # Double Rate deinterlace
Deint = (OUTPUT==0) ? Deint.SelectEven : Deint # If (OUTPUT==0) McDegrainSharp & SpotLess done after Deinterlace.SelectEven, Else after doublerate deinterlace.
Deint = (UNSPOT_I) ? Deint.SpotLess(Radt=RADT_I,thsad=ThSAD_I,thsad2=ThSAD2_I,blksz=BLKSZ_I,tm=Tm_I,glob=Glob_I,bBlur=bBlur_I) : Deint
Deint = (FRAMES_I>0) ? Deint.McDeGrainSharp(frames=FRAMES_I,csharp=CSHARP_I) : Deint
Deint = (OUTPUT==1) ? Deint.SeparateFields.SelectEvery(4,0,3).Weave : Deint # If singlerate Re-Interlacing
Deint = (SUBS) ? Deint.Subtitle(SUBTXT,Align=5) : Deint

# Progressive parts, Despot, Degrain sharp done still @ singlerate
Prog = Src
Prog = (UNSPOT_P) ? Prog.SpotLess(Radt=RADT_P,thsad=ThSAD_P,thsad2=ThSAD2_P,blksz=BLKSZ_P,tm=Tm_P,glob=Glob_P,bBlur=bBlur_P) : Prog
Prog = (FRAMES_P>0) ? Prog.McDeGrainSharp(frames=FRAMES_P,csharp=CSHARP_P) : Prog
Prog = (OUTPUT==2) ? Prog.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator) : (OUTPUT==3) ? Prog.JohnFPS() : Prog # Progressive to O/P rate

SrcTest = (OUTPUT<=1) ? Src : Src.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)
# Select Progressive / Deinterlaced frame, based on IsCombedTIVTC detection on original source clip frame.
Result=ConditionalFilter(SrcTest, Deint, Prog, COMBDET,"=","true")

SrcOut = (!TITLE)?SrcTest:(OUTPUT<=1) ? Src.TSub("Source",TITNUM) : Src.TSub("Source DblRate via ChangeFPS, ie dupes",TITNUM).ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)

Result = (!TITLE)?Result :(OUTPUT<=1) ? Result.TSub(String(OUTPUT,"%.0f) Result SingleRate"),TITNUM) : (OUTPUT==2) ? Result.TSub("2) Result DblRate (Progressive frames via ChangeFPS)",TITNUM)
\ : Result.TSub("3) Result DblRate (Progressive frames via JohnFPS, ie MvTools2)",TITNUM)

(STACK) ? Stackhorizontal(SrcOut, Result) : Result

(PRE_FETCH>1) ? Prefetch(PRE_FETCH) : NOP # I had problems using x64 and Prefetch [Access Violation], culprit was old IsCombedTIVTC, fixed via Pinterf version.
RT_SignalDar2(16,9)
Return Last

#####################
# Stack Overhead Subtitle Text, with optional FrameNumber shown.
Function TSub(clip c,string Tit,Bool "ShowFrameNo",Int "Col"){
c.BlankClip(height=20,Color=Default(Col,0))
(Default(ShowFrameNo,False))?ScriptClip("""Subtitle(String(current_frame,"%.f] """+Tit+""""))"""):Trim(0,-1).Subtitle(Tit)
Return StackVertical(c).AudioDubEx(c)
}


EDIT: I am just doing the lot [110 eps] with exact same script [no messing about].
EDIT: Resulting above scripts were fed direct into MeGUI x64. [all x64 plugs are available]
EDIT: I found that there are RadT=2 spots in seasons 1, 2 & 3, others unknown
(I scan through ep1 of a season, if single double spot found, then I do the lot)
I dont even bother with spots on interlaced sections, they're naff/indistinct anyways and dont need more over processing.

EDIT:
Wish I could give you a better answer, but [B]I dont got one :(
My "English" has been greatly enhanced by this here clip [Andrew Dice Clay, "Even bums don't not got a car", in "One night at McCools"]:-
[!!!WARNING!!!, Colourful language]:- https://www.youtube.com/watch?v=uthKzsSY2AY

[johnmeyer]

I pointed out in your thread over in the Videohelp format, that eliminating vertical lines is a tough challenge. However, unlike film scratches which are virtually impossible to remove, you can remove your scratches. However, the problem will be the inevitable blur that will result from blending pixels from the adjacent area. I suggested using the VirtualDub Delogo plugin, but I'm sure there are better choices.

Spotless is almost certainly the wrong tool because it is looking for differences between frames: when a spot shows up for just one frame, it is removed. Since your lines do not change from frame-to-frame, there are no difference vectors, and Spotless likely won't do anything.

Some of the experts on using AVISynth masking may have some more clever ideas. You might start your own thread here and see what you get.

[StainlessS]

I think he was just asking me because I said I was doin the same series, just in case I did something clever with those near invisible lines.
110 eps is one helluva lot of viewing never mind examining and special casing for those lines.
If you're getting big bucks for a total remaster of original source, then is maybe sane, otherwise not.

[StainlessS]

John or coolgit or anybody really,



Avisource("SH_2_secs.avi.avi") # Did not have codec for FOURCC, redid in ffmpeg to UT_Video so AVI.AVI
ConvertToYV24
O=Last

# Intention: Single pixel width, vertical line/scratch reduction.
GeneralConvolution(0, "
1 0 1
2 4 2
1 0 1
")

StackVertical(O,Last)


I thought above might work-ish, just made up the matrix string on gut feelin' no idea if is any kind of standard matrix
but it does seem to make some difference.
[I cant really see it properley lousy eyes and I'm on a smaller monitor, main machine busy doing something else, and not
got Avs stuff installed on it (actually messin with linux install on it)].

Can you test out above and see if it is of any worth at all [and if it blurs much].

Only my guess work and I got almost zero experience of same, so might be rubbish.

EDIT: Could be applied to frames/ranges using clipclop. Gotta be faster/less-user-interaction than delogo or descratch or whatever.

EDIT: In VDub2 at 200%, I'de say reduces scratch to about 50% visiblility, not perfect but does not seem to create additional artefacts
that I can see.

[coolgit]

Stainless

Is your template above for MeGui only?

The GeneralConvolution didn't do anything. I don't understand it either. Vdub2 has it and I've never used it. All it did what change the output to black with very faint movements.

What i am looking for, dunno if it can work, it a simple replace bad pixel type plugin. The way i see it would work like this.

Instruct start hor pixel(say 300) and end hor pixel (305) and full vert length - like masking and only work in this area and nowhere else.
Pixel 300 to 305 are considered bad.
Lets focus on 5 main colours. BWRGb. | = start and end mask boundary.


Small example

Good bad good
pixels pixels pixels
BBBB|WbbbWW|BBBB
BBBB|WWbbbW|BBBB
GGGG|WWbbbW|RRRR
GGGG|WbbbWW|RRRR

would become by using previous good hor pixel(299) and hor pixel(306) on both side alternately to overwrite the bad pixels

BBBB|BBBBBB|BBBB
BBBB|BBBBBB|BBBB
GGGG|GGGRRR|RRRR
GGGG|GGGRRR|RRRR

The above looks simple enough to me. No blurring, no band lines, no random guessing based on fancy algorithms, etc.

How hard can it be?

[coolgit]

I pointed out in your thread over in the Videohelp format, that eliminating vertical lines is a tough challenge.

I know but i have been making headway using photoshop and topaz denoise.

However, unlike film scratches which are virtually impossible to remove, you can remove your scratches. However, the problem will be the inevitable blur that will result from blending pixels from the adjacent area. I suggested using the VirtualDub Delogo plugin, but I'm sure there are better choices.

Topaz denoise removed source file faint bands and also removed blur which was bloody useful. It actually removed the grain inside the whites of the eyes making the eyes look visibly better. Faint white lines about 3/4 pixel width remains. I had an idea how to remove bad pixels and replace them with good pixels in my last post above. Is it feasible with avisynth?

Some of the experts on using AVISynth masking may have some more clever ideas. You might start your own thread here and see what you get.

This what i need, masking. I want to be able to focus on the 3 lines and nothing else. Hence why i am using photoshop.

[StainlessS]

I think I cocked up the initial post, and removed a ",8" after the matrix string, maybe you took a copy before I fixed it.

Anyways, here a comparison, top is result - Original [should be mid grey-ish],
bottom is same but differences amplified GREATLY, maximum amp'ed.
You can see that the light lines have been modified, and are darker, meaning darker than original, as intended.
It does produce global frame changes, as does most denoise type processing.


Avisource("SH_2_secs.avi.avi") # Did not have codec for FOURCC, redid in ffmpeg to UT_Video so AVI.AVI
ConvertToYV24
O=Last
GeneralConvolution(0, "
1 0 1
2 4 2
1 0 1
")
D1=ClipDelta(Last,O,false)
D2=ClipDelta(Last,O,true)
StackVertical(D1,D2) # Top Subtract, bottom amplified
Return last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}


Click Me
https://i.postimg.cc/zygf74S6/a-00.jpg (https://postimg.cc/zygf74S6)

Pretty much all avisynth scripts, can of course by loaded into Megui main screen top input Avisynth Script editbox.
[or just view in eg VDub2]

[coolgit]

StainMoreH:D:D

Will try your new script later.

Just played with photoshop (36 action commands) a bit more and boooooom, i manage to remove the white line in 98% of all the frames. Just a small 1 pixel thin, about 10 pixel high, line in some obscure location. At normal zoom i can not see anything. I would have to zoom to 200% or more before noticing faint band, small bits of line and i know where to look. A random person wouldn't. Will convert the frames to mp4 and see how that pans out.

[coolgit]

Click Me
https://i.postimg.cc/zygf74S6/a-00.jpg (https://postimg.cc/zygf74S6)



That is what i got using vdub2. I copied your script but changed Avisource to FFVideoSource.

What am i suppose to do with the grey clip?

[StainlessS]

That is just the difference between src and dest. Use the original (fixed) script.

EDIT: Fixed original

Avisource("SH_2_secs.avi.avi") # Did not have codec for FOURCC, redid in ffmpeg to UT_Video so AVI.AVI
ConvertToYV24
O=Last

# Intention: Single pixel width, vertical line/scratch reduction.
GeneralConvolution(0, "
1 0 1
2 4 2
1 0 1
")

StackVertical(O,Last)

[StainlessS]

Coolgit,
maybe of use [I've used my GeneralConvolution thingy as fixed clip, but could use some other de-streak'ed clip].


/*
Requires Avs+, FrameSel[v2.21 just posted new version], ClipClop, RT_Stats
Frames file specifies frames/ranges to fix.
*/

Avisource("SH_2_secs.avi.avi")
STACK = false # True, result is STACKED, else Not
SHOW = True # True, Show ClipClop metrics, else Not
Frames = ".\Frames.txt" # Frames/Ranges to fix, file, either single frame number or range eg 100,110 or 100 110.
###
Frames = Frames.RT_GetFullPathName
###
ConvertToYV24
O=Last

# We're gonna use GeneralConvolution de-streaker whotsit, is applied to ENTIRE clip.
De_Streak = GeneralConvolution(0, "
1 0 1
2 4 2
1 0 1
")

FixFrames(O,De_Streak,Frames,Show=SHOW) # Replace frames/ranges in frames file with De_Streaked frames

###
STK = StackHorizontal(O,Last) # Lft=Original, Rgt=Fixed
#STK = StackVertical(O,Last) # Top=Original, Bot=Fixed ## If Preferred

(STACK) ? STK : Last
ConvertToYV12
Return last

Function FixFrames(clip c,clip repc,string Frames,Bool "Show") {
Show = Default(Show,False)
Assert(Frames!="","FixFrames: Frames cannot be ''")
Frames = Frames.RT_GetFullPathName
Assert(Frames.Exist,"FixFrames: Frames file Does Not Exist")
# Temp filename for Frames file re-written as ClipClop Command file [with clip index of 1 for replacement clip].
ClipClopCmd=(".\~ClipClipCmd_" + RT_LocalTimeString(file=True) + ".Txt").RT_GetFullPathName
# Convert To ClipClop Cmd file, REQUIRES FrameSel v2.21 for PruneIx arg and ClipClop clip_index=1 conversion.
FrameSel_CmdReWrite(c,ClipClopCmd,cmd=Frames,Ordered=true,range=true,Prune=True,PruneIx=1)
ClipClop(c,repc,Cmd=ClipClopCmd,show=SHOW)
RT_FileDelete(ClipClopCmd) # Done with it.
return Last
}

EDITED: OOps, added in blue. RT_FileDelete(ClipClopCmd)

EDIT:
Contents of Frames file [Frames.txt] to fix frame 5, and range 10 to 20, and range 30 to 40

5
10,20
30,40

EDIT: 5; 10,20; and 30,40 picked out of the hat, use your own numbers.

[coolgit]

Stainless or anybody

Need help getting the output right. Currently both left and right viewers show the spotless output.
I want the left to stick with the original and the right to show the output so i can compare.

LoadPlugin("C:\Program Files (x86)\AviSynth\plugins+\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth\plugins+\MedianBlur2.dll")
source=FFVideoSource("E:\currser\BABYLON5\new\Soul Hunter.avi").convertTOYV12(interlaced=false).killaudio()

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.0) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

output=SpotLess(source)

return(output)

Tired interleave but got original left, and right both original and new output, very slow way of checking.

So far impressed with this one.
cheers

[StainlessS]

this what you're after


# ...

output=SpotLess(source)
StackHorizontal(source,output) # stack side by side, source left, output right
#StackVertical(source,output) # stack vertical, source top, output bot
return last

EDIT: Untested

EDIT:
Currently both left and right viewers show the spotless output.
At a guess, maybe you are using Vdub2, and seeing VD2 input and output panes.
Those panes are VD2 input and output, which are probably both the same, [assuming no editing in vd2].
What you want is input and output in avisynth, as stacked above. [I mostly have VD2 output pane @ tiny size].

EDIT: Also, maybe try with

output=SpotLess(source,tm=false,bblur=0.6)

Spotless will likely not be of much use for your streaky lines thingies, unless there is horizontal motion 'though' them.

[coolgit]

this what you're after


# ...

output=SpotLess(source)
StackHorizontal(source,output) # stack side by side, source left, output right
#StackVertical(source,output) # stack vertical, source top, output bot
return last


That worked, not perfect but better than nothing.



At a guess, maybe you are using Vdub2, and seeing VD2 input and output panes.
Those panes are VD2 input and output, which are probably both the same, [assuming no editing in vd2].
What you want is input and output in avisynth, as stacked above. [I mostly have VD2 output pane @ tiny size].

So there is no way one can override right side of vdub using avisynth?

EDIT: Also, maybe try with

output=SpotLess(source,tm=false,bblur=0.6)

Spotless will likely not be of much use for your streaky lines thingies, unless there is horizontal motion 'though' them.

The above threw an error column 29 missing , or ] edit: scrap that, i had [/COLOR] in my script.

Got it to work however output=SpotLess(source) is better. A defect 100% cleared up and the above only 70% cleared up. Culprit tm=false... when true is did better, say 95%.

I was just testing your script. It worked a lot of better when there are motion (fast and/or close to camera) as dirt cleaner took chunks of pixel and left artefacts behind.

However, 1st and last frame before scene changes are not processed. Does your script needs to look behind and ahead 1 frame for analysis to detect spot?

How i denoise a bit more?

[StainlessS]

That worked, not perfect but better than nothing.
Well if its still your streaks you're after fixing, then Spotless is probably not a good solution,
Spotless is intended to fix temporal anomalies, mainly Spots [hence the name].
A spot exists [usually] on a single frame, and it is that that enables detection, your stripey streaky things are
spatial, in the same spacial location, temporally kinda constant, so the spotless doofah dont work too well.

Did you try the script I recently gave, using ClipClop, can just use frames file containing "0,0" meaning all frames if you want to see all frames.
It will not be perfect but gottat be bettr than Spotless. [calms your streaks to maybe 50% as visible].

So there is no way one can override right side of vdub using avisynth?
Vdub has no idea at all what the input to avisynth was, only the output from it.

Does your script needs to look behind and ahead 1 frame for analysis to detect spot?
Yes, its a temporal filter, and only detects spots where the non spot pixels outnumber the spot pixels, with a temporal span of 2*RadT+1 frames. [RadT=temporal radius]

Give me a little while, I want to try something, only I'm a bit busy so might take some time.

[coolgit]

Coolgit,
maybe of use [I've used my GeneralConvolution thingy as fixed clip, but could use some other de-streak'ed clip].


/*
Requires Avs+, FrameSel[v2.21 just posted new version], ClipClop, RT_Stats
Frames file specifies frames/ranges to fix.
*/

Avisource("SH_2_secs.avi.avi")
STACK = false # True, result is STACKED, else Not
SHOW = True # True, Show ClipClop metrics, else Not
Frames = ".\Frames.txt" # Frames/Ranges to fix, file, either single frame number or range eg 100,110 or 100 110.
###
Frames = Frames.RT_GetFullPathName
###
ConvertToYV24
O=Last

# We're gonna use GeneralConvolution de-streaker whotsit, is applied to ENTIRE clip.
De_Streak = GeneralConvolution(0, "
1 0 1
2 4 2
1 0 1
")

FixFrames(O,De_Streak,Frames,Show=SHOW) # Replace frames/ranges in frames file with De_Streaked frames

###
STK = StackHorizontal(O,Last) # Lft=Original, Rgt=Fixed
#STK = StackVertical(O,Last) # Top=Original, Bot=Fixed ## If Preferred

(STACK) ? STK : Last
ConvertToYV12
Return last

Function FixFrames(clip c,clip repc,string Frames,Bool "Show") {
Show = Default(Show,False)
Assert(Frames!="","FixFrames: Frames cannot be ''")
Frames = Frames.RT_GetFullPathName
Assert(Frames.Exist,"FixFrames: Frames file Does Not Exist")
# Temp filename for Frames file re-written as ClipClop Command file [with clip index of 1 for replacement clip].
ClipClopCmd=(".\~ClipClipCmd_" + RT_LocalTimeString(file=True) + ".Txt").RT_GetFullPathName
# Convert To ClipClop Cmd file, REQUIRES FrameSel v2.21 for PruneIx arg and ClipClop clip_index=1 conversion.
FrameSel_CmdReWrite(c,ClipClopCmd,cmd=Frames,Ordered=true,range=true,Prune=True,PruneIx=1)
ClipClop(c,repc,Cmd=ClipClopCmd,show=SHOW)
RT_FileDelete(ClipClopCmd) # Done with it.
return Last
}

EDITED: OOps, added in blue. RT_FileDelete(ClipClopCmd)

EDIT:
Contents of Frames file [Frames.txt] to fix frame 5, and range 10 to 20, and range 30 to 40

5
10,20
30,40

EDIT: 5; 10,20; and 30,40 picked out of the hat, use your own numbers.

No function named RT_GetFullPathName error

[kedautinh12]

Requires Avs+, FrameSel[v2.21 just posted new version], ClipClop, RT_Stats

[coolgit]

Requires Avs+, FrameSel[v2.21 just posted new version], ClipClop, RT_Stats

Got all that.

[StainlessS]

Got all that.
RT_GetFullPathName is from RT_Stats,
Maybe you got x64 plug in x86 plugins, or something.

[coolgit]

dll was missing for some strange reason.

Got it working to and i can't see any changes at all.

[coolgit]

Stainless

Spotless worked great. Tried many ways and watched them on my TV. Small improvements here and there. Lines can't been seen on the TV, monitor yes but that's because it's in my face. On the TV the noticeable part is when the camera is still and there is a close up scene. The 3 lines shows up when there is dark background on the bottom 1/3.

So i decided to have cuppa break. Then a brainwave. Photoshop clone. Previously didn't work because one can not clone precisely frame by frame thus it created more flickering. My brainwave is do the cloning quickly for the noticeable parts, approx 75 frames, then apply spotless. The cloning differences between frames was removed and flickering non-existent. Now it looked great. Miles better than the original.

Pleased with my hard work I proceed to save it. Didn't work. I got a vid file with both org and new from the output in vdub.

How do i save the result of the work after spotless?

[StainlessS]

STACK = false # True, result is STACKED, else Not
SHOW = True # True, Show ClipClop metrics, else Not

Make STACK=False, that was intended for user configuration.
SHOW was for if you were using ClipClop. [where you would switch that off too].


Good Job :)

[coolgit]

STACK = false # True, result is STACKED, else Not
SHOW = True # True, Show ClipClop metrics, else Not

Make STACK=False, that was intended for user configuration.
SHOW was for if you were using ClipClop. [where you would switch that off too].


Good Job :)

The above isn't the spotless script?

[StainlessS]

OK, then just comment out StackHorizontal/StackVertical lines [precede line with '#'],
or delete them.

You really ortta be gettin to learn a bit of avisynth script yourself, people are not gonna be wantin' to spoon feed you forever. :)

[coolgit]

It is your script. It is not my fault if avisynth is complicated with crap 'manual'.

Showing a new output and saving it, is the most basic task that all software does in 5 seconds. 24 hours later and i still can't do the basic. I can only learn as i go along bit by bit.

This is the latest and obviously "the script's return value was not a video clip" pops up.

LoadPlugin("C:\Program Files (x86)\AviSynth\plugins+\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth\plugins+\MedianBlur2.dll")
source=FFVideoSource("E:\currser\BABYLON5\new\Soul Hunter.avi").convertTOYV12(interlaced=false).killaudio()



Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.3) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


#output=SpotLess(source,tm=true,bblur=0.6)
#output=SpotLess(source)
#StackHorizontal(source,output) # stack side by side, source left, output right
#StackVertical(source,output) # stack vertical, source top, output bot
#return source
return last
#return spotless(source)

[VoodooFX]

This is the latest and obviously "the script's return value was not a video clip" pops up.

You are trying to return "last" when not doing anything in your script.

Maybe this you want:
output=SpotLess(source)
return output

[coolgit]

You are trying to return "last" when not doing anything in your script.

Maybe this you want:
output=SpotLess(source)
return output


I tried that but the left viewer and right viewer of vdub is the same. Is it possible to have
return source output etc or does return only has 1 expression.

[StainlessS]

Vdub2 is not part of nor related to avisynth, it is a stand alone editor, with its own input and output viewing windows.
You can turn off VD2 windows, either input or output [it calls them panes I think], in the view menu, [again I think, I'm not on windows].
Avs users may use VD2 as a viewer for avs files, but you can view the output of AVS via eg PotPlayer, or MPC-HC, or in the AVS
dedicated GUI AvsPMod, your choice.

EDIT: VD2 does have an AVS script editor, added because I asked for it, by that nice guy "Shekh The Magnificent",.

EDIT: Unless you've done some filtering in VD2 of input Avisynth script or other video input,
both input and output panes will be exactly the same.
If avisynth outputs a stacked clip, then the stacked output will show in both VD2 panes.
As I said earlier, I usually make the output pane in VD2 as small as possible and only view a single pane. [or you can turn it off].

EDIT: Kinda nifty name, CoolGit, I'm sorta wishin' I'de gone with Ol'Fart, or I knida gotta hankering for BogBlocker too :)

[coolgit]

I do get the above but avisynth doesn't make it easy to work with.

So i can compare using StackHorizontal then when i am happy put a # before StackHorizontal then refresh then save new file. Next time i've got to remember to remove the #. Back and forth.

Surely avisynth can do better at outputting.

Is there not a save command for script? Use StackHorizontal for comparison then type save(newfilename) in script.

It's all working now. Is there a way of increasing threshold or whatever in spotless to increase effectiveness. Sometime the spot replacing pixels are a bit too light. ThSAD,10000 what does this do? I tried playing around with the numbers but couldn't see any effects.

[StainlessS]

Avisynth is video editing frameserver, it takes in some video clip, edits in some way and then produces an output video clip.
Thats all it does, thats all its meant to ever do.
I think that you originally used it with MeGUI, if avisynth were as you suggest, it would not be usable with MeGUI.
The purpose behind its use was to be able to eg feed into an encoder without saving the edited file to disk,
and the massive amount of disk space required to do that. [originated in about 2002 when hard drives were much smaller].

[EDIT: my 1st hard drive was 20MB and cost aboout £400.00, in late 80's, early 90's]

Maybe you would be happier with a more traditional video editor, you dont sound like you feel cut out for it.

Current beta of Spotless, with docs.
http://forum.doom9.org/showpost.php?p=1939724&postcount=69

If you need further background, you could read MvTools2 docs too.

EDIT: VirtualDub can [or older versons of it used to, probably still can] act as a FrameServer,
but with large fast disks that ability becomes maybe obsolete.
Avisynth has a pretty big bank of plugins and scripts, that can and are being used by many open source
projects, nobody wants to throw that in the bin. If you need an avs specific gui, try AvsPmod, thats what its for.

[some people use a text editor (me included) and some viewer eg MPC-HC player, that is their choice, you also have yours]

[coolgit]

[EDIT: my 1st hard drive was 20MB and cost aboout £400.00, in late 80's, early 90's]

Maybe you would be happier with a more traditional video editor, you dont sound like you feel cut out for it.

105mb approx £200 1991. A friend and I spent days deliberating between 105 and 115mb - costing £40 more.

Nowadays looking back it sound pathetic.

I do use video editor, a fair few according to tasks. But dust, scratches, spots, scene changes, all are pretty poor. Even pro software costing zillions are crap.

[StainlessS]

Not sure, maybe mine was more expensive than for typical PC, was for Atari-ST, and more specific interface, ACSI rather SCSI, I think,
but I guess coulda been as early as about 86. [EDIT: Actually, I still got it]
My [first] Sinclair QL 'Floppy' drive was about £150.00 - £200.00, [with some special interface, about 256KB (perhaps 128KB) RAM and maybe some toolkit], and I bought from the firm I worked for at the time [bit cheaper].

[VoodooFX]

I tried that but the left viewer and right viewer of vdub is the same. Is it possible to have
return source output etc or does return only has 1 expression.

AvS is not vdub's filter, output of AvS is input for vdub, you can run two instances of vdub and do compare switching between them.

[coolgit]

StainlessS

Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.

True works better. It took a nice longish scratchy defect and replaced it 99%. I tried false and the replacement was 60%.

[StainlessS]

True works better.
And there was me considering making false the default, guess I maybe leave as it is.

[coolgit]

Rendering spotless is 20 fps. Any way to make it faster?

Default(ThSAD,10000) - what does the number do? Tried playing with and couldn't see any effects.

[johnmeyer]

Are you using multi-threading? Set Prefetch() to use multiple cores.

[coolgit]

Are you using multi-threading? Set Prefetch() to use multiple cores.

Using vdub2 script for spotless. Is set prefetch suppose to go in the script?

[coolgit]

Had some fun yesterday/today using spotless on some Laurel and Hardy films. Those great B&W films had too many (5 to 20 per frame) flickering. Claims by various film restoration companies are outrageous. How they claim to have restore the film and not remove one flicker is beyond me. It takes no more than 10 minutes or less per film.

I improved the quality by 1000% simply by removing all the flickering. Some were the size of 2 pence coin. In some case I ran spotless twice on the same film.

All is great. Film Dirt Cleaner consigned to the recycle bin and eradicated forever.

[StainlessS]

I'm guessin' that some love the flicker. [There's nowt so queer as folk. https://en.wiktionary.org/wiki/there%27s_nowt_so_queer_as_folk ]

[StainlessS]

CoolGit,

Maybe try this out, could be useful in B&W restoration. [selective processing]


ColorBars(pixel_type="YV12").ShowFramenumber.KillAudio

# Need the latest Beta Spotless for DC clip.
DC_Radius=7 # Frames to average
DC_SC=25 # Scene change detect
DC=TemporalSoften(DC_Radius,255,255,DC_SC,2).Blur(0.6,0.6) # Detect clip for Spotless, hopefully less prone to be fooled by flickering [untested, tweak DC_Radius]

#DC = Blur(0.6,0.6) # Alternative prefiltered analysis clip, without TemporalSoften, or Roll-Your-Own DC.

V1 = SelectEvery(1,-1) # CopyFromPreviousFrame [replace current frame with its predecessor]
V2 = SelectEvery(1, 1) # CopyFromNextFrame [replace current frame with its successor]
V3 = Spotless(RadT=1,BlkSz=16,TM=False,Glob=true,dc=DC) # Un-spot Single frame spots. DC prefiltered clip for motion analysis.
V4 = Spotless(RadT=2,BlkSz=16,TM=False,Glob=true,dc=DC) # Un-spot Double frame spots. DC prefiltered clip for motion analysis.
V5 = Spotless(RadT=3,BlkSz=16,TM=False,Glob=true,dc=DC) # Un-spot Triple frame spots. DC prefiltered clip for motion analysis.
V6 = FlipVertical
V7 = Invert
# Etc

Nicks = """ # NickNames (pseudonyms) for use in SCMD or CMD file.
CP = 1 # CopyFromPreviousFrame ::: 1st replacement Clip V1 in clipclop args
CN = 2 # CopyFromNextFrame ::: 2nd replacement Clip V2
DS1 = 3 # DeSpot single spots ::: 3rd replacement Clip V3
DS2 = 4 # DeSpot double spots ::: etc
DS3 = 5 # DeSpot triple spots
FV = 6 # FlipVertical
INV = 7 # Invert
"""

SCMD = """ # ClipClop Commands given in SCMD string [can supply in CMD file too], can use Nickname or replacement clip index to specify clip.
CP 10 # Replace bad frame at 10 with frame before it [Maybe bad End Of Scene frame - glue @ Splice]
CN 20 # Replace bad frame at 20 with frame after it [Maybe bad Start Of Scene frame]
DS1 30 # Spot at 30
DS2 40,-2 # Double Spot at 40 [-2 = 2 frames replaced]
DS3 50,52 # Triple Spot at 50 [range 50->52 replaced]
1 60 # CopyFromPreviousFrame frame 60, we used clip replacment index 1 [ie V1] instead of nickname CP.
FV 70,75 # FlipVertical 70->75
INV 80,0 # Invert 80 -> End Of Clip
"""

CMD = "" # Not Used, Maybe eg CMD=".\ClipClopCommand.txt"

SHOW=True # Show ClipClop metrics

ClipClop(Last, V1,V2,V3,V4,V5,V6,V7, scmd=SCMD, cmd=CMD, NickName=Nicks, show=SHOW)


You can define up to 255 replacement clips in ClipClop.

EDIT: There is a Donald Graft plugin called DeFlick, I think, maybe use before Spotless for B&W.
EDIT: Maybe forget DeFlick before spotless, it would likely spread spots over multiple frames making them harder to remove. [could try after spotless though]

External Filters:- http://avisynth.nl/index.php/External_filters
and Internal filters:- http://avisynth.nl/index.php/Internal_filters

[coolgit]

Regarding your pal hybrid script, I am getting no function McDeGrainSharp. I got it and created a McDeGrainSharp.avs file and put it in avisynth 32/64 and vdub 32 and not working.

[StainlessS]

I is guessin you is usin v2.6, avs 2.6 requires .avsi extension, avs+ requires *.avs or *.avsi
mobile in public house :)

[coolgit]

using avs+ 3.7

[kedautinh12]

using avs+ 3.7

You need use AVSMeter or Avisynth Info Tool to check your AVS plugins and scripts. After you share your log here
https://forum.doom9.org/showthread.php?t=173259

[coolgit]

Log file created with: AVSMeter 3.0.9.0 (x64)
Command line switches: -avsinfo

[OS/Hardware info]
Operating system: Windows 7 (x64) Service Pack 1.0 (Build 7601)

CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz / Coffee Lake (Core i7)
MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, FMA3, RDSEED, ADX, AVX, AVX2
6 physical cores / 12 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, x86_64)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\System32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 15:47:06 (UTC)
PluginDir2_5 (HKLM, x64): C:\Program Files (x86)\AviSynth\plugins64
PluginDir+ (HKLM, x64): C:\Program Files (x86)\AviSynth\plugins64+


[C++ 2.5 Plugins (64 Bit)] [Version, Time stamp]
C:\Program Files (x86)\AviSynth\plugins64+\Decomb.dll [n/a, 2013-12-01]

[C++ 2.6 Plugins (64 Bit)] [Version, Time stamp]
C:\Program Files (x86)\AviSynth\plugins64+\AutoAdjust.dll [2.6.0.0, 2016-01-05]
C:\Program Files (x86)\AviSynth\plugins64+\ClipClop_x64.dll [1.27.0.0, 2021-07-27]
C:\Program Files (x86)\AviSynth\plugins64+\ConvertStacked.dll [n/a, 2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\DelogoHD.dll [1.0.0.0, 2019-12-09]
C:\Program Files (x86)\AviSynth\plugins64+\DePan.dll [2.13.1.6, 2020-05-22]
C:\Program Files (x86)\AviSynth\plugins64+\DePanEstimate.dll [2.10.0.4, 2020-05-22]
C:\Program Files (x86)\AviSynth\plugins64+\DeScratch64.dll [1.1.0.0, 2018-09-07]
C:\Program Files (x86)\AviSynth\plugins64+\DirectShowSource.dll [n/a, 2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\ffms2.dll [n/a, 2020-08-22]
C:\Program Files (x86)\AviSynth\plugins64+\FrameSel_x64.dll [2.21.0.0, 2021-07-27]
C:\Program Files (x86)\AviSynth\plugins64+\ImageSeq.dll [n/a, 2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\KNLMeansCL.dll [n/a, 2020-11-12]
C:\Program Files (x86)\AviSynth\plugins64+\LSMASHSource.dll [n/a, 2021-07-28]
C:\Program Files (x86)\AviSynth\plugins64+\MedianBlur2.dll [1.1.0.0, 2021-01-30]
C:\Program Files (x86)\AviSynth\plugins64+\mvtools2.dll [2.7.45.0, 2021-06-08]
C:\Program Files (x86)\AviSynth\plugins64+\neo-f3kdb.dll [1.0.0.0, 2021-07-05]
C:\Program Files (x86)\AviSynth\plugins64+\nnedi3.dll [0.9.4.59, 2021-07-19]
C:\Program Files (x86)\AviSynth\plugins64+\RemoveDirt.dll [0.9.3.0, 2021-02-23]
C:\Program Files (x86)\AviSynth\plugins64+\RoboCrop_x64.dll [1.13.0.0, 2020-02-07]
C:\Program Files (x86)\AviSynth\plugins64+\RT_Stats_x64.dll [2.0.13.0, 2020-12-29]
C:\Program Files (x86)\AviSynth\plugins64+\Shibatch.dll [n/a, 2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\TDeint.dll [1.8.0.0, 2020-12-14]
C:\Program Files (x86)\AviSynth\plugins64+\TimeStretch.dll [n/a, 2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\TIVTC.dll [1.0.26.0, 2021-02-22]
C:\Program Files (x86)\AviSynth\plugins64+\VDubFilter.dll [n/a, 2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\yadifmod2.dll [0.2.7.0, 2021-02-07]

[Scripts (AVSI)] [Time stamp]
C:\Program Files (x86)\AviSynth\plugins64+\colors_rgb.avsi [2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\FFMS2.avsi [2015-05-22]

[Uncategorized files] [Time stamp]
C:\Program Files (x86)\AviSynth\plugins64+\colors_rgb.txt [2021-01-11]
C:\Program Files (x86)\AviSynth\plugins64+\ffmsindex.exe [2020-08-22]
C:\Program Files (x86)\AviSynth\plugins64+\McDegrain.avs [2021-08-05]
C:\Program Files (x86)\AviSynth\plugins64+\McDegrainSharp.avs [2021-08-05]

[poisondeathray]

I am getting no function McDeGrainSharp. I got it and created a McDeGrainSharp.avs file and put it in avisynth 32/64 and vdub 32 and not working.


Rename it "McDeGrainSharp.avsi" (avsi) to autoload



Or import .avs script functions manually using

Import("PATH\McDeGrainSharp.avs")

[kedautinh12]

Change McDegrain.avs and McDegrainSharp.avs to .avsi after test again

[coolgit]

Ok thanks. Done it and now working. I was told avisynthplus doesn't care whether it is avs or avsi?

[coolgit]

OH well... that pal hybrid didn't work on deinterlacing properly.

I guess I have run out of tools.

[StainlessS]

This bit is config for deinterlace, I have the one in RED deliberately set high. Maybe try with lower. [blue is default]


# IsCombedTIVTC args, Defaults, 1st after comment hash.
CTHRESH = 12 # 10), Area combing threshold.
# Valid settings are from -1 (every pixel will be detected as combed) to 255 (no pixel will be detected as combed).
# This is basically a pixel difference value. A good range is between 8 to 12.
MI = 80 # 80), Number of combed pixels inside any of the blocky by blockx size blocks on the frame for the frame to be detected as combed.
CHROMA = True # True), Whether or not chroma is considered in the combed frame decision. [We want chroma as interlaced parts in color]
BLOCKX = 16 # 16), X-axis size of the window used during combed frame detection.
BLOCKY = 16 # 16), Y-axis size of the window used during combed frame detection.
# So in any single block, if 85[MI] out of 256[BLOCKX x BLOCKY] pixels pass CThresh as combed, then is Interlaced.
METRIC = 1 # 0) Detecto method: 0=Old, 1=New

# Deinterlace mode
BOBMOD=1 # 0=BOB, 1=YadifMod2, 2=QTGMC : QTGMC here is way memory/CPU intensive, req x64. Suggest YadifMod2

BOB_FUNC_0 = """Bob()""" # Cheap & cheerful Doublerate simple Bob.
BOB_FUNC_1 = """YadifMod2(Order=-1,Mode=1,edeint=NNedi3(Field=-2, nns=2))""" # Doublerate Yadifmod2 filter.
# # Field-2=double rate, internal parity value to start : Order-1=Internal Parity. Mode 1=double rate do spactial check (3=no spatial check).

BOB_FUNC_2 = """QTGMC(Preset="fast",NoisePreset="Fast",SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1, tr2=0)""" # Doublerate QTGMC filter.

FUNC = Select(BOBMOD, BOB_FUNC_0, BOB_FUNC_1, BOB_FUNC_2)
FNAME = Select(BOBMOD,"BOB", "YADIF", "QTGMC")


Make FINAL=False to view which are detected as interlaced.

[coolgit]

Have you ever deinterlace the first 14 seconds scene of midnight in the firing line?

This guy did something but haven't a clue what. https://www.modeemi.fi/~leopold/Babylon5/DVD/DVDTransfer.html#8

I was using TFM i think with debug true to see the info. Progressive and interlace looked mixed up.

[coolgit]

When the scene goes upwards there is jumping back and forth.

frame by frame u-up; ju- jump up; d-down

uuuujudud uuuujudud uuuujudud pattern. Sometimes a bit different.

[poisondeathray]

Have you ever deinterlace the first 14 seconds scene of midnight in the firing line?

This guy did something but haven't a clue what. https://www.modeemi.fi/~leopold/Babylon5/DVD/DVDTransfer.html#8

I was using TFM i think with debug true to see the info. Progressive and interlace looked mixed up.

When the scene goes upwards there is jumping back and forth.

frame by frame u-up; ju- jump up; d-down

uuuujudud uuuujudud uuuujudud pattern. Sometimes a bit different.



Post your script

Post source video sample

Consider posting in separate thread, as it does not necessarily pertain to the thread topic "SpotLess"

[coolgit]

Here the 14 seconds file.

https://mega.nz/file/by4GABxY#0mb4B8kPfmqmbJwFsU35DY1BLhAOo1T5ke1VOIveVn8

[StainlessS]

@Moderator,
Can you please move all posts from post #74, into new thread with user Coolgit [EDIT: post #74 owner] the main man in charge of thread.

post #74:- https://forum.doom9.org/showthread.php?p=1948448#post1948448

thanks very much :)

EDIT: Maybe thread title "Babylon5", CoolGit can change thread name once created.

[@CoolGit, edit 1st post, "Go Advanced", and change title]

[manono]

@Moderator,
Can you please move all posts from post #74, into new thread with user Coolgit the main man in charge of thread.
I, for one, have no idea how to do that. Maybe someone else does. Sorry.

coolgit, open a new thread with your latest question(s) as at this point you are hijacking this StainlessS thread about his very useful filter.

[coolgit]

hijacking this StainlessS thread about his very useful filter.

Hijacking:confused::confused: Nope, my posts is about spotless and pal/hybrid script. A couple about B5 but only because Stainlesss mentioned he was doing it too.

However, it is no skin off my nose if someone were to do what Stainlesss requested.

[StainlessS]

I, for one, have no idea how to do that. Maybe someone else does. Sorry.
I believe that tebasuna has done that in the past, [and of course DG many times when was moderator (EDIT: and of course Wilbert, but he aint been around lately)].
Dont matter I suppose, thanks anyway.

[LouieChuckyMerry]

Hi Bender, er, StainlessS, I hope you're well :) . Searching for help to improve my Blu-ray video of The Bad News Bears{1976}--it looks no better than my HDTV rip, which I find rather pathetic; seriously, if a relative noob such as myself can improve the video quality within a few days via a forum... anyway, you get the idea--I turned to videohelp.com (How To Better "The Bad News Bears{1976}" Blu-ray Video (https://forum.videohelp.com/threads/402938-How-To-Better-The-Bad-News-Bears%7B1976%7D-Blu-ray-Video)), whence jagabo The Magnificent turned me on to SpotLess. Wow, it really worked wonders with default settings. :thanks: . Now I'm here seeking information, information, from the new number two, er, StainlessS The Great. My denoiser of choice is SMDegrain, and that's all I've ever really used to improve my progressive sources: some denoising, some sharpening, and I'm generally quite pleased. SpotLess, however, has added a (very positive) step to my older sources, so I'm trying to figure out how much, if any, denoising it does on it's own. I read the .avsi file but a majority of the words entered my mind then left without sticking, as I understand quite well the concept of AviSynth but not all of the technical terms. Thanks for any clarity.

Edit: I also wondered if you ever gained any positive-negative feedback on using SpotLess as a prefilter in SMDegrain (beyond the quote near the beginning of this thread).

[StainlessS]

Sorry, I did my best to describe it already.
Suggest you see original Didee post [I think I linked it somewhere].
Other than that, see MvTools docs.

Again sorry, I just too pushed for time, and already did my best.
Perhaps someone else can do some kind of tutorial.

[LouieChuckyMerry]

Apologies, I think I wasn't clear. I don't require a tutorial, I'm just curious how much general denoising you think SpotLess does on it's own, independent of any other filters, and whether you'd received any other feedback, beyond the quote from ChaosKing, from people using it as a prefilter in SMDegrain.

No worries, though. I'll run some trim tests and see what I find. Thanks again for a really nice tool, be well :) .

[FranceBB]

Hi Bender, er, StainlessS

Whenever someone says "Bender" I don't think at the Futurama's robot but rather at Jimmy Carr saying "Mommy's little bender" xD

https://youtuberead.com/photos/hq/jimmy-carr-comedian-2007-full-82N5z_7M7Cw-1.jpg



my Blu-ray video of The Bad News Bears{1976}--it looks no better than my HDTV rip, which I find rather pathetic; seriously, if a relative noob such as myself can improve the video quality within a few days via a forum...

It's probably the same master which has been sent out for the BD... The problem with those kind of projects is that most of the time the people working on it know how to do and are also very skilled, but can't do anything because of rights management.
For instance, although I was able to Remaster several old movies, there are some exceptions (and they're not so uncommon unfortunately). For instance, when we've got the 4K Remaster of a certain batch of very old movies from a certain company (I won't reveal the name for the sake of avoiding a PR Incident) they were like: "You better not denoise those!" so I couldn't do anything to the masterfile other than color space conversion and dithering and they aired with tons of grain which made the 25 Mbit/s live encoder struggle which in turn resulted in the whole picture being blurred by the H.265 live encoder.

Anyway, when I was actually able to work on movies, I did use SpotLess() and I gotta agree with you, it saved my butt several times by getting rid of all the nasty dots and also of some stripes sometimes. Unfortunately I wasn't able to use it for sports 'cause it's very hard to tweak for ice hockey games 'cause there literally a black disk on a white surface which very much looks like a spot (black dot), so SpotLess() makes it disappear, leading to some hilarious results xD

But aside from that, it's great.

[LouieChuckyMerry]

Ciao, FranceBB, and thanks for the inside information. It's a shame that the people signing-off on the Blu-rays don't seem like they've even viewed the videos. Seriously, a couple posts at videohelp.com and I learned to add

ColorYUV(Levels="TV->PC")
ColorYUV(Gain_Y=8,Off_Y=-6,Gamma_Y=100)
ColorYUV(Levels="PC->TV")

to my script. The improvement is night and day (truly, the original video looks as if it were filmed through a sunglasses lens). Anyway, at least there's improvement to be made on a personal level ;) .

Any advice for using SpotLess in conjunction with SMDegrain? Would the end result be different using it before SMDegrain versus as a prefilter in SMDegrain?

[zorr]

Unfortunately I wasn't able to use it for sports 'cause it's very hard to tweak for ice hockey games 'cause there literally a black disk on a white surface which very much looks like a spot (black dot), so SpotLess() makes it disappear, leading to some hilarious results xD

Delta Restore could fix that kind of problems. If Spotless is the bread, Delta Restore is the knife and now we can have sliced bread. :)

https://forum.doom9.org/showthread.php?t=182831

[StainlessS]

If Spotless is the bread, Delta Restore is the knife, then zorr is the butter.
:)

EDIT: And I'll be the Marmite, real tasty treat, but an acquired taste.

[FranceBB]

Thanks for the insight, I'm gonna try Delta Restore. :)

If Spotless is the bread, Delta Restore is the knife, zorr is the butter and Stainless is Marmite


https://cdn-a.william-reed.com/var/wrbm_gb_food_pharma/storage/images/publications/food-beverage-nutrition/beveragedaily.com/article/2020/06/12/brewing-slowdown-hits-marmite-production/11477063-1-eng-GB/Brewing-slowdown-hits-Marmite-production.jpg

[StainlessS]

Zorr,
I note that in a number of posts in spotless related thread
eg here:- https://forum.doom9.org/showthread.php?p=1944487#post1944487


function RemoveSmallSpots(clip mask, int spotSize) {
mask_orig = mask

# remove too small spots ----------------------------------------------------
for (i = 1, spotSize) {
mask = mask.mt_inpand()
}
mask = mt_hysteresis(mask, mask_orig, chroma="-127")
return mask
}

think maybe above chroma="-127" should be chroma="-128",
Appears quite a few times in the linked thread.

From Masktools 2 original docs.

When defined, the value contained in this string will overwrite the u & v processing modes. This is a nice addition proposed by mg262 that makes the filter more user friendly. Allowed values for chroma are :

"process" : set u = v = 3.
"copy" or "copy first" : set u = v = 2.
"copy second" : set u = v = 4.
"copy third" : set u = v = 5.
"xxx", where xxx is a number : set u = v = -xxx.


EDIT: I recently found a script posted some time ago that now throws an error in recent Pinterf MaskTools, I had wrongly used
chroma="128" [instead of chroma="-128"]
and it now throws a wobbly, says should be setting chroma to -ve ie -128 [although original script did seem to work OK in older masktools].

EDIT: To below: Thankyou kind sir :) [you've gotta develop thick skin on the D9]

[zorr]

Zorr,
I note that in a number of posts in spotless related thread
...
think maybe above chroma="-127" should be chroma="-128",


Yes you're right. I'll fix it when I release the "official" Delta Restore version. It doesn't have any effect on the final result so it's nothing to worry about. :)

[coolgit]

Many times I had spots around the edges of the frames and always put back in by delta restore.

Check the coding and zpadding seems to be responsible for it. Don't know why it was there but I had changed it to zeros.

Spots now removed. Yippeeeee.

[zorr]

Many times I had spots around the edges of the frames and always put back in by delta restore.

Check the coding and zpadding seems to be responsible for it. Don't know why it was there but I had changed it to zeros.

Spots now removed. Yippeeeee.

Zpadding was used in the original case to restrict the edge detector from detecting the frame borders (the source was scanned film). For most material Zpadding is not needed.

[coolgit]

Hey StainlessS, what does olap do?

Had a big defect and was experimenting. Nothing worked 100% even when BlkSz,32. For a laugh I changed olap to BlkSz/3 and boooom, it work 100%.

[johnmeyer]

Hey StainlessS, what does olap do?It is the overlap parameter used by the MAnalyze function of MVTools2, which is the motion estimation plugin that is at the heart of Spotless. In MVTools2 you can specify the size of the block used to "track" objects in the video. Typical sizes are 4, 8, and 16. To improve the quality of certain operations the algorithm can overlap adjacent blocks. This sometimes improves noise reduction or synthesis of intermediate frames, but you don't know until you try, as you did in fact do.

[StainlessS]

OLap is the block [Blksz] Overlap.
Popular choices are sometimes BlkSz/2 or 2.

[MysteryX]

lol I'm trying Spotless on a clip while driving, and it's doing a slightly better job than VS mClean at removing noise (at higher speed!). And -- trees while driving are disappearing. Can I call it Tree-less?

[StainlessS]

lol I'm trying Spotless on a clip while driving, and it's doing a slightly better job than VS mClean at removing noise (at higher speed!). And -- trees while driving are disappearing. Can I call it Tree-less?
You can indeed call it TreeLessS,
However I have to point out that it is very dangerous to try out Spotless whilst you are driving.

EDIT: Especially at higher speed!

EDIT: Zorr added some recovery stuff for disappearing trees, arms, legs, balls, etc, here:- https://forum.doom9.org/showthread.php?t=182831

[MysteryX]

That ain't faster anymore. Waiting half-an-hour for the first frame of a 5K clip to come out. I give up.

Another way to restore details: run this as a prefilter to BM3D.

Edit: LOL it was resizing up by a factor of 3x

[MysteryX]

I'm comparing this to xClean. I'm very surprised by the results. Trees are back but....... look different. They actually look more like trees than in the source??? How is that possible? Image details are also a lot clearer. This is looking damn good. It's just that -- the way it completely alters moving trees is a bit risqué

Runs at 0.4fps on 5K

Compared to xClean, noise is about the same, image is clearer/sharper, and moving objects are... transformed, for better or worse. Haven't tested whether it works well for all types of content. On this test clip, Spotless gives better visible result (perhaps a bit too sharp), while xClean better respects the source.

[coolgit]

It is the overlap parameter used by the MAnalyze function of MVTools2, which is the motion estimation plugin that is at the heart of Spotless. In MVTools2 you can specify the size of the block used to "track" objects in the video. Typical sizes are 4, 8, and 16. To improve the quality of certain operations the algorithm can overlap adjacent blocks. This sometimes improves noise reduction or synthesis of intermediate frames, but you don't know until you try, as you did in fact do.

Is there an image example somewhere explaining how it works?

Are you saying that if a block is currently analyse the overlap can do the next block to exist the current block.

How does BlkSz/3 do a better job than BlkSz/2 - assuming / means divide.

[MysteryX]

Is there an image example somewhere explaining how it works?

Are you saying that if a block is currently analyse the overlap can do the next block to exist the current block.

How does BlkSz/3 do a better job than BlkSz/2 - assuming / means divide.
For blksize=16, if you set an overlap of 8, half-way through each block begins the next block.

[johnmeyer]

Is there an image example somewhere explaining how it works?

Are you saying that if a block is currently analyse the overlap can do the next block to exist the current block.

How does BlkSz/3 do a better job than BlkSz/2 - assuming / means divide.http://avisynth.org.ru/mvtools/mvtools2.html

https://forum.doom9.org/showthread.php?p=564686#post564686

Search them both for "overlap."

[StainlessS]

Assuming just 8 bit depth,
I nearly always avoid LSMash and ffms2, too many probs.
Maybe this of use [in a demo format]

DEMO.BAT

REM https://forum.doom9.org/showthread.php?p=1908619#post1908619
REM We DO NOT LIKE SPACES IN FILE NAMES (REM == REMark ie comment)
REM We DO NOT LIKE ACCENTS IN FILE NAMES.


setlocal

REM Where to Find ffmpeg
set FFMPEG="C:\BIN\ffmpeg.exe"

REM Where to get INPUT files, No terminating Backslash, "." = current directory (ie same as dir .bat file)
set INDIR="."

REM Where to place OUTPUT files, No terminating Backslash. "." would be same as .bat file, ".\OUT" = OUT folder in same directory as bat file.
set OUTDIR="D:"

REM Below, can add INPUT extensions as eg *.WEBM (SPACE separated, Batch Processes all INPUT type files in INDIR)
FOR %%A IN (*.AVI *.MKV *.MP4 *.MOV *.QT *.3GP *.DVB *.VOB *.MPG *.MPEG *M2P *.PS *.TS *.DIVX *.XVID *.FLV *.WMV *.ASF *.MXF) DO (

REM ****** Un-REM [ie SELECT] ONLY one of below lines, Comment out ie REM the remaining lines.
%FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec huffyuv -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec magicyuv -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec v410 -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec r210 -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec copy -acodec copy -sn "%OUTDIR%\%%~nxA.MKV"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -acodec copy "%OUTDIR%\%%~nxA.MKV"
REM %FFMPEG% -i "%INDIR%\%%A" -vn -acodec pcm_s16le "%OUTDIR%\%%~nxA.WAV"
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec rawvideo -f rawvideo -an "%OUTDIR%\%%~nxA.rawvideo"


REM *****************************************************************************************.

)
Pause

REM ... Above Command lines, What they do if UnREM'ed (and all others REM'ed, UnREM=UnCOMMENT ) :
REM (1) Convert Video to UtVideo lossless, Convert Audio to 16 bit PCM audio (output AVI).
REM (2) Convert Video to HuffYUV lossless, Convert Audio to 16 bit PCM audio (output AVI).
REM (3) Convert Video to MagicYUV lossless, Convert Audio to 16 bit PCM audio (output AVI).
REM (4) Convert Video to v410 Uncompressed 4:4:4 10-bit lossless, Convert Audio to 16 bit PCM audio (output AVI).
REM (5) Convert Video to r210 Uncompressed RGB 10-bit lossless, Convert Audio to 16 bit PCM audio (output AVI).
REM (6) Remux NO SUBTITLES, copy both video and audio but NOT subtitles(output MKV - see file extension at the end of the Remux line, ie MKV).
REM (7) Convert Video to UtVideo lossless, copy audio (output MKV).
REM (8) Skip any video, Convert Audio to 16 bit PCM (output WAV).
REM (9) Convert Video to rawvideo uncompressed, Convert Audio to 16 bit PCM audio (output y4m).
REM *****************************************************************************************.
REM In the UnREM'ed command Line [ie without a preceding REM]:-
REM '-vcodec utvideo' means convert video using utvideo codec.
REM '-vcodec copy' means copy video rather than convert.
REM '-vn' means no video output, use instead of eg '-vcodec utvideo'
REM '-acodec pcm_s16le' means convert audio using pcm_s16le codec.
REM '-acodec copy' means copy audio rather than convert.
REM '-an' means no audio output, use instead of eg '-acodec pcm_s16le'
REM The file extension at end of the line determines output container, eg '.AVI'
REM
REM From Command line with ffmpeg somewhere in your environment PATH:-
REM 'ffmpeg -codecs >D:\ffmpeg_codecs.txt'
REM Writes a txt file of ffmpeg available codecs to D:\
REM At start of txt file it shows a 'legend' or 'key' for codecs that can be used for video and audio,
REM use only ENCODING SUPPORTED codecs, prefereably LOSSLESS.
REM Supported codecs will vary with ffmpeg version.
REM *****************************************************************************************.


I have a dedicated conversion directory where the script lives.
Copy video files to that directory.
Click the bat, batch converts all in directory.
Job done. Use reliable AviSource for the lot.

[MysteryX]

I had accidently posted the ffsm2 issue in the wrong thread; but since you answered... not sure converting 5K video to UT Video is a great idea. It would need enormous space!

Edit: for a 94MB video of 12 seconds, it produces a 2.2GB video and lists these errors:
- Using non-standard frame rate 29/1
- deprecated pixel format used, make sure you did set range correctly
- Duplicate POC in a sequence
- Error parsing NAL unit #0
- Error while decoding stream #0:0: Invalid data found when processing input

Video is only missing the last frame and colours are wrong. lol.

[StainlessS]

Maybe, but they would work without problems.

I think GoPro would be mp4, just use LSmashVideoSource, that one is usually fine.

[MysteryX]

Maybe, but they would work without problems.

I think GoPro would be mp4, just use LSmashVideoSource, that one is usually fine.
I think I'll just denoise the videos and let my video editor fix remaining glitches. Next time, I'll encode in h264 instead of h265, with High-Bit-Depth enabled, and re-encode every clip while transferring to my computer at a desirable bitrate. I hope they have less bug in their H264 encoder.

Also taking GoPro pictures -- need to set sharpening to LOW, and then add sharpening and contrast with Photoshop and it then looks as good or better than my cellphone. GoPro is peculiar to use...

[MysteryX]

I tried Spotless+DR on other clips. As I suspected, it works great on some content and not so great on other types (by the nature of MVTools). Main issue is that MVTools blurs out what I call "fine textures", which is why xClean passes MVTools through BM3D which restores the textures; and then through KNLMeans which always improves upon the prefilter.

Other than lost textures, Spotless has nearly identical noise reduction as xClean. Sharpening is too high!

[kedautinh12]

You can try Restore_Objects to restore trees. It's not as complex as the zorr version
https://forum.doom9.org/showthread.php?p=1945014#post1945014

[StainlessS]

Sharpening is too high!
ssS dont have no sharpening.

[MysteryX]

I'm converting the basic SpotLess function to VapourSynth to compare... it's using undocumented parameters!

MVTools doc (http://avisynth.org.ru/mvtools/mvtools2.html)

MAnalyse has no Multi parameter, and MCompensate has no tr parameter.

VapourSynth has such parameters even less.

Edit: Those parameters are in pinterf version. This script is not VapourSynth compatible then?

[coolgit]

ssS dont have no sharpening.
I think mysteryx was using spotless with delta restore and it has sharpening in it.

By default i do not use it as qtgmc does sharpening during deinterlacing.

[StainlessS]

MX,
Pinterf version [not sure exactly when it happened] Mvtools2 has MDegrainN, and additional args to MAnalyse,
to avoid many nearly identical lines of script with differing Delta arg.
You can still do it the old way eg, getting fwrd/bkwrd vectors for each required/supported delta [ie RadT, temporal radius],
and choose the appropriate forward backward vectors and use eg MDegrain1, or MDegrain2, or MDegrain3 or for whatever
radius [I think pre MDegrainN supports up to MDegrain6].

https://forum.doom9.org/showthread.php?p=1923580#post1923580


So, with Pinterf version, could use eg RadT=10, which would not be supported in old MvTools2, where is capped at delta 6 max.
Also, a number of bug fixed [eg cant mrecalculate where overlap=0], also additional block sizes supported in Pinterf ver$.
Also some weird bug fixed in the masks produced in old mvtools2 [long time bugged].

EDIT: Does away with repetative script lines as below in blue
[just any old script, I found here:- https://forum.doom9.org/showthread.php?p=1710446#post1710446 ]


#Denoising function, progressive video
function MDegrainProg(clip source, int "blksize", int "overlap", int "dct")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
preNR = source.fft3dfilter(sigma=4,interlaced=false)

preNR_super = preNR.MSuper(pel=2, sharp=1)
src_super = source.MSuper(pel=2, sharp=1,levels=1)

backward_vec2 = MAnalyse(preNR_super,isb = true, delta = 2, blksize=blksize, overlap=overlap, dct=dct)
backward_vec1 = MAnalyse(preNR_super,isb = true, delta = 1, blksize=blksize, overlap=overlap, dct=dct)
forward_vec1 = MAnalyse(preNR_super,isb = false, delta = 1, blksize=blksize, overlap=overlap, dct=dct)
forward_vec2 = MAnalyse(preNR_super,isb = false, delta = 2, blksize=blksize, overlap=overlap, dct=dct)

MDegrain2(source,src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=800)
}

And above, only supports a single radius of 2.

Additional Documentation, seems only available in Pinterf ver$ zip.

[johnmeyer]

I'm converting the basic SpotLess function to VapourSynth to compare... it's using undocumented parameters!

MVTools doc (http://avisynth.org.ru/mvtools/mvtools2.html)

MAnalyse has no Multi parameter, and MCompensate has no tr parameter.As SpotlessS says, you need to use the Pinterf version. I switched to that in order to try out SpotlessS, and found that all my older scripts which use MVTools2 still seem to work just fine. I also find it more stable when used with AVISynth+ and multi-threading.

[StainlessS]

MX,
Old style MDegrain() script function [supporting temporal radius up to 3, in Frames]

Function MCDegrain(clip c, int "frames",int "plane",int "limit",int "limitC")
{ # By Didee, http://forum.doom9.org/showthread.php?t=161594
frames = default(frames, 2)
plane=Default(plane,0) # as FFT3DFilter
limit=Default(limit,255) # maximal change in pixl luma (default 255 [no limit])
limitC=Default(limitC,limit) # maximal change in pixl chroma (default as luma limit)
bs = (c.width>960) ? 16 : 8
super = c.MSuper(pel=2, sharp=1)
backward_vec3 = MAnalyse(super, isb = true, delta = 3, blksize=bs, overlap=bs/2)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=bs, overlap=bs/2)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=bs, overlap=bs/2)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=bs, overlap=bs/2)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=bs, overlap=bs/2)
forward_vec3 = MAnalyse(super, isb = false, delta = 3, blksize=bs, overlap=bs/2)
(frames<=0) ? c :\
(frames==1) ? c.MDegrain1(super, backward_vec1,forward_vec1,thSAD=400,plane=plane,limit=limit,limitC=limitC) :\
(frames==2) ? c.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,plane=plane,limit=limit,limitC=limitC) :\
c.MDegrain3(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400,plane=plane,limit=limit,limitC=limitC)
return(last)
}


New style [although this one is mod with extra args (no cap of temporal Radius <=3, on 'frames']

Function McDegrain(clip c, int "frames",
\ Float "Limit",Float "LimitC",
\ float "bblur", int "Pel",bool "Chroma", Int "BlkSz",Int "OLap",Int "Plane",bool "Tm",Bool "glob", Int "ThSAD",Int "ThSAD2") {
myName = "McDegrain: "
frames = Default(frames, 2)
Limit = Default(Limit,255)
LimitC = Default(LimitC,Limit)
bblur = Default(bblur,0.0) # Denoising for Super Clip.
bblur = (bblur>=0.0) ? bblur : (c.width>1920) ? 0.75 : (c.width>1280) ? 0.7 : (c.width>960 ) ? 0.65 : 0.6
pel = Default(pel,2)
chroma = Default(chroma,true) # Use chroma in MAnalyse for vectors
BlkSz = Default(BlkSz,(c.width>1920) ? 32 : (c.width>1280) ? 24 : (c.width>960 ) ? 16 : 8)
OLap = Default(OLap,BlkSz/2)
Plane = Default(Plane,4)
tm = Default(tm,true) # TrueMotion
Glob = Default(glob,Tm) # Default Tm, Allow set MAnalyse(global) independently of TrueMotion.
ThSAD = Default(ThSAD,400)
ThSAD2 = Default(ThSAD2,ThSAD)
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
Assert(0 <= Plane <= 4, myName + "0 <= Plane <= 4")
super = (bblur==0.0 ? c : c.blur(bblur)).MSuper(pel=pel, sharp=1)
super_rend = (bblur==0.0) ? super : c.MSuper(pel=pel, sharp=1)
MultiVec = super.MAnalyse(multi=true,delta=frames,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
(frames<=0)
\ ? c
\ : c.MDegrainN(super_rend, MultiVec, Frames, thSAD=ThSAD, plane=Plane, Limit=Limit, LimitC=LimitC, thsad2=ThSAD2)
return Last
}

[MysteryX]

Except that here it's MCompensate, not MDegrain. It takes a single vector clip.

[zorr]

I think mysteryx was using spotless with delta restore and it has sharpening in it.

Delta Restore doesn't have sharpening either. The script by chmars which used Spotless and Delta Restore did have sharpening though.

[StainlessS]

MX,

Something like this

/*
SpotLess v1.07Beta. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
v1.07Beta, bBlur Default changed from 0.0 OFF, to 0.6, slower but better for mc analysis. Maybe try 1.0 for HD to overcome HD 'grittiness'.
Added dc arg, detect clip.
*/

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)



sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


# Support 1 <= RadT <= 3
Function ZpotLezz(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "ZpotLezz: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT <= 3,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)

# equivalent


sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup

bv1 = sup.MAnalyse(isb=true, delta=1,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
bv2 = sup.MAnalyse(isb=true, delta=2,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
bv3 = sup.MAnalyse(isb=true, delta=3,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
fv1 = sup.MAnalyse(isb=false,delta=1,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
fv2 = sup.MAnalyse(isb=false,delta=2,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
fv3 = sup.MAnalyse(isb=false,delta=3,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)

bc1 = c.MCompensate(sup_rend, bv1, thSad=ThSAD, thSad2=ThSAD2)
bc2 = c.MCompensate(sup_rend, bv2, thSad=ThSAD, thSad2=ThSAD2)
bc3 = c.MCompensate(sup_rend, bv3, thSad=ThSAD, thSad2=ThSAD2)
fc1 = c.MCompensate(sup_rend, fv1, thSad=ThSAD, thSad2=ThSAD2)
fc2 = c.MCompensate(sup_rend, fv2, thSad=ThSAD, thSad2=ThSAD2)
fc3 = c.MCompensate(sup_rend, fv3, thSad=ThSAD, thSad2=ThSAD2)
/*
i1 = Interleave(fc1, c , bc1)
i2 = Interleave(fc2, fc1, c , bc1, bc2)
i3 = Interleave(fc3, fc2, fc1, c , bc1, bc2, bc3)
*/
i1 = Interleave(bc1, c , fc1)
i2 = Interleave(bc2, bc1, c , fc1, fc2)
i3 = Interleave(bc3, bc2, bc1, c , fc1, fc2, fc3)



(RadT==1) ? i1.MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT)
\ : (RadT==2) ? i2.MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT)
\ : i3.MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT)

SelectEvery(RadT*2+1,RadT) # Return middle frame
}

AviSource(".\Replace Bad Fields.avi").ConvertToYV12

O=Last
RADT=3
A=SpotLess(RadT=RADT)
B=ZpotLezz(RadT=RADT)
TOP=StackHorizontal(A,B)
D=ClipDelta(A,B)
D2=ClipDelta(A,B,true)
BOT=StackHorizontal(D,D2)
StackVertical(TOP,BOT)
return last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}


Above mod not exactly the same, but looks similar, dont know why is different, maybe I screwed up somewhere.
I'm struggling to stay awake and so am about to go to bed.
EDIT: Even produces the odd frame not identical when RadT=1, so I maybe my error.

[coolgit]

Delta Restore doesn't have sharpening either. The script by chmars which used Spotless and Delta Restore did have sharpening though.

Yes... I just thought it was part of delta restore but personally I didn't want it.

A question. I had a big spot, a penny size, 1st frame after scene change. Spotless does nothing but delta restore somehow fixed it 60% ish with lighter colour. I.e say the background is red and the next frame is the same, the white spot get replaced with pinkish colour. I suspect an opacity issue here but still can't get my head around your codes.

Any pointers?

[kedautinh12]

I'm converting the basic SpotLess function to VapourSynth to compare... it's using undocumented parameters!

MVTools doc (http://avisynth.org.ru/mvtools/mvtools2.html)

MAnalyse has no Multi parameter, and MCompensate has no tr parameter.

VapourSynth has such parameters even less.

Edit: Those parameters are in pinterf version. This script is not VapourSynth compatible then?

You can try spotless_mod to keep detail lost in spotless
https://forum.doom9.org/showthread.php?p=1945133#post1945133

[MysteryX]

Here's the translation in VapourSynth. But the image turns full of green horizontal lines. Can you see what's wrong?


def SpotLess(c: vs.VideoNode, radt: int = 1, thsad: int = 10000, thsad2: Optional[int] = None, pel: int = 2, chroma: bool = True,
blksize: int = 8, overlap: Optional[int] = None, truemotion: bool = True, pglobal: int = True,
blur: float = 0.0, ref: Optional[vs.VideoNode] = None) -> vs.VideoNode:
if radt < 1 or radt > 3:
raise ValueError("Spotless: radt must be between 1 and 3")
if not pel in [1, 2, 4]:
raise ValueError("Spotless: pel must be 1, 2 or 4")

icalc = c.format.bits_per_sample < 32
S = core.mv.Super if icalc else core.mvsf.Super
A = core.mv.Analyse if icalc else core.mvsf.Analyse
C = core.mv.Compensate if icalc else core.mvsf.Compensate
pad = max(blksize, 8)

sup = ref or (c.std.Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1]) if blur > 0.0 else c)
sup = S(sup, hpad=pad, vpad=pad, pel=pel, sharp=2)
sup_rend = S(c, hpad=pad, vpad=pad, pel=pel, sharp=2, levels=1) if ref or blur > 0.0 else sup

bv1 = A(sup, isb=True, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
bv2 = A(sup, isb=True, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
bv3 = A(sup, isb=True, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv1 = A(sup, isb=False, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv2 = A(sup, isb=False, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv3 = A(sup, isb=False, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)

bc1 = C(c, sup_rend, bv1, thsad=thsad, thsad2=thsad2)
bc2 = C(c, sup_rend, bv2, thsad=thsad, thsad2=thsad2)
bc3 = C(c, sup_rend, bv3, thsad=thsad, thsad2=thsad2)
fc1 = C(c, sup_rend, fv1, thsad=thsad, thsad2=thsad2)
fc2 = C(c, sup_rend, fv2, thsad=thsad, thsad2=thsad2)
fc3 = C(c, sup_rend, fv3, thsad, thsad2=thsad2)

i1 = core.std.Interleave([bc1, c, fc1])
i2 = core.std.Interleave([bc2, bc1, c, fc1, fc2])
i3 = core.std.Interleave([bc3, bc2, bc1, c, fc1, fc2, fc3])

output = core.tmedian.TemporalMedian(i3 if radt==3 else i2 if radt==2 else i1, radius=radt)
return output.std.SelectEvery(radt*2+1, radt) # Return middle frame


Edit: Compensate in VapourSynth doesn't have thsad2; and if I remove it, it's the output of Compensate that is corrupted with green lines.

[jmac698]

You're still going! Good on you.

I've become an image (color/distortion/vignetting/black frames) calibration expert since, and better programmer. I do data science. Still love image processing, and actually had a chance to use Avisynth at work! So fun.

[StainlessS]

MX,

Edit: Compensate in VapourSynth doesn't have thsad2; and if I remove it, it's the output of Compensate that is corrupted with green lines.
Not sure what to make of that, if it dont have thsad2, how come you can use it ??? [I dont speak ParselTongue (EDIT: Snake Speak, ie Python)]
I have not checked, but presume that ThSAD2 is new in Pinterf Multivec version MvTools,
and, maybe should calc individual ThSAD_Delta1, ThSAD_Delta2, ThSAD_Delta3, linear interp where ThSAD_Delta1 = thSAD and ThSAD_Delta3=ThSAD2 and ThSAD_Delta2=(ThSAD+ThSAD2)/2 [when RadT=3],
(and similar for RadT=2, and 1).
EDIT:

ThSAD_Delta1 = thSAD [when RadT=1]
ThSAD_Delta1 = thSAD and ThSAD_Delta2=ThSAD2 [when RadT=2]



fc2 = C(c, sup_rend, fv2, thsad=thsad, thsad2=thsad2)
fc3 = C(c, sup_rend, fv3, thsad, thsad2=thsad2)

Above, why is red different,from others.

Yo JMac, you still alive ? thought you'de be long dead by now.

EDIT:
Above mod not exactly the same, but looks similar, dont know why is different, maybe I screwed up somewhere.
Maybe I should not have set MCompensate ThSAD2, think maybe only for MultiVec usage.

[StainlessS]

MX, see above edit.

Mod

/*
SpotLess v1.07Beta. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
v1.07Beta, bBlur Default changed from 0.0 OFF, to 0.6, slower but better for mc analysis. Maybe try 1.0 for HD to overcome HD 'grittiness'.
Added dc arg, detect clip.
*/

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)



sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


# Support 1 <= RadT <= 3
Function ZpotLezz(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "ZpotLezz: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,10000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT <= 3,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup

th1 = thSAD
th2 = (RadT==3) ? (thSAD + thSAD2)/2 : thSAD2
th3 = thSAD2

bv1 = sup.MAnalyse(isb=true, delta=1,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
fv1 = sup.MAnalyse(isb=false,delta=1,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
bv2 = (RadT>=2) ? sup.MAnalyse(isb=true, delta=2,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob) : NOP
fv2 = (RadT>=2) ? sup.MAnalyse(isb=false,delta=2,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob) : NOP
bv3 = (RadT>=3) ? sup.MAnalyse(isb=true, delta=3,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob) : NOP
fv3 = (RadT>=3) ? sup.MAnalyse(isb=false,delta=3,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob) : NOP

bc1 = c.MCompensate(sup_rend, bv1, thSad=th1)
fc1 = c.MCompensate(sup_rend, fv1, thSad=th1)
bc2 = (RadT>=2) ? c.MCompensate(sup_rend, bv2, thSad=th2) : NOP
fc2 = (RadT>=2) ? c.MCompensate(sup_rend, fv2, thSad=th2) : NOP
bc3 = (RadT>=3) ? c.MCompensate(sup_rend, bv3, thSad=th3) : NOP
fc3 = (RadT>=3) ? c.MCompensate(sup_rend, fv3, thSad=th3) : NOP

ic = (RadT==1) ? Interleave(bc1, c , fc1) :
\ (RadT==2) ? Interleave(bc2, bc1, c , fc1, fc2) :
\ Interleave(bc3, bc2, bc1, c , fc1, fc2, fc3)

ic.MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT)

SelectEvery(RadT*2+1,RadT) # Return middle frame
}

AviSource(".\Replace Bad Fields.avi").ConvertToYV12

O=Last
RADT=1
A=SpotLess(RadT=RADT)
B=ZpotLezz(RadT=RADT)
TOP=StackHorizontal(A,B)
D=ClipDelta(A,B)
D2=ClipDelta(A,B,true)
BOT=StackHorizontal(D,D2)
StackVertical(TOP,BOT)
return last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}


EDIT: On a test sample, 4 frames out of 75 produce non identical result to SpotLess, maybe something like rounding in multivec
routine is a bit different [visually pretty much the same].

[MysteryX]

OK... this works in 32-bit. The dubhater plugin version has a bug outputing green horizontal lines.


def SpotLess(c: vs.VideoNode, radt: int = 1, thsad: int = 10000, thsad2: Optional[int] = None, pel: int = 2, chroma: bool = True,
blksize: int = 8, overlap: Optional[int] = None, truemotion: bool = True, pglobal: bool = True, blur: float = 0.0,
ref: Optional[vs.VideoNode] = None) -> vs.VideoNode:
if radt < 1 or radt > 3:
raise ValueError("Spotless: radt must be between 1 and 3")
if not pel in [1, 2, 4]:
raise ValueError("Spotless: pel must be 1, 2 or 4")

thsad2 = thsad2 or thsad
icalc = c.format.bits_per_sample < 32
S = core.mv.Super if icalc else core.mvsf.Super
A = core.mv.Analyse if icalc else core.mvsf.Analyse
C = core.mv.Compensate if icalc else core.mvsf.Compensate
pad = max(blksize, 8)

sup = ref or (c.std.Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1]) if blur > 0.0 else c)
sup = S(sup, hpad=pad, vpad=pad, pel=pel, sharp=2)
sup_rend = S(c, hpad=pad, vpad=pad, pel=pel, sharp=2, levels=1) if ref or blur > 0.0 else sup

th1 = thsad
th2 = (thsad + thsad2)/2 if radt==3 else thsad2
th3 = thsad2

bv1 = A(sup, isb=True, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv1 = A(sup, isb=False, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
if radt >= 2:
bv2 = A(sup, isb=True, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv2 = A(sup, isb=False, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
if radt >= 3:
bv3 = A(sup, isb=True, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv3 = A(sup, isb=False, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)

bc1 = C(c, sup_rend, bv1, thsad=th1)
fc1 = C(c, sup_rend, fv1, thsad=th1)
if radt >= 2:
bc2 = C(c, sup_rend, bv2, thsad=th2)
fc2 = C(c, sup_rend, fv2, thsad=th2)
if radt >= 3:
bc3 = C(c, sup_rend, bv3, thsad=th3)
fc3 = C(c, sup_rend, fv3, thsad=th3)

ic = core.std.Interleave([bc1, c, fc1]) if radt == 1 else \
core.std.Interleave([bc2, bc1, c, fc1, fc2]) if radt == 2 else \
core.std.Interleave([bc3, bc2, bc1, c, fc1, fc2, fc3])

output = core.tmedian.TemporalMedian(ic, radius=radt)
return output.std.SelectEvery(radt*2+1, radt) # Return middle frame

[MysteryX]

OK I compared xClean with mClean (MDegrain) vs SpotLess (Compensate) as the first of 3 passes. It doesn't suffer from missing objects because they come back with pass 2 (BM3D) and 3 (KNLMeansCL).

Can't test performance because only the 32-bit version of Compensate is working.

According to preliminary tests, there's no difference on noise. I'm unsure about mClean vs SpotLess radt=1 or 2, but if I set radt=3, the SpotLess version seems to have a bit more realistic volume to the output. It feels less flat.

Might be a worthy contestant if I can get it to run in 8-bit.

[StainlessS]

Maybe you can also persuade DubHater that MultiVec stuff is a good idea [whole helluva lot less script - EDIT: Especially when greater than RadT=3].

[coolgit]

Stainlesss

I have been doing some experiments to deal with spotty frames at scene changes. Also tried to find a better way of dealing with motion as delta restore is a bit of a hit and miss depending on colours etc. I came up with my first avs(i) script borrowing your spotless function.

1. Remove last and first frames of scene changes with 2nd last and 2nd frames. Currently did it with SCSelect with dfactor=5. A couple of false detections. Worked and saved hours.
Currently I produced a spotless file then use the new file for SCSelect. Stumbled on your SCSelect_HBD and trying that one out next. Tried to merge scene detection with spotless so it can do both functions at the same time, however I can't seem to get it to work. However and as usual flashes still an issue. All scene change when there one colour, black, white, orange, etc. An example white frame, cgi frame, becomes white frame, white frame. Not an issue when watching it on TV, just pointing out the weakness.

How many defects I had to fix using photoshop? 2. Two. TWO. TTTWWWOOO... Get it. Previously I was doing something like 30 to 50 per episode.

2. I created a script that allows spotless to output 2 files with truemotion=true and truemotion=false, with tweaks. Did some analysis and turned out better than delta restore in term of time spent and work level. Neither does any restoring.

SpotLessT=SpotLess(src,BlkSz=BlkSzT,OLap=OLapT,pel=pel,Tm=True,Bblur=BblurT,ThSAD=6000,RadT=RadT). The T after SpotLess etc is Tm=True. BlkSz of 16 is best.

SpotLessF=SpotLess(src,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=1000,RadT=RadT). The F after SpotLess etc is Tm=False. BlkSz of 8 is a must. Going higher creates more lost of motion details.

T thsad at 6000 is excellent at removing all spots and allow medium motion speed and distance from camera motion details to be retain. With thsad at 10000 there was more motion defects and thus more work replacing those frames and the spots fixed were the same as 6000. This is the primary spotless file where all frames are extracted.

F thsad at 1000 will do small spot but excellent at retaining motion, very much the same as source file except for flashes(as usual) and extreme motion speed close to the camera. This file is use to replace bad frames on the above primary T file. In some cases spots are removed in the first frame after scene change, never last frame, and never using T. Since above SCSelect is being use, I suppose this is now a moot point. The number of frames extracted was less compare to previous episodes, mostly because of SCSelect and T.

Source file is still used to replace primary file where there are flashes, but far less due to F and far less chance of having to fix spotty source frame. Today = zero.

I have enclosed a zip file with some images. You can compare the results of the above with other mod. After all the proof is in the pudding.

Here are the codes.

source=FFVideoSource("F:\BABYLON5\Season 2\7 A Race Through Dark Places\A Race Through Dark Places.avi").convertTOYV12(interlaced=false).killaudio()

# VARIABLES--------------------------------------------------------------------

txt_sz = 24 # Test size if infos displayed
src = source

# Spotless variables, to remove dirt and spots.-------------------------------
RadT = 1 # Temporal radius. (MCompensate arg)
Pel = 2 # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
BlkSzT = 16 # Default 8. MAnalyse BlkSize. 16 is better for eliminating all spots. Maybe also better where BIG noise.
BlkSzF = 8 # Default 8. MAnalyse BlkSize. Stay on 8 for greater accuracy in motion analysis and keeping details same as source file.
OLapT = (BlkSzT/2) #important not too low (eg 4 for blksz 12, because creates stairs). Can use 4 if blksz 16 in some cases.
OLapF = (BlkSzF/2) #important not too low (eg 4 for blksz 12, because creates stairs)
BblurT = 0.3 # Default 0.6. Suggest about 0.3 for denoised temporal clean clip.
BblurF = 0.6 # Default 0.6. Rare times about 0.9 helps with hands movement close to camera for better motion analysis, but a bit slower.

# Remove spots-----------------------------------------------------------------
SpotLessT=SpotLess(src,BlkSz=BlkSzT,OLap=OLapT,pel=pel,Tm=True,Bblur=BblurT,ThSAD=6000,RadT=RadT) # Truemotion=True for fixing almost all spots but will have missing details where there is too much motion.
SpotLessF=SpotLess(src,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=1000,RadT=RadT) # Truemotion=False for retaining almost all details when there is too much motion but less and smaller spots fixed.
Use these frames to overwrite above frames whenever necessary.

# FUNCTIONS
#------------------------------------------------------------------------------------------------------------------
function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,6000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,16) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap,BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true,delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend,MultiVec,tr=RadT,thSad=ThSAD,thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

output=SpotLess(src) # stack side by side, Source top left, Spotless TM true top right, Spotless TM false bottom left, ? bottom right.
S1 = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessT, "Spotless TM true", txt_sz)\
)
S2 = StackHorizontal(\
sub(SpotLessF, "Spotless TM false", txt_sz),\
sub(blankclip(src), "", txt_sz)\
)
output = StackVertical(S1,S2)

#return output
#return SpotLessT
return SpotLessF

Trying to merge scene change into spotless - not working.

source=FFVideoSource("F:\BABYLON5\Season 2\6 A Spider In The Web\A Spider In The Web.avi").convertTOYV12(interlaced=false).killaudio()

# VARIABLES--------------------------------------------------------------------

txt_sz = 24 # Test size if infos displayed
src = source

# Spotless variables, to remove dirt and spots.-------------------------------
RadT = 1 # Temporal radius. (MCompensate arg)
Pel = 2 # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
BlkSzT = 16 # Default 8. MAnalyse BlkSize. 16 is better for eliminating all spots. Maybe also better where BIG noise.
BlkSzF = 8 # Default 8. MAnalyse BlkSize. Stay on 8 for greater accuracy in motion analysis and keeping details same as source file.
OLapT = (BlkSzT/2) #important not too low (eg 4 for blksz 12, because creates stairs). Can use 4 if blksz 16 in some cases.
OLapF = (BlkSzF/2) #important not too low (eg 4 for blksz 12, because creates stairs)
BblurT = 0.3 # Default 0.6. Suggest about 0.3 for denoised temporal clean clip.
BblurF = 0.6 # Default 0.6. Rare times about 0.9 helps with hands movement close to camera for better motion analysis, but a bit slower.

# Remove spots-----------------------------------------------------------------
SpotLessT=SpotLess(src,BlkSz=BlkSzT,OLap=OLapT,pel=pel,Tm=True,Bblur=BblurT,ThSAD=6000,RadT=RadT) # Truemotion=True for fixing almost all spots but will have missing details where there is too much motion.
SpotLessF=SpotLess(src,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=1000,RadT=RadT) # Truemotion=False for retaining almost all details when there is too much motion but less and smaller spots fixed.
Use these frames to overwrite above frames whenever necessary.

# FUNCTIONS
#------------------------------------------------------------------------------------------------------------------
function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}

Function SCSelect_HBD(clip dClip,clip Start,clip End,clip Motion, float "dfactor",Float "MinDif",bool "show",String "S_Frames",String "E_Frames") {
dfactor = Float(Default(dfactor,4.5)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif ,1.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect
Show = Default(show,true)
S_Frames= Default(S_Frames,"")
E_Frames= Default(E_Frames,"")
Assert(!DClip.IsRGB,"SCSelect_HBD: DClip YUV Only")
Assert(1.0 <= dFactor,"SCSelect_HBD: 1.0 <= dFactor")
Assert(0.0 <= Mindif ,"SCSelect_HBD: 0.0 <= MinDif")
try { bpc = DClip.BitsPerComponent } catch(err_msg) { bpc = 8 }
DClip = (bpc!=8) ? DClip.ConvertBits(8) : DClip
Sz = Max(Round(Motion.Height*0.05),14)
S_Frames = (S_Frames!="") ? S_Frames.RT_GetFullPathName : ""
E_Frames = (E_Frames!="") ? E_Frames.RT_GetFullPathName : ""
(S_Frames !="") ? RT_FileDelete(S_Frames) : NOP
(E_Frames !="") ? RT_FileDelete(E_Frames) : NOP
Motion.GScriptClip("""
n = current_frame
pDf = Dclip.YDifferenceFromPrevious
nDf = Dclip.YDifferenceToNext
pTh = nDf*dfactor+MinDif
nTh = pDf*dfactor+MinDif
SC = 2 # Default To SC = MOTION (Last is MOTION clip)
if (n==0) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(n==FrameCount-1){ SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS, Fixes ScSelect SOS detect on Last frame
else if(pDf>pTh) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(nDf>nTh) { SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS
if(show) {
mx = Max(pDf,nDf)
mn = Min(pDf,nDf)
Can_dF = (mx > mn + Mindif) # Can any values of dfactor(1.0 and aove) cause scene detect ?
Can_Md = (mx > dFactor * mn) # Can any values of MinDif(0.0 and above) cause scene detect ?
if(Can_dF || Can_Md) {
E_dF = (Can_dF) ? String(Min((mx - Mindif) / max(mn,0.00001),99.9999),"dFact=%7.4f") : ""
E_Md = (Can_Md) ? String(mx - dFactor * mn,"MinD=%7.4f") : ""
EdfS = E_dF + ((Can_dF && Can_Md) ? ":" : "") + E_Md + "}"
EdfS = ((pDf==mx) ? "{SOS:" :"{EOS:") + EdfS
} else { EdfS = "" }
Subtitle(
\ String(n,"%.0f] ") + MidStr("SEG",SC+1,1) + " " + EdfS + "\n" +
\ String(pDf,"SOS: P{%7.3f} > ") + String(pTh,"pTh{%7.3f}") + (SC==0 ? " <==":"") + "\n" +
\ String(nDf,"EOS: N{%7.3f} > ") + String(nTh,"nTh{%7.3f}") + (SC==1 ? " <==":"")
\ ,Font="Courier New",Size=Sz,lsp=0)
}
Return Last
""",Args="dfactor,Start,End,Dclip,MinDif,show,S_Frames,E_Frames,Sz",Local=true) # Needs Grunt for args
return Last
}

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,6000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,16) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap,BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true,delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend,MultiVec,tr=RadT,thSad=ThSAD,thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}


prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessT=SCSelect_HBD(Last,prev,next)

output=SpotLess(src) # stack side by side, Source top left, Spotless TM true top right, Spotless TM false bottom left, ? bottom right.
S1 = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessT, "Spotless TM true", txt_sz)\
)
S2 = StackHorizontal(\
sub(SpotLessF, "Spotless TM false", txt_sz),\
sub(blankclip(src), "", txt_sz)\
)
output = StackVertical(S1,S2)

return output
#return SpotLessT
#return SpotLessF

[StainlessS]

Some interesting observations, thank you.

EDIT: this dont do anything [output overwritten later]

output=SpotLess(src) # stack side by side, Source top left, Spotless TM true top right, Spotless TM false bottom left, ? bottom right.


EDIT: Your sample PAL DVD ?
My B5 has Progressive, Interlaced and even Perverse Telecine sections.

[coolgit]

Some interesting observations, thank you.

EDIT: this dont do anything [output overwritten later]

output=SpotLess(src) # stack side by side, Source top left, Spotless TM true top right, Spotless TM false bottom left, ? bottom right.


EDIT: Your sample PAL DVD ?
My B5 has Progressive, Interlaced and even Perverse Telecine sections.

PAL yes. Same problems with yours.
However my dad just bought the new B5, yucky 4:3 live but brilliant cgi. I have now changed tack and now using original 16/9 live with new 4/3 cgi with black v bars. Brilliant, no need to worry about deinterlacing old cgi which is a mess.

[StainlessS]

OK thanks, gotcha.
Had no idea they had done that.
https://www.google.com/search?client=firefox-b-d&q=babylon+5+live+cgi

Some difference :) :- https://www.youtube.com/watch?v=p_oFTEAU9O8

[coolgit]

Amazingly got spotless and scene change to work.

Just need to output the frame numbers of detected scene frames to a text file for cross checking accuracy of detection. Your current output is of no use to me and I don't understand it either.

source=FFVideoSource("F:\BABYLON5\Season 2\9 The Coming Of Shadows\The Coming Of Shadows SA.avi").convertTOYV12(interlaced=false).killaudio()
#filename="F:\BABYLON5\Season 2\8 Soul Mates\Spotless and sc_hbd.txt"

# VARIABLES--------------------------------------------------------------------

txt_sz = 24 # Test size if infos displayed
src = source


# Spotless variables, to remove dirt and spots.-------------------------------
RadT = 1 # Temporal radius. (MCompensate arg)
Pel = 2 # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
BlkSzT = 16 # Default 8. MAnalyse BlkSize. 16 is better for eliminating all spots. Maybe also better where BIG noise.
BlkSzF = 8 # Default 8. MAnalyse BlkSize. Stay on 8 for greater accuracy in motion analysis and keeping details same as source file.
OLapT = (BlkSzT/2) #important not too low (eg 4 for blksz 12, because creates stairs). Can use 4 if blksz 16 in some cases.
OLapF = (BlkSzF/2) #important not too low (eg 4 for blksz 12, because creates stairs)
BblurT = 0.3 # Default 0.6. Suggest about 0.3 for denoised temporal clean clip.
BblurF = 0.6 # Default 0.6. Rare times about 0.9 helps with hands movement close to camera for better motion analysis, but a bit slower.

# Remove spots-----------------------------------------------------------------
SpotLessT=SpotLess(src,BlkSz=BlkSzT,OLap=OLapT,pel=pel,Tm=True,Bblur=BblurT,ThSAD=6000,RadT=RadT) # Truemotion=True for fixing almost all spots but will have missing details where there is too much motion.
SpotLessF=SpotLess(src,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=1000,RadT=RadT) # Truemotion=False for retaining almost all details when there is too much motion but less and smaller spots fixed.
# Use these frames to overwrite above frames whenever necessary.


# FUNCTIONS
#------------------------------------------------------------------------------------------------------------------
function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}

Function SCSelect_HBD(clip dClip,clip Start,clip End,clip Motion, float "dfactor",Float "MinDif",bool "show",String "S_Frames",String "E_Frames") {
dfactor = Float(Default(dfactor,4.5)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif,1.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect
Show = Default(show,true)
S_Frames= Default(S_Frames,"")
E_Frames= Default(E_Frames,"")
Assert(!DClip.IsRGB,"SCSelect_HBD: DClip YUV Only")
Assert(1.0 <= dFactor,"SCSelect_HBD: 1.0 <= dFactor")
Assert(0.0 <= Mindif ,"SCSelect_HBD: 0.0 <= MinDif")
try { bpc = DClip.BitsPerComponent } catch(err_msg) { bpc = 8 }
DClip = (bpc!=8) ? DClip.ConvertBits(8) : DClip
Sz = Max(Round(Motion.Height*0.05),12)
S_Frames = (S_Frames!="") ? S_Frames.RT_GetFullPathName : ""
E_Frames = (E_Frames!="") ? E_Frames.RT_GetFullPathName : ""
(S_Frames !="") ? RT_FileDelete(S_Frames) : NOP
(E_Frames !="") ? RT_FileDelete(E_Frames) : NOP
Motion.GScriptClip("""
n = current_frame
pDf = Dclip.YDifferenceFromPrevious
nDf = Dclip.YDifferenceToNext
pTh = nDf*dfactor+MinDif
nTh = pDf*dfactor+MinDif
SC = 2 # Default To SC = MOTION (Last is MOTION clip)
if (n==0) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(n==FrameCount-1){ SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS, Fixes ScSelect SOS detect on Last frame
else if(pDf>pTh) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(nDf>nTh) { SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS
if(show) {
mx = Max(pDf,nDf)
mn = Min(pDf,nDf)
Can_dF = (mx > mn + Mindif) # Can any values of dfactor(1.0 and aove) cause scene detect ?
Can_Md = (mx > dFactor * mn) # Can any values of MinDif(0.0 and above) cause scene detect ?
if(Can_dF || Can_Md) {
E_dF = (Can_dF) ? String(Min((mx - Mindif) / max(mn,0.00001),99.9999),"dFact=%7.4f") : ""
E_Md = (Can_Md) ? String(mx - dFactor * mn,"MinD=%7.4f") : ""
EdfS = E_dF + ((Can_dF && Can_Md) ? ":" : "") + E_Md + ""
EdfS = ((pDf==mx) ? "SOS:" :"EOS:") + EdfS
} else { EdfS = "" }
Subtitle(
\ String() + EdfS + "\n" +
\ String(pDf,"SOS") + (SC==0 ? " <":"") + "\n" +
\ String(nDf,"EOS") + (SC==1 ? " <":"")
\ ,Font="Courier New",Size=Sz,lsp=0)
}
Return Last
""",Args="dfactor,Start,End,Dclip,MinDif,show,S_Frames,E_Frames,Sz",Local=true) # Needs Grunt for args
return Last
}

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,6000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,16) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap,BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true,delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend,MultiVec,tr=RadT,thSad=ThSAD,thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCT=src.SCSelect_HBD(next,prev,SpotLessT,show=false)

S1 = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessT, "Spotless TM true", txt_sz)\
)
S2 = StackHorizontal(\
sub(SpotLessF, "Spotless TM false", txt_sz),\
sub(SpotLessSCT, "Spotless and scene change", txt_sz)\
)
output = StackVertical(S1,S2)

#Return output # Comparison

return SpotLessSCT # Conversion
#return SpotLessT
#return SpotLessF

# Debugging to check scene change detection------------------------------------
prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCTD=src.SCSelect_HBD(next,prev,SpotLessT,show=true)

SSCTD = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessSCTD, "Spotless and scene change", txt_sz)\
)

#Return SSCTD # Comparison

#WriteFileIf(SpotLessSCT, filename,
#\ "SC==0", "current_frame", """ ": Start" """,
#\ "SC==1", "current_frame", """ ": End" """, flush=true)

#WriteFileIf(src, filename, "(SC==0) || (SC==1)", "current_frame", """ "(: Start)" """ || """ "(: End)" """, flush=true)

See post 190 for updated script.

[coolgit]

OK thanks, gotcha.
Had no idea they had done that.
https://www.google.com/search?client=firefox-b-d&q=babylon+5+live+cgi

Some difference :) :- https://www.youtube.com/watch?v=p_oFTEAU9O8

Saw them years ago. There was one showing orangey tint making white people look like Donald Trump and i thought, cricky i ain't buying that 4/3 orangutan Trump 5.

[coolgit]

Stainlesss

How do I move your scselect_hbd text output higher to the added black bar?

I used addborder(0,100,0,0) in the debug section for debugging purpose only so the text can go in there. I got the spotless text in there but not the rest.

Checking that spot was fully removed is hard when the text is in the way.

[StainlessS]

If you are taling about the ScSelect_HBD metrics, then cannot be done, ScSelect_HBD knows nothing of the [ external to ScSelect_HBD() ] subtitled border that will be added later.
Your only option might be to eg create a clip with metrics, and chop off and keep the metrics, and another without metrics,
stack the metrics above the no metric clip.
You would also need a dummy [blankclip] clip so that your Stacked frames are same dimensions.

Do that asnwer your question ?

EDIT: I dont actually see anywhere in script that has an Addborders nor a BlackClip.

[coolgit]

The addborder was removed as it didn't work. I will do a small border for spotless script.

SO i should have source clip then clip with metrics then clip without metric then blank clip? 2x2 stacked.

[StainlessS]

2x2 stack sounds good.

[coolgit]

Final update. Works like a dream.

Changed to dfactor,5.0 and MinDif,5.0 to prevent false scene changes in static scenes like SCSelect did. On average most scene changes were 20 or above for both.

source=FFVideoSource("F:\whatever filename directory .avi").convertTOYV12(interlaced=false).killaudio()


# VARIABLES--------------------------------------------------------------------

txt_sz = 24 # Test size if infos displayed
src = source


# Spotless variables, to remove dirt and spots.-------------------------------
RadT = 1 # Temporal radius. (MCompensate arg)
BlkSzT = 16 # Default 8. MAnalyse BlkSize. 16 is better for eliminating all spots. Maybe also better where BIG noise.
BlkSzF = 8 # Default 8. MAnalyse BlkSize. Stay on 8 for greater accuracy in motion analysis and keeping details same as source file.
OLapT = (BlkSzT/2) #important not too low (eg 4 for blksz 12, because creates stairs). Can use 4 if blksz 16 in some cases.
OLapF = (BlkSzF/2) #important not too low (eg 4 for blksz 12, because creates stairs)
BblurT = 0.3 # Default 0.6. Suggest about 0.3 for denoised temporal clean clip.
BblurF = 0.6 # Default 0.6. Rare times about 0.9 helps with hands movement close to camera for better motion analysis, but a bit slower.

# Remove spots-----------------------------------------------------------------
SpotLessT=SpotLess(src,BlkSz=BlkSzT,OLap=OLapT,pel=2,Tm=True,Bblur=BblurT,ThSAD=6000,RadT=RadT) # Truemotion=True for fixing almost all spots but will have missing details where there is too much motion.
SpotLessF=SpotLess(src,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=1000,RadT=RadT) # Truemotion=False for retaining almost all details when there is too much motion but less and smaller spots fixed.
# Use these frames to overwrite above frames whenever necessary.


# FUNCTIONS
#------------------------------------------------------------------------------------------------------------------
function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}

Function SCSelect_HBD(clip dClip,clip Start,clip End,clip Motion, float "dfactor",Float "MinDif",bool "show",String "S_Frames",String "E_Frames") {
dfactor = Float(Default(dfactor,5.0)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif,5.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect
Show = Default(show,true)
S_Frames= Default(S_Frames,"")
E_Frames= Default(E_Frames,"")
Assert(!DClip.IsRGB,"SCSelect_HBD: DClip YUV Only")
Assert(1.0 <= dFactor,"SCSelect_HBD: 1.0 <= dFactor")
Assert(0.0 <= Mindif ,"SCSelect_HBD: 0.0 <= MinDif")
try { bpc = DClip.BitsPerComponent } catch(err_msg) { bpc = 8 }
DClip = (bpc!=8) ? DClip.ConvertBits(8) : DClip
Sz = Max(Round(Motion.Height*0.05),12)
S_Frames = (S_Frames!="") ? S_Frames.RT_GetFullPathName : ""
E_Frames = (E_Frames!="") ? E_Frames.RT_GetFullPathName : ""
(S_Frames !="") ? RT_FileDelete(S_Frames) : NOP
(E_Frames !="") ? RT_FileDelete(E_Frames) : NOP
Motion.GScriptClip("""
n = current_frame
pDf = Dclip.YDifferenceFromPrevious
nDf = Dclip.YDifferenceToNext
pTh = nDf*dfactor+MinDif
nTh = pDf*dfactor+MinDif
SC = 2 # Default To SC = MOTION (Last is MOTION clip)
if (n==0) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(n==FrameCount-1){ SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS, Fixes ScSelect SOS detect on Last frame
else if(pDf>pTh) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(nDf>nTh) { SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS
if(show) {
mx = Max(pDf,nDf)
mn = Min(pDf,nDf)
Can_dF = (mx > mn + Mindif) # Can any values of dfactor(1.0 and aove) cause scene detect ?
Can_Md = (mx > dFactor * mn) # Can any values of MinDif(0.0 and above) cause scene detect ?
if(Can_dF || Can_Md) {
E_dF = (Can_dF) ? String(Min((mx - Mindif) / max(mn,0.00001),99.9999),"dFact=%7.4f") : ""
E_Md = (Can_Md) ? String(mx - dFactor * mn,"MinD=%7.4f") : ""
EdfS = E_dF + ((Can_dF && Can_Md) ? ":" : "") + E_Md + ""
EdfS = ((pDf==mx) ? "SOS:" :"EOS:") + EdfS
} else { EdfS = "" }
Subtitle(
\ String() + EdfS + "\n" +
\ String(pDf,"SOS") + (SC==0 ? " <":"") + "\n" +
\ String(nDf,"EOS") + (SC==1 ? " <":"")
\ ,Font="Courier New",Size=Sz,lsp=0)
}
Return Last
""",Args="dfactor,Start,End,Dclip,MinDif,show,S_Frames,E_Frames,Sz",Local=true) # Needs Grunt for args
return Last
}

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,6000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,16) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap,BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true,delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend,MultiVec,tr=RadT,thSad=ThSAD,thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCT=src.SCSelect_HBD(next,prev,SpotLessT,show=false)

S1 = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessT, "Spotless TM true", txt_sz)\
)
S2 = StackHorizontal(\
sub(SpotLessF, "Spotless TM false", txt_sz),\
sub(SpotLessSCT, "Spotless and scene change", txt_sz)\
)
output = StackVertical(S1,S2)

#Return output # Comparison

#return SpotLessSCT # Conversion
#return SpotLessT
#return SpotLessF

# Debugging to check scene change detection------------------------------------
prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCTDM=src.SCSelect_HBD(next,prev,SpotLessT,show=true)

SSCTD1 = StackHorizontal(\
sub(src.AddBorders(0,30,0,0), "Source", txt_sz),\
sub(SpotLessSCTDM.AddBorders(0,30,0,0), "Spotless and scene change with metrics", txt_sz)\
)
SSCTD2 = StackHorizontal(\
sub(SpotLessSCT.AddBorders(0,30,0,0), "Spotless and scene change without metrics", txt_sz),\
sub(BlankClip(src).AddBorders(0,30,0,0), "Blank", txt_sz)\
)
outputD = StackVertical(SSCTD1,SSCTD2)

Return outputD # Comparison

[StainlessS]

CoolG, bout ScSecelct_HBD,
Did this lot make any sense to you, was it at all useful/usable ? [its not immediately obvious to me unless I got the below doc in front of me :) ].


Show, Default False. In original ScSelect, this arg is called Debug, and output some info to DebugView (google),
we show info on frame and so have changed arg name to Show but of roughly same functionality.

Metrics when Show=True (metrics provided as if in 8 bit range):
>>>>>>>>>>>>>>>>>
20531 ] S {SOS: dFact= 19.7365:MinD=54.6495}
SOS: P{ 66.214} > pTh{ 12.565}<==
EOS: N{ 3.304} > nTh{232.750}
<<<<<<<<<<<<<<<<<

20531, Is current Frame Number
S, Is flagging a detected Start-Of-Scene (Also 'E'=End-Of-Scene, and 'G' = Global motion.
SOS: P{ 66.214} > pTh{ 12.565} <== The arrow "<==" at the end denotes a SOS (Start Of Scene) detection.
'P' (short for pDf, screen space saving) is difference to previous frame.
pTh = dFactor * nDf + MinDif
EOS: N{ 3.304} > nTh{232.750} Much as Above, but no EOS (End Of Scene) detection.
'N' (short for nDf), is difference to next frame.
nTh = dFactor * pDf + MinDif

{SOS: dFact= 19.7365:MinD=54.6495}, NOT ALWAYS SHOWN.
The Scene change Arg, matching estimate info, in this case for SOS estimate.
Shown only if pDf, nDF, and arg limits on dFactor and MinDif could
produce either SOS or EOS. This indicator only makes any sense on frames
that YOU have decided should be a scene change frame, otherwise ignore.
dFact is the dFactor that will produce an EXACT match of the threshold
(SOS pTh or EOS nTh) to the higher difference {SOS pDf or EOS nDf), when
MinDif unchanged.
As this example shows SOS, so the threshold is pTh and higher difference is pDf,
so it balances this line,
P{ 66.214} == pTh{66.214}, or
P{ 66.214} == {dfact * nDf + MinDif)
P{ 66.214} == {19.7365 * nDf + MinDif)
So in this particular instance it shows that the maximum value for dFactor arg
that could produce SOS on this frame with current Mindif is LESS THAN 19.7365.
It shows that lowest value of dFactor that would fail to produce a SOS, so the maxium
value of dFactor that would succeed to detect SOS for current frame is just below this
value. (pDf has to be greater than pTh so when equal will fail SOS detect).
Similar can be deduced when EOS estimator is shown.
MinD is similar estimator (in this case again for SOS) which shows where MinDif
will balance (with current dFactor unchanged)
P{ 66.214} == pTh{66.214}, or
P{ 66.214} == {dfactor * nDf + MinD)
P{ 66.214} == {dfactor * nDf + 54.641195)
Similar can be deduced when EOS estimator is shown.
Hope some of that made at least a little sense, is intended to give you some
idea about how much you can raise (or lower) dFactor or Mindif to get
a successful detection on current frame.


Changed to dfactor,5.0 and MinDif,5.0 to prevent false scene changes in static scenes like SCSelect did
Did you actually experience false detects ?
Default values should have been pretty good. MinDif=1.0 (default) should avoid almost all false scene change detects [possible exception some anime].

[coolgit]

CoolG, bout ScSecelct_HBD,
Did this lot make any sense to you, was it at all useful/usable ?

A bit of it.

Only dfactor and mindif are important in deciding scene change. The rest doesn't do anything hence why I'd removed it from the subtitle. There is such a thing called 'information overload'. If both are above the variables of, currently, 5.0 then it is a scene change.

Test it on s2 ep7 A Race Through Dark Places, frames 3300 onwards where Bester was torturing a guy for info. Pay attention to the static frames where the background was black and white (like chess board pattern), behind Bester. Some frames jumped up 2 ish pixels then back down to normal. SCSelect flagged this as scene change and swap frames. Yours, because of mindif, didn't except for 1 frame because mindif was 1.0. Hence why putting it up to 5.0 is the solution and doesn't affect real scene changes.

In GROPOS I had 1 scene change that wasn't detected but was detected by vdub. My vdub is set at 240. It does the job and stops when there too much motion as those frames will mostly needs replacing by spotlessF file. The failed detection was because the background was the same except the camera moved from 1 actor to another. Does it matter? If vdub stops then i can see if there is any spots before moving on. If there isn't then detection failure doesn't matter, no frames are swapped. However in this case there was a spot and that is when I knew the detection failed. That was the only frame I had to fix in photoshop, just 1, bloody brilliant compare to 50 to 70.

In essence explaining dfactor and mindif is enough. P, Pth, ndf, are not important and just confused the hell out of me.

[StainlessS]

Thanks for your insight :)
[it is though of use to me to find out why something caused +ve/-ve detect, even if diags from numbers not immediately obvious, even to me].

EDIT: I could though add a "verbose" type arg to allow switch on/off dfact/minD estimates.

[coolgit]

Just curious how this happened. Clear scene change but not detected and low dfactor and 0 mindif. Vdub had no problems.

https://i.ibb.co/vHpKztV/029835.png

https://i.ibb.co/MfFphtG/029836.png

Also Mindif reduce to 3.0. It worked better as 5.0 was an overkill on my part.

[coolgit]

Full script in two parts. Include both functions explanations. Put it together ok.

/*
SpotLess v1.07Beta. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.

NOT FOR cartoon/anime, live video only, sorry.

v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.
v1.05, Additional checks on args.
v1.06, Glob Default true, Almost always better.
v1.07Beta, bBlur Default changed from 0.0 OFF, to 0.6, slower but better for mc analysis. Maybe try 1.0 for HD to overcome HD 'gritiness'.
Added dc arg, detect clip.

SpotLess(clip c,int "RadT"=1,int "ThSAD"=10000,int "ThSAD2"=ThSAD,int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,
\ bool "tm"=true,Bool "glob"=True,Float "bBlur"=0.6, clip "dc"=Undefined)

RadT, 1 or more, Default 1. Removes Spots on up to RadT [Temporal Radius] consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD, 0 < ThSAD2 < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.

Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)

Chroma, Default True. MAnalyse chroma arg. If true, use chroma in block matching when creating vectors. Maybe use False if source B&W or color noise.

BlkSz, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]

OLap, Default half BlkSz, Block overlap.

Tm, TrueMotion Default True. Some folk swear truemotion=false is better.

Glob, Default True (True v1.06, was same as Tm, true almost always better), Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable.

bBlur, Default 0.6. If used, Suggest about 0.6, where MAnalyse create vectors is performed on denoised (blurred) super clip
for better motion analysis. Maybe try 1.0 for HD clip to counteract HD 'grittiness'. bBlur ignored if dc clip specified,
If providing bBlur, then ALWAYS specify as named argument ie bBlur=bBlur, we will likely insert any additional MvTools args
before bBlur to keep them together.

dc, Default UnDefined. If dc Specified, then must be same size and colorspace as source clip.
If detection clip specified then bBlur prefilter arg is ignored, and analysis is performed on
dc.MSuper() clip [ instead of c.Blur(bBlur).MSuper() ]. Allows to provide your own prefiltered clip.
If providing dc clip, then ALWAYS specify as named argument ie dc=DC, we will likely insert any additional MvTools args
before bBlur to keep them together.

#---------------------------------------------------------------------------------------------------------------------------------------

ScSelect_HBD(), v0.01. by StainlessS @ Doom9. : https://forum.doom9.org/showthread.php?t=182392
Requires Grunt (c) Gavino. If S_Frames or E_Frames != "" then also requires RT_Stats.

SCSelect_HBD(clip dClip,clip Start,clip End,clip Motion,
float "dfactor"=3.5,Float "MinDif"=1.0,
bool "show"=False,String "S_Frames"="",String "E_Frames"="")

ScSelect_HBD(), Mod of ScSelect_Like():- https://forum.doom9.org/showthread.php?p=1644023#post1644023
Named with "_HDB" extender as should work ok with HBD in AVS+.
ScSelect_Like() emulated ScSelect() from RemoveDirt.
RemoveDirt's SCSelect(clip input, clip scene_begin, clip scene_end, clip global_motion, float dfactor, bool debug, bool planar)
To emulate original ScSelect results, use ScSelect_HBD(dFactor=4.0,MinDif=0.0).

Args:
dClip, Detection clip. May be different size and colorspace to Start, End and Motion clips. YUV only.
If High Bit Depth (greater than 8 bits per channel) then will be converted to 8 bit for scene change
detection stage. For larger source clips, you may want to downsize the dClip to speed up detection .

Start, Clip of frames that will replace frames from Motion clip when Start-Of-Scene detected.
End, Clip of frames that will replace frames from Motion clip when End-Of-Scene detected.
Motion, Clip the will be used where neither start nor end of scene detected.
Start, End, and Motion, clips can be different colorspace and size compared to dClip (unlike ScSelect),
but must all be same colorspace and size as each other, also can be high-bit-depth or RGB.

dFactor, Default 3.5. (1.0 <= dFactor) Detection factor, a multiplier applied to frame difference
metrics to decide if a scene change.
Mindif, Default 1.0. (0.0 <= Mindif, Supply MinDif as if in 8 bit range, 0.0 -> 255.0)
An additional component part of scene change detection.
Where difference measurements,
pDf = Average per pixel Luma difference, between dClip current frame and previous frame (0.0->255.0).
nDf = Average per pixel Luma difference, between dClip current frame and next frame (0.0->255.0).
pTh = dFactor * nDf + MinDif : If pDf greater than this then is SOS, Start-Of-Scene.
nTh = dFactor * pDf + MinDif : If nDf greater than this then is EOS, End-Of-Scene.
Detect method,
SOS = pDf > pTh : SOS = pDf > dFactor * nDf + MinDif, Also,
EOS = nDf > nTh : EOS = nDf > dFactor * pDf + MinDif.
(If pDf > pTh, then pDf is also greater than nDf as dFactor is at least 1.0 and MinDif at least 0.0)
Original ScSelect, used default dFactor=4.0, and equivalent of MinDif=0.0 (It had no MinDif arg).
Using only dFactor, and no Mindif (ie 0.0), is not too good as a detector, especially where
scene of low motion or near static scene, eg when nDf=0.0 then pDf value of anything above 0.0
would detect a scene change, not exactly as desired for near static scene, additinal component is needed.
In same situation where MinDif=1.0 and nDf=0.0, so SOS = pDf > pTh : SOS = pDf > 4.0 * 0.0 + 1.0
so, to detect Start of scene then pDf has to be greater than 1.0 for a start of scene detect.
In near static scene, we are most reliant upon Mindif being of sufficient magnitude to avoid a false
detection. In scene with plenty of motion, then value of dFactor becomes more important to prevent false
detection. In a very jerky or anime scene, then probably a high dFactor and high Mindif are required.
In this crazy mixed up world, you have to find a good compromise for dFactor and MinDif
so as to detect scene changes without producing false detections.
Our default of dFactor=3.5 and Mindif=1.0, is just one combination, you might prefer bigger Mindif
and smaller dFactor,(whatever you choose, some clips will require an alternative set).
I suggest dFactor usually no less than about 2.5 and no more than 4.0, and Mindif no less than 0.5, and
maybe no more than about 4.0. As you lower dFactor, you might want to raise Mindif, and vise versa.
Anime is likely to have different requirments with bigger MinDif (maybe 4 or more but depends upon the clip).

[coolgit]

Part 2

Show, Default False. In original ScSelect, this arg is called Debug, and output some info to DebugView (google),
we show info on frame and so have changed arg name to Show but of roughly same functionality.

Metrics when Show=True (metrics provided as if in 8 bit range):
>>>>>>>>>>>>>>>>>
20531 ] S {SOS: dFact= 19.7365:MinD=54.6495}
SOS: P{ 66.214} > pTh{ 12.565}<==
EOS: N{ 3.304} > nTh{232.750}
<<<<<<<<<<<<<<<<<

20531, Is current Frame Number
S, Is flagging a detected Start-Of-Scene (Also 'E'=End-Of-Scene, and 'G' = Global motion.
SOS: P{ 66.214} > pTh{ 12.565} <== The arrow "<==" at the end denotes a SOS (Start Of Scene) detection.
'P' (short for pDf, screen space saving) is difference to previous frame.
pTh = dFactor * nDf + MinDif
EOS: N{ 3.304} > nTh{232.750} Much as Above, but no EOS (End Of Scene) detection.
'N' (short for nDf), is difference to next frame.
nTh = dFactor * pDf + MinDif

{SOS: dFact= 19.7365:MinD=54.6495}, NOT ALWAYS SHOWN.
The Scene change Arg, matching estimate info, in this case for SOS estimate.
Shown only if pDf, nDF, and arg limits on dFactor and MinDif could
produce either SOS or EOS. This indicator only makes any sense on frames
that YOU have decided should be a scene change frame, otherwise ignore.
dFact is the dFactor that will produce an EXACT match of the threshold
(SOS pTh or EOS nTh) to the higher difference {SOS pDf or EOS nDf), when
MinDif unchanged.
As this example shows SOS, so the threshold is pTh and higher difference is pDf,
so it balances this line,
P{ 66.214} == pTh{66.214}, or
P{ 66.214} == {dfact * nDf + MinDif)
P{ 66.214} == {19.7365 * nDf + MinDif)
So in this particular instance it shows that the maximum value for dFactor arg
that could produce SOS on this frame with current Mindif is LESS THAN 19.7365.
It shows that lowest value of dFactor that would fail to produce a SOS, so the maxium
value of dFactor that would succeed to detect SOS for current frame is just below this
value. (pDf has to be greater than pTh so when equal will fail SOS detect).
Similar can be deduced when EOS estimator is shown.
MinD is similar estimator (in this case again for SOS) which shows where MinDif
will balance (with current dFactor unchanged)
P{ 66.214} == pTh{66.214}, or
P{ 66.214} == {dfactor * nDf + MinD)
P{ 66.214} == {dfactor * nDf + 54.641195)
Similar can be deduced when EOS estimator is shown.
Hope some of that made at least a little sense, is intended to give you some
idea about how much you can raise (or lower) dFactor or Mindif to get
a successful detection on current frame.

S_Frames, Default "". Name of Start-Of-Scene Frames text file. Req RT_Stats if non "".
E_Frames, Default "". Name of End-Of-Scene Frames text file. Req RT_Stats if non "".
*/

source=FFVideoSource("F:\whatever .avi").convertTOYV12(interlaced=false).killaudio()

# VARIABLES--------------------------------------------------------------------

txt_sz = 24 # Test size if infos displayed
src = source


# Spotless variables, to remove dirt and spots.-------------------------------
RadT = 1 # Temporal radius. (MCompensate arg)
BlkSzT = 16 # Default 8. MAnalyse BlkSize. 16 is better for eliminating all spots. Maybe also better where BIG noise.
BlkSzF = 8 # Default 8. MAnalyse BlkSize. Stay on 8 for greater accuracy in motion analysis and keeping details same as source file.
OLapT = (BlkSzT/2) #important not too low (eg 4 for blksz 12, because creates stairs). Can use 4 if blksz 16 in some cases.
OLapF = (BlkSzF/2) #important not too low (eg 4 for blksz 12, because creates stairs)
BblurT = 0.3 # Default 0.6. Suggest about 0.3 for denoised temporal clean clip.
BblurF = 0.6 # Default 0.6. Rare times about 0.9 helps with hands movement close to camera for better motion analysis, but a bit slower.

# Remove spots-----------------------------------------------------------------
SpotLessT=SpotLess(src,BlkSz=BlkSzT,OLap=OLapT,pel=2,Tm=True,Bblur=BblurT,ThSAD=6000,RadT=RadT) # Truemotion=True for fixing almost all spots but will have missing details where there is too much motion.
SpotLessF=SpotLess(src,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=1000,RadT=RadT) # Truemotion=False for retaining almost all details when there is too much motion but less and smaller spots fixed.
# Use these frames to overwrite above frames whenever necessary.


# FUNCTIONS
#------------------------------------------------------------------------------------------------------------------
function sub(c, label, txt_sz) {
return subtitle(c, label, size = txt_sz, align=8)
}

Function SCSelect_HBD(clip dClip,clip Start,clip End,clip Motion, float "dfactor",Float "MinDif",bool "show",String "S_Frames",String "E_Frames") {
dfactor = Float(Default(dfactor,3.5)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif,1.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect
Show = Default(show,true)
S_Frames= Default(S_Frames,"")
E_Frames= Default(E_Frames,"")
Assert(!DClip.IsRGB,"SCSelect_HBD: DClip YUV Only")
Assert(1.0 <= dFactor,"SCSelect_HBD: 1.0 <= dFactor")
Assert(0.0 <= Mindif ,"SCSelect_HBD: 0.0 <= MinDif")
try { bpc = DClip.BitsPerComponent } catch(err_msg) { bpc = 8 }
DClip = (bpc!=8) ? DClip.ConvertBits(8) : DClip
Sz = Max(Round(Motion.Height*0.05),12)
S_Frames = (S_Frames!="") ? S_Frames.RT_GetFullPathName : ""
E_Frames = (E_Frames!="") ? E_Frames.RT_GetFullPathName : ""
(S_Frames !="") ? RT_FileDelete(S_Frames) : NOP
(E_Frames !="") ? RT_FileDelete(E_Frames) : NOP
Motion.GScriptClip("""
n = current_frame
pDf = Dclip.YDifferenceFromPrevious
nDf = Dclip.YDifferenceToNext
pTh = nDf*dfactor+MinDif
nTh = pDf*dfactor+MinDif
SC = 2 # Default To SC = MOTION (Last is MOTION clip)
if (n==0) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(n==FrameCount-1){ SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS, Fixes ScSelect SOS detect on Last frame
else if(pDf>pTh) { SC = 0 Start S_Frames!=""?RT_WriteFile(S_Frames,"%d",n,Append=true):NOP} # SOS
else if(nDf>nTh) { SC = 1 End E_Frames!=""?RT_WriteFile(E_Frames,"%d",n,Append=true):NOP} # EOS
if(show) {
mx = Max(pDf,nDf)
mn = Min(pDf,nDf)
Can_dF = (mx > mn + Mindif) # Can any values of dfactor(1.0 and aove) cause scene detect ?
Can_Md = (mx > dFactor * mn) # Can any values of MinDif(0.0 and above) cause scene detect ?
if(Can_dF || Can_Md) {
E_dF = (Can_dF) ? String(Min((mx - Mindif) / max(mn,0.00001),99.9999),"dFact=%7.4f") : ""
E_Md = (Can_Md) ? String(mx - dFactor * mn,"MinD=%7.4f") : ""
EdfS = E_dF + ((Can_dF && Can_Md) ? ":" : "") + E_Md + ""
EdfS = ((pDf==mx) ? "SOS:" :"EOS:") + EdfS
} else { EdfS = "" }
Subtitle(
\ String(n,"%.0f ") + MidStr("SEG",SC+1,1) + " " + EdfS + "\n" +
\ String(pDf,"SOS: P{%7.3f} > ") + String(pTh,"pTh{%7.3f}") + (SC==0 ? " <==":"") + "\n" +
\ String(nDf,"EOS: N{%7.3f} > ") + String(nTh,"nTh{%7.3f}") + (SC==1 ? " <==":"")
\ ,Font="Courier New",Size=Sz,lsp=0)
}
Return Last
""",Args="dfactor,Start,End,Dclip,MinDif,show,S_Frames,E_Frames,Sz",Local=true) # Needs Grunt for args
return Last
}

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob", Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,6000) # SAD threshold at radius 1 (Default Nearly OFF).
ThSAD2 = Default(ThSAD2,ThSAD) # SAD threshold at radius RadT.
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,16) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap,BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.6) # Default 0.6. Suggest about 0.6 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true,delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend,MultiVec,tr=RadT,thSad=ThSAD,thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCT=src.SCSelect_HBD(next,prev,SpotLessT,show=false)

S1 = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessT, "Spotless TM true", txt_sz)\
)
S2 = StackHorizontal(\
sub(SpotLessF, "Spotless TM false", txt_sz),\
sub(SpotLessSCT, "Spotless and scene change", txt_sz)\
)
output = StackVertical(S1,S2)

#Return output # Comparison

return SpotLessSCT # Conversion
#return SpotLessT
#return SpotLessF

# Debugging to check scene change detection------------------------------------
prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCTDM=src.SCSelect_HBD(next,prev,SpotLessT,show=true)

SSCTD1 = StackHorizontal(\
sub(src.AddBorders(0,30,0,0), "Source", txt_sz),\
sub(SpotLessSCTDM.AddBorders(0,30,0,0), "Spotless and scene change with metrics", txt_sz)\
)
SSCTD2 = StackHorizontal(\
sub(SpotLessSCT.AddBorders(0,30,0,0), "Spotless and scene change without metrics", txt_sz),\
sub(BlankClip(src).AddBorders(0,30,0,0), "Blank", txt_sz)\
)
outputD = StackVertical(SSCTD1,SSCTD2)

#Return outputD # Comparison

[StainlessS]

You should not really change defaults inside the script,
Defaults

dfactor = Float(Default(dfactor,3.5)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif ,1.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect


Your mod

dfactor = Float(Default(dfactor,5.0)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif,3.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect

They are both still a bit on the high side, and i would only really suggest MinDif = 3.0 or 4.0 for some anime.

On 29836 [SOS] it shows that with current MinDif (3.0) the maximum DFact that could detect as SOS is 2.0028 which is a lot less
than your DFact setting of 5.0. Even reducing Dfact to suggested 3.5 and MidDif to suggested default 1.0, it would still not detect,
problem is the big frame 29836 difference to next frame 29837 which is 8.641,
Difference to Previous (ie 20.306) has to be bigger than (5.0 * 8.641 + 3.0) [ie 46.205] to be detected as SOS,
even with defaults, would have to be greater than (3.5 * 8.641 + 1.0) [ie 31.2435] to be detected as SOS.
EDIT: 29836 -> 29837, diff will be a bit of a jolt, maybe a small pan. [EDIT: We dont know, we cant see it]

SCSelect / SCSelect_HBD are not a perfect solution to detect scene change, sad but there you go.

[coolgit]

You should not really change defaults inside the script,
Defaults

dfactor = Float(Default(dfactor,3.5)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif ,1.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect


Your mod

dfactor = Float(Default(dfactor,5.0)) # Suggest 2.5 -> 4.0 ::: Change default to 4.0 to emulate ScSelect
Mindif = Float(Default(MinDif,3.0)) # Suggest 0.5 -> 4.0 ::: Change default to 0.0 to emulate ScSelect

They are both still a bit on the high side, and i would only really suggest MinDif = 3.0 or 4.0 for some anime.

Done.

On 29836 [SOS] it shows that with current MinDif (3.0) the maximum DFact that could detect as SOS is 2.0028 which is a lot less
than your DFact setting of 5.0. Even reducing Dfact to suggested 3.5 and MidDif to suggested default 1.0, it would still not detect,
problem is the big frame 29836 difference to next frame 29837 which is 8.641,
Difference to Previous (ie 20.306) has to be bigger than (5.0 * 8.641 + 3.0) [ie 46.205] to be detected as SOS,
even with defaults, would have to be greater than (3.5 * 8.641 + 1.0) [ie 31.2435] to be detected as SOS.
EDIT: 29836 -> 29837, diff will be a bit of a jolt, maybe a small pan.

SCSelect / SCSelect_HBD are not a perfect solution to detect scene change, sad but there you go.

Just curious how it works. Perhaps the lack of colours in 29839 threw off the calculations.

However some common sense has to come into play here.

If there is an end of scene change then obviously the next frame must be the start of scene change, no matter what the maths is. What else can it be?

[StainlessS]

then obviously the next frame must be the start of scene change
ScSelect always had the problem of SOS sometimes not following EOS or EOS not always before SOS, my mod does not change that.
EOS frame considers only itself and those either side, SOS frame considers only itself and those either side,
so EOS knows nothing of the frame following SOS and SOS knows nothing of the frame before EOS.
no matter what the maths is
It aint the math, its logic that fails, ie the algorithm, but IIRC, mvTools scene change detect has exact same problem. [ie mismatched EOS / SOS detects]

ScSelect_HBD is definitely better than ScSelect, but it aint perfect by any means.

EDIT: Maybe you can make some use of the stuff here:- https://forum.doom9.org/showthread.php?p=1954985#post1954985
EndOfSceneClip() and StartOfSceneClip.

[StainlessS]

Or maybe you could try this [not much testing at all].


Function SceneCutSelectClip(clip dClip,clip Start,clip End,clip Both,clip Motion,Int "thSCD1",Int "thSCD2") { # (c) ssS: https://forum.doom9.org/showthread.php?p=1955111#post1955111
thSCD1=Default(thSCD1,400) thSCD2=Default(thSCD2,130)
Sup = dClip.MSuper(pel=1,sharp=0,rfilter=2,hpad=16, vpad=16)
BvEos= Sup.MAnalyse(isb=True, delta=1,blksize=16)
FvSos= Sup.MAnalyse(isb=False, delta=1,blksize=16)
Eos = dClip.MSCDetection(BvEos,thSCD1=thSCD1,thSCD2=thSCD2).crop(0,0,4,4)
Sos = dClip.MSCDetection(FvSos,thSCD1=thSCD1,thSCD2=thSCD2).crop(0,0,4,4)
CondS="""
ix = (Sos.AverageLuma>0?1:0) + (Eos.AverageLuma>0?2:0)
Return ix==0?Motion:ix==1?Start:ix==2?End:Both
"""
ARGS="Motion,CondS,Motion,Start,End,Both,Eos,Sos"
Motion.GSCriptClip(CondS,Args=ARGS,After_Frame=True,Local=True) # Requires Grunt
}

EDIT: Added the Crop(0,0,4,4) things.

It works sorta like ScSelect/ScSelect_HBD, but you must also Provide a "BOTH" [ie both EOS and SOS detect] clip (you can choose whatever solution you like for that possible outcome).

AviSource("D:\hard sub - 01 WEBdlRip 720p, 23.976.mkv.AVI")
dclip = BilinearResize(320,240).Blur(1.0) # Whatever (just testing frame size can be different to other clips)
ConvertToRGB32 # Just testing works where Dclip colorspace is differenct from the other clips.
Motion = Last
Start = Subtitle("START",size=64,align=5)
End = Subtitle("END",size=64,align=5)
Both = Subtitle("BOTH",size=64,align=5)
SceneCutSelectClip(dClip,Start,End,Both,Motion,400,130)


EDIT: You will likely fire a BOTH if there is a single frame scene cut, ie an 'odd' frame that belongs neither with previous nor following scenes.
EDIT: There is at least one frame that will fire BOTH in either X-Men 1 or X-Men 2. [excluding intro where CGI flashes wildly and you cant tell where real scene changes are]

[coolgit]

ScSelect always had the problem of SOS sometimes not following EOS or EOS not always before SOS, my mod does not change that.
EOS frame considers only itself and those either side, SOS frame considers only itself and those either side,
so EOS knows nothing of the frame follwing SOS and SOS knows nothing of the frame before EOS.

It aint the math, its logic that fails, ie the algorithm, but IIRC, mvTools scene change detect has exact same problem. [ie mismatched EOS / SOS detects]

ScSelect_HBD is definitely better than ScSelect, but it aint perfect by any means.

EDIT: Maybe you can make some use of the stuff here:- https://forum.doom9.org/showthread.php?p=1954985#post1954985
EndOfSceneClip() and StartOfSceneClip.

Don't worry no software is perfect, otherwise Microsoft would have went bankrupt in the last century.

What I meant was something like this.

If EOS=True
Then nextframe=SOS (not relying on maths but a fact that SOS must be after EOS. Accept for start of video of course.)
Else end of video.

[StainlessS]

But if its wrong, then they're both then wrong.
And if SOS detected and not EOS, then how does it go back and make EOS true on previous frame [its already gone past it].
There is no easy solution testing only 3 frame, prev,curr,next, and testing 2 prev, curr, 2 next aint without its own problems either.

[coolgit]

But if its wrong, then they're both then wrong.
And if SOS detected and not EOS, then how does it go back and make EOS true on previous frame [its already gone past it].
There is no easy solution testing only 3 frame, prev,curr,next, and testing 2 prev, curr, 2 next aint without its own problems either.

If EOS is not detected then there can be no SOS.
If SOS is detected but not previous frame as EOS then there is no SOS.

For scene change to be true there must be EOS and SOS. It can't be either or none.

[StainlessS]

For scene change to be true there must be EOS and SOS. It can't be either or none.
Tell that to Kassandro (ScSelect) and the authors of MvTools (Manao, Fizick, Tsp, TSchniede, SEt, cretindesalpes, pinterf).

If you dont like'em, there are others to choose from:- http://avisynth.nl/index.php/SCDetect#Scene_Change_Detection

[MysteryX]

You could add the VapourSynth code version to the first post. I'll remove it from my code but don't want this to get lost.

In VapourSynth, it currently only works with the 32-bit library. The dubhater library has a bug.


# SpotLess denoising method (m1=4) EXPERIMENTAL
def SpotLess(c: vs.VideoNode, radt: int = 1, thsad: int = 10000, thsad2: Optional[int] = None, pel: int = 2, chroma: bool = True, blksize: int = 8,
overlap: Optional[int] = None, truemotion: bool = True, pglobal: bool = True, blur: float = 0.0, ref: Optional[vs.VideoNode] = None) -> vs.VideoNode:
if radt < 1 or radt > 3:
raise ValueError("Spotless: radt must be between 1 and 3")
if not pel in [1, 2, 4]:
raise ValueError("Spotless: pel must be 1, 2 or 4")

thsad2 = thsad2 or thsad
icalc = c.format.bits_per_sample < 32
S = core.mv.Super if icalc else core.mvsf.Super
A = core.mv.Analyse if icalc else core.mvsf.Analyse
C = core.mv.Compensate if icalc else core.mvsf.Compensate
pad = max(blksize, 8)

sup = ref or (c.std.Convolution(matrix=[1, 2, 1, 2, 4, 2, 1, 2, 1]) if blur > 0.0 else c)
sup = S(sup, hpad=pad, vpad=pad, pel=pel, sharp=2)
sup_rend = S(c, hpad=pad, vpad=pad, pel=pel, sharp=2, levels=1) if ref or blur > 0.0 else sup

th1 = thsad
th2 = (thsad + thsad2)/2 if radt==3 else thsad2
th3 = thsad2

bv1 = A(sup, isb=True, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv1 = A(sup, isb=False, delta=1, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
if radt >= 2:
bv2 = A(sup, isb=True, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv2 = A(sup, isb=False, delta=2, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
if radt >= 3:
bv3 = A(sup, isb=True, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)
fv3 = A(sup, isb=False, delta=3, blksize=blksize, overlap=overlap, chroma=chroma, truemotion=truemotion, pglobal=pglobal)

bc1 = C(c, sup_rend, bv1, thsad=th1)
fc1 = C(c, sup_rend, fv1, thsad=th1)
if radt >= 2:
bc2 = C(c, sup_rend, bv2, thsad=th2)
fc2 = C(c, sup_rend, fv2, thsad=th2)
if radt >= 3:
bc3 = C(c, sup_rend, bv3, thsad=th3)
fc3 = C(c, sup_rend, fv3, thsad=th3)

ic = core.std.Interleave([bc1, c, fc1]) if radt == 1 else \
core.std.Interleave([bc2, bc1, c, fc1, fc2]) if radt == 2 else \
core.std.Interleave([bc3, bc2, bc1, c, fc1, fc2, fc3])

output = core.tmedian.TemporalMedian(ic, radius=radt)
return output.std.SelectEvery(radt*2+1, radt) # Return middle frame

[StainlessS]

How bout I put link to it in 1st post.
I dont speak python/vapoursynth, so it means little to me and I dont really want to be tip-toe-ing around it, wondering why its there.

EDIT: Put link in first post.

[kedautinh12]

I think you need creat new post in Vapoursynth and add your script in that, MX

[coolgit]

Hey Stainless, s 3 ep17 war without end f 8013 to 8215 is weird. Had to do that from vob again using assume bff to get it right. Any idea why this happen in the first place? Had this in an earlier ep in s3, can't remember which now.

[StainlessS]

Is that the sequence starting with subs "Special guest star Michael O'hare".
Seems progressive in my VOB, no jumping with AssumeTFF or AssumeBFF. [EDIT: Assume??? then SeparateFields]

[coolgit]

No, sub produced by john copeland... Sinclair walking away to go indoors. I use assumetff but in some rare cases, need asssumebff.

[coolgit]

Stainless, can you check you vob filesize for season 5 ep 18. I got 38:42 length with teaser part at the beginning missing.

[StainlessS]

S05E18, Disk 5, ep Clip2, - "The Fall of Centauri Prime".
is 42:21
https://i.postimg.cc/18DMS14V/Untitled-00.jpg (https://postimg.cc/18DMS14V)

EDIT:
Stainless, can you check you vob filesize for season 5 ep 18. I got 38:42 with teaser part at the beginning missing.
I really dont know how to answer that, episodes are combined in DVD VOB so VOB size is all 4 episodes and who knows what else.
38:42 is a time, not file size and teaser is part of the clip.
OK, cutting out all of "teaser", and starting at first frame of B5 intro/credits with first frame showing very small but distinct "2258", [frame before is black]
leaves 58024 frames, 38:40.96 duration. [but it depends upon where you cut the black sequence - I cut on very first visible "2258" so you should get exactly the same if cut at that point]

EDIT: My season 5 is not boxed version as my other seasons, is single DVD case with multiple flip disk holders (6).

[coolgit]

Well it seems my dvd got a defect. Tried other software and can't read it. Tried vdub and as soon as i move to the first frame it crashes.

If you have time could you upload the missing vob clip up to the first keyframe of intro for me.

Cheers.

[StainlessS]

Check PM.

[StainlessS]

Any of you mask wizards out there improve on this ?


AviSourcE(".\somefile.avi")
ConvertToYV24
#ConvertBits(16)
Src=Last
Filtered=SpotLess(RadT=2)
YTH8 = 12
CTH8 = 12
CAMP = True
SHOW = True
SpotTest(Src,Filtered,YTH8,CTH8,CAMP,SHOW)
Return Last.ConvertToRGB32

/*
SpotTest(),
Purpose:
Spotless produces frame global changes [most pixels are changed], here we want to try alter only real spots where there is significant change in spotless Y [by YTh8]
and also that are near grey in source [by CTh8], these targeted pixels are then overlayed from Spotless result onto original src.

Src, Pre-Spotless source
Filtered, Spotless result
YTh8, threshold for Y in 8 bit range.
CTh8, threshold for Chroma in 8 bit range.
CAMP, If True amplify chroma MaskC when Show
Show, Show Masks. Mask is the final mask made from MaskY and MaskC.

*/
Function SpotTest(clip Src,clip Filtered,int "YTh8",int "Cth8",Bool "AmpC",Bool "Show") { # 8 -> 16 bit YUV444 only
YTh8 = Default(YTh8,12)
CTh8 = Default(CTh8,12)
AmpC = Default(AmpC,True)
Show = Default(Show,False)
try {bpc = Src.BitsPerComponent} catch(msg) {bpc=8}
try {Y444=Src.IsYV24||Src.Is444} catch(msg) {Y444=false}
Assert(Y444,"SpotTest: Must be YV24/YUV444")
YTh = BitLShift(YTh8,bpc-8)
CTh = BitLShift(YTh8,bpc-8)
MaskY = Mt_Lutxy(Src.ExtractY,Filtered.ExtractY,"x y - abs " + String(YTh) + " > range_max 0 ?") # "(abs(x-y) > YTh) ? range_max : 0"
MU = Mt_Lut(Src.ExtractU,"x range_half - abs") # "abs(x - range_half)"
MV = Mt_Lut(Src.ExtractV,"x range_half - abs") # "abs(x - range_half)"
MaskC = Mt_Lutxy(MU,MV,"x x * y y * + 0.5 ^") # "((x^2)+(y^2)) ^ 0.5" # 2D chroma (by Pythagoras)
Mask = Mt_Lutxy(MaskC,MaskY,"x " + string(CTh) + " < y 0 ?") # "(x < CTh) ? y : 0
Mask = Mask.mt_Expand.Blur(1.0)
MaskC = (!AmpC) ? MaskC : MaskC.mt_lut("x 8 *") # 2D chroma Amp'ed for view if AmpC
Overlay(Src,Filtered,Mask=Mask,Opacity=1.0)
CGREY = Mask.Mt_lut("range_half")
TOP=StackHorizontal(Src.Subtitle("Src"),Filtered.Subtitle("Filt"),YtoUV(CGrey,CGrey,MaskY).Subtitle("MaskY"))
BOT=StackHorizontal(Last.Subtitle("Result"),YtoUV(CGrey,CGrey,Mask).Subtitle("Mask"),YtoUV(CGrey,CGrey,MaskC).Subtitle((AmpC)?"MaskC Amp'ed":"MaskC"))
STK = StackVertical(TOP,BOT)
(Show) ? STK : Last
}


Hopefully reduces the number of changed pixels, and avoids arms and legs and balls and stuff going missing.

[Dogway]

It might be obvious but maybe looking into current algorithms (Wiki (https://en.wikipedia.org/wiki/Blob_detection#Spatio-temporal_blob_detectors)) can provide some insight. I plan to give it a stab in a few weeks.

[StainlessS]

Thanks doggie, looks like Gobble-De-Gook to me though.

Above script does pretty well with VOB where looks like original film stock had emulsion chipped off (or something),
white-ish dots every now and then.
[will undergo further processing and did not want to overdo it with spotless - RADT=2 was just to make it more difficult test (with repaired disappearing arm ),
RADT=1 is quite sufficient for my clip]

Just tried with dirty home movie where various color dirt, not so good.

Anyways, I think it does pretty much exactly as I want for current clip, and really quite simple.

EDIT: Here, two frames from very jerky InGoldie clip that you might remember [boy in purple throws ball in the air, Spotless removes it but we use original source pixels not spotless result].
[no white spots for demo here]

https://i.postimg.cc/rwX9npSB/PBDe-Spot-00.jpg (https://postimg.cc/8j4WsNX4)


https://i.postimg.cc/j5LgVFCm/PBDe-Spot-01.jpg (https://postimg.cc/KkhrBQPD)

[coolgit]

A new code....mmmm. The results looks good but shouldn't both source should have a 'spot' in it to ensure the code doesn't put it back in.

If it works it could replace spotlessF.

[StainlessS]

shouldn't both source should have a 'spot' in it to ensure the code doesn't put it back in.
No,
MaskY is white where change in pixel Y [for src->spotless] is > YTh, MaskC is amount Chroma in SOURCE pixels [*8 Amplified in images for view only],
will only overlay with Spotless pixels where MaskC is < CTh and MaskY = white.

Mask = Mt_Lutxy(MaskC,MaskY,"x " + string(CTh) + " < y 0 ?") # "(x < CTh) ? y : 0 # x is MaskC pixel and y = corresponding MaskY pixel


Its really a bit specific for white or black noise only. [and its where are white pixels in MASK that are overlaid from Spotless filtered onto source]
(we are selecting what we want rather than restoring what we dont want)
There are not usually that many white dots on final mask, probably down to all them there daisies.

[coolgit]

Inserted the code into Spotless and got it to work.

Does well in restoring hands (a common problem) but also does well in restoring all kinds of defects. It rarely removes anything.

What a pity.

[StainlessS]

Yeah, I processed the clip in question, removed a lot of emulsion chips, but not as many as I would like,
however the purpose was to do only very little denoise, and it did that as the purpose was to to avoid frame global changes.
Need to figure out at least one more conditional restriction so as to be able to relax YTh and CTh restriction a little.
I still want only a very light touch.
[The frames I was processing needed maybe up to about 3 small chips/dots removed on SOME frames only].

[coolgit]

My thinking is, fresh pot of coffee helped, do not restore if mask object(what you call the white blob) or any other means, doesn't exist in adjacent frames.

ns=no spot
s=spot
b=ball

frame 1 2 3
ns s ns
b b b

Spot in frame 2 removed and since nothing exist in either 1 or 3 then do not restore. If object like a ball exist in 1 or 3 then restore.
Spot are motionless and the ball isn't.

[StainlessS]

All you did is move the problem.
Spot are motionless and the ball isn't.
How do you tell if ball exists in frames 1,and 2, and 3, as you say, ball isnt motionless.
[And if MoComp could tell, then ball would not disappear in the first place via Spotless]
I'm less concerned with balls, more with hands and feet, and arms and legs (but balls would be nice too).
EDIT: Anyways, have not given up yet, still plugging away at it.

EDIT: What I got right now, take comments with pinch of salt, and havta figure out what its doin'.

#AviSource(".\10Bit_Standard8_Scan_16FPS.avi")
#AviSource(".\IG18.avi")
#AviSource(".\AMOLAD.demuxed.m2v.AVI")
AviSource("D:\DVD\PBDeSpot_1.avi")
ConvertToYV24
#ConvertToYUV444
#Return last.info
#ConvertBits(16)
Src=Last
Filtered=SpotLess(RadT=1)
YTH8 = 8
CTH8 = 12
YNOISETH8 = 2
SHOW = True
SpotTest(Src,Filtered,YTH8,CTH8,YNOISETH8,SHOW)
Return Last.ConvertToRGB32

/*
SpotTest(),
Purpose:
Spotless produces frame global changes [most pixels are changed], here we want to try alter only real spots where there is significant change in spotless Y [by YTh8]
and also that are near grey in source [by CTh8], these targeted pixels are then overlayed from Spotless result onto original src.

Src, Pre-Spotless source
Filtered, Spotless result
YTh8, threshold for Y in 8 bit range.
CTh8, threshold for Chroma in 8 bit range.
YNoiseTh8, ?????
Show, Show Masks.Mask is the final mask made from MaskY and MaskC.

*/
Function SpotTest(clip Src,clip Filtered,int "YTh8",int "Cth8",int "YNoiseTh8",Bool "Show") { # 8 -> 16 bit YUV444 only
YTh8 = Min(Max(Default(YTh8,12),0),255)
CTh8 = Min(Max(Default(CTh8,12),0),255)
YNoiseTh8 = Min(Max(Default(YNoiseTh8,8),0),255)
Show = Default(Show,False)
try {bpc = Src.BitsPerComponent} catch(msg) {bpc=8}
try {Y444=Src.IsYV24||Src.Is444} catch(msg) {Y444=false}
Assert(Y444,"SpotTest: Must be YV24/YUV444")
YTh = BitLShift(YTh8,bpc-8).String
CTh = BitLShift(CTh8,bpc-8).String
YNoiseTh = BitLShift(YNoiseTh8,bpc-8).String
SY = Src.ExtractY
FY = Filtered.ExtractY
PSY = SY.SelectEvery(1,-1)
NSY = SY.SelectEvery(1, 1)
MaskN = Mt_Lutxyz(PSY,SY,NSY,"x "+ YNoiseTh + " - y <= y z "+ YNoiseTh + " + <= & z "+ YNoiseTh + " - y <= y x "+ YNoiseTh + " + <= & | 0 range_max ?")
# ( ((x-YNoiseTh <= y) & (y <= z+YNoiseTh)) | ((z-YNoiseTh <= y) & (y <= x+YNoiseTh)) ) ? 0 : 255 ## Infix @ 8 bit
# ((YPrv-YNoiseTh <= YCur <= YNxt+YNoiseTh) || (YNxt-YNoiseTh <= YCur <= YPrv+YNoiseTh)) ? 0 : 255

MaskY = Mt_Lutxy(SY,FY,"x y - abs " + YTh + " > range_max 0 ?") # "(abs(x-y) > YTh) ? 255 : 0"
MU = Mt_Lut(Src.ExtractU,"x range_half - abs") # "abs(x - 128)"
MV = Mt_Lut(Src.ExtractV,"x range_half - abs") # "abs(x - 128)"
MaskC = Mt_Lutxy(MU,MV,"x 2 ^ y 2 ^ + .5 ^ round " + CTh + " <= range_max 0 ?") # "(round(((x^2)+(y^2))^0.5) <= CTh) ? 255 : 0" # 2D chroma distance from grey (by Pythagoras)
MaskN = MaskN.mt_Expand
MaskY = MaskY.mt_Expand
MaskC = MaskC.mt_Expand
MaskAnd = MaskC.Mt_Logic(MaskY,"and") #
MaskAnd = MaskAnd.Mt_Logic(MaskN,"and") #
MaskSelect = MaskAnd.mt_Expand.Blur(1.0)
#
Overlay(Src,Filtered,Mask=MaskSelect,Opacity=1.0)
CGREY = MaskSelect.Mt_lut("range_half")
SEP=$FF0000
HBAR=Src.BlankClip(length=1,Height=4,color=SEP)
LFT=StackVertical(Src.Subtitle("Src"),HBAR,Filtered.Subtitle("Filt"),HBAR,Last.Subtitle("Result"))
MID=StackVertical(YtoUV(CGrey,CGrey,MaskN).Subtitle("MaskN"),HBAR,YtoUV(CGrey,CGrey,MaskY).Subtitle("MaskY {abs(SrcY-FiltY) > YTh}"),HBAR,YtoUV(CGrey,CGrey,MaskSelect).Subtitle("MaskSelect=MaskAnd.mt_Expand.Blur(1.0) {Blk=Src used : Wht=Filt used}"))
RGT=StackVertical(YtoUV(CGrey,CGrey,MaskC).Subtitle("MaskC {Src Chroma distance from Grey <= Cth8"),HBAR,YtoUV(CGrey,CGrey,MaskAnd).Subtitle("MaskAnd {MaskN & MaskC & MaskY}"),HBAR,YtoUV(CGrey,CGrey,CGrey))
VBAR=LFT.BlankClip(length=1,width=4,color=SEP)
STK = StackHorizontal(LFT,VBAR,MID,VBAR,RGT)
(Show) ? STK : Last
}

EDIT: some cosmetic changes.
I'm gonna get some sleep soon.

MaskN (noiseMask) shows no noise = BLACK, noise = WHITE for current frame Cur yPixel where,
(PrvY - NoiseTh) <= CurY <= (NxtY + NoiseTh) OR (NxtY - NoiseTh) <= CurY <= (PrvY + NoiseTh) Then BLACK Else WHITE
If pixel lies between corresponding pixels in previous and next frames (with a little tolerance by NoiseTh), then not noise, else noise.
[ie No spot if BLACK, and MAYBE spot (or ball) if WHITE]

Currently, MaskC is the main problem [BLACK is colored ie dont use BALL/HAND/LEG from Filt clip, and WHITE if Copy from filt clip].
Detecting colored ball as BLACK, also tends to stop detect of what looks like white spots at high luma level (but with chroma distance above CTh).

[coolgit]

What I was thinking was the ball would exist in the previous or next frame, as in your example post 217, by searching a part of the frame (previous or next) for similar object, say 50 by 50 pixels search area (freedom to change the variables according to video), colour shouldn't be an issue. Where the ball disappeared would be the centre of 50 by 50. Surely this would narrow down the search and mistaken identifying of details by not searching the whole frame. If it was a spot then nothing would be found and therefore not restored. In your post 217 mask y frames made it clear where the ball was and is. Comparing the ball real colour from the source frames would be the final confirmation before restoring. Neither ball in Mask y and colour have to be 100% the same in both frames, perhaps 80% is enough. Obviously this would apply to hand and foot.

[StainlessS]

OK, then please supply script code to do that, and then we is cookin'.
[If it works as well as you say, then maybe can look forward to MvTools algo update, and then this script (or parts of it) may not be needed at all]

EDIT: At present, the function is only looking to target single pixels, [EDIT: blob-ish objects are also made up of single pixels] it is possible that it may extend to blob-ish objects,
but original requirement is for fast, simple, de-speckler, which causes little global frame changes, later other main denoiser may be also used.
Its definitely just experimental at the moment.

[StainlessS]

https://academic.microsoft.com/paper/2037299049/reference/search?q=Color%20space%20normalization%3A%20Enhancing%20the%20discriminating%20power%20of%20color%20spaces%20for%20face%20recognition&qe=Or(Id%253D2121647436%252CId%253D2163808566%252CId%253D2130660124%252CId%253D2914885528%252CId%253D2137659841%252CId%253D2106309274%252CId%253D3146003712%252CId%253D2994340921%252CId%253D2012352340%252CId%253D1591385104%252CId%253D2160835070%252CId%253D1761337995%252CId%253D2144119003%252CId%253D1974097586%252CId%253D2147933509%252CId%253D2096813265%252CId%253D2140907607%252CId%253D2112724657%252CId%253D1759219755%252CId%253D148999028%252CId%253D2156334937%252CId%253D2126687887%252CId%253D2010560725%252CId%253D2150892735%252CId%253D2043999711%252CId%253D2110249778%252CId%253D2113593239%252CId%253D2162557814%252CId%253D2143857284%252CId%253D2106887132%252CId%253D1968189504%252CId%253D2002552184%252CId%253D2511353375%252CId%253D2106822035%252CId%253D2134486056%252CId%253D1515707506%252CId%253D2122426897%252CId%253D2128293876%252CId%253D2028339207%252CId%253D1574019097%252CId%253D1997071866%252CId%253D2163626218%252CId%253D2111433705%252CId%253D2062199239%252CId%253D2136353882%252CId%253D2400676549%252CId%253D639253003)&f=&orderBy=0

Color space normalization: Enhancing the discriminating power of color spaces for face recognition
Abstract

This paper presents the concept of color space normalization (CSN) and two CSN techniques, i.e., the within-color-component normalization technique (CSN-I) and the across-color-component normalization technique (CSN-II), for enhancing the discriminating power of color spaces for face recognition. Different color spaces usually display different discriminating power, and our experiments on a large scale face recognition grand challenge (FRGC) problem reveal that the RGB and XYZ color spaces are weaker than the I"1I"2I"3, YUV, YIQ, and LSLM color spaces for face recognition. We therefore apply our CSN techniques to normalize the weak color spaces, such as the RGB and the XYZ color spaces, the three hybrid color spaces XGB, YRB and ZRG, and 10 randomly generated color spaces. Experiments using the most challenging FRGC version 2 Experiment 4 with 12,776 training images, 16,028 controlled target images, and 8,014 uncontrolled query images, show that the proposed CSN techniques can significantly and consistently improve the discriminating power of the weak color spaces. Specifically, the normalized RGB, XYZ, XGB, and ZRG color spaces are more effective than or as effective as the I"1I"2I"3, YUV, YIQ and LSLM color spaces for face recognition. The additional experiments using the AR database validate the generalization of the proposed CSN techniques. We finally explain why the CSN techniques can improve the recognition performance of color spaces from the color component correlation point of view.

Basically, Color Space Normalizing of YUV just involves forgetting all about the Y channel.
Used in Facial recognition, because skin tones have similar-ish U and V range values, no matter how light / dark the skin tone.

Some, above maybe of some use. [we are using only U and V in the chroma distance thingy].

EDIT: I did see a pdf saying similar to above for YUV but deleted it.
Maybe Color Space Normalizing of YUV or Facial recognition in YUV is specifically what we want, so above extract is maybe not the one we ideally want.

[coolgit]

Just chucking in ideas, was thinking of the top of my head. Obviously i don't know the full capabilities of avisynth.

[StainlessS]

Obviously i don't know the full capabilities of avisynth.
Obviously, Nobody knows the full capabilities of avisynth. :)

[coolgit]

Stainless - tried to message you but your inbox is full.

[StainlessS]

Coolgit, sorry, bit involved,
I is a bit busy,

[coolgit]

Remove line 306 downwards and insert this...

prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCT=src.SCSelect_HBD(next,prev,SpotLessT,show=false)

prev = SpotLessF.selectevery(1,-1)
next = SpotLessF.selectevery(1,1)
SpotLessSCF=src.SCSelect_HBD(next,prev,SpotLessF,show=false)

SpotLessSCF2P=SpotLess(SpotLessSCF,BlkSz=BlkSzF,OLap=OLapF,pel=1,Tm=False,Bblur=BblurF,ThSAD=3000,RadT=RadT) # Useful for many small defects per frame and not cleared on first pass.
# 2nd pass should clear up remaining defects and improve quality on 1st pass.
S1 = StackHorizontal(\
sub(src, "Source", txt_sz),\
sub(SpotLessT, "Spotless TM true", txt_sz),\
sub(SpotLessF, "Spotless TM false", txt_sz)\
)
S2 = StackHorizontal(\
sub(SpotLessSCT, "Spotless TM true and scene change", txt_sz),\
sub(SpotLessSCF, "Spotless TM false and scene change", txt_sz),\
sub(SpotLessSCF2P, "Spotless 2 pass TM false and scene change", txt_sz)\
)
output = StackVertical(S1,S2)

Return output # Comparison


#Return SpotLessSCT # Conversion
#Return SpotLessSCF
#Return SpotLessSCF2P
#Return SpotLessT
#Return SpotLessF

# Debugging to check scene change detection------------------------------------
prev = SpotLessT.selectevery(1,-1)
next = SpotLessT.selectevery(1,1)
SpotLessSCTDM=src.SCSelect_HBD(next,prev,SpotLessT,show=true)

SSCTD1 = StackHorizontal(\
sub(src.AddBorders(0,30,0,0), "Source", txt_sz),\
sub(SpotLessSCTDM.AddBorders(0,30,0,0), "Spotless and scene change with metrics", txt_sz)\
)
SSCTD2 = StackHorizontal(\
sub(SpotLessSCT.AddBorders(0,30,0,0), "Spotless and scene change without metrics", txt_sz),\
sub(BlankClip(src).AddBorders(0,30,0,0), "Blank", txt_sz)\
)
outputD = StackVertical(SSCTD1,SSCTD2)

#Return outputD # Comparison

Added two new choices.
1. Use SpotLessF with scene changes for clips with small defects only.
2. 2nd pass available to improve on 1st pass.

[StainlessS]

DoubleRate deinterlacer template, uses SpotLess. For DGIndex / DGIndexNV. [works with either, copy to their Templates Directory]

Need some tester to suggest optimal defaults, but pretty damn good as it is.
Uses Spotless(RadT=1) on separated Even/Odd fields, where removes spots and reduces any artifically added grain [added to hide how bad the clip is].

DespotDeinterlaceDbleRate.avs

### DgIndex / DgIndexNV, Template, DespotDeinterlaceDbleRate.avs

# Temp Pre-Processing step for Interlaced, with Spotless() on separated fields, prior to deinterlace.
# Spotless(RadT=1) nicely reduces any artificial grain as well as getting rid of single field/frame spots.
# We sort of prefer BOBMOD = 1, YadifMod2 deinterlace with little/no additional overprocessing at pre-proc stage. Although can BOBMOD=2, ie QTGMC.
# Output is YV24, save AVI from VDub2, with eg Fast Recompress, Codec UT_Video, YUV444 BT.601 VCM

VideoFileName = "__vid__"
AudioFileName = "__aud__"
AudioDelay = Value("__del__")

###############

vExt = GetFilenameExtension(VideoFileName)

If(vExt == ".dgi") { DGSource(VideoFileName) } # DGIndexNV
Else { Mpeg2Source(VideoFileName) } # DGIndex

AudioExt = GetFilenameExtension(AudioFileName)

# May prefer other audio source, but these usually work just fine.
Audio= (AudioExt==".ac3") ? NICAC3Source(AudioFileName,channels=2,DRC=0)
\ : (AudioExt==".mpa"||AudioExt=="mp1"||AudioExt==".mp2"||AudioExt==".mp3") ? NicMPG123Source(AudioFileName,Normalize=False)
\ : (AudioExt==".wav") ? RaWavSource(AudioFileName)
\ : (AudioExt==".dts") ? NicDTSSource(AudioFileName)
\ : (AudioExt==".w64") ? RaWavSource(AudioFileName,samplerate=6)
\ : 0

Assert(!isInt(Audio),"NO AUDIO")

(!isInt(Audio)) ? AudioDub(Audio).DelayAudio(AudioDelay).Trim(0,0) : NOP # Trim, chop/add audio to length

(!isInt(Audio) && AudioRate() <> 44100) ? ResampleAudio(44100) : NOP # If you want 44.1 KHz (I do)


####### CONFIG ##########

AssumeTFF # Whatever (check) : Almost always TFF for PAL DVD. Default field order is BFF.

ConvertToYV24(Interlaced=true)

FINAL = False # True, switch off SUBS, STACK, and Title

DBLRATE = true # Output rate Double input
RADT = 1 # Spotless() on separated fields only, before de-interlaced.
DEGRAIN_FIELDS=False # True, Degrain separated fields rather than deinterlaced frames (On one or other, but Not both). [done after Spotless when on separated fields].
BOBMOD=1 # 0=BOB, 1=YadifMod2, 2=QTGMC.

FRAMES = 0 # McDegrainSharp(frames=FRAMES), (0=Dont McDegrainSharp) # McDegrainSharp() on fields if DEGRAIN_FIELDS=true, else on frames [Never on both].
CSHARP = 0.03 # Very light McDegrainSharp sharpen arg



# For Spotless on separated fields.
ThSAD = 1000 # Ditto
ThSAD2 = 990 # Ditto
PEL = 2 # Ditto
CHROMA = True # Ditto
BlkSz = 16 # Ditto
TM = False # Ditto
DC = "RemoveGrain(1).Blur(0.3,0.3)" # DC = Detect Clip. Or Just "Blur(0.3)"

# Deinterlace mode

BOB_FUNC_0 = """Bob()""" # Cheap & cheerful Doublerate simple Bob.
BOB_FUNC_1 = """YadifMod2(Order=-1,Mode=1,edeint=NNedi3(Field=-2, nns=2))""" # Doublerate Yadifmod2 filter.
# # Field-2=double rate, internal parity value to start : Order-1=Internal Parity. Mode 1=double rate do spactial check (3=no spatial check).

BOB_FUNC_2 = """QTGMC(Preset="fast",NoisePreset="Fast",SourceMatch=2,Lossless=2,EZKeepGrain=0.4,Sharpness=0.1, tr2=0)""" # Doublerate QTGMC filter.

FUNC = Select(BOBMOD, BOB_FUNC_0, BOB_FUNC_1, BOB_FUNC_2)
FNAME = Select(BOBMOD,"BOB", "YADIF", "QTGMC")


PRE_FETCH=4 # 1 = No Prefetch, Else eg 4 = Prefetch(4) ie MT mode

### # Display stuff
SUBS = True
STACK = True # Show Src and Result
TITLE = true # Show Title Bar
TITNUM = True # Show Frame Number on Title Bar (Only if TITLE)

##### End Config ##########

Src=Last


PROCESS_FIELDS = (RADT>0 || (DEGRAIN_FIELDS && FRAMES>0)) # any Field pre-processing before deinterlacing ?

# DeSpot and/or McDeGrainSharp on fields
Sep = Src.SeparateFields
SEP_Even = Sep.SelectEven
SEP_Even = (RADT>0) ? SEP_Even.SpotLess(Radt=RADT,thsad=ThSAD,thsad2=ThSAD2,Pel=PEL,chroma=CHROMA,blksz=BLKSZ,tm=TM,glob=True,dc=SEP_Even.Eval(DC)) : SEP_Even
SEP_Even = (DEGRAIN_FIELDS && FRAMES>0) ? SEP_Even.McDeGrainSharp(frames=FRAMES,csharp=CSHARP) : SEP_Even
SEP_Odd = Sep.SelectOdd
SEP_Odd = (RADT>0) ? SEP_Odd.SpotLess(Radt=RADT,thsad=ThSAD,thsad2=ThSAD2,Pel=PEL,chroma=CHROMA,blksz=BLKSZ,tm=TM,glob=True,dc=SEP_Odd.Eval(DC)) : SEP_Odd
SEP_Odd = (DEGRAIN_FIELDS && FRAMES>0) ? SEP_Odd.McDeGrainSharp(frames=FRAMES,csharp=CSHARP) : SEP_Odd
PreProc = (PROCESS_FIELDS) ? Interleave(Sep_Even,Sep_Odd).Weave : Src

Deint = PreProc.Eval(FUNC) # Double Rate deinterlace
Deint = (!DEGRAIN_FIELDS && FRAMES>0) ? Deint.McDeGrainSharp(frames=FRAMES,csharp=CSHARP) : Deint
Result = (!DBLRATE) ? Deint.SelectEven : Deint

SUBS = (!FINAL&&SUBS)
TITLE = (!FINAL&&TITLE)
STACK = (!FINAL&&STACK)

SUBTXT = (!DBLRATE) ? FNAME+"_DEINTERLACED-EVEN" : FNAME+"_DblRate-DEINTERLACED"


RESULT = (SUBS) ? Result.Subtitle(SUBTXT,Align=5) : Result

SrcTest = (!DBLRATE) ? Src : Src.ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)

SrcOut = (!TITLE)?SrcTest:(!DBLRATE) ? Src.TSub("Source",TITNUM) : Src.TSub("Source DblRate via ChangeFPS, ie dupes",TITNUM).ChangeFPS(Src.FrameRateNumerator*2,Src.FrameRateDenominator)

RESULT = (!TITLE) ? RESULT :(!DBLRATE) ? Result.TSub("Result SingleRate",TITNUM) : Result.TSub("Result DblRate",TITNUM)

(STACK) ? Stackhorizontal(SrcOut, Result) : Result

(PRE_FETCH>1) ? Prefetch(PRE_FETCH) : NOP # I had problems using x64 and Prefetch [Access Violation], culprit was old IsCombedTIVTC, fixed via Pinterf version.

Return Last

#####################
# Stack Overhead Subtitle Text, with optional FrameNumber shown.
Function TSub(clip c,string Tit,Bool "ShowFrameNo",Int "Col"){
c.BlankClip(height=20,Color=Default(Col,0))
(Default(ShowFrameNo,False))?ScriptClip("""Subtitle(String(current_frame,"%.f] """+Tit+""""))"""):Trim(0,-1).Subtitle(Tit)
Return StackVertical(c).AudioDubEx(c)
}

Function GetFilenameExtension(String fn) {
fn=fn.RevStr i1=fn.FindStr("\") i2=fn.FindStr("/")
i=(i1==0) ? i2 : (i2==0) ? i1 : min(i1,i2)
fn=(i>0) ? Fn.LeftStr(i-1) : fn
i=fn.FindStr(".")
Return (i>0)?fn.LeftStr(i).RevStr:""
}


EDIT: Intended where entire input clip is known to be interlaced, not mix of interlaced and progressive.

Here a [vob source] frame picked out at nearly random, shows stacked with subs. [CLICK ME]
https://i.postimg.cc/N9XhSFjC/CL-A-01.jpg (https://postimg.cc/N9XhSFjC)
I did not look for frame with spots, but shows grain reduction.

EDIT:
Changed Frames=0 [MDegrainSharp off by default], and default CSHARP=0.03 [was 0.1]
Makes it a bit faster by default {less pre-processing}, and even milder sharpen if switched on.
https://i.postimg.cc/RqQpFyqD/Cold-Lazarus-Frames-0.jpg (https://postimg.cc/RqQpFyqD)

[johnmeyer]

StainlessS,

I just used (today!) Spotless on half an hour of 1956 NFL (American football) film. It nailed spots that the old RemoveDirtMC wouldn't touch.

I'm still looking for a way to add adjustments for spot size and color. These "spots" (they were mostly circles) were all white. At other times most spots are black. Adding this capability would add one other feature: ignore real objects, most of which are not near-black or near-white. I still lose a lot of shirt buttons as people move around. In this project, the ball often disappeared and I had to create a separate video that didn't have any dirt removal and cut to that when the ball disappeared. That is tedious work.

However, even as is, I'm finding it more useful than RemoveDirtMC for many projects.

So, thanks again for the excellent work!

Here is a before/after. All those white circles on the black uniforms on the left were moving around like a snowstorm from frame to frame, and it got 90% of them.
https://i.imgur.com/13zoiHx.png

[StainlessS]

Slight alterations to previously given DgIndex/DgIndexNV script template. [McDegrainSharp(Frames=0) by default, ie off, faster]

For Spot size, BlkSz maybe at least twice as wide/tall as spots. And also, ThSAD,ThSAD2 affects result. [ThSAD >= ThSAD2].

Also this lot from Docs [I doubt that I can easily make it less opaque].


ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle [EDIT: median] pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD, 0 < ThSAD2 < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.


Several guys earlier in thread did some mods where arms/legs were recovered, there are links to other scripts.

Also, Dogways posted here:- https://forum.doom9.org/showthread.php?p=1959548#post1959548
It might be obvious but maybe looking into current algorithms (Wiki) can provide some insight. I plan to give it a stab in a few weeks.
So maybe something in the pipeline one day.

[johnmeyer]

Thanks for those hints. That helps a lot.

[StainlessS]

SpotLess v1.07, see first post.

Repeat post here, and zip @ mediafire in sig below this post.

SpotLess.avsi v1.07


/*
SpotLess v1.07. temporal spot/noise remover, by StainlessS @ Doom9. https://forum.doom9.org/showthread.php?t=181777
Original idea from Didée post :- https://forum.doom9.org/showthread.php?p=1402690#post1402690
Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.

NOT FOR cartoon/anime, live video only, sorry.

v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
v1.02, Add some stuff.
v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.
v1.05, Additional checks on args.
v1.06, Glob Default true, Almost always better.
v1.07, bBlur Default changed from 0.0 OFF, to 0.3, slower but better for mc analysis. Maybe try 0.6 for HD to overcome HD 'grittiness'.
ThSAD default changed from 10,000 to 255 * (8/2 * 8/2), ie 1/4 of 8x8 block at 255, ie 1/4 BlkSz spots @ max SAD. Also ThSAD2 a bit less than ThSAD.
Added dc arg, detect clip.

SpotLess(clip c,int "RadT"=1,int "ThSAD"=255*(8/2 * 8/2),int "ThSAD2"=ThSAD-(8*8),int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,
\ bool "tm"=true,Bool "glob"=True,Float "bBlur"=0.3, clip "dc"=Undefined)

RadT, 1 or more, Default 1. Removes Spots on up to RadT [Temporal Radius] consecutive frames.
RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

ThSAD, Default 255 * (8/2 * 8/2) ie 4080 <ie 1/4 of 8x8 block at 255, ie 1/4 BlkSz spots @ max SAD>.
0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
in the result MCompensated frame].
Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
or some other mixture.
A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
middle pixel value.
I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
as we still had the temporal median filter to fall back on and pick the middle pixel value.

From mvtools2 docs:
ThSAD is SAD threshold for safe (dummy) compensation.
If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

ThSAD2, Default ThSAD-(8*8), 0 < ThSAD2 <= ThSAD < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
acquiring a SAD threshold linearly interpolated between the two.
From mvtools2 docs:
ThSAD2:
Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
of having too high a SAD threshold.

Pel, Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)

Chroma, Default True. MAnalyse chroma arg. If true, use chroma in block matching when creating vectors. Maybe use False if source B&W or color noise.

BlkSz, Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]

OLap, Default half BlkSz, Block overlap.

Tm, TrueMotion Default True. Some folk swear truemotion=false is better.

Glob, Default True (True v1.06, was same as Tm, true almost always better), Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable.

bBlur, Default 0.3. If used, Suggest about 0.3, where MAnalyse create vectors is performed on denoised (blurred) super clip
for better motion analysis. Maybe try 0.6 for HD clip to counteract HD 'grittiness'. bBlur ignored if dc clip specified,
If providing bBlur, then ALWAYS specify as named argument ie bBlur=bBlur, we will likely insert any additional MvTools args
before bBlur to keep them together.

dc, Default UnDefined. If dc Specified, then must be same size and colorspace as source clip.
If detection clip specified then bBlur prefilter arg is ignored, and analysis is performed on
dc.MSuper() clip [ instead of c.Blur(bBlur).MSuper() ]. Allows to provide your own prefiltered clip.
If providing dc clip, then ALWAYS specify as named argument ie dc=DC, we will likely insert any additional MvTools args
before bBlur to keep them together.

*/

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
myName = "SpotLess: "
RadT = Default(RadT,1) # Temporal radius. (MCompensate arg)
ThSAD = Default(ThSAD,255*(8/2 * 8/2)) # SAD threshold at radius 1. Here is max'ed out at 255 * 1/4 of 8*8 block <ie 4080> (MvTools Default 10,000 is nearly OFF, maybe way Too High).
# Maybe above 255*8/2*8/2 still a bit high, maybe 128*(8/2 * 8/2) Better <ie 2048>. For General noise perhaps somethimg like 24 * (8 * 8) <ie 1536>.
ThSAD2 = Default(ThSAD2,ThSAD-(8*8)) # SAD threshold at radius RadT(a bit less than ThSAD).
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
BlkSz = Default(BlkSz,8) # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better, esp for HD clips. Maybe also better where BIG noise.
OLap = Default(OLap, BlkSz/2) # Default half of BlkSz.
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
bBlur = Default(bblur,0.3) # Default 0.3. Suggest about 0.3 for better motion analysis, but a bit slower.
HasDC = dc.Defined # bblur ignored if HasDC, ie user already provided prefiltered clip.
Assert(1 <= RadT,myName + " 1 <= RadT")
Assert(0.0 <= bblur <= 1.58, myName + "0.0 <= bblur <= 1.58")
Assert(pel==1 || pel==2 || pel==4, myName + "pel==1 || pel==2 || pel==4")
pad = max(BlkSz,8)
sup = (HasDC ? dc : bBlur>0.0 ? c.blur(bBlur) : c ).MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2)
# Only 1 Level required where not MAnalyse-ing.
sup_rend = (HasDC||bBlur>0.0) ? c.MSuper(hpad=pad,vpad=pad,pel=pel, sharp=2,Levels=1) : sup
MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,chroma=Chroma,truemotion=Tm,global=Glob)
c.MCompensate(sup_rend, MultiVec, tr=RadT, thSad=ThSAD, thSad2=ThSAD2)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT) # Temporal median blur only [not spatial]
SelectEvery(RadT*2+1,RadT) # Return middle frame
}

[guest]

[QUOTE=StainlessS;1975475]SpotLess v1.07, see first post.

Repeat post here, and zip @ mediafire in sig below this post.

SpotLess.avsi v1.07

Can't find it :(

[kedautinh12]

[QUOTE=StainlessS;1975475]SpotLess v1.07, see first post.

Repeat post here, and zip @ mediafire in sig below this post.

SpotLess.avsi v1.07

Can't find it :(

Just copy script above

[guest]

[QUOTE=TDS;1975479]

Just copy script above

I did do that, but the other option is not there....that's all.

I had this working a while back, but something must have changed, as currently, I can't get it to work.

But it's a script I wouldn't use too often, anyway.

Thanks.

[StainlessS]

Sorry, and thanks guys, its on MediaFire now.

@ TDS,
maybe you missed out the final "}" at end of script, its easily done.

[Stereodude]

Sorry if this is a really dumb question, but is this mostly for removing spots like the examples posted and the name suggests, or can it also do the same sort of motion compensated general noise reduction as something like MCTD (https://forum.doom9.org/showthread.php?t=139766)?

I've started playing with SpotLess, but was curious what other's have found.

[kedautinh12]

Sorry if this is a really dumb question, but is this mostly for removing spots like the examples posted and the name suggests, or can it also do the same sort of motion compensated general noise reduction as something like MCTD (https://forum.doom9.org/showthread.php?t=139766)?

I've started playing with SpotLess, but was curious what other's have found.

Latest ver:
https://github.com/Asd-g/AviSynthPlus-Scripts/blob/master/MCTD_.avsi

[StainlessS]

At StereoDude,
Suggest try with about, RadT=1, and ThSAD = N * (8 * 8) for general denoise [where N about 6 ==>> 24].
Also, maybe something like "DC=Src.RemoveGrain(22).Blur(0.3)"

[Stereodude]

At StereoDude,
Suggest try with about, RadT=1, and ThSAD = N * (8 * 8) for general denoise [where N about 6 ==>> 24].
Also, maybe something like "DC=Src.RemoveGrain(22).Blur(0.3)"
Okay, I will give it a try. :thanks:

[guest]

Sorry, and thanks guys, its on MediaFire now.

@ TDS,
maybe you missed out the final "}" at end of script, its easily done.

Thanks SssS, but the } was there, so there must be something else missing.

LoadPlugin("%AVISYNTHPLUGINS%\MedianBlur2\MedianBlur2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
Import("%AVISYNTHPLUGINS%\SPOTLESS\Spotless.avs")
video=Spotless(video)

I know I had this working, a while ago :(

[StainlessS]

Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
With appropriate plugins, will be AVS+ colorspace and x64 compatible.
Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter.


Only those bolded above maybe necessary, I think.
Some modes of MvTools use fftw3.dll, but not usually neccesary, I think [dont think we use those modes].
Also, current AVS+ a good idea, but should work on other.
NOTE, should be recent Pinterf version dll's.

Apart from above, you dont give any indication of what went wrong, no error message or anything, so little chance of any further assistance.
Suggest try with no arguments at all, ie Spotless(SomeClip) to see if works.

[kedautinh12]

Only those bolded above maybe necessary, I think.
Some modes of MvTools use fftw3.dll, but not usually necesary, I think [dont think we use those modes].
Also, current AVS+ a good idea, but should work on other.
NOTE, should be recent Pinterf version dll's.

Apart from above, you dont give any indication of what went wrong, no error message or anything, so little chance of any further assistance.
Suggest try with no arguments at all, ie Spotless(SomeClip) to see if works.

Ripbot don't give any error message and he still use it :D

[StainlessS]

Thanks K,

then just load script into VDub2.
https://forum.doom9.org/showthread.php?t=172021

[guest]

Ripbot don't give any error message and he still use it :D

Come on K, I'm right here, you knew I'd read this :rolleyes:

So, I'd like to know how many of "you guy's" have even tried RipBot264 ??

And how many of you actually do a considerable amount of movie encoding ??

RipBot is the ONLY (free) app that can use remote PC's to assist with the encoding process, it's called "Distributed Encoding", and can use up to 16 PC's.

App's like MeGUI, VD2, Handbrake, Staxrip, etc, can only be used on the PC the app runs on.

I'm aware that RipBot has quite a few limitations, and not too many filtering options, but that's the way Atak_Snajpera has written it, and is VERY reluctant to change.....and his support for his app has declined a lot recently, and is a pretty well guarded code.

Fortunately, a guy called Pauly Dunne has incorporated a plethora of Dogway's SMDegrain based (and others) scripts & filter options, for RipBot, and provides regular updates to these and other important components of the app.

So don't knock it, 'til you try it, you might be surprised, and it would also be beneficial to the app if more "developers" like yourselves, supported it. ;)

Spread the word...

[kedautinh12]

Come on K, I'm right here, you knew I'd read this :rolleyes:

So, I'd like to know how many of "you guy's" have even tried RipBot264 ??

And how many of you actually do a considerable amount of movie encoding ??

RipBot is the ONLY (free) app that can use remote PC's to assist with the encoding process, it's called "Distributed Encoding", and can use up to 16 PC's.

App's like MeGUI, VD2, Handbrake, Staxrip, etc, can only be used on the PC the app runs on.

I'm aware that RipBot has quite a few limitations, and not too many filtering options, but that's the way Atak_Snajpera has written it, and is VERY reluctant to change.....and his support for his app has declined a lot recently, and is a pretty well guarded code.

Fortunately, a guy called Pauly Dunne has incorporated a plethora of Dogway's SMDegrain based (and others) scripts & filter options, for RipBot, and provides regular updates to these and other important components of the app.

So don't knock it, 'til you try it, you might be surprised, and it would also be beneficial to the app if more "developers" like yourselves, supported it. ;)

Spread the word...

If i change from Megui to other apps, i will change to Avspmod or selur's hybrid. And yeah, i still suprise although don't use it cause i don't know what errors when use with scripts and how to fix it :D

[guest]

If i change from Megui to other apps, i will change to Avspmod or selur's hybrid. And yeah, i still suprise although don't use it cause i don't know what errors when use with scripts and how to fix it :D

Well, you can always ask someone for help, like I do with you.

With your knowledge of MeGUI, etc, you could suss out script errors, whereas I'm not familiar with MeGUI or AvsPmod.

Hybrid is super complicated (to me anyway).

I would be interested for you to at least try RipBot, the Pauly Dunne build's, that is...

[kedautinh12]

Well, you can always ask someone for help, like I do with you.

With your knowledge of MeGUI, etc, you could suss out script errors, whereas I'm not familiar with MeGUI or AvsPmod.

Hybrid is super complicated (to me anyway).

I would be interested for you to at least try RipBot, the Pauly Dunne build's, that is...

It's hard when you and me use different ver of scripts and plugins

[guest]

It's hard when you and me use different ver of scripts and plugins

Maybe so, but there would be a way to get them working, I'm sure.

Maybe I need to spend a bit of time with MeGUI.

[StainlessS]

RipBot is the ONLY (free) app that can use remote PC's to assist with the encoding process, it's called "Distributed Encoding", and can use up to 16 PC's.

Ripbot don't give any error message and he still use it

@TDS,
Cant you remote connect via VNC [I used to use TightVNC on Windows or TigerVNC by same guys on Linux - have not used either for a couple of years].
You have to install VDub2 on problem machine.
Could connect to problem machine and run test script in VDub2. [and get sensible error message]

Taking wild guesses at problem is not the way to go, and a waste of time asking for assistance on D9 without any clues at all.
(maybe RipBot sends error message to log file - no error report at all would seem like a serious flaw in RipBot)


Virtual Network Computing (VNC) is a graphical desktop-sharing system that uses the Remote Frame Buffer protocol (RFB) to remotely control another computer. It transmits the keyboard and mouse input from one computer to another, relaying the graphical-screen updates, over a network.[1]

VNC is platform-independent – there are clients and servers for many GUI-based operating systems and for Java. Multiple clients may connect to a VNC server at the same time. Popular uses for this technology include remote technical support and accessing files on one's work computer from one's home computer, or vice versa.

Virtual Network Computing[wikipedia]:- https://en.wikipedia.org/wiki/Virtual_Network_Computing
You install the VNC server on problem/remote machine, and VNC client on your local machine, and then access problem/remote machine
as if you are sitting in front of it.

https://www.tightvnc.com/

EDIT: I did try othe VNC flavours, eg RealVNC and UltraVNC but neither worked well for me, Tight/Tiger VNC worked flawlessly.

[guest]

You have to install VDub2 on problem machine.
Could connect to problem machine and run test script in VDub2. [and get sensible error message]

Taking wild guesses at problem is not the way to go, and a waste of time asking for assistance on D9 without any clues at all.
(maybe RipBot sends error message to log file - no error report at all would seem like a serious flaw in RipBot)

As far as I know the only "checking" feature in RipBot is if you run a "new" script thru AVSMeter...sometimes it will hint as where the error might be, but rarely give details.

Doubt there is a log, hiding somewhere.

And like you suggest, I should install VD2, and get use to it :)

Cheers

[StainlessS]

and get use to it
Not much to get used to, just install and load script, it works or it dont, easy peasy.

[guest]

Not much to get used to, just install and load script, it works or it dont, easy peasy.

I just got the latest available build, so I will see if it IS, easy peasy :p

[guest]

If i change from Megui to other apps, i will change to Avspmod or selur's hybrid. And yeah, i still suprise although don't use it cause i don't know what errors when use with scripts and how to fix it :D

I just got the latest available build of MeGUI....

It looks very old, going on the versions & dates of all it's dependencies...

Is it x86 or X64 ???

[kedautinh12]

I just got the latest available build of MeGUI....

It looks very old, going on the versions & dates of all it's dependencies...

Is it x86 or X64 ???

I use x64 ver and i updated all plugins manually

[guest]

I use x64 ver and i updated all plugins manually

Yep, found & downloaded x64 :)

Yes, I will have to figure out how to do that...

[coolgit]

After a few days of experiments, I thought I will post my findings.

Sometimes I used
Despot(mthres=16,p1=24,p2=12,pwidth=4,pheight=4,ranked=true,sign=-1,maxpts=0,p1percent=10,show=0)
to remove small white star(ish) shaped defects. It worked to a point and fast too. However when you have outdoors scenes with trees, bushes, grass, etc, then the small white areas in the trees, for example, get removed and the trees will now flicker insanely when played back. Spotless doesn't have this problem because it use previous and next frames in its calculation.

The experiment is to get Spotless to do what DeSpot does.

The answer is
SpotLess(BlkSz=8, OLap=8/2, pel=1, Tm=False, Bblur=0.0, ThSAD=200, RadT=1)
Yes ThSAD is 200 and it is not a typing error ok. You can go as low as 150 for Despot(mthres=16,p1=24,p2=12,pwidth=3,pheight=3,ranked=true,sign=-1,maxpts=0,p1percent=10,show=0)

StainLesss, is it possible to add a 'sign' syntax like Despot has. Currently DeSpot 'sign' syntax is

sign = 0 - any spots and outliers (default)
sign = 1 - only black (dark) spots and outliers
sign = -1 - only white (light) spots and outliers
sign = 2 - only black (dark) spots, any outliers
sign = -2 - only white (light) spots, any outliers

Just curious if white or black only options, and white and black option, totalling 3 options. The rest of the colours are ignored since I have never come across coloured star(ish) shaped defects that flickers.

[StainlessS]

I would not really like to include that in SpotLess() function,
best I can suggest is make some kind of mask by combining source chroma close to 128, and luma hi/low, and use only results
of Spotless where masks comply with required.

There are others here way better than I at masking stuff, perhaps someone else can assist on that,
I also got my hands quite full with non AVS related stuff.

[coolgit]

What about doing luma or chroma only.
The thing is when i used Spotless to mimic DeSpot, it takes 30 mins extra to compress for an hour video clip.
I just thought if it does luma only then it would be faster.

[ChaosKing]

Try Chroma = false
Have you tried spotless with Prefetch() ?

[coolgit]

Prefetch yes as always.

Not sure what to do with chroma=false?

I am looking for a filter that separates luma and chroma, then i could use Spotless on luma, then merge new luma with chroma.

[coolgit]

Source whatever...

Luma=SpotLess(src, BlkSz=8, OLap=8/2, pel=1, Tm=False, Bblur=0.0, ThSAD=900, RadT=1)
MergeLuma(src, Luma)
StackHorizontal(src,last)

Tried the above and seems to work. ThSAD needs to be 800 to 1000 to do the same as SpotLess(BlkSz=8, OLap=8/2, pel=1, Tm=False, Bblur=0.0, ThSAD=200, RadT=1).

Testing to see if it will work faster.

EDIT: Much later... The above took 2 minutes longer than SpotLess(BlkSz=8, OLap=8/2, pel=1, Tm=False, Bblur=0.0, ThSAD=200, RadT=1). Crap idea.:o:o

[Mounir]

converttoyv12(matrix="rec601",interlaced=true) #
separatefields()
input=last
U = UToY().SpotLess() #
V=VToY().SpotLess() #
YToUV(U,V,last)
weave()

[coolgit]

converttoyv12(matrix="rec601",interlaced=true) #
separatefields()
input=last
U = UToY().SpotLess() #
V=VToY().SpotLess() #
YToUV(U,V,last)
weave()

Tried it and didn't work.

[coolgit]

Converting image to greyscale doesn't reduce Spotless processing time. DeSpot works differently and i am not sure why.
Despot took 1 minute extra to compressed whereas SpotLess took 30 to 32 minutes extra.

[Lathe]

I would not really like to include that in SpotLess() function,
best I can suggest is make some kind of mask by combining source chroma close to 128, and luma hi/low, and use only results
of Spotless where masks comply with required.

There are others here way better than I at masking stuff, perhaps someone else can assist on that,
I also got my hands quite full with non AVS related stuff.

Hi there! I'm kind of a newbie and I'm trying to employ your SpotLess script with 32bit Avisynth 2.6. I have your dll and the two support dll's that I believe were mentioned awhile back (mvtools2 and MedianBlur2) I tried running this script and I got a really weird error both when trying to preview with AVSMod and actually run the x264 CMD line. This is the script I was trying to run:

DirectShowSource("GC.mkv")
Import("C:\Program Files (x86)\AviSynth\plugins\Spotless.avsi")
LoadPlugin("C:\Program Files (x86)\AviSynth\plugins\MSharpen.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth\plugins\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth\plugins\MedianBlur2.dll")
assumeTFF()
TFM().TDecimate()
crop(24,12,-16,-20) #crop the crud off
Spotless()
MSharpen()
#greyscale() #convert to B&W
addborders(20,16,20,16) #pad to 720x480

This was the error the preview showed:

Traceback (most recent call last):
File "avsp.pyo", line 9119, in OnMenuVideoPlay
File "avsp.pyo", line 14672, in PlayPauseVideo
File "avsp.pyo", line 13855, in ShowVideoFrame
File "avisynth.pyo", line 462, in GetFrame
WindowsError: exception: access violation reading 0xFFFFFFFF

I really appreciate any help because I would LOVE to try your script if I can get it to run, thanks!

[Sharc]

Lathe, please don't hijack and litter this thread as well. The problem is not with Spotless(), it's a problem between your chair and your keyboard. See all your other posts here and over at the VideoHelp forum. You call yourself a newbie with 1077 posts at doom9 in almost 20 years? Start with learning the basics of video and AviSynth. Sorry to be a bit harsh :devil:

[Arx1meD]

Lathe, try opening your video with L-SMASH-Works. DirectShowSource is often buggy.

[coolgit]

crop(24,12,-16,-20) #crop the crud off
Spotless()


Spotless()
crop(24,12,-16,-20) #crop the crud off

[Bluedraft]

Hi, it's possible to remove the vertical lines and spots from this video using spotless? I tried with the default setting and using RadT=2 but could not eliminate them

sample
https://drive.google.com/file/d/1ybNWoe2Td_EvRTOdvQxcfMEoG54MpUjf/view?usp=drive_link

[johnmeyer]

Probably not vertical lines.

Couldn't access your Google Drive. You need to make the link public.

[Bluedraft]

Sample
https://mega.nz/file/Y8oQyK5a#Y14zTh259pxceNMqcLQx5dvMN_afcdWe-Ywnb1Are6Q

[Selur]

" I tried with the default setting and using RadT=2 "
For me, SpotLess seems to remove those scratches (works in Avisynth and Vapoursynth)
Script used:
ClearAutoloadDirs()
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\LSMASHSource.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mvtools2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\MedianBlur2.dll")
LoadPlugin("F:\Hybrid\64bit\Avisynth\avisynthPlugins\RgTools.dll")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\mtmodes.avsi")
Import("F:\Hybrid\64bit\Avisynth\avisynthPlugins\SpotLess.avs")
# loading source: C:\Users\Selur\Desktop\sample.mkv
# color sampling YV12@8, matrix: bt709, scantyp: progressive, luminance scale: limited
LWLibavVideoSource("C:\Users\Selur\Desktop\sample.mkv",cache=false,format="YUV420P8", prefer_hw=0,repeat=true)
# current resolution: 1920x1080
# despotting using SpotLess
SpotLess()
# Converting from 8 to 10bit for encoder
ConvertBits(10)
# setting output fps to 24.000fps
AssumeFPS(24,1)
PreFetch(16)
# output: color sampling YV12@10, matrix: bt709, scantyp: progressive, luminance scale: limited
return last
https://i.ibb.co/KLMQpfs/Spot-Less-01.png (https://ibb.co/LJLqXcg)
https://i.ibb.co/101m8TT/Spot-Less-02.png (https://ibb.co/0nTGhrr)
https://i.ibb.co/8KKG8TH/Spot-Less-03.png (https://ibb.co/JBB8k09)
https://i.ibb.co/JthGmtw/Spot-Less-04.png (https://ibb.co/vqtTVq6)

[Bluedraft]

Wonderful! I'll try it! Thanks a lot Selur!

[Bluedraft]

You seems to be using spotless by default, and looking at the pictures you upload it it works perfectly, I'm doing the same but from staxrip and using FFVideoSource to open the file instead LSMASHSource (because otherwise I get error) and it's like spotless was not being applied, the scratches still there.


SetFilterMTMode("DEFAULT_MT_MODE", 2)
tcFile = "%temp_file%_timestamps.txt" # timestamps file path
Exist(tcFile) ? FFVideoSource("%source_file%", cachefile="%source_temp_file%.ffindex", timecodes=tcFile) : FFVideoSource("%source_file%", cachefile="%source_temp_file%.ffindex")
LoadPlugin("C:\Users\Bluedraft\Downloads\Portables\StaxRip 2.12\Apps\Plugins\AVS\MVTools2\mvtools2.dll")
LoadPlugin("C:\Users\Bluedraft\Downloads\Portables\StaxRip 2.12\Apps\Plugins\AVS\MedianBlur2\MedianBlur2.dll")
LoadPlugin("C:\Users\Bluedraft\Downloads\Portables\StaxRip 2.12\Apps\Plugins\AVS\RgTools\RgTools.dll")
Import("C:\Users\Bluedraft\Downloads\Portables\StaxRip 2.12\Apps\Plugins\AVS\Spotless\Spotless.avsi")
Spotless()

PreFetch(16)


What I'm doing wrong?

[Selur]

No, clue, seems fine to me, but I have never used StaxRip.
SpotLess script I use: https://pastebin.com/2UDMcJw5

[coolgit]

This will do the whole video.

src=FFVideoSource("your dir\sample.mkv")

v1=Trim(src,0,394).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=3000, RadT=1).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=3000, RadT=1)
v2=Trim(src,394,405).FrameArea(600, 0, 700, 1080, "SpotLess(BlkSz=8, OLap=8/2, pel=2, Tm=true, Bblur=0.0, ThSAD=10000, RadT=3).SpotLess(BlkSz=8, OLap=8/2, pel=2, Tm=true, Bblur=0.0, ThSAD=10000, RadT=3)").
\ SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=3000, RadT=1).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=3000, RadT=1)
v3=Trim(src,406,457).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=3000, RadT=1).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=3000, RadT=1)
v4=Trim(src,457,540).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=700, RadT=1)
v5=Trim(src,541,543).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=3000, RadT=1)
v6=Trim(src,544,559).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=500, RadT=1)
v7=Trim(src,560,980).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=2000, RadT=1).SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=False, Bblur=0.0, ThSAD=2000, RadT=1)

alignedsplice(v1,v2,v3,v4,v5,v6,v7).Deleteframe(394,457)

return last

Using Vdub2, open script editor, needs both spotless.avsi and framearea.avsi in avisynth plugins folder.

[Bluedraft]

Thanks coolgit! I'll try it :)

[Bluedraft]

@coolgit RadT=3 certainly completely remove the scratches but it removes also the fine grain and it blur the picture

[coolgit]

RadT=3 only works in the specified framearea and not the whole picture. What i did was removed the defects. After that it is your choice. You can use sharpening and addgrainc or whatever your intention is.

[DTL]

You can now replace MedianBlurTemporal to vsTTempSmooth with pmode=1 from releases by Asd-g - https://github.com/Asd-g/AviSynth-vsTTempSmooth/releases/tag/1.2.6 (for tr >1 (2) ?)

It looks equal to median and with AVX2 and AVX512 run faster with tr>1 (2 ?). Also it supports 'very large' tr same as MDegrainN of 128. Also it have easy controls for YUV planes processing so allow to process chroma only for VHS de-rainbow (example at https://forum.videohelp.com/threads/414289-Flying-erase-head-noise#post2732760 ). Also it has some simple IIR optional additional features.

To be more user-friendly for processing chroma only it can be added script function params like 'process_chroma= true/false' and process_luma=true/false' . With decoding to params of vsTTempSmooth:

process_luma=true : y=3 (if false y=2)
process_chroma=true: u=3 v=3 (if false u=2 v=2).

Or for tr>1 it may be switched to vsTTempSmooth processing. Currently MediaBlurTemporal runs faster only with tr=1 (special case of 3 numbers median ?).

Also vsTTempSmooth has thresholding processing and allows the users to remove only differences below y,u,v, threshold. Though as typical large defects are much larger in amplitude in comparison with noise/grain it does not allow to keep small amplitude grain. But this can be easily inverted for experiments - and it will remove only sample differences above threshold (so will keep low amplitude noise and grain but the distortions of this amplitude also).

[LouieChuckyMerry]

StainlessS: Just another thanks for SpotLess; it works wonders (but not wonders of the universe, unfortunately). :thanks:

[LouieChuckyMerry]

After playing with SpotLess a bit more, I've noticed that it doesn't use 100% of my cpu as a same script without it would (AviSynth+ 3.73, x64); this is the case if I use it as either a separate call or a prefilter in SMDegrain. Is there any way to make SpotLess use more cpu? Thanks for any help.

[LouieChuckyMerry]

Hello, I hope all's well with all :) . I'm re(rerere...)encoding some of my favorite older movies, taking advantage of a better understanding of SMDegrain and the addition of SpotLess. So far the results are mostly stellar; however, I discovered an issue with Barry Lyndon and was looking for some settings advice. Barry Lyndon was filmed entirely with natural light; sunshine and candles only. The outdoor scenes with vast skies look amazing; the indoor scenes, unfortunately, have a problem. It's as if SpotLess can't always keep up with the flickering flames of the candles and produces a poor result. This test clip (https://www.mediafire.com/file/gwfqt6h4p0gkl9k/TestClip%2528Candle%2529%255BBarryLyndon%255D.7z/file) is a perfect example; SpotLess with default settings resulted in a video where the flame becomes obviously blocky-pixelated when the flickering is most extreme. Thanks for any help. (Aside: any suggestions for a video player that can process scenes like this without any screen tearing?)

[johnmeyer]

The clip looks fine to me. Is this the original, or is this the clip after you have applied Spotless? If it is the original, I'm not sure what defect you are trying to fix.

[LouieChuckyMerry]

It's the original clip.

[DTL]

SMDegrain is a linear temporal denoise via linear average (mean). Spotless is temporal-median filter (motion-compensated via mvtools) mostly to fix rare defects. You can try this as some denoise engine but not expect much. It is not linear and only selection-type engine so can not calculate smooth averages and at the fast action with bad motion compensation will produce large 'temporal aliasing' defects.

[LouieChuckyMerry]

SMDegrain is a linear temporal denoise via linear average (mean). Spotless is temporal-median filter (motion-compensated via mvtools) mostly to fix rare defects. You can try this as some denoise engine but not expect much. It is not linear and only selection-type engine so can not calculate smooth averages and at the fast action with bad motion compensation will produce large 'temporal aliasing' defects.

Thank you for the explanation. I've found SpotLess works wonders on older movies, but the candle-thing is a first. Would you know a change in the settings to fix this problem? Here's a clip post-SpotLess (https://www.mediafire.com/file/q69onz6vcmpl0y8/TestClip(CandleAfterSpotLess)[BarryLyndon].7z/file).

[DTL]

It is a nice warm analog scene - but fast flickering candle creates too much temporal aliasing with 24fps film. And simple enough motion compensation engine can not track such fast changes nicely. So simply do not apply non-linear temporal median to such footage. Use linear MDegrain - it will protect from such bad samples selection also it is block-based with smoothed overlap so will save you from badly cut areas with sharp edges.

Or you can try to redesign Spotless to use vsTTempSmooth (pmode=1) with non-full thresholds for output samples (this feature not present in TemporalMedian or you can use some external comparison and selection engine to 'limit' possible sample value changes). This will make defects at least lower while still providing some denoise activity.

[johnmeyer]

That flickering is going to be impossible to deal with because, in the middle of the clip, the flame is pulsing up and down exactly in sync with the frame rate. Thus, it is large in one frame, small in the next frame, large in the frame after that, and so on. Motion estimation software will not be able to track that and so I am not surprised that Spotless broke down in those scenes.

Since the original clip you provided looks so good, I would just turn off any processing on those scenes.

Actually, given how good that clip looks, if the rest of the movie looks like this, what improvement are you hoping to achieve?

[LouieChuckyMerry]

It is a nice warm analog scene - but fast flickering candle creates too much temporal aliasing with 24fps film. And simple enough motion compensation engine can not track such fast changes nicely. So simply do not apply non-linear temporal median to such footage. Use linear MDegrain - it will protect from such bad samples selection also it is block-based with smoothed overlap so will save you from badly cut areas with sharp edges.

Or you can try to redesign Spotless to use vsTTempSmooth (pmode=1) with non-full thresholds for output samples (this feature not present in TemporalMedian or you can use some external comparison and selection engine to 'limit' possible sample value changes). This will make defects at least lower while still providing some denoise activity.

Thank you for your reply; I understand it enough to know that I'm incapable of your suggestion (but really do appreciate your thought).


That flickering is going to be impossible to deal with because, in the middle of the clip, the flame is pulsing up and down exactly in sync with the frame rate. Thus, it is large in one frame, small in the next frame, large in the frame after that, and so on. Motion estimation software will not be able to track that and so I am not surprised that Spotless broke down in those scenes.

Since the original clip you provided looks so good, I would just turn off any processing on those scenes.

Actually, given how good that clip looks, if the rest of the movie looks like this, what improvement are you hoping to achieve?

The outdoor scenes are very, very noisy, given it's a 1975 movie filmed entirely with natural light.

[johnmeyer]

I saw the movie in the theaters in 1975 and it was indeed somewhat grainy. Kubrick worked with just-invented super-fast lenses in order to film the indoor scenes with just the candle light and even with that aid obviously had to use high-speed film. I'm sure he used similar stock even when outdoors, in order to match.

I haven't seen your transfer so maybe it is horrible. The clip you posted looked very good.

[LouieChuckyMerry]

It's the Criterion Blu-ray; here's a test clip of a typical outdoor scene (https://www.mediafire.com/file/obmdptasetvuq9y/TestClip(Outside)[BarryLyndon].7z/file). Some people seem to love grain; I find it a waste of bitrate as well as visual pollution. To each their own ;-) .

[Selur]

@LouieChuckyMerry: SpotLess, to me, seems like the wrong filter for such light grain&co,.... https://imgsli.com/Mjc1NDk3/0/3

[LouieChuckyMerry]

@LouieChuckyMerry: SpotLess, to me, seems like the wrong filter for such light grain&co,.... https://imgsli.com/Mjc1NDk3/0/3

I'm using SpotLess, in addition to my usual SMDegrain-FastLineDarken-F3KDB call, because it did such a nice job on my The Bad News Bears {1976} (https://forum.videohelp.com/threads/402938-How-To-Better-The-Bad-News-Bears%7B1976%7D-Blu-ray-Video) Blu-ray that I thought I'd re-attack some of my other favorite pre-digital film older movies that are extremely noisy, like Barry Lyndon, Solyaris, The Wild Bunch, etc.. After numerous tests comparing the results of

ConvertBits(Bits=16)
SMDegrain(TR=5,ThSAD=1300,RefineMotion=True,Plane=0,Chroma=False)
FastLineDarkenPlus()
Neo_F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)

to

ConvertBits(Bits=16)
SMDegrain(Prefilter=SpotLess(),TR=5,ThSAD=1300,RefineMotion=True,Plane=0,Chroma=False)
FastLineDarkenPlus()
Neo_F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)

to

ConvertBits(Bits=16)
SpotLess()
SMDegrain(TR=5,ThSAD=1300,RefineMotion=True,Plane=0,Chroma=False)
FastLineDarkenPlus()
Neo_F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)

I settled on the third option because it not only had the lowest bitrate output of the CRF20.0 tests--to my mind that means less bitrate wasted on grain, or am I wrong?--but also looked the best to my eyes. I'm definitely willing to try something new, however, and wondered if you'd please explain what "CAS" and "DPIRdeblock" are in the cool split-picture-with-drop-down-menus link. Thanks.

Edit: TemporalDegrain2 produces really really nice results with default settings.

EditEdit: Found CAS...

EditEditEdit: CAS is very nice...

[DTL]

As AVS environment looks like abandonware in mid-202x - you can try simple Topaz Denoise (with several different models and simple options). It is not as slow as new mvtools2 modes (AreaMode and other hacks) and can use GPU and provide some good looking image from film-grained sources. Yes - it also can lost some details but much simple to use for non-advanced user in AVS scripting.

[LouieChuckyMerry]

Please, does anyone have any information on DPIRdeblock? I was easily able to find TemporalDegrain2 and CAS, but have had no luck finding DPIRdeblock. Thanks for any help.

[coolgit]

THis ??

https://github.com/cszn/DPIR

https://lvsfunc.encode.moe/en/latest/submodules/deblock.html

Both vapoursynth i think

[LouieChuckyMerry]

THis ??

https://github.com/cszn/DPIR

https://lvsfunc.encode.moe/en/latest/submodules/deblock.html

Both vapoursynth i think

Thanks, coolgit, the first link, for DPIR, seems to be what I was looking for. Unfortunately you're right and it's for Vapoursynth. Do you happen to know of a similar filter-plugin for AviSynth?

[StvG]

DPIR can be used with avs-mlrt - https://github.com/Asd-g/avs-mlrt/tree/main/scripts

[coolgit]

Thanks, coolgit, the first link, for DPIR, seems to be what I was looking for. Unfortunately you're right and it's for Vapoursynth. Do you happen to know of a similar filter-plugin for AviSynth?

You can use staxrip then put the dpir in vapoursynth in plugin folder. Should work.

[LouieChuckyMerry]

DPIR can be used with avs-mlrt - https://github.com/Asd-g/avs-mlrt/tree/main/scripts

Thanks for that, StvG. After some effort, I'm to the point where the script

ConvertToPlanarRGB()
ConvertBits(Bits=32)
mlrt_DPIR(Model=3)

run with VirtualDub2 gives me the error "There is no function named 'mlrt_ncnn'.". After more effort, I realized that your link only has the sources, but I can't build the mlrt_ncnn.dll (or mlrt_ort.dll or mlrt_ov.dll) because I'm running Win 7. Any idea where I can find these .dll's (I've searched many ways without success...)?

[Emulgator]

These are found in Asd-g repos.
https://github.com/Asd-g/avs-mlrt/releases/tag/r63
Ah, just saw it, in #304 StvG mentioned it already...

[StvG]

Thanks for that, StvG. After some effort, I'm to the point where the script

ConvertToPlanarRGB()
ConvertBits(Bits=32)
mlrt_DPIR(Model=3)

run with VirtualDub2 gives me the error "There is no function named 'mlrt_ncnn'.". After more effort, I realized that your link only has the sources, but I can't build the mlrt_ncnn.dll (or mlrt_ort.dll or mlrt_ov.dll) because I'm running Win 7. Any idea where I can find these .dll's (I've searched many ways without success...)?

For e.g., the requirements of mlrt_DPIR (https://github.com/Asd-g/avs-mlrt/blob/main/scripts/mlrt_DPIR.avsi#L5-L7):
- mlrt_common.avsi (https://github.com/Asd-g/avs-mlrt/tree/main/scripts)
- avs-mlrt - this is a group of ML (machine learning) filters for avs. It contains mlrt_ncnn, mlrt_ov and mlrt_ort. Info here (https://github.com/Asd-g/avs-mlrt?tab=readme-ov-file#filters). They can be downloaded from here (https://github.com/Asd-g/avs-mlrt/releases). Make sure to read README of every filter because some of them have additional requirements.

[LouieChuckyMerry]

StvG: Thanks--again--for your help; and, thanks for giving me just enough information to not only make progress but actually have to apply myself to do so. I struggle with GitHub because it's in a foreign language, at least to me. Would you be willing to post a default mlrt_DPIR call so I can have a better idea of what I'm missing? I don't speak computer, but often I'm able to reverse engineer something. Thanks.

[StvG]

StvG: Thanks--again--for your help; and, thanks for giving me just enough information to not only make progress but actually have to apply myself to do so. I struggle with GitHub because it's in a foreign language, at least to me. Would you be willing to post a default mlrt_DPIR call so I can have a better idea of what I'm missing? I don't speak computer, but often I'm able to reverse engineer something. Thanks.

To download mlrt_DPIR.avsi click on the download button - https://i.ibb.co/Y0PSxfJ/Untitled1.png
To download mlrt_common.avsi click on the download button - https://i.ibb.co/NxmZvjK/Untitled2.png
To download mlrt_ncnn.dll (requires Vulkan device):
- click on the Releases button - https://i.ibb.co/W3q3Xtj/Untitled3.png
- the latest release is on top of the page - https://i.ibb.co/hZNWrnm/Untitled4.png
- scroll down to the Assets and download mlrt_ncnn-1.0.3.7z - https://i.ibb.co/rptDGR8/Untitled5.png
- above Assets there is "models can be downloaded from here." - https://i.ibb.co/gSgTXmr/Untitled6.png
- click on "here" and the scroll to the Assets and download models.7z - https://i.ibb.co/CtFX3h2/Untitled7.png
Put mlrt_DPIR.avsi, mlrt_common.avsi, mlrt_ncnn.dll and the folder "models" in your plugins folder.
Open the avs previewer (AvsPmod/VirtualDub...) and write:
BlankClip(pixel_type="y32")
mlrt_DPIR()

[LouieChuckyMerry]

To download mlrt_DPIR.avsi click on the download button - https://i.ibb.co/Y0PSxfJ/Untitled1.png

I already did this.

To download mlrt_common.avsi click on the download button - https://i.ibb.co/NxmZvjK/Untitled2.png

I already did this.

To download mlrt_ncnn.dll (requires Vulkan device):
- click on the Releases button - https://i.ibb.co/W3q3Xtj/Untitled3.png
- the latest release is on top of the page - https://i.ibb.co/hZNWrnm/Untitled4.png
- scroll down to the Assets and download mlrt_ncnn-1.0.3.7z - https://i.ibb.co/rptDGR8/Untitled5.png
- above Assets there is "models can be downloaded from here." - https://i.ibb.co/gSgTXmr/Untitled6.png
- click on "here" and the scroll to the Assets and download models.7z - https://i.ibb.co/CtFX3h2/Untitled7.png

I already did all this, but I think that my GPU(s) are too old to support Vulkan (which would explain the error message "Avisynth open failure: mlrt_ncnn: get_gpu_device failed".

Put mlrt_DPIR.avsi, mlrt_common.avsi, mlrt_ncnn.dll and the folder "models" in your plugins folder.

I already did this.

I wish I could afford a new computer, ha ha. Any thoughts on a standard AviSynth plugin-filter that would be similar to mlrt_DPIR?

[coolgit]

If your GPU too old then try staxrip.

[LouieChuckyMerry]

If your GPU too old then try staxrip.

I use MeGUI and VirtualDub2; why do you suggest StaxRip?

[coolgit]

I use MeGUI and VirtualDub2; why do you suggest StaxRip?

Staxrip can do both avisynth and vapoursynth. Just an idea.

[LouieChuckyMerry]

Ahhh, right, I remember your earlier post. Thanks for the thought, I'll see if I can sort it out :) .

[LouieChuckyMerry]

You can use staxrip then put the dpir in vapoursynth in plugin folder. Should work.

Would you be willing to explain this in more detail, please?

[coolgit]

Would you be willing to explain this in more detail, please?

StaxRip\Settings\Plugins\VapourSynth

Create folder dpir and put dpir files in folder

[StvG]

I already did all this, but I think that my GPU(s) are too old to support Vulkan (which would explain the error message "Avisynth open failure: mlrt_ncnn: get_gpu_device failed".

Then try mlrt_ov (https://github.com/Asd-g/avs-mlrt/tree/main/mlrt_ov) (make sure read the info about runtimes).
You can also try mlrt_ort (https://github.com/Asd-g/avs-mlrt/tree/main/mlrt_ort) (it has CPU and CUDA support).

BlankClip(pixel_type="y32")
mlrt_DPIR(backend="ov") # or mlrt_DPIR(backend="ort")

[LouieChuckyMerry]

StaxRip\Settings\Plugins\VapourSynth

Create folder dpir and put dpir files in folder

Thank you.

Then try mlrt_ov (https://github.com/Asd-g/avs-mlrt/tree/main/mlrt_ov) (make sure read the info about runtimes).
You can also try mlrt_ort (https://github.com/Asd-g/avs-mlrt/tree/main/mlrt_ort) (it has CPU and CUDA support).

BlankClip(pixel_type="y32")
mlrt_DPIR(backend="ov") # or mlrt_DPIR(backend="ort")

Thank you. I seem to be making progress, but I'm still not quite there. After putting the open_vino.dll, the tbb12.dll, and onnxruntime.dll in my Plugins folder, and all the other openvino_.dlls. and the "gna".dll in my "SysWOW64" folder I'm down to a single VirtualDub2 error message:

VirtualDub2: VirtualDub64.exe - Entry Point Not Found
The procedure entry point GetSystemTimePreciseAsFileTime could not be located in the dynamic link library KERNEL32.dll

before the AviSynth error message:

Avisynth open failure: mlrt_ov: [IE Exception] ReadNetwork(): [ NETWORK_NOT_READ ]: Unable to read the model. Please check if the model format is supported and model is correct.

[StvG]

Thank you. I seem to be making progress, but I'm still not quite there. After putting the open_vino.dll, the tbb12.dll, and onnxruntime.dll in my Plugins folder, and all the other openvino_.dlls. and the "gna".dll in my "SysWOW64" folder I'm down to a single VirtualDub2 error message:

These files must be in System32 not in SysWOW64.

[LouieChuckyMerry]

I wondered about that, so I tried that some time after my post (I forgot to edit it, sorry), but the error messages are the same. Ahhh, the script is

SetFilterMTMode("Default_MT_Mode",2)
BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend="ov")
PreFetch(3

[StvG]

I wondered about that, so I tried that some time after my post (I forgot to edit it, sorry), but the error messages are the same. Ahhh, the script is

SetFilterMTMode("Default_MT_Mode",2)
BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend="ov")
PreFetch(3

Do you have folder "models" next to mlrt_ov.dll? https://i.ibb.co/CsGKkDj/Untitled1.png

[kedautinh12]

64 bit plugins must be in the System32 folder and 32 bit plugins must be in the SysWOW64 folder. Don't ask me why, ask Microsoft :D

[LouieChuckyMerry]

Do you have folder "models" next to mlrt_ov.dll? https://i.ibb.co/CsGKkDj/Untitled1.png

Yes, the "models" folder (with the subfolder "dpir") is in my Plugins folder. For whatever reason, when I upgraded to AviSynth+ a couple years ago, the plugins in my "AviSynth 2.5/Plugins" folder still worked, so I never put any plugins in the (then) new folder "AviSynth+/plugins64" or "AviSynth+/plugins64+" folders.

So, I just tried copying all my plugins (and the "model" folder) to both of those folders, but the error messages are the same. I'm not sure what's wrong, but why did adding the gna.dll and all the openvino_"whatever".dlls to PATH make VirtualDub2 throw an error message?

Ahhh, I just tested a script I know works with VirtualDub2, and after the initial "KERNEL32.dll" error message, it seems to load fine...

64 bit plugins must be in the System32 folder and 32 bit plugins must be in the SysWOW64 folder. Don't ask me why, ask Microsoft :D

I'd certainly like a bit of time alone with Bill Gates, ha ha.

[StvG]

Avisynth open failure: mlrt_ov: [IE Exception] ReadNetwork(): [ NETWORK_NOT_READ ]: Unable to read the model. Please check if the model format is supported and model is correct.

Ok, I reproduced the error when I remove openvino_onnx_frontend.dll. Make sure you have this file in the path (in your case System32).

Yes, the "models" folder (with the subfolder "dpir") is in my Plugins folder. For whatever reason, when I upgraded to AviSynth+ a couple years ago, the plugins in my "AviSynth 2.5/Plugins" folder still worked, so I never put any plugins in the (then) new folder "AviSynth+/plugins64" or "AviSynth+/plugins64+" folders.

avs-mlrt is 64-bit only - meaning the avs 64-bit chain must be used (64-bit previewer, 64-bit plugins, plugins64/plugins64+ auto loading folders, System32 is part of the 64-bit system path).

So mlrt_ov.dll 1.0.2 must be in plugins64 or plugins64+ (unless is loaded manually with LoadPlugin). The files (all of them) from openvino_dll.7z if placed in system path, they must be in System32.
Also don't mix openvino_dll.7z from one release version with mlrt_ov from other release version (no the case here but mentioning it just for info).

Edit: Moderator can move the posts from #301 (including) in own thread because are off topic.

[LouieChuckyMerry]

Ok, I reproduced the error when I remove openvino_onnx_frontend.dll. Make sure you have this file in the path (in your case System32).

I triple-checked; openvino_onnx_frontend.dll is in my C:/Windows/System32 folder (along with all the other openvino_.dlls {including openvino.dll}, the gna.dll, the onnxruntime.dll, and the tbb12.dll).


avs-mlrt is 64-bit only - meaning the avs 64-bit chain must be used (64-bit previewer, 64-bit plugins, plugins64/plugins64+ auto loading folders, System32 is part of the 64-bit system path).

So mlrt_ov.dll 1.0.2 must be in plugins64 or plugins64+ (unless is loaded manually with LoadPlugin).

I'm loading it manually, which has changed things (see below).


The files (all of them) from openvino_dll.7z if placed in system path, they must be in System32.

They're all there; triple-checked.


Edit: Moderator can move the posts from #301 (including) in own thread because are off topic.

Thank you for remembering this; I'm sorry I forgot.


After manually loading the mlrt_ov.dll, I still receive the VD2 KERNEL32.dll error message--maybe not necessarily related to mlrt since it appears for scripts that didn't have an error message before beginning the current process--but after acknowledging the error message VD2 actually opens, but the screen looks like an LSD flashback with purples and greens that look more like a spectrum meter of some kind.

[Emulgator]

Repeated this here: VD2's (build 44282) DIB decoder indeed chokes on Y32.
I have the script extended with
BlankClip(Pixel_Type="Y32").Subtitle("This is Reptttuunnioern !", size=30, align=5)
and get rendered a pink/green LSB Plane while AvsPmod Preview behaves correct.

If the script is extended with
ConvertBits(10) at the end: VD's DIB decoder sees [0][0][0][0], the result is rendered top-down, and Cyan instead of white.
ConvertBits(8) at the end: VD's DIB decoder sees Y800, this is rendered as expected.

(Note to self: Maybe I should move on to AvsPmod and finally get behind making my own encoder presets there.
Tools -> Script Encoder (CLI)
Here is one for the now ubiquituous ProRes HQ apch 10bit
Add these two lines into ffmpeg.presets
[ProRes HQ apch 10bit]
ffmpeg.exe -i "$video_input" -c:v prores -profile:v 3 -vf "format=yuv422p, setsar=sar=$par_x/$par_y" -c:a pcm_s16le "$video_output".mov

[StvG]

I triple-checked; openvino_onnx_frontend.dll is in my C:/Windows/System32 folder (along with all the other openvino_.dlls {including openvino.dll}, the gna.dll, the onnxruntime.dll, and the tbb12.dll).

Do you pass the CPU system requirements from here (https://docs.openvino.ai/2023.3/system_requirements.html)?
If you pass them (the requirements) download Dependencies_x64_Release.zip (https://github.com/lucasg/Dependencies/releases), open DependenciesGui.exe, drag and drop openvino_onnx_frontend.dll and share a screenshot of the window.

[Guest]

I would appreciate some advice on trying to remove the "spots" & artifacts with this video.
https://www.mediafire.com/file/3nww1h98klm6o8s/PP+sample.mkv/file 142Mb
Regards

[LouieChuckyMerry]

Emulgator: I'll try to decipher your post this weekend :) .

StvG: I've an Intel Core i5-3320M CPU @ 2.60GHz, which I somehow doubt is 33rd generation...

[StvG]

Emulgator: I'll try to decipher your post this weekend :) .

StvG: I've an Intel Core i5-3320M CPU @ 2.60GHz, which I somehow doubt is 33rd generation...

Skylake (https://en.wikipedia.org/wiki/Skylake_(microarchitecture)) is the minimum CPU that is officially supported.

Maybe you can have luck with mlrt_ort (https://github.com/Asd-g/avs-mlrt/tree/main/mlrt_ort) (mlrt_DPIR(Backend="ort")).

[LouieChuckyMerry]

StvG: Many thanks for your continued help. I'm trying to use mlrt_ort, but after trying to understand the readme

For mlrt_ort(provider="cpu") - only onnxruntime_dll.7z is needed.
For mlrt_ort(provider="dml") - onnxruntime_dll.7z and dml_dll.7z are needed.
For mlrt_ort(provider="cuda") - onnxruntime_dll.7z and cuda_dll.7z are needed.
For all providers (mlrt_ort(provider="cpu/dml/cuda")) - onnxruntime_dll.7z, dml_dll.7z and cuda_dll.7z are needed.

(honestly, that's confusing, but all the .dlls from the three .7z files are in the mlrt_ort_rt folder) I'm having trouble with my script. I'm manually loading mlrt_ort.dll, and after the usual VD2 KERNEL32.dll error message, the script

BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend="ort")

results in the error message

Avisynth open failure:
mlrt_ort: failed loading "Path"/.../mlrt_ort_rt/onnxruntime.dll (the "mlrt_ort_rt" folder is in the same folder as the mlrt_ort.dll).

and, the script

BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend="ort"(Provider="cpu"))

results in the error message

Avisynth open failure:
Script error: 'string' cannot be called. Give me a function!

It's probably a simple solution, I just can't figure it out.

[anton_foy]

BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend="ort", Provider="cpu")
Should it be like this?

EDIT: misread and edited.

[StvG]

(honestly, that's confusing, but all the .dlls from the three .7z files are in the mlrt_ort_rt folder) I'm having trouble with my script. I'm manually loading mlrt_ort.dll, and after the usual VD2 KERNEL32.dll error message, the script

BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend="ort")

results in the error message

Download Dependencies_x64_Release.zip (https://github.com/lucasg/Dependencies/releases), open DependenciesGui.exe, drag and drop onnxruntime.dll and share a screenshot of the window. Here my output - https://i.ibb.co/NKdGd6W/Untitled1.png



and, the script

BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend="ort"(Provider="cpu"))

results in the error message



It's probably a simple solution, I just can't figure it out.

Backend (https://github.com/Asd-g/avs-mlrt/blob/main/scripts/mlrt_DPIR.avsi#L39-L42) parameter must be either string or array.
The correct syntax is:
BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend=["ort", """provider="cpu" """])

Btw CPU is the default provider for mlrt_ort.

Edit: You can try mlrt_ov 1.0.1 (https://github.com/Asd-g/avs-mlrt/releases/tag/r36) (make sure to download the corresponding openvino_dll.7z too). It uses older OpenVINO version than the latest mlrt_ov so it could work on your CPU.

[LouieChuckyMerry]

Download Dependencies_x64_Release.zip (https://github.com/lucasg/Dependencies/releases), open DependenciesGui.exe, drag and drop onnxruntime.dll and share a screenshot of the window. Here my output - https://i.ibb.co/NKdGd6W/Untitled1.png

Here's my output (https://www.mediafire.com/view/oalpao2tmf2gx8t/Dependencies%255Bonnxruntime.dll%255D.JPG/file).


Backend (https://github.com/Asd-g/avs-mlrt/blob/main/scripts/mlrt_DPIR.avsi#L39-L42) parameter must be either string or array.
The correct syntax is:
BlankClip(Pixel_Type="Y32")
mlrt_DPIR(Backend=["ort", """provider="cpu" """])

Btw CPU is the default provider for mlrt_ort.

Oooweeee!


Edit: You can try mlrt_ov 1.0.1 (https://github.com/Asd-g/avs-mlrt/releases/tag/r36) (make sure to download the corresponding openvino_dll.7z too). It uses older OpenVINO version than the latest mlrt_ov so it could work on your CPU.

Thanks for the idea; I'll try it if the latest version is proved to be impossible.

[StvG]

Here's my output (https://www.mediafire.com/view/oalpao2tmf2gx8t/Dependencies%255Bonnxruntime.dll%255D.JPG/file).

What Windows do you use?

[LouieChuckyMerry]

I'm sorry, Windows 7 SP1. Ahhh, and I learned that my kernel32.dll x64 is from 2020, so I found the latest but need to figure out how to replace it (Unlocker show a couple dozens processes; seems very popular).

[StvG]

I'm sorry, Windows 7 SP1. Ahhh, and I learned that my kernel32.dll x64 is from 2020, so I found the latest but need to figure out how to replace it (Unlocker show a couple dozens processes; seems very popular).

Don't bother. Min Windows 10 is required.

[LouieChuckyMerry]

D'oh! And thank you very much for the effort, StvG, I really appreciate it. Two questions:

1) Any idea why I'm now receiving the KERNEL32.dll error in VD2, even with scripts that I know are (well, were) OK before I started playing with mlrt?

2) Any suggestion for a good DeBlock plugin-filter for AviSynth?

EDIT: And thanks for showing me Dependencies, it seems more straightforward to me than Dependency Walker.

[Boulder]

2) Any suggestion for a good DeBlock plugin-filter for AviSynth?


Dogway has a collection of deblockers in his script package. A simple (and fast) Deblock_QED might already be enough for most purposes.
https://forum.doom9.org/showthread.php?t=182881

[StvG]

D'oh! And thank you very much for the effort, StvG, I really appreciate it. Two questions:

1) Any idea why I'm now receiving the KERNEL32.dll error in VD2, even with scripts that I know are (well, were) OK before I started playing with mlrt?

2) Any suggestion for a good DeBlock plugin-filter for AviSynth?

EDIT: And thanks for showing me Dependencies, it seems more straightforward to me than Dependency Walker.

1) Delete mlrt_ov.dll and it dependencies (all openvino runtimes). Delete mlrt_ort.dll and it dependencies (all onnxruntime files, directml eventually...).

2) The @Boulder suggestion (Deblock_QED) is popular and good. In addition to his suggestion about the Dogway collection there is a section External plugins (http://avisynth.nl/index.php/External_filters)->Deblocking (http://avisynth.nl/index.php/External_filters#Deblocking) on the front page of the wiki (http://avisynth.nl). Test the more popular filters and decide which fits best in your case.

[LouieChuckyMerry]

Dogway has a collection of deblockers in his script package. A simple (and fast) Deblock_QED might already be enough for most purposes.
https://forum.doom9.org/showthread.php?t=182881

Thanks, I'll try to get it working ASAP.


1) Delete mlrt_ov.dll and it dependencies (all openvino runtimes). Delete mlrt_ort.dll and it dependencies (all onnxruntime files, directml eventually...).

Thanks, that did the trick.


2) The @Boulder suggestion (Deblock_QED) is popular and good. In addition to his suggestion about the Dogway collection there is a section External plugins (http://avisynth.nl/index.php/External_filters)->Deblocking (http://avisynth.nl/index.php/External_filters#Deblocking) on the front page of the wiki (http://avisynth.nl). Test the more popular filters and decide which fits best in your case.

Thanks for the suggestions, and thanks again for all your help with mlrt :) .

[Guest]

I thought I'd bump this up, as I haven't had any response, it's all been for Louie....Another W7 user....OMG.

I would appreciate some advice on trying to remove the "spots" & artifacts with this video.
https://www.mediafire.com/file/hz00084l90t6p6b/spots.7z/file 142Mb
Regards
___________

[LouieChuckyMerry]

FTLOY: Here's my result (https://www.mediafire.com/file/mwz57dzabogukyv/SpotsTemp[CRF20.0].7z/file) with the following script at CRF20.0.

ConvertBits(Bits=16)
Deblock_QED()
SpotLess()
TemporalDegrain2()
CAS()
Neo_F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)

Any fine-tuning I leave to the experts ;-) .

[Guest]

FTLOY: Here's my result (https://www.mediafire.com/file/mwz57dzabogukyv/SpotsTemp[CRF20.0].7z/file) with the following script at CRF20.0.

ConvertBits(Bits=16)
Deblock_QED()
SpotLess()
TemporalDegrain2()
CAS()
Neo_F3KDB(Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)

Any fine-tuning I leave to the experts ;-) .

Wow, that looks pretty damn good to my old eyes, thank you so much for taking the time to do that for me :)

There must be some dependencies to support that script ???

I can't get the same result you did, what app did you use ??

My scripts need to look like this for RipBot264, I've probably missed something, I haven't done much with scripts, for a while :(

SetCacheMode()
SetMemoryMax(20480)
LoadPlugin("%AVISYNTHPLUGINS%\PD_TOOLS\DCTFilter\DCTFilter.dll")
LoadPlugin("%AVISYNTHPLUGINS%\PD_TOOLS\Deblock\Deblock-x64.dll")
LoadPlugin("%AVISYNTHPLUGINS%\masktools\masktools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\CAS\CAS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\PD_TOOLS\neo-f3kdb\neo-f3kdb.dll")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\Deblock_QED\Deblock_QED_MT2Mod.avs")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\SPOTLESS\Spotless newest.avs")
Import("%AVISYNTHPLUGINS%\PD_TOOLS\TemporalDegrain\TemporalDegrain2.avs")
video=ConvertBits(video,16)
video=Neo_F3KDB(video,Y=100,Cb=100,Cr=100,GrainY=0,GrainC=0)

[LouieChuckyMerry]

FTLOY: I use portable x64 MeGUI with the latest AviSynth+ (which autoloads most plugins-filters). As for dependencies, I use a search engine with "avisynth+ whateverplugin-filter" then head to the wiki or github page and reverse engineer it after reading the page (wiki) or readme (github) to learn the required dependencies, and if necessary I post here and plea for help from the geniuses. I'm a plodder, not a savant. Probably not the most efficient method, but it works for my limited knowledge of AviSynth.

[Guest]

FTLOY: I use portable x64 MeGUI with the latest AviSynth+ (which autoloads most plugins-filters). As for dependencies, I use a search engine with "avisynth+ whateverplugin-filter" then head to the wiki or github page and reverse engineer it after reading the page (wiki) or readme (github) to learn the required dependencies, and if necessary I post here and plea for help from the geniuses. I'm a plodder, not a savant. Probably not the most efficient method, but it works for my limited knowledge of AviSynth.

Fair enough, whatever works for you that's good enough.

That script I posted actually works, but it doesn't get rid of the little black artifacts like your sample clip did, so not really sure, maybe I need to experiment with your app.
:thanks:

[LouieChuckyMerry]

The little black dots are what SpotLess removes. Maybe try the script

SpotLess()

and check the results, then dig deeper.

[Guest]

The little black dots are what SpotLess removes. Maybe try the script

SpotLess()

and check the results, then dig deeper.

Thanks Louie, just running SpotLess(video) made all the difference, and default settings seems to be good enough. :)

[Bluedraft]

Hello, I'm trying to use spotless to clean some scenes but using it by default it ends affecting the image in some parts (it generate distortion in the image), I tried playing a little with some settings but couldn't get any better (probably because I not understand very well what I'm doing). I uploaded a sample of the scene, any help will be greatly appreciated.

https://drive.google.com/file/d/1FQvveK0k9CdU9sf1FzyLwGiqcQQK8D4U/view?usp=sharing

[StainlessS]

If distortion is around the edges, try adding padding of maybe 8 or 16 around distorted edges (before call to Spotless).
Padding()
http://forum.doom9.org/showthread.php?p=1596804#post1596804
Then chop (crop) it off afterwards.

[Bluedraft]

Hello StainlessS, the distortion appears throughout the whole image, see

Before Spotless
https://i.ibb.co/pBsSpkQT/1.png (https://ibb.co/pBsSpkQT)

After Spoless
https://i.ibb.co/prXSMZkP/2.png (https://ibb.co/prXSMZkP)

[StainlessS]

Try take a look at JohnMeyer post #20 for tips.

I think that there was some kind of mod of Spotless which tried to restore some removed spots, I've spent about an hour looking for it but cant find it.
Maybe somebody knows what I was looking for.

[Selur]

Maybe somebody knows what I was looking for.
You probably looking for: https://forum.doom9.org/showthread.php?t=182831 (SpotLess + DeltaRestore)

Cu Selur

Ps.: DeSpot, with tweaked settings, might be the better filter for such a source.

[StainlessS]

Thanks Selur, that looks to be what I was looking for.
Have added links at head of Spotless thread to that thread + SpotRemover
[film transfer] Remove dirt but keep grain? :- https://forum.doom9.org/showthread.php?t=182831
SpotRemover - clean video from spots:- https://forum.doom9.org/showthread.php?t=182407

[coolgit]

Hello, I'm trying to use spotless to clean some scenes but using it by default it ends affecting the image in some parts (it generate distortion in the image), I tried playing a little with some settings but couldn't get any better (probably because I not understand very well what I'm doing). I uploaded a sample of the scene, any help will be greatly appreciated.

https://drive.google.com/file/d/1FQvveK0k9CdU9sf1FzyLwGiqcQQK8D4U/view?usp=sharing

This is my standard to go for removing defects and keeping the quality intact. ThSAD should go no higher than 2000. Any more the image gets distorted. In some case i run spotless twice which is better than increasing ThSAD.

SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=false, Bblur=0.0, ThSAD=1000, RadT=1)

[rgr]

This is my standard to go for removing defects and keeping the quality intact. ThSAD should go no higher than 2000. Any more the image gets distorted. In some case i run spotless twice which is better than increasing ThSAD.

SpotLess(BlkSz=12, OLap=8/2, pel=2, Tm=false, Bblur=0.0, ThSAD=1000, RadT=1)

Whether on default or these settings -- I see that it can't handle flashes (left - org, right - SpotLess).

But in other cases -- pretty good.
https://gcdnb.pbrd.co/images/2lTDElnKRYhA.png?o=1

[coolgit]

Flashes are rare and better to copy the original flash frames back in. In fact flashes creates havoc for many filters.

[johnmeyer]

Because flashes are so bad for other filters, if I have a film or video which has them, I have a simple script which compares the current frame to both the adjacent frames using a YDiff comparison. This blows up in only one direction at scene changes, but when there is a bright flash, like a photographer's flash, it will replace that frame with either a duplicate of the previous frame or use motion interpolation to synthesize a new frame.

Finding individual "bad" frames in video; save frame number; or repair (https://forum.doom9.org/showthread.php?t=174104)

[rgr]

Because flashes are so bad for other filters, if I have a film or video which has them, I have a simple script which compares the current frame to both the adjacent frames

Yes, I know about that filter. But I want to keep the flashes.

[StainlessS]

Yes, I know about that filter. But I want to keep the flashes.

You can 'pluck' out your flash frame/s using FrameSel(),
process with whatever temporal filter (Spotless), then replace the plucked out frames
from whence they came [ FrameRep() ]

FrameSel:- https://forum.doom9.org/showthread.php?t=167971

EDIT: From the FrameSel() thread:

Post #2 of 4.

2) FrameRep

FrameRep(Clip c,Clip c2 int F1, ... , int Fn, string 'scmd',string 'cmd', bool 'show', bool 'ver',bool "reject",bool "debug")

Video:- All Valid colorspaces

Audio:- Returns source clip c audio.

Args identical to FrameSel, with exception of the compulsory 2nd clip arg which is a clip of frames to replace into the 1st source clip,
and also does not have an Ordered argument, as it is ALWAYS Ordered, and FrameRep, does not have the Extract arg.

See FrameSel() description for other arguments.

FrameRep() is complimentary to FrameSel, whereas FrameSel extracts frames, FrameRep replaces or reseats them (from the c2 clip) back from
whence they came (into the c clip) and using the EXACT same arguments and commands (whether directly supplied or in string command or
file command). If you wish to extract some frames and later replace back into the source clip, you MUST use the default Ordered=True in
FrameSel or the frames may not be replaced back into the positions they originally came from.

The number of Unique Selected frames in the combined frames commands MUST EQUAL the number of frames in the c2 clip (after taking Reject
into account), or it WILL throw an error, (will not know where to put the remainder or frame shortage). This is the reason you should
use the same frame commands in both FrameSel and FrameRep so that there will be an identical number of unique frames and Ordered
sequential. If Debug is set true, it will show a little info in DebugView (Google), eg number of unique frames in a command set.

# ----------------

Here, Example to do some simple editing on selected frames, try changing REJECT to True.

Colorbars.Trim(0,-10).ShowFrameNumber() # 10 frames
sCmd="2;5;6" # Frames in SCmd string
REJECT=False # False=modify SCmd frames : True=modify frames NOT in SCmd
SELECTED=FrameSel(scmd=sCMD,ordered=TRUE,reject=REJECT,debug=true) # We are going to use FrameRep, must use ORDERED=True (OR REJECT=True)
SELECTED=Invert(SELECTED) # Do some editing
FrameRep(Last,SELECTED,scmd=sCMD,reject=REJECT,debug=true) # Put edited frames back into original Last clip (audio from Last)
return Last

# ----------------

Example use when NOT selecting/replacing in exact same position:
Lets say you have a clip with a bad frame before each scene change and you want to replace these bad frames with the frame immediately
before it.

C=Colorbars().Trim(0,-50).ShowFrameNumber() # 50 frames (0 -> 49)
BADFRAMES = "10;20;30;40" # 4 bad frames in source clip
PREBAD = " 9;19;29;39" # 4 frames immediately before bad frames
REPAIR=FrameSel(C,scmd=PREBAD) # Get the 4 frames we will duplicate
Return FrameRep(C,REPAIR,scmd=BADFRAMES) # fix bad frames with REPAIR clip from PREBAD frames

Will produce a duplicate at frames 10, 20, 30, and 40 in source clip.

When used as above you must ensure that eg BADFRAMES and PREBAD have the same number of unique frames and sequential or results
may not be as expected (ie use Ordered).

# ----------------

Same Example again using Extract arg:-

C=Colorbars().Trim(0,-50).ShowFrameNumber() # 50 frames (0 -> 49)
BADFRAMES = "10;20;30;40" # 4 bad frames in source clip
EXTRACT=FrameSel(C,scmd=BADFRAMES,ordered=True,extract=3) # Extract Previous, Target and Next frames for each Target frame
REPAIR=EXTRACT.SelectEvery(3,0) # Select each frame previous to Target bad frames, 1st of each triplet
#REPAIR=REPAIR.Invert() # So we can easily see repaired frames
Return FrameRep(C,REPAIR,scmd=BADFRAMES) # Fix bad frames with REPAIR clip using exact same BADFRAMES spec.

# ----------------

Example scripts to show use with a paint program for manual frame repair.

# Extract Script
C=Colorbars().Trim(0,-50).ShowFrameNumber.ConvertToRGB24 # 50 frames (0 -> 49), RGB24
BADFRAMES = "10;20;30;40" # 4 bad frames in source clip
EDITCLIP=FrameSel(C,scmd=BADFRAMES,ordered=True) # Extract BAD frames
EDITCLIP.ImageWriter(file="Pic%06d.bmp",type="bmp",info=true) # Write bitmaps to edit
return Last
# Need to play all way through for ImageWriter to work properly.

# Here, EDIT bad frames in paint program

# Replace Script, after editing
C=Colorbars().Trim(0,-50).ShowFrameNumber.ConvertToRGB24 # 50 frames (0 -> 49), RGB24
BADFRAMES = "10;20;30;40" # 4 bad frames in source clip
END = 3 # unique_bad_frames - 1 (number of frames in BADFRAMES-1)
REPAIRED=ImageReader(file="Pic%06d.bmp",end=END)
#REPAIRED=REPAIRED.Invert() # Hilite repaired frames, easier to see
Return FrameRep(C,REPAIRED,scmd=BADFRAMES,show=true) # Fix bad frames with REPAIRED clip using exact same BADFRAMES spec.
# NOTE, you need to tell ImageReader the last frame number to read,
# the default for End is 1000, so imagereader will read in 4 frames (0 to 3) and 997 duplicates,
# FrameRep would throw an error as it does not know what to do with the unwanted 997 duplicate frames that were 'helpfully' appended.
# The 1st script, ImageWriter(Info=True) arg, shows the FrameNumber and name for each frame, and the number shown on the last frame
# can be assigned to END in 2nd script, shows eg "Frame 3 written to: D:\Pic000003.bmp".
# The Last frame number shown by Info=True, will be correct even if BADFRAMES list (in eg string or command file) has duplicates
# as they will be sorted and only unique frames delivered by FrameSel when Ordered=True is used, eg
# BADFRAMES="40;30;20;10;20;30;40", if used in 1st script FrameSel will select exactly the same frames as BADFRAMES="10;20;30;40",
# ie 4 frames and in sequential order (Ordered=True).
# ----------------

Example scripts to show use with a paint/rotoscope program that can read in bad frames together with previous and next frames,
and manually fix frames using image data from previous or next frames to repair damage.
# Extract Script
C=Colorbars().Trim(0,-50).ShowFrameNumber.ConvertToRGB24 # 50 frames (0 -> 49), RGB24
BADFRAMES = "10;20;30;40" # 4 bad frames in source clip
EDITCLIP=FrameSel(C,scmd=BADFRAMES,ordered=True,extract=3) # Extract Previous, Target and Next frames for each BAD frame
EDITCLIP.ImageWriter(file="Pic%06d.bmp",type="bmp",info=true) # Write bitmaps to edit
return Last

# Here, EDIT in Triple frame paint/rotoscope editing program

# Replace Script, after editing triplets
C=Colorbars().Trim(0,-50).ShowFrameNumber.ConvertToRGB24 # 50 frames (0 -> 49), RGB24
END = 11 # unique_bad_frames * Extract - 1
BADFRAMES = "10;20;30;40" # 4 bad frames in source clip
EDITCLIP=ImageReader(file="Pic%06d.bmp",end=END)
REPAIRED=EDITCLIP.SelectEvery(3,1) # Select repaired BAD frames, 2nd of each triplet
#REPAIRED=REPAIRED.Invert() # Hilite repaired frames, easier to see
Return FrameRep(C,REPAIRED,scmd=BADFRAMES,show=true) # Fix bad frames with REPAIRED clip using exact same BADFRAMES spec.


In 1st example above, selected frames were EDITED/Changed and then replaced back where they came from,
You, would want to extract the flash frames, change(spotless) the entire source clip and then replace extracted flash frames back into the spotless'ed clip.

EDIT: making small changes to 1st example [Untested]

Colorbars.Trim(0,-10).ShowFrameNumber() # 10 frames
sCmd="2;5;6" # Frames in SCmd string
REJECT=False # False=modify SCmd frames : True=modify frames NOT in SCmd
SELECTED=FrameSel(scmd=sCMD,ordered=TRUE,reject=REJECT,debug=true) # We are going to use FrameRep, must use ORDERED=True (OR REJECT=True)

#SELECTED=Invert(SELECTED) # Do some editing
Last=Invert(Last) # Do some editing to original source clip [Or use Spotless or whatever instead of invert]

FrameRep(Last,SELECTED,scmd=sCMD,reject=REJECT,debug=true) # Put SELECTED [eg Flash] frames back into EDITED Last clip (audio from Last)
return Last


EDIT: You could also use Clipclop for similar functionality.

[johnmeyer]

I'm doing some film transfer and restoration for the first time in several years. I'd used SpotLess before, but hadn't done a lot of testing until now. On these Super 8 and 8mm transfers I'm doing now, it did a better job than the old RemoveDirtMC.

The reason for posting is that I almost went back to RemoveDirtMC after I discovered some really horrible morphing artifacts on a girl's face in a swimming pool. I've seen this stuff before when using MAnalyze motion estimation for frame interpolation. On a hunch, I changed TrueMotion=False to TrueMotion=True.

Problem solved.

I changed the defaults in my script and will not be tempted to go back to TrueMotion=False.

I searched this entire thread to see if anyone had tested this. One person did test it, but never reported on the results because he was testing other things as well.

[StainlessS]

Thanks for the report John.

From first post,


Tm, TrueMotion Default True. Some folk swear truemotion=false is better.

Glob, Default True (True v1.06, was same as Tm, true almost always better), Allow set MAnalyse(global) independently of TrueMotion.
From MvTools2 docs for MAnalyse,
global
Estimate global motion (at every level) and use it as an additional predictor.
Only pan shift is estimated (no zoom and rotation).
Use false to disable, use true to enable.


Perhaps you had already changed defaults in your version script.

[johnmeyer]

I'm not sure why I had "false" in my script, other than the comment you put in your script the "some people" (whoever they are) preferred false. I just wanted people to know that the TrueMotion setting does make a difference in this script.

BTW, thanks for both SpotLessS (as I spell it) and also GamMac. I used them both yesterday in my restoration work. I still can't get GamMac to always produce better color with these old films, so I do one restoration with GamMac, and one without, and then cut between those in my NLE to get the best color. The one without GamMac already has the color correction that I do manually in my NLE, so the colors have already been corrected as best I can.

[StainlessS]

Pre v1.06, Glob default was same as tm, if your used script version was prior v.1.06, and you manually changed to default tm=false (because of the 'some 'people' comment),
then Glob default would be same as your default tm of false, that would probably not be very good, [Note Glob comment, "true almost always better"].

[DTL]

"I just wanted people to know that the TrueMotion setting does make a difference in this script."

This script uses the very very simple example of ME with the old mvtools2 plugin. Also the difference is not only with truemotion preset switch but also with some footage + mvtools ME settings used. This is why some other users may report with different footage the better results may be with TrueMotion=false.

More advanced way of ME used in QTGMC script (different prefiltering of the input to MAnalyse + up to 2 stages of block size refining + refining search around all predictors (including temporal predictor). These features are implemented in ver 3.392s https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.6%20and%20up/QTGMC.avsi . I think someone can export the ME part from QTGMC and make it into separate script function like MAnalysePlus(all ME arguments from QTGMC) and return value as MVs clip to be used as script function in the other scripts without copy of many text from QTGMC.

Other possible solution is to use exports of MV clips from QTGMC - https://github.com/realfinder/AVS-Stuff/blob/ff6c8682f82e073d2775564c8f96ad8e496ffc8b/avs%202.6%20and%20up/QTGMC.avsi#L753 . But they are in the format of a single MV clip for single time offset (not in the format of multi MV clip). So easy enough may be created a version of Spotless with RadT up to 6.

Example for RadT=1:

Function SpotLess_QTGMC_ME_RadT1(clip c,int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",
\ Float "bBlur", clip "dc" ) {
ThSAD = Default(ThSAD,255*(8/2 * 8/2)) # SAD threshold at radius 1. Here is max'ed out at 255 * 1/4 of 8*8 block <ie 4080> (MvTools Default 10,000 is nearly OFF, maybe way Too High).
# Maybe above 255*8/2*8/2 still a bit high, maybe 128*(8/2 * 8/2) Better <ie 2048>. For General noise perhaps somethimg like 24 * (8 * 8) <ie 1536>.
ThSAD2 = Default(ThSAD2,ThSAD-(8*8)) # SAD threshold at radius RadT(a bit less than ThSAD).
Pel = Default(pel,2) # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
Tm = Default(tm,True) # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
Chroma = Default(chroma,True) # MAnalyse chroma arg. If set to true, use chroma in block matching.
Glob = Default(glob,True) # Default True, Allow set MAnalyse(global) independently of TrueMotion.
# Only 1 Level required where not MAnalyse-ing.
sup_rend = c.MSuper(pel=pel, sharp=2,Levels=1)
QTGMC(c,InputType=1, GlobalNames="QTGMC_SL", preset="slow", SubPel=Pel, BlockSize=16, Overlap=8, TrueMotion=Tm, ChromaMotion=Chroma, GlobalMotion=Glob, \
Refinemotion=true, RefineMotionAll=true, search=3,searchparam=4, trymany=true, RefineMotionSearch=3, RefineMotionRadius=8, PNew=5)
#QTGMC(c,InputType=1, GlobalNames="QTGMC_SL", preset="slow", SubPel=Pel, BlockSize=32, Overlap=16, TrueMotion=Tm, ChromaMotion=Chroma, GlobalMotion=Glob, Refinemotion=true, Refinemotion2=true, \
#temporal=true, RefineMotionAll=true, search=3,searchparam=4, trymany=true, RefineMotionSearch=3, RefineMotionRadius=8, PNew=5)

mc_f1=c.MCompensate(sup_rend, QTGMC_SL_fVec1, thSad=ThSAD, thSad2=ThSAD2)
mc_b1=c.MCompensate(sup_rend, QTGMC_SL_bVec1, thSad=ThSAD, thSad2=ThSAD2)
Interleave(mc_b1, c, mc_f1)
MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=1) # Temporal median blur only [not spatial]
SelectEvery(3,1) # Return middle frame
}


Lots more settings for ME to play with. But I still can not get ME result matching close enough the simple SpotLess version. May be prefiltering in QTGMC make this difference.