SpotLess - Temporal denoise

[StainlessS]

SplotLess() v1.04, update. See 1st post.

I made the executive decision to make some script breaking changes, sorry.

    SpotLess v1.04. temporal spot/noise remover, by StainlessS @ Doom9.   https://forum.doom9.org/showthread.php?t=181777
    Original idea from Didée post :-                                      https://forum.doom9.org/showthread.php?p=1402690#post1402690
    Req:- Pinterf MvTools2(), Pinterf Medianblur2() with MedianBlurTemporal rather than the MedianBlurT() from Didée post.
          With appropriate plugins, will be AVS+ colorspace and x64 compatible.
          Fine using Avs+ Prefetch() so long as current Pinterf plugins, and Frame Accurate source. RequestLinear() following the source filter might suffice as frame accurate source filter. 

    NOT FOR cartoon/anime, live video only, sorry.

    v1.01, Remove RadT Max check. Add MSuper(hpad=16,vpad=16). Add BlkSz arg.
    v1.02, Add some stuff.
    v1.03, Frame size auto config of args removed (found instances where caused problems). Added Glob and bBlur args.
    v1.04, Script breaking changes, I guess is more flexible if can also set ThSAD, inserted ThSAD 3rd arg. RadT now default 1, was 2, dont over denoise unless requested.

    SpotLess(clip c,int "RadT"=1,int "ThSAD"=10000,int "ThSAD2"=ThSAD,int "pel"=2,bool "chroma"=true, int "BlkSz"=8,Int "Olap"=BlkSz/2,bool "tm"=true,Bool "glob"=Tm,Float "bBlur"=0.0)

    RadT, 1 or more, Default 1. Removes Spots on up to RadT [Temporal Radius] consecutive frames.
        RadT > 2 will usually be overkill. Setting too high could possibly result in blurring.
        Each pixel in result frame is median pixel value of (2*RadT+1) motion compensated frames (including source, ie current_frame-RadT to current_frame+RadT).

    ThSAD, Default 10000=NEARLY OFF(ie ignore hardly any bad blocks), 0 < ThSAD < 16320(8*8*255). 8x8 block SAD threshold at radius 1 (ie at current_frame +- 1) [SAD, Sum of Absolute (pixelwise) Differences].
        ThSAD and ThSAD2 suggested absolute minimum of maybe about 400.
        ThSAD and ThSAD2 thresholds are based on 8 bit 8x8 block, irrespective of colorspace depth or BlkSz, max=8x8x255=16320, use same thresholds where High Bit Depth.
        In mvTools MCompensate(), when creating a compensated block the SAD between compensated block and the same original block in current_frame, the 8 bit SAD is measured and if
        greater than SAD threshold then that block is ignored and uses original block from current frame instead. [The compensated block is judged too different, so ignored & original block used instead
        in the result MCompensated frame].
        Where eg ThSAD=64, AVERAGE absolute single pixel difference threshold would be 64/(8*8)=1, so AVERAGE absolute pixel difference greater than 1 would ignore that mcompensated block and use the
        block from current frame in the resulting mcompensated frame instead. This example allows for all pixels in a 8x8 block to be different by 1, or a single pixel in 8x8 block to be different by 64,
        or some other mixture.
          A problem with above is, if a low ThSAD and current_frame block is mostly noise, so compensated blocks could be judged bad because they are too different to the bad noisey block, and the result
        block may/will be just as bad as the noisy source block. A possible solution to this problem is to have a higher SAD threshold and/or have a bigger BlkSize so that the number of bad source pixels
        after converting/scaling to as if an 8x8 block, will contain fewer bad noise pixels. So, SpotLess BlkSz arg would ideally maybe 4 or more times the area of the largest spots that you have, and a SAD
        threshold big enough so as to not ignore the block [ wild guess minimum SAD threshold for big spot sizes of (8x8x255)/4 = 4080 ].
        Where a complete source frame is bad, then maybe should have very high (eg 10000) SAD threshold, and BlkSz may not really matter too much.
          It is not the end of the world if some of the compensated blocks are ignored and swapped for the original current_frame block. Nor is it the end of the world if
        no blocks were ignored because of high SAD threshold. The final result pixel is median pixel value of (2*RadT+1) motion compensated blocks, so allowing for some mistakes by choosing the
        middle pixel value.
        I've just tested real bad double frame, full frame luma and chroma corruption, with below line:
            SpotLess(RadT=5,ThSAD=1000000,ThSAD2=1000000,pel=2,chroma=false,BlkSz=8,Olap=4,tm=false,glob=false,bBlur=0.0)
        And although both SAD thresholds of 1 million, are totally impossible and so no blocks could possibly be ignored and yet we still got pretty good results, all frames were fixed
        as we still had the temporal median filter to fall back on and pick the middle pixel value.

        From mvtools2 docs:
          ThSAD is SAD threshold for safe (dummy) compensation.
              If block SAD is above the thSAD, the block is bad, and we use source block instead of the compensated block. Default is 10000 (practically disabled).

    ThSAD2, Default ThSAD, 0 < ThSAD2 < 16320(8*8*255), Lower removes fewer spots, but less chance of blurring.
        ThSAD2 sets the SAD [Sum of Absolute Differences] threshold for most distant frame from current_frame at distance RadT, with those frames that are distances in-between 1 and RadT
        acquiring a SAD threshold linearly interpolated between the two.
        From mvtools2 docs:
          ThSAD2:
              Defines the SAD soft threshold for the furthest frames at current_frame +- RadT.
              The actual SAD threshold for each reference frame is a smooth interpolation between the original thSAD (close to the current frame)
              and thSAD2. Setting thSAD2 lower than thSAD allows large temporal radii and good compensation for low SAD blocks while reducing the global error and the
              risk of bluring when the result of MCompensate is passed to a temporal denoising filter.
        EDIT: Although I have said that SAD threshold being too high could result in blurred frames, that is really taken from above "risk of bluring" line from mvtools docs,
        however, that warning says "temporal denoising filter", which might suggest pixel averaging, whereas we are using pixel median. I'm not sure that blurring would be the result
        of having too high a SAD threshold.

    Pel,     Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=half pixel, 4=quarter pixel)

    Chroma,  Default True. MAnalyse chroma arg. If true, use chroma in block matching when creating vectors. Maybe use False if source B&W or color noise.

    BlkSz,   Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps better for HD clips. [Info: current Pinterf MvTools allows for BlkSize=12, and overlap=6]

    OLap,    Default half BlkSz, Block overlap.

    Tm,      TrueMotion Default False. Some folk swear truemotion=false is better.

    Glob,    Default Tm, Allow set MAnalyse(global) independently of TrueMotion.
             From MvTools2 docs for MAnalyse,
               global
                 Estimate global motion (at every level) and use it as an additional predictor.
                 Only pan shift is estimated (no zoom and rotation).
                 Use false to disable, use true to enable. Default is like truemotion.

    bBlur,   Default 0.0 [OFF]. If used, Suggest about 0.6, where MAnalyse create vectors is performed on denoised (blurred) super clip.

EDIT: RequestLinear() following the source filter might suffice as frame accurate source filter.

[johnmeyer]

I just tried the new v1.04 on the full version of that B&W football game, but didn't see any significant differences.

I'm still getting a little too much removal of legs, etc., but I'm still trying to experiment with all the settings.

[StainlessS]

I have not as yet tried with that clip, I'm a bit knackered and probably heading for bed soon.
Maybe I should have given moniker of this filter StainLessS(), then my name would be remembered down the millennia,
me and Achilles, yes they would remember us.

EDIT: John, your torture clip is 29.976FPS with dupes, so assume 23.976 FPS.
You need to at least remove duplicates, they will mess stuff up, ie count bad frames twice and mess with median,
things look tricky enough without having same error counted multiple times.

[johnmeyer]

Originally Posted by StainlessS

John, your torture clip is 29.976FPS with dupes, so assume 23.976 FPS.
You need to at least remove duplicates, they will mess stuff up, ie count bad frames twice and mess with median,
things look tricky enough without having same error counted multiple times.

Let me go back and look at that. I just checked and it looks like proper 23.976 progressive without dups, but maybe something got screwed up along the way.

[edit]I just looked at the original MPEG video, as well as the clip that I losslessly extracted using Womble's MPEG Video Wizard, and both are indeed 23.976 progreesive without dups or drops. I looked at the test clip I posted in MediaInfo and it reports 23.976, with no hint of 29.97 w/ pulldown. My NLE (Vegas) also reports 23.976 and when I play the clip in Vegas, as well as when I serve from Vegas into your script, I see no evidence of mismatched frame rates.

Have another look tomorrow morning and see if something got glitched on your end.

[Boulder]

Have you tested if doing MRecalculate (or multiple recalculations) with a smaller blocksize on the vectors would help with those disappearing human parts issues?

[joka]

Spotless() is a motion compensated temporal median filter of n frames (n=2*RadT+1). In case of n=5 this is a very heavy filter that removes a lot of dirt, spots, grain, ...; but removes details and (parts of) moving objects too.
Such a filter but with n=3 (like the script proposed by Didee) is the cleaning part of RemoveDirtMC. In RemoveDirtMC the cleaning filter is followed by a motion area restoring part ("RestoreMotionBlocks") to remove the artifacts.

In my opinion spotless() is a try worth to use it as cleaning filter in RemoveDirtMC. For a stand alone usage there are too much articfacts.

However, as I understood the original intend was to remove overlapping spots on 2 consecutive frames. Spotless() can do it in no-motion areas, but the required "RestoreMotionBlocks" will restore it.

[johnmeyer]

Originally Posted by Boulder

Have you tested if doing MRecalculate (or multiple recalculations) with a smaller blocksize on the vectors would help with those disappearing human parts issues?

I never thought of doing that. I've re-written quite a few MVTools2 scripts to use MRecalculate so, if time permits today I'll try that with StainlessS' code and see what happens.

I made the observation, in my earlier post, that block size had the strongest impact on the filter of any variable and this impact was far larger than I'd ever seen in other uses of MVTools2. It never occurred to me to use MRecalculate, which I normally think of as a way to get the precision of the smaller block size, but with the speed of the larger blocks, but which also, I guess, has other impacts.

[wonkey_monkey]

Spotless() is a motion compensated temporal median filter of n frames (n=2*RadT+1).

Huh. There was me skimming this thread, not really paying much attention, seeing the video and thinking "What this really needs is a motion compensated temporal median filter of, ooh, around 5 frames." Great minds think alike

[Boulder]

Originally Posted by johnmeyer

I never thought of doing that. I've re-written quite a few MVTools2 scripts to use MRecalculate so, if time permits today I'll try that with StainlessS' code and see what happens.

I made the observation, in my earlier post, that block size had the strongest impact on the filter of any variable and this impact was far larger than I'd ever seen in other uses of MVTools2. It never occurred to me to use MRecalculate, which I normally think of as a way to get the precision of the smaller block size, but with the speed of the larger blocks, but which also, I guess, has other impacts.

I have no idea if it does help here, but I find it stabilizes the vector field quite a bit. Maybe SSS will add an MShow-related parameter to display the vectors to help determine the correct settings.

[johnmeyer]

I was unable to get MRecalculate to work with this script. I suspect that the problem is that the "delta" variable in MAnalyze is not supported by MRecalculate. This script uses large values for Delta. Once I used MRecalculate, the resulting video was from the wrong moment in time.

[StainlessS]

John, I suspect it [Mrecalculate thingy] just needs vectors=Multivec, and tr=RadT.

MRecalculate

MRecalculate (
clip super,
clip vectors,
int thSAD (200),
int smooth (1),
int blksize,
int blksizeV,
int search,
int searchparam,
int lambda,
bool chroma,
bool truemotion,
int pnew,
int overlap,
int overlapV,
string outfile,
int dct,
int divide,
int sadx264,
bool isse,
int tr
int scaleCSAD (0)
)

Refines and recalculates motion data of previously estimated (by MAnalyse) motion vectors with different super clip or new parameters set (e.g. lesser block size), after divide, etc. The two-stage method may be also useful for more stable (robust) motion estimation. The refining is at finest hierarchical level only. Interpolated vectors of old blocks are used as predictors for new vectors, with recalculation of SAD. Only bad quality new vectors with SAD above threshold thSAD will be re-estimated by search. thSAD value is scaled to 8x8 block size. Good vectors are not changed, but their SAD will be re-calculated and updated.

You must set the tr variable if you process "multi" motion vector clips.

Parameters not described below have the same meaning as in MAnalyse, but you can use other values. Their default settings are same as in MAnalyse.

smooth

This is method for dividing coarse blocks into smaller ones.
0 Use motion of nearest block.
1 Bilinear interpolation of 4 neighbors.

tr

This is the temporal radius for motion vector clips generated by MAnalyse with multi = true. Default 0 (normal vector clip).

EDIT:
vectors is not an optional named argument for Mrecalculate, so skip the "vectors=".

MAnalyse     "c[blksize]i[blksizeV]i[levels]i[search]i[searchparam]i[pelsearch]i[isb]b[lambda]i[chroma]b[delta]i[truemotion]b[lsad]i[plevel]i[global]b[pnew]i[pzero]i[pglobal]i
                  "[overlap]i[overlapV]i[outfile]s[dct]i[divide]i[sadx264]i[badSAD]i[badrange]i[isse]b[meander]b[temporal]b[trymany]b[multi]b[mt]b[scaleCSAD]i"
MBlockFps    "cccc[num]i[den]i[mode]i[ml]i[blend]b[thSCD1]i[thSCD2]i[isse]b[planar]b[mt]b"
MCompensate  "ccc[scbehavior]b[recursion]f[thSAD]i[fields]b[time]f[thSCD1]i[thSCD2]i[isse]b[planar]b[mt]b[tr]i[center]b[cclip]c[thSAD2]i"
MDegrain1    "cccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain2    "cccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain3    "cccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain4    "cccccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain5    "cccccccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrain6    "cccccccccccccc[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[mt]b[out16]b"
MDegrainN    "ccci[thSAD]i[thSADC]i[plane]i[limit]f[limitC]f[thSCD1]i[thSCD2]i[isse]b[planar]b[lsb]b[thsad2]i[thsadc2]i[mt]b[out16]b"
MDepan       "cc[mask]c[zoom]b[rot]b[pixaspect]f[error]f[info]b[log]s[wrong]f[zerow]f[range]i[thSCD1]i[thSCD2]i[isse]b[planar]b"
MFlow        "ccc[time]f[mode]i[fields]b[thSCD1]i[thSCD2]i[isse]b[planar]b[tclip]c"
MFlowBlur    "cccc[blur]f[prec]i[thSCD1]i[thSCD2]i[isse]b[planar]b"
MFlowFps     "cccc[num]i[den]i[mask]i[ml]f[blend]b[thSCD1]i[thSCD2]i[isse]b[planar]b[optDebug]i"
MFlowInter   "cccc[time]f[ml]f[blend]b[thSCD1]i[thSCD2]i[isse]b[planar]b[tclip]c"
MMask        "cc[ml]f[gamma]f[kind]i[time]f[Ysc]i[thSCD1]i[thSCD2]i[isse]b[planar]b"

MRecalculate "cc[thsad]i[smooth]i[blksize]i[blksizeV]i[search]i[searchparam]i[lambda]i[chroma]b[truemotion]b[pnew]i[overlap]i[overlapV]i[outfile]s[dct]i[divide]i[sadx264]i[isse]b[meander]b[tr]i[mt]b[scaleCSAD]i"
               ^
               |


MRestoreVect "c[index]i"
MScaleVect   "c[scale]f[scaleV]f[mode]i[flip]b[adjustSubPel]b[bits]i"
MSCDetection "cc[Ysc]i[thSCD1]i[thSCD2]i[isse]b"
MShow        "cc[scale]i[sil]i[tol]i[showsad]b[number]i[thSCD1]i[thSCD2]i[isse]b[planar]b"
MStoreVect   "c+[vccs]s"
MSuper       "c[hpad]i[vpad]i[pel]i[levels]i[chroma]b[sharp]i[rfilter]i[pelclip]c[isse]b[planar]b[mt]b"

[johnmeyer]

I thought that's what I did, but this code didn't work.

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
    RadT     = Default(RadT,2)                              # Temporal radius. (MCompensate arg)
    ThSAD    = Default(ThSAD,10000)   # SAD threshold at radius 1 (Default Nearly OFF).
    ThSAD2   = Default(ThSAD2,10000)                        # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)
    Pel      = Default(pel,2)                               # Default 2. 1, 2, or 4. Maybe set 1 for HD+. (1=precision to pixel, 2=precision to half pixel, 4=quarter pixel)
    Chroma   = Default(chroma,True)                         # MAnalyse chroma arg. If set to true. Use chroma in block matching.
    BlkSz    = Default(BlkSz,8)                             # Default 8. MAnalyse BlkSize. Bigger blksz quicker and perhaps  better, esp for HD clips.
    OLap     = Default(OLap, BlkSz/2)                       # Default half of BlkSz.
    Tm       = Default(tm,True)                             # TrueMotion, Some folk swear MAnalyse(truemotion=false) is better.
    Glob     = Default(glob,Tm)                             # Default Tm, Allow set MAnalyse(global) independently of TrueMotion.
    Bblur    = Default(bblur,0.0)                           # Default OFF
    Assert(1 <= RadT,"SpotLess: 1 <= RadT")
    pad      = max(BlkSz,8)
    sup      = (bBlur<=0.0  ? c   : c.blur(bblur)).MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2)
    sup_rend = (bBlur<=0.0) ? sup : c.MSuper(hpad=pad,vpad=pad,pel=pel,sharp=2,levels=1) # Only 1 Level required where not MAnalyse-ing.
    MultiVec = sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,search=4,chroma=Chroma,truemotion=Tm,global=Glob)

    MultiVec =    sup.MAnalyse(multi=true, delta=RadT,blksize=BlkSz,overlap=OLap,search=4,chroma=Chroma,truemotion=Tm,global=Glob)
    MultiVec_re = MRecalculate(sup,MultiVec, chroma=Chroma,blksize=BlkSz/2, overlap=OLap/2,truemotion=Tm,tr=1) 
    c.MCompensate(sup_rend,MultiVec_re,tr=RadT,thSad=ThSAD,thSad2=ThSAD2)
    MedianBlurTemporal(radiusY=0,radiusU=0,radiusV=0,temporalradius=RadT)  # Temporal median blur only [not spatial]
    SelectEvery(RadT*2+1,RadT)                                             # Return middle frame
}

[StainlessS]

MultiVec_re = MRecalculate(sup,MultiVec, chroma=Chroma,blksize=BlkSz/2, overlap=OLap/2,truemotion=Tm,tr=RadT)

[johnmeyer]

That works. Thanks!

I'll play around with the variables and see if I can come up with something which cleans as well or better than RemoveDirt, and removes the same or fewer arms and legs.

[edit]If you ever decide to play with my torture clip, this set of variables did a pretty good job of cleaning the basic dirt (forget about the chemical stains because they are too big to be dealt with automatically). It also didn't completely amputate arms and legs, although socks and wristbands sometimes disappear. As long as that doesn't happen too often, most people won't notice.

This is getting tantalizingly close to being one of the truly great scripts.

SpotLess(source,ThSAD=5000,ThSAD2=5000,RadT=1,Chroma=false,BlkSz=8,OLap=4,Tm=false)

[StainlessS]

I will have a play with your clip, I'm just about done recovering a working linux after moving Linux / Windows onto SSHD (Solid State Hybrid Drive), it killed/crippled my setup for some days [well a week].
I had to look, I'm currently on Linux.

EDIT: This,

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
    RadT     = Default(RadT,2)                              # Temporal radius. (MCompensate arg)
    ThSAD    = Default(ThSAD,10000)   # SAD threshold at radius 1 (Default Nearly OFF).
    ThSAD2   = Default(ThSAD2,10000)                        # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)

Should be

Function SpotLess(clip c,int "RadT",int "ThSAD",int "ThSAD2",int "pel",bool "chroma", int "BlkSz",Int "Olap",bool "tm",Bool "glob",Float "bBlur") {
    RadT     = Default(RadT,1)                              # Temporal radius. (MCompensate arg)
    ThSAD    = Default(ThSAD,10000)   # SAD threshold at radius 1 (Default Nearly OFF).
    ThSAD2   = Default(ThSAD2,ThSAD)                        # Setting lower, sets SAD thresold to limit blurring effect of more distant reference frame BAD matching blocks. (MCompensate arg)

[johnmeyer]

Yeah, changing RadT to 1 is what made it behave the way I expected. Of course RadT>1 is part of what makes the script so powerful for other applications, like when you removed that massive color cast, so I'm sure I'll be using larger numbers for other applications.

[FranceBB]

And this is where things went completely wrong.
Sadly those athletes were all black and SpotlessS clearly thought they were big giant spots and tried its best to remove them xD






StainlessS, I think your filter might be bit racist xD

[kedautinh12]

Originally Posted by FranceBB

And this is where things went completely wrong.
Sadly those athletes were all black and SpotlessS clearly thought they were big giant spots and tried its best to remove them xD






StainlessS, I think your filter might be bit racist xD

I think they are ghost

[StainlessS]

StainlessS, I think your filter might be bit racist

Yeh well in this crazy mixed up PC (EDIT: Black Legs Matter) world, you gotta have a bit of balance somewhere.

EDIT: Although the guy in yellow at the back seems to have lost a bit more than his legs.

[johnmeyer]

Originally Posted by FranceBB

And this is where things went completely wrong.

What settings did you use? Without that information, your observations don't mean much.

I too noticed the same thing (limb amputation), and posted about it earlier in this thread. I then spent some time to learn and understand the filter, and found settings that completely avoid the disappearance of arms and legs that you posted.

Posting that, without providing any information about settings is exactly like complaining that a noise reduction filter eliminates all detail after you turn all the settings to "max." Such an observation does a disservice to the people reading the thread who may think that the filter/plugin has no utility and is not worth much.

SpotlessS is most definitely a very worthwhile script.