Temporal filter w/ ME and scene change detection?

[`Orum]

I've been using several temporal filters to try and get one that does what I want but I've been unable to find one so far. Basically I'm looking for one that understands motion estimation (ME) and also detects scene changes ("cuts").

I've found some close ones; TTempSmooth (detects cuts with a user-set threshold, but no ME) and MVDegrain/MVCompensate (ME but no cut detection). I've also tried some 3D filters, but my favorite one, dfttest, doesn't detect scene changes, leading to some ugly ghosting, and is very, VERY slow when doing anything other than just 2D filtering--which it's not terribly fast at either.

The ideal thing would be to somehow hack Didée's Scripted_MVDegrain3() to allow for a scene change threshold to be added, as the filter is otherwise excellect, especially the ME (though I did modify it to use MVDegrain1/2 instead of 3 to improve its speed a bit). I'm not sure how to do this, or if it's even possible in the scripting language, but I'm open to suggestions.

Thanks in advance.

Edit: MaskTools appears to have this capability w/ MVSCDetection(), but unfortunately returns a full-frame clip that's either "green" or "purple" instead of an easy to use boolean value for the script. Any way around this?

[Sagekilla]

Why do you need scenecut detection? Also, MVTools already has a scenecut detection parameter for many of it's functions. See this (http://avisynth.org.ru/mvtools/mvtools.html#functions) page for more information, specifically thSCD1 and thSCD2. Honestly, what you describe isn't hard at all for MVTools.

thSCD1 determines if a block has changed, while thSCD2 determines how many blocks need to change for it to determine there's been a scene change. Read the documentation I linked to for more information.


function MVDegrain( clip source, int "degrain", int "blksize", int "overlap",
\ int "pel", int "thSCD1", int "thSCD2", int "idx" )
{
degrain = default( degrain, 2)
blksize = default( blksize, 8)
overlap = default( overlap, blksize/2)
pel = default( pel, 2)
thSCD1 = default( thSCD1, 400)
thSCD2 = default( thSCD2, 130)
idx1 = default( idx, 1)

global idx1 = idx1 + 1

bvec3 = (degrain>=3) ? source.MVAnalyse(isb=true, delta=3, blksize=blksize, overlap=overlap, pel=pel, idx=idx1) : NOP()
bvec2 = (degrain>=2) ? source.MVAnalyse(isb=true, delta=2, blksize=blksize, overlap=overlap, pel=pel, idx=idx1) : NOP()
bvec1 = (degrain>=1) ? source.MVAnalyse(isb=true, delta=1, blksize=blksize, overlap=overlap, pel=pel, idx=idx1) : NOP()
fvec1 = (degrain>=1) ? source.MVAnalyse(isb=false, delta=1, blksize=blksize, overlap=overlap, pel=pel, idx=idx1) : NOP()
fvec2 = (degrain>=2) ? source.MVAnalyse(isb=false, delta=2, blksize=blksize, overlap=overlap, pel=pel, idx=idx1) : NOP()
fvec3 = (degrain>=3) ? source.MVAnalyse(isb=false, delta=3, blksize=blksize, overlap=overlap, pel=pel, idx=idx1) : NOP()

denoise = (degrain>=3) ? source.MVDegrain3(bvec1, fvec1, bvec2, fvec2, bvec3, fvec3, thSCD1=thSCD1, thSCD2=thSCD2,idx=idx1) :
\ (degrain==2) ? source.MVDegrain2(bvec1, fvec1, bvec2, fvec2, thSCD1=thSCD1, thSCD2=thSCD2, idx=idx1) :
\ (degrain==1) ? source.MVDegrain1(bvec1, fvec1, thSCD1=thSCD1, thSCD2=thSCD2, idx=idx1) : NOP()

return(denoise)
}

[`Orum]

I need scene cut detection because as you approach a cut (when filtering with Scripted_MVDegrain3(), for example) you get noticeable artifacts that increase the entropy. I can upload some examples if you're interested.

While it has scenecut detection on its own internal filters, scripted filters making use of the internal filters and combining the results manually (once again, Scripted_MVDegrain3() suffers from this) cannot benefit from these parameters designed for the internal filtering.

[Didée]

Show those examples, please.

Generally, the internal "Scenecut detection" of MVTools' filters do exactly what they're supposed to do. (They sometimes miss a SC, but that's another story.)

The other part of the story is that IF there is a SC, then temporal filtering gets less efficient, because the weights of the not-used frames (from the other side of the SC) get assigned back to the actual current frame, which simply results in "less filtering". As a consequence, the filtered frames around the SC do keep more noise. Perhaps this is what you name as "increased entropy" ... ?

[Fizick]

`Orum,
I (we) do not understand what do you want :), but I can make some change of MVSCDetection() in v2.0 for you :).
Please clarify how you intent to use boolean and scene cuts in hypothetic script, and why you can not use masktools binarize or similar.

[`Orum]

Ok, here's a screenshot from the only source I have handy at the moment, the fast and the furious DVD. It's 4 frames leading up to a scene change and then the first frame of the new scene. On the left is the unfiltered (well, just deblocked) frame, in the center one run though my ever so slightly modified Scripted_MVDegrain3(), and on the right is a luma histogram of the difference between the two. As you move from top to bottom you go further forward in time til it hits the scene change.

Visible Ghosting (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghosting.jpg) (warning, high res)

Look at the vertical line on the forehead, the area to the right where the glasses rack is after the scene change, and the noise on the beard (in fact, the whole frame suffers, but these are the most noticeable areas). As you can see, there's some visible ghosting from the future frame(s) after a scene cut (remember, because we're temporally smoothing forwards and backwards, we can ghost in both directions). Now, what I propose is to use a scene change detection algorithm (the one already in MVTools is fine) to figure out where scene changes are. If a frame after a scene change would normally be considered in the temporal smoothing, we instead substitute the next-closest frame within our current frame's scene.

For example, say we are filtering frame x, and there is a scene change at x+2 (x+2 is part of the new scene). What we currently have is:
x-3, x-2, x-1, x, x+1, x+2, x+3
^
new scene
but what we want is:
x-3, x-2, x-1, x, x+1, x+1, x+1
^
stay within our current scene
substituting the last frame in our scene for future frames, more heavily weighting it without accumulating all this false noise from the scene change. This would of course work backwards as well, repeating early frames when filtering frames immediately following a scene change. In areas of extremely high motion, where it would detect several frames in a row as a "scene change", there isn't much to be gained from temporal smoothing anyway and thus we'd just pass those frames strait through.

So, what do you think?

[Sagekilla]

Is there any chance you can post the actual video so we can help you? Posting pictures (jpgs at that, are not recommended, you should use pngs most of the time) only helps so much.

The following avs script should help anyone who's using AvsP to see what's going on:

Imagesource("ghosting.jpg")

c0 = Crop(0,1456,720,364)
c1 = Crop(0,1092,720,364)
c2 = Crop(0,728,720,364)
c3 = Crop(0,364,720,364)
c4 = Crop(0,0,720,364)

f0 = Crop(720,1456,720,364)
f1 = Crop(720,1092,720,364)
f2 = Crop(720,728,720,364)
f3 = Crop(720,364,720,364)
f4 = Crop(720,0,720,364)

clean = Interleave(c0,c1,c2,c3,c4)
filter = Interleave(f0,f1,f2,f3,f4)

Interleave(clean,filter)

[Fizick]

I can say, that currently in MVDegrain3 plugin it is implement approximately as a follows:

x-3, x-2, x-1, x, x1, x, x



But IMHO ghosting at scenechange is not important and may be considered as usual crossfade transition.

you still do not provide a (pseudo)script :)

[Sagekilla]

Through some google magic I came up with this:


function Scripted_MVDegrain3(clip c, clip "mvbw", clip "mvfw", clip "mvbw2", clip "mvfw2", clip "mvbw3", clip "mvfw3",
\ int "thSAD", int "plane", int "limit", clip "pelclip", int "idx")
{
thSAD = default(thSAD, 400)
plane = default(plane, 4)
limit = default(limit, 255)
_idx = default(idx, -11)

thSAD = thSAD / 8

# alt = c.FFT3DFilter(sigma=10,sigma2=6,sigma3=4,sigma4=2,bw=16,bh=16,ow=8,oh=8,bt=1)

SAD_fw3 = c.MVMask(mvfw3, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_fw2 = c.MVMask(mvfw2, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_fw1 = c.MVMask(mvfw, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_bw1 = c.MVMask(mvbw, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_bw2 = c.MVMask(mvbw2, kind=1, ml=thSAD, gamma=0.999, Ysc=255)
SAD_bw3 = c.MVMask(mvbw3, kind=1, ml=thSAD, gamma=0.999, Ysc=255)

comp_fw3 = c.MVCompensate(mvfw3, idx=_idx) # .mt_merge(alt,SAD_fw3,U=3,V=3)
comp_fw2 = c.MVCompensate(mvfw2, idx=_idx) # .mt_merge(alt,SAD_fw2,U=3,V=3)
comp_fw1 = c.MVCompensate(mvfw, idx=_idx) # .mt_merge(alt,SAD_fw1,U=3,V=3)
comp_bw1 = c.MVCompensate(mvbw, idx=_idx) # .mt_merge(alt,SAD_bw1,U=3,V=3)
comp_bw2 = c.MVCompensate(mvbw2, idx=_idx) # .mt_merge(alt,SAD_bw2,U=3,V=3)
comp_bw3 = c.MVCompensate(mvbw3, idx=_idx) # .mt_merge(alt,SAD_bw3,U=3,V=3)

black = blankclip(c,color_yuv=$008080)
long = interleave( comp_fw3,comp_fw2,comp_fw1, c, comp_bw1,comp_bw2,comp_bw3 )
long_SAD = interleave( SAD_fw3, SAD_fw2, SAD_fw1, black, SAD_bw1, SAD_bw2, SAD_bw3 )

long.TTempSmooth(3,255,255,1,1,strength=4,pfclip=long_SAD,fp=false,scthresh=99.9)
# long.temporalsoften(3,255,255,24,2) # instead of TTempSmooth - if and only IF alt-clip is used
SelectEvery(7,3)
}



My question is, why are you using a "scripted" MVDegrain3? I remember the only reason why that was ever invented was because a true MVDegrain3 wasn't out yet.

[Didée]

@ Sagekilla: that Scripted_~ function probably is even worse at scenechanges. "scthresh" would need to be altered. (The first ad-hoc version didn't work correctly because it considered many many correct neighbors as scenechange, cause I had forgotten about that parameter, and that it's related to 0-255 SAD masks, where ttempsmooth default value is related to normal image content. After discovering, I just quickly put a high value to make it work generally.)

@ 'Orum: That kind of artifacts appears when the actual SAD values do *not* trigger MVTools' SC detection. No SC detected, and false frames are taken into the filtering mix.
You need to lower thSCD1 and/or thSCD2 in MVDegrain.

Default: MVDegrainX( [vectors], [thSAD], thSCD1=400, thSCD2=128 )

Instead of 400|128, try e.g. 300|95.


Edit:
Oh, actually you are using the Scripted_~ function. Well, if the commented sections are still commented, then there's hardly any justification to use that one instead of MVDegrain. ;)

Try simply with MVDegrain, and toy a little with the thSCD values. If it has to be the Scripted_~ one for whatever reason, try lowering the "scthresh" value in ttempsmooth until the scenechanges are okay.

[Sagekilla]

Didee, the script that I posted was one I found, it was the only copy of Scripted_MVDegrain3() that I could find (I searched for that exact name) so I don't know if there are other versions I might have missed.

@ `Orum: In any case, I HIGHLY recommend you don't use the Scripted MVDegrain3 when a true MVDegrain3 already exists. Use the one I posted a while ago, since you can use MVDegrain(1|2|3) easily, and call it multiple times in a script if you -really- need to. For that matter, did you even -try- the script I posted to see if it would work?

[`Orum]

@Sagekilla
Yeah, I uploaded it as a jpg because photobucket limits you to 1MB image size. If you want the individual frames as pngs, you can see them here:

src t-4 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/As-4.png) src t-3 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Bs-3.png) src t-2 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Cs-2.png) src t-1 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Ds-1.png) src t (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Es0.png)
flt t-4 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Af-4.png) flt t-3 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Bf-3.png) flt t-2 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Cf-2.png) flt t-1 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Df-1.png) flt t (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Ef0.png)
lma t-4 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Ad-4.png) lma t-3 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Bd-3.png) lma t-2 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Cd-2.png) lma t-1 (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Dd-1.png) lma t (http://img.photobucket.com/albums/v16/yoitsmeremember/tmp/ghst/Ed0.png)

And yes, that's the script I'm referring to. Why don't I use the standard MVDegrain3()? For some sources I like to use the alt clip for heavier denoising. Also, the two filters (not using the alt) do give *slightly* different denoising, though it's usually not noticeable (but depends somewhat on the footage).

@Fizick
Well, this is more for passing it off to an encoder. If you blur the scene changes, that's more entropy you have to deal with, and it's also less likely to keyframe at the right point or as efficiently.

@Didée
Yeah, I realize that, but that's only if I use the standard MVDegrain1/2/3, but I usually do lower the thresholds slightly.

[Sagekilla]

If you need "heavier denoising" I recommend try raising the thSAD threshold in MVAnalyse, so it gets processed more. It's very unlikely that a standard call of MVDegrain3 will actually end up ghosting unless you start specifying unrealistic values. But, for values that are generally in the normal range you can easily increase the amount MVDegrain will denoise.

Try thSAD=500 or thSAD=550, and see if it helps any.

Edit: Also, the "alt" clip is nothing more than using preprocessing, you can easily modify another script to do this. Can you please post your modified SC_MVD3 script so we can advise you?

[`Orum]

Can you please post your modified SC_MVD3 script so we can advise you?
While the screenshots I've posted so far use exactly what you've already pasted, the times I use it for heavier denoising I change it slightly. It's essentially what's already been posted, only I have uncommented everything for the alt clip and commented out the line specific to use when NOT using the alt-clip. I also take the params for FFT3DFilter in the function so I can easily change them.

However, if it's as you say, and I can just prefilter with something like FFT3DFilter, I suppose there's no real point to using the scripted function over MVDegrain3(), assuming they get the same results.

I have tried upping thSAD on MVDenoise3(), but it's not quite the same as the "alt clip" filtering (or the prefiltering you suggested, but I'll have to test to verify).

[Didée]

Well, of course upping thSAD in MVDegrain is not the same as using "alt" in the scripted function. And it's not the same as simply using some pre-filtering, either.

MVDegrain denoises less when SAD of a compensated block is high. Basic principle: a noisy spot that cannot be compensated will not be filtered. It stays noisy.

With pre-filtering, all of the frame gets additional filtering before MVDegrain even gets its hands on the clip, i.e. this is in no way adaptive.

What the scripted function with "alt" does is this: process like MVdegrain does, but weight-in the "alternatively" filtered clip according to block SAD. Effect: if compensation is good (SAD is low), alt filtering does not take place. As compensation gets worse (SAD gets bigger), use more&more of the alt filtered clip.

(Small excursus: A similar, more simple & faster, but less accurate way is to just average all SAD masks in the filtering window, and use that average to merge-in an alternative filter. IIRC, *.mp4 guy showed that way in a few scripts. But it's less accurate, since it evenly reduces weights for all neighbor frames (as much for the "good" ones as for the "bad" ones).
The way shown in Scripted_MVDegrain3 is the correct way of what generally should be done (or: should be an available option), though I've little hope it'll ever be implemented in MVTools: because an additional clip is used, it is no more a "generic" filter in the way that all MVTools filters are.)

[`Orum]

Ah, so they are different. Is there a way to use the alternatively filtered clip with scene change detection?

[Didée]

Wrong question. The alt clip already IS used for scene changes.

Example for a valid question: "Shouldn't the thSCD1|2 parameters also be implemented in that function?"

- Yes, they should. Along with several other modifications, to make that tepid function a little hotter. minSAD thresholding and and inloop sharpening can be plugged-in rather easily. Usability should be improved, too ... variable temporal radius, alt clip parametrisation /or/ using a custom "external" alt clip. Lots of handicraft that could be done ...