How to use a prefiltered clip for MDegrain2 motion estimation

[cy]

Hi,

Could someone please show me how to modify my script, in order to use a pre-filtered clip to feed MDegrain2?
I've read this would give more accurate motion vectors and thus more accurate de-noising/de-graining?

How "clean" must the pre-filtered clip be for improved results with MDegrain2 ?

My current script:

SetMTMode (5)

# Set DAR in encoder to 163 : 86. The following line is for automatic signalling
global MeGUI_darx = 163
global MeGUI_dary = 86

DGDecode_mpeg2source("Y:\MOVIES\84 Charing Cross Road (1987)  PAL DVD\mkv\VTS_01_1.d2v", info=3)
ColorMatrix(hints=true, threads=0)

SetMTMode (2)

super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

crop( 0, 14, -8, -14)

[Didée]

DGDecode_mpeg2source("... .d2v", info=3)
ColorMatrix(hints=true, threads=0)
src = last

SetMTMode (2)

preNR = src.FFT3DFilter( with_appropriate_settings )  # as an example

preNR_super = preNR.MSuper(pel=2, sharp=1)
src_super   = src.MSuper(pel=2, sharp=1, levels=1)

backward_vec2 = MAnalyse(preNR_super, isb = true,  delta = 2, overlap=4)
backward_vec1 = MAnalyse(preNR_super, isb = true,  delta = 1, overlap=4)
forward_vec1  = MAnalyse(preNR_super, isb = false, delta = 1, overlap=4)
forward_vec2  = MAnalyse(preNR_super, isb = false, delta = 2, overlap=4)

src.MDegrain2(src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

The prefiltering should be just strong enough to eliminate all noise.

[cy]

Thank you Didée !

[Blue_MiSfit]

Didee:

In your example of using fft3dfilter - I might worry a bit about banding at high sigma values. Would banding goof up the motion estimation? My guess is probably a lot less than big chunks of dirt and grain

~MiSfit

[Sagekilla]

It actually doesn't hurt ME. the function I made out of his degraining routine for 300 (TemporalDegrain) uses a absurdly high sigma values (sigma=16, sigma2=12, sigma3=10, sigma4=4) but it doesn't hurt ME at all. Only problem where it might hurt is in the limiting portion where banding might seep through, since the 1st degrain stage is limited against what FFT3DFilter did. And FFT3DFilter denoised -a lot-

[Nephilis]

Originally Posted by Didée

DGDecode_mpeg2source("... .d2v", info=3)
ColorMatrix(hints=true, threads=0)
src = last

SetMTMode (2)

preNR = src.FFT3DFilter( with_appropriate_settings )  # as an example

preNR_super = preNR.MSuper(pel=2, sharp=1)
src_super   = src.MSuper(pel=2, sharp=1, levels=1)

backward_vec2 = MAnalyse(preNR_super, isb = true,  delta = 2, overlap=4)
backward_vec1 = MAnalyse(preNR_super, isb = true,  delta = 1, overlap=4)
forward_vec1  = MAnalyse(preNR_super, isb = false, delta = 1, overlap=4)
forward_vec2  = MAnalyse(preNR_super, isb = false, delta = 2, overlap=4)

src.MDegrain2(src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)

The prefiltering should be just strong enough to eliminate all noise.

i think this script with fft3d is too strong and removes many details, especially for not much dirty sources..Isn't it?

[Didée]

The procedure with prefiltering-for-ME is often suited for sources with strong grain/noise. For sources with only low noise, prefiltering-for-ME is mostly not needed.

So, yes, a sledge hammer is not well suited to stick a pushpin into a cork board. However it is well suited to drive stakes into earth.

It is expected that users are aware of the difference between pushpin-into-cork and stake-into-earth. Answering each'n every question beginning with Adam & Eve is too time intensive.

[rkalwaitis]

Didee, If the source isnt that noisey could you use MVflow.

I read about this under MVTools.

It says MVFlow produces no artifacts, like other methods of denoising. Which confuses me because right after the MVFlow portion of the little script they denoise again in another way, DeGrainMedian().

super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
forward_compensation = MFlow(super, forward_vectors, thSCD1=400) # or use MCompensate
backward_compensation = MFlow(super, backward_vectors, thSCD1=400)
# create interleaved 3 frames sequences
interleave(forward_compensation, last, backward_compensation)

DeGrainMedian() # place your preferred temporal (spatial-temporal) denoiser here

selectevery(3,1)

as always I wait for the wisdom of Didee.

[Didée]

Well, that's Off Topic. Answer-in-pictures:

Source:


MCompensate:


MFlow:

[rkalwaitis]

Didee,

My test of it didnt have electricity shooting out of peoples hands, Im jealous.

how did you use it?

MCompensate gave me lots of blocks

[Sagekilla]

rkalwaitis: There's only two denoising functions in MVTools, MDegrain and MDenoise. MCompensate is meant for creating intermediary frames to help temporal denoisers. They're not meant to be used standalone without using a denoiser.

[rkalwaitis]

Perhaps that is why Didee thought I was off base. I was thinking that Mflow was a type of denoising by itself. But now I understand that it has to be used in conjuction with a denoiser. The documentation for MVTools2 stated that the MvFlow motion compensation does not produce any artifacts. It said also it is recommended to use true motion estimation (is strongly suggested)

Of course the documentation only gave me the example below.

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
vectors = MAnalyse(super, isb = false)
compensation = MCompensate(super,vectors) # or use MFlow function here
# prepare blurred frame with some strong blur or deblock function:
blurred = compensation.DeBlock(quant=51) # Use DeBlock function here
badmask = MMask(vectors, kind = 2, ml=50)
overlay(compensation,blurred,mask=badmask) # or use faster MaskedMerge function of MaskTools


But this is compenstation not estimation, right? Im not even sure how I would do a similar script for estimation.

[Sagekilla]

MCompensate and MFlow both create intermediary frames suitable for denoising. Their method of doing so differs slightly: MCompensate uses block based compensation, and MFlow uses pixel based compensation. Because of how they work, you can sometimes get blocking in MCompensate (which can screw up some temporal averaging schemes) but you won't in MFlow.

And yes, compensation is a form of estimation. You're creating a frame, X-0.5, in between X-1 and X, but to do so you need to estimate how everything in the frame moved from X-1 to X.

[rkalwaitis]

So if I understand correctly it would possibly be more beneficial to use Mflow. Of course with a proper denoiser to help avoid artifacts? Im gonna see which is the best type of denoiser to use. Makes sense to start with the MDegrains.

k

[Didée]

The reason why I thought you were off base is: this thread is about how to use pre-filtering for getting a more stable vector field. Kind of an advanced question. Then you hop in with a question about how all of this mo-comp denoising stuff is functioning at all. That's much less advanced, and doesn't fit the topic.

Best way is to try all methods, and judge the results by yourself. The coarse crack-down is this: for denoising purposes, utilizing MFlow will introduce (much) more potential problems than benefit. Look again at the small pics above. That's a "naked" MCompensate vs. a "naked" MFlow. It's obvious which one has more artifacts ... and that's not only a selected case example, but the general characteristic. When building a denoiser that (internally) uses motion compensation, I'd rather chose that method out of two available that produces less artifacs.

The wording "MFlow is artifact free" is highly misleading ... probably it's a relict of the times when block overlapping wasn't yet available. Back then, the normal compensation methods were prone to produce very blocky results, which the "flow" method did not. Nowadays with block overlapping available, the raw compensation quality of the blockbased methods is superior to the flow method.


@ Sagekilla: "compensation is a form of estimation" - that's not a good statement.

a) motion estimation: the procedure of searching motion vectors.

b) motion compensation: the act of shifting a block to a new location, by 100% of the length of its attached motion vector.

c) motion interpolation: the act of shifting a block to a new location, by only a fraction of the length of its attached motion vector.

Step a) has always to be done. Step b) is for fully undoing the motion from one frame to another, so that temporal filters can act safely. Step c) is for partly undoing inter-frame motion, i.e. creating a new intermediate step of motion, mostly for framerate conversion or motion blur effects.

[Fizick]

I never said, that "MVFlow produces no artifacts".
From documentation: "any blocking artefactes"

[rkalwaitis]

Sorry Fizick, you are absolutely correct. Excuse me.

[rkalwaitis]

So if the goal is to always do motion estimation followed by motion compensation then, of course one may need prefiltering to enhance the deonising.

So a suitable script for this may be........

src=last
shrp=awarpsharp()
src_super =src.Msuper()
shrp_super=shrp.Msuper(levels=1)

forward_vectors = MAnalyse(src_super, isb = false)
backward_vectors =Manalyse(src_super,isb=true)
forward_compensation = MCompensate(src_super, forward_vectors, thSCD1=350)
backward_compensation = MCompensate(src_super, backward_vectors, thSCD1=350)
interleave(forward_Compensation, last, backward_compensation)
DeGrainMedian(limitY=2,limitUV=3,mode=2).DeGrainMedian(limitY=2,limitUV=3,mode=2)
SelectEvery(3,1)

Anyone have any tips. I tried to sharpen the image before MSuper in the hopes of keeping a sharper image. Of course Im sure that sharpening prior to does not necessarily count as prefiltering

[Sagekilla]

note aWarpSharp isn't a traditional sharpener! It doesn't work like LimitedSharpenFaster does, so you may get very odd results.

Also, if you read what Didee said, prefiltering is only necessary when there's excessive noise. Unless your video looks like 300, you probably don't need to use prefiltering. If you want to do that sharpening trick Didee posted earlier, you need to use MDegrain, which means you can't use the MCompensate -> Interleave -> DegrainMedian -> SelectEvery method.

[Jawed]

This is the function I developed a while back:

function Killer(clip source, int temporal, int "blksize", int "overlap", int "sharp", int "thSAD", bool "RefineMotion")
{ # Motion compensated denoiser for progressive source clip with prefiltering for strength and repair for de-artefacting
  # Uses MVTools2, ReduceFlicker and RemoveGrain/Repair

blksize       = default(blksize,16)          # blksize value (4, 8 or 16)
overlap       = default(overlap,blksize/2)   # overlap value (0 to half blksize)
sharp         = default(sharp,2)             # 0=bilinear softest, 1=bicubic, 2=Wiener sharpest
thSAD         = default(thSAD,300)           # higher risks motion ghosting and swimming, lower risks blotchy denoising
RefineMotion  = default(RefineMotion,true)   # true means MRecalculate will be used to improve motion vectors

halfblksize   = blksize/2                    # MRecalculate works with half block size
halfoverlap   = overlap/2                    # Halve the overlap to suit the halved block size
halfthSAD     = thSAD/2                      # MRecalculate uses a more strict thSAD, which defaults to 150 (half of function's default of 300)

dct=5

source        = source.assumeframebased()    # MSuper pel=2 is faster with this

# Prefilter the clip
calm          = source.reduceflicker(strength=3,aggressive=true)
calm          = calm.repair(source,mode=1)
calm          = calm.removegrain(mode = 17)
calm          = calm.removegrain(mode = 17)
calm          = calm.removegrain(mode = 17)

calm_super    = calm.MSuper(pel=2, hpad=blksize, vpad=blksize, sharp=sharp)
source_super  = source.MSuper(pel=2, hpad=blksize, vpad=blksize, sharp=sharp,levels=1)
recalculate   = calm.MSuper(pel=2, hpad=blksize, vpad=blksize, sharp=sharp,levels=1)

backward_vec3 = MAnalyse(calm_super, blksize=blksize, isb = true, delta = 3, overlap=overlap, dct=dct)
backward_vec3 = RefineMotion ? MRecalculate(recalculate, backward_vec3, blksize=halfblksize, overlap=halfoverlap, thSAD=halfthSAD) : backward_vec3
backward_vec2 = MAnalyse(calm_super, blksize=blksize, isb = true, delta = 2, overlap=overlap, dct=dct)
backward_vec2 = RefineMotion ? MRecalculate(recalculate, backward_vec2, blksize=halfblksize, overlap=halfoverlap, thSAD=halfthSAD) : backward_vec2
backward_vec1 = MAnalyse(calm_super, blksize=blksize, isb = true, delta = 1, overlap=overlap, dct=dct)
backward_vec1 = RefineMotion ? MRecalculate(recalculate, backward_vec1, blksize=halfblksize, overlap=halfoverlap, thSAD=halfthSAD) : backward_vec1

forward_vec1  = MAnalyse(calm_super, blksize=blksize, isb = false, delta = 1, overlap=overlap, dct=dct)
forward_vec1  = RefineMotion ? MRecalculate(recalculate, forward_vec1, blksize=halfblksize, overlap=halfoverlap, thSAD=halfthSAD) : forward_vec1
forward_vec2  = MAnalyse(calm_super, blksize=blksize, isb = false, delta = 2, overlap=overlap, dct=dct)
forward_vec2  = RefineMotion ? MRecalculate(recalculate, forward_vec2, blksize=halfblksize, overlap=halfoverlap, thSAD=halfthSAD) : forward_vec2
forward_vec3  = MAnalyse(calm_super, blksize=blksize, isb = false, delta = 3, overlap=overlap, dct=dct)
forward_vec3  = RefineMotion ? MRecalculate(recalculate, forward_vec3, blksize=halfblksize, overlap=halfoverlap, thSAD=halfthSAD) : forward_vec3

temporal == 3 ? MDegrain3(source, source_super, backward_vec1, forward_vec1, backward_vec2, forward_vec2, backward_vec3, forward_vec3, thSAD=thSAD) : \
temporal == 2 ? MDegrain2(source, source_super, backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=thSAD) : \
                MDegrain1(source, source_super, backward_vec1, forward_vec1, thSAD=thSAD)
repair(source,mode=17)
}

It's brutally strong hence thSAD defaulting to 300. You may prefer lower

To use it, just do:

Killer(3)

for 3 frames backward and 3 frames forward, or 2 or 1 if you prefer to use less frames.

I prefer to keep it at 3 and tune thSAD between 50 and 300, e.g.:

Killer(3,thSAD=150)

It's pretty slow, too.

Oh and it'll prolly break if you use blksize=4 and RefineMotion=true (the default). Can't remember if I tested that, since blksize=4 is ridiculously slow.

There's a FizzKiller version too. Maybe I should update my FizzKiller thread.

Jawed