Motion compensated denoising - need help

[Mystery Keeper]

Here's my motion compensated denoising function. (https://bitbucket.org/mystery_keeper/templinearapproximate-vapoursynth/src/540e78888fcaa3b5144ed5ee8e68ccb55290cade/MCDenoise.py?at=master)
Here's my script. (http://pastebin.com/8P1ASjah)
Here's the sample of the video I'm processing. (http://www.mediafire.com/watch/k53nulffz7kvae7/kanomono_cut.m2v)

For most of video the denoising function works like wonder. But on some frames dark lines on bright background get lighter or vanish completely. Not just in this video.

http://s018.radikal.ru/i521/1502/0b/a0316ec102bft.jpg (http://s018.radikal.ru/i521/1502/0b/a0316ec102bf.png)

See the umbrella spokes. What am I doing wrong? How can I fix it? What can I do to generally improve the denoising function?

[pinterf]

I had similar problems in Avisynth (RemoveDirtMC)
Left: Original, Right: filtered in AviSynth
http://i.imgur.com/dkteXcZm.jpg (http://imgur.com/dkteXcZ)

Somehow in VapourSynth it works, though I use Compensate instead of the not yet ported MFlow.
Left: VapourSynth Right: Original
http://i.imgur.com/sagRrSQm.jpg (http://imgur.com/sagRrSQ)

I called RemoveDirtMC with limit=60

def RemoveDirt(input, repmode=16, remgrainmode=17, _grey=False, limit=10):
core = vs.get_core()
clensed = core.rgvs.Clense(input)
alt = core.rgvs.RemoveGrain(input, mode=[remgrainmode,remgrainmode,1]) # alt= input.removegrain(2)
return core.rdvs.RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=2, gmthreshold=40,dist=3,dmode=2,noise=limit,noisy=4, grey=_grey)
# Alternative settings
# return core.rdvs.RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,dmode=2,noise=limit,noisy=4, grey=_grey)
# return core.rdvs.RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,noise=limit,noisy=12,grey=_grey)
# return core.rdvs.RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=6,cthreshold=8, gmthreshold=40,dist=3,tolerance=12,dmode=2,noise=limit,noisy=12,grey=_grey)

def RemoveDirtMC(input, limit=6, _grey = False):
core = vs.get_core()
qpel = core.rgvs.RemoveGrain(input, mode=[12,0,1]) # blur the luma for searching motion vectors orig avs: mode=12, modeU=-1
i = core.mv.Super(qpel, pel=2) # avs: i=MSuper(clip,pel=2, isse=false)

# blksize 8 is much better for 720x576 noisy source than blksize=16
block_size = 8 #block size of MAnalyze
block_over = 0 #block overlapping of MAnalyze 0! 2 or 4 is not good for my noisy b&w 8mm film source

bvec = core.mv.Analyse(super=i,isb=True, blksize=block_size,overlap=block_over, delta=1, truemotion=True, chroma= not _grey)
fvec = core.mv.Analyse(super=i,isb=False, blksize=block_size,overlap=block_over, delta=1, truemotion=True, chroma= not _grey)
backw = core.mv.Compensate(clip=quad,super=i,vectors=[bvec]) # No MFlow in VS! Try it instead of backw = MFlow(quad,i,bvec)
forw = core.mv.Compensate(clip=quad,super=i,vectors=[fvec])
clp = core.std.Interleave([backw,quad,forw])
clp = RemoveDirt(clp, remgrainmode=2, limit=limit,_grey=_grey)
clp = core.std.SelectEvery(clp,3,1)
return clp

[feisty2]

rgvs.repair

[Mystery Keeper]

rgvs.repair
What is the right way to repair without bringing the noise back?
Why does it happen at all? Shouldn't dark lines on bright background be an easy target for motion estimation?

[GMJCZP]

What is the right way to repair without bringing the noise back?
Why does it happen at all? Shouldn't dark lines on bright background be an easy target for motion estimation?

That was an aspiration of Didée, using the contrast to estimate the motion:
Here (http://forum.doom9.org/showthread.php?p=1567569#post1567569).

But since SAD is present I recommend down thSAD until the problem is gone.

Vapoursynth not know, but if you could adding a prefiltering could compensate the thSAD minor values.

[Mystery Keeper]

Vapoursynth not know, but if you could adding a prefiltering could compensate the thSAD minor values.So far I've failed to find a prefiltering that wouldn't kill the accuracy. Suggestions?

[GMJCZP]

Take a look around.

Here. (http://forum.doom9.org/showthread.php?p=1263083#post1263083)

You can adapt to your needs, for example, use MDegrain3 and other filter (or others) in preNR.

If you have problems with cleaning edges you could try the Camembert function, or one of its variants. Maybe works in your case.

[Mystery Keeper]

If I lower thscd1 - the edges are not destroyed, but the noise gets back. Using RemoveGrain for prefiltering makes things worse. Using DFTTest for prefiltering allows to use lower thscd1, but with the same result - when edges get back, noise gets back as well. Looking further into it.

[GMJCZP]

Try frfun3b(S=8,T=2) in preNR (You can down "S" to increase the strength of the filter, at the expense of increasing the processing time). Or add DeGrainMedian(mode=3) or Deen (2,4).

Please consider using Camembert, as the edges is something that does not touch very well MDegrain.

You could also try ContraSharpening.

[Mystery Keeper]

Try frfun3b(S=8,T=2) in preNR (You can down "S" to increase the strength of the filter, at the expense of increasing the processing time). Or add DeGrainMedian(mode=3) or Deen (2,4).

Please consider using Camembert, as the edges is something that does not touch very well MDegrain.

You could also try ContraSharpening.
We're talking VapourSynth, remember?

[feisty2]

motion estimation does not estimate REAL MOTION, it's designed to do the work for sure, but ain't no perfect, truth is, it collects all matched (similar) blocks (SAD lower than whatever value you set) of frames within the time radius you set together for further process to minimize temporal redundancy, that's pretty much it. it gets nasty when similar blocks are not enough, some not so actually similar blocks (SAD still below the set value tho) would still be assumed similar enough and get matched and shit happens. the sucker punch always hits the clip like hell when motions are complex, you know, fast moving objects, fades...
really no way out of it, "repair" or "dither_limit_dif16" are only two solutions might work on my mind, both would try to keep mild difs (noise removal) and discard wild difs (artifacts)

[feisty2]

BM4D might be a real solution, it matches blocks at both spatial and temporal dimensions, much bigger chance to get enough similar blocks, shame no one ever wrote a plugin for it