BM3D-r7 | state-of-the-art image/video denoising filter

[asarian]

Okay, thanks for the explanation.

[mawen1250]

It's the V-BM3D that requires lots of memory, for spatial BM3D it's mostly acceptable.
Original NLMeans and BM3D were both designed for Gaussian white noise. In the paper, their models are built upon Gaussian white noise assumption, and their performance is also measured with Gaussian white noise.
From my experience, KNLMeans works great with anime-style sources, it'll smooth flat area and won't filter edges much.
On the contrary, BM3D will filter edges more, resulting in a more uniform de-noising. it's more suitable for non-anime content, and it's also really good to eliminate blocking and ringing in highly-compressed sources.

[feisty2]

Originally Posted by mawen1250

KNLMeans works great with anime-style sources, it'll smooth flat area and won't filter edges much.

depends, generally larger s = smoother edge but more ringings...
s=4 leaves lots of ringing like residual noise around edges on VERY grainy clips
s=0/1 slays at ringing removal, it vaporizes ringings with little costs to the edge sharpness

[mawen1250]

With s=0, NLMeans falls back to something like Bilateral filter with box spatial weighting. Narrowly speaking it's not NLMeans any more (whose weighting is based on neighborhood similarity).

[feisty2]

Originally Posted by mawen1250

With s=0, NLMeans falls back to something like Bilateral filter with box spatial weighting. Narrowly speaking it's not NLMeans any more (whose weighting is based on neighborhood similarity).

right, and actually s=0 and 1 are for different kinds of ringings..
s=0 (like an alternative kind of blurring, weighting on error instead of spatial distance) is good for overshoot kind of stuff
s=1, the real NLMeans is good for mosquito noise

[asarian]

Originally Posted by feisty2

different algorithms, these two work best when combined together, not one replacing another, you could over de-noise the image a tiny little bit with BM3D and then refine it with an NLMeans loop, the result will be stunning and better than both plain BM3D or plain NLMeans

So, I'd do a BM3D (radius1=0), for a Gaussian noise reduction (per frame) first, and then follow up with a KNLmeansCL pass, right?

[mawen1250]

Originally Posted by asarian

So, I'd do a BM3D (radius1=0), for a Gaussian noise reduction (per frame) first, and then follow up with a KNLmeansCL pass, right?

If you want a very clean output, it's right. KNLMeans after BM3D will give a noise-free result with smooth flat area.
Though I think feisty2 was referring to combining them in frequency domain or to using one as the reference for another, which is mainly for less detail loss.

[feisty2]

exactly, I was talking about refining with NLMeans, not simply appending an NLMeans after BM3D

[Elegant]

Would you be better off with BM3D refined with NLMeans over just NLMeans for anime content? Or would their still be too much filtering of the edges?

[asarian]

Originally Posted by feisty2

exactly, I was talking about refining with NLMeans, not simply appending an NLMeans after BM3D

I'm sure you were. Can you give a crude example, though, please, of how you would 'nest' the two like that, though?

[feisty2]

Originally Posted by asarian

I'm sure you were. Can you give a crude example, though, please, of how you would 'nest' the two like that, though?

import vapoursynth as vs
import mvmulti
core = vs.get_core()

fmtc_args           = dict (fulls=True, fulld=True)
msuper_args         = dict (chroma=False, hpad=32, vpad=32, sharp=2, levels=0) 
manalyze_args       = dict (search=3, chroma=False, truemotion=False, trymany=True, levels=0, badrange=-24) 
mrecalculate_args   = dict (chroma=False, truemotion=False, search=3, smooth=1) 

def freq_merge (low, hi, p=8):
    core            = vs.get_core ()
    Resample        = core.fmtc.resample
    MakeDiff        = core.std.MakeDiff
    MergeDiff       = core.std.MergeDiff
    def gauss (src):
        upsmp       = Resample (src, src.width * 2, src.height * 2, kernel="gauss", a1=100, **fmtc_args)
        clip        = Resample (upsmp, src.width, src.height, kernel="gauss", a1=p, **fmtc_args)
        return clip
    hif             = MakeDiff (hi, gauss (hi))
    clip            = MergeDiff (gauss (low), hif)
    return clip

def padding (src, left=0, right=0, top=0, bottom=0):
    core            = vs.get_core ()
    Resample        = core.fmtc.resample
    w               = src.width
    h               = src.height
    clip            = Resample (src, w+left+right, h+top+bottom, -left, -top, w+left+right, h+top+bottom, kernel="point", **fmtc_args)
    return clip

def getvectors (src, pelclip=None, tr=6, pel=1, thsad=400):
    core            = vs.get_core ()
    MSuper          = core.mvsf.Super
    MAnalyze        = mvmulti.Analyze
    MRecalculate    = mvmulti.Recalculate
    supersoft       = MSuper (src, pelclip=pelclip, rfilter=4, pel=pel, **msuper_args)
    supersharp      = MSuper (src, pelclip=pelclip, rfilter=2, pel=pel, **msuper_args)
    vmulti          = MAnalyze (supersoft, overlap=16, blksize=32, tr=tr, **manalyze_args)
    vmulti          = MRecalculate (supersoft, vmulti, overlap=8, blksize=16, tr=tr, thsad=thsad/2, **mrecalculate_args)
    vmulti          = MRecalculate (supersharp, vmulti, overlap=4, blksize=8, tr=tr, thsad=thsad/2, **mrecalculate_args)
    vmulti          = MRecalculate (supersharp, vmulti, overlap=2, blksize=4, tr=tr, thsad=thsad/2, **mrecalculate_args)
    return vmulti

clp = rule6
clp = core.fmtc.bitdepth(clp, bits=32, flt=True)
clp = padding(clp,36,36,36,36)
ref = core.bm3d.VBasic (clp, radius=6, sigma=24.0, block_size=8, block_step=8).bm3d.VAggregate(6, 1)
cln = core.bm3d.VFinal (clp, ref,radius=6, sigma=24.0, block_size=8, block_step=8).bm3d.VAggregate(6, 1)
cln0 = cln

vec = getvectors(cln)
blk = core.std.Expr(cln,"0.5")

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 32, 4, 12, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 16, 3, 10.5, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 8, 2, 9, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 4, 1, 7.5, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 2, 0, 6, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)
cln = freq_merge(cln0,cln,8)

cln.set_output()

[asarian]

Originally Posted by feisty2

[code]
import vapoursynth as vs
import mvmulti
core = vs.get_core()

You're the best! Thank you very much! Very illuminating!

[feisty2]

BUG REPORT

frames  ∈ {[0, radius - 1] ∪ [last frame - radius +1, last frame]} are misplaced after VBasic/VFinal

[mawen1250]

Update r6

VAggregate bug fix: For Gray input and/or float output, frames [0, radius-1] are replaced by frame 'radius'. frames [num_frames - radius, num_frames - 1] are replaced by frame 'num_frames - radius - 1'
Re-licensed to MIT

[mawen1250]

Update r7

Update FFTW to 3.3.5

[MonoS]

Hi, i'm encountering a very strange artifact with this script

import vapoursynth as vs
import nnedi3_resample as edi

core = vs.get_core()

src = core.ffms2.Source("C:/U.S. Theatrical Trailer (Red Band).m2ts23.mkv")

blksize=16
overlap=8
pad = blksize + overlap

deint = core.fmtc.bitdepth(src, bits=16).std.CropRel(left=250, right=250)

deint = core.fmtc.resample(deint, deint.width+pad, deint.height+pad, sw=deint.width+pad, sh=deint.height+pad, kernel="point")

RGB = edi.nnedi3_resample(deint, deint.width, deint.height,csp=vs.RGB48, fast=False)

OPPF = core.bm3d.RGB2OPP(RGB, 1)
OPP = core.fmtc.bitdepth(OPPF, bits=16, flt=False, dmode=1, fulls=True, fulld=True)

super = core.mv.Super(OPP)

bvec2 = core.mv.Analyse(super, isb = True, delta = 2, blksize=blksize, overlap=overlap, truemotion=False)
bvec1 = core.mv.Analyse(super, isb = True, delta = 1, blksize=blksize, overlap=overlap, truemotion=False)
fvec1 = core.mv.Analyse(super, isb = False, delta = 1, blksize=blksize, overlap=overlap, truemotion=False)
fvec2 = core.mv.Analyse(super, isb = False, delta = 2, blksize=blksize, overlap=overlap, truemotion=False)

ref = core.mv.Degrain2(OPP, super, bvec1,fvec1,bvec2,fvec2, 425)
ref = core.fmtc.bitdepth(ref, flt=True)


flt = core.bm3d.VFinal(OPPF, ref, radius=1, matrix=100, sigma=[11,7,7]).bm3d.VAggregate(radius=1, sample=1)
"""
denf = core.bm3d.OPP2RGB(flt, sample=1)

den = core.fmtc.bitdepth(denf, bits=16, flt=False, dmode=0)

den = edi.nnedi3_resample(den[0:55], 946, 720, matd="709", fulld=False, csp=vs.YUV444P16, fulls=True, src_width=1420, src_height=1080, curves="709", curved="709", mats="RGB", sigmoid=True, kernel="spline64") + edi.nnedi3_resample(den[55::], 946, 720, matd="709", fulld=False, csp=vs.YUV444P16, fulls=True, src_width=1420, src_height=1080, curves="709", curved="709", mats="RGB", sigmoid=True, invks=True)
"""
flt.set_output()

The incriminated file is this one https://mega.nz/#!b14RDa6Z!82LdVsl0S...vRjKk1sDqDiQ9U and the artifact can be seen at frame 648 https://abload.de/img/artifactgvq1k.png , [the screen was made with the last line uncommented] the more the luma sigma the more the artifact is visible.

Did i make some mistake in my script or did i find a bug in the implementation?

Thanks for the attention.

[royia]

Anyone encountered mawen1250 lately?

[dipje]

Click on name, view public profile, last activity is 12th November 2016.

I don't think doom9 was his primary forum btw, but not sure about stuff like that.

[Jindadil007]

Originally Posted by feisty2

import vapoursynth as vs
import mvmulti
core = vs.get_core()

fmtc_args           = dict (fulls=True, fulld=True)
msuper_args         = dict (chroma=False, hpad=32, vpad=32, sharp=2, levels=0) 
manalyze_args       = dict (search=3, chroma=False, truemotion=False, trymany=True, levels=0, badrange=-24) 
mrecalculate_args   = dict (chroma=False, truemotion=False, search=3, smooth=1) 

def freq_merge (low, hi, p=8):
    core            = vs.get_core ()
    Resample        = core.fmtc.resample
    MakeDiff        = core.std.MakeDiff
    MergeDiff       = core.std.MergeDiff
    def gauss (src):
        upsmp       = Resample (src, src.width * 2, src.height * 2, kernel="gauss", a1=100, **fmtc_args)
        clip        = Resample (upsmp, src.width, src.height, kernel="gauss", a1=p, **fmtc_args)
        return clip
    hif             = MakeDiff (hi, gauss (hi))
    clip            = MergeDiff (gauss (low), hif)
    return clip

def padding (src, left=0, right=0, top=0, bottom=0):
    core            = vs.get_core ()
    Resample        = core.fmtc.resample
    w               = src.width
    h               = src.height
    clip            = Resample (src, w+left+right, h+top+bottom, -left, -top, w+left+right, h+top+bottom, kernel="point", **fmtc_args)
    return clip

def getvectors (src, pelclip=None, tr=6, pel=1, thsad=400):
    core            = vs.get_core ()
    MSuper          = core.mvsf.Super
    MAnalyze        = mvmulti.Analyze
    MRecalculate    = mvmulti.Recalculate
    supersoft       = MSuper (src, pelclip=pelclip, rfilter=4, pel=pel, **msuper_args)
    supersharp      = MSuper (src, pelclip=pelclip, rfilter=2, pel=pel, **msuper_args)
    vmulti          = MAnalyze (supersoft, overlap=16, blksize=32, tr=tr, **manalyze_args)
    vmulti          = MRecalculate (supersoft, vmulti, overlap=8, blksize=16, tr=tr, thsad=thsad/2, **mrecalculate_args)
    vmulti          = MRecalculate (supersharp, vmulti, overlap=4, blksize=8, tr=tr, thsad=thsad/2, **mrecalculate_args)
    vmulti          = MRecalculate (supersharp, vmulti, overlap=2, blksize=4, tr=tr, thsad=thsad/2, **mrecalculate_args)
    return vmulti

clp = rule6
clp = core.fmtc.bitdepth(clp, bits=32, flt=True)
clp = padding(clp,36,36,36,36)
ref = core.bm3d.VBasic (clp, radius=6, sigma=24.0, block_size=8, block_step=8).bm3d.VAggregate(6, 1)
cln = core.bm3d.VFinal (clp, ref,radius=6, sigma=24.0, block_size=8, block_step=8).bm3d.VAggregate(6, 1)
cln0 = cln

vec = getvectors(cln)
blk = core.std.Expr(cln,"0.5")

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 32, 4, 12, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 16, 3, 10.5, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 8, 2, 9, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 4, 1, 7.5, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)

dif = core.std.MakeDiff(clp, cln)
dif = core.knlm.KNLMeansCL(dif, 0, 2, 0, 6, rclip=cln)
sup = core.mvsf.Super (dif, rfilter=2, pel=1, **msuper_args)
dif = mvmulti.DegrainN (blk, sup, vec, tr=6, thsad=10000)
cln = core.std.MergeDiff(cln, dif)
cln = freq_merge(cln0,cln,8)

cln.set_output()

Hi... Can you please inform where to import the video source...a stupid but newbie's question... My current working script is :

import vapoursynth as vs
import mvsfunc as mvf
import mvmulti
core = vs.get_core()
core.max_cache_size = 12000
video = core.d2v.Source(r'C:\DVD\Sample.d2v')
video = mvf.BM3D(video, sigma=[3,3,3], radius2=0)
video.set_output()

Thanks !!!

[Are_]

Originally Posted by Jindadil007

Hi... Can you please inform where to import the video source...

From feisty2's post, this line:

clp = rule6

rule6 is supposed to be your source filter, but if you are a newbie I don't think using fesity's scripts is a good idea.