FizzKiller - Death to High Contrast Noise

[Jawed]

[See post 30 for current version]

I've been using MVDegrain2 for the cleaning up of "noise" for a while now. I've noticed that there's one type of noise, high-contrast noise, that utterly defeats MVD2.

I have provided two clips for the purposes of this thread, both from the same film, Concursante:

1. Smoke - this clip features a night scene with smoke blowing across the face of a building. The smoke is brightly lit, showing the kind of fizzy noise I dislike. Additionally other brightly lit surfaces feature this fizz, a great example of this is the man's face at the end of the clip. Smoke and MVD2 tend not to mix well, producing some swimming motion artefacts, too. The flashing lights also create a rather entertaining motion artefact - this can be seen most clearly on the dark diagonal line on the front corner of the ambulance in the latter half of the clip. The full-screen white flashes also turn out to be quite useful tests for obscure reasons…
   
2. Blackboard - several faces in this clip show how annoying this fizzy noise is. The blackboard, with its low contrast scribblings, provides a fairly sensitive test for ghosting and swimming motion artefacts.

http://www.megaupload.com/?d=R0XDV4MT

This is how I've been using MVD2 for a while now:

function MVDegrain2p(clip source, int "blksize", int "overlap", int "sharp", int "thSAD",  int "idx")
{ # Motion compensated denoiser for progressive source clip, adapted from http://avisynth.org.ru/mvtools/mvtools.html
  # uses MVTools plugin
  
blksize = default(blksize, 8)         #  blksize value (4, 8 or 16)
overlap = default(overlap, blksize/2) #  overlap value (0 to half blksize)
sharp   = default(sharp, 1)           #  0=bilinear softest, 1=bicubic, 2=Wiener sharpest
thSAD   = default(thSAD, 400)         #  higher risks motion ghosting and swimming, lower risks blotchy denoising
idx     = default(idx, 1)             #  use various idx for different sources in same script

backward_vec2 = source.MVAnalyse(isb = true,  delta = 2, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
backward_vec1 = source.MVAnalyse(isb = true,  delta = 1, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
forward_vec1  = source.MVAnalyse(isb = false, delta = 1, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
forward_vec2  = source.MVAnalyse(isb = false, delta = 2, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
source.MVDegrain2(backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=thSAD, idx=idx)
}

Normally I call this with:

MVDegrain2p(last, blksize = 16, sharp = 2)

as it's considerably faster than running blksize 8, and the "extra" cleaning power of 16 is acceptable to me. The motion artefacts are reduced with blksize 8 and lowered further with blksize 4 - but on balance I can live with what I get at 16.

For cleaning this high contrast noise I've been inspired by various discussions of "prefiltered" MVD2 using a "calm" clip. I've tried using FFT3DFilter/FFT3DGPU (various block sizes up to 128, bt 1, 3 and 5, sigma from 2 to 64 and various strengths of sigma2/3/4), then tried combining this with RemoveGrain(mode=17), luma only or luma+chroma, in various orderings and counts.

So, this is where I've got to:

function FizzKiller(clip source, int "blksize", int "overlap", int "sharp", int "thSAD")
{ # Motion compensated denoiser for progressive source clip with prefiltering optimised to reduce high-contrast noise
  # Uses MVTools and RemoveGrain

blksize     = default(blksize,8)         # blksize value (4, 8 or 16)
overlap     = default(overlap,blksize/2) # overlap value (0 to half blksize)
sharp       = default(sharp,1)           # 0=bilinear softest, 1=bicubic, 2=Wiener sharpest
thSAD       = default(thSAD,400)         # higher risks motion ghosting and swimming, lower risks blotchy denoising

idx         = 20 # "safe" idx for use within this function only

# Prefilter the clip
calm  = source.invert("Y").levels(0, 0.5, 255, 0, 255, coring = false)
calm  = calm.removegrain(mode = 17)
calm  = calm.removegrain(mode = 17)
calm  = calm.removegrain(mode = 17)

backward_vec2 = calm.MVAnalyse(isb = true,  delta = 2, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
backward_vec1 = calm.MVAnalyse(isb = true,  delta = 1, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
forward_vec1  = calm.MVAnalyse(isb = false, delta = 1, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
forward_vec2  = calm.MVAnalyse(isb = false, delta = 2, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
source.MVDegrain2(backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD = thSAD, idx = idx + 1)
}

I call it like so:

last.FizzKiller(blksize = 16, sharp = 2)

So, any ideas for tweaking this to improve it? Less motion artefacts would always be welcome (my CPU makes blksize 8 uncomfortable, halving encode rate)? Also perhaps it could be made a bit stronger on "flat" areas of the picture, but not at the cost of making the motion artefacts worse. I haven't finished playing with this, but I've been playing so long I thought I'd share the results so far.

I'd be interested in clips that cause problems. I've tested other clips and I have yet more clips to test with...

Jawed

[Sagekilla]

This sounds very much like the degraining filter created by Didee that I turned into a function. I'll test yours against mine, I'm curious as to how different the output is from the two methods. No doubt yours is much faster though, since Temporal Degrain calls FFT3DFilter and then MVDegrain2 twice. Will post results soon, see if there's anything I can do to help improve our functions.

Edit: By the way, try out mediafire next time Much better site to use for files.

[Terranigma]

Yes, but the function you came up with, doesn't fare well against hot pixels or Mosquito noise, so maybe you shouldn't be passing it off as a miracle function.

[Sagekilla]

Found a bug by the way. You have to change the source from clip "source", to clip source. No quotations for your source video! Here's the fixed function until you come around to fix it:

function FizzKiller(clip source, int "blksize", int "overlap", int "sharp", int "thSAD")
{ # Motion compensated denoiser for progressive source clip with prefiltering optimised to reduce high-contrast noise
  # Uses MVTools and RemoveGrain

blksize     = default(blksize,8)         # blksize value (4, 8 or 16)
overlap     = default(overlap,blksize/2) # overlap value (0 to half blksize)
sharp       = default(sharp,1)           # 0=bilinear softest, 1=bicubic, 2=Wiener sharpest
thSAD       = default(thSAD,400)         # higher risks motion ghosting and swimming, lower risks blotchy denoising

idx         = 20 # "safe" idx for use within this function only

# Prefilter the clip
calm  = source.invert("Y").levels(0, 0.5, 255, 0, 255, coring = false)
calm  = calm.removegrain(mode = 17)
calm  = calm.removegrain(mode = 17)
calm  = calm.removegrain(mode = 17)

backward_vec2 = calm.MVAnalyse(isb = true,  delta = 2, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
backward_vec1 = calm.MVAnalyse(isb = true,  delta = 1, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
forward_vec1  = calm.MVAnalyse(isb = false, delta = 1, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
forward_vec2  = calm.MVAnalyse(isb = false, delta = 2, pel = 2, blksize=blksize, overlap=overlap, sharp=sharp, idx = idx)
source.MVDegrain2(backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD = thSAD, idx = idx + 1)
}

[Terranigma]

otoh, mc_spuds handles hot pixels very well, but only with frames above 2. But sometimes, frames 3 & 4 gives unpleasing results to the eyes. I wonder if spuds noticed anything.

[Jawed]

Originally Posted by Sagekilla

This sounds very much like the degraining filter created by Didee that I turned into a function. I'll test yours against mine, I'm curious as to how different the output is from the two methods.

I tried Didee's function, on the way...

No doubt yours is much faster though, since Temporal Degrain calls FFT3DFilter and then MVDegrain2 twice. Will post results soon, see if there's anything I can do to help improve our functions.

I've been ildly wondering if I should recode in mask tools, maybe that's faster.

Edit: By the way, try out mediafire next time Much better site to use for files.

The 100MB limit might be problematic for general usage (it has certainly tripped me up on other big files I've tried to share), though wouldn't have been an issue here. Spanned-RARs yeah, sure, but inertia's powerful.

Jawed

[Morte66]

[Jawed is an old friend of mine.]

What prompted you to use the inverted/levelled clip for motion analysis? What do you reckon to gain by it?

[Jawed]

Originally Posted by Sagekilla

Found a bug by the way. You have to change the source from clip "source", to clip source.

Sigh, forgot that the quotes make an argument "optional".

Thanks,
Jawed

[Jawed]

Originally Posted by Terranigma

otoh, mc_spuds handles hot pixels very well, but only with frames above 2. But sometimes, frames 3 & 4 gives unpleasing results to the eyes. I wonder if spuds noticed anything.

I tried:

last.MC_Spuds(frames=2,strength=5)

but didn't go any higher as I don't have MVDegrain3 and decided that if MVD2 doesn't solve the problem, a longer temporal filter wouldn't be any better.

Jawed

[Didée]

Originally Posted by Terranigma

Yes, but the function you came up with, doesn't fare well against hot pixels or Mosquito noise, so maybe you shouldn't be passing it off as a miracle function.

This function here isn't a miracle, either. Truth is, there're no big surprises to expect. Whether MVDegrain "bites" something or not depends mostly on the SAD of a compensated block. When using a pre-denoised search clip, consequently it depends mostly on the quality of the pre-denoising.

Here, the pre-denoising is done pure spatially by a RG(17) cascade. It works out quite OK for this source, but: like already told in Zep's thread, the caveat is that pure spatial pre-denoising leaves too much flicker in flat areas. That's why Jawed gets those "swimming" artefacts.

Also, the gamma-adjustment done for the search clip isn't a bad idea, but it's not well balanced. gamma=2.0 between 0->0 & 255->255 will map 16->64 ... one quarter of the available gammut isn't used anymore at all, and the bright parts lose information.

[Jawed]

Originally Posted by Morte66

[Jawed is an old friend of mine.]

What prompted you to use the inverted/levelled clip for motion analysis? What do you reckon to gain by it?

I played with various gamma changes to see what happens and reasoning that this noise is problematic due to contrast reasoned that an inverted gamma might do the trick better than just gamma.

Jawed

[Terranigma]

Originally Posted by Jawed

I tried:

last.MC_Spuds(frames=2,strength=5)

but didn't go any higher as I don't have MVDegrain3 and decided that if MVD2 doesn't solve the problem, a longer temporal filter wouldn't be any better.

Jawed

Well, if you don't have access to mvdegrain 3, like me, then instead use flow=true.
So like:
MC_Spuds(frames=3,strength=5,flow=true)

Originally Posted by Didée

Here, the pre-denoising is done pure spatially by a RG(17) cascade. It works out quite OK for this source, but: like already told in Zep's thread, the caveat is that pure spatial pre-denoising leaves too much flicker in flat areas. That's why Jawed gets those "swimming" artefacts.

I've noticed that a few weeks ago. Since then, i've been using very weak pre-filtering settings. Just enough so that the remaining noise (if any) is tolerable.

[Jawed]

Originally Posted by Didée

This function here isn't a miracle, either. Truth is, there're no big surprises to expect. Whether MVDegrain "bites" something or not depends mostly on the SAD of a compensated block.

I think the trick here is that I'm biasing the input data for MVAnalyse so that the SAD in MVDegrain doesn't cause the high contrast noise to be ignored. Though I admit this is justification after the fact.

When using a pre-denoised search clip, consequently it depends mostly on the quality of the pre-denoising.

Here, the pre-denoising is done pure spatially by a RG(17) cascade. It works out quite OK for this source, but: like already told in Zep's thread, the caveat is that pure spatial pre-denoising leaves too much flicker in flat areas. That's why Jawed gets those "swimming" artefacts.

I think I've got less motion artefacts with this function than all other variants of pre-filtered MVD2 I've tried. All forms of pre-filtering that I've tested cause an increase in motion artefacts, when compared with MVD2 used in the default fashion.

Also, the gamma-adjustment done for the search clip isn't a bad idea, but it's not well balanced. gamma=2.0 between 0->0 & 255->255 will map 16->64 ... one quarter of the available gammut isn't used anymore at all, and the bright parts lose information.

It looks like I've done gamma 2 but the two are not the same If I take out Invert("Y") and simply use:

levels(0, 2, 255, 0, 255, coring = false)

I get worse motion artefacts - the bitrate is notably lower, about 3.5% on the Smoke clip. Clearly I could tweak that "2" to be something else, after all the whole thing is empirical... it's just where I landed.

I started experimenting with a gamma change (I set about changing the image into linear space) because I'm always suspicious that image processing software handles gamma incorrectly:

http://www.4p8.com/eric.brasseur/gamma.html

The asymmetry between Invert("Y").gamma(0.5) and gamma(2) makes me suspicious. The error might be in Invert or Levels or it might be in MVTools. Or everywhere.

I've got more experimentation to do...

Jawed

[Jawed]

Originally Posted by Jawed

It looks like I've done gamma 2 but the two are not the same

Where's that :oops: smiley?

I don't know why, but for some reason I convinced myself that Invert.gamma(0.5) is the same as gamma(2). ARGH. ARGH ARGH.

Apologies for that ruse :confused:

Jawed

[Morte66]

Originally Posted by Jawed

don't have MVDegrain3 and decided that if MVD2 doesn't solve the problem, a longer temporal filter wouldn't be any better

Well, I donated my GBP 13.66 and got the beta of mvdegrain3. So, for your reference:

MVDegrain2 17.95fps
MVDegrain3 13.66fps
FizzKiller 16.91fps (replaces borked version)

The script:

SetMTMode(5)
DGDecode_mpeg2source("smoke.d2v",idct=4,info=3)
SetMTMode(2)

Tweak(bright=-16, cont=1.164, coring=false)

DeBlock_QED_mt2( quant1=20, quant2=24, aOff1=2, bOff1=4, aOff2=4, bOff2=8 )

#crop( 10, 80, -6, -80) #original, erroneous
crop( 10, 80, -6, -80, align=true)

#mvdegrain2p (source=last,blksize=16,overlap=8,sharp=2,thSAD=400,idx=1)
#mvdegrain3p (source=last,blksize=16,overlap=8,sharp=2,thSAD=400,idx=1)
FizzKiller (blksize=16,overlap=8,sharp=2,idx=1)

gradfunkmirror(1.2)

[Jawed]

Originally Posted by Morte66

Well, I donated my GBP 13.66 and got the beta of mvdegrain3.

Thanks for posting those clips Joel, nice work. Prepare to SCREAM (hmm, maybe you've already started).

But first, comparing MVD2 and MVD3 I can discern a useful reduction in the strength of the noise in the sky at the beginning of the clip. The rest of the clip makes this comparison very difficult. MVD3 only seems to have very slightly worse motion artefacts, judging by the black line on the ambulance (frames 1352 and 1365) - I'd say ignorable, which is a relief. Hopefully FizzKiller will be quite happy working with MVD3...

Now, to that scream. FK is completely borked on frames 7, 72, 356+357, 435, 508, 610, 678, 713, 753, 818, 991, 1059, 1201, 1237, 1490. I can't reproduce that problem here. The error is very similar to errors I've encountered when doing silly extreme things to the calm clip. I can't find anything "special" about the corresponding frames in the source clip, or about the neighbouring pairs of frames on both sides.

I suppose it's either MVTools beta (did you use it for the FK encode?) or multi-threading. Or some horrid combination of the two. Or, erm...

For what it's worth, FK encodes to 34MB here, at ~2.8fps on my lickle A64 3500+, single core. This is for a slightly larger frame, 704x424 versus 704x416. Here's my script:

global MeGUI_darx = 2361
global MeGUI_dary = 1000
DGDecode_mpeg2source("Smoke.d2v")

crop(0,64,0,-64)

tweak(bright=-16, cont=1.164, coring=false)
DeBlock_QED_mt2( quant1=20, quant2=24, aOff1=2, bOff1=4, aOff2=4, bOff2=8 )
last.FizzKiller(blksize=16,sharp=2)

crop( 4, 10, -12, -14)

gradfunkmirror()

and x264 command line:

--crf 16 --ref 5 --mixed-refs --no-fast-pskip --bframes 8 --b-pyramid --b-rdo --bime --weightb --subme 6 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --me umh --merange 32 --no-dct-decimate --aq-strength 0.3 --aq-sensitivity 5

Jawed

[Morte66]

Originally Posted by Jawed

Now, to that scream. FK is completely borked on frames 7, 72, 356+357, 435, 508, 610, 678, 713, 753, 818, 991, 1059, 1201, 1237, 1490.

The flashgun effect?

I suppose it's either MVTools beta (did you use it for the FK encode?) or multi-threading. Or some horrid combination of the two. Or, erm...

Rebooted, ran it again, problem still there.

Switched to script with FizzKiller single-threaded, problem gone.

I may have mixed single and multi-threaded this afternoon during the smoke sequence. I dimly remember forgetting to switch it on at and "fixing" it some time. I've done a lot of clips today between mvd3 and variance AQ, they run together in my mind...

I've uploaded a new version, single-threaded, no flashgun. You can compare now.

I've been running multithreaded mvd3 since this afternoon on various stuff without problems. But not FizzKiller.

Just did multithreaded mvd3 and mvd2 again on the smoke clip, they work fine.

But FizzKiller seems to have a problem with MT, at least here it does. I note that it is a meaningfully different use of mvtools from mvdegrain2p/3p, e.g. it uses two values for idx instead of one, so one script might expose issues that the other doesn't.

I will work on this some more tomorrow, and try to either find where I screwed up or work up a useful bug report for somebody.

[Jawed]

Originally Posted by Morte66

The flashgun effect?

Yeah. I was like WTF is that when I first watched the clip, I don't remember that happening.

I've uploaded a new version, single-threaded, no flashgun. You can compare now.

Cool, yeah, that looks fine.

But FizzKiller seems to have a problem with MT, at least here it does. I note that it is a meaningfully different use of mvtools from mvdegrain2p/3p, e.g. it uses two values for idx instead of one, so one script might expose issues that the other doesn't.

I suppose the obvious thing to try is someone else's known-good prefiltering MVD2 script. Or null FizzKiller by setting calm=source. Or edit a regular MVD2 script so that it uses two values of idx (or remove all assignments to idx so that each MV function gets a unique idx - though that seems extremely unlikely - actually these all seem unlikely).

I will work on this some more tomorrow, and try to either find where I screwed up or work up a useful bug report for somebody.

Are encodes made with, say, MVD2, identical if using MVTools released versus MVTools beta?

With single-threaded FK, see how frames 1395+1396 look, with gamma set to 0.05. That is much like the problem with MT-FK. The error simply looks like the same blocks (mostly bright ones) being used all over the place - on frame 1396 you can see this in action by changing blksize to 8 or 4.

You didn't scream - I guess that's your hardened beta tester skin coming into its own...

Jawed

[Sagekilla]

It seems odd that FizzKiller is having issues with MT. The function I use, I always use SetMTMode(2) to (mostly) double the speed of the function. Only difference is you're using RemoveGrain as a prefilter -- perhaps that's what has issues with MT?

[Morte66]

*sigh*
Is seems you're meant to add "align=true" to the crop command when using MT, or stuff sometimes goes wonky. It further seems that I never encountered anything that provoked the problem until I tried your FizzKiller script. It's all OK now.