Lock stock and two smoking barrels - Dancing noise, unable to remove

[tormento]

I am trying to clean the BD of Lock stock and two smoking barrels but the usual scripts are useless.

Here (http://www.mediafire.com/?uie9la36bpm7322) is a crop of the m2ts.

I have tried to apply the following scripts:

SetMTMode(5)
SetMemoryMax(1000)
LoadPlugin("D:\eseguibili\media\dgdecnv\x64 binaries\DGDecodeNV.dll")
DGSource("E:\in\lock_stock\lock_stock2.dgi")
ChangeFPS(last,last,true)
SetMTMode(2)
SMDegrain (tr=3,thSAD=400,blksize=16,overlap=8,pel=1)

Result (http://www.mediafire.com/?7ie5bhsghknmrks)

SetMemoryMax(1000)
LoadPlugin("D:\eseguibili\media\dgdecnv\x64 binaries\DGDecodeNV.dll")
DGSource("E:\in\lock_stock\lock_stock2.dgi")
ChangeFPS(last,last,true)
MT("""
MCTemporalDenoise(settings="high",useEEDI2=true,bwbh=16,owoh=4,chroma=true,sharp=true,pel=1)
""",threads=4, overlap=8)

Result (http://www.mediafire.com/?l3mbrdsz1e3vn88)

SetMemoryMax(1000)
LoadPlugin("D:\eseguibili\media\dgdecnv\x64 binaries\DGDecodeNV.dll")
DGSource("E:\in\lock_stock\lock_stock2.dgi")
ChangeFPS(last,last,true)
MT("""
MCTemporalDenoise(settings="very high",useEEDI2=true,bwbh=16,owoh=4,chroma=true,sharp=true,pel=1)
""",threads=4, overlap=8)

Result (http://www.mediafire.com/?2fpmijwdjshv6dv)

Could you please give me advice on how to remove the dancing noise? Look at the blue screen...

Thanks ;)

[Didée]

Multilevel-MDegrain?

http://www.mediafire.com/?tbqgtfi7r7mb47v

[Vitaliy Gorbatenko]

Luxurious result! How do you pick up the levels for Multilevel-MDegrain?

[tormento]

Didée, my hat off.

You know you have to post me the avs you used, don't you =)

[dfan]

Multilevel-MDegrain?

http://www.mediafire.com/?tbqgtfi7r7mb47v

amazing!! :script:

[Didée]

Multi"Level" in the sense of using multiple "scalings". The observation was that when downscaling the source to a smallish resolution, then a vanilla MDegrain can produce a very stable result. Hence, it's an obvious approach to first make a small-but-stable-denoised clip, and then work the way upwards to the original resolution.

This is the main filterchain that was used:

DGsource("lock_stock2.dgi")
crop(0,20,-0,-20,true)

o = last
ox = o.width
oy = o.height
ox1 = 1280
oy1 = 704
ox2 = 640
oy2 = 360

sm1 = o.bicubicresize(ox1,oy1) # medium scale
sm2 = sm1.bicubicresize(ox2,oy2) # small scale

D12 = mt_makediff(sm2.bicubicresize(ox1,oy1),sm1,U=3,V=3) # residual of (small)<>(medium)
D1o = mt_makediff(sm1.bicubicresize(ox,oy),o,U=3,V=3) # residual of (medium)<>(original)

lev2 = sm2.MCDegrain(3,soft=true) # filter on smalle scale
up1 = lev2.bicubicresize(ox1,oy1)
up2 = up1.bicubicresize(ox,oy)

D12.MCDegrain(3,up1,soft=false,thSAD=480) # filter on medium scale
lev1 = up1.mt_makediff(last,U=3,V=3)
up3 = lev1.bicubicresize(ox,oy)

D1o.MCDegrain(3,up2,soft=true,thSAD=480) # filter on original scale
up3.mt_makediff(last,U=3,V=3)

return(last)

Also required is this modification of the "MCDegrain" script: (originally made to help an MVTools novice, lately)
function MCDegrain(clip c, int "frames", clip "srch", bool "soft", int "thSAD")
{
frames = default(frames, 2)
srch = default(srch,c)
soft = default(soft,false)
thSAD = default(thSAD,400)
bs = (c.width>960) ? 16 : 8
rg = (c.width<640) ? 11 : (c.width<1200) ? 11 : 20
rg = soft ? rg : 0
super = srch.MSuper(pel=2, sharp=1)
super2 = soft ? c.mt_adddiff(mt_makediff(c,c.removegrain(rg)),U=2,V=2).MSuper(pel=2,sharp=1,levels=1)
\ : c.MSuper(pel=2,sharp=1,levels=1)
bvec3 = MAnalyse(super, isb = true, delta = 3, blksize=bs, overlap=bs/2)
bvec2 = MAnalyse(super, isb = true, delta = 2, blksize=bs, overlap=bs/2)
bvec1 = MAnalyse(super, isb = true, delta = 1, blksize=bs, overlap=bs/2)
fvec1 = MAnalyse(super, isb = false, delta = 1, blksize=bs, overlap=bs/2)
fvec2 = MAnalyse(super, isb = false, delta = 2, blksize=bs, overlap=bs/2)
fvec3 = MAnalyse(super, isb = false, delta = 3, blksize=bs, overlap=bs/2)
(frames<=0) ? c :\
(frames==1) ? c.removegrain(rg).MDegrain1(super2, bvec1,fvec1,thSAD=thsad) :\
(frames==2) ? c.removegrain(rg).MDegrain2(super2, bvec1,fvec1,bvec2,fvec2,thSAD=thsad) :\
c.removegrain(rg).MDegrain3(super2, bvec1,fvec1,bvec2,fvec2,bvec3,fvec3,thSAD=thsad)
return(last) }

There's a bunch of hardcoded assumptions (e.g. scale-resolutions) that should be generalized, and probably some refinements missing (e.g. evolving the edges of the searchclip.)

But in general, it seems to do a reasonable job.

Oh, one point: In particular on 1080-ish FullHD, it is ...

... Slooooow!

[tormento]

Thanks, I'll try it as soon as I'll be at home! ;)

[Nephilis]

Didée; Are you sure about this line ?
rg = (c.width<640) ? 11 : (c.width<1200) ? 11 : 20
Perhaps you meant
rg = (c.width<640) ? 11 : (c.width<1200) ? 12 : 20
or
rg = (c.width<640) ? 11 : (c.width<1200) ? 19 : 20

[Didée]

Flutterby. First I was thinking about 17/11/20, but then left it at 11/11/20. Maybe it should rather be sbr()/11/20, that would be the logical adaption to the frame scales.

Howsoever. A strength option instead of a lazy 100% on/off would be more important than the kernel itself. :p

[Nephilis]

You have to pay mr. jiminy 2 pennys for saying flutterby (http://www.urbandictionary.com/define.php?term=flutterby) :p

[tormento]

... Slooooow!
Indeed. I'll try to MT it. Where do you thing is the step that takes more time to be performed? Resize or MC?

[Didée]

Where do you thing is the step that takes more time to be performed? Resize or MC?
Erh, surely not the resize. There are three calls of MDegrain3, with the 2nd and 3rd call being reliant on the earlier ones.(!!) - Now what do you think where all the time is spent .....


Aahh, you also need a SetMemoryMax command. I found that a serious fps breakdown happens with SMM(500) (which is Avisynth's default upper boundary). With SMM(600) the fps were ~almost~ on par. To have some safety margin, I'd go for a SMM(800) or so.
BTW: Single-threaded, I got 1.33 fps on an i7-860.

Also, after a short re-visit I found the smalle scale actually is set a little too big. (Originally had it at 480x260, then set it to 640x360 and had let it there.) I think that sm2=[576x312] and sm1=[1088x584] should be okay.

The script is heavyweight with ressources. Most probably you will not get it successfully multithreaded in 32bit Avisynth. Maybe in 64bit Avisynth, but there you'll very likely get crashes from the buggyness in itself. :p

If you have enough free HD space, then in this case I'd really go for a (near-) lossless intermediate, with [CPU cores] instances running in parallel.

And finally, it's not like this would be the one & only way that can work out. I just tried the idea (because it hasn't been done yet in this way), and although it seems to work out generally, it's quite possible that other approaches can work out quite as well. With the "traditional" way of FFT3dFilter-prefiltering, maybe the culprit is that the usually-used sigma cascade (16/10/6/4) has too low sigma2/3 for this kind of noise.

[tormento]

What about reducing pel? Do you think it could impact too much about quality?

[Didée]

Loosely guessing:
- The smallest scale should ultimately have pel=2.
- The original(largest) scale probably can get away with pel=1.
- Medium scale: maybe yes, maybe no.

"Try, compare, judge."

[tormento]

Mmm.. the scripts works well. Any idea of why x264 crashes after a random time if I multithread it? It's a pity as I can get almost 4 fps.

No other script I currently use is so greedy, I suppose.

[Didée]

Look again at post#12 - I already predicted that multithreading won't work with this script.
You have to ask the Avisynth fairies to cast a spell making FullHD+MVTools+MultiThreading less ressource hungry. With what we have now, it is like it is. Nothing that could be done.

Be assured my disappointment about this situation is not smaller than yours. However:

Spending two days to run a script single-threaded still is clearly more efficient than spending a week trying to run it multithreaded, to no avail. ;)

[tormento]

You are the wise man, I'll listen to your words of wisdom ;)

[tormento]

Ok, I did it with a MT thread. I compared the result with MCTD(very high) and I found that the Didee script is very good to remove bigger dancing noise, while MCTD is better in smaller dancing noise. As a matter of fact Didee's one produces a bigger file, probably meaning there is more noise/detail left.

Didee, what should I do? Increase thsad in your/mcdegrain script? Process the output again with a "simple" mcdenoise?

Thanks for the time ;)

[Didée]

what should I do?

x264 --preset fast, lay on couch, watch movie. :p

[tormento]

I watched the movie and I was disapponted to see the same behaviour of dancing noise on some "pure colors", such as red, again.

Are you interested in investigating more or should I go to hell? =)

[SilaSurfer]

Didée, very nice job. Tried it on SD Dvd source which was resized before to 720x304. I used scalling based on my resolution and did some tweaking on Mdegrain part. Awasome results. Even better then using prefiltered search clip for my source. This is definitely going to my filter folder. Thanks ;)

[tormento]

x264 --preset fast, lay on couch, watch movie. :p
Didée, another challenge for you: Pi (http://www.mediafire.com/?1f6uc613r8gsvfv).
Do you have some spare time to adjust your script using the newer mdegrain libraries such as the one used in SMDegrain and the work from cretinesdesalpes? Plain SMDegrain is almost useless. Thanks in advance.

[Reel.Deel]

Sorry to bump this thread but I had a question regarding the script in post #6 (http://forum.doom9.org/showthread.php?t=161848#post1512464):


There's a bunch of hardcoded assumptions (e.g. scale-resolutions) that should be generalized, and probably some refinements missing (e.g. evolving the edges of the searchclip.)

"evolving the edges of the searchclip" - I'm not really sure what this means. I take it as apply some sorta edge-enhancement on the search clip(s). I know Didée's no longer around but maybe someone else has a better understanding.

[feisty2]

that's a simple temporal averaged (SAD weighted) sharpening

[Reel.Deel]

that's a simple temporal averaged (SAD weighted) sharpening

Uhh, are you talking about the "evolving the edges of the searchclip" statement or the script in general?

[feisty2]

both, I guess
the script blurs blocks with a large SAD (poor motion match)
denoises and sharpens blocks with an SAD value lower than thSAD
"evolving the edges of the searchclip"
guess he was tryna say he could do some more to "up1", "up2" (like, maybe blur them a little bit? edges are mostly where all the insane SADs come from)
but I'm not quite sure what was he saying either, I got Ds at English tests after all