Masks creation

[Dogway]

Is there a way to make a mask to protect low-mid frequence details, like say a wall in an animation source, so I can denoise the flat colors of character cels?
Maybe through a pixel similarity mask

You can test with this source. The most I got was this mask, by using msmooth denoiser, but I think I can do similar or better inside masktools. (Before making the mask I denoised with mdegrain)
http://www.mediafire.com/?z2gwu6uh5gdvy3z
op=MSmooth(threshold=8, strength=4, highq=true, chroma=false, mask=false, show=false, debug=false)
mt_makediff(op,last,U=-128,V=-128)
mt_lut("x 91 < 255 x 157 > 0 255 x 91 - 255 157 91 - / * - ? ?",U=1,V=1)
mt_binarize(threshold=120).RemoveGrain(19).mt_expand(U=-128,V=-128)
http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_620.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/620.png)
I also would like some insight on how to use mt_motion, basically as a reinforcement for manalyse+mdegrain. I have it using, just Im not sure how it works or take the most out of it.

[mastrboy]

Is there a way to make a mask to protect low-mid frequence details, like say a wall in an animation source, so I can denoise the flat colors of character cels?



Might be what you are looking for: http://forum.doom9.org/showthread.php?p=1443463#post1443463 (I have not tested it at all though)

[Dogway]

Thank you, that's something to look at. But it doesn't look to do anything objective, it just creates a mask based on different factors but not strictly in local pixel similarity. It kinda looks to work, but doesnt.
LEM, mixes some masks upon L for luma, masks bright zones, E, masks out edges through a mask created with msharpen, M, creates a mask for motion based on MAnalyse (interesting). Im curious to know why he didn't use either mt_edge or mt_motion.

In any case the filter Im having problems with is tnlmeans, it denoises this frequency detail too much, so I need to isolate it through a mask. To my surprise a difference mask shows that everything is evenly denoised, so it might not be of much use, although perceptively the wall is the first thing to draw your attention.

raw________________denoised__________________diff
http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_raw-1.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/raw-1.png)http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_denoised.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/denoised.png)http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_diff.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/diff.png)

[Zarxrax]

Maybe you are going about it backwards. Instead of trying to mask the flat areas, try to mask the not flat areas. For example, things like edges are what you want to preserve right? So if you mask those, its easy to filter the other parts of the image.

[Dogway]

Zarxrax: That was my first try, mask through mt_edge, but it doesnt detect all the wall as detail, nor all the flat area as flat (because of noise), so this is kind of plan B, I post an updated test, using as a base a better degrained source with mdegrain.

mask _______________tnlmeans through mask
http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_mask.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/mask.png)http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_mt_edge.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/mt_edge.png)
processed source for test: source (http://www.mediafire.com/?o758y1hh6664p5p) 20Mb

Not so good.
Im going to desist for now until someone unveils some magic secrets of masktools...

[cretindesalpes]

My attempt. A bit rough but should be close enough for rock 'n' roll.

e1 = mt_edge (mode="prewitt", thY1=0, thY2=255)
e2 = mt_edge (mode="min/max", thY1=0, thY2=255)
mt_makediff (e1, e2)
mt_lut ("x 128 - 0 max 256 / 0.25 ^ 256 *")
mt_deflate ().mt_deflate ()
w = width ()
h = height ()
r = 3
BicubicResize (w / (r*2) * 2, h / (r*2) * 2, 1, 0)
TemporalSoften (2, 20, 255)
mt_expand ().mt_inpand ().mt_inpand (mode="both")
BicubicResize (w, h, 1, 0)
mt_lut ("x 10 - 6 *")

MergeRGB (last, last, last) # Display

http://img225.imageshack.us/img225/1808/part100.png

[Dogway]

ay mama thanks a bunch, I didnt have much hope on this one. After a test there were areas I still wished to be denoised, some closer look and found out they all were darkish zones, so I added a luma mask into the mix. This got something in its own, probably useful for others as well. I post the improvement and the code wrapped in a function.
Dark areas are commonly detroyed by denoisers specially spatial ones, so that is a personal compromise, in this case I think Im gonna use it.
mt_merge (Your denoiser,
\ last,
\ flatmask, luma=true, y=3, u=3, v=3)


function flatmask(clip a, int "str", int "Lstr", bool "luma", int "r", bool "debug"){
luma = default(luma,true)
r = default(r, 3)
str = string(default(str, 6))
th = default(Lstr,80)
debug= default(debug, false)

e1 = a.mt_edge (mode="prewitt", thY1=0, thY2=255)
e2 = a.mt_edge (mode="min/max", thY1=0, thY2=255)
mt_makediff (e1, e2)
mt_lut ("x 128 - 0 max 256 / 0.25 ^ 256 *")
mt_deflate ().mt_deflate ()
luma ? mt_logic(last,a.mt_binarize(threshold=th),"min") : last
w = width ()
h = height ()
BicubicResize (w / (r*2) * 2, h / (r*2) * 2, 1, 0)
TemporalSoften (2, 20, 255)
mt_expand ().mt_inpand ().mt_inpand (mode="both")
BicubicResize (w, h, 1, 0)
mt_lut ("x 10 - "+str+" *")
debug ? MergeRGB (last, last, last) : last}

http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_flatmasked.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/flatmasked.png)http://i212.photobucket.com/albums/cc35/Dogway/Misc/th_luma.png (http://i212.photobucket.com/albums/cc35/Dogway/Misc/luma.png)