LSD: LimitEdge Smooth Denoise | Updated 10/4/10

[TheProfileth]

LimitEdge Smooth Degrain
LSD for short
Original idea came from HQdering and how it used an edgemask to limit the smoothing,
I was looking through Tritical's filters and I saw Tcanny and I knew it would have some practical usage, and so I created this.
Its a multilayered edgemask limited Degrainer/Smoother/Denoiser, that helps to improve compression without damaging details
Its extremely accurate because of the limiting and how sensitive it is, it is even chroma sensitive so it will not oversmooth lightly shaded areas that lack hard edges.
Its uses canny edge detection so anything that messes with edges or lines or even noise (I said it was sensitive) should probably be done after using this filter.
It is very specific in the way it works, it creates a multilevel mask that uses chroma as a meter of how much to smooth, higher chroma means less smoothing.
example here

http://farm5.static.flickr.com/4146/5017047660_16c6f130f7_o.png
http://farm5.static.flickr.com/4109/5015897963_4792185b33_o.png
Darker areas get less smoothing and black areas get none
LSDlite is a moderately stronger version of LSDverylite or LSDvlite for short, LSDvlite will basically cause absolutely zero damage to your source.
LSDlite might cause extremely minor blurring of some soft details but at a relatively nice increase in compression and at zero speed cost.

And of course everyone knows that if you want to test degraining or smoothing you should test it on the oh so lovely Sailor Moon dvds
Comparisons
LSDvlite (http://comparescreenshots.slicx.com/comparison/81780)
LSDlite (http://comparescreenshots.slicx.com/comparison/81779)
LSDvlite VS LSDlite (http://comparescreenshots.slicx.com/comparison/81783)

It also preforms well on bluray noise
BD Comparisons
LSDvlite on Martian Successor Nadesico Bluray (http://comparescreenshots.slicx.com/comparison/82053)
You can call LSDvlite twice for extremely good results (but it is relatively slow, but totally worth it)
Before
http://farm5.static.flickr.com/4104/5018427711_11a286008f_o.png
After LSDvlite.LSDvlite
http://farm5.static.flickr.com/4113/5019034638_612a003783_o.png
It works even better if you remove some of the fluctuating noise with a temporal filter such as fluxsmooth BEFORE you use it
FluxsmoothT(temporal_threshold=4)
LSDvlite.LSDvlite
http://farm5.static.flickr.com/4088/5019043666_e6b9b1b16d_o.png
NOW that is what I call a result, removing the fluctuating noise helps the mask to be more accurate

Download (http://www.mediafire.com/download.php?iggnj2q5imss9x5)
Needed filters (incase you do not have them, I would get them just so you do not end up complaining it does not work) (http://www.mediafire.com/download.php?ue0l85quzolhgqb)
Usage Notes
Usage Notes
## the filter itself really has no hard "parameters" because one of the things I kept from the conversion from HQdering is that the script has no set method of smoothing
## I have found that Tbilateral fits extremely well into what I wanted this to do, and I have configured it to work well with this filter,
## lite and vlite are just different tbilateral settings with the same internals, currently I do not know how to store presets or else I would have them all as 1 filter
## The only thing this shares in common with HQdering is the input system and the maskmerging
## You can actually use any filter you want (within reason, also if you tell a filter to give you a mask as the output as far as I know it will not work)
## If you desire to change what smoother is used then just put Smoother="filtername" and then fill in the parameters
## with params="what ever they should be"
## only issue is if there are quotes already in there then you need to use Chr(34) like this params=Chr(34)+"w2d"+Chr(34)+", 2, 15, 15, 20"
BTW I would really like to be able to make and store preset modes instead of having to have separate scripts for the modes, if someone could show me how to do this easily then I would be very grateful and add in another mode.

Please excuse any errors I have , I will fix them post haste

Please leave feedback and if you have suggestions leave those too

Changelog
LSDvlite 0.6 LSD v0.7 : initial release
LSDvlite 0.61 LSD v0.71 : changed maskedmerge to mt_merge
LSDvlite 0.6 LSD v0.7 : fixed issue with mask

[TheProfileth]

Updated it and added more examples

[Usedocne]

Looks like quite a useful function, thanks.

Unfortunately I haven't encoded any anime in quite a while. But once I do, I'll be giving this a whirl.

[TheProfileth]

Looks like quite a useful function, thanks.

Unfortunately I haven't encoded any anime in quite a while. But once I do, I'll be giving this a whirl.

No problem, I do ALOT of encoding anime with terrible sources, so I have had to deal with lots and lots of filters, so I just saw this as the next logical step, lol well I actually started on this as a proof of concept type of thing and ended up turning it into a extremely useful filter, I am currently employing it in Hare+Guu to great effect, and will most likely use this for other sources too, whats nice about this filter is it can be used for general compression and you do not really have to worry about hurting details (atleast with LSDvlite, and only slightly more so with LSDlite)

I have no idea how well this works on live action video, ( I would guess poorly) but then again most filters preform poorly on live action video.

[TheProfileth]

BTW if you want to see what the mask looks like then you need to use this
input = last
input.tcanny(sigma=1.5,mode=1,t_l=1.5).blur(1).tcanny(sigma=1,mode=1,t_l=2.5)
normalmask = last

normalmask.lsfmod(defaults="slow",edgemode=0).tcanny(sigma=1.5,mode=0)Levels(0, 3.3, 90, 0, 255).blur(0.5)
amplifiedmask = last

normalmask.tcanny(sigma=0.3,mode=1,t_l=1,plane=3).Levels(0, 3.3, 90, 0, 255).removegrain(2).Levels(0, 3.3, 255, 0, 255).daa3
thickmask = last

Overlay(amplifiedmask.Invert(), thickmask, mode="multiply").greyscale


ringingmask = last

mt_merge(input, ringingmask)

[Vitaliy Gorbatenko]

normalmask.lsfmod(defaults="slow",edgemode=0).tcanny(sigma=1.5,mode=0)Levels(0, 3.3, 90, 0, 255).blur(0.5)
'.' Absent before Levels function. Is this correct?

[TheProfileth]

yea that is correct, its a small error on my part, I forgot to correct earlier in the script, it has no actual bearing on the script as far as I know, if I make a new version then I will fix that, just for the sake of keeping things tidy

[mandarinka]

normalmask.lsfmod(defaults="slow",edgemode=0).tcanny(sigma=1.5,mode=0)Levels(0, 3.3, 90, 0, 255).blur(0.5)
'.' Absent before Levels function. Is this correct?

Can someone explain why the script doesn't error out with that?

I got only a very basic understanding of what avisynth accepts and what it barfs out, but when I omit dots like that, it doesn't like it...

[Motenai Yoda]

maskedmerge => masktools v1, mt_merge => masktools v2
why u use the maskedmerge instead of mt_merge?

[Didée]

Can someone explain why the script doesn't error out with that?
because

filter1().filter2().filter3()

is the same as

filter1()
filter2()
filter3()

And since the parser doesn't require linebreaks, the latter is equivalent to

filter1()filter2()filter3()

However, this doesn't work in all cases. When the expression starts with an explicit assignment, like

theclip = filter1().filter2().filter3()

then it's another story.


why u use the maskedmerge instead of mt_merge?
Because that's what mf's original HQDering was using? The variable names are the same too ... and even the levels() parameters are still the same. Just that they don't fit anymore for this function.

There's big effort going on to produce the mask (supersampled lsf, and NNEDI3-based antialiasing, just for the mask!), however the mask is not good. The mask should be basically black/white. What I see in the opening post is a greyish mask - not much black, not much white. All over the frame, only a fraction of the used denoiser is used ... meaning that noise won't be removed, just a little reduced. Also, there are black areas in the middle of no-detail regions ... which will lead to uniform areas having spots that are not denoised at all.

Try FRFun7 on those sample pics. Same detail, much better noise removal, and very much faster.

[mandarinka]

Thanks for the explanation, Didée.

[TheProfileth]

I am just making due with that I have, and atleast in my opinion I am extremely content with the results. You say the mask is not good, but can you point out another smoother that is limited using masks that produces the results you see here? I limited the filter ON PURPOSE, I wanted to allow for MAXIMUM detail preservation with modest denoising and smoothing. Its not meant to completely degrain the entire source with just 1 filter, because when you do that you lose detail, frfun7 is a good filter but it really suffers from lines being killed and it tends to leave sort of a washed out and blurry looking clip, see here for what I mean
http://comparescreenshots.slicx.com/comparison/82226
http://comparescreenshots.slicx.com/comparison/82229
Yea so if you do not care about minor details or lines just disappearing or entire sections of video losing clarity then use FRfun7 as it is as Didée stated much faster and on some sources it works very very well, but if you want more consistently precise and exact or just general use compression you might want to try this.
I am open to suggestions and criticism and will try to improve apon what I have.

[Didée]

Well okay, I won't argue against your justification. Still, in my opinion it's an overly big effort to remove only a fractional amount of noise.

A little script for comparison:
o = last
fr = o.pointresize(o.width+8,o.height+8,-4,-4,o.width+8,o.height+8).frfun7(234.5,6,7).crop(4,4,-4,-4)
fr1 = o.mergeluma(fr,0.85).mergechroma(fr,0.4).mt_lutxy(o,"x y < x 1 + x y > x 1 - y ? ?",U=3,V=3)
fr11 = fr1.repair(fr1.removegrain(2),1) # the result


Source:
http://img97.imageshack.us/img97/9009/animesource.th.png (http://img97.imageshack.us/i/animesource.png/)


Your suggestion: FluxSmoothT(4).LSDvlite().LSDvlite()
(single-threaded: 0.75 fps on an i7-860 !!!!) (on 720x576 PAL)
http://img18.imageshack.us/img18/8338/anime2xlsdvlite.th.png (http://img18.imageshack.us/i/anime2xlsdvlite.png/)


The little frfun7-script:
(single threaded: ~11.0 fps)
http://img821.imageshack.us/img821/7752/animefrfun7custom.th.png (http://img821.imageshack.us/i/animefrfun7custom.png/)


When I compare those, and consider that the little frfun script would render 7 hours where 2*LSDvlite would render 100 hours, then ....

.... then I am very happy that I don't ever deal with Anime content. ;)

[TheProfileth]

the script is undeniably slow, that is not the point though,
its only meant to handle noise in general removal and is adaptable to handle both the small and the huge levels of noise to produce something that will not be drastically different from the source.
Also this script does not leave artifacts like removegrain and is able to remove HEAVY noise or alteast help to lessen it without blurring out the picture. Its less of a smoother and more of noise/grain defuser, it removes the noise and makes the grain less noticeable, it provides excellant compression while still maintaining the details, this filter is not for everyone, hell its not for most people, its for people who do not want to tamper with the source and ONLY want a to increase compression without compromising on quality, regardless of speed.
lol well 100 hours is not quiet reasonable, as most people have atleast somewhat better , for me with my 2.2GHz Quadcore, the script is not too slow and if it is IDC really, aslong as it encodes above .3FPS I am fine with it .
Realistically this filter gets about ~1-5fps depending on the scene, and if you have a worse computer or a better one it will be different.

[TheProfileth]

Updated it and the requirements with the newer versions
also updated both scripts to use mt_merge instead of maskedmerge , should make it a bit faster.

[TheProfileth]

Reverted back, as mt_merge was causing weird issues with the mask being output

[Didée]

Reverted back, as mt_merge was causing weird issues with the mask being output
Why not use mt_merge correctly.

(set u/v parameters)

[TheProfileth]

Why not use mt_merge correctly.

(set u/v parameters)

Because unfortunately I do not know how to use it correctly :(