Heavy Weight Denoisers, critique? (Anime)

[xekon]

From the denoisers ive tried, I settled on FFT3Dgpu two pass after reading this thread: http://forum.doom9.org/showthread.php?t=139028

Let me start by saying that I dont really care for grain in my anime, some people love it, some do not.
The cleaning power of the two pass FFT3D was the best of everything I had tried.

I started to notice however that it did blur out more detail than I would have liked.
Lines that were fairly clean/sharp in the original were noticeably softer after FFT3D.

So then I had a look at this thread with the chalkboard pictures: http://forum.doom9.org/showthread.php?t=162598

I picked out all of the images that resulted in a strong denoise without losing the fine details. (Mdegrain/Nlmeans/dfttest/MCTemporalDenoise)

Mdegrain seemed to do the best job at retaining detail, but it just wasnt doing Heavy enough denoising for my taste (for anime)

Nlmeans seemed to do an amazing job at denoising and retaining detail, however it was noted it was too slow to be useable in most cases, then I found NLMeansCL on here.

NLMeansCL works best spatialy from what I can tell, so alone this denoises well, but there are still a large amount of fluctuating pixels on grainy sources.(because no temporal)

So I tried using the two together and I really love the result that I am seeing.
It gives me the same amount or more denoising than what I got from the two pass FFT3D but retains detail better!
I even get a smaller file size than two pass FFT3D.(not that file size mattered to me)

original:
http://i1208.photobucket.com/albums/cc361/xekon/sdbz4-orig001096_zps0f37c79f.png

SMDegrain+NLMeansCL:
http://i1208.photobucket.com/albums/cc361/xekon/sdbz0-both001096_zpsbf940ae1.png


FPS when encoding this clip (q9450@3.2+gtx480):
SMDegrain+NLMeansCL 1.11 fps (7,084KB)
SMDegrain 2.70 fps (8,936KB)
NLMeansCL 1.36 fps (9,350KB)
fft3dgpu 3.73 fps (7,255KB)
no denoise-x264 8.93 fps (18,772KB)
original-m2v 23.9 fps (51,900KB)



.avs using both:

MPEG2Source("F:\AviSynth\095.d2v")
ColorMatrix()
inH = height
inW = width
Spline64Resize(inW*2, inH*2).Stab(dxmax = 1, dymax = 1, range = 1, mirror = 15).Spline64Resize(inW, inH)
SMDegrain(tr=3,thSAD=500, RefineMotion=False, Search=5)
NLMeansCL(hC=2.2, A=10, S=3, B=0, plane=4, aa=100)
LSFmod(strength=50)
Masked_DHA(rx=1.5, ry=1.5, darkstr=0.6, brightstr=1, lowsens=50, highsens=50, ss=1.0)
HQDeringmod()
Spline64Resize(640, 480)
GradFun3(0.465)

The attached file has a small test .m2v clip (1,417 frames), 5 comparison encodes of the test clip, the .avs files used, and the .bat used to do the encoding.
http://www.mediafire.com/?e9geqru9q0gajth

I am still relatively new at denoising when compared to some of the Avisynth users with years experience under their belt.

So although I like what I see when I look at the results of Mdegrain + NLMeansCL, I am worried that I might be overlooking something that could use improvement.

So if you notice artifacts, or anywhere I could make an improvement please let me know what you noticed and what you think would improve it.

Also if you manage to speed the SMDegrain+NLMeansCL .avs up at all I would love to hear about that as well.

One Thing that I did notice is that there is some banding on some of the frames, I am still trying to figure out what I need to adjust to resolve that. Looking closer the frames where I am noticing banding, the banding appears to be present in the source, but the amount of grain is pretty heavy so the banding goes unnoticed mostly, I am thinking I could live with the small amount of banding that I am noticing if I had to.

SMDegrain() Mod v2.1d: http://doom10.org/index.php?topic=2178.0
NLMeansCL v0.3.2: http://forum.doom9.org/showthread.php?t=158925
ColorMatrix v2.5: http://web.missouri.edu/~kes25c/
Stab: http://avisynth.org/mediawiki/Image:Stab.avsi
LSFmod v1.9: http://forum.doom9.org/showthread.php?t=142706
Masked DeHalo_alpha: http://forum.doom9.org/showthread.php?t=148498
hqdering_mod_v0.7: http://pastebin.com/SYMK6Mh8
dither 1.22.1: http://forum.doom9.org/showthread.php?p=1386559

[S_Prince]

If Banding Is The Problem Then You can try 'gradfun2db' to deal with that..

u may add the line

gradfun2db()

after the denoising line
Hope this helps..

also u can try dither for banding removal

You can check this link http://forum.doom9.org/showthread.php?t=153589
:)

Regards,

[xekon]

Thank you for the reply. I am currently using GradFun3() from the dither package. cretindesalpes outlines that it is intended to replace GradFun2DB(mod) and that it should have better compressibility. I think it is helping, but maybe there is more that I could do with the Dither package than what I am currently doing. I will have to read over the dither package documentation again and look for some good examples.

It really looks like the banding (or what looks like banding) is part of the original source and not a result of denoising, so that could be the problem. Anytime I have had banding that was a result of denoising GradFun3() has stomped it out nicely, and 10bit x264 does a good job of saving the applied dither.

[yup]

hI xekon!
I think this will be little better.
MPEG2Source("F:\AviSynth\095.d2v")
ColorMatrix()
inH = height
inW = width
NLMeansCL(hC=2.2, A=10, S=3, B=0, plane=4, aa=100)
Spline64Resize(inW*2, inH*2).Stab(dxmax = 1, dymax = 1, range = 1, mirror = 15).Spline64Resize(inW, inH)
SMDegrain(tr=3,thSAD=500, RefineMotion=False, Search=5)
LSFmod(strength=50)
Masked_DHA(rx=1.5, ry=1.5, darkstr=0.6, brightstr=1, lowsens=50, highsens=50, ss=1.0)
HQDeringmod()
Spline64Resize(640, 480)
GradFun3(0.465)
Non local means not sense to size. Also You can increase A and S respectively for high remove noise.
yup.

[xekon]

you think its better to do the spatial denoising before the temporal denoising?

or were you just going for it to be before resizing for Stab()? (more sensitivity)

going to try it out now, thanks for your reply :)

[S_Prince]

Thank you for the reply. I am currently using GradFun3() from the dither package. cretindesalpes outlines that it is intended to replace GradFun2DB(mod) and that it should have better compressibility. I think it is helping, but maybe there is more that I could do with the Dither package than what I am currently doing. I will have to read over the dither package documentation again and look for some good examples.

It really looks like the banding (or what looks like banding) is part of the original source and not a result of denoising, so that could be the problem. Anytime I have had banding that was a result of denoising GradFun3() has stomped it out nicely, and 10bit x264 does a good job of saving the applied dither.

Well Yes.
Gradfun3 can also do the same work as gradfun2db.
anyways.use what which gives you the maximum of results :)


Rgds,

[xekon]

I will give gradfun2db a shot too, thanks :)

[mandarinka]

The critique is pretty simple - they destroy the source. The banding is just one problem .
(And to be blunt, to use gradfun on this source is a sign of you being incompetent - you are basically using the filter only because you screwed up the source before it. Bad thing to do. You should know that cel animation DOES NOT have inherent banding problems, it is always caused by bad filtering like yours is. Sorry.)

It isn't just the banding - they kill detail in low contrast areas - if there are for example black lines in darker/low contrast area, fft3dfilter and similar horrible denoisers will simply blur it all away.

Temporal denoisers that have some semblance of sanity are much much better idea for these kinds of sources, especially if they are motion compensated. Try mdegrain2, it is the sanest approach if you want to decrease the filesize of this source's encode. Maybe look at the SMDegrain wrapper function.

Please don't be one of those ........ who commit this smoothing crime against anime. [I spit on the floor at this point :)]

[xekon]

Lol, thank you for your honesty mandarinka. I completely agree with the bit about small/thin black lines in dark or low contract areas being destroyed by fft3dfilter, I could actually show some comparison pictures where it was doing just that.

It is the reason I am now using nlmeans for my spatial filtering. If I comment out the line for nlmeans and just use smdegrain it still leaves too much of a static/grainy look for me. (personal preference)

Did you notice any lines in particular that got blured away in my test? If so on what frame & area of the picture? Thank you for your response :) I appreciate it.

[mandarinka]

Having a completely smooth output is wrong. That is not how hand-drawn iamge looks. It is supposed to have some sort of texture. If you change it into solid-color areas, it is unnatural and looks like a computer stuff. So a good denoising must leave the frames a bit dirty/textury. Uniformity is an undesirable result.

Heh...
<tyroz> i still don't know why people denoise so much
<tyroz> i think they dont know either :D

[xekon]

ahh I see what ya mean. I actually plan on encoding a few episodes with and without nlmeans and watching them both. Just to see how it looks in motion. before I decide how I am going to encode it for my collection.

Even if I decide to use only smdegrain instead of smdegrain+nlmeans, I still see some very good uses for adding nlmeans into your chain like I have done here.

It could be very beneficial if you were making AMVs or anything for youtube.

I understand what you mean by keeping it how a hand-drawn image looks.

some people denoise more than others because of perceived quality to them (this will differ from person to person, and for some its a matter of file size)

Just like not everyone uses the same crf value when they encode using x264.

[mandarinka]

Well, higher crfs are because some people are more stingy than others :)

[xekon]

ahhh but at what point does it go from picking the appropriate crf value to being stingy? 16? 18? 20? 22? 26? you see where I am going with this? (otherwise I suppose we should all just encode at crf 0?) :)

[TheProfileth]

ahhh but at what point does it go from picking the appropriate crf value to being stingy? 16? 18? 20? 22? 26? you see where I am going with this? (otherwise I suppose we should all just encode at crf 0?) :)

When you get blocking and banding..... didn't really think that needed to be pointed out ;)
Also I agree with tyroz and mandarinka, you gotta maintain at least SOME part of the original feel or else it just looks unnatural in most scenes and in detailed scenes it will look bad.
Either way my main point is that CRF's aren't arbitrary things in general you shouldn't need to go outside of the range of 16-18 to get good quality at a decent size. Any higher crf will look bad any lower will often be more bloat than quality. Then again the other x264 settings are equally if not more important things such as qcomp for example. Here is what I like to use in general for my encodes
--preset veryslow --crf 17.0 --bframes 12 --chroma-qp-offset -2 --qcomp 0.8 --rc-lookahead 50 --merange 32 --direct spatial

[xekon]

Thanks for your reply, my only point was crf gets set by perceived quality, or as you said, as high as the person encoding can set it without noticing issues such as blocking and banding.

I watched a couple full length episodes today with and without nlmeans, and I agree too that it looks better with just smdegrain. I still also believe that nlmeans has some great uses when a smaller file size is critical, as it really did do a MUCH better job than fft3dfilter, It managed to save even small lines in dark areas, unlike fft3dfilter.

I really appreciate the comment about the qcomp, after reading about it I have added it to my encode line as well, should help to preserve quality in fast paced scenes, I also did away with my --ref constraint, I used to use that for device compatibility, but now that I am encoding in 10bit only my htpcs can play it back anyway:

x264_10 --preset veryslow --crf 18 --bframes 16 --no-fast-pskip --psy-rd 0.0 --aq-strength 0.2 --qcomp 0.8

On your encode line, is your override of the veryslow preset for --direct and --rc-lookahead to get a speedboost?

Also what was your reason for limiting your bframes?(speedboost also?) my understanding is the encoder decides how many it will use and that the setting is really only a limiter, and that with a higher bframe setting it can be beneficial sometimes, I guess it really just depends on the source.

I have read that there are instances where a higher --merange can result in worse quality instead of better, I am not sure at what point that is, so I have always left it as the veryslow preset.

If you take into account that many people actually still actively use fft3dfilter and fft3dgpu and like the results, and that using this method retains more detail at a smaller file size then FFT3D, I dont see what is not to like for a lot of people. (other than the slower encoding speed of coarse)