The state of anime denoising filters

[Zarxrax]

I've been around for a number of years now, and have seen a ton of denoising filters come through over the years. If you are anything like me though, you might tend to just find something that works, and then stick with it for ages. Lately, I'm starting to become quite dissatisfied with the denoising filters that I have at my disposal. Deen has been my old standby for about as long as I can remember. It's honestly not that great of a filter though, and I can sometimes see it slightly shifting the color in my video.
I've probably at least tried almost every denoiser out there. Improper usage, the choice of testing material, and other factors might lead one to come to bad conclusions about a particular filter though. For instance, I often hear people rave about RemoveGrain... but I usually can't even tell that it's doing anything to the video. For this reason, I'd like people to share their thoughts on which denoisers they are fond of using on anime material, and why.

[Sagekilla]

You can also try complex script chains which denoise the image and slightly blur edges, with a follow up edge enhancement and sharpening. I remember there being a post a while back about removing noise from fairly flat (it works with semi-detailed anime thougrh) anime, making it sharper, and then upsizing it. It was quite slow though, on the order of 1 - 2 fps.

[cestfait]

I think mvdegrain is the best I've tried for anime-- at least, it compresses best without blocking. I know that technically temporal median filters are supposed to work better, and they look better frame by frame often, but usually block here and there most annoyingly when encoded.

I don't even bother with spatial filters-- x264 does a nice job of stabilizing leftover grain by itself, in my opinion. You can never REMOVE noise without removing detail, so I settle for noise supression (with quite clean results, usually).

However, I always use x264 (with settings to preserve grain), and I don't know much about other codecs. I am finally getting to the point where I am satisfied with my results after a LOT of tests. I shoot for between 1200 and 1500kbps, usually. No doubt mvdegrain3 will seal the deal for any future anime encoding (or at least the new mvtools will speed up the mvanalyse in my current script!).

[Mug Funky]

fwiw, i think the current generation of anime stuff doesn't so much need denoising as deblocking. anime has been all digital for a number of years now, so the only noise introduced would come from encoding (or the bizarre practise of dubbing to digibeta via composite cables, which i hope has been stamped out by now).

however, i've been off the DVD/anime scene for nearly a year now, and have not seen anything new since then. it was getting pretty good quality-wise when i left though.

my current denoise interest is in getting 20+ year old standard 16mm film looking good in HD... so far it's looking like puke :)

[kumi]

Here's a chunk out of a Lucky Star R2 DVD (originally posted here (http://forum.doom9.org/showthread.php?p=1081277#post1081277)):
http://rapidshare.com/files/80202244/sample.demuxed.m2v.html

Would you guys say that needs denoising? Deblocking? All I can personally see is mosquito noise around the edges.

[pablobhz]

Here's a chunk out of a Lucky Star R2 DVD (originally posted here (http://forum.doom9.org/showthread.php?p=1081277#post1081277)):
http://rapidshare.com/files/80202244/sample.demuxed.m2v.html

Would you guys say that needs denoising? Deblocking? All I can personally see is mosquito noise around the edges.

heh, its what i posted ;P
Oh well kumi.
Yeah, looks like it does not need anything. But, when you analyze frame per frame, you start to see noise, sharpening, halos. Halos are still present, even in the best dvds. But i remember that i had in my hands a R2 DVD (Votoms), it was nice. Just needed a colormatrix in order to get perfect in h264. My actual problem is about gradients, nothing more. To be specific, about the vmtoon strength i'm using, since its really destroyng my video =)

btw thanks for all the help you provided in that post. Yeah, i was doing wrong, but learned a lot of things from there ^^

BYe
Pablo

[cestfait]

holy god. I have never seen an anime source so clean! that's amazing. those japanese have actually figured out how to use mpeg2!

I am working with an r2 haibane renmei (r1 has major ghosting, the kind that is not field-related), and the dvd encoders' solution to making it look clean was to pepper the whole thing with not-too-major noise. it blocks terribly when compressed though, and there is as much chromatic noise as luma noise. It's also pretty blurry and light.

Looks like anime dvds have come a long way! Now I know why everyone wants things remastered! :eek:

>.<

[Zarxrax]

Well, yea the newer stuff is mostly VERY nice, and things have come a long way. I'm mainly referring more to the traditional stuff though (almost all of my anime dvds are older titles). Case in point, right now I've been messing around with Millennium Actress. Very beautiful, but its got some film grain on it. After encoding, it results in luminance values flickering all over the frame. I had some amazing results with temporal soften, but it also caused artifacts at settings high enough to remove the noise and flickering.

[DeathAngelBR]

For grainy animes I use fluxsmooth, ttempsmooth and fft3dfilter with sigma=0.75, in this order. I think mvdegrain blurs too much in some high detailed scenes.

[TheRyuu]

I use motion compensated fft3dfilter (low sigma) or mvdegrain plus another temporal one (like temporalcleaner).

I adjust the settings if the anime is clean/grainy, etc....

[Chainmax]

When grain is not as prevalent, I use MVDeGrain2.

For noisier sources, I add DeGrainMedian (mostly with default settings) to the mix. It's an awesome filter that in addition to its main task is also pretty damn good at reducing dotcrawl and blocking. Now, It has been said a lot of times in here that it produces artifacting in the form of unneeded blurring, but I never spotted it. Then again, I wouldn't know where to start looking for it either.


There's a new filter called dfttest, which was recently released by tritical, that seems as good as DeGrainMedian from the couple of tests I did and apparently doesn't have the blurring artifact, but I haven't dabbled with it enough yet (don't even know how to use its temporal portion :o).

Other things I might use are DeBlock_QED_MT2 and gradfun2db. And of course, after resize aWarpSharp followed by LimitedSharpenFaster.

Also, TComb is an amazing rainbow and dotcrawl remover. Another good filter for rainbow removal is FFT3DFilter on the chroma channel. I don't use it for the luma channel after seeing it cause some artifacting when denoising pictures and on a DVD-->HD conversion I was toying with.



...
I always use x264 (with settings to preserve grain)
...

Which settings do you use?

[cestfait]

For grainy animes I use fluxsmooth, ttempsmooth and fft3dfilter with sigma=0.75, in this order. I think mvdegrain blurs too much in some high detailed scenes.

blocking... banding...

... and I think fft3d at a sigma like that removes much more detail than mvdegrain (I stick with thSADs 200 or lower mostly). not every inch of a frame always needs ridiculous denoising. I just can't get those temporal averagers not to block, and fft3d invariably bands like hell. Also, fluxsmoothst even at (1,1) blurs a lot in the motion in many of my sources (for some reason motion-compensating it seems to drastically reduce its power).

oops! didn't see that request there chainmax-- give me a day to revive the old encoding pc and I'll get back to you on the exact settings!

[check]

ft3d invariably bands like hell.
I suspect this is mostly just fft3dfilter revealing banding, which is at least what my experience is.

[cestfait]

however you want to call it, some denoisers "unveil the truth" in a bit more obvious of a way than suits my eyes. I also usually add some gradfun (the scripted kind) at the end, as it has absolutely no effect on the final size with my x264 settings and seems to clean anime up quite nicely (because some aq can actually preserve its effects as a pre-processor!).

oh, by the way chainmax, this is what I have finally settled at for crf encodes:

--crf [18-ish] --deadzone-inter 15 --deadzone-intra 6 --ref 8 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime
--weightb --direct none --subme 7 --analyse all --8x8dct --me umh --threads auto --thread-input --cqmfile "prestige.cfg"
--progress --no-dct-decimate --no-psnr --no-ssim --[THE OUTPUT AND INPUT STUFF] --aq-strength 1 --aq-sensitivity 20

notes to cover my ass:

I have done a number of tests with trellis, no-fast-p-skip, and no aq. I like these settings, and they fit my bitrate.

and, yes, you probably SHOULD max out refs and bframes for anime, but I am a bit of a speed lover and am willing to eat the extra bitrate. :)

@zarxrax

try lowering your scenechange threshold on temporalsoften and/or luma and chroma thresholds. I would also not dip below 5 radius frames for safety with that filter. remember that at the only really safe values for scenechange on clean sources (REALLY low, like 2-4) you wont have much effect at all on high motion scenes.

also try out mcnr_simple2 if you haven't already, a nice scripted version of the whole mv-comped-average method.

still, I would stick with mvdegrain if you want to avoid blocking.


good luck, and HAPPY NEW YEAR!!!

:D

[discofever]

fwiw, i think the current generation of anime stuff doesn't so much need denoising as deblocking. anime has been all digital for a number of years now, so the only noise introduced would come from encoding (or the bizarre practise of dubbing to digibeta via composite cables, which i hope has been stamped out by now).

however, i've been off the DVD/anime scene for nearly a year now, and have not seen anything new since then. it was getting pretty good quality-wise when i left though.


It's mostly good, except for the rare title where someone think's it's good to go with a composite dub.

Except for Haruhi. In order to get around the opening (flashing colours and stars and streaks of light) the Japanese decided to blur it. The US dvd just leaves it blocky...

Now if I could only find the time to clean up some of those transformer episodes...

[Sagekilla]

@cestfait, you can safely enable 16 B-frames without having to worry about losing encoding speed or decoding speed. In fact, it should be faster with 16 B-frames since they can be coded faster and on anime you usually do end up getting more B's when you crank it up to 16.

In general, I never use a setting lower than 16 for B-frames since my PS3 and computer both handle streams with it perfectly fine. Plus, I sometimes get a (slight) increase in speed, and a drop in bitrate. Hope this helps.

[cestfait]

wow! read that a while ago and totally forgot about it!

:thanks:

[[MA]Debito]

*cough*

TemporalSoften(4,4,2,15,2)

*cough*

[Soulhunter]

I suspect this is mostly just fft3dfilter revealing banding, which is at least what my experience is.
Banding is caused by abstinence of adequate bitdepth/dithering... And noise is nothing else than dithering! As you cant do much about the insufficient bitdepth, you have to keep the noise or use something like GradFun :]

[MfA]

Noise is used to create dither but they aren't the same thing.

Ideally a denoiser should function in higher color depth so it can remove the real noise (and dither, but that's not important since it can recreate the gradients) and add new dither at the end. Not too many 16 bpc denoisers out there though.

[Zarxrax]

TTempsmooth seems to be doing a pretty awesome job, so it looks like this one is definitely going to start going into my arsenal :)
TemporalSoften(4,4,2,15,2) did really great too, perhaps even better than ttempsmooth, but with this filter I worry a bit about creating artifacts, so I think I will go with ttempsmooth instead, since it seems to have much better protection against artifacts, although it is a lot slower.

I'm still going through some of these other suggestions, it's looking quite promising though.