The DCT Block vs. Grain Anime paradox

[mahsah]

...or whatever...

I have done a lot of searching; I have an anime dvd source that is grainy, but when the grain is cleaned up, DCT blocks appear (shaky blocks is dark areas, right?).

The only solution seems to be adding dither... which seems to make it pointless to even degrain the anime to begin with!

But most anime doesn't look good grainy--it can distort lines, or plain just look bad.

Is grain/dither really the only solution to getting rid of DCT blocks? It seems that its a lose/lose situation -- either have your anime look like it is covered in sand or have dark areas full of dancing blocks.

[Dark Shikari]

Tried dfttest?

[mahsah]

I have not, what settings would you recommend for anime?

[Dark Shikari]

I have not, what settings would you recommend for anime?Try dfttest(sigma=1) and raise/lower sigma from there.

Its like fft3dfilter, but better.

[mahsah]

dfttest isn't really helping... it just seems to replace the dancing blocks with pulsating blobs, like most other denoisers.

[Zanejin]

You could try a motion-compensated dfttest. This is what I'm testing for a similarly grainy and blocky source:

source=last
# Motion-compensation code based on one of Dark Shikari's posts (https://forum.doom9.org/showthread.php?p=1098270#post1098270).
backward_vectors = source.MVAnalyse(isb = true, truemotion=false, delta = 1, idx = 1,overlap=4)
forward_vectors = source.MVAnalyse(isb = false, truemotion=false, delta = 1, idx = 1,overlap=4)
forward_compensation = source.MVCompensate(forward_vectors, idx=1)
backward_compensation = source.MVCompensate(backward_vectors, idx=1)
interleave(forward_compensation, source, backward_compensation)
dfttest(sigma = 0.5)
dull=last
sharp = dull.LimitedSharpenFaster(smode=4, strength = 20, undershoot = 80, ss_x = 1.0, ss_y = 1.0)
Soothe(sharp, dull)
selectevery(3,1)
gradfun2db(1.3)

[mahsah]

That helps a little bit, but I still get blocks that tend to flicker every second or so. Its really strange.

EDIT: I tried the motion compensated fft3dgpu script that you linked to, and it seems to work much better then dfttest. Except for the banding, but gradfun2db can help that.

Also is it better to include the sharpening in the motion compensated section of the script? What about line darkening?

[Adub]

Line darkening can be run after the motion compensation, as it shouldn't gain any difference in the quality of darkened lines if motion was factored in. Edges are edges, whether they are moving or not.

Edit: Also, why is dfftest better than fft3dfilter/gpu?

[TheRyuu]

Try dfttest(sigma=1) and raise/lower sigma from there.

Its like fft3dfilter, but better.

sigma=1 is way to high. Would destroy WAY too many details. As long as the source is at least semi-clean, anything under sigma=0.5 should be fine (even that is high). Once you go higher, you loose too many details.

It's a trade off, but I'd take the little noise over lost details any day of the week.

Also is it better to include the sharpening in the motion compensated section of the script? What about line darkening?

I believe both of those are spatio and not temporal so it wouldn't matter if it was in or not. Only thing it would effect would be speed since it's processing twice as many frames (more then twice I think).

Everything is a trade off. You can have the grain, you can have the blocks, or you can have a little of both. I'd try and keep as many details as possible though no matter which way you go.

[check]

OP, can you post a sample?

sigma=1 is way to high. Would destroy WAY too many details. As long as the source is at least semi-clean, anything under sigma=0.5 should be fine (even that is high). Once you go higher, you loose too many details.
It's a trade off, but I'd take the little noise over lost details any day of the week.

Are you talking about sigma=1 being too high for live action, or for anime?
Also, seeing as the suggestion was to go up or down as needed, it shouldn't matter too much :)

[Didée]

In respect to:

You could try a motion-compensated dfttest. This is what I'm testing for a similarly grainy and blocky source:

source=last
# Motion-compensation code based on one of Dark Shikari's posts (https://forum.doom9.org/showthread.php?p=1098270#post1098270).
backward_vectors = source.MVAnalyse(isb = true, truemotion=false, delta = 1, idx = 1,overlap=4)
forward_vectors = source.MVAnalyse(isb = false, truemotion=false, delta = 1, idx = 1,overlap=4)
forward_compensation = source.MVCompensate(forward_vectors, idx=1)
backward_compensation = source.MVCompensate(backward_vectors, idx=1)
interleave(forward_compensation, source, backward_compensation)
dfttest(sigma = 0.5)
dull=last
sharp = dull.LimitedSharpenFaster(smode=4, strength = 20, undershoot = 80, ss_x = 1.0, ss_y = 1.0)
Soothe(sharp, dull)
selectevery(3,1)
gradfun2db(1.3)

and

EDIT: I tried the motion compensated fft3dgpu script that you linked to, and it seems to work much better then dfttest.


Note that with default settings, dfttest works purely spatial (tbsize=1), so it won't benefit at all from the compensation.
You should instead use dfttest(sigma=[something], tbsize=3) in that script, to actually make use of the motion compensation.


Also is it better to include the sharpening in the motion compensated section of the script?
The sharpening is not affected by motion compensation, but the soothing of the sharpening effect is. Thus there's indeed a point to put the sharpen+soothe combo in the motion compensated section.


Also, dfttest is a bit odd in how it manages some things. It seems that every pixel gets filtered several times, as a consequence of the spatial window overlapping. With the default of sbsize=12/sosize=9, every pixel gets filtered 4 times. That's why the denoising strength of dfttest appears to be much stronger than what one would expect from the specified sigma.

Compare the results of these two lines:

> dfttest(sigma=3, sbsize=12, sosize=9) # default
versus
> dfttest(sigma=3, sbsize=12, sosize=3) # less overlapping


both use the same sigma, so one would expect a similar denoising strength. However, the 1st one does waaaaay more strong denoising than the 2nd one.
I'd call this counter-intuitive.

[mahsah]

Huh, something strange. Using that script that Zanejin posted (with tbsize=3 and sigma=.5), motion became very "jolty", almost like frames were being outright dropped.

I got better results by dropping dfttest into the example script provided with mvtools:

source=last
backward_vectors = source.MVAnalyse(isb = true, truemotion=true, delta = 1, idx = 1)
# we use explicit idx for more fast processing
forward_vectors = source.MVAnalyse(isb = false, truemotion=true, delta = 1, idx = 1)
forward_compensation = source.MVFlow(forward_vectors, idx=1, thSCD1=500) # or use MVCompensate
backward_compensation = source.MVFlow(backward_vectors, idx=1, thSCD1=500) # or use MVCompensate
# create interleaved 3 frames sequences
interleave(forward_compensation, source, backward_compensation)

dfttest(sigma=.5,tbsize=3) # place your preferred temporal (spatial-temporal) denoiser here

selectevery(3,1) # return filtered central (not-compensated) frames only


This does a very good job of temporally stabalizing the clip as well as smoothing noise. While the "blobs" are still there, they are much less noticeable, only moving about every second or so.

EDIT: Also is Limitedsharpenfaster really good for anime? I have found it always introduces some form of haloing. Is seesaw any good for anime?

[Adub]

VMtoon is usually the way to go with Anime. LSF does have a tendency to produce halo's unless using lower settings (obviously).

Give VMtoon at try and see what you think.

[mahsah]

Yeah, but VMtoon is very slow (isn't it's sharpening just a supersampled xsharpen?)

I usually use fastlinedarkenmod...

[Adub]

huh, looks like you are right. I will have to try fastlinedarkenmod. I think I had a bad experience with it in the past, so that's why I wasn't using it.

As to sharpening, it's all a matter of preference/source. If LSF works, then great! You may also want to try awarpsharp, Seesaw(although its rarely used for anime), xsharpen, and whatever else you have up your sleeve.

[TheRyuu]

huh, looks like you are right. I will have to try fastlinedarkenmod. I think I had a bad experience with it in the past, so that's why I wasn't using it.

As to sharpening, it's all a matter of preference/source. If LSF works, then great! You may also want to try awarpsharp, Seesaw(although its rarely used for anime), xsharpen, and whatever else you have up your sleeve.

LSF with smode=3 generally works better for anime from what I've found.

warpsharpening is a matter of preference (some people like it, others don't).

As for a line darkener. try using vmtoon(sharpen=false,thinning=0,strength=35) or there abouts. Use LSF instead as the sharpener (call it after, or before, try it out both ways). I've found vmtoon to have higher quality over fastlinedarkenmod and the likes (Fastlinedarkenmod caused jaggies for me and I just liked the overall look of vmtoon better (WITH SHARPEN=FALSE). It may be a little slower, but the quality is probably worth it.

[mahsah]

What is the difference between the different smodes anyways?

Also jaggies from fastlinedarken mod arn't a big deal, as I am calling aaf() (TWICE, my source is really jaggy) at the end of my script.

Also has anyone tried mvdegrain3 with anime? I noticed the new version of mvtools has it, but I'm not sure how to best use it.

[Zanejin]

Note that with default settings, dfttest works purely spatial (tbsize=1), so it won't benefit at all from the compensation.
You should instead use dfttest(sigma=[something], tbsize=3) in that script, to actually make use of the motion compensation.

Hm, I didn't realize that. Thanks for the note!

What is the difference between the different smodes anyways?

This is well-documented here (http://avisynth.org/mediawiki/LimitedSharpen). I prefer smode = 4 for usual anime-filtering (note: "Each mode has progressively less haloing in its sharpening.").

Also jaggies from fastlinedarken mod arn't a big deal, as I am calling aaf() (TWICE, my source is really jaggy) at the end of my script.

To anti-alias (remove jaggies), you can also try TIsophote. It's faster than AAF and may or may not produce satisfactory results.

You may actually be able to accomplish sharpening, line-darkening, and anti-aliasing all with LimitedSharpenFaster. You can use a high undershoot value (above 80, perhaps) to darken lines and super-sampling (controlled by ss_x and ss_y) to anti-alias.

Also has anyone tried mvdegrain3 with anime? I noticed the new version of mvtools has it, but I'm not sure how to best use it.

MVDegrain3 will just stabilize and remove grain to a greater degree and with greater accuracy than MVDegrain2 at a cost to speed. It's basically the standard MVDegrain algorithm using a radius of 3 frames. The very basic usage would be:

source = last
backward_vector_1 = source.MVAnalyse(isb = true, delta = 1, idx = 1)
forward_vector_1 = source.MVAnalyse(isb = false, delta = 1, idx = 1)
backward_vector_2 = source.MVAnalyse(isb = true, delta = 2, idx = 1)
forward_vector_2 = source.MVAnalyse(isb = false, delta = 2, idx = 1)
backward_vector_3 = source.MVAnalyse(isb = true, delta = 3, idx = 1)
forward_vector_3 = source.MVAnalyse(isb = false, delta = 3, idx = 1)
source.MVDegrain3(backward_vector_1, forward_vector_1, backward_vector_2, forward_vector_2, backward_vector_3, backward_vector_3, idx = 1)

You may be interested in experimenting with the pel and blksize parameters for MVAnalyse.

[mahsah]

Thanks!

But I think I'm gonna stay with my current script, as supersampling or tisophote isn't enough to get rid of the aliasing in my source, as undershoot isn't really enough darkening for my taste.

I ended up with
fastlinedarkenmod(thinning=0, strength=35)
and
limitedsharpenfaster(smode=3, strength=50)