Color banding and noise removal

[cretindesalpes]

Overlap: check the content of the planes by converting them to luma only. I don't know how your software converts YV12 to RGB for display, this may be an interpolation issue.

Conversion to YUY2: Don't use Dither_get_msb() to convert from 16 to 8 because it would leave the lsb information aside, use only DitherPost(). Anyway your example doesn't even need 16-bit processing, because you just pointresize (no interpolation) and keep the result in 8 bits. And it's even simpler on Avisynth 2.6: ConvertToYUY2 (chromaresample="point").

[Gavino]

Originally Posted by cretindesalpes

And it's even simpler on Avisynth 2.6: ConvertToYUY2 (chromaresample="point").

Because of differing chroma placement between YV12 and YUY2,
ConvertToYUY2 (chromaresample="point")
has to be followed by
MergeChroma(PointResize(width, height, 0, 1))
to get the desired effect of doubling every chroma line.

[atra dies]

I thought it looked like a resize artifact but I see it at 100% too if that matters which I suspect it doesn't since you say it's an YV12 to RGB issue. I am using AvsPmod 2.2.1 and at sat=0 I don't see it. Avisynth stackvertical isn't doing it, I don't think, and I see it with mdegrain lsb out but since it adds a border I am guessing it doesn't matter for that. I see no signs of it with dither post either (but maybe I don't know what I'm talking about). Used dither display as srgb and still saw it with point kernel.

After finding out about resize and converttoyuy2 bugs for 2.5.8 I don't trust internal filters and want to use dither for almost everything. Point resize doesn't alter lsb so I cut it off and it also doesn't lowpass, right? I don't want to unnecessarily lowpass or supersample the video when processing.

[atra dies]

Nevermind, it went away when I converted to yuy2 using what I posted above although I changed the resize width/height to accept any clip like Gavino's yv12toyuy2.

[tormento]

cretindesalpes could you please put some older version online? I am getting strange glitches in latest encodings I made and I am trying to understand why.

[Are_]

http://ldesoras.free.fr/src/avs/

[Yellow_]

hi, I notice in dither tools docs examples using Gradfun3 as final op to go from 16bit to 8bit instead of a dither function, it is under the simple debanding heading I think, is using Gradfun3 a suitable alternative to a dither method specifically if further processing outside a avisynth is envisaged.

Also when going to RGB48 is a dither method involved, if so which is default, same query for going to 10bit lossless h264.

Cheers

[cretindesalpes]

GradFun3() calls DitherPost() internally, but does other things before to deband the input clip. Dithering is only the process of reducing the bitdepth with less perceptual loss than the naive way. If you don't need debanding, just call DitherPost() on your 16-bit clip.

When you're going to RGB48Y, there is no dithering involved, because dithering is only used when reducing the bitdepth. However Dither_quantize() does dithering and has the same default settings as DitherPost().

[Yellow_]

Great, thanks.

GradFun3() calls DitherPost() internally

Of coarse, slaps head

[tormento]

cretindesalpes: there's a bit of discussion about the glitch I found. If you mind it's there.

[cretindesalpes]

Dither v1.19.0:

• Added kernel inversion to Dither_resize16. Similar to debilinear(), but works differently.
• Added cropping parameters to Dither_srgb_display.
• Fixed a bug preventing to use the 4:1:1 colorspace in some scripted functions.
• MVTools v2.6.0.5: MCompensate, MDegrainN: fixed a bug causing occasionally horizontal magenta stripes in multithreading mode.

[tormento]

Thx. I'll try ASAP.

[XMEN3]

Sorry if it's been asked before...is possible x64 version of this filter?

thanks

[cretindesalpes]

Not a the moment.

[raido]

I stumbled on this thread after searching for "Death Note" and "debanding". I have some debanding and I wanted to get rid of it. I am using the following script...

# functions that load clip here...

animeivtc(mode=1,aa=0,killcomb=2)

crop(0,2,0,0)
blackmanresize(720,400,taps=4)

fft3dfilter(sigma=3,bw=32,bh=32,ow=16,oh=16,plane=4)
limitedsharpenfaster(strength=200)
toon(0.5)
ediaa()
awarpsharp2()
gradfun3(thr=0.4,smode=2)
tweak(sat=1.1)

I still get banding, although it is much less noticable. I remembered reading a while ago that for sources with banding that dithering or adjusting the video levels can fix the problem. Since I'm relativily new to avisynth I was hoping someone here could explain to me what that means or give me a general idea about how to combat banding in general. From what I can tell it is a problem that occurs primarily in areas with low tonal/color range. I read that it can be fixed by dithering or increasing the bits per pixel. I understand that the dither package uses ordered dithering. I'm assuming that the other technique would be to increase the bit per pixel of the image, right? Is increasing the bits per pixel similar to adjusting video levels? Is this an x264 setting? Sorry if these questions have obvious answers, like I said, I'm new to this and I'm just trying to understand the potential solutions to the problems with my video source.

[mastrboy]

The Death Note dvd's should not any noticeable banding as i remember, so you are probably creating the banding with: "fft3dfilter(sigma=3,bw=32,bh=32,ow=16,oh=16,plane=4)"

[TheSkiller]

You can be almost sure that Tweak(sat=1.1) does, more or less, introduce (chroma) banding because it does not work with high precision and dithering.

Therefore, I'd suggest to put Gradfun3() at the end and use SmoothTweak(saturation=1.1) instead.


I don't think adjusting levels would reduce banding, that seems illogical. Well, maybe exept if you adjust the levels so that the banding gets hidden in total blackness.

[raido]

Originally Posted by mastrboy

The Death Note dvd's should not any noticeable banding as i remember, so you are probably creating the banding with: "fft3dfilter(sigma=3,bw=32,bh=32,ow=16,oh=16,plane=4)"

Is there an alternative denoiser that I should try on this anime? Fft3dfilter has always been my goto filter. I'll remove it when I get home and test it. Is there a more subtle alternative? This source has quite a bit of grain in it for my tastes so I went for an aggressive denoiser.

Originally Posted by TheSkiller

You can be almost sure that Tweak(sat=1.1) does, more or less, introduce (chroma) banding because it does not work with high precision and dithering.
Therefore, I'd suggest to put Gradfun3() at the end and use SmoothTweak(saturation=1.1) instead.
I don't think adjusting levels would reduce banding, that seems illogical. Well, maybe exept if you adjust the levels so that the banding gets hidden in total blackness.

I was not aware of this plugin. Thanks! I'll give it a shot. Also, I didn't know what was meant by "adjusting video levels". Does this mean adjusting brightness/contrast/gamma etc. etc.? Hate to ask a novice question, but I'm not a pro like you guys.

[Wilbert]

You can be almost sure that Tweak(sat=1.1) does, more or less, introduce (chroma) banding because it does not work with high precision and dithering.

Therefore, I'd suggest to put Gradfun3() at the end and use SmoothTweak(saturation=1.1) instead.

It didn't in the old days (at least the dithering). Try

Tweak(sat=1.1, dither=true)

[TheSkiller]

Originally Posted by raido

Is there an alternative denoiser that I should try on this anime? Fft3dfilter has always been my goto filter.

There are a dozen of denoising filters and scripts that you could try but my advice to you being new to this is: don't overdo the filtering. A light bit of noise acts as a dither somewhat, filtering a video so that there is absolutely zero noise left makes it easy for banding to become visible. Try it.

If you want to try other denoisers you could give MC_Spuds a shot for example. I have used that on a pretty grainy VHS tape Walt Disney animated movie and got great results.
It's not a very newb-friendly script though, you need tons of plugins.



@ Wilbert Thanks, I didn't know Tweak now comes with a dithering option.