Color banding and noise removal [Archive] - Page 12

cretindesalpes

21st June 2012, 09:00

The default transfer curve in the gamma-linear functions has always been sRGB, and still is. It is mostly used for still pictures and computer display. For SD video, use the "601" curve, which is the same as for HD. I think these formulas are right unless I did something wrong and nobody noticed. Anyway the video curve is clearly different from the sRGB one, not just for the darkest shades. It looks brighter in general, when going from gamma to linear.

Also, thank you for reporting the copypasta mistake in the documentation.

atra dies

21st June 2012, 22:32

Ugh this is why we lowly end users need example scripts explaining what to use for SD or HD. Now I have to do some encodes over, but only a couple HAHAHAHA!, cause I waited and did things that didn't require a resize until I was sure of what was going on and what settings to use (and saved my vafs). I tell you what though, I compared a default to a 601 linear resize and didn't see a difference. But when I put 601 on to linear and srgb back to gamma then there were big problems.

OK it's my fault cause I saw it was srgb and misread a page when checking to see if that meant 601 too (or the page had it wrong) but I think 601/709 should be default cause avisynth is mainly for yv12 digital video IMO. At the time 601 was not listed and I thought 709 was just for 709.

I also wish for a 16bit colormatrix mod cause the convert to rgb solution for 709 to 601 is really slow (for rgb48) and doesn't look all that different from tritical's Colormatrix plugin as it is.

Edit: OK I just saw this wi-pedi page (http://en.wikipedia.org/wiki/File:SRGB_gamma.svg) and the difference between the srgb and 601 curves is the red and black respectively, right? They don't look much different to me. :confused:

cretindesalpes

22nd June 2012, 08:06

The black curve on the wikipedia page is the ideal gamma 2.2 transfer curve. Here are the transfer curves that the Dither gamma functions use:

http://sadpanda.us/images/1033118-45XNOXO.png

Akai-Shuichi

22nd June 2012, 13:28

I don't know if this is the right place to post this, but I was wondering is there any way at all that it is possible to fix the banding in this?
http://i49.tinypic.com/30sjzt3.jpg

I have tried GradFun3, but it doesn't seem to do much to it... I'm kind of noobish at avisynth for the record.

Are_

23rd June 2012, 02:52

Hi,

I found gradfun3() speed is inconsistent between modes on my cpu (smode 2 is way faster than it should be).

I ran a few short tests using this simple script:
ColorBars(pixel_type="YV12")
trim(0,1000)
gradfun3(smode=3)
Test report on Windows 7 (http://paste.kde.org/506108/)
Test report on Linux (wine) (http://paste.kde.org/506120/)
Test report on Windows 7 (laptop) (http://paste.kde.org/506156/)

On Linux it only gets worst :(

At first, I was thinking it was cause my amd cpu on my main box, because a friend of mine with an I5 reported to me he was getting the expected fps from the readme, but the quick test on my i5 laptop reported about the same results as my main box. Is the speed of smode 2 ok?

All test were done with last build of Avisynth 2.6 MT and the latest version of all plugins (I also tested a little Avisynth 2.5.8 but it was the same).

atra dies

23rd June 2012, 03:04

This happened to me before where I thought "this is way faster than I remember" then I realized I had smode=3 which is an "undocumented mode". 2 is the bilateral slow one. In the code you just posted, it says smode=3, which was fast for me months ago when it happened.

Akai-Shuichi, looks like jpeg or mpeg artifacts which I use mdegrain almost always and that smooths it a little but something stronger would be deblock_qed or deblock or some type of smoothing. You could use the smoothing in smoothadjust16 as well as adjust levels.

Keiyakusha

23rd June 2012, 04:24

cretindesalpes

23rd June 2012, 11:14

smode 3 is an experiment I started but I was unhappy with it. I had to make a new release for other reasons and I decided to leave this unfinished code in the script so I could continue working on it later. I thought it was funny to mention a dissuasive "undocumented mode" in the doc, but it is even funnier to see that people actually use it (http://www.google.fr/search?hl=all&q="gradfun3"+"smode%3D3")!

As for speed, smode 2 is not that slow nowadays, mainly because partial sampling is activated for the bilateral filter. Set subspl=1 for maximum slowness. It also relies on threading from avstp.dll, so the discrepancies may be coming from that too.

Are_

23rd June 2012, 17:40

Oh I see, thx for the clarification. Even at the maximum radius (48), smode 2 is still faster than smode 1, that's a pretty impressive speed boost from initial versions.

Thx for your great tools, now I can use them with more relief :)

atra dies

25th June 2012, 22:54

Maybe I'm being dumb and paranoid here, but is convert to 16 supposed to have this overlap at the border?

And also I want to double the chroma instead of interpolating it to convert to yuy2. Is this right for 720x480 clips? I'm pretty sure it is but I don't want to make a stupid mistake since I mainly do normal, simple things with avisynth.

u=UToY().Dither_convert_8_to_16().dither_resize16(360,480,kernel="point").Dither_get_msb ().converttoyuy2
V=VToY().Dither_convert_8_to_16().dither_resize16(360,480,kernel="point").Dither_get_msb ().converttoyuy2
YToUV(U, V, last.converttoyuy2)

cretindesalpes

26th June 2012, 07:01

Overlap: check the content of the planes by converting them to luma only. I don't know how your software converts YV12 to RGB for display, this may be an interpolation issue.

Conversion to YUY2: Don't use Dither_get_msb() to convert from 16 to 8 because it would leave the lsb information aside, use only DitherPost(). Anyway your example doesn't even need 16-bit processing, because you just pointresize (no interpolation) and keep the result in 8 bits. And it's even simpler on Avisynth 2.6: ConvertToYUY2 (chromaresample="point").

Gavino

26th June 2012, 09:12

And it's even simpler on Avisynth 2.6: ConvertToYUY2 (chromaresample="point").
Because of differing chroma placement between YV12 and YUY2,
ConvertToYUY2 (chromaresample="point")
has to be followed by
MergeChroma(PointResize(width, height, 0, 1))
to get the desired effect of doubling every chroma line.

atra dies

26th June 2012, 22:42

I thought it looked like a resize artifact but I see it at 100% too if that matters which I suspect it doesn't since you say it's an YV12 to RGB issue. I am using AvsPmod 2.2.1 and at sat=0 I don't see it. Avisynth stackvertical isn't doing it, I don't think, and I see it with mdegrain lsb out but since it adds a border I am guessing it doesn't matter for that. I see no signs of it with dither post either (but maybe I don't know what I'm talking about). Used dither display as srgb and still saw it with point kernel.

After finding out about resize and converttoyuy2 bugs for 2.5.8 I don't trust internal filters and want to use dither for almost everything. Point resize doesn't alter lsb so I cut it off and it also doesn't lowpass, right? I don't want to unnecessarily lowpass or supersample the video when processing.

atra dies

28th June 2012, 02:56

Nevermind, it went away when I converted to yuy2 using what I posted above although I changed the resize width/height to accept any clip like Gavino's yv12toyuy2.

tormento

6th July 2012, 09:19

cretindesalpes could you please put some older version online? I am getting strange glitches in latest encodings I made and I am trying to understand why.

Are_

6th July 2012, 13:16

http://ldesoras.free.fr/src/avs/

Yellow_

7th July 2012, 15:16

hi, I notice in dither tools docs examples using Gradfun3 as final op to go from 16bit to 8bit instead of a dither function, it is under the simple debanding heading I think, is using Gradfun3 a suitable alternative to a dither method specifically if further processing outside a avisynth is envisaged.

Also when going to RGB48 is a dither method involved, if so which is default, same query for going to 10bit lossless h264.

Cheers

cretindesalpes

8th July 2012, 10:09

GradFun3() calls DitherPost() internally, but does other things before to deband the input clip. Dithering is only the process of reducing the bitdepth with less perceptual loss than the naive way. If you don't need debanding, just call DitherPost() on your 16-bit clip.

When you're going to RGB48Y, there is no dithering involved, because dithering is only used when reducing the bitdepth. However Dither_quantize() does dithering and has the same default settings as DitherPost().

Yellow_

8th July 2012, 12:03

Great, thanks.

GradFun3() calls DitherPost() internally

Of coarse, slaps head :o

tormento

9th July 2012, 07:49

cretindesalpes: there's a bit of discussion about the glitch I found. If you mind it's there (http://forum.doom9.org/showthread.php?t=165306).

cretindesalpes

18th July 2012, 20:41

Dither v1.19.0 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Added kernel inversion to Dither_resize16. Similar to debilinear(), but works differently.
Added cropping parameters to Dither_srgb_display.
Fixed a bug preventing to use the 4:1:1 colorspace in some scripted functions.
MVTools v2.6.0.5: MCompensate, MDegrainN: fixed a bug causing occasionally horizontal magenta stripes in multithreading mode.

tormento

19th July 2012, 19:21

Thx. I'll try ASAP.

XMEN3

20th July 2012, 20:47

Sorry if it's been asked before...is possible x64 version of this filter?

thanks

cretindesalpes

21st July 2012, 07:48

Not a the moment.

raido

26th July 2012, 13:59

I stumbled on this thread after searching for "Death Note" and "debanding". I have some debanding and I wanted to get rid of it. I am using the following script...

# functions that load clip here...

animeivtc(mode=1,aa=0,killcomb=2)

crop(0,2,0,0)
blackmanresize(720,400,taps=4)

fft3dfilter(sigma=3,bw=32,bh=32,ow=16,oh=16,plane=4)
limitedsharpenfaster(strength=200)
toon(0.5)
ediaa()
awarpsharp2()
gradfun3(thr=0.4,smode=2)
tweak(sat=1.1)

I still get banding, although it is much less noticable. I remembered reading a while ago that for sources with banding that dithering or adjusting the video levels can fix the problem. Since I'm relativily new to avisynth I was hoping someone here could explain to me what that means or give me a general idea about how to combat banding in general. From what I can tell it is a problem that occurs primarily in areas with low tonal/color range. I read that it can be fixed by dithering or increasing the bits per pixel. I understand that the dither package uses ordered dithering. I'm assuming that the other technique would be to increase the bit per pixel of the image, right? Is increasing the bits per pixel similar to adjusting video levels? Is this an x264 setting? Sorry if these questions have obvious answers, like I said, I'm new to this and I'm just trying to understand the potential solutions to the problems with my video source.

mastrboy

26th July 2012, 15:12

The Death Note dvd's should not any noticeable banding as i remember, so you are probably creating the banding with: "fft3dfilter(sigma=3,bw=32,bh=32,ow=16,oh=16,plane=4)"

TheSkiller

26th July 2012, 15:16

You can be almost sure that Tweak(sat=1.1) does, more or less, introduce (chroma) banding because it does not work with high precision and dithering.

Therefore, I'd suggest to put Gradfun3() at the end and use SmoothTweak (http://forum.doom9.org/showthread.php?t=154971)(saturation=1.1) instead.

I don't think adjusting levels would reduce banding, that seems illogical. Well, maybe exept if you adjust the levels so that the banding gets hidden in total blackness. :p

raido

26th July 2012, 17:23

The Death Note dvd's should not any noticeable banding as i remember, so you are probably creating the banding with: "fft3dfilter(sigma=3,bw=32,bh=32,ow=16,oh=16,plane=4)"

Is there an alternative denoiser that I should try on this anime? Fft3dfilter has always been my goto filter. I'll remove it when I get home and test it. Is there a more subtle alternative? This source has quite a bit of grain in it for my tastes so I went for an aggressive denoiser.

You can be almost sure that Tweak(sat=1.1) does, more or less, introduce (chroma) banding because it does not work with high precision and dithering.
Therefore, I'd suggest to put Gradfun3() at the end and use SmoothTweak (http://forum.doom9.org/showthread.php?t=154971)(saturation=1.1) instead.
I don't think adjusting levels would reduce banding, that seems illogical. Well, maybe exept if you adjust the levels so that the banding gets hidden in total blackness. :p

I was not aware of this plugin. Thanks! I'll give it a shot. Also, I didn't know what was meant by "adjusting video levels". Does this mean adjusting brightness/contrast/gamma etc. etc.? Hate to ask a novice question, but I'm not a pro like you guys.

Wilbert

26th July 2012, 21:55

You can be almost sure that Tweak(sat=1.1) does, more or less, introduce (chroma) banding because it does not work with high precision and dithering.

Therefore, I'd suggest to put Gradfun3() at the end and use SmoothTweak(saturation=1.1) instead.

It didn't in the old days (at least the dithering). Try

Tweak(sat=1.1, dither=true)

TheSkiller

27th July 2012, 14:25

Is there an alternative denoiser that I should try on this anime? Fft3dfilter has always been my goto filter.There are a dozen of denoising filters and scripts that you could try but my advice to you being new to this is: don't overdo the filtering. ;) A light bit of noise acts as a dither somewhat, filtering a video so that there is absolutely zero noise left makes it easy for banding to become visible. Try it.

If you want to try other denoisers you could give MC_Spuds (http://avisynth.org/mediawiki/MC_Spuds) a shot for example. I have used that on a pretty grainy VHS tape Walt Disney animated movie and got great results.
It's not a very newb-friendly script though, you need tons of plugins. :p

@ Wilbert Thanks, I didn't know Tweak now comes with a dithering option.

raido

28th July 2012, 14:34

I tried MC_spuds and I keep getting a error that says manalyse does not have an argument named "pel" I have mvtools2 and I can see in the documentation that Manalyse does indeed not have an argument named pel. Why is this passed as an argument to the function?

StainlessS

28th July 2012, 14:40

mvtools v1 uses pel arg in MVAnalyse().

MVAnalyse (clip, int "blksize", int "blksizeV", int "pel", int "level", int "search", int "searchparam", int "pelsearch", bool "isb", int "lambda", bool
"chroma", int "delta", bool "truemotion", int "lsad", int "pnew", int "plevel", bool "global", int "pzero", int "overlap", int "overlapV", string
"outfile", int "sharp", int "rfilter", clip "pelclip", int "dct", int "divide", bool "mc", int "idx", int "sadx264")

v2

MAnalyse (clip super, int "blksize", int "blksizeV", int "levels", int "search", int "searchparam", int "pelsearch", bool "isb", int "lambda", bool
"chroma", int "delta", bool "truemotion", int "lsad", int "plevel", bool "global", int "pnew", int "pzero", int "pglobal", int "overlap", int "overlapV",
string "outfile", int "dct", int "divide", int "sadx264", int "badSAD", int "badrange", bool "isse", bool "meander", bool "temporal", bool "trymany")

raido

29th July 2012, 00:37

Ah, I see....

/slapshead

I added the mvtools listed here (http://avisynth.org/mediawiki/MC_Spuds) which links to mvtools2. When the script said it couldn't find MVAnalyse, I thought it meant MAnalyse.

raido

29th July 2012, 20:54

Okay I did a few encodes with denoisers such as fft3dfilter, mc_spuds, and dfttest (and zzz_denoise). I was hoping to find a combination that removed the noise without introducing color banding but I wasn't able to do this very well. It seems that what everyone in this thread has been saying all along is true; that you have to leave in some grain in order to combat banding. The only way I was able to do both was using the following technique...

My source had minor banding in it but running fft3dfilter(sigma=3,bw=32,bh=32,ow=16,oh=16,plane=4) resulted in even more color banding. I compensated for this by following it up with gradfun3(thr=0.96) which removed the banding. When I encoded it using the x264 animation preset I still got banding. Then I found a couple of threads that recommended raising AQ, raising Psy-RD, and raising Psy-Trellis. I increased these values to 1.2, 1.00, and 0.8 respectivily. I encoded using high profile with a crf value of 18. My encoded video was much larger but the dithering was preserved. There was no noise and color banding was virtually eliminated. The cost of this was a large file size and fairly aggressive x264 settings that preserved the dithering. Personally, the picture still looks a little soft but I won't sharpen it because that seems to bring out the banding again.

jmartinr

30th July 2012, 09:05

The cost of this was a large file size and fairly aggressive x264 settings that preserved the dithering. Personally, the picture still looks a little soft but I won't sharpen it because that seems to bring out the banding again.

Was the result any better than just no filtering at all and encoding with crf 18?

raido

30th July 2012, 13:08

I won't make the noob mistake of saying my encodes are better since the word better invariable boils down to "in what way".

I was in a position where I was forced to choose between leaving in the grain/noise (i.e. no filtering or very minor smoothing/grain removal) or taking the grain out and dealing with color banding. I hate both grain and color banding. If you are the type that can tolerate either color banding or noise, then no, my result will not look any better. However, if you want a denoised and debanded result, then yes, my result will look better. My result is 856x480 and it even looks pretty good on my HD TV. I guess the trade off is file size, my files went from 65-70 MB to 185 MB when I increased AQ, Psy-RD, and Psy-Trellis.

TheSkiller

30th July 2012, 13:15

raido

30th July 2012, 18:20

Another option would be to deband during playback, using ffdshow (either with it's "Deband" option or by implementing Gradfun3 or Gradfun2dbmod via the AviSynth option). Of course, that's only possible when using a PC for playback. File sizes would be just as small as they used to be and chances are the played back video looks even better debanded.

Unfortunately I use XBMC for my playback. It uses Mplayer. I read that Mplayer does have a deband on playback option but that it is not implemented in XBMC.

Edit: Btw, GradFun3(thr=0.96) is quite aggressive, it will soften the picture in certain areas.

Quite right. I think it is on the soft side. I tried doing some sharpening after gradfun3, but that just reintroduced the banding when encoding. I tried several sharpeners but none seemed to work. In my opinion, the picture is softer, but it was tolerable (well, more tolerable than the banding). I don't know enough about avisynth but I was hoping that there was a way to create a mask that selected areas with low tonal range and applied the debanding in those areas. I may play around with lower thr values in the future.

cretindesalpes

30th July 2012, 18:28

I was hoping that there was a way to create a mask that selected areas with low tonal range and applied the debanding in those areas.
That's what GradFun3 does; it automatically sets thr_det according to thr. If you need to keep thr high because of important banding steps, try to lower thr_det a bit (2 or 3, but not less) to preserve more details and textures.

jmartinr

30th July 2012, 19:07

@raido

fft3d introduces banding. I'd recommend motion adaptive denoising:

# degrain
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1, forward_vec1, backward_vec2, forward_vec2, thSAD=400, lsb=true)
DitherPost()

raido

31st July 2012, 13:39

@cretindesalpes

Ah, good to know. I found this to work quite set to 3. Thank you for the wonderful plugin!

@jmartinr

I pulled out fft3dfilter and used your script. It was really REALLY effective a "calming" the grain. I use the term calming because it still seemed to keep a minor amount, enough to prevent color banding but not enough to be very noticeable. The result was comparable to what I was getting using my aggressive gradfun3 script except that yours preserved detail much better. The thing is, I have no idea what that script does. Can you give me a brief explanation? It looks like it creates a "super" clip to do some type of vector analysis. How exactly does this work?

ajp_anton

31st July 2012, 20:23

The thing is, I have no idea what that script does. Can you give me a brief explanation? It looks like it creates a "super" clip to do some type of vector analysis. How exactly does this work?MSuper just prepares the clip for motion analysis. It creates multiple stacked versions of the same clip at different sizes for motion vectors of different length and precision.
MAnalyse finds the motion vectors using the super clip.
MDegrain2 uses these vectors for what it does. The 2 stands for its temporal resolution (2 frames forward, 2 backward), MDegrain1/3 also exist. Afaik it overlaps the frame block by block with matches it made from neighboring frames, strength depending on settings (thSAD and maybe others).

jmartinr

1st August 2012, 00:06

@ raido

I can't give you a brief explanation, sorry. See http://avisynth.org.ru/mvtools/mvtools2.html

raido

1st August 2012, 04:20

@ajp_anton
Thank you for the explanation, so MDegrain3 would stomp out the grain even further. I would just have to create another backward and forward vector and add them as arguments.

@jmartinr
No worries, you've helped me a lot already. I think I have a basic understanding of how MDegrain works. I see from that link that mvtools has many more useful applications. Thanks for bringing it to my attention.

Gonna go generate some CPU cycles now...

ajp_anton

6th August 2012, 13:40

Is there a way to use a non-16bit-aware linear filter in ~16 bits?
I had some vague ideas that may lose one bit (so 15 bits), but haven't had access to a computer with Avisynth to test anything.

Abyssal

8th August 2012, 20:37

For someone like me who does not make magic happen when it comes to avisynth (new to this), is there anything I should look at particularly when setting Gradfun3 up? Meaning what parameters are the things you generally want to look at when working with simple debanding? I mean, it's a lot of parameters and many that I don't think is necessary to learn about if you know nothing to begin with. For now I'm only messing with thr, thr detection, range, mode and mask - Is there any other important stuff I need to know when setting it up for the particular source I want to use it on?

cretindesalpes

8th August 2012, 21:45

ajp_anton:

This was more or less what Dither1Pre and Dither2Pre (8-bit input, 16-bit output) tried to achieve but this method isn't very effective and is subject to artifacts. The main problem is the limited output precision of the main filter.

Abyssal:

Mostly:

- thr: the intensity of the debanding. Tradeoff between debanding power and preservation of the textures and low-contrast details.
- radius: a large radius is able to smooth very low gradients when color steps are distant, but may fail on steeper ones.
- mode: tradeoff between visual smoothness and compressability
- smode: quality of the restored gradients, misc artifacts.

You can also play with the mask, depending on the amount of noise and details on the source.

cretindesalpes

12th August 2012, 19:03

Dither 1.20.0 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Added Dither_removegrain16 (modes 1–4, 11, 12, 19 and 20 only) and Dither_repair16 (modes 1–4 and 11–14 only).
Added threshold, radius and elasticity settings for chroma to GradFun3, as well as the usual plane filters. Better compatibility with the 4:1:1 colorspace.
Added Dither_sub16 and optimised Dither_add16 for SSE2.
Fixed a regression for 4:2:0 DV-PAL chroma placement in Dither_resize16.
Colorspace conversions using point-sampling for chroma are now more consistent regarding chroma placement.