Color banding and noise removal

[cretindesalpes]

You should not use SmoothGrad alone to reduce banding, unless using extremely light settings. It’s a simple building block. It requires edge and detail masking to avoid the mentioned problems. Prefer using GradFun3 or f3kdb. If you’re processing dark scenes, you’ll probably need a luma mask too.

[Boulder]

Out of interest, is it generally useful to apply GradFun3 if the output is then encoded using x264 (with general settings not targeted for grain retention, and usually crf 18-19)?

[asarian]

You should not use SmoothGrad alone to reduce banding, unless using extremely light settings. It’s a simple building block. It requires edge and detail masking to avoid the mentioned problems. Prefer using GradFun3 or f3kdb. If you’re processing dark scenes, you’ll probably need a luma mask too.

Thx. I figured I'd need masking. :)

As for GradFun3, I thought that was only to *prevent* banding (whereas SmoothGrad is used to *remove* existing banding).

[feisty2]

all smoothing filters (denoise filters, blurring filters, etc) can deband your clip more or less, cuz banding it very self is a kind of noise too. as for PREVENTING banding occurring on banding free clips, it's a job for dither filters, not smoothing filters

[asarian]

all smoothing filters (denoise filters, blurring filters, etc) can deband your clip more or less, cuz banding it very self is a kind of noise too. as for PREVENTING banding occurring on banding free clips, it's a job for dither filters, not smoothing filters

Exactly. :) So, I thought GradFun3 was to *prevent* banding (like GradFun2DBmod does) when compressing starts (x264).

[feisty2]

gradfun3 is a smoothing filter, dither filter is something like ditherpost

[feisty2]

a deband (smoothing) filter takes low precision banding clips and outputs high precision banding free clips
a dither filter takes high precision banding free clips and outputs low precision clips via dither algorithms, so banding won't jump outta noplace on your downsampled low precision clips

[asarian]

a deband (smoothing) filter takes low precision banding clips and outputs high precision banding free clips
a dither filter takes high precision banding free clips and outputs low precision clips via dither algorithms, so banding won't jump outta noplace on your downsampled low precision clips

Thx. :)

Well, the Dither manual has some excellent examples for using masks with these. I guess I should go try em out, instead of just applying SmoothGrad() on the entire frame.

[feisty2]

gradfun3 is simply a shortcut of smoothgrad+mask filters+limit filters, everything you need has been built in inside the func, that's why smoothgrad is just a building block of gradfun3, use gradfun3, you don't have to worry about mask stuff anymore

[Sparktank]

I just noticed the document is updated and very organized!

Excellent work! :) I love the new TOC.

Any chance for cretindesalpes to hijack thread?
Like copy/move his main post (with updates) to be post #0 (or -1) and push the first post (thread starter) to post #1?
Or maybe just move to new thread where he gains control over thread starter.
I'm pretty used to how it is now, but for simplistic reasons, I think it would fair better with Dither_package being first post and title of thread for newcomers.

Keep up the good work, cretindesalpes.

[StainlessS]

Any chance for cretindesalpes to hijack thread?

I guess that would require osgZach consent, he is still around last on-line 26 aug 2014.

It is not without precedent, I took over a thread originally by Forensic.

[Motenai Yoda]

Probably something like cl_exprxyz(last,pointresize(width,height,-1),pointresize(width,height,1),"x y z max max",lsb=true).
I don't think this can work, maybe a more complex one like (also note pointresize is in 8bit domain)

h=cl_exprxyz(last,dither_resize16(width,height /2,-1,0,-1,0,kernel="point"),\
dither_resize16(width,height /2,1,0,width +1,0,kernel="point"),"x y z max max",lsb=true)
v=cl_exprxyz(last,dither_resize16(width,height /2,0,-1,0,-1,kernel="point"),\
dither_resize16(width,height /2,0,1,0,(height /2) +1,kernel="point"),"x y z max max",lsb=true)
cl_exprxy(h,v,"x y max",lsb=true)

btw my gpu doesn't support opencl over 1.0.

[cretindesalpes]

Dither 1.26.4 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Fixed bug in Dither_limit_dif16 and SmoothGrad when thr and elast are both too low or too high. Thanks to mirkosp for reporting it.
The radius value is more thoroughly checked in GradFun3 in order to make related error messages more explicit.

[asarian]

Dither 1.26.4 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Fixed bug in Dither_limit_dif16 and SmoothGrad when thr and elast are both too low or too high. Thanks to mirkosp for reporting it.
The radius value is more thoroughly checked in GradFun3 in order to make related error messages more explicit.

Hmm, when I upgraded, I saw both avsi files were identical to the one I already had, LOL. :confused:

Also, your mvtools also comes with its own avstp.dll file. The dither package also has one (which appears to be created an hour later). So, which one should I be using now?

Thx.

[cretindesalpes]

dither.avsi has changed. Check the version on the header (1.26.4)
The avstp.dll should be the same too, the date difference probably comes from conversions to local time (with and w/o DST).

[asarian]

dither.avsi has changed. Check the version on the header (1.26.4)
The avstp.dll should be the same too, the date difference probably comes from conversions to local time (with and w/o DST).

Thx. :)

[cretindesalpes]

Dither 1.26.5 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Fixed Dither_resize16 to make pure vertical resizing multi-threaded.

[bxyhxyh]

Hi, I can't understand Dither_lut8, Dither_lutxy8 and Dither_lutxyz8.
Documentation says These functions are the equivalent of the Masktools mt_lut, mt_lutxy and mt_lutxyz functions. Here, they generate 16-bit data from 8-bit clips.
But it generates clip like this

source - https://dl.dropboxusercontent.com/u/58215671/source.png
dither_lut8() - https://dl.dropboxusercontent.com/u/58215671/result.png

What am I doing wrong?

[colours]

bxyhxyh, it doesn't look like there's anything wrong there.

Remember that you need to scale your output by 256 when going from 8-bit to 16-bit to get something meaningful. In other words, expr="x 256 *" would be the "no-op" for Dither_lut8, not expr="x". Likewise for the other 8-to-16 LUT functions.

(Of course there are caveats and occasionally it makes sense to not scale the input, but I'm guessing this is not one of those times.)

[bxyhxyh]

From documentation, I thought dither_lut8() and dither_lutxy8() work on normal 8bit clip. And result would be 16bit stacked clip.

But it is not even 16bit stacked clip.

Do you have an example that uses dither_lut8() properly?

[feisty2]

very simple, you can keep your old 8bpc lut expr and add "256 *" at the end of it, and it will work on dither_lut8

[colours]

But it is not even 16bit stacked clip.

It does have twice the height of the input clip, so I'm not sure why you don't think it's a stack16 clip.

Like I said, you can check it for yourself with Dither_lut8("x 256 *",u=3,v=3), which would be a slower version of Dither_convert_8_to_16.

[bxyhxyh]

Ok. As you and feisty said it works. But is it 256 or 257?
0=0 and 255(8bit white)*257=65535(16bit white) (not 256=65536 here)

[colours]

The convention is to multiply by 256; both DitherPost and f3kdb_dither assume this convention when reducing the bit depth. BT.709 and BT.2020 also specify this depth conversion convention.

Unless you intentionally want to screw up the levels or the source is full-range 8-bit and you want the output to be full-range 16-bit, don't multiply by 257.

[StainlessS]

Ok. As you and feisty said it works. But is it 256 or 257?
0=0 and 255(8bit white)*257=65535(16bit white) (not 256=65536 here)

I dare say that you spotted your little faux pas, but just in case you did not,

255 * 256 = 65280 ($FF * $100 = $FF00)
256 * 256 = 65536 ($100 * $100 = $10000)

[bxyhxyh]

The convention is to multiply by 256; both DitherPost and f3kdb_dither assume this convention when reducing the bit depth. BT.709 and BT.2020 also specify this depth conversion convention.

Unless you intentionally want to screw up the levels or the source is full-range 8-bit and you want the output to be full-range 16-bit, don't multiply by 257.

Ok. Thanks.

[bxyhxyh]

Hi, sorry for double post

This shifts my video. It shouldn't do shift, right?
Dither_convert_8_to_16().Dither_resize16(1280,720,0,5,0,0,kernel="spline16").Ditherpost()

So I'm using.
Dither_convert_8_to_16().Dither_resize16(1280,720,0,5,0,-0.0001,kernel="spline16").Ditherpost()

Is it a bug or not?

[TheSkiller]

RTFM. ;)


dither_resize16 (
clip src,
int width,
int height,
float src_left (0),
float src_top (0),
float src_width (0),
float src_height (0),
string kernel ("spline36"),
float fh (1.0),
float fv (1.0),
int taps (4),
float a1 (undefined),
float a2 (undefined),
float a3 (undefined),
int kovrspl (1),
bool cnorm (true),
bool center (true),
string cplace ("mpeg2"),
int y (3),
int u (3),
int v (3),
string kernelh (""),
string kernelv (""),
float totalh (0),
float totalv (0),
bool invks (false),
bool invksh (invks),
bool invksv (invks),
int invkstaps (5)
)


You are telling dither_resize16 to crop before resizing.

[bxyhxyh]

RTFM. ;)
You are telling dither_resize16 to crop before resizing.

Yes, but it is not cropping like standard resizers, it is shifting.

[atra dies]

My avsi doesn't have function dither_resize16, ditherpost, latest 1.26.5 and says 1.26.3 at the top? Am I missing something?

[Reel.Deel]

That's odd. Try redownloading? Anyways dither_resize16, ditherpost are part of dither plugin not the script.

[atra dies]

Oops, updated from a sandbox which doesn't work.
Alright now.

[Boulder]

cretindesalpes,

have you happened to have taken a look at the improvements mentioned by chainik_svp here: http://forum.doom9.org/showthread.php?p=1695899#post1695899 ? Just wondering if those could be included in your MVTools2 build :)

[Overdrive80]

Hi, using gradfun3(mode=2,3,4 or 5) in flat areas I see vertical lines, is it normal???

Source: https://dl.dropboxusercontent.com/u/19135067/Sample.demuxed.m2v

Script used:

assumetff()

tfm(order=1,pp=6,mode=4).tdecimate(mode=1)

#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-
#FILTRADO
#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-

original=last

#/*Prefiltrado*/#

pre=original.dfttest(sigma=8.5,tbsize=1,lsb=true).ditherpost()

mascara=original.mt_edge("hprewitt",thY1=28, thY2=55,thc1=0,thc2=255).mt_inflate(thY=16)

#return mascara

prefiltrado=mt_merge(pre, original, mascara,u=2,v=2)

#/*Filtrado temporal*/#

tr = 6

super= MSuper(prefiltrado,mt=true)

multi_vec = MAnalyse (super, multi=true, delta=tr, overlap=4,search=4)

prefiltrado.MDegrainN (super, multi_vec, tr, thSAD=470, thSAD2=170,plane=4,lsb=true)

#/*Otros*/#

gradfun3(mode=2,lsb_in=true)



For seeing better this effect I had resized to * 2, and this is the result:

Mode 2: http://i57.tinypic.com/12192yv.png

[Izuchi]

If one were to apply 16bit filtering and then encode to 10bit using x264, which method would be better quality wise?


16bit filtering, then using Dither_Out() and --input-depth 16; or
16bit filtering, then using Dither_quantize (10,reducerange=true) + Dither_out () and --input-depth 10


Would inputting a 16bit clip versus a 10bit clip be better since it has higher bits?

[wOxxOm]

Izuchi, 10-bit dithering by DitherTools is better because it's more sophisticated and customizable whereas x264 internal 16->10bit transformation is rather primitive. The only case where 16-bit undithered input would have been justified is a not-yet-supported 16-bit Hi16P profile. Anyway whichever method you choose I doubt there'll be a noticeable visual difference unless you encode textureless 3D-rendered stuff with smooth artificial color transitions. However there might be a size difference if you go with Dither_quantize's mode=1 which is better compressible.

[feisty2]

Direct 16 bit input to x264 = rounding to 10 bit

[the_weirdo]

The only case where 16-bit undithered input would have been justified is a not-yet-supported 16-bit Hi16P profile.

H.264 doesn't have such profile and I don't think they'll ever add that profile to H.264 specs.

[mawen1250]

x264 always uses Sierra-2-4A error diffusion for bit depth down-conversion, which is an encode-friendly dithering method.
fmtconv supports it but DitherTools doesn't, so I'd rather directly output 16bit for TV range content. (for PC range content x264 and DitherTools always applies bit shifting for bit depth conversion which is not correct)

[fabioseixal]

Hello, I just want to thank cretindesalpes for his oustanding job :)
Thank you very much, you made me have a lot more quality in my personal encodings.

[Dogway]

x264 always uses Sierra-2-4A error diffusion for bit depth down-conversion, which is an encode-friendly dithering method.

Friendlier than ordered dither?

Dither documentation encourages the pre-dither step with Dither_quantize(), which is defaulted to mode=0.

I thought that high bitdepth input would benefit from other x264 float point calculations like motion analysis, etc. As this is put it looks like feeding x264 a 8-bit Sierra-2-4A dithered source would yield the same results as if the input was the same source in it undithered 16-bit form, is this right?

[Izuchi]

Direct 16 bit input to x264 = rounding to 10 bit

Yes, that's the part I wasn't quite sure of: whether to output 16-bit and let x264 handle the bit depth conversion to 10-bit; or output directly to 10-bit in which case x264 doesn't need to perform any further bit depth conversions since both the input script and output video will be the same bit depth.

[Sparktank]

Yes, that's the part I wasn't quite sure of: whether to output 16-bit and let x264 handle the bit depth conversion to 10-bit; or output directly to 10-bit in which case x264 doesn't need to perform any further bit depth conversions since both the input script and output video will be the same bit depth.

Well, letting the Dither filter dither down to 10bit, you have options of which dither algorithm you want.

The documentation has some info on how certain dither algorithms affect the output.

Actually, I think it was flash3kyuu_deband that specified how different dither algorithms affected the outcome (especially in an x264 encode).

From the flash3kyuu help.txt:
dither_algo:
3: Floyd-Steinberg dithering
* Visual quality of mode 3 is the best, but the debanded pixels may easily be destroyed by x264, you need to carefully tweak the settingsto get better result.

I would think having the option to choose different algorithms (suited to your encoding needs) would be a better choice than being limited a singular option (now that we know how x264 dithers).
Entirely source-dependent.
Blu-ray movies would yield different results than an HD camcorder.

There's more to read in the flash3kyuu help.txt. Or whatever it's called.

But, I can't imagine the dither algorithm is proprietary and different than most other dither algorithms.

[cretindesalpes]

Direct 16 bit input to x264 = rounding to 10 bit
No. As mawen1250 stated, x264 uses Sierra-2-4A to reduce bitdepth. This algorithm is similar to Floyd-Steinberg quality-wise, but a bit more computationally-efficient.

It was better to convert to 10 bits in Avisynth in the early days of 10-bit encoding because of a bug in x264 (or a wrong design), but this is not required anymore. As Dogway said, you can still use 10-bit ordered dithering to keep the best possible gradients, but I’m not sure it’s worth the effort, as 10-bit banding is more or less beyond the visual perception or hardware capabilities. You’d better spend these precious bits on other features.

[feisty2]

No. As mawen1250 stated, x264 uses Sierra-2-4A to reduce bitdepth. This algorithm is similar to Floyd-Steinberg quality-wise, but a bit more computationally-efficient.


I blame it for the harm of rumors :)

[mawen1250]

Friendlier than ordered dither?

Dither documentation encourages the pre-dither step with Dither_quantize(), which is defaulted to mode=0.

I thought that high bitdepth input would benefit from other x264 float point calculations like motion analysis, etc. As this is put it looks like feeding x264 a 8-bit Sierra-2-4A dithered source would yield the same results as if the input was the same source in it undithered 16-bit form, is this right?

Obviously ordered dither is more encode-friendly, but it will generate undesired patterns in the image.
When encoding in low bitrate 8bit, I prefer ordered dither to avoid banding. For 10bit or high bitrate 8bit, random dither is a better choice.

[Dogway]

Maybe I had it wrong from the beginning but to put in other words, what's the benefit of feeding high bitdepth video to x264 instead of a pre-dithered source? Is it only for the Sierra algo that Dither lacks?


Also wanted to ask if it's possible to change chroma placement within Dither tools, as for now I'm using the next for a YV12 DV PAL clip. I guess I am right to assume the clip uses DV chroma placement but, is there a way to know for sure? (I captured using WinDV as Type 1, and then ignoring I couldn't edit with that converted to Type 2 with Enosoft DV Processor)
ConvertToYV12(matrix="Rec601", interlaced=true,ChromaInPlacement="DV",ChromaOutPlacement="MPEG2")


This is as far as I went using Dither but I'm not sure if converting to rgb and back is worth, result differs a lot from the above.
Dither_convert_yuv_to_rgb(lsb_in=false,cplace="DV",chromak="spline36",output="rgb48y",interlaced=true)
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)
Dither_convert_rgb_to_yuv(r,g,b,lsb=false,cplace="mpeg2",chromak="spline36",mode=6,interlaced=true)

[TheSkiller]

I guess I am right to assume the clip uses DV chroma placement but, is there a way to know for sure?Depends on the decompression Codec. If you use Cedocida (http://forum.doom9.org/showthread.php?t=94458) (highly recommended) it's configuration allows you to output the raw YV12 (DV chroma placement) or a MPEG2 corrected one – your choice.

[Izuchi]

I just want to confirm something to make sure I read the Dither documentation right (you can never be too safe).

If I were to only use 16-bit filters and then reduce the bitdepth down to, say 8-bit, would the following filter-chain as an example be correct?


src
Dither_convert_8_to_16()
HQDeringmod(lsb=true,lsb_in=true,lsb_out=true)
SMDegrain(lsb=true,lsb_in=true,lsb_out=true)
GradFun3(lsb=true,lsb_in=true)
DitherPost()


So basically, what I'm asking is that I should be able to use as many 16-bit filter calls as I want without the results going awry, right?
As long as I remember to include the lsb=true,lsb_in=true,lsb_out=true parameters for each successive filter to make sure its stacked 16-bit data, of course.

[cretindesalpes]

Dogway: I’m not sure about the interlaced 4:2:0 DV chroma placement. If someone knows a serious technical source, I’m all ears. What is currently implemented in Dither is described here (http://www.mir.com/DMG/chroma.html). The code in Avisynth (avisynth/src/convert/convert_planar.cpp, ConvertToPlanarGeneric constructor) looks different.

Izuchi: Yes, you’re right.