Color banding and noise removal

[wOxxOm]

well it might need some tweaking to make it usable for your type of content, but even in this state it might be handy to conceal dither pattern of GradFun3 in default mode=0. Requires variableblur plugin.

#your GradFun3 settings
vGF3=GradFun3(0.5,8,2,1,thr_det=3,thr_edg=20,dthr=0.05,debug=0,smode=1)

#build motion mask
blk=8
vMA=msuper(rfilter=1,hpad=blk,vpad=blk).manalyse(blksize=blk)
vSC=MSCDetection(vMA,thscd1=300).mt_invert()
vMotionMask=mmask(vMA,ml=10,kind=0).mt_lut(y=0,w=blk).mt_lut(y=0,h=blk).mt_lut(y=0,offx=width-blk,w=blk).mt_lut(y=0,offy=height-blk)
vMotionMask=vMotionMask.mt_logic(vMotionMask.trim(1,0),"or").mt_logic(vSC,"and").mt_logic(vSC.trim(1,0),"and")
vMotionMask=vMotionMask.bilinearresize(ceil(width/blk/2)*2,ceil(height/blk/2)*2).mt_binarize(16)
vMotionMask=vMotionMask.removegrain(4).mt_expand().removegrain(11).bicubicresize(width,height).binomialblur(200,u=1,v=1)

#create motion-adaptive grain
vGray=mt_lut(y=-128,u=-128,v=-128)
vGrain=vGray.addgrainC(1,0,0,0,-1,true).mt_merge(vGray.addgrainC(1,0,0,0,-1,false),vMotionMask,true)

#grain will be added to dark areas only
vGrainMask=mt_lut("255 x 32 - 2 << 255 / 2 ^ 255 * -")
#contract mask down to areas touched by GF3 only, apply extreme blur to feather edges of the mask
vGrainMask=vGrainMask.mt_logic(mt_lutxy(vGF3,last,"x y - abs 8 << ").mt_expand().binomialblur(200,u=1,v=1),"min")

#apply grain to GF3
vGF3.mt_merge( vGF3.mt_adddiff(vGrain,u=2,v=2), vGrainMask,true )

[SilaSurfer]

Thanks man.

[upyzl]

Excuse me

Does dither.dll & dither.avsi support x64 now?
It's pretty tools and I want to use it for 10-bit x264 encoding

[atra dies]

Originally Posted by atra dies

I am using it to smooth horrible compression artifacts, bands of blocks for flat animation colors and nothing removes it so completely as gradfun3 .66 strength and I tried a lot of filters and pp settings. For non-animation I have found that smoothd works wonders at removing lots of noise but blurs the picture, set adaptive mode to 1. Seems promising if someone used that principle for another filter let me know. (Edit: sorry, should have tried mdegrain2, it is great but slow and one of the mod16 filters.)

Make that .41 strength and default radius (defaults are good). Took a better look at it with histogram "luma". I would like to apply only to Y. Is there a command to show Y or U or V separately in avisynth?

The live video problem was 16mm grain. I would use ttempsmoothf but I see banding in the motion parts (in 8bit land). I tried all kinds of denoisers but I won't accept softening instead of grain. I usually deblock live stuff with cpu=6 or cpu2 on just the luma but this also leads to banding and what would I use on that? haha

[Dogway]

In tone of my last post.
I haven't done any tests still but Im sketching some theories to put in practice later and see the results.
edit: updated with results. This is against ffv1 lossless codec, Im unsure if ordered dithering is rather targeted to h.264 encodings but results are more or less the expected.

EX01
ditherpost(mode=6)
+
mdegrain in mode=0
theory = consumes lots of bitrates because of mode=6 left overs
result = to my surprise this one was the best in compressibility o_O!
clip = 48.37Mb

EX02
ditherpost(mode=0)
+
mdegrain in mode=0
theory = motion estimation is(could be) altered
result = This was the worst for compressibility. It showed heavy overlapping ordered dithering
clip = 50.49Mb

EX03
ditherpost(mode=0)
+
motion estimation of ditherpost(mode=6) and mdegrain in mode=0
theory = ideal but probably overlapping ordered dithering, plus denoising "ordered dither" which is not recommended.
result = Looks like motion estimation has some effect when mode=0, so this compresses better than EX02. Actually results or more towards EX01 than EX02 !O_o
clip = 49.08Mb

EX04
ditherpost(mode=0)
+
mdegrain in mode=0 over mode=6 dummy
+
Ditherbuildmask workaround
theory = paired with motion estimation over the mode=6 dummy this could work.
result = The thinking behind stays true but the code is a bit different, see the code box below.
clip = 48.4Mb

o=dfttest(tbsize=1,sigma=10,lsb=true)
la2=o.DitherPost(mode=6)
la1=o.DitherPost(mode=-1)

super   = la2.MSuper(pel=2, sharp=2) #hpad=0,vpad=0, para +velocidad
b3vec     = super.MAnalyse(isb = true, delta = 3, overlap=2, blksize=8, search=4)
b2vec     = super.MAnalyse(isb = true, delta = 2, overlap=2, blksize=8, search=4)
b1vec     = super.MAnalyse(isb = true, delta = 1, overlap=2, blksize=8, search=4)
f1vec     = super.MAnalyse(isb = false, delta = 1, overlap=2, blksize=8, search=4)
f2vec     = super.MAnalyse(isb = false, delta = 2, overlap=2, blksize=8, search=4)
f3vec     = super.MAnalyse(isb = false, delta = 3, overlap=2, blksize=8, search=4)

ms=la1.MDegrain3(super, b1vec, f1vec, b2vec, f2vec , b3vec, f3vec, thSAD=400, limit=255, limitc=255,lsb=true)
ms2=la2.MDegrain3(super, b1vec, f1vec, b2vec, f2vec , b3vec, f3vec, thSAD=400, limit=255, limitc=255,lsb=true)

msk=DitherBuildMask(la1,ms.ditherpost(mode=-1))
Dither_merge16_8(o,ms2,msk)

DitherPost(mode=0)

Conclusion: EX04 could possibly be better than EX01 but differences are too small to be judged fairly, plus processing time would be longer and compressibility reduced. So Im more or less like in the beginning, in wonder. : /

I decided to spatially analyse the dither, checking with the used masks:


What I see: This would probably not be the best example as there is little motion (showed by the mdegrain mask). The relevant "mask" to check out is the one in 2nd picture, the enclosed regions belongs to the pure dfttest parts, where the rest is mdegrain. Here you can see, specially on the cloak how error diffusion changes to ordered dithering, making it ideal for encoding.

[06_taro]

Originally Posted by upyzl

Excuse me

Does dither.dll & dither.avsi support x64 now?
It's pretty tools and I want to use it for 10-bit x264 encoding

you can find the answer here and here.

[SSH4]

I thought about upscaling some source after dfttest 16bit mod. So i thought that will be great dither 16 bit data after upscaling, this must not upscale ditherings.
I was play a little bit with 16 bit MSB/LSB data and upscale with nnedi3. And found that not good idea for this moment. LSB part have aliased edges when value drop from 255 to 0. and nnedi and most of other scalers smooth them, so in result after ditherpost() we have worst result
Is there any another way for simulate 16 bit data in avisinth but without MSB/LSB and sharp edges in LSB?
My brain not work and i cant imagine anything %)

[cretindesalpes]

You can resize stacked 16-bit MSB/LSB data with Dither_resize16(). Probably not as sharp as nnedi3, but it works.
It's also possible to combine 8-bit nnedi3 with Dither_resize16(). For example:

# 16 bit input

nw = 1280	# new width
nh = 960	# new height

upnn8  = DitherPost (mode=-1)
upnn8  = upnn8.nnedi3_rpow2 (rfactor=2, fwidth=nw, fheight=nh, cshift="Spline36Resize")
upnn16 = upnn8.Dither_convert_8_to_16 ()

Dither_resize16 (nw, nh, kernel="bicubic")

last.Dither_limit_dif16 (upnn16, thr=1.0, elast=2.0)
DitherPost ()

[TheProfosist]

right now im trying to use:

Dither2Pre (flt="FFT3DGpu(Sigma=4)")
SmoothGrad (radius=16, thr=0.25, elast=2)
DitherPost ()

but it it throws a error that asks me to debug:


i would like to use dither+smoothgrad with the current plugins as best as possible. current sample of the filters i normally use in a script.

#RemoveGrain()
#RemoveGrainHD()

#crop(0,0,-0,-0,align=true)

#FFT3dGPU(sigma=0.2, precision=2)
#FFT3DFilter(sigma=0.2, ncpu=4)

#EdgeCleaner()

#DAA()
#MAA()

#LSFmod(strength=20, preblur="OFF", ss_x=2.0, defaults="slow")

#DeHalo_alpha(darkstr=1.0, brightstr=1.0, ss=2.0)

#GradFun2DBmod()

i would like to keep that order if possible (not required)
i would like the script to be able to output 16bit for 10bit x264 but also be possible to output 8bit for 8bit x264 (should be 1 line of code difference DitherPost()?)

Since i dont think anyone would want to work on the whole script i plan to work on getting one filter at a time as i need them.

For now i just need Dither+SmoothGrad to output 8bit&16bit to 8bit&10bit x264 respectively.

[cretindesalpes]

Originally Posted by TheProfosist

right now im trying to use:
Dither2Pre (flt="FFT3DGpu(Sigma=4)")

IIRC FFT3DGpu can be run only once at a time. Dither2Pre instantiates many of them, so you cannot use it here. You can try to use the classic FFT3DFilter instead, but it will be very slow. A much a better alternative would be using dfttest(lsb=true) (no need for Dither2Pre).

i would like the script to be able to output 16bit for 10bit x264 but also be possible to output 8bit for 8bit x264 (should be 1 line of code difference DitherPost()?)

For 8/8 bits, use DitherPost() as usual. For 16/10 bits, replace DitherPost() with Dither_convey_yuv4xxp16_on_yvxx(). No further filtering will be possible past this line, so this should be the last one of the script. Then encode with the appropriate command line (see the Dither documentation).

Edit:

A more general way to insert 8-bit processing between 16-bit filters is the following:

dfttest (lsb=true) # or whatever generating stack16 data

# Insert 16-bit filters here

s16 = last
DitherPost (mode=-1)

# Insert 8-bit filters here

Dither_convert_8_to_16 ()
s16.Dither_limit_dif16 (last, thr=1.0, elast=2.0)

# Insert 16-bits filters again...

# Finally
DitherPost () # or Dither_convey_yuv4xxp16_on_yvxx ()

To work correctly, the 8-bit filters must leave the low-gradient or flat area intact, i.e. touch only the edges, details or their immediate surrounding. It won't work with level/curve things (use SmoothAdjust in 16 bits instead).

Set thr and elast according to taste. You can also use masking instead of Dither_limit_dif16(), but this is a bit more work.

[TheProfosist]

is there a way to dither when going to 16bit i.e.: Dither_convey_yuv4xxp16_on_yvxx () ?

[cretindesalpes]

Originally Posted by TheProfosist

is there a way to dither when going to 16bit i.e.: Dither_convey_yuv4xxp16_on_yvxx () ?

This is a planned feature. In the meantime… here is a quick and dirty hack:

Edit: rewritten, and made a function of it (but it's still a suboptimal hack)

# n = target bitdepth (8–14)
# outn = true : output is scaled to the specified bitdepth
#      = false: output is kept in 16 bits, the lowest bits are set to 0 after dithering.
# other parameters: transmitted to DitherPost().
#
# Example:
#
# .avs:
# ...
# GradFun3(lsb=true) # Stack16 clip
# fslg_dither_to_n_bits (10, outn=true)
# Dither_convey_yuv4xxp16_on_yvxx ()
# 
# .bat:
# avs2yuv -raw script.avs -o - | x264-10bit --demuxer raw --input-depth 10 --input-res 1280x720 --fps 24000/1001 --output video.mkv -

Function fslg_dither_to_n_bits (clip o, int n, bool "outn", int "mode", int "ampo", int "ampn", bool "dyn")
{
	outn = Default (outn, false)

	o
	msb = Dither_get_msb ()
	lsb = Dither_get_lsb ()
	hr = 2

	m = String (Round (Pow (2, 16 - hr - n)) - 1)
	l = msb.mt_lut ("x     "+m+"   &u", y=3, u=3, v=3)
	h = msb.mt_lut ("x 255 "+m+" - &u", y=3, u=3, v=3)

	s1 = String (n - 8)
	ofs = String (Round (Pow (2, 7 - hr)))
	a = StackVertical (l, lsb)
	a = a.Dither_lut16 ("x "+s1+" <<u "+ofs+" 256 * +", y=3, u=3, v=3)

	b = a.DitherPost (mode=mode, ampo=ampo, ampn=ampn, dyn=dyn)

	(outn)
\	? fslg_dither_to_n_bits_out_n (b, h, n, hr, s1, ofs, m)
\	: fslg_dither_to_n_bits_out_16 (b, h, n, hr, s1, ofs)
}

Function fslg_dither_to_n_bits_out_16 (clip b, clip h, int n, int hr, string s1, string ofs)
{
	s2 = String (16 - n)
	ofs2 = String (Round (Pow (2, 15 - hr - n)))
	b_msb = b.mt_lut ("x "+s1+" >>u 255 &u "+ofs2+" - 128 +", y=3, u=3, v=3)
	b_lsb = b.mt_lut ("x "+ofs+" - "+s2+" <<u 255 &u", y=3, u=3, v=3)

	c = mt_adddiff (h, b_msb, y=3, u=3, v=3)

	StackVertical (c, b_lsb)
}

Function fslg_dither_to_n_bits_out_n (clip b, clip h, int n, int hr, string s1, string ofs, string m)
{
	ofs2 = String (Round (Pow (2, 7 - (n-8) - hr)))
	b2 = b.mt_lut ("x "+s1+" >>u "+ofs2+" - 128 + 255 "+m+" - &u", y=3, u=3, v=3)
	h2 = mt_adddiff (h, b2, y=3, u=3, v=3)
	msb = h2.mt_lut ("x "+String(16-n)+" >>u", y=3, u=3, v=3)

	m2 = String (Round (Pow (2, 8 - hr)) - 1)
	lsb = mt_lutxy (h2, b, "x "+s1+" <<u 255 "+m2+" - &u y "+ofs+" - "+m2+" &u +", y=3, u=3, v=3)

	StackVertical (msb, lsb)
}

Note: by sending 10-bit content to x264 (instead of 8- or 16-), you make sure that it will encode the exact values instead of scaling and dithering them again. Thus, you can encode safely with a vanilla 10-bit x264 instead of having to use a patched build.

Note 2: scaling issue mentioned in the previous note is now solved in the official x264 build (from r2164).

[Yellow_]

Originally Posted by cretindesalpes

Dither 1.9.5 released:

• Bug fixed in Dither_resize16(), displaying green bars when SSE2 optimisations are disabled.
• Better check of stack16 clip dimensions.

TV range is all about keeping a headroom for the ringing causing by some filters (resizers, sharpeners…), so with PC range this headroom is lost. But this is not a big difference, and the bottom headroom is cleared by the gamma/linear conversion anyway.


Yes.


GammaYUV -> linearYUV -> linearRGB will give you wrong colors. Inverse colorspace conversions should be done in the same gamma/linearity as the forward conversion. For gamma YUV to linear RGB conversion you can use:

Dither_convert_yuv_to_rgb (output="rgb48y")
Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false)
Dither_convey_rgb48_on_yv12 (
\   SelectEvery (3, 0),
\   SelectEvery (3, 1),
\   SelectEvery (3, 2) )

You're right, it appears that BT709 doesn't use the same transfer curve as sRGB, the slope at 0 in sRGB is much steeper than in BT709.

If you want to linearize the R'G'B' components converted from the BT709 Y'CbCr, apply on each one : R = R'/4.5 if R' < 0.081 or R = ((R' + 0.099) / 1.099) ^ (1/0.45) if R' ≥ 0.081.The formula are the same as in sRGB, but with different values. I will add the BT709 mode to Dither conversion functions in a future release.

Could you clarify, do you mean with Imagemagick via avs2yuv as this will be 48bit RGB or can this be done with the stacked in avisynth?

[Zero1]

I downloaded and placed the dither package in the plugins directory and all the 16 bit stuff seems to be working fine, however I can not get gradfun3 to work (in normal YV12). I simply just get a green screen. I've deleted a bunch of plugins in an attempt to troubleshoot and this is what I'm left with:

15/05/2011 21:14 192,512 dfttest.dll
21/12/2008 22:46 26,624 DirectShowSource.dll
20/07/2011 22:03 44,994 dither.avsi
21/07/2011 08:18 233,472 dither.dll
21/06/2010 11:12 1,527,650 libfftw3f-3.dll
31/12/2010 18:13 966,144 mt_masktools-25.dll
31/12/2010 18:12 966,656 mt_masktools-26.dll
19/03/2006 20:56 430,080 mt_masktools.dll
11/09/2010 14:39 2,787 mt_xxpand_multi.avsi
20/01/2005 21:58 212,992 MVTools.dll
11/05/2011 22:09 753,664 mvtools2.dll
01/08/2005 00:08 65,536 RemoveGrainS.dll

I'm using avisynth 2.5.8. Any ideas?

[wOxxOm]

remove mt_masktools-25.dll, mt_masktools.dll, MVTools.dll and make sure that mt_masktools-26.dll is alpha48

[cretindesalpes]

Originally Posted by Yellow_

Could you clarify, do you mean with Imagemagick via avs2yuv as this will be 48bit RGB or can this be done with the stacked in avisynth?

I meant this can be done with avisynth. It's just a few coefficients to change in Dither_y_gamma_to_linear and Dither_y_linear_to_gamma.

Originally Posted by wOxxOm

remove mt_masktools-25.dll, mt_masktools.dll, MVTools.dll and make sure that mt_masktools-26.dll is alpha48

Zero1 wrote he was using 2.5.8, so it's the other way round: keep mt_masktools-25 (alpha 48) and remove mt_masktools-26.dll

[Zero1]

Thanks for the info. I think I have inadvertantly found out what is going wrong. I tried to run dfttest() earlier and it returns and error saying it cannot load libfftw3f-3.dll.
When I moved it to another directory and used loadplugin, it tells me that it is not an Avisynth 2.5 pluginr?!

Edit
I googled and found that I needed to put that in my syswow64 dir, so now dfttest works. Also after deleting the numerous masktools dlls, gradfun3 appears to work too. Much thanks guys, these are really going to come in handy.

[Yellow_]

Originally Posted by cretindesalpes

I meant this can be done with avisynth. It's just a few coefficients to change in Dither_y_gamma_to_linear and Dither_y_linear_to_gamma.

I've looked through dither.avsi at the above functions and quite simply I'm lost, so thanks anyway, think I'll just look out for the BT709 mode in some future version of Dither and find an alternative route for the linearizing bit in the meantime.

[MADAJ]

Hi,

I fed an 8-bit source to x264-10bit, and I converted the 8-bit to 16 bit (using dither scripts) then fed it to x264-10bit.
However color wise, I noticed that the 1st encoding is fine and looks like the 8-bit source, while the later encoding is overlayed with greenish color.
So which one is correct?

[cretindesalpes]

Is this related with this issue? But the difference should be hardly noticeable. If you really have a green tint, first check your masktools version (must be alpha 48 or later), then post your script and encoding command.