Color banding and noise removal

[Dogway]

Thanks for the new version, I really liked Dither_lut*8() functions, but I wondered whether Dither_lut*16() would be possible, seems more logical having also Dither_convert_8_to_16 in the pack.

[cretindesalpes]

Originally Posted by Dogway

but I wondered whether Dither_lut*16() would be possible

It's possible, but not implemented as lookup tables, and it would be much slower than the mt_lut*. Also, this is not a trivial task.

But wait, there is a new dither version today:

• Dfttest has been modified to make it accept 16-bit input clips (v1.8 mod16b). It's now possible to apply a curve on the input and the inverse curve to the output to protect dark shades for example, without resolution loss or risk of color banding.
• Made available the two internal components of the SmoothGrad function: Dither_box_filter16 and Dither_limit_dif16.
• Added Dither_bilateral16, a bilateral filter giving cleaner results than the box filter for gradient smoothing.
• Improved GradFun3:
  • Better detail/edge masking with adjustable thresholds and optional mask display for debugging purpose.
  • Added two smoothing modes using the modified dfttest and the new bilateral filter. The latter is slow as hell and should be considered as experimental.
• Added a Dither_crop16 function to properly crop stacked 16-bit clips.
• Added a luma parameter to Dither_merge16_8 and fixed a slight chroma bug.

Next, I'm planning to write a 16-bit resizer. This could be used to do high bitdepth colorspace conversions.

[Dogway]

ooh shhh**. You are the man! so 16-bit resizer and colorspace conversion, then by being linear, gamma isn't an issue here, right? That's better news than expected if done right. I hope to use your spline36 then next time. Although lately I discovered MultiSWAR resizer by *.mp4 guy which works wonders for animated content.

Also Im going to test your Dither_bilateral16 function. Although it is only space dimensional I think I couldn't afford more through dither. I really try to refrain using dfttest as much for animation, not only it produces temporal artifacts/ghosting but it also blurs lineart or details, and use instead a combination of a good temporal with a bilateral, like ttempsmooth+tnlmeans.

[Heaud]

Tried out that smode 2 and it indeed is slow as all hell. Visually, it's slightly better than smode 1 and both new modes are way much better than the old method. Looks like I no longer have to do a 5 pass smoothgrad any more!

[SilaSurfer]

Originally Posted by Dogway

Although lately I discovered MultiSWAR resizer by *.mp4 guy which works wonders for animated content.

For upsizing I presume?

Nice work cretindesalpes!

[Dogway]

Originally Posted by SilaSurfer

For upsizing I presume?

Ye, didn't try downsizing tho. In my pack I included a modified version for awarpsharp2 and masktools2, but Id like to remove/replace more dependencies of old or I-dont-use-much plugins.

I tested Dither_bilateral16. It works ok a bit too raw maybe, I will try to use some combination of edge/luma/difference mask. Also the get_msb/lsb function saves any speed or is just for convenience?

[cretindesalpes]

Originally Posted by Dogway

I tested Dither_bilateral16. It works ok a bit too raw maybe, I will try to use some combination of edge/luma/difference mask.

Indeed, this function alone hasn't any protection. Its main target is blurring everything to smooth the gradients, but in a smarter way than the box filter do (taking only close colors into account). I recommend adding a first protection with Dither_limit_dif16(). Then, use a mask on the result. This is how it is implemented in GradFun3(). I have ideas to optimize the bilateral filter, but unfortunately I wouldn't expect significant speed gains.

Also the get_msb/lsb function saves any speed or is just for convenience?

It is just for convenience. Well, it's just a Crop(), so it would be difficult to write something faster!

[ESPARO]

I have a problem when I use GradFun3 ()

when i used it no error appears but the video turn to " Green" screen

i install Masktools 2 a48 before I install the filters.. any feedback?

i also re install the avisynth .. but no different ..

[cretindesalpes]

Could you please post your script? Which Avisynth version do you use ?

[ESPARO]

this is my script :

LoadPlugin("C:\Program Files (x86)\dgmpgdec158\DGDecode.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ColorMatrix.dll")
DGDecode_mpeg2source("C:\Users\Luffy\Desktop\Ao-No\onepiece-498.D2V", info=3)
tfm(D2V="C:\Users\Luffy\Desktop\Ao-No\onepiece-498.D2V")
trim(4625,9110)+trim(11220,34018)+trim(40612,54817)
gradfun3()

and I use AVIsynth 2.5.8 .. AND I tried 2.5.7 too
and there is no news just green clip ..

I tried it at many script but no different and

DitherPost and the others is work fine with me ..!
only gradfun3 () is not working fine.

[cretindesalpes]

I think you still have an old Masktools version installed somewhere and autoloaded by Avisynth. This is the only way I could reproduce this green screen. Search on your disk mt_masktools.dll, mt_masktools-25.dll and possibly mt_masktools-26.dll, remove the culprit(s), then install the latest version.

[ESPARO]

Thx a lot .. that's works with me ^_^

[cretindesalpes]

Dither 1.8.1:

• Optimised Dither_bilateral16 with SSE(2) and introduced a subsampling parameter. This gives more than a 150-fold speed increase with the default settings at the cost of a minor quality loss, making the function usable now.
• Still in Dither_bilateral16, fixed a bug related to the use of a ref clip.
• Added a troubleshooting section to the documentation.

[SilaSurfer]

Example of your functions from user guide that came in the package


o = last
Dither2Pre ("TTempSmooth (lthresh=16, cthresh=20, lmdiff=8, cmdiff=12, scthresh=48)", overlap=0, tvopt=true)
ref = last
mask = DitherPost (mode=-1).mt_edge (mode="min/max", thY1=0, thY2=255, thC1=0, thC2=255)
mask = mask.mt_binarize (1, u=3, v=3).mt_expand (u=3, v=3).mt_expand (u=3, v=3)
mask = mask.RemoveGrain (20).RemoveGrain (20)
Dither_bilateral16 (radius=16, thr=1.5, wmin=1, flat=0.75)
Dither_bilateral16 (radius=8, thr=2.5, wmin=1, flat=0.33, ref=ref)
Dither_limit_dif16 (ref, thr=0.3)
DitherPost ()
mt_merge (last, o, mask, u=3, v=3)

Could Ttempsmooth be replaced with modified dfttest so Ditherpre wouldn't be needed

o = last
Dfttest(settings) #16bit output
ref = last
mask = DitherPost (mode=-1).mt_edge (mode="min/max", thY1=0, thY2=255, thC1=0, thC2=255)
mask = mask.mt_binarize (1, u=3, v=3).mt_expand (u=3, v=3).mt_expand (u=3, v=3)
mask = mask.RemoveGrain (20).RemoveGrain (20)
Dither_bilateral16 (radius=16, thr=1.5, wmin=1, flat=0.75)
Dither_bilateral16 (radius=8, thr=2.5, wmin=1, flat=0.33, ref=ref)
Dither_limit_dif16 (ref, thr=0.3)
DitherPost ()
mt_merge (last, o, mask, u=3, v=3)

Nice work on optimizations.

[cretindesalpes]

Yes, of course you can, but then you should add all the motion compensation code if tbsize is not set to 1, otherwise you'll get ghosting.

[SilaSurfer]

I think your Mdegrain1/2/3 16bit mod instead of dfttest would be a better idea.

[Dogway]

hey cretin, Im still testing and learning your tools. I just tested smoothcurve16 and everything's ok, but in this certain case Im doing the color correction before denoising. I use it like this:

Dither_convert_8_to_16()
smoothcurve16(ycurve="string")
dfttest(lsb_in=true,lsb=true)
#Dither1Pre("dfttest()",stacked=true)
DitherPost(stacked=true,prot=false,mode=6)

You see a commented line, this is me trying to reproduce the same behaviour with dither1pre, but it doubles the height of my already doubled(msb+lsb) input. The lsb_in parameter is something I guessed myself because I couldn't find any reference in the help.

I learned finally how to use smoothgrad but what differentiates it from gradfun3? Can I use them together? I tested gradfun3 yesterday and I remember performing better if added at the very end of the script (after ditherpost) rather than added through the masking workaround, normally right after my sharpening stage. I can elaborate more on this later on.

I also realised I had some kind of banding or dither alteration when doing a resize using spline36. I attach a sample and my script just in case something is wrong, but it could easily be the resizer being 8bit. With prot=true I was having more severe banding than this

raw=last
filtered=MT("""Dither1Pre(flt="tnlmeans(settings)",stacked=true)""",2,2,true)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
msb_sharp=Contrasharpening(filtered,raw)
msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3).Spline36Resize(720,480)
msb=msb.Spline36Resize(720,480)
mask=DitherBuildMask(msb_sharp,msb)
SmoothGrad(msb_sharp,lsb.BilinearResize(720,480),radius=16, thr=0.25, elast=2).DitherPost(prot=false,mask=mask)

[SilaSurfer]

Hello cretindesalpes.

I'm would like to use this workflow for a perticular movie:

spatial = MinBlur(1)
temporal = spatial.FluxsmoothT()
mixed = temporal.merge(spatial,0.251)
Fft=mixed.sbr()

pre = fft.MSuper(pel=2, sharp=1)
src_super = o.MSuper(pel=2, sharp=2)
backward_vec2 = MAnalyse(pre, isb = true, delta = 2, overlap=4,truemotion=false)
backward_vec1 = MAnalyse(pre, isb = true, delta = 1, overlap=4,truemotion=false)
forward_vec1 = MAnalyse(pre, isb = false, delta = 1, overlap=4,truemotion=false)
forward_vec2 = MAnalyse(pre, isb = false, delta = 2, overlap=4,truemotion=false)
Nr1=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90).contra(o)


x1=subtract(Nr1.deen("a2d",3,4,0),Nr1)
subtract(Nr1,x1)

DitherPost()

I would like to have contrasharpening, and deen enhancer between Mdegrain2 and Ditherpost with using masks and I just cant get it working. I would appreciate your help if it can be done.

[Dogway]

@SilaSurfer: try this if it works

filtered=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
con=Contra(filtered,o)
x1=subtract(con.deen("a2d",3,4,0),con)
msb_sharp=subtract(con,x1)
msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
mask=DitherBuildMask(msb_sharp,msb)
DitherPost(msb_sharp,lsb,prot=true,mask=mask)

[SilaSurfer]

Originally Posted by Dogway

@SilaSurfer: try this if it works

filtered=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
x1=subtract(filtered.deen("a2d",3,4,0),filtered)
msb_sharp=Contra(filtered,o).subtract(filtered,x1)
msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
mask=DitherBuildMask(msb_sharp,msb)
DitherPost(msb_sharp,lsb,prot=true,mask=mask)

Thanks for the effort Dogway, can't try it right know (encoding), but I did notice something about contrasharpening function. I have a modified version.

taken from my example above:

o=last
spatial = o.MinBlur(1)
temporal = spatial.FluxsmoothT()
mixed = temporal.merge(spatial,0.251)
Fft=mixed.sbr()

pre = fft.MSuper(pel=2, sharp=1)
src_super = o.MSuper(pel=2, sharp=2)
backward_vec2 = MAnalyse(pre, isb = true, delta = 2, overlap=4,truemotion=false)
backward_vec1 = MAnalyse(pre, isb = true, delta = 1, overlap=4,truemotion=false)
forward_vec1 = MAnalyse(pre, isb = false, delta = 1, overlap=4,truemotion=false)
forward_vec2 = MAnalyse(pre, isb = false, delta = 2, overlap=4,truemotion=false)
Nr1=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90).contra(o)

You can use it in chain reaction and only specify a source in this case (o). You don't need to input filtered source. It was modified by Didée in thread where I found it. Here is the function itself:

function contra (clip denoised, clip original)
{
s    = denoised.minblur(1,1)                       # Damp down remaining spots of the denoised clip.
allD = mt_makediff(original,denoised)              # The difference achieved by the denoising.
ssD  = mt_makediff(s,s.removegrain(11,-1))         # The difference of a simple kernel blur.
ssDD = ssD.repair(allD,1)                          # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

denoised.mt_adddiff(ssDD,U=2,V=2)                  # Apply the limited difference. (Sharpening is just inverse blurring.)

return( last )
}

Thanks for your script I will try it as soon as I can.