Color banding and noise removal

[Dogway]

Thanks for the new version, I really liked Dither_lut*8() functions, but I wondered whether Dither_lut*16() would be possible, seems more logical having also Dither_convert_8_to_16 in the pack.

[cretindesalpes]

but I wondered whether Dither_lut*16() would be possible
It's possible, but not implemented as lookup tables, and it would be much slower than the mt_lut*. Also, this is not a trivial task.

But wait, there is a new dither (http://ldesoras.free.fr/src/dither-1.8.zip) version today:
Dfttest has been modified to make it accept 16-bit input clips (v1.8 mod16b). It's now possible to apply a curve on the input and the inverse curve to the output to protect dark shades for example, without resolution loss or risk of color banding.
Made available the two internal components of the SmoothGrad function: Dither_box_filter16 and Dither_limit_dif16.
Added Dither_bilateral16, a bilateral filter giving cleaner results than the box filter for gradient smoothing.
Improved GradFun3: Better detail/edge masking with adjustable thresholds and optional mask display for debugging purpose.
Added two smoothing modes using the modified dfttest and the new bilateral filter. The latter is slow as hell and should be considered as experimental.
Added a Dither_crop16 function to properly crop stacked 16-bit clips.
Added a luma parameter to Dither_merge16_8 and fixed a slight chroma bug.

Next, I'm planning to write a 16-bit resizer. This could be used to do high bitdepth colorspace conversions.

[Dogway]

ooh shhh**. You are the man! so 16-bit resizer and colorspace conversion, then by being linear, gamma isn't an issue here, right? That's better news than expected if done right. I hope to use your spline36 then next time. Although lately I discovered MultiSWAR (http://forum.doom9.org/showthread.php?t=112526)resizer by *.mp4 guy which works wonders for animated content.

Also Im going to test your Dither_bilateral16 function. Although it is only space dimensional I think I couldn't afford more through dither. I really try to refrain using dfttest as much for animation, not only it produces temporal artifacts/ghosting but it also blurs lineart or details, and use instead a combination of a good temporal with a bilateral, like ttempsmooth+tnlmeans.

[Heaud]

Tried out that smode 2 and it indeed is slow as all hell. Visually, it's slightly better than smode 1 and both new modes are way much better than the old method. Looks like I no longer have to do a 5 pass smoothgrad any more!

[SilaSurfer]

Although lately I discovered MultiSWAR (http://forum.doom9.org/showthread.php?t=112526)resizer by *.mp4 guy which works wonders for animated content.

For upsizing I presume?

Nice work cretindesalpes!;):p

[Dogway]

For upsizing I presume?
Ye, didn't try downsizing tho. In my pack I included a modified version for awarpsharp2 and masktools2, but Id like to remove/replace more dependencies of old or I-dont-use-much plugins.

I tested Dither_bilateral16. It works ok a bit too raw maybe, I will try to use some combination of edge/luma/difference mask. Also the get_msb/lsb function saves any speed or is just for convenience?

[cretindesalpes]

I tested Dither_bilateral16. It works ok a bit too raw maybe, I will try to use some combination of edge/luma/difference mask.
Indeed, this function alone hasn't any protection. Its main target is blurring everything to smooth the gradients, but in a smarter way than the box filter do (taking only close colors into account). I recommend adding a first protection with Dither_limit_dif16(). Then, use a mask on the result. This is how it is implemented in GradFun3(). I have ideas to optimize the bilateral filter, but unfortunately I wouldn't expect significant speed gains.

Also the get_msb/lsb function saves any speed or is just for convenience?
It is just for convenience. Well, it's just a Crop(), so it would be difficult to write something faster!

[ESPARO]

I have a problem when I use GradFun3 ()

when i used it no error appears but the video turn to " Green" screen

i install Masktools 2 a48 before I install the filters.. any feedback?

i also re install the avisynth .. but no different ..

[cretindesalpes]

Could you please post your script? Which Avisynth version do you use ?

[ESPARO]

this is my script :
LoadPlugin("C:\Program Files (x86)\dgmpgdec158\DGDecode.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ColorMatrix.dll")
DGDecode_mpeg2source("C:\Users\Luffy\Desktop\Ao-No\onepiece-498.D2V", info=3)
tfm(D2V="C:\Users\Luffy\Desktop\Ao-No\onepiece-498.D2V")
trim(4625,9110)+trim(11220,34018)+trim(40612,54817)
gradfun3()

and I use AVIsynth 2.5.8 .. AND I tried 2.5.7 too
and there is no news just green clip ..

I tried it at many script but no different and

DitherPost and the others is work fine with me ..!
only gradfun3 () is not working fine.

[cretindesalpes]

I think you still have an old Masktools version installed somewhere and autoloaded by Avisynth. This is the only way I could reproduce this green screen. Search on your disk mt_masktools.dll, mt_masktools-25.dll and possibly mt_masktools-26.dll, remove the culprit(s), then install the latest version.

[ESPARO]

Thx a lot .. that's works with me ^_^

[cretindesalpes]

Dither 1.8.1 (http://ldesoras.free.fr/src/dither-1.8.1.zip):

Optimised Dither_bilateral16 with SSE(2) and introduced a subsampling parameter. This gives more than a 150-fold speed increase with the default settings at the cost of a minor quality loss, making the function usable now.
Still in Dither_bilateral16, fixed a bug related to the use of a ref clip.
Added a troubleshooting section to the documentation.

[SilaSurfer]

Example of your functions from user guide that came in the package


o = last
Dither2Pre ("TTempSmooth (lthresh=16, cthresh=20, lmdiff=8, cmdiff=12, scthresh=48)", overlap=0, tvopt=true)
ref = last
mask = DitherPost (mode=-1).mt_edge (mode="min/max", thY1=0, thY2=255, thC1=0, thC2=255)
mask = mask.mt_binarize (1, u=3, v=3).mt_expand (u=3, v=3).mt_expand (u=3, v=3)
mask = mask.RemoveGrain (20).RemoveGrain (20)
Dither_bilateral16 (radius=16, thr=1.5, wmin=1, flat=0.75)
Dither_bilateral16 (radius=8, thr=2.5, wmin=1, flat=0.33, ref=ref)
Dither_limit_dif16 (ref, thr=0.3)
DitherPost ()
mt_merge (last, o, mask, u=3, v=3)

Could Ttempsmooth be replaced with modified dfttest so Ditherpre wouldn't be needed :rolleyes:

o = last
Dfttest(settings) #16bit output
ref = last
mask = DitherPost (mode=-1).mt_edge (mode="min/max", thY1=0, thY2=255, thC1=0, thC2=255)
mask = mask.mt_binarize (1, u=3, v=3).mt_expand (u=3, v=3).mt_expand (u=3, v=3)
mask = mask.RemoveGrain (20).RemoveGrain (20)
Dither_bilateral16 (radius=16, thr=1.5, wmin=1, flat=0.75)
Dither_bilateral16 (radius=8, thr=2.5, wmin=1, flat=0.33, ref=ref)
Dither_limit_dif16 (ref, thr=0.3)
DitherPost ()
mt_merge (last, o, mask, u=3, v=3)

Nice work on optimizations.

[cretindesalpes]

Yes, of course you can, but then you should add all the motion compensation code if tbsize is not set to 1, otherwise you'll get ghosting.

[SilaSurfer]

I think your Mdegrain1/2/3 16bit mod instead of dfttest would be a better idea.

[Dogway]

hey cretin, Im still testing and learning your tools. I just tested smoothcurve16 and everything's ok, but in this certain case Im doing the color correction before denoising. I use it like this:
Dither_convert_8_to_16()
smoothcurve16(ycurve="string")
dfttest(lsb_in=true,lsb=true)
#Dither1Pre("dfttest()",stacked=true)
DitherPost(stacked=true,prot=false,mode=6)
You see a commented line, this is me trying to reproduce the same behaviour with dither1pre, but it doubles the height of my already doubled(msb+lsb) input. The lsb_in parameter is something I guessed myself because I couldn't find any reference in the help.

I learned finally how to use smoothgrad but what differentiates it from gradfun3? Can I use them together? I tested gradfun3 yesterday and I remember performing better if added at the very end of the script (after ditherpost) rather than added through the masking workaround, normally right after my sharpening stage. I can elaborate more on this later on.

I also realised I had some kind of banding or dither alteration when doing a resize using spline36. I attach a sample and my script just in case something is wrong, but it could easily be the resizer being 8bit. With prot=true I was having more severe banding than this
raw=last
filtered=MT("""Dither1Pre(flt="tnlmeans(settings)",stacked=true)""",2,2,true)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
msb_sharp=Contrasharpening(filtered,raw)
msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3).Spline36Resize(720,480)
msb=msb.Spline36Resize(720,480)
mask=DitherBuildMask(msb_sharp,msb)
SmoothGrad(msb_sharp,lsb.BilinearResize(720,480),radius=16, thr=0.25, elast=2).DitherPost(prot=false,mask=mask)
http://i212.photobucket.com/albums/cc35/Dogway/Misc/artifact.png

[SilaSurfer]

Hello cretindesalpes.

I'm would like to use this workflow for a perticular movie:

spatial = MinBlur(1)
temporal = spatial.FluxsmoothT()
mixed = temporal.merge(spatial,0.251)
Fft=mixed.sbr()

pre = fft.MSuper(pel=2, sharp=1)
src_super = o.MSuper(pel=2, sharp=2)
backward_vec2 = MAnalyse(pre, isb = true, delta = 2, overlap=4,truemotion=false)
backward_vec1 = MAnalyse(pre, isb = true, delta = 1, overlap=4,truemotion=false)
forward_vec1 = MAnalyse(pre, isb = false, delta = 1, overlap=4,truemotion=false)
forward_vec2 = MAnalyse(pre, isb = false, delta = 2, overlap=4,truemotion=false)
Nr1=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90).contra(o)


x1=subtract(Nr1.deen("a2d",3,4,0),Nr1)
subtract(Nr1,x1)

DitherPost()

I would like to have contrasharpening, and deen enhancer between Mdegrain2 and Ditherpost with using masks and I just cant get it working. I would appreciate your help if it can be done.

[Dogway]

@SilaSurfer: try this if it works
filtered=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
con=Contra(filtered,o)
x1=subtract(con.deen("a2d",3,4,0),con)
msb_sharp=subtract(con,x1)
msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
mask=DitherBuildMask(msb_sharp,msb)
DitherPost(msb_sharp,lsb,prot=true,mask=mask)

[SilaSurfer]

@SilaSurfer: try this if it works
filtered=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
x1=subtract(filtered.deen("a2d",3,4,0),filtered)
msb_sharp=Contra(filtered,o).subtract(filtered,x1)
msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
mask=DitherBuildMask(msb_sharp,msb)
DitherPost(msb_sharp,lsb,prot=true,mask=mask)

Thanks for the effort Dogway, can't try it right know (encoding), but I did notice something about contrasharpening function. I have a modified version.

taken from my example above:

o=last
spatial = o.MinBlur(1)
temporal = spatial.FluxsmoothT()
mixed = temporal.merge(spatial,0.251)
Fft=mixed.sbr()

pre = fft.MSuper(pel=2, sharp=1)
src_super = o.MSuper(pel=2, sharp=2)
backward_vec2 = MAnalyse(pre, isb = true, delta = 2, overlap=4,truemotion=false)
backward_vec1 = MAnalyse(pre, isb = true, delta = 1, overlap=4,truemotion=false)
forward_vec1 = MAnalyse(pre, isb = false, delta = 1, overlap=4,truemotion=false)
forward_vec2 = MAnalyse(pre, isb = false, delta = 2, overlap=4,truemotion=false)
Nr1=o.MDegrain2(lsb=true, src_super, backward_vec1,forward_vec1,backward_vec2,forward_vec2thSAD=300,thscd1=300,thscd2=90).contra(o)

You can use it in chain reaction and only specify a source in this case (o). You don't need to input filtered source. It was modified by Didée in thread where I found it. Here is the function itself:

function contra (clip denoised, clip original)
{
s = denoised.minblur(1,1) # Damp down remaining spots of the denoised clip.
allD = mt_makediff(original,denoised) # The difference achieved by the denoising.
ssD = mt_makediff(s,s.removegrain(11,-1)) # The difference of a simple kernel blur.
ssDD = ssD.repair(allD,1) # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

denoised.mt_adddiff(ssDD,U=2,V=2) # Apply the limited difference. (Sharpening is just inverse blurring.)

return( last )
}

Thanks for your script I will try it as soon as I can.

[Dogway]

But here this is a variable chain so you need to specify. If you dont you are implying "last", which is the latest filter used outside the variable chain. I use this all the time with the same contra function and it works, but just give it a try and tell me. Once you get the grasp of dither mechanics it becomes easy (and fun!)

[SilaSurfer]

Yeah, can't wait! Gracias Dogway

[mandarinka]

Is there a way to apply gradfun3 but keep the final result in 16bits (the stacked MSB/LSB format), for 10bit x264 encoding? I tried to remove the ditherpost line from the function, but that didn't go so well (I don't understand scripting, sadly...)

I would like to do this: 8bit source with banding -> debanded 16bit -> 10bit h264.

[mp3dom]

Hi cretindesalpes, amazing work and I found the result of Gradfun3 more pleasant now than gradfun2db. Actually I need to do a dithering on a YUY2 clip and I'm getting errors if I don't convert it to YV12 (the script is asking for a "isYV16()" function that doesn't exists in AVS 2.58 and is only available on 2.60). I still would prefer to dither natively on YUY2 because the encoder accepts YUY2 and I would like to avoid colorspace conversions. I've tried to convert the clip to planar (with SSE2Tools available with RemoveGrain package) but the script always ask for that function. Is there a way to 'fix' it? Thank you very much!

[tormento]

Any x64 compatible code? Please ;)

[LaTo]

Hi cretindesalpes, amazing work and I found the result of Gradfun3 more pleasant now than gradfun2db. Actually I need to do a dithering on a YUY2 clip and I'm getting errors if I don't convert it to YV12 (the script is asking for a "isYV16()" function that doesn't exists in AVS 2.58 and is only available on 2.60). I still would prefer to dither natively on YUY2 because the encoder accepts YUY2 and I would like to avoid colorspace conversions. I've tried to convert the clip to planar (with SSE2Tools available with RemoveGrain package) but the script always ask for that function. Is there a way to 'fix' it? Thank you very much!

YUY2 <-> YV16 conversion is lossless... It's the same colorspace but YV16 is planar.

[mp3dom]

Yes, but under AVS 2.58 (which I'm currently use) there's no function named "isYV16()". There are other functions too that are called to check the source colorspace: IsYV12, IsYV16, IsYV24, IsYV411, IsY8. With AviSynth 2.58 seems that only YV12 source can be processed so I'm basically asking if it's possible to add AviSynth 2.58 support for planar YUY2. Thanks.

[cretindesalpes]

You see a commented line, this is me trying to reproduce the same behaviour with dither1pre, but it doubles the height of my already doubled(msb+lsb) input. The lsb_in parameter is something I guessed myself because I couldn't find any reference in the help.
DitehrXPre () is for 8 to 16 bit conversion with filtering. Here, your clip has already a 16 bit depth, no need for these functions. Your version with lsb_in is right. This option is documented is in the dfttest-1.8-mod16b.zip readme file.

I learned finally how to use smoothgrad but what differentiates it from gradfun3?
SmoothGrad is a 100% 16-bit (in and out) building block. GradFun3 is a no-brainer function based on SmoothGrad, and is to be called on 8-bit clips like gradfun2db(mod). Don't call it on 16-bit clips, it would give unexpected results. Actually GF3 calls internally SmoothGrad multiple times, with different radius and thresholds.

I also realised I had some kind of banding or dither alteration when doing a resize using spline36.
Your script looks correct. Unfortunately, using prot=true doesn't solve all the problems related to resizing. There are no real solution for clean resizing in the 8-bit world. That's why I'm currently working on a 16-bit resizer, which will do the job correctly, without the prot=true hack.

Is there a way to apply gradfun3 but keep the final result in 16bits (the stacked MSB/LSB format), for 10bit x264 encoding? I tried to remove the ditherpost line from the function, but that didn't go so well (I don't understand scripting, sadly...)
Removing everything after the "# Dithering" line in GradFun3 should output regular stacked 16 bits data. Then if you want to output it to x264-10bit, put just after the call the code snippet I wrote in the SmoothAdjust thread (http://forum.doom9.org/showpost.php?p=1504502&postcount=216). Anyway, I don't know if there are players which can play correctly 10-bit AVC streams. Recent ffmpeg builds can decode them, but the extended accuracy is available only for yuv420p16 output...

Actually I need to do a dithering on a YUY2 clip and I'm getting errors
Indeed, the functions are not designed for non-planar data. I should investigate how Interleaved2Planar() works as it is not well documented on a technical POV, and find if it's really interesting to implement, because mt_merge(luma=true) doesn't work with YUY2.

In the meantime, you can use this workaround:
y = ConvertToYV12 ()
u = UToY ().ConvertToYV12 ()
v = VToY ().ConvertToYV12 ()
YToUV (u, v, StackVertical (y, y))

GradFun3 (mask=0)

y = Crop (0, 0, 0, Height () / 2).ConvertToYUY2 ()
u = UToY ().ConvertToYUY2 ()
v = VToY ().ConvertToYUY2 ()
YToUV (u, v, y)
Theses conversions are lossless. However in the converted clip, luma and chroma planes are not spatially synchronised. Everything mixing them should be avoided, like the GradFun3's mask functionality using mt_merge(luma=true). If you really want masking, you can still modify the script this way:

ca = YToUV (u, v, StackVertical (y, y))
cb = YToUV (u, v, y.PointResize (Width (), Height () * 2))

Interleave (ca, cb)
GradFun3 ()
MergeChroma (SelectEven (), SelectOdd ())

(BTW note to myself: make the mask threshold vary according to the main threshold, because increasing only the latter is completely useless to combat serious banding).

Any x64 compatible code? Please ;)
Not in my priority list, unless the source code for x64 dfttest and MVTools2 suddenly becomes available. Unfortunately JoshyD hasn't connected to this forum for more than a year.

[mandarinka]

Removing everything after the "# Dithering" line in GradFun3 should output regular stacked 16 bits data. Then if you want to output it to x264-10bit, put just after the call the code snippet I wrote in the SmoothAdjust thread (http://forum.doom9.org/showpost.php?p=1504502&postcount=216). Anyway, I don't know if there are players which can play correctly 10-bit AVC streams. Recent ffmpeg builds can decode them, but the extended accuracy is available only for yuv420p16 output...

Thanks! I'll give it a try.

[SilaSurfer]

Hello everyone. Dogway I just tried your example that you posted and it works. For another source I swapped Mdegrain with dfttest like this:

filtered=o.dfttest(lsb=true,sigma=1.5, sbsize=48,tbsize=3,sosize=16,U=false,V=false)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
con=Contra(filtered,o)
x1=subtract(con.deen("a2d",3,4,0),con)
msb_sharp=subtract(con,x1)
msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
mask=DitherBuildMask(msb_sharp,msb)
DitherPost(msb_sharp,lsb,prot=true,mask=mask)

but I have a little trouble again :p, adding Smoothgrad in there to do some debanding which was covered by grain that was removed. Appreciate any help.

[cretindesalpes]

Just replace your last line with the following code:
SmoothGrad (msb_sharp, clsb=lsb)
DitherPost (prot=true, mask=mask)

[SilaSurfer]

cretindesalpes Thank you very much, BTW how is that 16-bit resizer coming along?

[mp3dom]

Thanks cretindesalpes!

[SilaSurfer]

cretindesalpes does Smoothgrad remove details since it's operation is smoothing gredients to eliminate banding?

[cretindesalpes]

does Smoothgrad remove details since it's operation is smoothing gredients to eliminate banding? It depends on the thr value, but with the default settings the damages will be minimal. Anyway, I would recommend to use a mask (see the GradFun3 implementation).
Thank you very much, BTW how is that 16-bit resizer coming along?
The resizer is ready and working but I just installed Avisynth 2.6 on another computer and I'm trying (struggling?) to make a single dll work on both 2.5.x and 2.6.0 while supporting correctly the new planar colorspaces.

[SilaSurfer]

Thanks for your response. I had another idea but I need your opinion once more.;)

Taken from my example above, just removing deen:

o=last
filtered=o.dfttest(lsb=true,sigma=1.5, sbsize=48,tbsize=3,sosize=16,U=false,V=false)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
con=Contra(filtered,o)
msb_sharp=mt_makediff (con,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
mask=DitherBuildMask(msb_sharp,msb)
SmoothGrad (thr=0.15,msb_sharp, clsb=lsb)
DitherPost (prot=true, mask=mask)

to use Smoothgrad after dfttest like this so any detail loss will be returned with contrasharpening

o=last
filtered=o.dfttest(lsb=true,sigma=1.5, sbsize=48,tbsize=3,sosize=16,U=false,V=false)-16-bit output.SmoothGrad (thr=0.15, clsb=lsb)
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
con=Contra(filtered,o)
msb_sharp=mt_makediff (con,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
mask=DitherBuildMask(msb_sharp,msb)
DitherPost(msb_sharp,lsb,prot=true,mask=mask)

Didn't test this since I'm at work, just wanted your opinion.Thank you.

PS:
Can't wait for that resizer :p

[cretindesalpes]

Yes it should work (without the typos and the clsb parameter, useless here because the dfttest output is stack16). But be careful, the contrasharpening might restore the banding as well. You'll have to check with your source.

[cretindesalpes]

Dither 1.9 (http://forum.doom9.org/showthread.php?p=1386559#post1386559) update:

The threshold of the detail mask in GradFun3() now automatically adjusts to the debanding threshold. This helps removing serious color banding, when band boundaries are masked because they are detected as details. Also, the radius for smode 0 has been internally reduced to make the value reflect more accurately the actual range in pixels.
Added an interlaced mode for DitherPost().
Added Dither_resize16(), a 16-bit resizer which can also work as a chained horizontal and vertical convolution engine.
Added y, u and v parameters to almost all the dither.dll functions. Their behaviour is the same as in masktools2.
Added Dither_y_gamma_to_linear() and Dither_y_linear_to_gamma(), for gamma <-> linear conversions in 16 bits.
Added Dither_convey_yuv4xxp16_on_yvxx() and Dither_convey_rgb48_on_yv12(), functions to export clips with 16 bits per component when Avisynth's output is read as rawvideo via a pipe.
Added Dither_convert_yuv_to_rgb() and Dither_convert_rgb_to_yuv() to convert 8 or 16 bits YUV to RGB and back with dithering. It's also possible to output RGB48 conveyed on YV12 via a rawvideo pipe.
Added Dither_max_dif16() to get from two clips the pixels having the maximum difference with a third reference clip.
Fixed an error message about wrong colorspace.
Bugfix: removed artifacts on flat pure white areas (luma == 255) when using error diffusion dithering (modes 6–8).
Non-YV12 planar Y(UV) colorspaces should now work correctly on Avisynth 2.6.0. I haven't tested all the combinations with all the functions, so please report me any defect!
All these new things work correctly but are not well optimised. Consequently, they are insanely slow and resource hungry, especially the colorspace conversion function which would deserve a true C++/asm implementation instead of a scripted hack using the convolver. Currently, consider this more as a proof of concept and base for experiments rather than full-fledged production tools. Moreover, the stack16 format is nice and convenient for previewing, but horrendous for programming.

[Dogway]

Wow, you are really the game changer. Avisynth to the next level, thanks for all, its gonna take me a few days to assimilate.
The 16b resizer also fixed the banding problem I had in my other post (http://forum.doom9.org/showthread.php?p=1504967#post1504967)

I use it like this (+adding prot=true at the end)
filtered=MT("""Dither1Pre(flt="tnlmeans(settings)",stacked=true)""",2,2,true).Dither_resize16(720,480,kernel="spline36")
msb=filtered.dither_get_msb
lsb=filtered.dither_get_lsb
filtered=filtered.DitherPost(mode=-1,stacked=true,prot=false)
msb_sharp=Contrasharpening(filtered,raw.Spline36Resize(720,480))
...
Is this ok? it makes sense, but just wondering, I can't resize after sharpening right? not at least in 16b

edit: In the mean time, Im gonna try to port contrasharpening to 16b if its possible.

[cretindesalpes]

Is this ok? it makes sense, but just wondering, I can't resize after sharpening right? not at least in 16b
The code looks OK but I'm not sure to understand your question. Do you mean you want to put the resizing code after having obtained msb_sharp ? I think the safest way to do this is merging a 16 bit conversion of the 8-bit sharp clip with the denoised 16 bit clip, and resizing the result. Use a mask keeping only the pixels modified by the (contra)sharpener.

[Dogway]

I think the safest way to do this is merging a 16 bit conversion of the 8-bit sharp clip with the denoised 16 bit clip, and resizing the result. Use a mask keeping only the pixels modified by the (contra)sharpener.
This?
filtered1=MT("""Dither1Pre(flt="tnlmeans(settings)",stacked=true)""",2,2,true)
msb=filtered1.Crop(0,0,706,478)
lsb=filtered1.Crop(0,478,706,478)
filtered=filtered1.DitherPost(mode=-1,stacked=true,prot=false)
msb_sharp=Contrasharpening(filtered,raw)

msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
to16=msb_sharp.Dither_convert_8_to_16()

mask=DitherBuildMask(to16,filtered1)
Dither_merge16(filtered1,to16,mask).Dither_resize16(720,480,kernel="spline36")
SmoothGrad(radius=16, thr=0.25, elast=2).DitherPost()

# SmoothGrad(msb_sharp,lsb,radius=16, thr=0.25, elast=2).DitherPost(prot=true,mask=mask)

I dont know if the last line is correct, the commented line was what I was using before.

[SilaSurfer]

cretindesalpes Thanks for the new package. Could your resizer be modified to output 8 bit like Gradfun3 for example? The reason is that I denoise after downsize rather then denoising 1080p source which is very slow with about any filter.

[Dogway]

@SilaSurfer: Simply use DitherPost(). Dither is a bit of a group of tools and functions spread around. You need to join pieces like a puzzle for "user friendly" usage. That's why I just made a function for myself for resizing, its optimized for rgb input, but you have parameters to change them.

function LinearResize(clip a, int "w", int "h", bool "rgb", string "kernel", bool "linear", bool "TVrange",string "matrix"){

w=default(w, a.width)
h=default(h, a.height)
kernel = default(kernel, "spline36")
rgba = default(rgb, true)
linear = default(linear, true)
TV = default(TVrange, false)
m = default(matrix, "601")
a
rgba ? Dither_convert_rgb_to_yuv(lsb=true,matrix=m,tv_range=tv,output="yv24") : Dither_convert_8_to_16()
linear ? Dither_y_gamma_to_linear(tv_range_in=tv,tv_range_out=tv) : last
Dither_resize16(w,h,kernel=kernel)
linear ? Dither_y_linear_to_gamma(tv_range_in=tv,tv_range_out=tv) : last
rgba ? Dither_convert_yuv_to_rgb(lsb_in=true,matrix=m,tv_range=tv,ampn=0.5,mode=6) : DitherPost}


Implemented matrix conversion and automatizations depending on input, not sure if it works flawlessly, but I did my tests...
Linear=true doesn't work when downsizing from HD to SD, it is to take advantage of the high resolution input for color conversion, it is that or linear resizing which I still consider experimental.

function LinearResize(clip a, int "w", int "h", string "kernel", bool "linear", bool "TVrange", string "matrix", \
string "matrixOut", float "fh", float "fv", bool "rgb", bool "lsb_in", bool "lsb_out"){

w = default(w, a.width)
h = default(h, a.height)
kernel = default(kernel, "spline36")
linear = default(linear, false)
rgba = default(rgb, a.isrgb ? true : false)
TV = default(TVrange,rgba ? false : true)
m1 = default(matrix, rgba || a.width<1280 ? "601" : "709")
m2 = default(matrixOut, Defined(matrix) ? matrix : ( rgba || w<1280 ? "601" : "709"))
fh = default(fh, 1.0) # Shrink (>1.0 Sharpen) or expand (<1.0 Blur) the Kernel
fv = default(fv, fh) # idem but vertically
lsb_in = default(lsb_in, false)
lsb_out = default(lsb_out, false)

function mmod2(clip a){
bh = 2 - ((a.Width() -1)%2 + 1)
bv = 2 - ((a.Height()-1)%2 + 1)
a.crop(bh/2,bv/2,-bh,-bv)}

size = a.width>=1280 && w<1280 ? true : false
mtrx = m1!=m2 ? true : false
if1 = mtrx && size ? true : false
if2 = linear && (!size || rgba) ? true : false

if1 ? Assert(linear==false, "HD to SD downsize requires linear=false") : nop
rgba ? Assert(mtrx==false, "You can't change matrix in RGB") : nop

!lsb_in && rgba ? a.mmod2 : a
!lsb_in ? (rgba ? Interleave(showred("yv12"), showgreen("yv12"), showblue("yv12")).Dither_convert_8_to_16 \
: (if1 ? Dither_convert_8_to_16.Dither_convert_yuv_to_rgb(lsb_in=true,matrix=m1,tv_range=tv,output="rgb48y") \
: Dither_convert_8_to_16)) \
: (if1 ? Dither_convert_yuv_to_rgb(lsb_in=true,matrix=m1,tv_range=tv,output="rgb48y") : last)

if2 ? Dither_y_gamma_to_linear(tv_range_in=tv,tv_range_out=false) : last
Dither_resize16(w%2+w,h%2+h,kernel=kernel,fh=fh,fv=fv,u=rgba?1:3,v=rgba?1:3)
if2 ? Dither_y_linear_to_gamma(tv_range_in=false,tv_range_out=tv) : last


!lsb_out && rgba ? Ditherpost(ampn=0.5,mode=6,u=1,v=1) : last
mtrx && !size ? Dither_convert_yuv_to_rgb(lsb_in=true,matrix=m1,tv_range=tv,output="rgb48y") : last
r = mtrx || rgba ? SelectEvery (3, 0) : nop
g = mtrx || rgba ? SelectEvery (3, 1) : nop
b = mtrx || rgba ? SelectEvery (3, 2) : nop

!lsb_out ? (rgba ? Mergergb(r,g,b) \
:(mtrx ? Dither_convert_rgb_to_yuv(r,g,b,lsb=false,matrix=m2,tv_range=tv) : Ditherpost)) \
: (mtrx ? Dither_convert_rgb_to_yuv(r,g,b,lsb=true ,matrix=m2,tv_range=tv) : last)}

[SilaSurfer]

Yeah I'm bombarding you guys with question. :( Work takes a lot of my time. Thanks Dogway for your presentation. I decided to spend some time with these great tools in Dither package and learn. Thanks again

[cretindesalpes]

msb_sharp=Contrasharpening(filtered,raw)

msb_sharp=mt_makediff (msb_sharp,filtered,y=3,u=3,v=3)
msb_sharp=mt_adddiff(msb,msb_sharp,y=3,u=3,v=3)
to16=msb_sharp.Dither_convert_8_to_16()

mask=DitherBuildMask(to16,filtered1)

This would more likely become:

sharp8 = Contrasharpening (filtered, raw)
mask = DitherBuildMask (sharp8, filtered)
to16 = sharp8.Dither_convert_8_to_16()

but the mask is 8 bit here, use Dither_merge16_8() instead of Dither_merge16.

edit: also what is the most accurate dither algorithm, similar to gradfun2db. Is for images.
The most accurate dithering algorithm are the error diffusion ones (6 to 8). Mode=6 is a classic serpentine Floyd-Steinberg and should give good results. It's a good idea to add a slight bit of noise before (ampn=0.5) to avoid artifacts related to this kind of algorithm. Anyway, I don't think gradfun2db dither can really be considered as good! But it does its job well and very quickly.

http://img225.imageshack.us/img225/6927/dither.png
Top: gradfun2db,
bottom: serpentine Floyd-Steinberg + noise

[Dogway]

yes, sorry cretin, I read up a few pages back and help and found out by myself mode=6. :angry:

Im gonna test the code.
edit:
This then?
filtered1=MT("""Dither1Pre(flt="tnlmeans(settings)",stacked=true)""",2,2,true)
msb=filtered1.Crop(0,0,706,478)
lsb=filtered1.Crop(0,478,706,478)
filtered=filtered1.DitherPost(mode=-1)

sharp8 = Contrasharpening (filtered, raw)
mask=DitherBuildMask (sharp8, filtered)
to16=sharp8.Dither_convert_8_to_16()

Dither_merge16(filtered1,to16,stackvertical(mask,mask)).Dither_resize16(720,480,kernel="spline36")
SmoothGrad(radius=16, thr=0.25, elast=2).DitherPost()

I dont know why, but this shifts my colors a bit.
edit2: No problem, it is because Im taking the ditherpost version instead of msb, so Im doing the right thing.

edit3: Ive been doing some more tests and Dither_convert_yuv_to_rgb is likely to be bugged, well, it just crashes when input is not mod4

[mandarinka]

While testing smoothtweak, I encountered a color hue shift when using this (http://forum.doom9.org/showthread.php?p=1504502#post1504502) code to import 16bit video into x264cli via avs2yuv. I have no idea where this minor difference comes from (it could be due to difference in x264's conversion between 16>10 and 8>10; it could be due to x264 expecting slightly different format of the 16bit raw data?). In any case, I thought I better post this in case it is some real/fixable issue.

I'm not sure if the tint appears in the 8bit encodes too, but it is perceivable to me when I compare the 10bit encodes (2 screens from each - most obvious on the hair in #1 and the door curtain in #2):
10bit encode using 16bit input via avs2yuv: #1 (http://img137.imageshack.us/img137/9174/000000011016bitavs2yuv.png) #2 (http://img715.imageshack.us/img715/9970/000000021016bitavs2yuv.png)
10bit encode using avs2yuv normaly (8bit yv12 all along): #1 (http://img89.imageshack.us/img89/9108/00000001108bitavs2yuv.png) #2 (http://img830.imageshack.us/img830/9984/00000002108bitavs2yuv.png)


---------------------------------
What I did & 8bit encode screens:

encoding from 16bit avs2yuv input:

AVS:
smoothtweak(contrast=0.84)
Dither_convert_8_to_16()

Interleave (Dither_get_lsb (), Dither_get_msb ())
TurnRight ()
AssumeFieldBased ().AssumeTFF ()
Weave ()
TurnLeft ()
CLI:
08bit: avs2yuv -raw XXXX-16bit.avs -o - | x264-1995.exe - --demuxer raw --input-depth 16 --input-res 704x480 --fps 23.976 --preset ultrafast --qp 0 -o XXXX.mkv
10bit: avs2yuv -raw XXXX-16bit.avs -o - | x264-1995-10.exe - --demuxer raw --input-depth 16 --input-res 704x480 --fps 23.976 --preset ultrafast --qp 0 -o XXXX_10.mkv

Results:
10bit encode #1 (http://img137.imageshack.us/img137/9174/000000011016bitavs2yuv.png) 10bit encode #2 (http://img715.imageshack.us/img715/9970/000000021016bitavs2yuv.png)
8bit encode #1 (http://img101.imageshack.us/img101/9924/00000001816bitavs2yuv.png) 8bit encode #2 (http://img864.imageshack.us/img864/9234/00000002816bitavs2yuv.png)


encoding from 8bit avs2yuv input:

AVS:
smoothtweak(contrast=0.84)
CLI:
08bit: avs2yuv -raw XXXX.avs -o - | x264-1995.exe - --demuxer raw --input-depth 8 --input-res 704x480 --fps 23.976 --preset ultrafast --qp 0 -o XXXX.mkv
10bit: avs2yuv -raw XXXX.avs -o - | x264-1995-10.exe - --demuxer raw --input-depth 8 --input-res 704x480 --fps 23.976 --preset ultrafast --qp 0 -o XXXX_10.mkv


Results:
10bit encode #1 (http://img89.imageshack.us/img89/9108/00000001108bitavs2yuv.png) 10bit encode #2 (http://img830.imageshack.us/img830/9984/00000002108bitavs2yuv.png)
8bit encode #1 (http://img29.imageshack.us/img29/1655/0000000188bitavs2yuv.png) 8bit encode #2 (http://img7.imageshack.us/img7/1516/0000000288bitavs2yuv.png)


The screenshots were taken with mplayer2 build by lachs0r (http://mplayer2.srsfckn.biz/):
mplayer2-high-bit-depth-20110609.exe -vo png XXXX.mkv

P.S.
Here are (for reference) the same screens using smoothtweak16 (no ditherpost) and 16bit avs2yuv output as in case 1: #1 (http://img828.imageshack.us/img828/5058/000000011016in.png) #2 (http://img821.imageshack.us/img821/1589/000000021016in.png)

[Yellow_]

Hi, I'm trying a simple script, import 8bit h264AVC, upto 16bit and out as 10bit lossless h264 using avs2yuv.

But can't get past the error, dfttest: unable to load libfftw3f-3.dll

I've downloaded it from the fftw site and using the 3.2.2.pl1 32bit version as all other stuff is 32bit, although I'm using Win7 64bit.

Read dither.html and have mvtools2, masktools alpha48, removegrain beta etc but still can't get it to work. :-(

Any ideas? Apart from usual stuff I have these lines for above.

dfttest (lsb=true)
Dither_convey_yuv4xxp16_on_yvxx()

[Yellow_]

Ok sorted, had to put libfftw3-3.dll etc in Windows/SysWOW64 :devil:

[Dogway]

A few notes on testing with my wrapped resizer.
Dither_convert_rgb_to_yuv: Fails on big inputs (greater than 1500px or so for any dimension)
Dither_convert_yuv_to_rgb: Fails on big inputs. Fails on not mod4
Linear also occasionaly makes some strong black outline/ringing, almost the same as ResampleHQ, I wouldn't had taken it as a bug if Archimedes didnt state (http://forum.doom9.org/showthread.php?p=1492573#post1492573) otherwise.