Color banding and noise removal

Im liking the results from your script, I just dislike how GradFun3(smode=1) is so slow. I went from a 4 hour pass on a movie to a 12 hour pass. Is this supposed to be like that? because regular GradFun3() runs MUCH faster it just doesnt give quite as good of results as smode1.

[cretindesalpes]

Originally Posted by mswaino2

I just dislike how GradFun3(smode=1) is so slow. I went from a 4 hour pass on a movie to a 12 hour pass. Is this supposed to be like that?

smode=1 is indeed slower than smode=0, but not that slower. Something must be wrong. Are you sure your figures are right? Anyway, if you do a 2-pass encoding, render first to a lossless file then encode it, you'll save one avisynth pass.

Originally Posted by cretindesalpes

smode=1 is indeed slower than smode=0, but not that slower. Something must be wrong. Are you sure your figures are right? Anyway, if you do a 2-pass encoding, render first to a lossless file then encode it, you'll save one avisynth pass.

I dunno, maybe its just Im using it in the script wrong or something. Could it be because Im using a denoiser and sharpener along with it?

I been doing a 2 pass encode with this for a script

LoadPlugin("C:\Program Files\megui\tools\dgindex\DGDecode.dll")

Import("C:\Program Files\AviSynth 2.5\plugins\LimitedSharpenFaster.avsi")

LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\mt_masktools-25.dll")
Import("C:\Program Files\AviSynth 2.5\plugins\dither.avsi")
Import("C:\Program Files\AviSynth 2.5\plugins\GradFun2DBmod.avsi")
Import ("C:\Program Files\AviSynth 2.5\plugins\mt_xxpand_multi.avsi")
DGDecode_mpeg2source("C:\Users\Owner\Desktop\Case Closed - The Last Wizard Of The Century\VTS_06_1.d2v", info=3)
LoadPlugin("C:\Program Files\Megui\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
#deinterlace
crop( 0, 0, -2, 0)
LanczosResize(852,480) # Lanczos (Sharp)
LoadPlugin("C:\Program Files\megui\tools\avisynth_plugin\UnDot.dll")
Undot() # Minimal Noise
deen()
LimitedSharpenFaster(ss_x=2.0, ss_y=2.0, strength=150, overshoot=0, undershoot=0, soft=0, edgemode=0)
GradFun3(smode=1)
__film = last
__t0 = __film.trim(0, 142520)
__t0

the only thing I have changed is the script and its taking a very long time for some reason. I was getting 11-12 FPS now Im at 3 FPS after adding it.

[Dogway]

cretindesalpes: Is it possible to add lsb_in feature in mvtools2. It would be very useful for certain sources with blocking+noise, where I would run a spatial dfttest pass then mdegrain. I read that normally spatial denoisers should go before temporal.

[mandarinka]

You can use a prefilter for the motion vector search (which I assume is completely fine to do in 8bit), to partially achieve that.


Something like this I think (hopefuly I got it right).

prefilt = FFT3Dgpu(sigma=2,bw=32,bh=32,bt=3,ow=16,oh=16,plane=4)
 
superfilt = prefilt.MSuper(pel=2, sharp=1)
super = MSuper(pel=2, sharp=1)
backward_vec2 = MAnalyse(superfilt, isb = true, delta = 2, overlap=4)
backward_vec1 = MAnalyse(superfilt, isb = true, delta = 1, overlap=4)
forward_vec1 = MAnalyse(superfilt, isb = false, delta = 1, overlap=4)
forward_vec2 = MAnalyse(superfilt, isb = false, delta = 2, overlap=4)
 
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=300,lsb=true)

[Dogway]

Hello mandarinka. Thanks for the input, I meant to denoise, not feed a better suited (prefiltered) clip for motion estimation. I know that lsb_in in mvtools is going to be quite annoying for cretindesalpes to make since he would probably need to adapt all the functions (manalyse,mmask,mcompensate,etc) for lsb_in in mdegrain. But I think it makes sense because spatial+temporal denoisers normally operate internally first spatial then temporally (dfttest,tnlmeans,etc)

By the way I found probably a bug in Dither_resize16 when processing a .png file. I tested and the error comes from the fh and fv parameters when resize dimensions are small:
This is my code, pitty I couldn't take advantage of high bitdepth rgb to yuv conversion : P

converttoyv12(matrix="PC.601")
MT("""Dither1Pre(flt="tnlmeans()",stacked=true)""",2,2,true)
dither_resize16(224,316,kernel="spline64",fh=1.1,fv=1.1)
Dither_convert_yuv_to_rgb(lsb_in=true,tv_range=false,output="rgb32",mode=6,ampn=0.5)

[cretindesalpes]

Originally Posted by Dogway

I know that lsb_in in mvtools is going to be quite annoying for cretindesalpes

Yes it would... but I could just leave the analysis part in 8 bits and focus on MSuper/MDegrain. Still, it's a lot of work, so I don't plan to do this in the near future.

By the way I found probably a bug in Dither_resize16 when processing a .png file.

I can't reproduce it here, everything looks OK. What is the bug, exactly? What are the dimensions of the original picture?

[Dogway]

Originally Posted by cretindesalpes

Still, it's a lot of work, so I don't plan to do this in the near future.

No problem, I knew it was. You hard work on Dither is already impressive for a one man.

Originally Posted by cretindesalpes

I can't reproduce it here, everything looks OK. What is the bug, exactly? What are the dimensions of the original picture?

Dimensions are 610x1114. I think its relevant because the other images work.

[cretindesalpes]

Originally Posted by Dogway

Dimensions are 610x1114. I think its relevant because the other images work.

OK I spotted the problem, a tile size calculation error again, resulting in a buffer overflow.

Fixed (I hope so) in Dither 1.9.4.

[TheProfosist]

Originally Posted by cretindesalpes

smode=1 is indeed slower than smode=0, but not that slower. Something must be wrong. Are you sure your figures are right? Anyway, if you do a 2-pass encoding, render first to a lossless file then encode it, you'll save one avisynth pass.

im getting crazy slower speed with smode=1 as well

smode=0 1.87FPS
smode=1 0.12FPS
this is with 1080p

[cretindesalpes]

Originally Posted by TheProfosist

im getting crazy slower speed with smode=1 as well

Weird.

I ran a few speed tests with 1280x720 progressive input, in single-threading and multi-threading modes. I added a naked dfttest() because it's the core of GradFun3(smode=1) and I would suspect this is the one slowing down the whole thing. Also added GradFun2DBMod for reference.

#SetMTMode (5, 4)
FFVideoSource ("random 720p source")
#SetMTMode (2)
Trim (0, -1000)

# Select one of these lines
#NOP ()
#GradFun3 (smode=0, mask=0)
#GradFun3 (smode=0)
#GradFun3 (smode=1, mask=0)
#GradFun3 (smode=1)
#GradFun3 (smode=2, mask=0)
#GradFun3 (smode=2)
#dfttest (sigma=35, tbsize=1, sbsize=36, sosize=27, lsb=true)
#GradFun2DBmod ()

PointResize (32, 32)
# Encoded with: x264 --preset ultrafast --output NUL "speed.avs"

Results (Phenom II X4 965 3.4 GHz):

                             |  Single-threaded  | Multi-threaded x4
                             |     fps    CPU    |     fps    CPU
-----------------------------+-------------------+------------------
NOP ()                       |   453.93   33 %   |   444.44   33 %
GradFun3 (smode=0, mask=0)   |    30.72   27 %   |    89.63   90 %
GradFun3 (smode=0)           |     7.38   25 %   |    26.74   99 %
GradFun3 (smode=1, mask=0)   |     7.96   78 %   |     9.53   97 %
GradFun3 (smode=1)           |     4.38   55 %   |     7.65   97 %
GradFun3 (smode=2, mask=0)   |    10.03   26 %   |    32.80   98 %
GradFun3 (smode=2)           |     4.93   25 %   |    17.25   99 %
dfttest (...)                |     8.87   85 %   |    10.11   98 %
GradFun2DBmod ()             |     5.19   25 %   |    18.95   96 %

So, GradFun3(smode=1) is 3.5 times slower than GradFun3(smode=0) in 4x MT mode, and just 1.6 times slower in single-threaded mode.

I'll check later if I get very different figures for 1080p input.

[Dogway]

Another error: 1434x1476 dimensions for Dither_resize16 resizing to half. Hope you get this kind of problems sorted, sorry I can't help more.

[cretindesalpes]

Originally Posted by Dogway

Another error: 1434x1476 dimensions for Dither_resize16 resizing to half. Hope you get this kind of problems sorted, sorry I can't help more.

I can't reproduce it. Is 1434x1476 the input size? Halving 1434 gives 717 which is odd. Could you please post the script?

[Yellow_]

I'm using Dither to go from 8bit h264AVC to 16bit OpenEXR image sequences but really those should be linear not gamma encoded, the compositing application I'm importing them into assumes linear. So does the Dither_y_gamma_to_linear function serve this purpose?

Is it only necessary to linearize the luma, more accurate and technically correct to do it that way before the conversion to RGB, rather than a typical 0.45 reverse gamma on all channels in RGB data after conversion?

Also as the source was encoded with a BT709 transfer curve not sRGB 2.2, to undo that ie: linearize YCbCr, I should be assuming something like the reciprocal of 2.35? If I understand correctly this helps prevent compressing shadow detail that can occur applying 0.45 to BT709 source?

[Dogway]

This is the code I'm testing with:

Interleave(showred("yv12"), showgreen("yv12"), showblue("yv12")).Dither_convert_8_to_16
Dither_resize16(1434/2,1476/2,kernel="spline36",y=3,u=1,v=1)

The error is very ugly, I thought Dither_resize did some kind of rounding. Anyway in the LinearResize function I changed target dimensions to (w%2+w,h%2+h). Nice to know it wasn't THAT error again.

As a last question, Im forcing to proces Dither_y_gamma_to_linear in PC range to have more data to work with. Do you think this is more detrimental than beneficial?

[cretindesalpes]

Dither 1.9.5 released:

• Bug fixed in Dither_resize16(), displaying green bars when SSE2 optimisations are disabled.
• Better check of stack16 clip dimensions.

Originally Posted by Dogway

As a last question, Im forcing to proces Dither_y_gamma_to_linear in PC range to have more data to work with. Do you think this is more detrimental than beneficial?

TV range is all about keeping a headroom for the ringing causing by some filters (resizers, sharpeners…), so with PC range this headroom is lost. But this is not a big difference, and the bottom headroom is cleared by the gamma/linear conversion anyway.

Originally Posted by Yellow_

the compositing application I'm importing them into assumes linear. So does the Dither_y_gamma_to_linear function serve this purpose?

Yes.

Is it only necessary to linearize the luma, more accurate and technically correct to do it that way before the conversion to RGB, rather than a typical 0.45 reverse gamma on all channels in RGB data after conversion?

GammaYUV -> linearYUV -> linearRGB will give you wrong colors. Inverse colorspace conversions should be done in the same gamma/linearity as the forward conversion. For gamma YUV to linear RGB conversion you can use:

Dither_convert_yuv_to_rgb (output="rgb48y")
Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false)
Dither_convey_rgb48_on_yv12 (
\   SelectEvery (3, 0),
\   SelectEvery (3, 1),
\   SelectEvery (3, 2) )

Also as the source was encoded with a BT709 transfer curve not sRGB 2.2, to undo that ie: linearize YCbCr, I should be assuming something like the reciprocal of 2.35? If I understand correctly this helps prevent compressing shadow detail that can occur applying 0.45 to BT709 source?

You're right, it appears that BT709 doesn't use the same transfer curve as sRGB, the slope at 0 in sRGB is much steeper than in BT709. If you want to linearize the R'G'B' components converted from the BT709 Y'CbCr, apply on each one : R = R'/4.5 if R' < 0.081 or R = ((R' + 0.099) / 1.099) ^ (1/0.45) if R' ≥ 0.081. The formula are the same as in sRGB, but with different values. I will add the BT709 mode to Dither conversion functions in a future release.

[SilaSurfer]

Hey cretindesalpes. Just wanted to stop by and say outstanding work. Finished off an encode using your ordered dithering in my filter chain. Awasome results. Thanks again for all of your hard work.

[Dogway]

hey I just needed to use addborders inside stacked 16b. Thought on sharing the function (primitive but useful). Thanks for the new version too!

Function Dither_addborders16 (clip src, int "left", int "top",
\	int "right", int "bottom")
{
	left   = Default (left,   0)
	top    = Default (top,    0)
	right  = Default (right,  0)
	bottom = Default (bottom, 0)
	
 src
	msb = crop(0,0,width,height/2).addborders (left, top, right, bottom)
	lsb = crop(0,height/2,width,height/2).addborders (left, top, right, bottom)

	StackVertical (msb, lsb)}

btw Im also looking forward the v&c implementation

[mandarinka]

Originally Posted by SilaSurfer

Hey cretindesalpes. Just wanted to stop by and say outstanding work. Finished off an encode using your ordered dithering in my filter chain. Awasome results. Thanks again for all of your hard work.

Indeed. Dither brought quite some fresh momentum into the abilities of avisynth, single-handedly. Thanks from me too.

[Alek93j]

Function zzz_denoise (clip src, float "sigma", int "thr", bool "mask", int "sad")
{
	sigma = Default (sigma,    16)
	thr   = Default (thr,       5)
	mask  = Default (mask,  False)
	sad   = Default (sad,     200)

	w = src.Width ()
	h = src.Height ()

	# Motion analysis
	super   = MSuper (src)
	super_a = MSuper (src.TTempSmooth ().RemoveGrain (12))

	fwd_vect_3 = super_a.MAnalyse (isb=false, delta=3, overlap=4)
	fwd_vect_2 = super_a.MAnalyse (isb=false, delta=2, overlap=4)
	fwd_vect_1 = super_a.MAnalyse (isb=false, delta=1, overlap=4)
	bck_vect_1 = super_a.MAnalyse (isb=true,  delta=1, overlap=4)
	bck_vect_2 = super_a.MAnalyse (isb=true,  delta=2, overlap=4)
	bck_vect_3 = super_a.MAnalyse (isb=true,  delta=3, overlap=4)

	fwd_comp_2 = src.MCompensate (super, fwd_vect_2, thSAD=sad)
	fwd_comp_1 = src.MCompensate (super, fwd_vect_1, thSAD=sad)
	bck_comp_1 = src.MCompensate (super, bck_vect_1, thSAD=sad)
	bck_comp_2 = src.MCompensate (super, bck_vect_2, thSAD=sad)

	# Spatio-temporal denoising using modified dfttest
	c_dft = Interleave (fwd_comp_2, fwd_comp_1, src, bck_comp_1, bck_comp_2)
	c_dft = c_dft.dfttest (sigma=sigma, lsb=true)	# Double height
	c_dft = c_dft.SelectEvery (5, 2)

	# Temporal-only denoising using modified MDegrain
	c_deg = src.MDegrain3 (super, bck_vect_1, fwd_vect_1, bck_vect_2, fwd_vect_2, bck_vect_3, fwd_vect_3, thSAD=sad, lsb=true)	# Double height

	# Spatio-temporal denoising smoothes too much the details,
	# therefore we use pure temporal denoising on edges or detailed areas.
	edge_src = c_deg.Crop (0, 0, w, h)
	edge_mask = edge_src.mt_edge (mode="prewitt", thY1=thr, thY2=thr)
	edge_mask = edge_mask.mt_expand ()
	edge_mask = StackVertical (edge_mask, edge_mask)	# Double height
	c_hyb = mt_merge (c_dft, c_deg, edge_mask, luma=true, y=3, u=3, v=3)

	return (mask ? edge_mask.GreyScale () : c_hyb)
}

Maybe i'm gonna say something stupid, but this line:

super = MSuper (src)

It's used only by MCompensate and MDegrain3, which don't need to use levels=0 (if I understood right), so, it would be:

super = MSuper (src,levels=1)

to get the same result with a little increase of the velocity?