Color banding and noise removal

[ajp_anton]

No, it definitely works. The extra chroma resolution is clearly visible on video game footage, and even more so when loading the encoded video into Avisynth and comparing the chroma planes directly.
Maybe you forgot the "--output-csp i444", which is always needed for 4:4:4 encoding?

edit:
So the full command line would be
avs2pipe26mod -rawvideo script.avs | x264-10b - --demuxer raw --input-res wxh --input-depth 16 --input-csp i444 --output-csp i444 --fps x --crf x -o o.mkv

[cretindesalpes]

Oh you're right, I only had the --input-csp. Thanks for the tip! I'll add it to the documentation.

[Yellow_]

I'm getting weird ghosting exporting 16bit to 10bit x264 with 1.12.0 that with the same script worked fine on 1.10.0 using the following:

LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\mt_masktools_26.dll")

FFmpegSource2("MVI_8068.MOV")

trim(0,10)

Dither_convert_8_to_16 ()

Dither_convey_yuv4xxp16_on_yvxx()

Most of the image is intact but there is a semi transparent overlap starting from the left which travels across the frame as it plays, just seems to affect chroma and only certain colors, lots of grey appearing too.

I'll try to upload a image frame later.

[cretindesalpes]

I'm getting weird ghosting exporting 16bit to 10bit x264 with 1.12.0 that with the same script worked fine on 1.10.0
This is weird, because dither.avsi hasn't changed from 1.10.0 to 1.12.0. Are you there wasn't any problem with 1.10.0?

One request if I may, would it be possible to add to your function Dither_Add_Grain_16 to be able to add 3 layers of grain like what one would get with Grainfactory3 from Didée? (hope he's well) Thanks.
I might consider this later, but it's currently not in my priority task list.

[Yellow_]

Ok, my bad I left the -raw out of the avs2yuv command line. so dumb, sorry for confusion. :-(

[SilaSurfer]

I might consider this later, but it's currently not in my priority task list.

Thanks cretindesalpes. No preasure. Keep on the good work. ;)

[Dogway]

Several questions in Dither_convert_yuv_to_rgb could you implement rgb24? This in case it saves something because I never use alpha.
Also it's strange, when I convert yuv red color AFAIK (81, 90, 240) to RGB in TV range I get 238,14,14, is it ok?
Do you still have planned v&c algorithm, Im curious to know how it compares to bayer dithering

[cretindesalpes]

Several questions in Dither_convert_yuv_to_rgb could you implement rgb24?
OK.

Also it's strange, when I convert yuv red color AFAIK (81, 90, 240) to RGB in TV range I get 238,14,14, is it ok?
Are you sure ?

BlankClip (pixel_type="YV12", color_yuv=(81*256+90)*256+240)
Dither_convert_yuv_to_rgb (tv_range=true)

gives me the expected values around (255, 0, 0), and I obtain (238, 14, 14) with tv_range=false. Have you this problem with a specific setting?

Do you still have planned v&c algorithm, Im curious to know how it compares to bayer dithering
I haven't made any progress on this topic. I'm currently trying to multithread internally the MAnalyse/MRecalculate part of the MVTools because this is a major speed bottleneck for lots of scripts, but understanding how all this works is a quite complex and time-consuming task.

[Dogway]

No, its ok, it just looked like wanting to be 240,16,16. Im converting a bunch of frames for manual editing and wanted to be sure, but I will probably use tvrange true so no range conversion is done.


No problem on v&c, at your pace. Now that you are on mvtools2 it could be interesting if you could have a look on how to make it more flexible, like
mdegrain not relying on vectors/mask, so one could mix spatial denoiser and mdegrain through the mvtools motion mask, you can see my discussion in the mvtools thread. This could also enable a more orthodox way for "fake" lsb_in in mdegrain, and play a role on the masktools pipeline.
Multithreading the motion search stage is going to be a big leap, you might already know the SVP group are on the GPU direction with this, but I prefer your tools because they work rock steady. No need to hurry.

[Yellow_]

Is there anything that can be done to improve memory usage when using Dither_convey_yuv4xxp16_on_yvxx() -> AVS2yuv -> Imagemagick written 16bit image sequences? On larger video files the process fails, with out of memory errors by trying to do the whole video in memory I guess before writing out of IM. :-(

LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
LoadPlugin("c:\Program Files (x86)\AviSynth 2.5\plugins\mt_masktools_26.dll")

FFmpegSource2("MVI_8068.MOV")

Dither_convert_8_to_16 ()

Dither_convert_yuv_to_rgb(matrix="601", tv_range=false, cplace="MPEG2", chromak="bicubic", lsb_in=true, output="rgb48y")
Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false, curve="709")
Dither_convey_rgb48_on_yv12 (SelectEvery (3, 0),SelectEvery (3, 1),SelectEvery (3, 2) )



Although I could probably batch write scripts to trim() so many frames into individual .avs files and batch convert, is there a more efficient way. Reduced bit depth to 12bit? But memory would just run out later in the process?

On another query, I see in the Dither.html that blending frames is possible, so is it possible to use Dither tools to do typical blend modes like overlay, multiply etc in linear light at 16bit depth? I'm interested in working this with new high bit depth tools in Avisynth now. http://forum.doom9.org/showpost.php?p=617523&postcount=219 & here: http://www.dcs.gla.ac.uk/~jhw/filmy/, not so much simulating film grain, don't see much point in that unless mixing computer generated imagery with film scans, but the build up of layers.

[Dogway]

The interlaced parameter in ditherpost is for interlaced inputs or (separated)fields input? I wonder because it would be unpractical to work 16 bit in real interlaced material.
Also I have been wondering about the motion search in x264, isn't it fooled by the ordered dither as in mvtools2? or does it use another approach for it.

At last Im trying to optimize smdegrain depending on processed planes. But there are some behaviours I don't get to grasp.

Shouldn't this be true?


dfttest(lsb=true,U=true,v=true)
DitherPost(mode=-1,u=2,v=2)
==
dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=3,v=3)

I also would think that in the first case UV output would be the MSB part of the clip which is 8 bit, but that also doesn't apply if I compare it to :

dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=2,v=2)
dfttest(Y=false)

[cretindesalpes]

Is there anything that can be done to improve memory usage when using Dither_convey_yuv4xxp16_on_yvxx() -> AVS2yuv -> Imagemagick
If I understand correctly your description of the problem, data accumulate in the pipe until Imagemagick decides to process them, probably after the pipe is closed at the AVS2yuv end. I don't know if replacing the pipe with an intermediate file would work, but you can try.

On another query, I see in the Dither.html that blending frames is possible, so is it possible to use Dither tools to do typical blend modes like overlay, multiply etc in linear light at 16bit depth?
It's possible to blend 8-bit pictures into a 16-bit frame using the masktools. However blending 16-bit pictures is another story, mainly because we lack functions that can combine two or three 16-bit clips.

The interlaced parameter in ditherpost is for interlaced inputs or (separated)fields input?
For interlaced input. It differs from progressive input only when using the error diffusion modes.

Also I have been wondering about the motion search in x264, isn't it fooled by the ordered dither as in mvtools2? or does it use another approach for it.
Sure but this is not a problem here if motion estimation is inaccurate in nearly flat areas. x264 isn't a temporal denoiser, it doesn't need to perfectly overlay fine details or textures. It has to take advantage of the data redundancy, and this is what an ordered dither can offer.

dfttest(lsb=true,U=true,v=true)
DitherPost(mode=-1,u=2,v=2)
==
dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=3,v=3)
No, because in the first case, the U and V plane have been filtered by dfttest. Whatever the conversion to 8 bits done afterward, you'll never obtain the input back. Using u=2, v=2 in DitherPost will keep only the MSB (truncation) instead of rounding, like cropping the top half of a stack16 frame.

[Dogway]

For interlaced input. It differs from progressive input only when using the error diffusion modes.
I see, although I don't know when would it apply since interlaced inputs are commonly filtered after separated fields, in any case I rarely use error diffusion.


Using u=2, v=2 in DitherPost will keep only the MSB (truncation) instead of rounding, like cropping the top half of a stack16 frame.
Yes that makes sense, as I explained afterwards it would more likely keep the msb part when using u=2 v=2, but I compared it also to filtering chroma after ditherpost in 8 bits and it isn't the same so the question is whether the MSB part is like the result of filtering in 8 bit or it is something different.

dfttest(lsb=true,U=true,v=true)
DitherPost(mode=-1,u=2,v=2)
==
dfttest(lsb=true,U=false,v=false)
DitherPost(mode=-1,u=2,v=2)
dfttest(Y=false)

[Yellow_]

cretindesalpes, thanks. Investigating further I've found that Imagemagick has a function called -limit, which can be set to restrict how much memory IM can use before it has to write temp files to disk. Using this on the CLI:

identify -list resource

Gives info on what resources IM thinks is available. Then setting -limit memory some GiB and -limit map some GiB functions on the avs2yuv command line, to levels within those resources keeps memory errors away. :-)

[TheProfosist]

DitherPost dithers? a 16bit clip to 8bit. Then what does GradFun3 dither? What im trying to ask is how is using SmoothGrad+DithePost different from using GradFun3?

[AzraelNewtype]

DitherPost dithers? a 16bit clip to 8bit. Then what does GradFun3 dither? What im trying to ask is how is using SmoothGrad+DithePost different from using GradFun3?

DitherPost turns stacked 16bit, usually created by using a denoiser that is designed to take 8bit input and yield that special output format, into dithered single height 8bit. GradFun3 takes your 8bit video and dithers it to attempt to deband source that's already been filtered, in much the same way GradFun2 and its modifications did before it. Which you should use depends on your workflow.

[TheProfosist]

DitherPost turns stacked 16bit, usually created by using a denoiser that is designed to take 8bit input and yield that special output format, into dithered single height 8bit. GradFun3 takes your 8bit video and dithers it to attempt to deband source that's already been filtered, in much the same way GradFun2 and its modifications did before it. Which you should use depends on your workflow.

will DitherPost help with debanding at all? I know that SmoothGrad should by smoothing the gradients.

[Dogway]

TheProfosist: Gradfun3 basically is the same as:

Dither_convert_8_to_16()
put a low pass denoiser here...
ditherpost

with some masks, etc. Cretindesalpes explained it at some point in the thread.

Ditherpost just takes whatever 16 bit processing clip and converts it to 8 bit with the consequent dither so you don't see banding. So ditherpost in nature doesn't deband but rather prevents banding when doing filtering (among other things).

For example GradFun3 with smode=3 is the same code as above but with a bunch of smoothgrad() lines in between with varying settings. So I guess smoothgrad is also some kind of a very lowpass denoiser.

If you never filter in 16 bit, use gradfun3 to deband. If instead you filter in 16 bit (denoisers, resizers, etc) then before ditherpost add a smoothgrad() line, and I think you are good to go, I do this oftenly.

[TheProfosist]

If instead you filter in 16 bit (denoisers, resizers, etc) then before ditherpost add a smoothgrad() line, and I think you are good to go, I do this oftenly.
This is what I have been doing. Thanks for the info.

[SSH4]

This is a planned feature. In the meantime… here is a quick and dirty hack:

# Assume stack16 clip here

n = 10 # Final bitdepth (range: 8 to 14)
msb = Dither_get_msb ()
lsb = Dither_get_lsb ()
hr = 2
m = String (Pow (2, 16 - hr - n) - 1)
l = msb.mt_lut ("x "+m+" &u", y=3, u=3, v=3)
h = msb.mt_lut ("x 255 "+m+" - &u", y=3, u=3, v=3)

s1 = String (n - 8)
ofs = String (Pow (2, 7 - hr))
a = StackVertical (l, lsb)
a = a.Dither_lut16 ("x "+s1+" <<u "+ofs+" 256 * +", y=3, u=3, v=3)

b = a.DitherPost ()

s2 = String (16 - n)
ofs2 = String (Pow (2, 15 - hr - n))
b_msb = b.mt_lut ("x "+s1+" >>u 255 &u "+ofs2+" - 128 +", y=3, u=3, v=3)
b_lsb = b.mt_lut ("x "+ofs+" - "+s2+" <<u 255 &u", y=3, u=3, v=3)

c = mt_adddiff (h, b_msb, y=3, u=3, v=3)
StackVertical (c, b_lsb)

Dither_convey_yuv4xxp16_on_yvxx ()

how about this kind dither LSB from 8bit to 2bit (16bit to 10bit)


# Assume stack16 clip here

a=Dither_get_lsb()
b=Dither_get_msb()
a.mt_lut("x 6 >>",u=3,v=3)
DitherPost (last,a,mode=6, u=3, v=3)
mt_lut("x 6 <<",u=3,v=3)



And why ampn not work on mode 2-5 ?

And what can give better encoding result (quality/size on same CRF) on x264-10bit encode:
• dithered (ordered or floid steinberg + static noise) 8-bit source;
• 16-bit (MSB/LSB)
• 10-bit dithered (probably with static noise too) from 16-bit
?

[cretindesalpes]

how about this kind dither LSB from 8bit to 2bit (16bit to 10bit)
Yes it works too, but I think you omitted a StackVertical() at the end. Actually I developed the more complex solution to ensure the quantized values can spread across the 2-bit levels, if required (but it's not perefect). Extreme example (http://screenshotcomparison.com/comparison/88852) using ampo=4. Also, significant improvement can be done by shifting the LSB before dithering. Example (http://screenshotcomparison.com/comparison/88862) showing the amplified difference between a 16-bit gradient directly dithered to 8 bits and the same gradient using the 10-bit intermediate. Here is a fixed version:
a=Dither_get_lsb()
b=Dither_get_msb()
c1 = a.mt_lut("x 6 >>",u=3,v=3)
c2 = a.mt_lut ("x 2 << 255 &u", u=3, v=3)
DitherPost (c1,c2,mode=6, u=3, v=3)
mt_lut("x 6 <<",u=3,v=3)
StackVertical (b, last)

and why ampn not work on mode 2-5 ?
Because the modes 2–5 use a piece of code different from mode 0–1 and modes 6–8, and I was too lazy to add the noise stuff to it. Moreover, I now consider these modes obsolete. The patterns for values +1/4 and +3/4 look less resilient to compression than ordered dithering, although they may be perceptually a bit better (but this is questionable). I might remove them later to simplify further code maintenance, if nobody complains.

Edit:

And what can give better encoding result (quality/size on same CRF) on x264-10bit encode:
• dithered (ordered or floid steinberg + static noise) 8-bit source;
• 16-bit (MSB/LSB)
• 10-bit dithered (probably with static noise too) from 16-bit
?
I haven't a firm answer to this question, one should conduct tests with different types of content. I would think noise/grain of very low level could play a key role between choosing dithering to 8 or 10/16 bits. Compression size drops drastically when the picture looks almost stable and clean after dithering, with distinct patterns on flat areas, compared to more obvious grain.

[SSH4]

nice.


#put here any source with banding :)

n=6 # bit dithering
o=String(8 - n) r=String(n)

a=last
d=a.mt_lut("x " + o + " >>",u=3,v=3)
a.mt_lut("x " + r + " << 255 &u", u=3, v=3)

b=DitherPost (d,last,mode=6,ampn=1,staticnoise=true, u=3, v=3).mt_lut("x " + o + " <<",u=3,v=3)

Interleave(a.Subtitle("orig"),b.Subtitle("dither"))

[SSH4]

oops. why src_left not work right for me (on all version dither.dll)?

http://i30.fastpic.ru/big/2011/1023/65/_2b4cda3e71e3d176768bf917f8406465.png?noht=1

Dither_resize16 (1920,1080,src_left=9,src_width=-7,kernel="bicubic",a1=0,a2=0.33)

while BicubicResize(1920,1080,0,0.33,src_left=9,src_width=-7) work right on already dithered source.

[cretindesalpes]

You're right, I spotted the bug a few days ago and it is now fixed, along with some other bugs related to Dither_resize16 and extreme horizontal/vertical resizing ratios. I need to do a bit more testing. I'll make a new release as soon as possible.

[SSH4]

May i be a beta tester? :) One my project stalled because i can't out for this moment 1080p in 16/10bit.

And here i made some tests with 16/10/8 bit output and encoding in 10-bit x264:
BD source VC1 "opening" filtered with dfttest (16bit mod) with motion compensation.

Three output:
8 bit (DitherPost(mode=6,ampn=1,staticnoise=false)
10 bit (MSB+LSB dithered with DitherPost(mode=6,ampn=1,staticnoise=false)
16 bit LSB+MSB
encoded with hq settings with 2-pass in 5000K/s and 3000K/s

16 bit 5Mb/s - http://www.mediafire.com/?3bxu032qsbcyxdx
10bit 5Mb/s - http://www.mediafire.com/?oodq8xyrz6brmcj
8 bit 5Mb/s - http://www.mediafire.com/?62jlpy7m3gclbtf
16 bit 3Mb/s - http://www.mediafire.com/?1sj5iavs85xryux
10 bit 3 Mb/s - http://www.mediafire.com/?cf9l942zmc1ecaz
8 bit 3 Mb/s - http://www.mediafire.com/?yo3m33c5mrf7m8o

And dumn i can't understand what look better. 8bit sometime looks better because grain a little bit more visible than 10Bit or 16. Sometime 10 or 16 looks better than 8 :(

[Keiyakusha]

SSH4
Just a random thought. Where is the source we can compare these results to? Also are you sure you not throwing away extra information in 10-16 bit files? For example you can't open them in avisynth like any other 8bit ones. What decoders you use - also matters.
EDIT: also it doesn't sounds right that in 8 bits grain looks better, preserving grain and dither is the point of 10 bit.

[SSH4]

hmm, you want me upload luseless HUFFYUV LSB/MSB avi in >10Gb??

avs2yuv rawpipe to x264-10bit
avs2yuv raw dither_convey_... MSB/LSB 16bit pipe to x264 raw input 16bit
and as we talk later in http://forum.doom9.org/showthread.php?p=1533008#post1533008 10bit (2bit dithered LSB) and MSB/LSB 16bit pipe to x264 raw input 16bit....

Just check results. what result look better than other, not what result better or more like original.
I understand that 16 bit without dither vs 10 bit (2-3bit dithered) or 8 bit (3-4 bit dithered) not the same sources. and probably we must use different psy-rd, tree and other settings for compare results...

But anyway. with identical encode settings in same bitrate what file looks better?
Because if i try encode in CRF 10/16 bit source beat 8bit (4-5Mb/s vs 8-10Mb/s)

[Keiyakusha]

hmm, you want me upload luseless HUFFYUV LSB/MSB avi in >10Gb??
Eh, no. I was referring to unprocessed VC1 stream. Or better to say small part of it. I just realized that you didn't tell how exactly you filtered stuff so I won't be able to replicate it anyway. But whatever.
By the way I misunderstood that by 8bit you only mean 8bit source but output from x264 is still 10 bit.
To me 16 and 10 bits looks better than 8 bits. 8 bits has more artifacts, that's all. Between 10 and 16 bits my choice is 10, because 16 is an overkill, none of them is better they just different.
EDIT: however, personally I use different deband/dither so I'm not sure which one (10 or 16) bits consumes less resources in your case. If that's 16bit one, then my choice is 16bit.
BTW, I'm talking about 5mbps samples, all 3mbps looks not good to even compare.

[SSH4]

I think you understand why x264-10 bit can made better result on 8bit stream (specially on gradients) than 8bit x264...

This is not about filtering, this question about 8/10/16 bit
So my test is compare similar source but in 8/10/16 bit (3-4bit/2-3bit/0 bit dither) and encode it in 10bit x264 only.

ok, 10bit looks more better for me too. on 8bit sometime looks like grain can help protect a little bit more textures than in 10/16bit.
But mostly clean source better in 10/16 bit because it made less artifacts.
But... clean source "CLI" ^___^ looks not real.
So my choice use 10bit with add small grain in MSB for more realistic result. And i hope i can test it on my Mushishi "remaster" :)

[Keiyakusha]

I think you understand why x264-10 bit can made better result on 8bit stream (specially on gradients) than 8bit x264...
Sure.
Yet again, If you filter stuff in 16 bit, I don't see reason to go to 8 as you seems to be doing in (8 bit (DitherPost(mode=6,ampn=1,staticnoise=false)).
But... clean source "CLI" ^___^ looks not real.
I wonder if it was really meant to look real :Р

[cretindesalpes]

Dither 1.12.1 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Fixed a bug with Dither_resize16, occuring with positive values for src_left or src_top.
Fixed other issues in Dither_resize16, related to huge differences between vertical and horizontal resizing ratios. It's not perfect yet, but still much better than before.
Dither_convert_yuv_to_rgb now supports RGB24 output colorspace.
Simplified Dither_crop16 for pure horizontal cropping.

[SSH4]

cretindesalpes Thank you!

Sure.
Yet again, If you filter stuff in 16 bit, I don't see reason to go to 8 as you seems to be doing in (8 bit (DitherPost(mode=6,ampn=1,staticnoise=false)).


Yep, i think you right.

But:
• x264 just clip 6 bites from 16 bit, without any dither.
• 8bit data because of rounding in the encoding process take 10bit values.
• Most of sources i ever seen have or need some grain.

So i try find best workflow for me.
Original style avisynth 8bit, or avialable with dither.dll 16bit LSB/MSB, but more difficult to implement.

[Dogway]

is it possible to add u and v to dither_merge16_8?
Also is there a way to extract a plane of a 16 bit clip? I suspect this involves using luts functions...

[cretindesalpes]

Yes, I'll probably rewrite this function as a native plugin, and then I will add the y, u and v plane selectors.

But what do you mean "extract a plane"?

[Dogway]

The idea came from the Color Vibrance thread (http://forum.doom9.org/showthread.php?t=162882)
Vtoy(),Utoy() and YToUV(). I didn't know how to do that in 16 bit. There might be more uses for these functions than I can think now, maybe it's not important, but I felt a bit limited when I thought about it. Same for mergechroma and mergeluma.


edit:
To put an example:

16 bit input
...
o=ditherpost(mode=6)
a=Dither_lutxy8(o,o.SwapUV(),"x 128 - 2 ^ y...",y=2,u=3,v=3)
...
16 bit output (but luma comes from a ditherpost stage)


So now I need to mix in 16 bit the luma previous to the ditherpost stage with the chroma of the dither_lutxy8().
My initial thoughts are:

Dither_merge16(last,a,mt_lut("65535",u=-65535,v=-65535),y=2,u=4,v=4)

Then I guess this can become sort of a mergeluma, mergechroma workaround for 16bit.

Greyscale or how to get Y plane:
dither_lut16("x",y=2,u=-128,v=-128)

Ditherpost if applied in this case dithers chroma too despite being flat, so one gets 129 values in UV planes.

About UtoY, VtoY, YtoUV, I have no idea, nor I know if they would be of any use in 16 bit. I'm just thinking loud I guess, so sorry for that...

[cretindesalpes]

You can use all these functions (UToY, VToY, MergeXXX, …) with 16 bit clips. Of course, for functions with multiple clip arguments, all clips must be of the same bitdepth.

[redfordxx]

Hi I am little bit confused with so many parameters with the Dither_resize16, so I ask plainly.
What should I do, when I need to do on the stacked 16bit clip exactly same thing as following does on 8bit:
mt_convolution(horizontal=h,vertical=v,total=t,...)Thank you

[cretindesalpes]

Currently, there is no direct mapping between mt_convolution and Dither_resize16. You'll have to scale the kernel coefficients manually (dividing them by t, if required). Also, it's not possible to use a different kernel for vertical and horizontal directions, so you'll have to do it in 2 passes:
Dither_resize16 (Width(), Height()/2, kernel="impulse "+h, fh=-1, center=false)
Dither_resize16 (Width(), Height()/2, kernel="impulse "+v, fv=-1, center=false)
I'll add some parameters in a next release to make Dither_resize16 more compatible with mt_convolution.

[vampiredom]

One simple thing I have done is to use FFT3dFilter, with fairly strong sigma, then blend that with the original clip to a certain percentage, like:

orig = last
filt = FFT3DFilter(sigma=2)
Merge(filt, orig, weight=0.25)

This seems to do a fairly good job of reducing visible noise while maintaining detail and avoiding banding (depending on the source, of course).

[redfordxx]

One simple thing I have done is to use FFT3dFilter, with fairly strong sigma, then blend that with the original clip to a certain percentage, like:My personal experience:
dfttest keeps more detail than dctfilter

[redfordxx]

I made a function for convolutions, so I am sharing it here. function Convolve(clip c, val hor, val ver, float "tot", bool "lsb", int "hctr", int "vctr", int "y", int "u", int "v", bool "info")
{
#Performs flat convolution of given dimensions...v0.1 by redfordxx
#needed Dither 1.12
# issues: Dither_resize16 applies kernel in reverse
# Dither_resize16 normalizes kernel, so tot parameter is ignored in 16bits
#Usage parameters:
#hor, ver arguments...dimensions of the convolution or kernels
#tot...divider, negative values are relative to sum of elements, so for 3x3 convo and tot=-2 the resul is tot=18, tot=0 is automatic. Doesnot work in 16bits
#lsb...is input stacked 16bit?
#hctr,vctr...centerpoints of the convolution string, by default in the middle...ignored if given hor and ver are kernels
#info...shows convolution strings on screen
y=default(y,3)
u=default(u,3)
v=default(v,3)
htext = IsString(hor) ? hor : ""
vtext = IsString(ver) ? ver : ""
hi = IsString(hor) ? 1 : int(hor)
vi = IsString(ver) ? 1 : int(ver)
lsb=default(lsb,false)
info=default(info,false)
horcenter=default(hctr,(hi+1)/2)
vercenter=default(vctr,(vi+1)/2)
sum=hi*vi
tot=default(tot,0.0)
tot = (tot>0) ? tot
\ : (tot<0) ? -sum*tot
\ : 0
hcoef = 1
vcoef = 1
uh = " "+ string(hcoef)
uv = " "+ string(vcoef)
uhl =strlen(uh)
uvl =strlen(uv)
flt=((int(tot)!=tot)&&(lsb==false)) ? ".0" : ""
shl=(hi-2*horcenter+1)*(lsb ? -1 : 1 ) #Dither applies convolution in reverse
svl=(vi-2*vercenter+1)*(lsb ? -1 : 1 )
strh=uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh+uh
strv=uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv+uv
str0=" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0"
hh=leftstr(str0,max(2*shl,0))+leftstr(strh,uhl*hi)+flt+leftstr(str0,max(-2*shl,0))
vv=leftstr(str0,max(2*svl,0))+leftstr(strv,uvl*vi)+flt+leftstr(str0,max(-2*svl,0))
hh=(htext=="") ? hh : htext
vv=(vtext=="") ? vv : vtext
hvsame=(hh==vv) # when hor and vertical is same, do 16bit in one pass
c
(!lsb && tot==0) ? mt_convolution(horizontal=hh,vertical=vv,y=y,u=u,v=v) : nop
(!lsb && tot>0) ? mt_convolution(horizontal=hh,vertical=vv,total=tot,y=y,u=u,v=v) : nop
(lsb && hh!="1" && hvsame) ? Dither_resize16 (Width(), Height()/2, kernel="impulse "+hh, fh=-1, fv=-1, center=false) : nop
(lsb && hh!="1" && !hvsame) ? Dither_resize16 (Width(), Height()/2, kernel="impulse "+hh, fh=-1, center=false) : nop
(lsb && vv!="1" && !hvsame) ? Dither_resize16 (Width(), Height()/2, kernel="impulse "+vv, fv=-1, center=false) : nop
info ? Subtitle(" h=("+hh+") v=("+vv+")") : nop
info ? Subtitle(" 16bit="+string(lsb)+" total=("+string(tot)+") size=("+string(hor)+","+string(ver)+") center=("+string(horcenter)+","+string(vercenter)+") ",y=24) : nop
return last
}
I hope it is understandable what it does, so won't comment more, if you wanna know just play with parameters with info=true, see what is the difference eg on the following
convolve(5, 4, info=true)
convolve(5, 4, tot=-2, lsb=false, hctr=-3, vctr=1, info=true)
convolve(5, 4, tot=0, lsb=true, hctr=-3, vctr=1, info=true)
convolve("1 0 -1 0 1", 4, tot=0, lsb=false, hctr=-3, vctr=1, info=true)

There are two issues introduced by dither, see them in the comments
@cretindesalpes: I wonder whether these are intents or bugs
Now I believe that it was main reason why I was confused with using Dither_resize16

[vampiredom]

dfttest keeps more detail than dctfilter

Ah, interesting. I am certainly not that well-versed in all the available denoisers for AviSynth. I will have to check it out. My comment was not really intended to suggest that FFT was the best way, but rather the concept of blending the denoised output with the original video in small degrees as a method of reducing visible banding.

[cretindesalpes]

Dither_resize16 applies kernel in reverse
Dither_resize16 normalizes kernel, so tot parameter is ignored in 16bits
[...]
I wonder whether these are intents or bugs
Not really a bug nor an intent. I implemented the convolution in the simplest possible way, and didn't care about the kernel traversal sign, because resizing kernels are symmetric. I'll fix these issues in the next release.

[ajp_anton]

Avisynth open failure:
Evaluate: operands of '==' and '!=' must be comparable
(dither.avsi, line 850)
(dither.avsi, line 814)
I tried to set u=3,v=3 in dither_lut8.
In the function dither_lut8_lsb, line 850, it checks if u or v are defined, but if they are, it then tries to check what y is. But if y is not defined, it fails. You need to set y=default(y,3) somewhere.
Don't know if there are other places with this error.

Also, a feature request...
Would it be possible to convert into any colorspace by defining the matrix yourself?

[cretindesalpes]

Dither (http://forum.doom9.org/showthread.php?p=1386559#post1386559) updated to v1.13.0:

Added kernelh, kernelv, totalh and totalv to Dither_resize16. Please note that the custom convolution coefficients are now read in the correct order.
Fixed an error with the plane selectors in the Dither_lut*8 functions.
Updated MVTools to v2.5.13.1: MAnalyse, MRecalculate: added an SATD approximation for blksize > 16 and dct >= 5. Previously, the SATD calculation was silently bypassed and always returned a null SAD value. Actually this is more a mixed SAD-SATD calculation, because the SATD for large blocks is implemented as a sum of the SATD of smaller blocks, instead as a SAD of the Hadamard transform of the whole block. It was already the case for blksize=8 or above.
Functions now check that the pel setting is the same in the superclip and the motion vectors, instead of crashing if not.


Would it be possible to convert into any colorspace by defining the matrix yourself?
Do you have a specific use in mind? I could add other standard matrices, if required. Or give control over the kr, kg and kr parameters. But defining the 9 matrix coefficients + 3 constants would be a bit more complex.

[ajp_anton]

Well I was thinking of YCgCo, but then thought there might be others so a more general solution might be possible.

[Dogway]

I'd like to ask if currently the staticnoise option of ditherpost works like the wOxxOm workflow (http://forum.doom9.org/showpost.php?p=1517049&postcount=241), or instead noise is spread evenly in the image. Ideally one would want noise only where ditherpost is dithering more aggresively.

I also noticed gradfun3 needs the staticnoise option. It could be a RAM problem but gradfun for smode=2 at least feels very unstable I usually have crashes, raising ampo over 1 also makes it unstable but I'm not sure which one is the trigger. My tests are in 16bit pipeline. I'm tackling fine banding here.
GradFun3(0.6,14,lsb_in=true,smode=2,lsb=true)
ditherpost(mode=0,ampn=0.5,staticnoise=true,ampo=1.2)

BTW I'd like to know if it's ok to use ampn with ampo at the same time when using ordered dither as in the example.

Another idea I had is to use floyd-steingberg on detailed areas to shoot for accuracy, but ordered dither on low gradients, if it leaves banding then use the staticnoise as mentioned. The drawback is having to use 2 ditherposts. I don't know if mixing 2 different algos harms too much.

At last to confirm something I didnt get clear:
(interlaced source)
separatefields
dfttest(lsb=true,tbsize=1)
ditherpost(mode=6,interlaced=true)
weave
Is this ok?

[cretindesalpes]

I'd like to ask if currently the staticnoise option of ditherpost works like the wOxxOm workflow (http://forum.doom9.org/showpost.php?p=1517049&postcount=241), or instead noise is spread evenly in the image. Ideally one would want noise only where ditherpost is dithering more aggresively.
No, currently the noise (static or not) is global. If the mask is active, only the non-edge area will be debanded and dithered, whatever the noise settings. The function is very basic actually.

I also noticed gradfun3 needs the staticnoise option. It could be a RAM problem but gradfun for smode=2 at least feels very unstable I usually have crashes, raising ampo over 1 also makes it unstable but I'm not sure which one is the trigger. My tests are in 16bit pipeline. I'm tackling fine banding here.
GradFun3(0.6,14,lsb_in=true,smode=2,lsb=true)
ditherpost(mode=0,ampn=0.5,staticnoise=true,ampo=1.2)
I couldn't reproduce any crash here. Could you post the full script? What is the size of your input clip? Which Avisynth version do you run?

BTW I'd like to know if it's ok to use ampn with ampo at the same time when using ordered dither as in the example.
Yes, the noise will just add to the dither pattern before bit reduction.

Another idea I had is to use floyd-steingberg on detailed areas to shoot for accuracy, but ordered dither on low gradients, if it leaves banding then use the staticnoise as mentioned. The drawback is having to use 2 ditherposts. I don't know if mixing 2 different algos harms too much.
There shouldn't any problem with this.

At last to confirm something I didnt get clear:
(interlaced source)
separatefields
dfttest(lsb=true,tbsize=1)
ditherpost(mode=6,interlaced=true)
weave
Is this ok?
Noting that tbsize=1, yes it should work but Weave() before DitherPost(), that's how it is intended to work in interlaced mode.

[cretindesalpes]

The Dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559) has been updated to v1.13.1:
Fixed height checks in DitherPost with interlaced content.
Updated dfttest to v1.9: added the quiet parameter to deactivate the filter spectrum output when sigma is specified with sstring/ssx/ssy/sst.
Updated MVTools to v2.5.14.0: Added MStoreVect, MRestoreVect and MScaleVect, from the Vit's MVExtras plugin. Function names have been changed to avoid collisions with MVExtras.

[Nevilne]

This is very sweet!
Seeing as vector scaling is being developed again, can I ask if this is possible:

Manalyze on half size clip with pel=1
Mflow/blur/whatever on full clip with pel 2 or 4

I've tried tinkering with adjustSubPel and other settings but there seems to be no way,
and using pelsearch=0 + blank pel clip is not very clean.

Also, vector scaling doesn't work with divide>0.
I didn't want to bug Vit with this but since you've included vector scaling it would be great if you could look into it :o