Color banding and noise removal

[Dogway]

cretindesalpes: Yes, I followed the same page diagrams and ended up with this (http://forum.videohelp.com/threads/369658-Converting-chroma-placement-from-DV-to-MPEG2-manually?p=2369301&viewfull=1#post2369301).
So it can be done manually with Dither, it matches Cedocida's conversion, and as you say Avisynth conversion is different, wrong IMO.
In theory the above Dither conversion should also work, but it's different too, although not an ideal method (due two a double chroma resize) maybe it could be fixed, I'm not sure.

Mainly I wanted to ask about high bitdepth input in x264, trying to clear up this post (http://forum.doom9.org/showthread.php?p=1704794#post1704794). I thought it did something else other than a "ditherpost" to Sierra algo, like some high bitdepth motion analysis, or else. In that regard an imaginary 8 bit Sierra dithered input would yield the same output as a 16-bit input? If that was the case the only benefit of high bitdepth input x264 builds would be for "10bit in->10bit out".

[Dogway]

I still think the above should be double checked. Converting a DV chroma encoded YUV format to RGB should produce a chroma cositing output (all planes aligned). Then from RGB back to YUV I use cplace="mpeg2", so my output should have MPEG2 chroma placement. Unless I'm reading the "cplace" setting wrongly.
It's possible that the interlaced flag could be causing the error, although I haven't checked without it.

Also a reminder of the above, is there any benefit using high bitdepth input for x264 when rendering to 8bit?

What I really came for is that I think there's a bug in Dither_resize16nr(), I was doing some tests and when using for output a resolution around 3K the noring cancellation wasn't working. I'm not sure if it's due to high demanding memory, I compared against Resize8 where it was working.

[feisty2]

Read the script and you will find out noring does not work on large factor upscale
It's not a bug, it's designed like this

[Dogway]

yes, found the note in the script, it's undocumented though.

[foxyshadis]

Does anyone know a way to deal with LWLibAVVideoSource's broken stack16 format? It opens 10-bit files as (msb/lsb) xxxxxx12/34567890 instead of 12345678/90xxxxxx.

Edit: Never mind, Dither_lut16("x 64 *",u=3,v=3) works perfectly.

[feisty2]

Does anyone know a way to deal with LWLibAVVideoSource's broken stack16 format? It opens 10-bit files as (msb/lsb) xxxxxx12/34567890 instead of 12345678/90xxxxxx.

Edit: Never mind, Dither_lut16("x 64 *",u=3,v=3) works perfectly.

LWLibavvideosource ("xxx",xxx,format="yuv4xxp16")

[foxyshadis]

Thanks! I didn't realize that was an option.

[Motenai Yoda]

Edit: Never mind, Dither_lut16("x 64 *",u=3,v=3) works perfectly.
or use "x 6 <<u"

[Boulder]

cretindesalpes,

sorry to bother you again, but would you be willing to add the speedup changes of svp's MVTools and the optimized SATD functions that the VapourSynth version has to your latest MVTools build?

I've been investigating switching to VapourSynth but changing some of my custom functions might not be too easy if not feasible at all :(

[Sparktank]

Would it be possible to add Jinc to the resizers (for upscale only) ?
http://forum.doom9.org/showthread.php?p=1655388#post1655388

[cretindesalpes]

Boulder:
Maybe if I can find the time to do it, but it is a lot of work and I’m not sure it is worth it (IIRC most of the SVP speed-up was related to private GPU code).

Sparktank:
Not at the moment. The JINC resizer is not of the separable kind and would require a different code path which is more work than just adding a kernel.

I’m currently busy merging the resizer codes from fmtconv and Dither, adding AVX2 support and a few other improvements while working on 32/64-bit/VS-dfttest/eedi3 at the same time, so things go slower than ever.

[Sparktank]

Sparktank:
Not at the moment. The JINC resizer is not of the separable kind and would require a different code path which is more work than just adding a kernel.

I thought it might be different due to being an upscaler only.
Just wanted to check.

I’m currently busy merging the resizer codes from fmtconv and Dither, adding AVX2 support and a few other improvements while working on 32/64-bit/VS-dfttest/eedi3 at the same time, so things go slower than ever.

Sounds like a workload!
Can't wait to see the future updates. :thanks:

[Boulder]

Is it possible to keep this whole package in high bit depth and just use Ditherpost over "output" at the end? I am very unsure how the Masktools2 calls react.


cleanedclip = MDegrain1(c,supermdg,bv1,fv1,thsad=200,lsb=true)
limited = mt_lutxy(cleanedclip,c,"x y < x 1 + x y > x 1 - x ? ?",U=3,V=3)
output = CSMDG(original,limited,supermdg,bv1,fv1,true)
return output

Function CSMDG(clip original, clip denoised, clip cssuper, clip cb1, clip cf1,\
bool HD, bool "chroma", bool "preblur", bool "preblurc", int "preR", bool "limit",\
string "blur", int "RGmode", int "RGmodeU", int "Repmode", int "RepmodeU", int "overshoot")
{
# contra-sharpening: sharpen the denoised clip, but don't add more to any pixel than what was removed previously.
# script function from Didee from the VERY GRAINY thread

chroma = Default(chroma, true )
preblur = Default(preblur, true )
preblurc = Default(preblurc,chroma )
preR = Default(preR, 1 )
limit = Default(limit, true )
RGmode = Default(RGmode, 11 )
RGmodeU = Default(RGmodeU, RGmode )
Repmode = Default(Repmode, 1 )
RepmodeU = Default(RepmodeU,Repmode)
overshoot = Default(overshoot, 0 )

HD ? eval("""
cb1=original.MCompensate(csSuper, cb1)
cf1=original.MCompensate(csSuper, cf1)
pmax = original.mt_logic(cb1, "max").mt_logic(cf1, "max")
pmin = original.mt_logic(cb1, "min").mt_logic(cf1, "min")""") : NOP()

s = denoised.MinBlur(HD?2:1,1) # Damp down remaining spots of the denoised clip.
allD = mt_makediff(original,denoised) # The difference achieved by the denoising.
ssD = mt_makediff(s,HD?s.removegrain(20,-1).\
removegrain(20,-1):\
s.removegrain(11,-1)) # The difference of a simple kernel blur.
ssDD = ssD.repair(HD?ssD.repair(allD,1):allD,HD?12:1) # Limit the difference to the max of what the denoising removed locally.
ssDD = SSDD.mt_lutxy(ssD,"x 128 - abs y 128 - abs < x y ?") # abs(diff) after limiting may not be bigger than before.

denoised.mt_adddiff(ssDD,U=2,V=2) # Apply the limited difference. (Sharpening is just inverse blurring)
HD ? mt_clamp(last,pmax,pmin,overshoot,overshoot,chroma="copy first") : last

return last
}

[cretindesalpes]

The contra-sharpening part alone can be converted to 16 bits (it has already been done). It think the mt_logic min/max can be converted too. However MCompensate is not compatible with 16-bit input or output.

[colours]

It think the mt_logic min/max can be converted too.

Someone wrote that too. (http://pastebin.com/D25HuLQ5) It is unfortunately a bit of a hack and also kinda slow. It could probably be optimised a little, but it'd definitely be slower than native code anyhow.

Using CLExpr would be faster if available (e.g. cl_exprxy(a,b,"x y max",lsb=true)), but apparently not everyone can use CLExpr.

[Boulder]

Fortunately MCompensate is run on a regular 8-bit super clip so that can be ignored.

Could someone please give me a hand in converting that blurb? I've been trying to get my head around converting the Masktools-related calls but I don't seem to understand the logic behind it. I did find Contra-Sharpen mod 16 but it seems to do many other things than the old simple contrasharpening. Using OpenCL shouldn't be a problem, at least NLMeansCL2 runs on my GPU.

[cretindesalpes]

rean: There is no definitive answer. Use ordered dithering (default setting for DitherPost and GradFun3) on the Dither side.
On the x264 side, increase --aq-strength a bit, and maybe the psy-trellis too. The exact values will obviously depend on the actual content and other compression settings.

[kuchikirukia]

Ok, how do I deband for 10 bit encoding? I shouldn't need dither since that's just a band-aid for 8 bit, right?
I've got a source that needs debanding, I'm resizing from 1080p to 720p (since I notice Dither can resize I'm throwing that in), and I'm encoding x264 10 bit through MeGUI. Suggestions?

[cretindesalpes]

kuchikirukia: On your video converted to 16 bits, use gradfun3() with lsb = true and lsb_in = true. Start with a low thr, like 0.2, and increase it by 0.1 steps until the banding is gone. Then send the result to your encoding application via Dither_out().

[kuchikirukia]

So I need avs2yuv?

Also, I'm antialiasing with santiag which can downscale as a part of its output. Dither can also resize. Which should I use? And how do I combine the 8 bit antialiasing with 16 bit debanding?

And, finally, does debanding improve compressibility? Since I'm 3/4ths done with this encode, I probably won't bother starting over if it doesn't save space. If it does, that's certainly an incentive add a high bit depth workflow to my repertoire.

[cretindesalpes]

kuchikirukia:

1. Yes, avs2yuv, avs2pipemod, or any modified x264 build that can take 16-bit data natively. You can also convert to 8 bits with DitherPost(mode=6) and let x264 convert it back to 10 bits before encoding.

2.a If resizing in 16 bits does a significant improvement for you, use Santiag to AA then call Dither_resize16 to resize. Otherwise, just go with Santiag. A bit better solution would be writing a small wrapping function around Dither_resize16 taking the same arguments (resize + crop) as the regular Avisynth resizers, and pass it to Santiag as scaler_post argument. However it will have to take 8-bit data as input and output.
But there is a third way: use an edge mask to select between Santiag and Dither_resize16 outputs.

2.b For mixed processing, please refer to the Dither doc: “Combining 8- and 16-bit processing.”

3. Sometimes, but by a tiny amount. It depends on your source.

[kuchikirukia]

If I just put the 8 bit first I shouldn't need anything special to combine them, right?

8 bit antialiasing/resize --> <convert to 16 bit> --> deband --> output.

Also, it was said earlier that x264's 16->10bit conversion was primitive. How do I use Dither to convert and what do I set for input? The wiki seems to show two different ways and I can't figure out where it's 10 bit output and where it's actually 16 bit.

a = Dither_get_lsb ()
b = Dither_get_msb ()
c1 = a.mt_lut ("x 6 >>", y=3, u=3, v=3)
c2 = a.mt_lut ("x 2 << 255 &u", y=3, u=3, v=3)
DitherPost (c1, c2) # Add the mode you want and other parameters here
mt_lut ("x 6 <<", y=3, u=3, v=3)
StackVertical (b, last)
Dither_out ()


Dither_quantize (
clip src,
int bitdepth,
bool reducerange (false),
int mode,
float ampo,
float ampn,
int pat,
bool dyn,
bool prot,
clip mask,
float thr,
bool interlaced,
int y,
int u,
int v,
bool staticnoise
)
Dither a clip to a given bitdepth. It’s possible to keep the resulting clip at the initial bitdepth of 16 or to reduce it to the specified depth.
To encode a clip in 10 bits using x264, set bitdepth=10 and reducerange=true, followed by Dither_out and set --input-depth 10 in the x264 parameters.

For a set of tools supposedly for high bit depth use, that wiki sure is light on how to actually use it.

[foxyshadis]

Also, it was said earlier that x264's 16->10bit conversion was primitive. How do I use Dither to convert and what do I set for input? The wiki seems to show two different ways and I can't figure out where it's 10 bit output and where it's actually 16 bit.

Don't know where you read that, but it isn't true. It uses Sierra-2-4A to dither both up and down. Also, in x264 10bit everything is internally 16-bit (except for decoded refs, which have to equal the bit-depth), so dithering down to 10 just so it can scale back up to 16 is crazy. x265 also uses Sierra-2-4A, but it internally keeps all frames at exactly the bit-depth of the encoding.

[cretindesalpes]

If I just put the 8 bit first I shouldn't need anything special to combine them, right?
Right. But you were previously mentioning using Dither’s 16-bit resizer.

Also, it was said earlier that x264's 16->10bit conversion was primitive.
Not really. There was a bug in the early 10-bit x264 versions but it has been fixed for a long time.

How do I use Dither to convert and what do I set for input? The wiki seems to show two different ways and I can't figure out where it's 10 bit output and where it's actually 16 bit.

a = Dither_get_lsb ()
b = Dither_get_msb ()
c1 = a.mt_lut ("x 6 >>", y=3, u=3, v=3)
c2 = a.mt_lut ("x 2 << 255 &u", y=3, u=3, v=3)
DitherPost (c1, c2) # Add the mode you want and other parameters here
mt_lut ("x 6 <<", y=3, u=3, v=3)
StackVertical (b, last)
Dither_out ()
This was the old method. I’ll remove it from the doc, as Dither_quantize now exists and is preferred. Thanks for having reported this inconsistency.

In short, use Dither_quantize(bitdepth=10, reducerange=true).Dither_out() if you really want to send 10-bit data to x264. Just use Dither_out() to send 16-bit data (preferred).

For a set of tools supposedly for high bit depth use, that wiki sure is light on how to actually use it.
The wiki looks outdated. Not surprising, as trying to maintain an exact copy of the official doc in a different format must be extremely boring (I’m not responsible for this).

[kuchikirukia]

Ooh, that's nice.

Thank you!

[Reel.Deel]

The wiki looks outdated. Not surprising, as trying to maintain an exact copy of the official doc in a different format must be extremely boring (I’m not responsible for this).

Thanks for pointing that out. I'm not responsible for that either but I'll try to update it to coincide with the current documentation. The dither wiki page is fairly new (created in March 2015) so it should be current but it seems that v1.23.0 was used as reference :(.


Regarding the official dither documentation there's some dead/outdated links in the troubleshooting section.

You surely have an old Masktools 2 version installed somewhere and autoloaded by Avisynth. Search on your disks the filenames mt_masktools.dll, mt_masktools-25.dll and mt_masktools-26.dll. Remove the culprits and install the latest Masktools 2 version from there.

The link to manao's homepage is dead. It's probably best to link to the updated MaskTools b2 (https://github.com/tp7/masktools/releases).

Also this:
Your RemoveGrain version is too old. Remove RemoveGrain*.dll, Repair*.dll and SSE?Tools.dll and install the version 1.0b "beta" from there, but do not copy the SSE3 files. This is not the latest version, but the most compatible with all the existing scripts around there.

The correct RemoveGrain link: http://avisynth.nl/index.php/RemoveGrain
It also might be best to recommend users to use RgTools (https://github.com/tp7/RgTools/releases). No more doubts on which binary to use :).

[kuchikirukia]

While you're at it, there's a syntax error:
mask_details= details.mt_binarize (lvl_details)>

And if you want to edit the santiag page, there's a mistake there as well:
http://avisynth.nl/index.php/Santiag

scaler_pos

pos doesn't work. It's "post" as in the example:
scaler_post="Spline36Resize")

[Reel.Deel]

@kuchikirukia

I am responsible for the santiag page :p. Fixed; thanks for pointing that out.

[kuchikirukia]

All I can say is thank God for great error reporting. I would've given up on avisynth in a day if MeGUI didn't tell me exactly where to look.

[cretindesalpes]

Reel.Deel: thanks. I updated the documentation for the upcoming 1.27 release.

[Desbreko]

Also, I'm antialiasing with santiag which can downscale as a part of its output. Dither can also resize. Which should I use? And how do I combine the 8 bit antialiasing with 16 bit debanding?

If you don't need santiag specifically, you could use xaa (http://www.mediafire.com/download/sygi04y47eknvc2/xaa_v1.1.1.avsi) for this. When outputting 16-bit, it will retain the lsb from working in 16-bit whenever possible, including combining the 8-bit antialiased clip with the 16-bit resized clip with an edge mask.

So, you could do something like this:

# 8-bit 1080p source
xaa(ow=1280, oh=720, mode="nnedi3", lsb=true)
GradFun3(thr=0.2, lsb=true, lsb_in=true)
Dither_out()

[kuchikirukia]

Tried it and it was slower than santiag for no apparent gain. I only noticed one difference in one scene and it was in santiag's favor.

[Desbreko]

How fast xaa runs is highly dependent on what settings you use, so it might be worth tinkering to see if you can get a better result than my example. Or santiag may just be better suited to your source since it works somewhat differently from xaa.

[Dogway]

for confirmation, does this apply as "Random Dither"?:
DitherPost(mode=-1, ampn=1)

[sneaker_ger]

I don't understand. Why does ampn do anything when we are only rounding?

/edit:
It's just adding noise? Then I don't think it's equal to dithering. Noise is just noise, it does not preserve gradients.

[cretindesalpes]

Dogway: yes.

sneaker_ger: We are just rounding, but after the noise addition. Actually rounding is implemented as dithering using a pattern full of zeroes.

Dithering is related to noise. It breaks correlated quantization noise and turns it into uncorrelated noise, perceptually much less annoying. Adding noise before quantization is a valid way to do it but has a poor global SNR (depends on the noise amount).

[Dogway]

Do you think it can work as an intermediate for further filtering despite the poor SNR? IMO rounding creates some ugly banding and F-S which I was using before is meant to be an "end look" due to its pattern based condition from what I read here (http://forum.doom9.org/showthread.php?p=1727107#post1727107).

I'm kinda switching the old DitherBuildMask method for hacky lsb_in with sub16/add16, which hopefully makes more sense, even more with the above random dither intermediate.

[cretindesalpes]

You’d better use error-diffusion dithering like Floyd-Steinberg rather than noise + quantization. The result will always be much better, no matter what the further processing steps will be. The only drawback is that it will be a bit slower.

[Dogway]

edit:nevermind, just read your post here (http://forum.doom9.org/showthread.php?p=1728059#post1728059).

[feisty2]

or better, do not dither at all, add native high bit depth support to filters lack of that

[cretindesalpes]

Dither 1.27.0 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Added cplaces, cplaced and csp to Dither_resize16.
Backported some code from fmtconv to Dither_resize16.
Added the slice parameter to disable multi-threaded slicing for error-diffusion dithering (DitherPost and related functions).
Added Dither_min_dif16.
AVX2 optimisations for Dither_resize16, Dither_add16, Dither_sub16, Dither_median16, Dither_max_dif16, Dither_min_dif16 and Dither_limit_dif16.
Added the BT.1886 transfer cruve to the gamma conversion functions.
Added the dispm parameter to change the conversion mode in Dither_srgb_display.
Improved the quality of the subsampling in Dither_bilateral16.
Radius is no longer limited in Dither_bilateral16 as well as in GradFun3 in smode 2.
Fixed a bug in the chroma filling (u or v < 0) with non-YV12 colorspaces.

[Sparktank]

Thanks a lot for the continual work!
Nice update.

Looking forward to playing with it.

[Bloax]

Clearly the future is motion-based dither that is easily predicted by lossy encoders.

[feisty2]

Clearly the future is motion-based dither that is easily predicted by lossy encoders.

Clearly the future is the death of low bit depth

[real.finder]

after update to 1.27.0

I can't use nnedi3_resize16 in upscale

in avspmod will show this

Traceback (most recent call last):
File "avsp.pyo", line 9061, in OnMenuVideoToggle
File "avsp.pyo", line 13855, in ShowVideoFrame
File "avisynth.pyo", line 462, in GetFrame
WindowsError: [Error -1073741795] Windows Error 0xC000001D

[StainlessS]

Looks like 0xC000001D might be Illegal Instruction (there seem to be other things attributed if you google it eg bad device driver).
Provide cretindesalpes with details of OS and CPU, perhaps some new assembler instructions added.
EDIT: Would not hurt to mention Avisynth version too.

[real.finder]

Looks like 0xC000001D might be Illegal Instruction (there seem to be other things attributed if you google it eg bad device driver).
Provide cretindesalpes with details of OS and CPU, perhaps some new assembler instructions added.
EDIT: Would not hurt to mention Avisynth version too.

cpu intel(r) core(tm) i7 cpu x 980 @ 3.33ghz

os windows server 2008 r2

avs http://forum.doom9.org/showthread.php?t=148782

2015.02.20 or 2013.03.09

[Groucho2004]

after update to 1.27.0

I can't use nnedi3_resize16 in upscale

in avspmod will show this

Traceback (most recent call last):
File "avsp.pyo", line 9061, in OnMenuVideoToggle
File "avsp.pyo", line 13855, in ShowVideoFrame
File "avisynth.pyo", line 462, in GetFrame
WindowsError: [Error -1073741795] Windows Error 0xC000001D
I can confirm the same crash. It doesn't seem to matter which Avisynth version is used.
I also tried to insert a another simple 8 Bit filter (sharpen) using the recommended methods, crashes with the same error.

[cretindesalpes]

Fixed in dither 1.27.1 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Fixed a wrong code path in Dither_limit_dif16 causing a crash on CPU without AVX2 instruction set. Thanks to real.finder and Groucho2004 for their report.
Documentation typos fixed thanks to Colours.

[LouieChuckyMerry]

Hello and thanks in advance for any help :) . Please, I'm wondering if the following script has the proper order and syntax:

Blu-raySourceVideo
SMDegrain(Lsb=True,Lsb_Out=True)
F=DitherPost(Mode=-1)
S=F.FastLineDarkenMod()
D=MT_MakeDiff(S,F).Dither_Convert_8_To_16()
Dither_Add16(Last,D,Dif=True,U=2,V=2)
GradFun3(Radius=16,Lsb_In=True,Lsb=True)
# DitherPost()
Dither_Out()

for encoding with 10-bit x264 (adding "--demuxer raw --input-depth 16 --sar 1:1" to the x264 command line). That is, is it correct to go: denoise, sharpen, deband? And are the Dither lines for FastLineSharpenMod then onto GradFun3 the best method? Thanks in advance for any help.