Color banding and noise removal

[mastrboy]

Added Dither_removegrain16 (modes 1–4, 11, 12, 19 and 20 only) and Dither_repair16 (modes 1–4 and 11–14 only).


Nice, speed wise, is it SSE(2/3) optimized?
I want to run some benchmarks compared to the original removegrain and need to know which dll file to use....

[cretindesalpes]

Nice, speed wise, is it SSE(2/3) optimized?
I want to run some benchmarks compared to the original removegrain and need to know which dll file to use....

Yes, it is optimised for SSE2, but don’t expect something anywhere close to the original RemoveGrain. First because I suck at optimizing and I don’t want to spend weeks on assembly stuffs that I couldn’t read and fix two months after. Secondly because it has to handle 16-bit data instead of 8 which divides the optimal pixel throughput by 2. Third because the stack16 format has a large overhead for small operations like these.

However the good thing is that it is internally multithreaded, so it’s not that bad.

[mastrboy]

I see, so no real point in comparing them speed wise.
Understand completely about the assembly code though, once tried to change a hardcoded timeout value in a assembly bootcode app, took me hours to just find the timeout code within that mess (Not commenting code should be illegal).

The multithreading is based on avstp i guess? So we get the advantages of having a shared thread pool...

[Reel.Deel]

So I noticed that Dither_removegrain16 modes 0-4 dont output the LSB while modes 11-20 do. Is that the expected behaviour?

PS it's very nice to see removegrain16 work on all planar colorspace. :)

[cretindesalpes]

Reel.Deel:

If your input is 8 bits and comes straight from Dither_convert_8_to_16(), it’s normal. RG modes 1–4 don’t create new values, they just shuffle existing pixels.

mastrboy:

Yes it uses avstp.

[TheProfileth]

cretindesalples it is really cool to see you porting removegrain modes to dither tools. Do you plan to also port the removegrainHD modes/filters as well? I think it would be really cool to see some of those filters function with the help of 16bit accuracy.
On a side note, have you ever considered just doing a 16bit median filter with a variable radius?
Currently the only viable median filter with a variable radius that I know of is medianblur.
Would be really cool to see a proper 16bit median filter, and while I don't do much coding myself a median filter shouldn't be that hard to make given that you already have removegrain(mode=4) in dither tools
Much love for your excellent work at advancing avisynth as a whole.
Take care

[Keiyakusha]

I was using dither 1.19 and avisynth 2.6 32bit mt from 16may 2012 (don't remember who built it, maybe SEt) on win7 x64
then i copied following files from 1.20 to avisynth plugin folder: avstp.dll; dither.avsi; dither.dll; mt_xxpand_multi.avsi (with overwrite)
then copied dither's mvtools2.dll and avstp.dll on top of that. (with overwrite)
After that when I open avs script in some software and reload it few times it crashes. I did something wrong?
After rolling back to dither 1.19 all is fine again.
I didn't used any part of the dither package in the actual script when it crashed.

[cretindesalpes]

I can’t see what you could have done wrong but I would need a little more information to know if there is a problem with v1.20. What kind of crash? (some programs can export a detailed crash report) Is it reproducible? Is it script-dependant? Is it dependant on the host application? Are you auto-loading the plug-ins?

[Keiyakusha]

What kind of crash? (some programs can export a detailed crash report)Is it dependant on the host application?Sorry forgot that. Avsp says "error loading avisynth" and says probably some plugins are unstable. Then crashes, or survives sometimes(but preview window resets). VirtualDub just hangs so no reports for you... Probably not dependant, not sure that i have other apps where i need/can to reload scripts often.

Is it script-dependant?
I don't have much time trying lots of scripts. I used, few colorspace conversions, removegrain, resize, f3kdb, mpeg2source
Videoclip was containing 1 frame only, odd-width yv24 was involved, if that matters...
EDIT: i realise that removegrain doesn't supports yv24 i used it before conversion
EDIT2: and this is classic removegrain, not new one from dither of course.

Is it reproducible?
From 10 reloads it for sure happens once. Wasn't able to reproduce with 1.19 unless I'm really unlucky. No other plugins or scripts was changed for some time And I was already doing similar processing before and never noticed crash.

Are you auto-loading the plug-ins?
It is probably obvious now but yes. I wouldn't be posting here otherwise. It looks like autoload-related issue.

[nhope]

From the 3rd post:

dfttest 1.9.2 (mod16)
MVTools 2.6.0.5 (mod16)

Does the "(mod16)" mean that these versions only work with mod16 resolution? Do they differ from the original versions in that respect?

[cretindesalpes]

Keiyakusha:

Thanks for your report, tell me if you have news about this crash.

nhope:

No, it just means it’s a modification to support 16-bit clips.

[Keiyakusha]

cretindesalpes
I realize that its probably impossible to fix something or even identify where is the problem, I'm fine with that. But just in case want to ask what news you expect? Since the changes introduced in 1.20 is not interesting to me I planned to just use 1.19, which I used since you 1st time posted it and never experienced crash (or how should i call this, autoload fail?)... I haven't updated other plugins. No one really develops them these days...
Howewer I just remembered, one more plugin I updated. AddgrainC... but i used it since you posted it and it was ok. And to dither 1.20 updated only today.

[nhope]

@cretindesalpes Thanks. I was confusing it with the abbreviation for "Modulus 16".

[zero9999]

Using Avisynth 2.6 Dither_convert_yuv_to_rgb(), Dither_srgb_display(), etc. fail because Dither_convert_yuv_to_rgb() passes Y8 data to Dither_lut16_msb/Dither_lut16_lsb which passes it directly to mt_lutxy. But mt_lutxy doesn't support Y8 so it fails.

[cretindesalpes]

Masktools2 do support Y8. Be sure you have only "mt_masktools-26.dll" in your plug-in folder, not the "mt_masktools-25.dll" version which is for Avisynth 2.5.x.

[zero9999]

Masktools2 do support Y8. Be sure you have only "mt_masktools-26.dll" in your plug-in folder, not the "mt_masktools-25.dll" version which is for Avisynth 2.5.x.
now i feel like a complete idiot. somehow the 2.5 masktools seems to have made it back into my plugins folder along with the 2.6 version.

[Yellow_]

hi, cretindesalpes, do the dither tools with Avisynth 2.5 cover the problem I'm having with some odd raw UYVY:

http://forum.doom9.org/showthread.php?p=1590264&posted=1#post1590264

The source is JFIF encoded full range chroma + luma, chroma +- 128

Described here by Poynton:

http://books.google.co.uk/books?id=dSCEGFt47NkC&pg=PA175&lpg=PA175&dq=chroma%2BJFIF%2Bvideo&source=bl&ots=OL4zeJ7Vyr&sig=aoNstetOv-HN7PpZ3i-dOj2JxVs&hl=en#v=onepage&q=chroma%2BJFIF%2Bvideo&f=false

One solution suggested makes use of kassandro's RemoveGrain:

http://home.pages.at/kassandro/RemoveGrain/

[cretindesalpes]

Yellow_:

If it’s a signed/unsigned integer problem, I would replace 127 with 128 in the lut expression: "x 128 + 256 %". Unfortunately I don’t know much about JFIF.

—

Dither 1.21.0 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Added the seed parameter to Dither_add_grain16. Reworked Dither_srgb_display for mor accurate aspect ratios and transfer characteristics, added gamma parameter. Gamma transfer functions have now an optional gamma correction parameter. Added Dither_median16 for median and quantile clipping of any radius, but slow and without temporal component at the moment.

[TheProfileth]

—

Dither 1.21.0 (http://forum.doom9.org/showthread.php?p=1386559#post1386559):
Added Dither_median16 for median and quantile clipping of any radius, but slow and without temporal component at the moment.

Thank you so much :o this is what I have been waiting for, for a long time.

[TheProfileth]

BTW cretindesalpes, regarding the temporal element of the current 16 bit median that you recently (and thankfully) added I was wondering if you were planning on implementing the basic limitational system from removegrain that worked quite well in degrainmedian that being this


We consider the same 3x3x3 cube.
New pixel candidate value (newp) is clipped by values of neighbor pixels from pair (bound1 and bound2).
But mode1-mode5 use more safe criterion (weight) of optimal pixel pair,
We take into account also the difference of new value from central pixel old value (oldp) .

Mode=0 of DeGrainMedian (similar to mode=9 of RemoveGrain) uses weight=|bound1 -bound2|, it is strongest.
Mode=1 of DeGrainMedian (more strong than mode=8 of RemoveGrain) uses weight=|oldp - newp|+4* |bound1 - bound2|
Mode=2 of DeGrainMedian (similar to mode=8 of RemoveGrain) uses weight=|oldp - newp|+2* |bound1 - bound2|
Mode=3 of DeGrainMedian (similar to mode=7 of RemoveGrain) uses weight=|oldp - newp|+ |bound1 - bound2|
Mode=4 of DeGrainMedian (similar to mode=6 of RemoveGrain) uses weight=2*|oldp - newp|+ |bound1 - bound2|
Mode=5 of DeGrainMedian (similar to mode=5 of RemoveGrain) uses weight=|oldp - newp|, it is weakest.

I do understand that this would probably require some more inventive thinking or complex coding in order to extend this past the originally intended 3 frames being processed at once. While we are on the topic of degrainmedian, if you were to implement a limit parameter that was to limit the amount of change per pixel, would it she represented in standard 8bit form to us then converted to its 16bit equivalent or would you allow for direct 16bit numbers? By that I mean will the limit parameter be between 0 and 255 or between 0 and 65535? Maybe I am being a bit presumptuous by asking this in advance though.

Also I would like to ask some questions regarding the currently implemented bilateral filter. You stated in the documentation that the this implementation is an approximation of a bilateral filter, like many other image processing tools do. The spatial weighting is not a gaussian curve but a box, and the value weighting is a trapezoid.
I assume this is for the sake of speed and from what I read regarding using, what is probably a similar implementation here
http://people.csail.mit.edu/sparis/siggraph07_course/slides08/06_implementation.pdf
it probably works fine.
While I don't necessarily know if that is even a viable option, I was interested to read/see this explanation of how to fix the "staircase effect" caused by the weighting in bilateral filters here http://people.csail.mit.edu/sparis/siggraph07_course/slides08/09_limitations.pdf
Not sure if this is necessarily useful however I thought it was interesting.
However could you instead use something akin to the method utilized in variableblur's binomialblur which
works by repeating a 5x5 or 3x3 kernel based on pascals triangle multiple times to blur the image. With a variance above 1 the result is very close to a true gaussian blur and much faster
Actually regarding this, I was also wondering if you could possibly port binomialblur to a 16bit version,with the exception of the box filter, as I don't particularly like the effects of box filtering, there are currently no good ways to blur things in 16bit. I would really appreciate this.
And as always keep up the good work cretindesalpes :)

[cretindesalpes]

I was wondering if you were planning on implementing the basic limitational system from removegrain that worked quite well in degrainmedian
Not at the moment. I'll stay with pure median/quantile for now, and we'll see if people use it to build more elaborate tools.

if you were to implement a limit parameter that was to limit the amount of change per pixel, would it she represented in standard 8bit form to us then converted to its 16bit equivalent or would you allow for direct 16bit numbers?
16 bit of course. It wouldn't make sense to use 8-bit values as you couldn't limit the changes below one 8-bit step. Anyway it's already possible to limit changes for any filter by using diff, lut (limiting) and add. Or you can use Dither_limit_dif16, but it limits differently.

For the staircase effect of the bilateral filter, have you encountered some cases where it shows unexpectedly? I designed it mainly for debanding purpose and I think the current implementation does correctly its job, thanks to the trapezoid approximation of the gaussian value weighting.

If you want gaussian blur, you can approximate it efficiently with a cascade of identical box filters (2–4 would do the job). This is the Pascal's triangle trick you mentioned previously, and it works with large box kernels too. Another possibility is to use Dither_resize16 with a gaussian kernel, without changing the picture size but using (negative) values for fh and fv, between -1 and 0. Increase the number of taps too. It's slow but accurate.

[TheProfileth]

cretindesalpes thanks for your responce, sorry I haven't responded sooner.
However I do have a question. Do you think you could implement a 16bit version of mt_average? I think something like this would be very nice to have, currently I am using redaverage (http://forum.doom9.org/showthread.php?t=163018) to do this, speaking of which it would be nice if you could port the parts of RAverageW and RAverageM to dithertools.

[nibus]

(edited to remove incorrect information)

[wOxxOm]

nibus, try replacing RemoveGrainSSE2.dll with SSE3 one (or plain RemoveGrain.dll), also ensure you have a new beta version from the developer's forum.

[cretindesalpes]

TheProfileth:

Why would you need an average function in Dither tools since you already have it in RedAverage?

nibus:

I don't own a 2600K but I'll try to reproduce this issue when I have time on my poor Phenom II 965. In the meantime, could you test if it crashes also with source plugins different of DGSource, as I don't have it and could not test in the same conditions as you. Also, what is the input resolution? I guess 1920x1080.

[TheProfileth]

TheProfileth:

Why would you need an average function in Dither tools since you already have it in RedAverage?

Because I have had issues with RedAverage plus the fact that I assume it would benefit from avstp and such multithreading and therefor be faster.
Plus the way that RedAverage handles it's settings is a bit off-putting.

[Keiyakusha]

nibus
If you not using new things that was added in current mvtools2, you can try different mvtools2 versions, as dither one is slowest out of three variants (including original) if you use it with mt avisynth.

BTW, back then when I reported about crashes I was using the same CPU as nibus. And they also was non-deterministic. Didn't tested if anything changed with new dither versions though as I'm no longer using it.

[nibus]

I don't own a 2600K but I'll try to reproduce this issue when I have time on my poor Phenom II 965. In the meantime, could you test if it crashes also with source plugins different of DGSource, as I don't have it and could not test in the same conditions as you. Also, what is the input resolution? I guess 1920x1080.

I apologize cretindesalpes, but it was not a problem with your modified mvtools2.dll. I just needed to adjust my usage of SetMTMode. I have it working again without crashes. Sorry for the false alarm :o

[Yellow_]

hi cretindesalpes, could you suggest what causes this grid appearance in this image extracted from AVSPmod window on this occasion but I see it in the final written 16bit images when they are sharpened as a post operation.

http://dl.dropbox.com/u/74780302/convert_yuv_rgb_RGB48Y.png

Its 16bit lsb=true output. I'm using a 16bit modded version of MCTD (Motion Compensated Temporal Denoise) to denoise and convert to 16bit RGB, then convey as rgb48_on_YV12 to Imagemagick for 16bit tif's.

I don't think this is a problem with MCTD script as I think the weave appearance happens even with a simple denoise lsb=true output and then piped to Imagemagick.

[cretindesalpes]

It looks like packed (not stacked) 16-bit data (YUV or even RGB) displayed as 8-bit YUV. How did you achieve this (script details)?

[Yellow_]

Here's the script minus the Loadplugins("").

SetMemoryMax(512)

ffmpegsource2("ffv1.mkv", threads=1)

MCTD(dfttest(sigma=0.25, lsb=true), chroma=true, settings="very low")

Dither_convert_yuv_to_rgb(matrix="601", tv_range=false, cplace="MPEG2", lsb_in=true, output="rgb48y")
#Dither_y_gamma_to_linear (tv_range_in=false, tv_range_out=false, curve="709")
Dither_convey_rgb48_on_yv12 (SelectEvery (3, 0),SelectEvery (3, 1),SelectEvery (3, 2) )

Opened in AVSPmod and using vdcrimms excellent PipeRGB to Imagemagick.

http://forum.doom9.org/showthread.php?p=1599178#post1599178

Wicked script for batch processing to 16bit image sequences avoiding high memory usage / crashing.

But this wierdness has been going on for a while even just using the CLI for piping, also just simply using 16bit mods of SMDegrain or dfttest, without MCTD, not sure whether it's just my set up, or bad usage of Dither tools or a bug.

Here's a link to a straightforward denoised frame to stacked:

http://dl.dropbox.com/u/74780302/grid.png

Using this:

baselayer = ffmpegsource2("Mov.MOV", threads=1)

dfttest(baselayer, sigma=0.25, lsb=true, Y=true, U=true, V=true)

To Illustrate grid I added ConvertToRGB(matrix="PC.601") to the bottom of the above script and saved the image frame from AVSPmod.

But I see same grid in 8bit images converted from 16bit tifs written from Imagemagick if the 8bit images are sharpened for example. Perhaps this is due to the 16bit to 8bit image conversion via numerous methods outside of Avisynth like The Gimp, ffmpeg and IM not dithering in that conversion?

[cretindesalpes]

Is the grid you are referring to the occasional patterns shown in the LSB part of the picture? If so, there is nothing wrong. They must result from blocking or other compression artefacts. Please post a few untouched frames of your source if you want me to check this issue.

[TheSkiller]

cretindesalpes, I have a question about chaining dither_convert_X_to_X(). :)

I'm in the position where I need to convert a native RGB24 clip to YV12 to process it with certain filters. After that I need RGB24 again for further processing in a NLE. It's an unfortunate situation but there's no way around it.

My question therefore is: Do you recommend chaining dithered color space conversions, or should I rather use AviSynth's internal conversions except for the very last conversion?

My concern is the dither gets "stacked" which might cause artifacts or blurring?

[cretindesalpes]

Converting with dithering (error diffusion mode without additional noise) will always be better than no dithering at all. If the filters you use only affect specific parts of the picture, you can use a mask and copy the untouched areas from the original RGB24 clip instead of the double conversion. This will at least save the effects of the chroma subsampling.

[TheSkiller]

Thanks a lot for your input. :)
Chroma subsampling is another issue right... :rolleyes:

[Yellow_]

hi, thanks here's a link to the file:

http://dl.dropbox.com/u/74780302/MVI_0561.mp4

I wonder if the 'grid' appearance is made to look worse by my set up, I recently updated my NVidia graphics card driver and I've noticed today seeing a definite pattern viewing images and even images uploaded from an ipad, so looks like display scaling issues outside of avisynth / AVSPmod making things look worse.

[cretindesalpes]

Indeed, the "grid effect" comes from blocking beeing smoothed by the denoising filter. So be reassured, there’s nothing wrong here.
o = FFVideoSource ("MVI_0561.mp4", threads=1)
f = o.dfttest(sigma=0.25, lsb=true, Y=true, U=true, V=true)
l = f.Dither_get_lsb ()
Interleave (l, o.Dither_convert_8_to_16 ().histogram_luma_16 (), f.histogram_luma_16 ())

Function histogram_luma_16 (clip src, int "amp", bool "tv_range")
{
amp = Default (amp, 16)
tv_range = Default (tv_range, true)

mi = (tv_range ? 16 * 256 : 0)
ma = (tv_range ? 235 * 256 : 65535)
rpa = (ma + 1 - mi) / amp - 1
mis = String (mi)
rpas = String (rpa)
amps = String (amp)
e = rpas+" x "+mis+" - "+rpas+" 2 * % "+rpas+" - abs - "+amps+" * "+mis+" +"

src
Dither_lut16 (e, u=1, v=1)
DitherPost (mode=6, u=-128, v=-128)
}

[cretindesalpes]

Indeed, this is a bug related to Dither_quantize in general. It's not specifically caused by the ampo value, althought it increases the probability to get this kind of artefact, as well as reducerange=true. I'm not sure I can do something unless rewriting the quantization function in a native form instead of scripting.

[mandarinka]

So the problem is from some sort of error in Dither_quantize?
I usually use that to dither to 10-bit for encoding. Does this affect other modes (6-8) besides the default?

[cretindesalpes]

Actually as soon as the dither amplitude is greater than unity (ampo + ampn > 1), there is a risk of getting this kind of inaccuracy in Dither_quantize, whatever the dither method. When reducing to 10 bits, it affects a small range of pixel values below any multiple of 16 (on a 8-bit scale). Keeping ampo + ampn ≤ 1 shouldn't create any noticeable artefact.

[Jenyok]

How to use Dither functions in the script below (super resolution upscale) ?
Thanks.
.

LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\MVTOOLS-V2_5_11_3\mvtools2.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\NNEDI3\nnedi3.dll")
LoadPlugin("C:\PROGRAM FILES\AVISYNTH 2.5\PLUGINS\NNEDI_20070920\nnedi.dll")



#
# http://forum.doom9.org/showthread.php?t=142704
# Author: Efenstor
#
function MSR(clip clp)
{
# Constants
# Change those constants to tune for the best result of interpolation

# MSuper
#
hpad = 8 # 4
vpad = 8
pel = 4 # 2
rfilter = 4 # 4
sharp = 2 # 2
isse = true

# MAnalyse
#
blks = hpad
blksV = vpad
overlap = blks / 2
overlapV = blksV / 2
search = 5
searchparam = 3
dct = 5

# MCompensate
#
tSAD = 10000
tCD1 = 400
tCD2 = 130

# Overlay
#
opacity = 0.5



brc = BilinearResize(clp, Width(clp) * 2, Height(clp) * 2)
# function MSuper(clip, int "hpad", int "vpad", int "pel", int "levels", bool "chroma", \
# int "sharp", int "rfilter", clip "pelclip", bool "isse", bool "planar")
#
super = MSuper(brc, \
hpad = hpad, \
vpad = vpad, \
pel = pel, \
levels = 0, \
chroma = true, \
sharp = sharp, \
rfilter = rfilter, \
isse = isse, \
planar = false)

# function MAnalyse(clip super, int "blksize", int "blksizeV", int "level", int "search", int "searchparam", \
# int "pelsearch", bool "isb", int "lambda", bool "chroma", int "delta", bool "truemotion", \
# int "lsad", int "plevel", bool "global", int "pnew", int "pzero", int "pglobal", int "overlap", \
# int "overlapV", string "outfile", int "dct", int "divide", int "sadx264", int "badSAD", \
# int "badrange", bool "isse", int "full", bool "meander", bool "temporal")
#
vec1 = MAnalyse(super, \
isb = true, \
delta = 2, \
blksize = blks, \
blksizeV = blksV, \
levels = 0, \
search = search, \
searchparam = searchparam, \
pelsearch = pel, \
lambda = 0, \
chroma = true, \
truemotion = true, \
lsad = 1200, \
plevel = 0, \
global = true, \
pnew = 50, \
pzero = 50, \
pglobal = 0, \
overlap = overlap, \
overlapV = overlapV, \
dct = dct, \
divide = 0, \
sadx264 = 0, \
badSAD = 10000, \
badrange = 24, \
isse = isse, \
meander = true, \
temporal = false, \
trymany = false)


# function MAnalyse(clip super, int "blksize", int "blksizeV", int "level", int "search", int "searchparam", \
# int "pelsearch", bool "isb", int "lambda", bool "chroma", int "delta", bool "truemotion", \
# int "lsad", int "plevel", bool "global", int "pnew", int "pzero", int "pglobal", int "overlap", \
# int "overlapV", string "outfile", int "dct", int "divide", int "sadx264", int "badSAD", \
# int "badrange", bool "isse", int "full", bool "meander", bool "temporal")
#
vec2 = MAnalyse(super, \
isb = true, \
delta = 1, \
blksize = blks, \
blksizeV = blksV, \
levels = 0, \
search = search, \
searchparam = searchparam, \
pelsearch = pel, \
lambda = 0, \
chroma = true, \
truemotion = true, \
lsad = 1200, \
plevel = 0, \
global = true, \
pnew = 50, \
pzero = 50, \
pglobal = 0, \
overlap = overlap, \
overlapV = overlapV, \
dct = dct, \
divide = 0, \
sadx264 = 0, \
badSAD = 10000, \
badrange = 24, \
isse = isse, \
meander = true, \
temporal = false, \
trymany = false)

# function MAnalyse(clip super, int "blksize", int "blksizeV", int "level", int "search", int "searchparam", \
# int "pelsearch", bool "isb", int "lambda", bool "chroma", int "delta", bool "truemotion", \
# int "lsad", int "plevel", bool "global", int "pnew", int "pzero", int "pglobal", int "overlap", \
# int "overlapV", string "outfile", int "dct", int "divide", int "sadx264", int "badSAD", \
# int "badrange", bool "isse", int "full", bool "meander", bool "temporal")
#
vec3 = MAnalyse(super, \
isb = false, \
delta = 1, \
blksize = blks, \
blksizeV = blksV, \
levels = 0, \
search = search, \
searchparam = searchparam, \
pelsearch = pel, \
lambda = 0, \
chroma = true, \
truemotion = true, \
lsad = 1200, \
plevel = 0, \
global = true, \
pnew = 50, \
pzero = 50, \
pglobal = 0, \
overlap = overlap, \
overlapV = overlapV, \
dct = dct, \
divide = 0, \
sadx264 = 0, \
badSAD = 10000, \
badrange = 24, \
isse = isse, \
meander = true, \
temporal = false, \
trymany = false)

# function MAnalyse(clip super, int "blksize", int "blksizeV", int "level", int "search", int "searchparam", \
# int "pelsearch", bool "isb", int "lambda", bool "chroma", int "delta", bool "truemotion", \
# int "lsad", int "plevel", bool "global", int "pnew", int "pzero", int "pglobal", int "overlap", \
# int "overlapV", string "outfile", int "dct", int "divide", int "sadx264", int "badSAD", \
# int "badrange", bool "isse", int "full", bool "meander", bool "temporal")
#
vec4 = MAnalyse(super, \
isb = false, \
delta = 2, \
blksize = blks, \
blksizeV = blksV, \
levels = 0, \
search = search, \
searchparam = searchparam, \
pelsearch = pel, \
lambda = 0, \
chroma = true, \
truemotion = true, \
lsad = 1200, \
plevel = 0, \
global = true, \
pnew = 50, \
pzero = 50, \
pglobal = 0, \
overlap = overlap, \
overlapV = overlapV, \
dct = dct, \
divide = 0, \
sadx264 = 0, \
badSAD = 10000, \
badrange = 24, \
isse = isse, \
meander = true, \
temporal = false, \
trymany = false)

# clp = nnedi(clp, field=0, dh=true)
# clp = TurnLeft(clp)
# clp = nnedi(clp, field=0, dh=true)
# clp = TurnRight(clp)
# clp = NNEDI3_rpow2(clp, rfactor=2, cshift="spline64resize", qual=2, nsize=2, nns=2)
clp = NNEDI3_rpow2(clp, rfactor=2, cshift="lanczos4resize", qual=2, nsize=2, nns=2)

clp = Sharpen(clp, 1)

# function MSuper(clip, int "hpad", int "vpad", int "pel", int "levels", bool "chroma", \
# int "sharp", int "rfilter", clip "pelclip", bool "isse", bool "planar")
#
super = MSuper(clp, \
hpad = hpad, \
vpad = vpad, \
pel = pel, \
levels = 0, \
chroma = true, \
sharp = sharp, \
rfilter = rfilter, \
isse = isse, \
planar = false)

# function MCompensate(clip source, clip super, clip vectors, bool "scbehavior", float "recursion", \
# int "thSAD", bool "fields", int "thSCD1", int "thSCD2", bool "isse", bool "planar")
#
c1 = MCompensate(clp, \
super, \
vec1, \
scbehavior = true, \
recursion = 0, \
thSAD = tSAD , \
fields = false, \
thSCD1 = tCD1, \
thSCD2 = tCD2, \
isse = isse, \
planar = false)

# function MCompensate(clip source, clip super, clip vectors, bool "scbehavior", float "recursion", \
# int "thSAD", bool "fields", int "thSCD1", int "thSCD2", bool "isse", bool "planar")
#
c2 = MCompensate(clp, \
super, \
vec2, \
scbehavior = true, \
recursion = 0, \
thSAD = tSAD, \
fields = false, \
thSCD1 = tCD1, \
thSCD2 = tCD2, \
isse = isse, \
planar = false)

# function MCompensate(clip source, clip super, clip vectors, bool "scbehavior", float "recursion", \
# int "thSAD", bool "fields", int "thSCD1", int "thSCD2", bool "isse", bool "planar")
#
c3 = MCompensate(clp, \
super, \
vec3, \
scbehavior = true, \
recursion = 0, \
thSAD = tSAD, \
fields = false, \
thSCD1 = tCD1, \
thSCD2 = tCD2, \
isse = isse, \
planar = false)

# function MCompensate(clip source, clip super, clip vectors, bool "scbehavior", float "recursion", \
# int "thSAD", bool "fields", int "thSCD1", int "thSCD2", bool "isse", bool "planar")
#
c4 = MCompensate(clp, \
super, \
vec4, \
scbehavior = true, \
recursion = 0, \
thSAD = tSAD, \
fields = false, \
thSCD1 = tCD1, \
thSCD2 = tCD2, \
isse = isse, \
planar = false)

t1 = Overlay(clp, c1, opacity=opacity)
t2 = Overlay(clp, c2, opacity=opacity)
t3 = Overlay(clp, c3, opacity=opacity)
t4 = Overlay(clp, c4, opacity=opacity)
f1 = Overlay(t1, t2, opacity=opacity)
f2 = Overlay(t3, t4, opacity=opacity)
last = Overlay(f1, f2, opacity=opacity)

return (last)
}

[cretindesalpes]

What "Dither functions" do you want to use? The question is strange: "How to use a screwdriver in this pie recipe?" What exactly do you expect to achieve or improve with the Dither tools?

[Jenyok]

cretindesalpes
.
I need very good quality in super resolution upscale (up to 2 times, up to 4 times) with enhanced small and very small details in video.

[TheProfileth]

cretindesalpes
.
I need very good quality in super resolution upscale (up to 2 times, up to 4 times) with enhanced small and very small details in video.
And I need him to finish that filter called AddDetail() ;)
By just saying what you "need" or want it doesn't really give much direction on what to change, that is, if there is anything at all to change. At most your function might be more accurate or have less banding but dithertools isn't magic and in general super resolution is more of a misnomer than an actual thing. It assumes that magic things can occur when you interpolate frame data and that all frame data is static because if it isn't then your clip ends up either full of artifacts or no more detailed than it started.

[Elvellon]

Thank you for the dither package! I use it constantly.
I have a problem.
If the 16 bit clip is YV12 or YV24 (then Dither_convey_YUV4xxP16_on_YVxx()) then everything encodes OK. Now I try to encode 4:2:2 (YV16 or YUY2) and everything is garbled.
Can it be fixed? Is there a workaround with some lossless adding/removing of chroma with YV24 in AVS and YV16 in x264?

[sneaker_ger]

Try SwapUV() between YV16 source and dither. Otherwise post the complete script and piping/encoding parameters.

[Elvellon]

FFVideoSource("random mkv, 8 bit 4:2:0")
BilinearResize(640,360) http://i53.fastpic.ru/big/2012/1226/83/da347a2923101ebe735a7d6c3efdfa83.jpeg
ConvertToYV16() # or YUY2
Dither_convert_8_to_16() # looks ok/expected up to this point
# SwapUV() http://i52.fastpic.ru/big/2012/1226/e0/32b2502aa3f7a23d8301bc0c712d95e0.jpeg
Dither_convey_YUV4xxP16_on_YVxx() # almost the same with and without swap http://i53.fastpic.ru/big/2012/1226/ac/418b64ccec0fcf63896dcbaaa83814ac.jpeg

Then
avs4x264mod -L x264_10bit_x86 "in.avs" --demuxer raw --input-depth 16 --input-res 640x360 --input-csp i422 --output-csp i422 --seek 100 --frames 1 --qp 0 --output "out.mkv"
Load via hacked 10 bit ffms: http://i51.fastpic.ru/big/2012/1226/e8/bcaae42ea362acdf922e65221ef940e8.jpeg

avs2yuv_x86 -raw -csp i422 "in.avs" -o - | x264_10bit_x86 --demuxer raw --input-depth 16 --input-res 640x360 --input-csp i422 --output-csp i422 --output "out.mkv" --seek 100 --frames 1 --qp 0 -
same result

If I change YV16 to YV24 in the script and i422 to i444 in the encoding parameters, everything is fine: http://i53.fastpic.ru/big/2012/1227/50/d9693137b5c31ca5d4c981c3dfea1450.jpeg

The "random mkv" was for reproducibility, the file I've been struggling with is ProRes. The result is the same, green blotches etc. Of course if I input it directly to x264 it encodes 4:2:2 fine. So somehow the colorspaces in the AVS > pipe > x264 process don't connect somehow.

[sneaker_ger]

I'm also having trouble making it work. Let's wait for cretindesalpes.

Or would fmtconv or flash3kyuu be an alternative for you? I don't know exactly why you need to use the dither tools.

[Elvellon]

Thanks, let's wait.
Actually I mostly do some basic denoising/resizing (+ QTGMC outside of dither). So I don't want to get into VapourSynth just yet.
Besides others have noticed some banding problems in the video that I can't see/fix so I'll just report the fix/workaround to them.

[cretindesalpes]

Sorry, my bad. Dither_convey_yuv4xxp16_on_yvxx() implementation was a bit naive and doesn’t work when the chroma subsampling is not the same on the X and Y axis. Here is a fixed version, before I release an update:

Function Dither_convey_yuv4xxp16_on_yvxx (clip src, bool "bigendian")
{
bigendian = Default (bigendian, false)

src
msb = Dither_get_msb ()
lsb = Dither_get_lsb ()

(bigendian) ? Interleave (msb, lsb) : Interleave (lsb, msb)

d = (Dither_get_chroma_subspl_h () != Dither_get_chroma_subspl_v ())
m = (d) ? ConvertToY8 () : last
u = (d) ? UToY8 () : last
v = (d) ? VToY8 () : last

m = m.TurnRight ().AssumeFieldBased ().AssumeTFF ().Weave ().TurnLeft ()
u = (d) ? u.TurnRight ().AssumeFieldBased ().AssumeTFF ().Weave ().TurnLeft () : u
v = (d) ? v.TurnRight ().AssumeFieldBased ().AssumeTFF ().Weave ().TurnLeft () : v

(d) ? YToUV (u, v, m) : m
}