flash3kyuu_deband 1.5.1 / 2.0pre2 with native VapourSynth support [2012-12-03]

[SAPikachu]

Hi all,

I've ported a deband filter for AviUtl (http://www.geocities.jp/flash3kyuu/auf/banding17.zip) to AviSynth. Its algorithm is simple but it is quite effective for some anime sources.

2.0pre2 with native VapourSynth support: http://nmm.me/tr

Some notes:

* This is still an experimental version, it has not been fully 
  tested, so please use it with caution.
* sample_mode=0 and dither_algo=0 are removed in this 
  version since nobody is using them.
* Document has not been updated yet, please refer to test.vpy 
  for sample usage. All parameters are in lower case.
* Because input_mode/input_depth/output_mode can be 
  implied from clip properties, they are not settable in VS 
  interface. output_depth is still available though.

2.0pre2:
* Fixes compatibility with XP
* Exports public API, can be used as a standalone library

Change log:
1.5.1 (2012-04-07)
* Supports setting StdDev (sigma) for the Gaussian random number generator

1.5.0 (2012-03-12)
* (There isn't any new feature in this version, only some parameters are modified to reduce user confusion)
* ditherY/ditherC are renamed to grainY/grainC
* dynamic_dither_noise is renamed to dynamic_grain
* precision_mode is renamed to dither_algo, mode 4 and 5 are removed
* random_algo_dither is renamed to random_algo_grain
* enable_fast_skip_plane is removed, this optimization will be enabled mplicitly whenever possible (Filter result won't be changed by this optimization)

1.4.2 (2011-11-10)
* Fixed crash on some non-mod16 videos

1.4.1 (2011-11-05)
* Fixed broken YUY2 support (still slow)
* Improved default value handling of bitdepth-related parameters
* precision_mode 4 / 5 are now deprecated and may be removed in future versions, you can use output_mode 1 / 2 to achieve the same result

1.4.0 (2011-10-30)
* 9 ~ 16 bit-depth input/output
** Related parameters: input_mode/input_depth/output_mode/output_depth
* New random number generator, reference position and dither noise can be generated in uniform or gaussian distribution
** Related parameters: random_algo_ref / random_algo_dither
* diff_seed is replaced with dynamic_dither_noise, when enabled, noise pattern will be different for each frame
* Another new parameter: enable_fast_skip_plane
* Short filter alias: f3kdb
* Now the ICC-compiled DLL should be runnable on pre-SSE2 systems (untested)
* Several bug fixes

1.3.0 (2011-09-07)
* Added x64 version
* Added a downsample filter: f3kdb_dither
* Internal precision is increased to 16bit
* New parameter: keep_tv_range, please see readme.txt for details
* Default sample_mode is changed to 2 as it is better in most cases
* Fixed: Floyd-Steinberg dithering may produce incorrect result for full-range videos
* Fixed: Broken YUY2 debanding
* Minor optimizations

1.2.0 (2011-08-01)
* Added support for YUY2 (not optimized yet)
* Added support for all planar YUV format in AviSynth 2.6
* The filter is now compatible with both AviSynth 2.5 and 2.6
* 16bit output (precision_mode = 4/5)
* Note: The internal processing precision is still 14bit, this will be
improved in future versions

1.1.0 (2011-06-18)
* Fixed a bug that high threshold values would produce incorrect result in high precision modes.
* Threshold values was scaled based on other parameter in previous versions, it is unscaled now to increase flexibility. Using same value has weaker effect in new version. Effect of default parameter set is also changed.
* SSE optimization for high precision mode.
* Rejects some invalid parameter combination instead of silently skips them

1.0.2
* High precision mode
(currently non-optimized, SSE routine will be added later)
* Frame edges are properly processed now
* Fix crash in some cases (unaligned frames are handled correctly)
* Other bug fixes

Binary: x86 / x64 / MSVC
(The MSVC build is slower, use that only if the x86 build has problems)

Documentation: https://raw.github.com/SAPikachu/fla...yuu_deband.txt

Source code: https://github.com/SAPikachu/flash3k...and/tree/1.5.1

Samples: #1 #2 #3 #4 (1st in post #4) #5 (2nd in post #4) All

(Note: I used different image source for samples starting from 1.0.2, the original sources are not very banded indeed.)

Script for generating the samples (changed again on 6-19):

ImageReader("wallorig.png", end=0)
ConvertToYV12
pic=last

global i=0
    
LoadPlugin("flash3kyuu_deband.dll")
StackVertical(last.croppic.subtitle("original"), last.croppic.luma_histogram.subtitle("original"))


addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3)""") #1
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3,y=64,cb=64,cr=64,grainy=32,grainc=32)""") #2
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3,y=64,cb=64,cr=64,grainy=16,grainc=16)""") #3
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3,y=64,cb=64,cr=64,grainy=0,grainc=0)""") #4
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=2,y=64,cb=64,cr=64,grainy=32,grainc=32)""") #5
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=1,y=64,cb=64,cr=64,grainy=32,grainc=32)""") #6
addtest(last,pic,"""gradfun3(smode=1,mask=0)""") #7
addtest(last,pic,"""gradfun3(smode=2,mask=0)""") #8



ConvertToRGB
ImageWriter("out",type="png")

function luma_histogram(clip c)
{
    return c.histogram("luma").ColorYUV(levels="PC->TV")
}

function croppic(clip c)
{
    return c.crop(100,100,400,400)
}

function addtest(clip c, clip pic, string expr)
{
    pic
    eval(expr)
    croppic
    
    
    global i=i+1
    expr="#" + string(i) + " " + expr
    
    StackVertical(last.subtitle(expr), luma_histogram.subtitle(expr))
    return StackHorizontal(c,last)
}

Comments are appreciated.

[CruNcher]

Comparison with http://forum.doom9.org/showthread.ph...59#post1386559 seems missing

[mastrboy]

could you provide some screenshots comparing this to GradFun2DBmod and using histogram(mode="luma") ?

[cretindesalpes]

Three comparisons done with flash3kyuu_deband v1.0.0, without High precision mode:


orig | gf2m | f3kds1 | f3kds2 | gf3sm0 | gf3sm1 | gf3sm2 (normal)
orig | gf2m | f3kds1 | f3kds2 | gf3sm0 | gf3sm1 | gf3sm2 (histogram)


orig | gf2m | f3kds1 | f3kds2 | gf3sm0 | gf3sm1 | gf3sm2 (normal)
orig | gf2m | f3kds1 | f3kds2 | gf3sm0 | gf3sm1 | gf3sm2 (histogram)


orig | gf2m | f3kds1 | f3kds2 | gf3sm0 | gf3sm1 | gf3sm2 (normal)
orig | gf2m | f3kds1 | f3kds2 | gf3sm0 | gf3sm1 | gf3sm2 (histogram)

orig = original
gf2m = Gradfun2dbMod (str=0)
f3kds1 = flash3kyuu_deband (sample_mode=1)
f3kds1 = flash3kyuu_deband (sample_mode=2)
gf3sm0 = GradFun3 (smode=0)
gf3sm1 = GradFun3 (smode=1)
gf3sm2 = GradFun3 (smode=2)

Conversion to RGB done with yv12torgb24hq ()

Edit 1: Histogram(mode="luma") looks broken and outputs PC-range lumas whatever the input range. I fixed the screenshots
Edit 2: One more comparison
Edit 3: imageshack.us resized silently most of the pictures so the comparison is now irrelevant.

[SAPikachu]

Added some more samples. Also thanks cretindesalpes for testing.

[mandarinka]

I see zero banding in the image in post #1 and only very very little banding in post #4 (not worht filtering imho). Monitor is Samsung SyncMaster 226bw (not exactly expensive TN, 4 years old), without any special calibration...

Anyone with good/calibrated display can comment? It's tricky if one can without knowing run debanding just because of deficiences of particular lcds...

[SAPikachu]

The original image have banding in lower sky. I can't find an image with more serious banding at this time... The third image in #4 should be more visible.

[Hiritsuki]

May I ask which is this filter used? GPU or CPU?

[SAPikachu]

It runs on CPU. It should be pretty fast with current CPU though.

[Hiritsuki]

Thank you. It's seems amazing.
I'll try it.

[Hiritsuki]

BTW, this filter support with MT?

[SAPikachu]

I didn't test that, but it should support MT mode 2 with appropriate overlap.

[Chikuzen]

Hi.
@SAPikachu
some Japanese on 2ch complain that "AviUtl's original can adjust params by 32 stages(0 to 31), but this can be adjusted only by three stages(value 15 of original corresponds to value 1 of this) except range".

would you mind improving it?

[SAPikachu]

Yes, I plan to add 12-bit (or even 16-bit) processing when I have time. Parameter stages will be the same as the original plugin then.

[Chikuzen]

Originally Posted by SAPikachu

Yes, I plan to add 12-bit (or even 16-bit) processing when I have time. Parameter stages will be the same as the original plugin then.

i see, thx

[Hiritsuki]

My friend say it's cannot use MT with over 4threads, I have 980X 6c12t to run and it's slow.

[SAPikachu]

Originally Posted by Hiritsuki

My friend say it's cannot use MT with over 4threads, I have 980X 6c12t to run and it's slow.

Is there any error message? Could you try setting mt=false when using AVS MT? For some reasons I can't test AVS MT at the moment...

[Hiritsuki]

No error message.
I means this filter only use 4threads for multithreads to run.

[SAPikachu]

Actually the filter only uses 2 threads internally. It is more complicated to use more threads, I won't change it until 12-bit processing is completed. You may want to try MT-related filter to speed up.

[SAPikachu]

Released 1.0.2.