flash3kyuu_deband 1.5.1 / 2.0pre2 with native VapourSynth support [2012-12-03]

[SAPikachu]

Hi all,

I've ported a deband filter for AviUtl (http://www.geocities.jp/flash3kyuu/auf/banding17.zip) to AviSynth. Its algorithm is simple but it is quite effective for some anime sources.

2.0pre2 with native VapourSynth support: http://nmm.me/tr

Some notes:

* This is still an experimental version, it has not been fully
tested, so please use it with caution.
* sample_mode=0 and dither_algo=0 are removed in this
version since nobody is using them.
* Document has not been updated yet, please refer to test.vpy
for sample usage. All parameters are in lower case.
* Because input_mode/input_depth/output_mode can be
implied from clip properties, they are not settable in VS
interface. output_depth is still available though.

2.0pre2:
* Fixes compatibility with XP
* Exports public API, can be used as a standalone library



Change log:
1.5.1 (2012-04-07)
* Supports setting StdDev (sigma) for the Gaussian random number generator

1.5.0 (2012-03-12)
* (There isn't any new feature in this version, only some parameters are modified to reduce user confusion)
* ditherY/ditherC are renamed to grainY/grainC
* dynamic_dither_noise is renamed to dynamic_grain
* precision_mode is renamed to dither_algo, mode 4 and 5 are removed
* random_algo_dither is renamed to random_algo_grain
* enable_fast_skip_plane is removed, this optimization will be enabled mplicitly whenever possible (Filter result won't be changed by this optimization)

1.4.2 (2011-11-10)
* Fixed crash on some non-mod16 videos

1.4.1 (2011-11-05)
* Fixed broken YUY2 support (still slow)
* Improved default value handling of bitdepth-related parameters
* precision_mode 4 / 5 are now deprecated and may be removed in future versions, you can use output_mode 1 / 2 to achieve the same result

1.4.0 (2011-10-30)
* 9 ~ 16 bit-depth input/output
** Related parameters: input_mode/input_depth/output_mode/output_depth
* New random number generator, reference position and dither noise can be generated in uniform or gaussian distribution
** Related parameters: random_algo_ref / random_algo_dither
* diff_seed is replaced with dynamic_dither_noise, when enabled, noise pattern will be different for each frame
* Another new parameter: enable_fast_skip_plane
* Short filter alias: f3kdb
* Now the ICC-compiled DLL should be runnable on pre-SSE2 systems (untested)
* Several bug fixes

1.3.0 (2011-09-07)
* Added x64 version
* Added a downsample filter: f3kdb_dither
* Internal precision is increased to 16bit
* New parameter: keep_tv_range, please see readme.txt for details
* Default sample_mode is changed to 2 as it is better in most cases
* Fixed: Floyd-Steinberg dithering may produce incorrect result for full-range videos
* Fixed: Broken YUY2 debanding
* Minor optimizations

1.2.0 (2011-08-01)
* Added support for YUY2 (not optimized yet)
* Added support for all planar YUV format in AviSynth 2.6
* The filter is now compatible with both AviSynth 2.5 and 2.6
* 16bit output (precision_mode = 4/5)
* Note: The internal processing precision is still 14bit, this will be
improved in future versions

1.1.0 (2011-06-18)
* Fixed a bug that high threshold values would produce incorrect result in high precision modes.
* Threshold values was scaled based on other parameter in previous versions, it is unscaled now to increase flexibility. Using same value has weaker effect in new version. Effect of default parameter set is also changed.
* SSE optimization for high precision mode.
* Rejects some invalid parameter combination instead of silently skips them

1.0.2
* High precision mode
(currently non-optimized, SSE routine will be added later)
* Frame edges are properly processed now
* Fix crash in some cases (unaligned frames are handled correctly)
* Other bug fixes

Binary: x86 (http://nmm.me/ff) / x64 (http://nmm.me/fh) / MSVC (http://nmm.me/fg)
(The MSVC build is slower, use that only if the x86 build has problems)

Documentation: https://raw.github.com/SAPikachu/flash3kyuu_deband/1.5.1/flash3kyuu_deband.txt

Source code: https://github.com/SAPikachu/flash3kyuu_deband/tree/1.5.1

Samples: #1 (https://picasaweb.google.com/lh/photo/XtvTmJ70F7sZTV6JBP_3LA?feat=directlink) #2 (https://picasaweb.google.com/lh/photo/jnmFoK_2WZgpRS_sFN1Dow?feat=directlink) #3 (https://picasaweb.google.com/lh/photo/vLCEJD0bsMY1WmFxrDGlfg?feat=directlink) #4 (1st in post #4) (https://picasaweb.google.com/lh/photo/FC9fwXrs-kMuHvSI_weieQ?feat=directlink) #5 (2nd in post #4) (https://picasaweb.google.com/lh/photo/1QtAx-sz59WwSLM2w-Oj0A?feat=directlink) All (https://picasaweb.google.com/SAPikachu/Flash3kyuu_deband110SampleRev2?feat=directlink)

(Note: I used different image source for samples starting from 1.0.2, the original sources are not very banded indeed.)

Script for generating the samples (changed again on 6-19):

ImageReader("wallorig.png", end=0)
ConvertToYV12
pic=last

global i=0

LoadPlugin("flash3kyuu_deband.dll")
StackVertical(last.croppic.subtitle("original"), last.croppic.luma_histogram.subtitle("original"))


addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3)""") #1
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3,y=64,cb=64,cr=64,grainy=32,grainc=32)""") #2
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3,y=64,cb=64,cr=64,grainy=16,grainc=16)""") #3
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=3,y=64,cb=64,cr=64,grainy=0,grainc=0)""") #4
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=2,y=64,cb=64,cr=64,grainy=32,grainc=32)""") #5
addtest(last,pic,"""flash3kyuu_deband(sample_mode=2,dither_algo=1,y=64,cb=64,cr=64,grainy=32,grainc=32)""") #6
addtest(last,pic,"""gradfun3(smode=1,mask=0)""") #7
addtest(last,pic,"""gradfun3(smode=2,mask=0)""") #8



ConvertToRGB
ImageWriter("out",type="png")

function luma_histogram(clip c)
{
return c.histogram("luma").ColorYUV(levels="PC->TV")
}

function croppic(clip c)
{
return c.crop(100,100,400,400)
}

function addtest(clip c, clip pic, string expr)
{
pic
eval(expr)
croppic


global i=i+1
expr="#" + string(i) + " " + expr

StackVertical(last.subtitle(expr), luma_histogram.subtitle(expr))
return StackHorizontal(c,last)
}


Comments are appreciated. :)

[CruNcher]

Comparison with http://forum.doom9.org/showthread.php?p=1386559#post1386559 seems missing

[mastrboy]

could you provide some screenshots comparing this to GradFun2DBmod and using histogram(mode="luma") ?

[cretindesalpes]

Three comparisons done with flash3kyuu_deband v1.0.0, without High precision mode:

http://img143.imageshack.us/img143/6048/bikesorig.th.png http://img52.imageshack.us/img52/6237/hbikesorig.th.png
orig (http://img143.imageshack.us/img143/6048/bikesorig.png) | gf2m (http://img607.imageshack.us/img607/7351/bikesgf2m.png) | f3kds1 (http://img269.imageshack.us/img269/121/bikesf3kds1.png) | f3kds2 (http://img198.imageshack.us/img198/6526/bikesf3kds2.png) | gf3sm0 (http://img638.imageshack.us/img638/73/bikesgf3sm0.png) | gf3sm1 (http://img857.imageshack.us/img857/9995/bikesgf3sm1.png) | gf3sm2 (http://img847.imageshack.us/img847/6822/bikesgf3sm2.png) (normal)
orig (http://img52.imageshack.us/img52/6237/hbikesorig.png) | gf2m (http://img35.imageshack.us/img35/9393/hbikesgf2m.png) | f3kds1 (http://img853.imageshack.us/img853/7394/hbikesf3kds1.png) | f3kds2 (http://img151.imageshack.us/img151/1512/hbikesf3kds2.png) | gf3sm0 (http://img829.imageshack.us/img829/9907/hbikesgf3sm0.png) | gf3sm1 (http://img101.imageshack.us/img101/711/hbikesgf3sm1.png) | gf3sm2 (http://img15.imageshack.us/img15/6818/hbikesgf3sm2.png) (histogram)

http://img835.imageshack.us/img835/315/redorig.th.png http://img198.imageshack.us/img198/3162/hredorig.th.png
orig (http://img835.imageshack.us/img835/315/redorig.png) | gf2m (http://img194.imageshack.us/img194/4851/redgf2m.png) | f3kds1 (http://img24.imageshack.us/img24/6929/redf3kds1.png) | f3kds2 (http://img805.imageshack.us/img805/3986/redf3kds2.png) | gf3sm0 (http://img97.imageshack.us/img97/2558/redgf3sm0.png) | gf3sm1 (http://img228.imageshack.us/img228/7789/redgf3sm1.png) | gf3sm2 (http://img263.imageshack.us/img263/1290/redgf3sm2.png) (normal)
orig (http://img198.imageshack.us/img198/3162/hredorig.png) | gf2m (http://img84.imageshack.us/img84/3571/hredgf2m.png) | f3kds1 (http://img225.imageshack.us/img225/690/hredf3kds1.png) | f3kds2 (http://img339.imageshack.us/img339/118/hredf3kds2.png) | gf3sm0 (http://img339.imageshack.us/img339/7392/hredgf3sm0.png) | gf3sm1 (http://img839.imageshack.us/img839/7782/hredgf3sm1.png) | gf3sm2 (http://img9.imageshack.us/img9/6792/hredgf3sm2.png) (histogram)

http://img828.imageshack.us/img828/6518/wallorig.th.png http://img696.imageshack.us/img696/7694/hwallorig.th.png
orig (http://img828.imageshack.us/img828/6518/wallorig.png) | gf2m (http://img163.imageshack.us/img163/475/wallgf2m.png) | f3kds1 (http://img69.imageshack.us/img69/646/wallf3kds1.png) | f3kds2 (http://img838.imageshack.us/img838/49/wallf3kds2.png) | gf3sm0 (http://img59.imageshack.us/img59/7411/wallgf3sm0.png) | gf3sm1 (http://img845.imageshack.us/img845/9733/wallgf3sm1.png) | gf3sm2 (http://img21.imageshack.us/img21/7774/wallgf3sm2.png) (normal)
orig (http://img696.imageshack.us/img696/7694/hwallorig.png) | gf2m (http://img703.imageshack.us/img703/599/hwallgf2m.png) | f3kds1 (http://img688.imageshack.us/img688/977/hwallf3kds1.png) | f3kds2 (http://img806.imageshack.us/img806/6481/hwallf3kds2.png) | gf3sm0 (http://img197.imageshack.us/img197/4183/hwallgf3sm0.png) | gf3sm1 (http://img801.imageshack.us/img801/2826/hwallgf3sm1.png) | gf3sm2 (http://img6.imageshack.us/img6/3691/hwallgf3sm2.png) (histogram)

orig = original
gf2m = Gradfun2dbMod (str=0)
f3kds1 = flash3kyuu_deband (sample_mode=1)
f3kds1 = flash3kyuu_deband (sample_mode=2)
gf3sm0 = GradFun3 (smode=0)
gf3sm1 = GradFun3 (smode=1)
gf3sm2 = GradFun3 (smode=2)

Conversion to RGB done with yv12torgb24hq ()

Edit 1: Histogram(mode="luma") looks broken and outputs PC-range lumas whatever the input range. I fixed the screenshots
Edit 2: One more comparison
Edit 3: imageshack.us resized silently most of the pictures so the comparison is now irrelevant.

[SAPikachu]

Added some more samples. Also thanks cretindesalpes for testing.

[mandarinka]

I see zero banding in the image in post #1 and only very very little banding in post #4 (not worht filtering imho). Monitor is Samsung SyncMaster 226bw (not exactly expensive TN, 4 years old), without any special calibration...

Anyone with good/calibrated display can comment? It's tricky if one can without knowing run debanding just because of deficiences of particular lcds...

[SAPikachu]

The original image have banding in lower sky. I can't find an image with more serious banding at this time... The third image in #4 should be more visible.

[Hiritsuki]

May I ask which is this filter used? GPU or CPU?

[SAPikachu]

It runs on CPU. It should be pretty fast with current CPU though.

[Hiritsuki]

Thank you. It's seems amazing.
I'll try it.

[Hiritsuki]

BTW, this filter support with MT?

[SAPikachu]

I didn't test that, but it should support MT mode 2 with appropriate overlap.

[Chikuzen]

Hi.
@SAPikachu
some Japanese on 2ch complain that "AviUtl's original can adjust params by 32 stages(0 to 31), but this can be adjusted only by three stages(value 15 of original corresponds to value 1 of this) except range".

would you mind improving it?

[SAPikachu]

Yes, I plan to add 12-bit (or even 16-bit) processing when I have time. Parameter stages will be the same as the original plugin then.

[Chikuzen]

Yes, I plan to add 12-bit (or even 16-bit) processing when I have time. Parameter stages will be the same as the original plugin then.

i see, thx:)

[Hiritsuki]

My friend say it's cannot use MT with over 4threads, I have 980X 6c12t to run and it's slow.

[SAPikachu]

My friend say it's cannot use MT with over 4threads, I have 980X 6c12t to run and it's slow.

Is there any error message? Could you try setting mt=false when using AVS MT? For some reasons I can't test AVS MT at the moment...

[Hiritsuki]

No error message.
I means this filter only use 4threads for multithreads to run.

[SAPikachu]

Actually the filter only uses 2 threads internally. It is more complicated to use more threads, I won't change it until 12-bit processing is completed. You may want to try MT-related filter to speed up.

[SAPikachu]

Released 1.0.2.

[kolak]

I see zero banding in the image in post #1 and only very very little banding in post #4 (not worht filtering imho). Monitor is Samsung SyncMaster 226bw (not exactly expensive TN, 4 years old), without any special calibration...

Anyone with good/calibrated display can comment? It's tricky if one can without knowing run debanding just because of deficiences of particular lcds...

Yes- I have Eizo monitor and see a lot of banding in image 1.
Debanding does a quite good job.


Andrew

[Chikuzen]

Yes- I have Eizo monitor and see a lot of banding in image 1.
Debanding does a quite good job.

It is not mysterious that you praise this plugin.

I heard that the original author(AviUtl version) is a professional who is working in an authoring studio as you.

[sneaker_ger]

Totally missed this one, sample_mode=2 looks amazing.
Anything to watch out for concerning pre-x264-encode debanding?

[Groucho2004]

Very impressive. And fast!

[SAPikachu]

Yes- I have Eizo monitor and see a lot of banding in image 1.
Debanding does a quite good job.


Andrew

I changed test image source for 1.0.2 version, the original image has less banding indeed. Forgot to mention this in OP, sorry.

[SAPikachu]

By the way, I recently found a serious (stupid) bug in 1.0.2 that makes high threshold values useless. I will release a new version next week, along with SSE optimization for high precision mode.

[SAPikachu]

Totally missed this one, sample_mode=2 looks amazing.
Anything to watch out for concerning pre-x264-encode debanding?

I haven't done research for this, you may need to tweak parameters yourself. :p

[SAPikachu]

1.1.0 and new samples are updated.

[cretindesalpes]

flash3kyuu_deband in high precision mode has the job very well done, indeed. And it is fast. Congratulations to you and to the author of the original filter.

BTW, GradFun3 current default settings are a bit on the conservative side, therefore to get comparable results you should add mask=0 to your tests.

[SAPikachu]

flash3kyuu_deband in high precision mode has the job very well done, indeed. And it is fast. Congratulations to you and to the author of the original filter.

BTW, GradFun3 current default settings are a bit on the conservative side, therefore to get comparable results you should add mask=0 to your tests.

Thanks. :) I've added mask=0 to my scripts and uploaded new samples.

[CruNcher]

yup definitely one of the optimized quality/speed ones here great work :)

[kolak]

Thanks for your great work SAPikachu.

What abut YUY2 support?

There seams to be quite a lot work going on around dithering/debanding.

Is there a solution, which would dither 10bit (or higher) source to 8bit output?

As far as I understand this tool is for debanding and designed to remove existing banding on 8bit sources. What about dithering eg. v210 source- can it do this?


Andrew

[SAPikachu]

Thanks for your great work SAPikachu.

What abut YUY2 support?

There seams to be quite a lot work going on around dithering/debanding.

Is there a solution, which would dither 10bit (or higher) source to 8bit output?

As far as I understand this tool is for debanding and designed to remove existing banding on 8bit sources. What about dithering eg. v210 source- can it do this?


Andrew

YUY2 may be added in next version.

Actually, the high precision mode will upsample pixels to 14bit first, and downsample to 8bit after processing. Dithering occurs in downsampling. The dither algorithm are well-known ones (Floyd-Steinberg dithering (https://secure.wikimedia.org/wikipedia/en/wiki/Floyd–Steinberg_dithering) and Ordered dithering (https://secure.wikimedia.org/wikipedia/en/wiki/Ordered_dithering)). But since AviSynth does not support higher bit-depth natively, it may be difficult to import the video source. If it can be loaded, it shouldn't be very hard to add support for it.

[kolak]

There were some attemps to import v210 Qt into avisynth- not sure if it's working and compatible with your work.


http://forum.doom9.org/showthread.php?t=158985&highlight=v210


Andrew

[ryrynz]

Performance and quality wise how does this compare to ffdshows deband filter?

[SAPikachu]

There were some attemps to import v210 Qt into avisynth- not sure if it's working and compatible with your work.


http://forum.doom9.org/showthread.php?t=158985&highlight=v210


Andrew

OK, I will add a dither function in future version. You may also try cretindesalpes's dither tools (http://forum.doom9.org/showthread.php?p=1386559#post1386559), it has built-in dither function (DitherPost). It has some advanced dither algorithms that may be better than mine. Of course the data need to be restructured to meet its requirement.

[SAPikachu]

Performance and quality wise how does this compare to ffdshows deband filter?

As far as I know, ffdshow uses gradfun2dbmod, its quality is not very good and my filter should be better than it, but I don't have time to test them now. To be sure you can test them yourself.

[naoan]

For real time debanding, this filter is noticeably better than ffdshow's one, tried on several anime source. Thank you for sharing this SAPikachu. :D

[SAPikachu]

For real time debanding, this filter is noticeably better than ffdshow's one, tried on several anime source. Thank you for sharing this SAPikachu. :D

Thanks for testing. :)

[Groucho2004]

I have used this filter in combination with MDegrain for several sources and I'm very happy with the results.

I also played around with the "diff_seed" parameter but can't really see any difference (apart from the fact that "diff_seed = true" is quite a bit slower). Do you have any more info on the usefulness of that parameter?

[SAPikachu]

I have used this filter in combination with MDegrain for several sources and I'm very happy with the results.

I also played around with the "diff_seed" parameter but can't really see any difference (apart from the fact that "diff_seed = true" is quite a bit slower). Do you have any more info on the usefulness of that parameter?

When it is enabled, it will use different seed to generate the reference info and noise pattern for each frame. It's not very useful indeed, I keep it for completeness only.

[ryrynz]

cretindesalpes, any chance of getting a sample of that clip used in the first image? I notice everything but gf3 wipes out all the detail on the bikes in the shadow, gf3 looks the best to me but I assume it's also the slowest.
That image seems to me to be the best test for a deband filter I would like to use it for my own comparisons if I could, thanks.

[cretindesalpes]

Here are some samples of the original clips:
http://www.mediafire.com/?ge0ntbznzbcv76c

Anyway, the screenshots displayed in post #4 of this thread were done with the first version of flash3kyuu_deband. The new high precision mode retains much better the details.

[SAPikachu]

Just added the other 2 images to sample collection. Quality of latest version should be better than the previous one.

PS: I'm going on vacation tomorrow for a week. I will reply to new posts after I return home.

[ryrynz]

Here are some samples of the original clips:
http://www.mediafire.com/?ge0ntbznzbcv76c

Anyway, the screenshots displayed in post #4 of this thread were done with the first version of flash3kyuu_deband. The new high precision mode retains much better the details.

Thanks, I did some testing and flash3kyuu kicks ffdshows deband in terms of quality AND performance, so this filter is a keeper. I find setting the DitherY and DitherC strength to 16 is a good blend of filtering and detail retention, Excellent work!!

Oh and cheers for the introduction to Kyubey. Love the face plant, adorable.

[hetrinity]

SAPikachu,

Thank you for a great filter. I have been using your filter in combination with two other filters to radically improve on the quality of my encodes. Below is an example of my avisynth scripts...

---------------------------------------------------------------------
"SetMemoryMax(1024)

MPEG2Source("myvob5.d2v")

audio=DirectShowSource("myvob5.ac3")
audiodub(audio)

Crop(4,0,-4,-2)
Spline64Resize(720,400)

FFT3DGPU(sigma=2,bw=8,bh=8,bt=1,sharpen=0,plane=0,precision=2,ow=4,oh=4,oldfft=false)
FFT3DGPU(sigma=2,bw=8,bh=8,bt=3,sharpen=0,plane=4,precision=2,ow=4,oh=4,oldfft=false)
flash3kyuu_deband(opt=3,mt=true,precision_mode=1)
---------------------------------------------------------------------

OR (for IVTC)

---------------------------------------------------------------------
SetMemoryMax(1024)

MPEG2Source("myvob1.d2v",idct=0)
TFM(d2v="myvob1.d2v",opt=3)
TDecimate(opt=3)
audio=DirectShowSource("myvob1.ac3")
audiodub(audio)
#Crop(8,0,-6,0)
Spline64Resize(720,400)

FFT3DGPU(sigma=2,bt=1,bw=8,bh=8,sharpen=0,mode=1,plane=0,precision=2,wintype=2,oldfft=false)
FFT3DGPU(sigma=2,bt=3,bw=8,bh=8,sharpen=0,mode=1,plane=4,precision=2,wintype=2,oldfft=false)
flash3kyuu_deband(opt=3,mt=true,precision_mode=1)
---------------------------------------------------------------------

The audio directshowsource would be ac3filter. These, with mediacoder 2011, provide excellent x264 quality for me... and I encode them at my 1100 bitrate. In most cases, the quality of the encode will visibly look superior to the original due to the vast reduction in noise. In comparison to the original, there is some loss of detail, particularly in space scenes where some of the stars are lost. Your filter helped to minimize/eliminate the banding created by my using the FFT3DGPU filters.

I'm no expert or specialist in this area, though, I am an IT person. So, you guys may be way ahead of me in visual quality encodes. Let me know what you think.

[SAPikachu]

SAPikachu,

Thank you for a great filter. I have been using your filter in combination with two other filters to radically improve on the quality of my encodes. Below is an example of my avisynth scripts...

---------------------------------------------------------------------
"SetMemoryMax(1024)

MPEG2Source("myvob5.d2v")

audio=DirectShowSource("myvob5.ac3")
audiodub(audio)

Crop(4,0,-4,-2)
Spline64Resize(720,400)

FFT3DGPU(sigma=2,bw=8,bh=8,bt=1,sharpen=0,plane=0,precision=2,ow=4,oh=4,oldfft=false)
FFT3DGPU(sigma=2,bw=8,bh=8,bt=3,sharpen=0,plane=4,precision=2,ow=4,oh=4,oldfft=false)
flash3kyuu_deband(opt=3,mt=true,precision_mode=1)
---------------------------------------------------------------------

OR (for IVTC)

---------------------------------------------------------------------
SetMemoryMax(1024)

MPEG2Source("myvob1.d2v",idct=0)
TFM(d2v="myvob1.d2v",opt=3)
TDecimate(opt=3)
audio=DirectShowSource("myvob1.ac3")
audiodub(audio)
#Crop(8,0,-6,0)
Spline64Resize(720,400)

FFT3DGPU(sigma=2,bt=1,bw=8,bh=8,sharpen=0,mode=1,plane=0,precision=2,wintype=2,oldfft=false)
FFT3DGPU(sigma=2,bt=3,bw=8,bh=8,sharpen=0,mode=1,plane=4,precision=2,wintype=2,oldfft=false)
flash3kyuu_deband(opt=3,mt=true,precision_mode=1)
---------------------------------------------------------------------

The audio directshowsource would be ac3filter. These, with mediacoder 2011, provide excellent x264 quality for me... and I encode them at my 1100 bitrate. In most cases, the quality of the encode will visibly look superior to the original due to the vast reduction in noise. In comparison to the original, there is some loss of detail, particularly in space scenes where some of the stars are lost. Your filter helped to minimize/eliminate the banding created by my using the FFT3DGPU filters.

I'm no expert or specialist in this area, though, I am an IT person. So, you guys may be way ahead of me in visual quality encodes. Let me know what you think.

I don't use FFT3DGPU much, but as far as I know sigma=2 is very high for normal videos, and due to your plane settings, the luma plane is filtered twice, so it is not surprising that many details are lost. To gain more details, you may try lowering the sigma values, and/or change plane in the second FFT3DGPU to 3, make it filter only the chroma planes.

Also for flash3kyuu_deband, opt and mt should usually be left out unless error occurs. The default settings are rather strong, so you may want to lower them a bit, particularly ditherY and ditherC.

Have a happy encoding. :)

[ryrynz]

there is some loss of detail, particularly in space scenes where some of the stars are lost.

Setting DitherY and DitherC to a lower value (I prefer 16 for each) as stated by SAPikachu and myself will help improve detail.

[hetrinity]

Thanks everyone for your input. I have a quick question. I'm not sure if the avsinput for MediaCoder will work with 64 bit avisynth, but, I have all my filters in 64 bit, except for your flash3kyuu_deband. Do you have a current 64 bit version made, or planned for later down the road? Thanks again!

[SAPikachu]

I have plan to make a 64bit version, but it has low priority and I have been very busy recently, so it may take some time to complete...

[SAPikachu]

Released 1.2.0.

[ryrynz]

Thank you SAPikachu, looking forward to the 16bit internal precision.

[SAPikachu]

Released 1.3.0. Added x64 version.

[mp3dom]

VERY very nice!

[kolak]

f3kdb_dither, but how to feed 10bit file :)


Andrew

[SAPikachu]

f3kdb_dither, but how to feed 10bit file :)


Andrew

You need a source filter that can output high bit-depth data. But I don't know any h264 source filter with high output at this time (Actually f3kdb_dither is originally made for raw source filters, not 10bit h264). Maybe someone can modify ffms2 to add this functionality.

[mp3dom]

I think kolak means not to input a 10bit AVC file but rather a raw 10bit (like v210 avi/mov). AVS plugins actually manages 8bit (truncated or dithered) from a 10bit source and BlackMagic codecs or AE internally made a colorspace conversion to RGB. What we needs is something like a v210 input plugins that outputs a (16bit?) stacked clip so flash and other dither tools can then be used to dither a 10bit clip directly and output a final 8bit clip.

[kolak]

Yes- I would like to feed v210 for dithering to 8bit.

[SAPikachu]

Sorry that I misunderstood the problem. As far as I know you need RawReader (Sashimi) and mt_lut to load the file as correct format, like http://forum.doom9.org/showthread.php?t=158985&highlight=v210 . I will try to figure it out later today.

[SAPikachu]

Just completed the script. I tested it on two samples from http://samples.mplayerhq.hu/V-codecs/v210/ (the 720p one need to be converted to mov container first). Script needs avisynth 2.6.

file_head needs to be manually figured out for other files. Open the file in a hex editor, find last "mdat" tag in ASCII view (won't be too far from start), file_head is offset of the immediately followed byte.

EDIT: There are special cases, see http://forum.doom9.org/showthread.php?p=1526877#post1526877 .


# modified from http://forum.doom9.org/showthread.php?p=1469679#post1469679

function readv210(string fn, int file_head, int frame_width, int frame_height, bool "flip") {

line_size = (frame_width * 16 / 6 + 127) / 128 * 128 # all lines are padded to 128 bytes boundary

base=RawReader (fn, format="y8", width=line_size, height=frame_height, numframes=0, filehead=file_head, framehead=0, flip=default(flip,false))


p0=base.every(4,0)
p1=base.every(4,1)
p2=base.every(4,2)
p3=base.every(4,3)

lsb0=mt_lut(p0,yexpr="x 6 <<u 255 &u")
msb0=mt_lutxy(p0,p1,yexpr="x 2 >>u y 6 <<u |u 255 &u")
out0=StackVertical(msb0, lsb0)

lsb1=mt_lut(p1,yexpr="x 2 >>u 6 <<u 255 &u")
msb1=mt_lutxy(p1,p2,yexpr="x 4 >>u y 4 <<u |u 255 &u")
out1=StackVertical(msb1, lsb1)

lsb2=mt_lut(p2,yexpr="x 4 >>u 6 <<u 255 &u")
msb2=mt_lutxy(p2,p3,"x 6 >>u y 2 <<u |u 255 &u")
out2=StackVertical(msb2,lsb2)

weave3h(out0,out1,out2)

y=last.every(2,1)
u=last.every(4,0)
v=last.every(4,2)

# return last
YToUV(u, v, y)

Crop(0,0,frame_width,0)

f3kdb_dither(stacked=true)


}

function every(clip v, int n, int offset) {#select every n bytes horizontally with offset, works on yv12 only
v
w=width
h=height
pointresize(v,w*2,h)
crop(offset*2,0,0,0).addborders(0,0,offset*2,0)#shift left offset pixels
pointresize(w/n,h)
}

function weave3h(clip a, clip b, clip c) {#horizontally weave 3 clips
a=a.turnright
b=b.turnright
c=c.turnright

interleave(a,c,b,c) # a c b c -> ac bc -> abcc
assumefieldbased
assumetff
weave
assumefieldbased
assumetff
weave

# From http://avisynth.org/mediawiki/Advanced_Scripting_Tips:

# In RGB, PointResize deletes the first of each group of 4 lines.
# However, in YUV modes, it deletes the last of each group of 4 lines.
pointresize(width,height*3/4)
turnleft
}


LoadPlugin("flash3kyuu_deband.dll")

# readv210("output.mov", 36, 640, 480, flip=true)

# readv210("coff3.mov", 1103, 176, 144)

readv210("v210_720p.mov", 36, 1280, 720)

[kolak]

Thanks a lot! I will test it (once have a bit of free time) and let you know how it behaves with v210 files.
File has to be be v210 mov- yes?

[mp3dom]

Thanks!! Now one final question... is it possible to convert a v210 AVI into MOV without recompress/bit-depth lost/colorspace conversion? (a Direct-stream copy changing only the container). Thanks.

[kolak]

FFmpeg can do it- just use vcodec copy option:)

[mp3dom]

Doh! Thanks! :)

[kolak]

Just completed the script. I tested it on two samples from http://samples.mplayerhq.hu/V-codecs/v210/ (the 720p one need to be converted to mov container first). Script needs avisynth 2.6.

file_head needs to be manually figured out for other files. Open the file in a hex editor, find last "mdat" tag in ASCII view (won't be too far from start), file_head is offset of the immediately followed byte.



It works ! :)

I had some 5sec v210 sample- some footage from Arri Alexa.

Does it do dithering inside function?
f3kdb_dither(stacked=true) - can I customise settings here?
If I remove f3kdb_dither(stacked=true) does it mean I can pass data to other tools which support 10bit in stacked mode?

Why does it have to be MOV?

Going to do some tests :)

Thanks a lot,
Andrew

[kolak]

I found some small problem on v210 file generated by AE (my other file is fine).

http://img196.imageshack.us/img196/6905/unledlfk.png

Top left corner has few bad pixels- files is 1920x1080.

Any idea why- is it AE?

Andrew

[poisondeathray]

I found some small problem on v210 file generated by AE (my other file is fine).

Top left corner has few bad pixels- files is 1920x1080.

Any idea why- is it AE?




No, I don't think it's AE

I noticed this in avspmod or vdub (just the readv210 function), but the defect was on bottom right for me

I used the greyscale v210 ramps I prepared earlier in the v210 thread
http://www.mediafire.com/?cxc8bmw5haceclo

[kolak]

I'm trying with AVIs and it shows picture, but it all gets shifted. Is it due to some padding in AVI container? RIFF chunk list shows data and some padding between chunks- is this padding causing problems? I also notice that some software will create AVIs with constant padding (so there is pattern) and some with one which changes. All bit of black magic for me :)

Does MOV stores all data in one go (more like raw yuv file)?

[kolak]

No, I don't think it's AE

I noticed this in avspmod or vdub (just the readv210 function), but the defect was on bottom right for me

I used the greyscale v210 ramps I prepared earlier in the v210 thread
http://www.mediafire.com/?cxc8bmw5haceclo

Hmmm- AE creates problematic AVIs- I had strange problems- some software don't like these streams. My other sample (coming from MAC I assume- no it comes from Adobe software also!) works fine.

[poisondeathray]

Hmmm- AE creates problematic AVIs- I had strange problems- some software don't like these streams. My other sample (coming from MAC I assume) works fine.


But the samples I posted were MOV wrapped :)


What v210 decoder are you using in your avs script for your screenshot ? (or are you using the script in this thread?)

[kolak]

Yes- script from this thread and Vdub.
I use my own samples- some work, some (one from AE) have this small artefacts- not sure why. They work fine in other apps which can read v210.

AVIs don't work.

[poisondeathray]

Yes- script from this thread and Vdub.
I use my own samples- some work, some (one from AE) have this small artefacts- not sure why. They work fine in other apps which can read v210.

AVIs don't work.



For AE, I notice it depends on the decoder. Sometimes internal, vs. BM, vs. AJA, vs. Drastic will give different results, it's unpredictable . When you hover over the asset in the clip bin, it will tell you which decoder it is using. You can uninstall that decoder and use another (restart AE)

But that defect in your screenshot I'm fairly certain is from the avs script

[kolak]

IT's other way around- in AE I don't have problems. With files (AVIs mainly) created by AE I have sometimes strange problems- something to do with headers which AE creates. It always puts lots of metadata there. Re-savin file in Vdub fixes this- strange.

[mp3dom]

Tried to convert v210 AVI to MOV with ffmpeg but using the above script I get all messed colors (pink become cyan, reds become blu, light yellow become light green and so on).

[poisondeathray]

Tried to convert v210 AVI to MOV with ffmpeg but using the above script I get all messed colors (pink become cyan, reds become blu, light yellow become light green and so on).


Did you use the correct file_head value with the hex editor ?

I found the same results as you when using the incorrect values

The only problem I still see is the defective corner pixels

[SAPikachu]

@kolak @poisondeathray
I tried the grayscale ramp sample but I don't get any defective pixels, I used the following line:


readv210("radialramp_1080p29.97_16bitRGB_v210.mov", 48, 1920, 1080)


Maybe file_head is wrong?

@kolak
Yes, f3kdb_dither does Floyd-Steinberg dithering by default. You can see readme.txt for parameter details. But only change "mode" and "keep_tv_range" as other parameters are to specify source data format. To get stacked 16bit clip just remove the f3kdb_dither line.

Need to be mov because avi may add variable-length data between frames. Actually raw v210 bitstream may also work, but I haven't test that.

[poisondeathray]

@kolak @poisondeathray
I tried the grayscale ramp sample but I don't get any defective pixels, I used the following line:


readv210("radialramp_1080p29.97_16bitRGB_v210.mov", 48, 1920, 1080)


Maybe file_head is wrong?


Thanks for reply,

yes, file_head was wrong. I used 64

Can you describe how you got "48" ? I don't know much about hex editors... I highlighted "mdat" and the value next to it was 64

[SAPikachu]

I use WinHex. Here is an demo:

https://lh5.googleusercontent.com/-mnbVS0B7Nas/TnFKHbDepEI/AAAAAAAAAcA/72Is0h6_Ek0/s800/hex.JPG

(Note the offset value is hex number, need to be converted to decimal for parameter)

EDIT: There are special cases, see http://forum.doom9.org/showthread.php?p=1526877#post1526877 .

[poisondeathray]

Thank you for explanation and screenshot, I understand now :)

So "30" from the "offset" is expressed in hexadecimal, but is "48" in decimal value

[SAPikachu]

Yes, that's right. :)

[mp3dom]

Did you use the correct file_head value with the hex editor ?

I found the same results as you when using the incorrect values

The only problem I still see is the defective corner pixels

I've used the right value, I think... the byte next to 'mdat' is at location 1C and I've used 28 as offset. The image doesn't have any corruption, just wrong colors.

Edit: OK, I get it... after a bit of 'brute forcing', using 36 was successfully. But 36 (hex: 24) it's not immediately after mdat. The previous bytes contains '10'.
Anyway, If I want to deband the clip with 16bit precision, can I use flash3kyuu? Should I put it just before the call to f3kdb?

[SAPikachu]

I've used the right value, I think... the byte next to 'mdat' is at location 1C and I've used 28 as offset. The image doesn't have any corruption, just wrong colors.

Edit: OK, I get it... after a bit of 'brute forcing', using 36 was successfully. But 36 (hex: 24) it's not immediately after mdat. The previous bytes contains '10'

36 is not correct. If you don't see corrupted pixel, 28 is probably right value. Try adding SwapUV after loading the file?

EDIT: I was wrong. Please see http://forum.doom9.org/showthread.php?p=1526877#post1526877 for details.

[SAPikachu]

.
Anyway, If I want to deband the clip with 16bit precision, can I use flash3kyuu? Should I put it just before the call to f3kdb?

It is not possible with f3kdb now. I will add high bit-depth input in next version. Right now you can use the dither package. Just remove f3kdb_dither and treat it as stacked 16bit clip.

[kolak]

@kolak @poisondeathray
I tried the grayscale ramp sample but I don't get any defective pixels, I used the following line:


readv210("radialramp_1080p29.97_16bitRGB_v210.mov", 48, 1920, 1080)


Maybe file_head is wrong?

@kolak
Yes, f3kdb_dither does Floyd-Steinberg dithering by default. You can see readme.txt for parameter details. But only change "mode" and "keep_tv_range" as other parameters are to specify source data format. To get stacked 16bit clip just remove the f3kdb_dither line.

Need to be mov because avi may add variable-length data between frames. Actually raw v210 bitstream may also work, but I haven't test that.

I'm not talking about ramp file- I have created file in AE (which is actually also ramp file- hehe) and I have issue on this file. I have also other file and this one is fine.

I will post this file, so we can verify- I believe my ofset is correct, but we can double check.

AVIs- yes, I've noticed that some v210 have not constant dummy data, but some do have constant. I can also post such a sample.

Andrew

[qyot27]

Are there certain hardware requirements (CPU, RAM, etc.) that need to be met? I noticed that flash3kyuu_dither has a base requirement of an SSE2-capable processor, but there's no such warning in the README's flash3kyuu_deband section.

Even with opt=0 set, I'm getting an "Evaluate: System exception - Illegal instruction" error pointing to the line where flash3kyuu_deband gets called, and I can only see that error when trying to load the script into VirtualDub - if I try to play the script in WMP6.4 the error text doesn't show up at all. Commenting it out allows the script to load as usual.

I wouldn't be surprised if it's just that my hardware is way too old (Coppermine-based Celeron = no SSE2 at all), but I would have thought that since the opt parameter defaults to autoselecting the right instruction set level and opt=0 disables optimizations, that it wouldn't error out like that, and just be a lot slower.

[kolak]

I've tried radialramps (http://www.mediafire.com/?cxc8bmw5haceclo) 1080 file and this works fine for me also.

I came back to my problematic file and.... works fine:)
Sorry for wrong report- it was an user error :)

I also tried yet another file- ProRes from Alexa converted to AJA v210 in QT- also works fine :)

[kolak]

readv210 works with RAW v210 files also :) (ofset=0)


Andrew

[SAPikachu]

@kolak
No problem, it is great to get it working. :)

If the AVI file has constant frame header, it is possible to read it by setting framehead in the script. But I think it is still easier to convert the file first.

@qyot27
That's weird, I checked the disassembly, C version don't have SSE-related code (except memset, but it will check CPU at runtime to select instruction set), and should be runnable on Coppermine... I don't have that CPU so I can't find the root cause, but I suspect it is an ICC problem. Can you try this build (http://nmm.me/x), with and without opt=0, and see if it works? Thanks.

[mp3dom]

No problem, it is great to get it working. :)

Yes, definitely! It's great to have the possibility to correctly manage 10bit in AVS and use dither/debanding plugins without colorspace conversion!
Thanks for your efforts!

[qyot27]

@qyot27
That's weird, I checked the disassembly, C version don't have SSE-related code (except memset, but it will check CPU at runtime to select instruction set), and should be runnable on Coppermine... I don't have that CPU so I can't find the root cause, but I suspect it is an ICC problem. Can you try this build (http://nmm.me/x), with and without opt=0, and see if it works? Thanks.
The new build worked in both instances, with or without opt=0. Thanks for taking a look.

[SAPikachu]

The new build worked in both instances, with or without opt=0. Thanks for taking a look.

Got it. :) So ICC generates instructions that older CPU can't support, I need to maintain a msvc version for them.

[kolak]

mp3dom- don't use it on files which are re-wrapped from AVIs- it looks like it does not work. Padding data is not constant, so there are distortions- different on different frames.
You can always force ffmpeg to re-encode video and than put to MOV container- I think it can do v210 to v210 properly.


Andrew

[mp3dom]

Tried -vcodec copy and -vcodec v210 on ffmpeg and both options outputs the exact same file. It's bit per bit identical. Regarding my offset (28) using SwapUV I see correct colors but also some corruption (probably already there, not due to SwapUV). So it seems that 36 is the real offset (watching almost all frames, seems correct) but it's not immediately after 'mdat'

Just for reference, I have this:
http://thumbnails52.imagebam.com/14973/11ca0e149720767.gif (http://www.imagebam.com/image/11ca0e149720767)
Red arrow: What I should use
Blue arrow: What I'm using

Edit: kolak, I know hex numbers, 0x1c=28, 0x24=36

[kolak]

Make sure you understand numbers. These are HEX, so you can convert in windows calc to decimal. There is only one valid value (I believe), so another is wrong :)

28 HEX is 40 dec and this is what my file needed.

[poisondeathray]

@mp3dom - SAPikachu said to use the offset number (as in the screenshot in the previous page), so in your screenshot the offset for the next byte is "10" and converted is "16" ?

[kolak]

@mp3dom - SAPikachu said to use the offset number (as in the screenshot in the previous page), so in your screenshot the offset for the next byte is "10" and converted is "16" ?

No- ofset is 1C, so it's 28 dec. Use 28 in v210reader.

Andrew

[poisondeathray]

No- ofset is 1C, so it's 28 dec. Use 28 in v210reader.



OK I see , that makes sense . I was reading from the column, not the actual offset

Then why is he getting swapped colors ?

[mp3dom]

I don't know, probably a strange v210 AVI. I'm start to think that it can be a mac conversion from v210 QT to v210 AVI. Unfortunately the files comes from an external company, not from our capture so I don't have a lot of infos about it.

[kolak]

OK I see , that makes sense . I was reading from the column, not the actual offset

Then why is he getting swapped colors ?

It looks like re-wrapping from AVI to MOV- I and the same problem and it was also after re-wrapping.

[mp3dom]

Makes sense, but I have recompressed it again to uncompressed mov but the file is exactly the same (unless -vcodec v210 doesn't mean 'recompress to v210')

[kolak]

Makes sense, but I have recompressed it again to uncompressed mov but the file is exactly the same (unless -vcodec v210 doesn't mean 'recompress to v210')

Maybe ffmpeg does "smart rendering" so it actually does the same as copy option. :)

[SAPikachu]

@mp3dom @kolak @poisondeathray

I just checked the qt format spec and found the reason. mp3dom was right and I was wrong, the offset should be 0x24 = 36. This mdat chunk has 64bit length - that is, the 4 bytes before "mdat" is "00 00 00 01", and a 8-byte length follows the tag, real data then follows the length. So file_head should be 8 bytes after mdat.

[SAPikachu]

Makes sense, but I have recompressed it again to uncompressed mov but the file is exactly the same (unless -vcodec v210 doesn't mean 'recompress to v210')

v210 is an uncompressed format, so recompress the file won't change the data.

[jmac698]

@Sap...
That's great, I wrote the original script. Can I package it up with your credit? I wanted to know the quick way to find the file head, but I didn't want to read the huge manual.. now I know and I think I can fix this in script to be fully automatic, which is great!
I think we're really making progress on some quite usable high bit depth processing, even ffmeg is going to have ProRes soon...

ps what would be really useful is a type of parsing file reader that you could say fileread(int a, float b, str c ) etc. something like that, but choose word length to read only bytes, and nops to skip bytes. We need to read data files. Maybe a version of sashimi that reads to script variables instead of clips?

[kolak]

@mp3dom @kolak @poisondeathray

I just checked the qt format spec and found the reason. mp3dom was right and I was wrong, the offset should be 0x24 = 36. This mdat chunk has 64bit length - that is, the 4 bytes before "mdat" is "00 00 00 01", and a 8-byte length follows the tag, real data then follows the length. So file_head should be 8 bytes after mdat.

OK- so we need swapuv after script, otherwise colors are wrong. I will check with different files, but it looks like this is a rule (so script can be adjusted inside function).

btw...I compared (just quickly for now) quality and it looks very good and promising. Conversion was slow thought 1.5fps for full HD, but using only one core. Not sure how I can make it using many cores, but in theory it can run at acceptable speed. Looking good :)

Andrew

[SAPikachu]

@Sap...
That's great, I wrote the original script. Can I package it up with your credit? I wanted to know the quick way to find the file head, but I didn't want to read the huge manual.. now I know and I think I can fix this in script to be fully automatic, which is great!
I think we're really making progress on some quite usable high bit depth processing, even ffmeg is going to have ProRes soon...

ps what would be really useful is a type of parsing file reader that you could say fileread(int a, float b, str c ) etc. something like that, but choose word length to read only bytes, and nops to skip bytes. We need to read data files. Maybe a version of sashimi that reads to script variables instead of clips?

Of course you can. Just do anything to it as you like. :)

I don't understand the second part. Do you want to directly parse the file header in script? I think that will be very hard and hackish since AviSynth lacks some useful features (like loops). A better method is integrate a more powerful scripting environment into it. Actually I am thinking of creating a python plugin environment, but I don't have much time to do it recently...

[kolak]

Another thing- is this Floyd dithering just pure implementation?

Filter which I compared it to use noise shaping/error diffusion plus optional random noise.
When noise is added, an R250 random number generator is used. The noise is injected into the quantizer, not into the signal, so the error propagated to the adjacent pixels does not include the random noise.

Can we do the same- is it good idea?


Andrew

[SAPikachu]

OK- so we need swapuv after script, otherwise colors are wrong. I will check with different files, but it looks like this is a rule (so script can be adjusted inside function).

btw...I compared (just quickly for now) quality and it looks very good and promising. Conversion was slow thought 1.5fps for full HD, but using only one core. Not sure how I can make it using many cores, but in theory it can run at acceptable speed. Looking good :)

Andrew

We just need to adjust the file_head based on the mdat chunk mode. SwapUV is unneeded and wrong.

It is hard to get speed improvement using pure script implementation. Better to integrate it into a source plugin, like Sashimi.

[kolak]

We just need to adjust the file_head based on the mdat chunk mode. SwapUV is unneeded and wrong.

It is hard to get speed improvement using pure script implementation. Better to integrate it into a source plugin, like Sashimi.

But when I use new (as you said- correct offset) colors are wrong :(
Tried 2 totally different files- without swpuv colors are wrong.


Andrew

[SAPikachu]

Another thing- is this Floyd dithering just pure implementation?

Filter which I compared it to use noise shaping/error diffusion plus optional random noise.
When noise is added, an R250 random number generator is used. The noise is injected into the quantizer, not into the signal, so the error propagated to the adjacent pixels does not include the random noise.

Can we do the same- is it good idea?


Andrew

f3kdb_dither is pure dithering. I think noise is unneeded unless the clip needs debanding. In next version you can use f3kdb on the 16bit clip.

[mp3dom]

OK- so we need swapuv after script, otherwise colors are wrong.

No, wrong colors was with wrong offset (0x1c) and in that case the SwapUV fixed it. With right offset (0x24) the colors are ok from the start.


When noise is added, an R250 random number generator is used. The noise is injected into the quantizer, not into the signal, so the error propagated to the adjacent pixels does not include the random noise.

Xscaler :D

f3kdb_dither is pure dithering. I think noise is unneeded unless the clip needs debanding. In next version you can use f3kdb on the 16bit clip.
Is it possible (in a future release, without hurry) to implement it as an optional parameter? About f3kdb, isn't it already working on 16bit clip as source? Or are you referring to the deband part (flash3kyuu)? In that case it would be very helpful to deband a 16bit clip and then dither to 8bit.

[SAPikachu]

But when I sue new (as you said- correct offset) colors are wrong :(



Andrew

There is 2 types of mdat chunk. If the 4 bytes before mdat is 00 00 00 01, you need to add 8 to the offset after mdat. Otherwise the original offset is correct.

[kolak]

There is 2 types of mdat chunk. If the 4 bytes before mdat is 00 00 00 01, you need to add 8 to the offset after mdat. Otherwise the original offset is correct.

OK- understand now- thanks. Didn't get your original message: Will check and confirm.
Yes- my files don't have 00 00 00 01, so "normal" offset works fine.

[SAPikachu]

Is it possible (in a future release, without hurry) to implement it as an optional parameter? About f3kdb, isn't it already working on 16bit clip as source? Or are you referring to the deband part (flash3kyuu)? In that case it would be very helpful to deband a 16bit clip and then dither to 8bit.

Yes I meant flash3kyuu_deband. I will make the new version when I have time.

[jmac698]

Yes the script can be sped up alot, that is the reason I requested some new features in sashimi. Those features were finalized just yesterday, so now it is time to rewrite readv210 with the new features, which should speed up load the files a lot.

My understanding of dither is that it can be fixed or random. I read a whole textbook on it once (long time ago), and it seems that uncorrelated noise with shaping looked best. That old floyd-steinberg is easy to understand but causes it's own banding (there's only 5 shades for 4 dots).

Btw, I worked on my own algorithm for fixed dither, that always sums to the target value for the area, meaning you could perfectly reverse it back into a high bit-depth image at full resolution! It can't always work for some pixels though.

As for the file parser, I could use this idea for other things, speed is not an issue, like reading the histograms from colorlike per frame, it's only 256 numbers of text, or reading the in-pattern results in dgindex, just one line per frame.. and in this case, reading 8 bytes to determine file header - such a siimple task, that I don't want to write a whole plugin just for those numbers... because ultimiately I find coding up an idea in script far, far faster to get a practical result (even though it may be slow). And yes you can do loops in avisythn, however I don't need them to parse files, the plugin will have some simple logic for that.

[Gavino]

AviSynth lacks some useful features (like loops).
See GScript in my sig.
For the purist, you can also use recursive functions to create loops in vanilla Avisynth.

[jmac698]

recursion and gscript are documented here
http://avisynth.org/mediawiki/Advanced_Scripting_Tips

[kolak]

My understanding of dither is that it can be fixed or random. I read a whole textbook on it once (long time ago), and it seems that uncorrelated noise with shaping looked best. That old floyd-steinberg is easy to understand but causes it's own banding (there's only 5 shades for 4 dots).



I think thit is the reason why MS filter adds noise- to cover FS own banding.

To be honest- going from 10 to 8bit with MS filter looks good enough- I would rather like to have more speed.

What about these methods:

Jarvis, Judice, and Ninke dithering diffuses the error also to pixels one step further away. The dithering is coarser, but has fewer visual artifacts. It is slower than Floyd–Steinberg dithering because it distributes errors among 12 nearby pixels instead of 4 nearby pixels for Floyd–Steinberg.

Stucki dithering is based on the above, but is slightly faster. Its output tends to be clean and sharp.

Burkes dithering is a simplified form of Stucki dithering that is faster, but less clean than Stucki dithering.

As a goal I would see 2x faster than RT dithering for full HD on eg. 12 core PC, with quality as FS.


Andrew

[kolak]

Current solution can give about 12fps on 8 core machine (older Xeons- not i7)- just need to run many instances. MS filter is a bit faster.


Andrew

[Fullmetal Encoder]

Yes I meant flash3kyuu_deband. I will make the new version when I have time.

Thank you very much for this. I eagerly await the full higher bit depth capability. Thanks also for providing us with such a high quality dithering tool. It's very impressive compared to what else is available.

[SAPikachu]

@Gavino @jmac698
So I must revise my wording - AviSynth doesn't have some convenient language features. At least it is very hard to parse raw data directly in script. Of course there are hackish ways to do that. :)

@jmac698 @kolak
Adding random noise can indeed reduce banding. Actually flash3kyuu_deband does that between core debanding and dithering.

[kolak]

Can you add optional (with possibility to set strength) add noise to fkdb_dither?


Andrew

[SAPikachu]

Can you add optional (with possibility to set strength) add noise to fkdb_dither?


Andrew

Actually that's what flash3kyuu_deband does. After adding high bit-depth input, it will be superset of f3kdb_dither.

[kolak]

OK- you're adding high bit-depth input for flash3kyuu_deband and than f3kdb_dither is not needed- cool :)

[kolak]

Not sure if it's a good place, but:

is there anyone interested to write frameserver for one of the editing software (preferably in v20 and YUY2- I believe theses are native formats for software for 10 and 8bit mode). Not for free.

Something like debugmode for Premiere/Vegas.

You can contact me via PM.

Thanks,
Andrew

[mp3dom]

The problem is that a lot of NLE internally works in RGB colorspace or have RGB filters. A YUY2/v210 output makes sense if no colorspace occurs during all the editing stage. Anyway Vegas and Premiere already can use DebugMode, Avid (and I suppose FinalCut) uses QuickTime Reference (which is similar to DebugMode).

[kolak]

Different software, which is way more stable and faster than all of them and it does work internally in YUV (maybe that's why it's so fast).


Andrew

[kolak]

Actually that's what flash3kyuu_deband does. After adding high bit-depth input, it will be superset of f3kdb_dither.

Done some test an additional noise does help- file from Xscaler is "smoother" than from f3kdb_dither and the difference is due to additional noise added by Xscaler filter.

flash3kyuu_deband with high bit-depth support will be very powerful :)

[SAPikachu]

Finally escaped from another project and get some free time. :)

New version should be available in 1 or 2 weeks.

[kolak]

Cool :)

Andrew

[mp3dom]

Awesome! I've the right project to test on :)

[SAPikachu]

A little update. My work has suddenly became very busy recently, I only completed about 50% of the improvement in the last 2 weeks... Need maybe several weeks more to fully implement the changes.

I also planned to add some more features to the new version:

* 9 - 15 bit output
* dynamic noise

[kolak]

Dynamic noise sounds interesting :)

Can't wait to see it in action.

[mp3dom]

What should do "dynamic noise"? You mean put noise only on zones where debanding take place (without adding noise to, example, a flat area of the same image)? Sounds cool! Can't wait to see how much is the gain (or detail preservation) in debanding an 8bit source vs debanding directly a 10bit source.

[SAPikachu]

What should do "dynamic noise"? You mean put noise only on zones where debanding take place (without adding noise to, example, a flat area of the same image)? Sounds cool! Can't wait to see how much is the gain (or detail preservation) in debanding an 8bit source vs debanding directly a 10bit source.

"dynamic noise" means different noise pattern on different frames, somewhat like diff_seed but much faster. (Actually diff_seed will be replaced with that). I am finalizing the last new functionality (9 ~ 15 bit output), new version should be available in a few days. (Hope the estimation is accurate this time :P )

[mp3dom]

Fantastic! Thanks a lot!!

[Keiyakusha]

What is the possible benefit it different noise pattern over frames? For compression, all I can see is that it makes harder to compress video, and harder to preserve deband. For real-time, it will behave more like noise and pattern is not something visible by your eye anyway...

[SAPikachu]

What is the possible benefit it different noise pattern over frames? For compression, all I can see is that it makes harder to compress video, and harder to preserve deband. For real-time, it will behave more like noise and pattern is not something visible by your eye anyway...

I heard from somewhere that dynamic noise is good for some videos, but I don't remember the exact theory. Anyway, it is only a switch and not very hard to implement. Just try it yourself. :)

[Nevilne]

It will make encoders allocate more bits to low detail areas - such as gradients.
Though if done right the dithering noise should not matter too much.

[Keiyakusha]

I heard from somewhere that dynamic noise is good for some videos, but I don't remember the exact theory. Anyway, it is only a switch and not very hard to implement. Just try it yourself. :)

Its OK I was just wondering what is the real use for it. Anyway I'm really looking for 10bit output :)

It will make encoders allocate more bits to low detail areas - such as gradients.
While probably true, this only hurt compression overall. Because these bits will be wasted. Allocating more bits while pattern is static is good, but when it is dynamic more bits needed to achieve the same perceived quality level. So you'll end up taking bits from high detail areas for nothing. On the opposite way, what we want for encoding is calming down fluctuations to reduce difference between similar frames and then allocating more bits by tweaking encoder itself.

[SAPikachu]

Released 1.4.0.

[mp3dom]

Great!! Thanks!!
Just to be sure to do the right thing... I'm importing a v210 mov via your ReadV210 so I'm having a stacked clip.
Is this the right cmd to do the debanding at 10/16bit and then dithering to 8bit regular clip?

ReadV210(...)
f3kdb(precision_mode=4,input_mode=1,input_depth=16)
f3kdb_dither(mode=1,stacked=true,input_depth=16)

It seems the right way. In any case, the result is amazing!! Thanks!

[SAPikachu]

Great!! Thanks!!
Just to be sure to do the right thing... I'm importing a v210 mov via your ReadV210 so I'm having a stacked clip.
Is this the right cmd to do the debanding at 10/16bit and then dithering to 8bit regular clip?

ReadV210(...)
f3kdb(precision_mode=4,input_mode=1,input_depth=16)
f3kdb_dither(mode=1,stacked=true,input_depth=16)

It seems the right way. In any case, the result is amazing!! Thanks!

Don't need to add f3kdb_dither after f3kdb, it's OK to just use precision_mode = 3, the result will be the same since the algorithm is identical. :)

[mp3dom]

Wow, so we can do it now in only 'one pass'. So f3kdb_dither now is only meant if someone wants to dither down a clip without debanding, right?
Thanks again very much! The results look impressive!

[SAPikachu]

Wow, so we can do it now in only 'one pass'. So f3kdb_dither now is only meant if someone wants to dither down a clip without debanding, right?
Thanks again very much! The results look impressive!

Yes, actually f3kdb_dither is only a shortcut now, same result can be achieved with f3kdb by setting Y/Cb/Cr/ditherY/ditherC to 0.

[kolak]

Once I have bit of time going to test it- thanks for your work SAPikachu:thanks:

[kolak]

When I tried ver 1.4 with avisynth 2.6 I got out of memory error in your dll reported by Vdub.
Same conditions, just with older version- no problem.

[DarkT]

GradFun3 results, in comparison pix from page 1 look better to me. Is the new version better? What are the adventages over GF3?

[cretindesalpes]

f3kdb is faster than GradFun3, at least with settings giving similar results.

[mp3dom]

I get some exception too when a stacked clip (1920x2160) is feeded as YUY2 (not YV16).
Regarding the differences, I found that f3kdb correctly debands zones that GradFun3 doesn't detect, but GradFun3 seems to preserve very dark areas (and edges with similar colors to contiguous zones) better.

To enhance the edges detection, since now we can work on 16bit, isn't it possible to work with curves (increase contrast, get a mask to get where to dither, return to original curve and dither, skipping edges) or to create an edge mask on the source and then copy it (only the edges detected) to the debanded clip?

[SAPikachu]

When I tried ver 1.4 with avisynth 2.6 I got out of memory error in your dll reported by Vdub.
Same conditions, just with older version- no problem.

Maybe there is a bug in my code, can you post your parameter and colorspace?

[SAPikachu]

I get some exception too when a stacked clip (1920x2160) is feeded as YUY2 (not YV16).
Regarding the differences, I found that f3kdb correctly debands zones that GradFun3 doesn't detect, but GradFun3 seems to preserve very dark areas (and edges with similar colors to contiguous zones) better.

To enhance the edges detection, since now we can work on 16bit, isn't it possible to work with curves (increase contrast, get a mask to get where to dither, return to original curve and dither, skipping edges) or to create an edge mask on the source and then copy it (only the edges detected) to the debanded clip?

I will check the YUY2 problem later in this week, but I recommend to use YV16, because YUY2 code is not well-tested nor optimized, so it will be slow and maybe buggy.

Regarding curves and edgemask, they are good ideas, but I have a promise with my friend and need to have some research on a denoise algorithm(maybe a new avs filter), so I won't work on new features of f3kdb for a while. Will work on that after I have some progress on the algorithm. :)

[kolak]

Maybe there is a bug in my code, can you post your parameter and colorspace?

It was yuy2 source, so maybe this is the issue. Standard settings with keep tv range.

[Fullmetal Encoder]

To enhance the edges detection, since now we can work on 16bit, isn't it possible to work with curves (increase contrast, get a mask to get where to dither, return to original curve and dither, skipping edges) or to create an edge mask on the source and then copy it (only the edges detected) to the debanded clip?

This was just the idea I had when I had discovered dither tools. It would seem to me that masktools would do the job quite nicely and could be used with f3kdb inside AviSynth without the need to code any kind of edge mask functionality into f3kdb itself. The only problem I can see though is that masktools would have to be modified/re-written with at least 16-bit internal precision, would it not? A minor detail.

Ideally it would be wonderful if we could get a full 16-bit pipeline established like so: QTGMC(dfttest, nnedi3, masktools) --> f3kdb --> 10-bit x264

I would love to contribute to this goal myself but while I know a little C++ and am studying it it will be a long time before I can do anything meaningful at that level.

[Fullmetal Encoder]

I just recently began experimenting with 10-bit encoding. I'm using f3kdb and I think I understand what it's doing through the encoding stage but I was hoping that someone here could help by confirming if my understanding and approach is correct. At the moment I am working with interlaced Blu-ray material. My goal is to deband the source and then dither down from the 16-bit internal precision to 10-bits using f3kdb before sending the output through to x264-10bit. What I'm thinking is that sending the 16-bit results directly to x264-10bit would truncate the 16 bits to 10 while a much higher quality way to do it is to dither to 10 bits inside f3kdb first. Is this correct? My script is as follows:

LoadPlugin("DGAVCDecode.dll")
AVCSource(dga="00041.dga", deblock=false)
f3kdb(sample_mode=2, dynamic_dither_noise=true, precision_mode=2, output_depth=10, output_mode=2, keep_TV_range=true)

Also, my understanding is that since the source is in YUV 4:2:0 the 16-bit processing in f3kdb is in the same format only with the values in higher decimal precision (internally). When the dithering is done something that goes beyond what simply truncating the values is performed to reduce that precision to 10 bits which are still in the YUV 4:2:0 format. From that point, x264-10bit happily accepts the YUV 4:2:0 10-bit input (from what I've read YUV 4:2:0 can contain 12 bits) to produce the final 10-bit encode. Or is there some kind of format conversion forced upon the values when the 16-bit values are generated? My concern is limiting or if possible eliminating any destructive color conversions in the chain.

[SAPikachu]

Released 1.4.1. Fixed YUY2 support.

[SAPikachu]

This was just the idea I had when I had discovered dither tools. It would seem to me that masktools would do the job quite nicely and could be used with f3kdb inside AviSynth without the need to code any kind of edge mask functionality into f3kdb itself. The only problem I can see though is that masktools would have to be modified/re-written with at least 16-bit internal precision, would it not? A minor detail.

Ideally it would be wonderful if we could get a full 16-bit pipeline established like so: QTGMC(dfttest, nnedi3, masktools) --> f3kdb --> 10-bit x264

I would love to contribute to this goal myself but while I know a little C++ and am studying it it will be a long time before I can do anything meaningful at that level.

Yes, it will better to do masking in masktools since that will be much more flexible. We really need a high bit-depth version of masktools...

I just recently began experimenting with 10-bit encoding. I'm using f3kdb and I think I understand what it's doing through the encoding stage but I was hoping that someone here could help by confirming if my understanding and approach is correct. At the moment I am working with interlaced Blu-ray material. My goal is to deband the source and then dither down from the 16-bit internal precision to 10-bits using f3kdb before sending the output through to x264-10bit. What I'm thinking is that sending the 16-bit results directly to x264-10bit would truncate the 16 bits to 10 while a much higher quality way to do it is to dither to 10 bits inside f3kdb first. Is this correct? My script is as follows:

LoadPlugin("DGAVCDecode.dll")
AVCSource(dga="00041.dga", deblock=false)
f3kdb(sample_mode=2, dynamic_dither_noise=true, precision_mode=2, output_depth=10, output_mode=2, keep_TV_range=true)



If your source is interlaced, I think you need to IVTC/deinterlace first. f3kdb should be put at the end of script.

In theory, dither to 10bit is better than directly feed 16bit to x264, but you need to use a hacked x264 to read non-8/16 bit source. You can use this version: http://j.mp/x264tMod , and specify --input-depth 10 in your command line.




Also, my understanding is that since the source is in YUV 4:2:0 the 16-bit processing in f3kdb is in the same format only with the values in higher decimal precision (internally). When the dithering is done something that goes beyond what simply truncating the values is performed to reduce that precision to 10 bits which are still in the YUV 4:2:0 format. From that point, x264-10bit happily accepts the YUV 4:2:0 10-bit input (from what I've read YUV 4:2:0 can contain 12 bits) to produce the final 10-bit encode. Or is there some kind of format conversion forced upon the values when the 16-bit values are generated? My concern is limiting or if possible eliminating any destructive color conversions in the chain.

From my understanding, 4:2:0 is a format of chroma-subsampling, it is not related to bit depth. You can check it here: http://en.wikipedia.org/wiki/Chroma_subsampling

[sneaker_ger]

I knew that cretindesalpes said that ordered dithering is better for compression, but I didn't expect that much of a difference. I debanded a test clip and then encoded to 8 bit x264 (2106, preset slower, tune animation, 2 pass 2079 kb/s):

source (floyd 8bit):
http://www.abload.de/img/8bit_floydsteinberg_smrulk.png

source (ordered 8bit):
http://www.abload.de/img/8bit_ordered_sourcevku98.png

source (no dither 8bit):
http://www.abload.de/img/8bit_no_dither_sourcet9u2z.png

f3kdb (floyd-steinberg 8bit):
http://www.abload.de/img/8bit_floydsteinberg1pudl.png

f3kdb (ordered 8bit):
http://www.abload.de/img/8bit_orderedyguhy.png

f3kdb (no dither 8bit):
http://www.abload.de/img/8bit_no_dithervquql.png

f3kdb (floyd-steinberg 16 bit, x264 8bit):
http://www.abload.de/img/16bit_floysteinberg_tcouu3.png

My conclusion: directly letting f3kdb dither to 8 bit with ordered dithering looks the best and not dithering looks very similar, followed by using 16 bit output and letting x264cli dither to 8 bit. f3kdb's 8 bit floyd-steinberg dithering gets totally destroyed at the encoding stage.
So for encoding I will from now on directly dither to the target bitdepth (be it 8 bit or 10 bit) using ordered dithering. I didn't see any problem before encoding between ordered and floyd-steinberg - or at least not in a still.
Any comments? Anyone want to share their findings?

Outside the competition:
10 bit x264/10 bit floyd-steinberg (http://www.abload.de/img/10bit_floydsteinberg7yu64.png)
8 bit x264 preset placebo/8-bit ordered (http://www.abload.de/img/8bit_ordered_placebofvuiy.png)


On a side node:
avs2yuv always crashes at the end of a script for me and avs2pipe(26)(mod) doesn't seem to support the high bit depth hack. Any alternatives? I'm using 32 bit AviSynth 2.6.0.

[Keiyakusha]

On a side node:
avs2yuv always crashes at the end of a script for me and avs2pipe(26)(mod) doesn't seem to support the high bit depth hack. Any alternatives? I'm using 32 bit AviSynth 2.6.0.

There was somewhere avs4x264mod_v0.2 on the forum. Only mod supports high bitdepth. It worked well for me

As for your results... it is clear to me that you say that ordered dithering is better, but not sure If i understand 2nd part.
directly letting f3kdb dither to 8 bit with ordered dithering looks the best
looks best compared to what? you seems to be not comparing f3kdb --> ordered 8bit output but encoded as 10bit.
10bit encode is always winner regardless of what your actual input is. Means it is good to use it with pure 8bit input.

[sneaker_ger]

looks best compared to what? you seems to be not comparing f3kdb --> ordered 8bit output but encoded as 10bit.

Except for the single 10 bit screenshot at the bottom of my post, all screenshots were done either directly from the source script (titled "source") or from the final 8 bit x264 encode (titled "f3kdb"). I didn't do any 10 bit x264 tests (again: except the one at the bottom, titled "outside the competition".)
It looks better compared to (1) using floyd-steinberg 8 bit dithering, i.e. precision_mode=3, output_depth=8, (2) using 16 bit floyd-steinberg and letting x264cli dither to 8 bit, i.e. precision_mode=3, output_depth=16 (3) using "no dithering" , i.e. precision_mode=1, output_depth=8

I hope it becomes clearer now. (Maybe I should rename the titles, as they are confusing.)

[sneaker_ger]

There was somewhere avs4x264mod_v0.2 on the forum. Only mod supports high bitdepth. It worked well for me

Thanks, that seems to work. It shows some warning ("avs [info]: Avisynth 2.6+ detected, forcing conversion to YV12"), but I guess that can be ignored in this context.

[Keiyakusha]

I hope it becomes clearer now. (Maybe I should rename the titles, as they are confusing.)
No it is ok. its just a language barrier. I agree, making direct dithering to desired bitdepth before encoding is better.

Thanks, that seems to work. It shows some warning ("avs [info]: Avisynth 2.6+ detected, forcing conversion to YV12"), but I guess that can be ignored in this context.

This is normal, x264 shows this warning for ages when it takes input from avisynth 2.6+. It is not because of avs4x264. Don't really remember now why it does so (because x264 hates possible YV24 as input?). Probably you didn't saw this warning before because avs2yuv makes x264 think it takes raw input and avs4x264 communicates with x264 in a bit "smarter" way so you don't need to specify resolution or framerate for example.

[SAPikachu]

Released 1.4.2, a bug fix build.

[SAPikachu]

On a side node:
avs2yuv always crashes at the end of a script for me and avs2pipe(26)(mod) doesn't seem to support the high bit depth hack. Any alternatives? I'm using 32 bit AviSynth 2.6.0.

You can also try patched x264 build, like http://j.mp/x264tMod , it can directly read interleaved high bit-depth clips without external tools.

[kemuri-_9]

This is normal, x264 shows this warning for ages when it takes input from avisynth 2.6+. It is not because of avs4x264. Don't really remember now why it does so (because x264 hates possible YV24 as input?). Probably you didn't saw this warning before because avs2yuv makes x264 think it takes raw input and avs4x264 communicates with x264 in a bit "smarter" way so you don't need to specify resolution or framerate for example.

That warning no longer occurs in the official x264 builds.
the original issue is because the new 2.6 colorspaces were designed a way that a 2.5-style isYV12() check would always be true.
so before I added 2.6 csp support in the C interface for x264, it needed to force a YV12 conversion because it had no way of telling if it was really a YV12 csp or some new 2.6 csp.

[Fullmetal Encoder]

That warning no longer occurs in the official x264 builds.
the original issue is because the new 2.6 colorspaces were designed a way that a 2.5-style isYV12() check would always be true.
so before I added 2.6 csp support in the C interface for x264, it needed to force a YV12 conversion because it had no way of telling if it was really a YV12 csp or some new 2.6 csp.

When was this warning removing? I have the latest 10-bit x264 and it will still give me this warning.

Also, can I confirm that it isn't making any changes to the input so long as the source starts as YUV 4:2:0? I believe I read this in the original thread where this issue was discussed but I'd like to make sure since it kind of freaks me out.

Thank you.

[Fullmetal Encoder]

Outside the competition:
10 bit x264/10 bit floyd-steinberg (http://www.abload.de/img/10bit_floydsteinberg7yu64.png)
8 bit x264 preset placebo/8-bit ordered (http://www.abload.de/img/8bit_ordered_placebofvuiy.png)


I am curious, for the 10-bit x264 floyd-steinberg image did you first dither to 10 bits with f3kdb and then send the 10-bit output to x264-10bit or did you have x264-10bit cut it down to 10 from 16? I am interested in comparing the difference between the two. I will upload my own comparisons when I get a chance.

[kolak]

Small bit out of topic- ffmpeg gets ProRes encoder with 10bit support :)

[sneaker_ger]

I am curious, for the 10-bit x264 floyd-steinberg image did you first dither to 10 bits with f3kdb and then send the 10-bit output to x264-10bit or did you have x264-10bit cut it down to 10 from 16?

The former.

[Nevilne]

Afaik vanilla x264 doesn't dither 16 to 10. You should passing 10bit avs directly to taro's x264 (which is correct way anyway). I think jeeb's builds support that as well?

[Fullmetal Encoder]

Afaik vanilla x264 doesn't dither 16 to 10.

This is what I thought. I expect x264-10bit to simply truncate the 16 bit values to 10. I would like to see how visible the difference will be.

[kemuri-_9]

When was this warning removing? I have the latest 10-bit x264 and it will still give me this warning.

that warning was removed in this commit (http://git.videolan.org/?p=x264.git;a=commit;h=1a6fd476c477a7870d15d0c7f005ae34813d2f9d), which is actually a long time ago now.

However, this does not necessarily cover any use of avs4x264 or other similar programs that are intended to pipe from 32bit avisynth to 64bit x264.
So if you're still getting the warning now, it's probably originating from that and not x264 itself...

Also, can I confirm that it isn't making any changes to the input so long as the source starts as YUV 4:2:0? I believe I read this in the original thread where this issue was discussed but I'd like to make sure since it kind of freaks me out.

We originally left this to avisynth, where ConvertToYV12() with a YV12 source is a No-Op...
Not sure if you could actually confirm it outside of modifying the source to print the AVS_VideoInfo's pixel_type before and after the ConvertToYV12 call to see that it didn't change.

Afaik vanilla x264 doesn't dither 16 to 10. You should passing 10bit avs directly to taro's x264 (which is correct way anyway). I think jeeb's builds support that as well?

what? if your input to x264 is generally >8 bit depth and it's compiled to output 10-bit files, then there will be a 16->10 dither.

[Nevilne]

Sorry if that was incorrect!

[SAPikachu]

what? if your input to x264 is generally >8 bit depth and it's compiled to output 10-bit files, then there will be a 16->10 dither.

The official x264 build will always do 16->10 dither indeed, but taro's build will skip this dithering when input and output bit-depth are the same.

[killazys]

A question about f3kdb(). If I want to make use of the dithering function only, would I run f3kdb(clip,0,0,0,0,output_mode=2,output_depth=16)? Would that work?

The docs say that "If set to 0, the corresponding plane will be untouched regardless of dither settings." for banding threshold. Does that mean it will not dither either?

[sneaker_ger]

There's f3kdb_dither() (see readme), but dithering is for reducing bitdepth, it does not really make sense to dither to 16 bit output.

[SAPikachu]

A question about f3kdb(). If I want to make use of the dithering function only, would I run f3kdb(clip,0,0,0,0,output_mode=2,output_depth=16)? Would that work?

The docs say that "If set to 0, the corresponding plane will be untouched regardless of dither settings." for banding threshold. Does that mean it will not dither either?

I think what you meant is bit-depth conversion? If input bit-depth != output bit-depth, the plane will always go through the filtering process. I will note that in the readme later. It's OK to set Y/Cb/Cr/ditherY/ditherC to all 0, f3kdb will only convert bit-depth with this parameter set.

BTW: You missed one "0" in your parameter list. ditherC is still the default value. :)

[sneaker_ger]

The official x264 build will always do 16->10 dither indeed, but taro's build will skip this dithering when input and output bit-depth are the same.

Are you sure this is true? In my test that I posted here about a month ago I used 10 bit output and I didn't use taro's build.

Btw, is there any reason for f3kdb_dither() being limited to 8 bit output depth?

[SAPikachu]

Are you sure this is true? In my test that I posted here about a month ago I used 10 bit output and I didn't use taro's build.

Btw, is there any reason for f3kdb_dither() being limited to 8 bit output depth?

Yes, at least at the time of writing, x264 will always convert high bit-depth clips to 16bit first, then dither down to 10bit. Taro's build can skip this process. It can also directly read high bit-depth avs script without avs4x264(mod).

Reason is simple - I was too lazy to add that, since it can now be done entirely in the main f3kdb filter. :P

[killazys]

Thanks SAPikachu! This will help me make quasi-improvements on sources that don't suffer from heavy banding :D

Edit: more like real improvements. Really decreases the whole "purple" issue.

[killazys]

Ah, sneaker_ger. I just read your comment. It actually helps a lot with "purple" color hues related to encoder compression. I can give you screenshot comparisons if you'd like.

Well... in my case, I did use the dither function. So in truth, I am unsure if disabling dither will also disable the purple addressing.

[sneaker_ger]

Yes, at least at the time of writing, x264 will always convert high bit-depth clips to 16bit first, then dither down to 10bit. Taro's build can skip this process. It can also directly read high bit-depth avs script without avs4x264(mod).

Found Taro's builds the other day and his website indeed explicitly mentions this capability - I would have never known. I wonder why the x264 dev's don't at least skip processing when input and output depth are 10 bit.

Ah, sneaker_ger. I just read your comment. It actually helps a lot with "purple" color hues related to encoder compression. I can give you screenshot comparisons if you'd like.

Well... in my case, I did use the dither function. So in truth, I am unsure if disabling dither will also disable the purple addressing.

Yes, I'd be interested in those screenshots, but what are your input and output/encoding bit depths? I don't really understand what f3kdb does when debanding is deactivated and input depth is smaller than output depth. My assumption would have been that it'd just do a skew operation, but from your discussion with SAPikachu this does not seem to be the case. Or do you first skew and then let x264 dither and the result would then be different from the original version?

[SAPikachu]

Yes, I'd be interested in those screenshots, but what are your input and output/encoding bit depths? I don't really understand what f3kdb does when debanding is deactivated and input depth is smaller than output depth. My assumption would have been that it'd just do a skew operation, but from your discussion with SAPikachu this does not seem to be the case. Or do you first skew and then let x264 dither and the result would then be different from the original version?

If output_depth > input_depth and deband-related parameters are all 0, pixels will be simply shifted to output bit-depth.

Thanks SAPikachu! This will help me make quasi-improvements on sources that don't suffer from heavy banding :D

Edit: more like real improvements. Really decreases the whole "purple" issue.

I don't quite understand, what's the "purple" issue?

[killazys]

Here's a screenshot comparison: http://screenshotcomparison.com/comparison/98877

Notice especially the coloring in the top right corner.

[SAPikachu]

I can see the difference. Did you capture the screenshot from 10bit video encoded by official x264 build?

[killazys]

No. I used JEEB's 2106 10-bit x64 build and captured the screenshots by using FFVideoSource(in.mkv) in VirtualDub (copy source to clipboard)

[SAPikachu]

Then I am not sure about the reason. There are too many factors that can cause this problem. Maybe you can try taro's build? I just checked one of my recent encode (8 bit input to taro's 10 bit x264), and didn't find this problem. You can see it here: http://screenshotcomparison.com/comparison/99003 (captured using FFMS2 in AvsP)

[sneaker_ger]

Here's a screenshot comparison: http://screenshotcomparison.com/comparison/98877

Notice especially the coloring in the top right corner.

1. Are these from the same frame?
2. TV or PC range encode?

[sneaker_ger]

Then I am not sure about the reason. There are too many factors that can cause this problem. Maybe you can try taro's build? I just checked one of my recent encode (8 bit input to taro's 10 bit x264), and didn't find this problem. You can see it here: http://screenshotcomparison.com/comparison/99003 (captured using FFMS2 in AvsP)

I think it's very hard to spot anyways, here's a comparison with blue and red colors and I honestly can't see any color skew:
http://screenshotcomparison.com/comparison/99027

It's really only visible with these kinda beige colors, like the one posted by killazys.

[killazys]

Hm. sneaker_ger, I am unsure as to how Coalgirls does its encodes. Those frames are close enough; I assure you the problem can be seen throughout in the grey scenes.

Granted, it's not very noticeable but I'm picky enough to want to dither up and back down to correct it.

And why are you comparing vanilla to JEEB's build?

[sneaker_ger]

And why are you comparing vanilla to JEEB's build?

JEEB's builds supposedly have the corrected TV range 8 bit -> 10 bit conversion, which did not find its way into the vanilla builds yet. I used 8 bit to feed x264cli for my screenshots.
http://x264.fushizen.eu/?p=381
http://forum.doom9.org/showthread.php?p=1517620#post1517620

[killazys]

Oh hm. I also use keep_tv_range in f3kdb if that matters.

[SAPikachu]

Oh hm. I also use keep_tv_range in f3kdb if that matters.

It shouldn't matter if you set ditherC/Cb/Cr to 0, and if your source is TV range (IIRC Madoka is correctly in TV range).

[killazys]

SAPikachu, for that Madoka encode I actually used
f3kdb(mt=true,range=31,Y=96,Cb=96,Cr=96,ditherY=16,ditherC=16,keep_tv_range=true,sample_mode=2,output_mode=2,output_depth=16)

So the color change might be because of the dithering, not the conversion to 16bit. Would that even be possible?

[SAPikachu]

Just thought about that, in theory it shouldn't affect the color in this way, since noises is added randomly. But I don't have time to test it now. Can you try setting ditherC/Cb/Cr to 0 and compare the result?

[killazys]

Sorry.. I don't have that raw anymore. After I finish some more encodes for my group I'll run a short comparison encode with another suitable raw. Likely some other madoka episode.

[Keiyakusha]

Hi, I want to ask something about true 16bit input.
Here (http://o.imgbox.com/aaiHN5lm.png) is how my source looks (yuv444p16le). And here (http://o.imgbox.com/aap666pT.png) is how it looks after I apply the following: f3kdb(input_mode=1,input_depth=16,output_mode=1,output_depth=16,keep_tv_range=true)

My question is... is everything is OK here? I mean, to me it looks like before there was some valuable information in LSB part, but after I applied debanding there is nothing but random noise. Is it not recommended to apply debanding on true 16bit? If so what if my source is mixed with true 16bit parts and 8bit parts that need to be debanded and these parts even can be mixed within the one frame?

[cretindesalpes]

Add ditherY=0 and ditherC=0 to avoid generating noise.

[Keiyakusha]

Thanks, now it seems to be alright. Here (http://o.imgbox.com/aawiEwnv.png) is what I'm getting.
However I'm afraid I have another question. If I want to get 10bit YV24 output I use this:
f3kdb(input_mode=1, input_depth=16, output_mode=1, output_depth=10)
And the output looks like this (http://o.imgbox.com/aasmd7sS.png). While I think the LSB looks really cool, MSB part looks kind of suspicious. Is everything ok with it?

BTW test image that I created you can get here (http://www.mediafire.com/?isvj65dx1qqrpor) (raw yuv444p16le stream)

[cretindesalpes]

This looks right. In 10 bits, the MSB values are confined to 0-3 for all planes, which is out of the 8-bit YUV range fitting in the RGB cube (esp. for chroma). Thus everything will look greenish.

[Keiyakusha]

Thank you! Now that I know everything works, I can think about rendering some of my projects in 16 bits. However I'm still searching of some way to open image sequence directly. Intermediate transcoding to yuv444p16le really eats my time and space. Its like 2.4 size of the source sequence (which is not that small by itself).

[SAPikachu]

Thank you! Now that I know everything works, I can think about rendering some of my projects in 16 bits. However I'm still searching of some way to open image sequence directly. Intermediate transcoding to yuv444p16le really eats my time and space. Its like 2.4 size of the source sequence (which is not that small by itself).

Which format is the image sequence? For regular format like PNG, JPG, BMP etc., I think you can just use ImageSource.

[Keiyakusha]

Which format is the image sequence? For regular format like PNG, JPG, BMP etc., I think you can just use ImageSource.

These are 16bit PNG or 16bit LZW-compresed tif... I think without any custom avisynth plugin they can't be loaded... Of course I want to keep them 16bit.
There is hacked version of ffms2 that can load 16bit stuff into avisynth but it doesn't supports 4:4:4 subsampling and i was able to open only 1 image, not sequence.

[TheFluff]

FFMS2 doesn't support image sequences, no. That hacked version most likely does the upper 8 bits wrong, too.

[SAPikachu]

These are 16bit PNG or 16bit LZW-compresed tif... I think without any custom avisynth plugin they can't be loaded... Of course I want to keep them 16bit.
There is hacked version of ffms2 that can load 16bit stuff into avisynth but it doesn't supports 4:4:4 subsampling and i was able to open only 1 image, not sequence.

Just had some research, DevIL actually supports 16 bits per channel internally, so we can build a hacked ImageSource to read 16 bit image sequence. If no one is interested to code it, I will make it when I get some time. :)

[Keiyakusha]

This would be so awesome to have this kind of plugin! Will look forward to it. In the meantime I'll try to convert my current project into yuv444p16le...
I usually work with image sequences because this is the most convenient format due to various reasons and it is easy to re-render some part of he project where I made some change. There can be some other options, especially on Mac (like Prores4444) but I'm a windows user and so far don't even see a way to render into raw 16bit stream directly... (the only real option is 4:2:2 10bit, but instead of it I better chose 4:4:4 8bit).

[SAPikachu]

Released 1.5.0. This version doesn't have any new feature, just changed some parameters to reduce confusion based on user feedback.

[sneaker_ger]

Thx for the new version. From the readme on dither_algo:
Use mode 1/2 for speed, or mode 3 for quality.

I don't agree with that. If you remember my small test I posted in this thread ordered dithering actually improves quality over Floyd-Steinberg when encoding lossy afterwards (which is probably what most people do with AviSynth), and I remember cretindesalpes posting a comment that says the same.

[SAPikachu]

Thx for the new version. From the readme on dither_algo:


I don't agree with that. If you remember my small test I posted in this thread ordered dithering actually improves quality over Floyd-Steinberg when encoding lossy afterwards (which is probably what most people do with AviSynth), and I remember cretindesalpes posting a comment that says the same.

Well, actually I totally forgot this part when updating the documentation. :P This should indeed be noted in the file. I will update it later today, thanks.

EDIT: Just updated the online documentation.

[sneaker_ger]

Can anyone give me an example of how to correctly pipe 10 bit 4:2:2 to x264-10bit?
I tried
ffvideosource("source.mkv")
ConvertToYUY2() # (or YV16)
f3kdb(output_mode=2, output_depth=10)

then:
avs2pipemod -rawvideo source.avs | x264-10bit - --demuxer raw --input-depth 10 --input-res 1280x720 --input-csp i422 (or YV16) --output-csp i422 -o output.h264

Either the complete picture or just the colors are messed up.

[SAPikachu]

Can anyone give me an example of how to correctly pipe 10 bit 4:2:2 to x264-10bit?
I tried
ffvideosource("source.mkv")
ConvertToYUY2() # (or YV16)
f3kdb(output_mode=2, output_depth=10)

then:
avs2pipemod -rawvideo source.avs | x264-10bit - --demuxer raw --input-depth 10 --input-res 1280x720 --input-csp i422 (or YV16) --output-csp i422 -o output.h264

Either the complete picture or just the colors are messed up.

You can try x264 tMod (http://www.mediafire.com/?118vb9r45qixy), it supports directly loading high bit-depth avs scripts. If only the colors are messed up, use YV16 and place SwapUV() before f3kdb may fix that.

[sneaker_ger]

Thx, swapping U and V in YV16 did the trick.

[Chyrka]

Please, tell me what is the correct way to convert avi with v210 in mov with v210? I've tried this in Premiere CS5, but after import in Avisynth i've got such picture:
http://i35.fastpic.ru/thumb/2012/0419/91/7150d3b01677d46fd967bbf134448e91.jpeg (http://fastpic.ru/view/35/2012/0419/7150d3b01677d46fd967bbf134448e91.png.html)
With my video i've got similar results.
Thank you.

[mp3dom]

You can use FFMBC or FFMPEG. It's probably even faster. Use -vcodec copy to do a direct stream without any re-compression.

[Chyrka]

mp3dom, thank you for the answer, but my problem remains. With correct offset i have good 1st frame (for 16-bit I disabled dithering):
http://i35.fastpic.ru/thumb/2012/0419/68/_579e67ab2809b75714a601a900765268.jpeg (http://fastpic.ru/view/35/2012/0419/_579e67ab2809b75714a601a900765268.png.html)
but other frames (both 8-bit and 16-bit) are messed up and shifted. 2-nd frame:
http://i37.fastpic.ru/thumb/2012/0419/62/_0305ec5071f0eed225bd85122f78f162.jpeg (http://fastpic.ru/view/37/2012/0419/_0305ec5071f0eed225bd85122f78f162.png.html)
:(
Maybe, I'm doing something wrong with readv210, but I cann't realize what. I guess, it's something with offset. :confused:
Script:
ReadV210("I:\Test\v210_720p.mov",36,1280,720)
My mov file: http://www.mediafire.com/?bshu20n0rf8quoj
In addition, I have not problem with coff3.mov from example. Why??? ((((

[mp3dom]

It's the AAC audio that is screwing something. Removing it resolved the issue.
ffbmc -i <file.mov> -vcodec copy -an <out.mov>

[Chyrka]

mp3dom, thank you so much. It works. :cool:

[Keiyakusha]

I have a question regarding readv210 too. What output should i expect see when i use it? For example if I use readv210("coff3.mov", 1103, 176, 144) or different command for opening my own file readv210("comp.mov", 48, 1280, 720) i see perfectly fine YV16 video... but i kind of expected to see stacked image... And mine is probably just MSB part. Any tips?
Edit: wow i found f3kdb_dither(stacked=true) in the script, after commenting it, it seems i'm getting what i expected... probably i need to sleep more...

[AiDz0r]

Why can't I use precision_mode?

[SAPikachu]

Why can't I use precision_mode?

precision_mode is renamed to dither_algo in 1.5.0.

[SAPikachu]

Released 2.0pre.

[sneaker_ger]

I get "Invalid plane requested" even on simple vs scripts:
import vapoursynth as vs
import sys

core = vs.Core()

core.std.LoadPlugin(path=r'C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll')
ret = core.ffms2.Source(source=r'c:\sample.mkv') #8 bit, YV12

core.std.LoadPlugin(path=r'c:\flash3kyuu_deband.dll')

ret = core.f3kdb.F3kdb(ret, output_depth=10)

ret.output(sys.stdout, y4m=True)

[SAPikachu]

I get "Invalid plane requested" even on simple vs scripts:
import vapoursynth as vs
import sys

core = vs.Core()

core.std.LoadPlugin(path=r'C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll')
ret = core.ffms2.Source(source=r'c:\sample.mkv') #8 bit, YV12

core.std.LoadPlugin(path=r'c:\flash3kyuu_deband.dll')

ret = core.f3kdb.F3kdb(ret, output_depth=10)

ret.output(sys.stdout, y4m=True)

I just confirmed again, it works in both AvsP and raw output for x264. y4m output also works for 8bit output but broken for 10bit output (shows "y4m [error]: bad header magic", not sure it is VS or x264's fault). Which version of VS and ffms2 are you using? Can you post the full error output?

[qyot27]

This is the commit where x264 gained the extended y4m support that VS and FFmpeg use. (http://git.videolan.org/?p=x264.git;a=commit;h=28ddb0dd533154b58f9147932fb1dec4c74127c8) It corresponds to r2226, but for practical reasons you'd probably be using either r2230 or r2216; r2216 and prior don't have that commit, r2230 does.



Personally, 2.0pre caused errors for me if it was in the autoload folder. AviSynth would fail to play the script - which was just a simple DirectShowSource test (but because of how autoload works, it loaded f3kdb even though I didn't have it in the script). If I tried wavi or FFmpeg, Windows itself actually threw up an error dialog: The procedure entry point FlsAlloc could not be located in the dynamic link library KERNEL32.dll.

Rolling back to 1.5.1, the issues disappear. The script can play, wavi and ffmpeg can open it and convert without issue. Checking what FlsAlloc is, it seems like it's probably not be available on XP. If I compile from git with VS2010 Express, it's fine, though.

Are there any plans to make f3kdb cross-platform like VapourSynth is?

[SAPikachu]

This is the commit where x264 gained the extended y4m support that VS and FFmpeg use. (http://git.videolan.org/?p=x264.git;a=commit;h=28ddb0dd533154b58f9147932fb1dec4c74127c8) It corresponds to r2226, but for practical reasons you'd probably be using either r2230 or r2216; r2216 and prior don't have that commit, r2230 does.


Thanks, I will try r2230 later today.


Personally, 2.0pre caused errors for me if it was in the autoload folder. AviSynth would fail to play the script - which was just a simple DirectShowSource test (but because of how autoload works, it loaded f3kdb even though I didn't have it in the script). If I tried wavi or FFmpeg, Windows itself actually threw up an error dialog: The procedure entry point FlsAlloc could not be located in the dynamic link library KERNEL32.dll.

Rolling back to 1.5.1, the issues disappear. The script can play, wavi and ffmpeg can open it and convert without issue. Checking what FlsAlloc is, it seems like it's probably not be available on XP. If I compile from git with VS2010 Express, it's fine, though.


I just found the culprit of the error, executables compiled by VS2012 does not work on XP by default. Will recompile an XP-compatible version after I installed a big patch for VS.


Are there any plans to make f3kdb cross-platform like VapourSynth is?

Yes, my next plan after finalizing this version is make it GCC-compilable. After that it is just a little more work to make it runnable on Linux.

[sneaker_ger]

I just confirmed again, it works in both AvsP and raw output for x264. y4m output also works for 8bit output but broken for 10bit output (shows "y4m [error]: bad header magic", not sure it is VS or x264's fault). Which version of VS and ffms2 are you using? Can you post the full error output?

I'm using VS R16 and tried the official ffms2-725-icl-4 build and ffms2-r734-libav-e6160bd + ffms2-r734-ffmbc-0.7rc7 from your signature. I don't think they're working incorrectly, as the script works as long as I outcomment the f3kdb line. As qyot implied I was already using x264cli 2230 with 10 bit y4m support. I tried 8 bit output anyways, but to no avail.

I even get the error message when I set y4m=False and redirect to NUL.

How do I get "the full error output"?

I'm on Win 7 x64, core i7-860

[SAPikachu]

I get "Invalid plane requested" even on simple vs scripts:
import vapoursynth as vs
import sys

core = vs.Core()

core.std.LoadPlugin(path=r'C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll')
ret = core.ffms2.Source(source=r'c:\sample.mkv') #8 bit, YV12

core.std.LoadPlugin(path=r'c:\flash3kyuu_deband.dll')

ret = core.f3kdb.F3kdb(ret, output_depth=10)

ret.output(sys.stdout, y4m=True)

Can you try again with pre2? I just found that I uploaded a buggy build as pre...

[sneaker_ger]

fixed

Btw, maybe you should call the function "Deband" instead of "F3kdb". Kinda redundant with the namespace already being "f3kdb" and would be more in line with the other filters.

[Bloax]

And if don't want to use a base-of-functionality name, then f3Deband would probably do. (And would be much easier to remember than f3kdb, which is kind of semi-random.)

[Keiyakusha]

I support sneaker_ger as there was dither too. (or this functionality wasn't ported?). Will be f3kdb.Deband and f3kdb.Dither.
Not sure what "base-of-functionality" means, but by all means do not change f3kdb to f3Deband or something, it is horrible. The name is "flash3kyuu", this is the only option if f3kdb is not good. Or something like f3k.Deband maybe?(don't think it is easier to remember though)

[06_taro]

+1 for f3k.Deband.

BTW, the f3kdb_dither for avs is already deprecated as it equals to f3kdb(Y=0, Cb=0, Cr=0, grainY=0, grainC=0, output_depth=8).

[SAPikachu]

I think f3kdb.Deband is better, since flash3kyuu has other aviutl plugins, if I used f3k, the namespace may conflict with plugins ported later. Any opinions?

f3kdb_dither will be kept in AviSynth interface for compatibility since the real work can be easily delegated to f3kdb core now. But I am still considering whether I should add a separate function in VS for it. Which one of the following do you like?


# Plan A
core.f3kdb.Deband(clip, preset="depth", output_depth=8)
# "preset=" may be omitted

# Plan B
core.f3kdb.Depth(clip, output_depth=8)



Personally I like plan A better, because plan B seems redundant, all works are done in the same place anyways.

[Keiyakusha]

I'm not sure how it works, but many internal filters use "std" many external "avs". Is it really a problem for different plugins to be "f3k"? For some reason I though this is OK... If not, then leave it f3kdb of course.

As for dither/depth, personally I'm ok with anything as long as it works.

[SAPikachu]

Internal filters and AviSynth compatibility layer are implemented in the core so they can share a same namespace, AFAIK other external plugins must all use different namespaces (correct me if I am wrong).

[sneaker_ger]

The download links appear to be down?

[SAPikachu]

Seems the server is down, please use the mirror in my signature for this moment.

[jackoneill]

Yes, my next plan after finalizing this version is make it GCC-compilable. After that it is just a little more work to make it runnable on Linux.

How is that coming along?

[SAPikachu]

How is that coming along?

I have been very busy recently and haven't touched the code for some time... Will continue on it after my current project is completed.

[Snowknight26]

YUY2 input seems to have an issue when using dither_algo=3.

Using this (http://stfcc.org/pics/i/3643363663c1946a94d5aadf66ff445c.jpg) as the source, the following produces a color shift vertically down the center of the image:
ImageSource("file.jpg")
ConvertToYUY2()
flash3kyuu_deband(dither_algo=3)
Example output: http://stfcc.org/pics/i/b06495640d07168aff65615139dd8064.png

If I change ConvertToYUY2 to ConvertToYV12 or dither_algo to 1/2, it's fine and looks like so:
http://stfcc.org/pics/i/9634210b51335e568a8b2b70be749827.png

Edit:
Seems like I was using the filter incorrectly? Just followed the example doing:
ImageSource("test.jpg",end=0)
ConvertToYV12()
f3kdb(dither_algo=3)
ConvertToRGB()
trim(0,0)
ImageWriter("C:\temp\out",type="png")
and it worked nicely (apart from the terrible chroma conversion).

What's the proper way of doing the color conversion? Using AviSynth 2.6.0 and ConvertToYV24 -> ConvertToRGB?

[SAPikachu]

YUY2 input seems to have an issue when using dither_algo=3.

Using this (http://stfcc.org/pics/i/3643363663c1946a94d5aadf66ff445c.jpg) as the source, the following produces a color shift vertically down the center of the image:
ImageSource("file.jpg")
ConvertToYUY2()
flash3kyuu_deband(dither_algo=3)
Example output: http://stfcc.org/pics/i/b06495640d07168aff65615139dd8064.png

If I change ConvertToYUY2 to ConvertToYV12 or dither_algo to 1/2, it's fine and looks like so:
http://stfcc.org/pics/i/9634210b51335e568a8b2b70be749827.png

Yes the YUY2 implementation may contain bugs since it is cumbersome to code for it and I can't write test case for it for some reasons. Actually I dropped YUY2 support in 2.0pre. You can convert to YV16 before processing, it is well-tested and possibly faster since YUY2 code is non-optimized.


Edit:
Seems like I was using the filter incorrectly? Just followed the example doing:
ImageSource("test.jpg",end=0)
ConvertToYV12()
f3kdb(dither_algo=3)
ConvertToRGB()
trim(0,0)
ImageWriter("C:\temp\out",type="png")
and it worked nicely (apart from the terrible chroma conversion).

What's the proper way of doing the color conversion? Using AviSynth 2.6.0 and ConvertToYV24 -> ConvertToRGB?

Well, if you want to process image you should really use YV24 to prevent chroma loss, and use Dither package (http://forum.doom9.org/showthread.php?p=1386559#post1386559) to do dithered colorspace conversion between YUV/RGB, that will give you better visual result than internal conversion filters.

[ryrynz]

Gave 2.0pre2 a shot today with 2.6MT and had my media player crash a few times and also an a bluescreen, have gone back to 1.5.1.

[SAPikachu]

Gave 2.0pre2 a shot today with 2.6MT and had my media player crash a few times and also an a bluescreen, have gone back to 1.5.1.

2.0 is an experimental version so it may have bugs that will crash the player, but in theory it shouldn't be able to cause BSOD... Anyway the core algorithm is not (yet) changed in 2.0, you can stay at 1.5.1 for now. I have been too busy to touch the code recently so it still needs some time for next version...

[ryrynz]

It doesn't appear that 2.0 is the issue, I've been using 1.5.1 in a script with other filters using MT mode 2 and TBH it's always been a little unstable but generally working fine. I've switched to the MSVC build and while it's early days, I haven't had a BSOD yet.

I wonder if there's a compiler option that will attain that same stability as the MSVC build but keep some performance advantage? I assume many won't experience any instability, I think it may depend a little on the filters being used in conjunction along with it in MT mode.

[SAPikachu]

It doesn't appear that 2.0 is the issue, I've been using 1.5.1 in a script with other filters using MT mode 2 and TBH it's always been a little unstable but generally working fine. I've switched to the MSVC build and while it's early days, I haven't had a BSOD yet.

I wonder if there's a compiler option that will attain that same stability as the MSVC build but keep some performance advantage? I assume many won't experience any instability, I think it may depend a little on the filters being used in conjunction along with it in MT mode.

Well, then it seems that the problem is in your hardware setup. Have you overclocked your CPU? All ICL does is optimize the assembly code so that CPU power can be better utilized. But that will cause your CPU become hotter than when you use the MSVC build. So if your CPU is overclocked it may be the culprit. If not overclocked, can you try limiting your CPU frequency (maybe switch to Power Saver mode) and see if the problem goes away?

[ryrynz]

Everything in the BIOS is stock. No memory errors, ran Lynx for 10 passes all okay, ran Prime95 blend mode with no issues so it's hard to say what's going on but my hardware appears stable.

I'm not fussed anyway, if the MSVC build fixes it then that's fine with me. It could be the smallest of instabilities somewhere in the hardware or maybe it's a conjunction of things that don't like
working together, either way nobody else is having any issues so I won't look into it any further at this point I just wanted to let you know at least it didn't appear to be a code issue with 2.0.

I'll look at it again some months from now when I upgrade to a Haswell based system, that should rule out the hardware at least if it still happens at that point.

[SAPikachu]

OK that's fine. Thanks for reporting anyways. :)

[turbojet]

ryrynz: What other filters are in the chain? No problem here with any of the f3kdb builds with mtmode2 but finesharp crashed quite often and occasional bsod with mtmode2, mtmode5 worked fine however.

SAPikachu: Do you know if f3kdb can be translated into a pixel shader? There isn't any very effective deband pixel shaders and could use some competition. f3kdb is the most effective debanders I've found. This would let me get ffdshow out of the playback chain.

[SAPikachu]

SAPikachu: Do you know if f3kdb can be translated into a pixel shader? There isn't any very effective deband pixel shaders and could use some competition. f3kdb is the most effective debanders I've found. This would let me get ffdshow out of the playback chain.

It may be possible, but I won't have time to do it in a few months. If someone is willing to write it I can provide some algorithm information though.

[ryrynz]

ryrynz: What other filters are in the chain? No problem here with any of the f3kdb builds with mtmode2 but finesharp crashed quite often and occasional bsod with mtmode2, mtmode5 worked fine however.

Psharpen, Hysteria, Vaguedenoiser, Awarpsharp2 and Flash3kyuu.

I'll look at the ICL build after my hardware update and troubleshoot it then. If the MSVC build does indeed fix it for me (for whatever reason) it I'm happy enough with continuing to use it.
I don't want to invest time trying to figure it out and posting further on it when when nobody else has any issues and it's solved by simply running a different compiled version, I can live with that for now.

A pixel shader would be very cool.

[travolter]

@SAPikachu thanks a lot!!!! the filter is really impressive and a daily use one. Perfect for realtime usage.

[ryrynz]

It definitely is the go to deband filter, I recommend playing with the settings a little to get it perfect to suit your media.

[x265]

Can any of you post the code for loading high-bitdepth avs script into x264tmod?

[SAPikachu]

Can any of you post the code for loading high-bitdepth avs script into x264tmod?

You need to specify output_mode=2 in parameters of f3kdb, that will make it output data that x264-tMod recognizes, like this:


f3kdb(... , output_mode=2, output_mode=10)

[x265]

Should i use the 10bit build within the FFmpeg folder?

[ryrynz]

Well, then it seems that the problem is in your hardware setup.

BIOS setting up improper RAM voltage. Back to ICL, all okay.

[SAPikachu]

Should i use the 10bit build within the FFmpeg folder?

Any 10bit tMod build should be OK, since they are all built with AVS support.

BIOS setting up improper RAM voltage. Back to ICL, all okay.

My guess was right. :)

[x265]

I'm getting this error

http://i50.tinypic.com/2vm7nup.png

AVS Script :

MPEG2Source("C:\Users\Abhijith Nair\Desktop\Death note vol1.d2v", cpu=0)
AnimeIVTC(mode=1, ifade=true, aa=0)
ClipTwo=trim(1918,30688)
ClipFour=trim(32368,32727)
ClipThree=MPEG2Source("C:\Users\Abhijith Nair\Desktop\ED.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
ClipOne=MPEG2Source("C:\Users\Abhijith Nair\Desktop\OP.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
AlignedSplice(ClipOne, ClipTwo, ClipThree, ClipFour)
Crop(2, 4, -2, -0)
Dehalo_alpha_mt(ss=3.0)
SMDegrain(lsb=true, refinemotion=true, pel=4, subpixel=3, tr=6, thSAD=300)
LSfmod(defaults="slow", strength=50)
flash3kyuu_deband(range=17, dither_algo=3, input_depth=8, output_depth=10, output_mode=2)


Sample :
http://www.mediafire.com/?y9vqkd98gku7hrs

[SAPikachu]

I'm getting this error

http://i50.tinypic.com/2vm7nup.png

AVS Script :

Well I made a mistake about any build will work. You should use x86 build.

[x265]

How do i confirm whether i should set tv range =true or false?

[SAPikachu]

How do i confirm whether i should set tv range =true or false?

You can use Histogram("levels") on your unfiltered source to check whether it is TV range. Be aware, I heard from my friends that some (badly-produced) anime have parts in TV range and parts in full range, so please check as many scene as you can.

[06_taro]

A simple function to scan the whole video and output a log file, in which out-of-tv-range frames are logged.

Function OutRange(clip c, float "thr", int "plane", string "log")
{
plane = Default(plane, 0)
thr = String(Default(thr, 0))
Yeval = "global ymax = YPlaneMax(" + thr + """)
global ymin = YPlaneMin(""" + thr + ")"
Ueval = "global umax = UPlaneMax(" + thr + """)
global umin = UPlaneMin(""" + thr + ")"
Veval = "global vmax = VPlaneMax(" + thr + """)
global vmin = VPlaneMin(""" + thr + ")"
expe = plane == 0 ? Yeval
\ : plane == 1 ? Ueval
\ : plane == 2 ? Veval
\ : plane == 3 ? Ueval + Chr(10) + Veval
\ : plane == 4 ? Yeval + Chr(10) + Ueval + Chr(10) + Veval
\ : Assert( False, """OutRange: "plane"(0~4) value error!""" )
expy = "ymax > 235 || ymin < 16"
expu = "umax > 240 || umin < 16"
expv = "vmax > 240 || vmin < 16"
expc = plane == 0 ? expy
\ : plane == 1 ? expu
\ : plane == 2 ? expv
\ : plane == 3 ? expu + " || " + expv
\ : plane == 4 ? expy + " || " + expu + " || " + expv
\ : Assert( False, """OutRange: "plane"(0~4) value error!""" )
wriy = """ "YMax = " + String(ymax) + "; YMin = " + String(ymin) """
wriu = """ "UMax = " + String(umax) + "; UMin = " + String(Umin) """
wriv = """ "VMax = " + String(vmax) + "; VMin = " + String(vmin) """
expw = plane == 0 ? wriy
\ : plane == 1 ? wriu
\ : plane == 2 ? wriv
\ : plane == 3 ? wriu + """ + "; " + """ + wriv
\ : plane == 4 ? wriy + """ + "; " + """ + wriu + """ + "; " + """ + wriv
\ : Assert( False, """OutRange: "plane"(0~4) value error!""" )

c
FrameEvaluate(expe)
WriteFileIf(Default(log, "OutRange.log"),
\ expc,
\ "current_frame", """ ": " """, expw)
}

thr: A percentage on how many percent of the pixels are allowed above or below minimum. Optional and defaults to 0.
plane: Which plane to check, same as mvtools 2 - 0:Luma; 1:Chroma U; 2:Chroma V; 3:Both chromas; 4:All. Optional and defaults to 0.
log: The filename of output log. Optional and defaults to "OutRange.log"

Example:
AVISource("source.avi")
OutRange(0, 0, log="sample.log")
use avs2avi/FFmpeg/x264 or anything to run it, and you'll find the log file named "sample.log".
Original frame is returned untouched, so you can add any other filters after it or even use it in encoding as only a check for assurance.
Scene-change not implemented, so only frame-based detection, no scene-based detection.

Or if you don't even want to manually override false detection, you can simply use ConditionalFilter to do frame level range-adaptive filtering:
AVISource("source.avi")
tv = f3kdb(keep_tv_range=True)
pc = f3kdb(keep_tv_range=False)
ConditionalFilter(pc, tv, "YPlaneMax > 235 || YPlaneMin < 16", "==", "True")

[x265]

http://www.mediafire.com/?y4zndt3cf34fm4ca

Could you analyze the log and tell me whether i should set tv-range=true or false?

Code:
mpeg2source("C:\Users\Abhijith Nair\Desktop\Anime\Vol1..d2v")
OutRange(log="sample.log")

[Gavino]

A simple function to scan the whole video and output a log file, in which out-of-tv-range frames are logged.
A neatly written function, especially the way the processing of plane is all done at compile-time (by building different expressions to be evaluated at run-time).

For information, a couple of small points on the following code:

Yeval = "global ymax = YPlaneMax(" + thr + """)
global ymin = YPlaneMin(""" + thr + ")"
Ueval = "global umax = UPlaneMax(" + thr + """)
global umin = UPlaneMin(""" + thr + ")"
Veval = "global vmax = VPlaneMax(" + thr + """)
global vmin = VPlaneMin(""" + thr + ")"
- The variables ymax, etc, don't actually need to be global, as they are only set and used at runtime, where they have script-level scope.
- Triple quotes are not needed in order to include a newline in a string (only needed when the string itself contains quote characters).

So the code could (if you like) be written simply as:

Yeval = "ymax = YPlaneMax(" + thr + ")
ymin = YPlaneMin(" + thr + ")"
Ueval = "umax = UPlaneMax(" + thr + ")
umin = UPlaneMin(" + thr + ")"
Veval = "vmax = VPlaneMax(" + thr + ")
vmin = VPlaneMin(" + thr + ")"
I suppose you could also do the plane range-check once at the start rather than repeating it each time plane is used.

[06_taro]

http://www.mediafire.com/?y4zndt3cf34fm4ca

Could you analyze the log and tell me whether i should set tv-range=true or false?

Code:
mpeg2source("C:\Users\Abhijith Nair\Desktop\Anime\Vol1..d2v")
OutRange(log="sample.log")

At lease those 0-255 frames are clearly PC range, if not badly edited animations I've seen somewhere with PC-range credit overlayed on TV-range background....

It is suggested to manually check those 14-236 frames or somewhat similar, as those range flows might be resulted from encoding artefacts but not editing, hence might still be TV-range. Anyway, in either cases, I couldn't say it need keep_tv_range=True, some flows from grain generator shouldn't be that harmful to a source which already contains over/under-flows.

A neatly written function, especially the way the processing of plane is all done at compile-time (by building different expressions to be evaluated at run-time).

For information, a couple of small points on the following code:

- The variables ymax, etc, don't actually need to be global, as they are only set and used at runtime, where they have script-level scope.
- Triple quotes are not needed in order to include a newline in a string (only needed when the string itself contains quote characters).

So the code could (if you like) be written simply as:

Yeval = "ymax = YPlaneMax(" + thr + ")
ymin = YPlaneMin(" + thr + ")"
Ueval = "umax = UPlaneMax(" + thr + ")
umin = UPlaneMin(" + thr + ")"
Veval = "vmax = VPlaneMax(" + thr + ")
vmin = VPlaneMin(" + thr + ")"
I suppose you could also do the plane range-check once at the start rather than repeating it each time plane is used.

Thanks for those suggestions. My quick written script without any double check suffered some dirty formats.:D

Updated here:
Function OutRange(clip c, float "thr", int "plane", string "log")
{
plane = Default(plane, 0)
thr = String(Default(thr, 0))
Assert( plane >= 0 && plane <= 4, """OutRange: "plane"(0~4) value error!""" )

Yeval = "ymax = YPlaneMax(" + thr + ")
ymin = YPlaneMin(" + thr + ")"
Ueval = "umax = UPlaneMax(" + thr + ")
umin = UPlaneMin(" + thr + ")"
Veval = "vmax = VPlaneMax(" + thr + ")
vmin = VPlaneMin(" + thr + ")"
expe = plane == 0 ? Yeval
\ : plane == 1 ? Ueval
\ : plane == 2 ? Veval
\ : plane == 3 ? Ueval + Chr(10) + Veval
\ : Yeval + Chr(10) + Ueval + Chr(10) + Veval

expy = "ymax > 235 || ymin < 16"
expu = "umax > 240 || umin < 16"
expv = "vmax > 240 || vmin < 16"
expc = plane == 0 ? expy
\ : plane == 1 ? expu
\ : plane == 2 ? expv
\ : plane == 3 ? expu + " || " + expv
\ : expy + " || " + expu + " || " + expv

wriy = """ "YMax = " + String(ymax) + "; YMin = " + String(ymin) """
wriu = """ "UMax = " + String(umax) + "; UMin = " + String(Umin) """
wriv = """ "VMax = " + String(vmax) + "; VMin = " + String(vmin) """
expw = plane == 0 ? wriy
\ : plane == 1 ? wriu
\ : plane == 2 ? wriv
\ : plane == 3 ? wriu + """ + "; " + """ + wriv
\ : wriy + """ + "; " + """ + wriu + """ + "; " + """ + wriv

c
FrameEvaluate(expe)
WriteFileIf(Default(log, "OutRange.log"),
\ expc,
\ "current_frame", """ ": " """, expw)
}


Actually I personally still prefer triple quotes with line wrapping, to keep a good habit for some other language users like python:)

[x265]

How do i find a good value for Y Cb Cr?

[SAPikachu]

How do i find a good value for Y Cb Cr?

You need to find a balance between detail retention and debanding effect by your eye. You can use Histogram("luma") to make it easier to check effect of the filter.

[Yellow_]

Did anyone work on a 16bit image sequence to Stacked LSB/MSB mentioned a few pages back? Hoping to find a method to import 16bit tif's and png's.

[sneaker_ger]

I tested 16 bit png with VapourSynth and it worked just fine. I used vsimagereader (http://forum.doom9.org/showthread.php?t=166088) for that - maybe you can poke Chikuzen about tiff support.

[Yellow_]

sneaker_ger, thanks for the reply, I'm happy with 16bit png so no probs. Currently using Avisynth with Dither Tools, http://forum.doom9.org/showthread.php?p=1632396#post1632396 but I should really get back into pursuing VapourSynth.

[jackoneill]

gcc support:
https://github.com/dubhater/flash3kyuu_deband/commits/gcc

I just hit it with a hammer until it worked, so many of those changes are probably Doing It Wrong™. I'm pretty sure I broke the SSE code paths with those "#if 1". The C code seems to work fine though. No idea if msvc or icl still compile it.

Criticism and suggestions welcome.

[SAPikachu]

gcc support:
https://github.com/dubhater/flash3kyuu_deband/commits/gcc

I just hit it with a hammer until it worked, so many of those changes are probably Doing It Wrong™. I'm pretty sure I broke the SSE code paths with those "#if 1". The C code seems to work fine though. No idea if msvc or icl still compile it.

Criticism and suggestions welcome.

Thanks! Just had a quick look, I think your aligned buffer declaration won't work with VS, IIRC __declspec(align(*)) must be put before type name.

Will look into it if I can get some free time in the coming weekend...

[jackoneill]

Thanks! Just had a quick look, I think your aligned buffer declaration won't work with VS, IIRC __declspec(align(*)) must be put before type name.

Will look into it if I can get some free time in the coming weekend...

This claims it's fine: http://stackoverflow.com/questions/7895869/cross-platform-alignx-macro/12654801#12654801

[SAPikachu]

This claims it's fine: http://stackoverflow.com/questions/7895869/cross-platform-alignx-macro/12654801#12654801

Maybe I was wrong then. Will confirm in VS later.

[sl1pkn07]

thanks for the GCC port!

[SAPikachu]

The master branch is now compilable by GCC 4.8.1 (Tested compilation under Ubuntu 13.04). Be warned that I haven't got time to actually try to run it yet, so it is likely to have bugs. The next step is port the unit test project to GCC too, to ensure the SSE optimization part is correct.

@jackoneill, thanks for your patch. Though I only used the patch for reference only, most fixes are rewritten to be better fit into the project.

To compile it, ensure GCC 4.8.1+ (we need g++ actually) and Python 3 is installed, then run:


./waf configure # If you need to specify where GCC is located at, use this: CXX=/path/to/your/g++-4.8 ./waf configure
./waf build

[jackoneill]

Cool!

Unfortunately, './waf build' fails here.

./waf configure -vvv: https://dpaste.de/ZKJW3/
./waf build -vvv: https://dpaste.de/QGEfR/
Maybe you can find something useful there.

I have gcc 4.8.1 and python 3.3.2.

[SAPikachu]

Cool!

Unfortunately, './waf build' fails here.

./waf configure -vvv: https://dpaste.de/ZKJW3/
./waf build -vvv: https://dpaste.de/QGEfR/
Maybe you can find something useful there.

I have gcc 4.8.1 and python 3.3.2.

Should be fixed in latest commit, can you try again?

[jackoneill]

That works.

Now it crashes in a most bizarre place, with a most bizarre error: https://dpaste.de/Sh7t0/

I call it like this:

c.f3kdb.Deband(ret, y=128)

Input is a 720×480 YUV420P8 clip from d2vsource.

[SAPikachu]

That works.

Now it crashes in a most bizarre place, with a most bizarre error: https://dpaste.de/Sh7t0/

I call it like this:

c.f3kdb.Deband(ret, y=128)

Input is a 720×480 YUV420P8 clip from d2vsource.

OK this needs some debugging.. Will check it when I got time.

EDIT: Fixed in latest commit.

[madshi]

I've been working on a debanding algorithm for madVR. I've taken the core algorithm idea of flash3kyuu_deband (which is really extremely simple), but modified it a bit. Wanted to let you know what I did, so you can implement the same thing in flash3kyuu_deband, if you like. Basically there's one point where the algorithm decides whether to use the original pixel value or the average of the 4 reference pixels. I've improved this decision making which allowed me to increase the thresholds a little bit. This results in stronger debanding, with hopefully not much more detail loss.

Of course the additional checks will eat up quite a bit of performance (and might be hard to implemented with SSE?), so I'm not sure if you want to do this, but that's your choice, of course. Some of the checks might be a bit redundant, I'm not sure. But I thought I'd rather add a few more checks to make sure the higher thresholds don't come with too many negative side effects...

Basically my decision making looks like this:

// orgPixel = original pixel value
// refPixel = one of the 4 reference pixels selected by the algorithm
// surPixel = one of the 8 pixels directly surrounding the original pixel
// refPixelsAvg = simple mean average of the 4 refPixels
// localContrast = max dif between the 8 surPixels and the orgPixel
// surroundContrast = max dif between the 4 * 9 surPixels left, top, right and bottom of the "localContrast" 9 pixel block
// maxRefPixelsDif = max dif between the 4 refPixels and the orgPixel
// refPixelsDifSum = sum of the absolute dif between each refPixel and the orgPixel

float3 result = ( (abs(refPixelsAvg - orgPixel) > 2.0 / 255.0) ||
(localContrast > 2.5 / 255.0) ||
(surroundContrast > 3.5 / 255.0) ||
(maxRefPixelsDif > 3.5 / 255.0) ||
(refPixelsDifSum > 6.5 / 255.0) ) ? orgPixel : refPixelsAvg;

// in pixel shaders one 8bit step is 1.0 / 255.0

Here's the end result, with no grain and no dithering as part of the debanding algorithm. madVR processes in high bitdepth and applies TPDF random dithering as a last step, anyway, so grain/dithering is not needed as part of the algorithm:

http://madshi.net/madVR/deband.png

[SAPikachu]

I've been working on a debanding algorithm for madVR. I've taken the core algorithm idea of flash3kyuu_deband (which is really extremely simple), but modified it a bit. Wanted to let you know what I did, so you can implement the same thing in flash3kyuu_deband, if you like. Basically there's one point where the algorithm decides whether to use the original pixel value or the average of the 4 reference pixels. I've improved this decision making which allowed me to increase the thresholds a little bit. This results in stronger debanding, with hopefully not much more detail loss.

Of course the additional checks will eat up quite a bit of performance (and might be hard to implemented with SSE?), so I'm not sure if you want to do this, but that's your choice, of course. Some of the checks might be a bit redundant, I'm not sure. But I thought I'd rather add a few more checks to make sure the higher thresholds don't come with too many negative side effects...

Basically my decision making looks like this:

// orgPixel = original pixel value
// refPixel = one of the 4 reference pixels selected by the algorithm
// surPixel = one of the 8 pixels directly surrounding the original pixel
// refPixelsAvg = simple mean average of the 4 refPixels
// localContrast = max dif between the 8 surPixels and the orgPixel
// surroundContrast = max dif between the 4 * 9 surPixels left, top, right and bottom of the "localContrast" 9 pixel block
// maxRefPixelsDif = max dif between the 4 refPixels and the orgPixel
// refPixelsDifSum = sum of the absolute dif between each refPixel and the orgPixel

float3 result = ( (abs(refPixelsAvg - orgPixel) > 2.0 / 255.0) ||
(localContrast > 2.5 / 255.0) ||
(surroundContrast > 3.5 / 255.0) ||
(maxRefPixelsDif > 3.5 / 255.0) ||
(refPixelsDifSum > 6.5 / 255.0) ) ? orgPixel : refPixelsAvg;

// in pixel shaders one 8bit step is 1.0 / 255.0

Here's the end result, with no grain and no dithering as part of the debanding algorithm. madVR processes in high bitdepth and applies TPDF random dithering as a last step, anyway, so grain/dithering is not needed as part of the algorithm:

http://madshi.net/madVR/deband.png

Thanks madshi, the result looks good to me. I will try to implement this when I have time later. In the meantime, can you explain a bit more about surroundContrast? I am not quite understand how to calculate it, does it work like http://imgur.com/M2JShFi ?

[madshi]

In the meantime, can you explain a bit more about surroundContrast? I am not quite understand how to calculate it, does it work like http://imgur.com/M2JShFi ?
Yes, almost like that. I've done this in 2 passes. In the first pass I'm calculating the "localContrast" for every pixel and store that in a helper texture/buffer. In the second pass I'm reading the "localContrast" for the pixels (x+3,y), (x-3,y), (x,y+3) and (x,y-3) and calculate the max of those 4 contrasts. The result is the "surroundContrast".

As I wrote before I'm not sure if we really need to do so many checks. Maybe the whole 2-pass thing and "surroundContrast" is overkill. Maybe you could skip the whole thing and get similar results. The key thing is to at least add *some* more checks because the default check is not good enough if you increase the thresholds (at least I thought so after testing that). In my first try I only checked "abs(refPixelsAvg - orgPixel)" and "maxRefPixelsDif" and that already worked quite nicely. But I found one case where detail suffered due to the increased thresholds, so I added some more checks to reduce the detail loss. I'm not even sure how much the added checks help, to be honest. Maybe you can try first without the "surroundContrast" check. Maybe that already works well enough...

[SAPikachu]

Yes, almost like that. I've done this in 2 passes. In the first pass I'm calculating the "localContrast" for every pixel and store that in a helper texture/buffer. In the second pass I'm reading the "localContrast" for the pixels (x+3,y), (x-3,y), (x,y+3) and (x,y-3) and calculate the max of those 4 contrasts. The result is the "surroundContrast".

As I wrote before I'm not sure if we really need to do so many checks. Maybe the whole 2-pass thing and "surroundContrast" is overkill. Maybe you could skip the whole thing and get similar results. The key thing is to at least add *some* more checks because the default check is not good enough if you increase the thresholds (at least I thought so after testing that). In my first try I only checked "abs(refPixelsAvg - orgPixel)" and "maxRefPixelsDif" and that already worked quite nicely. But I found one case where detail suffered due to the increased thresholds, so I added some more checks to reduce the detail loss. I'm not even sure how much the added checks help, to be honest. Maybe you can try first without the "surroundContrast" check. Maybe that already works well enough...

Thanks for your explanation. I think I will try the algorithm without "surroundContrast" first, since it seems this will have big performance impact, if the result is fine without this that will be great.

[turbojet]

Looking forward to it. If there's one thing I dislike about f3kdb is the added grain at times but I'll take it over the banding. It would be nice to see some comparison pics of real world source to see how much detail is lost or I could just wait.

Speaking of comparisons, it's unfortunate the second post of this thread hasn't been updated with pics of current f3kdb. Going by them f3kdb is the worst performer to me, which was once true and I abandoned it because of it but things changed at some point and now is on top imo.

[madshi]

I've done some more testing and I think we can forget the whole "surroundContrast" thing. It did reduce detail loss ever so slightly, but the difference was really small, and I think the performance loss isn't worth it. I've removed this check from madVR now, too.

If you want to check for detail loss, I can suggest this image (http://madshi.net/madVR/debandDetailTest.png). I've also created a small comparison image (http://madshi.net/madVR/debandComparison.png) which shows the difference between f3kdb(Y=128,Cb=128,Cr=128), GradFun3(smode=2,thr=0.7) and madVR. The debanding smoothness is comparable between all three with these settings. But f3kdb loses a lot of details in this case. With the improved checks madVR loses much less detail and is on a similar level as GradFun3 with these test images.

However, when using the default f3kdb and GradFun parameters, the extra checks I added don't seem to help much. With the default parameters, f3kdb and GradFun3 produce comparable results, but both leave quite a bit of banding in the test images. So the extra checks are mainly useful for users who want to use higher thresholds.

@turbojet, I'd recommend to turn on error diffusion (Floyd Steinberg) in f3kdb and to set grain to 0. IMHO when using error diffusion, there's no need to add grain, too.

[mandarinka]

I've done some more testing and I think we can forget the whole "surroundContrast" thing. It did reduce detail loss ever so slightly, but the difference was really small, and I think the performance loss isn't worth it.

Well, for encoding-time filtering, it might still make sense, because performance cost might not be a problem for many users and those details can be precious.

[turbojet]

madshi: Thanks for the comparison, I'm mainly paying attention to the real life source as I rarely see banding half as bad as the animated pic. Madvr definitely looks good in for both. So does gradfun3 but in realtime, gradfun3 gives me some nasty edge artifacts, less so with smode=2 but still very visible. Sharpening may have something to do with this but f3kdb and no debanding doesn't show the issue. My cpu can't handle smode=2 in realtime on HD sources, SD is fine however. A few questions:

1, Is that an encode or realtime comparison?
2. Where does the 64 and 128 come from in f3kdb? The first param, range, max value is 31 and I couldn't find anything besides grainc/grainy that allows it but the comparison doesn't look grainier but instead stronger or blurred.

As for the unwanted grain at times, setting grainc=0, grainy=0 definitely resolves the issue but also reveals blocks and ringing the grain was masking. Reducing grain from 64 to 16 seems to have mostly fixed the issue without the side effect and also changing random algo from uniform to gaussian made a positive impact. This is what I settled with on the movie I'm currently watching, an older, grainy source BD: f3kdb(grainy=8, grainc=8,blur_first=false,random_algo_ref=2,random_algo_grain=2). Has anyone played with the grain parameters?

[madshi]

Well, for encoding-time filtering, it might still make sense, because performance cost might not be a problem for many users and those details can be precious.
Well, we're talking about *significantly* slower speed for a benefit which is barely noticeable (even when pixel peeping), so I don't think it's worth it, even for encoding-time filtering. But of course that's only my personal opinion. I don't mind if SAPikachu implements it.

Yesterday I also played with trying to analyze & compare the gradiant direction/angles, as another check to reduce detail loss. The speed penalty of doing this is even worse, and the benefit once more barely visible, so at some point I stopped and gave up.

So does gradfun3 but in realtime, gradfun3 gives me some nasty edge artifacts, less so with smode=2 but still very visible. Sharpening may have something to do with this but f3kdb and no debanding doesn't show the issue. My cpu can't handle smode=2 in realtime on HD sources, SD is fine however.
Ah, ok. Didn't see such edge artifacts with GradFun3 here, but I've only worked with a limited set of samples, and didn't sharpen afterwards.

1, Is that an encode or realtime comparison?
I've tested on still images, so I don't have to frame step to a suitable frame all the time. But what I can say is that madVR's algorithm consumes about 15ms render time per 1080p frame on my Intel HD4000. So no problem for 24p, but probably a bit too slow for 60p with the HD4000. Hmmmm... Maybe I can remove the extra checks for the "low" setting (there will be a low and a high setting in madVR), to make it 60p capable, I'll give that a triy... I haven't tested f3kdb and GradFun3 for speed, only for quality.

2. Where does the 64 and 128 come from in f3kdb?
Sorry for being unclear, that's the value for the three "Y, Cb, Cr" thresholds. Increasing these thresholds in f3kdb improves banding removal strength, but also reduces detail and if you go too high, it introduces artifacts around edges. The detail loss is noticeably reduced by the checks I've added and the edge artifacts are totally gone.

As for the unwanted grain at times, setting grainc=0, grainy=0 definitely resolves the issue but also reveals blocks and ringing the grain was masking.
Well, f3kdb is meant to reduce banding, not ringing. But then, the madVR dithering is somewhat similar to the grain which f3kdb adds. So when using madVR the end result should be roughly comparable to using f3kdb with grain.

changing random algo from uniform to gaussian made a positive impact.
Hmmmm... Not sure right now, does the guassian change only improve the grain, or does it also help even if you turn grain off in f3kdb?

[SAPikachu]

Well, we're talking about *significantly* slower speed for a benefit which is barely noticeable (even when pixel peeping), so I don't think it's worth it, even for encoding-time filtering. But of course that's only my personal opinion. I don't mind if SAPikachu implements it.

Yesterday I also played with trying to analyze & compare the gradiant direction/angles, as another check to reduce detail loss. The speed penalty of doing this is even worse, and the benefit once more barely visible, so at some point I stopped and gave up.


I think I will make it togglable and let user choose.


Hmmmm... Not sure right now, does the guassian change only improve the grain, or does it also help even if you turn grain off in f3kdb?

Just FYI, random_algo_grain controls grain distribution, it has no effect when grainy/c is turned off. On the other hand random_algo_ref affects reference pixel selection and always have effect. @turbojet, which one did you mean?

[turbojet]

I had madvr dither off last night, enabling it definitely changed things. I ran across a scene (http://www.sendspace.com/file/7s9qsj) last night from an old SD xvid encode (where banding is most prevelant) that didn't look good at all at with previously mentioned settings and enabling dither didn't help much. It was improved by either increasing grain or increasing strength, I prefer the former on this particular scene but prefer latter on other scenes, defaults are really effective on this one. Is there a way to use a lot more grain on flat surfaces, like walls, then the everything else?

Both reference and grain were changed to guassian but the former made the bigger impact on the old BD. On this other scene, it doesn't make much of a difference. Not sure what guassian does, the math is well above me, but changing lumasharpen to it made a noticeable improvement and removed a uniform pattern I was starting to notice often. So I decided to try it in f3kdb, a filter I had been meaning to tweak for over a year and finally found a good reason to.

E: Does random_algo_grain matter when dynamic_grain=false (default)?

[madshi]

Do you have a sample where the difference between gaussian "reference" and default is clearly visible? I'd be interested in testing that...

[SAPikachu]

I had madvr dither off last night, enabling it definitely changed things. I ran across a scene (http://www.sendspace.com/file/7s9qsj) last night from an old SD xvid encode (where banding is most prevelant) that didn't look good at all at with previously mentioned settings and enabling dither didn't help much. It was improved by either increasing grain or increasing strength, I prefer the former on this particular scene but prefer latter on other scenes, defaults are really effective on this one. Is there a way to use a lot more grain on flat surfaces, like walls, then the everything else?


Not possible with f3kdb (at least for now), I think you may try masktools to filter details and flat area separately, but performance may suffer...


E: Does random_algo_grain matter when dynamic_grain=false (default)?

Yes, dynamic_grain just controls whether noise pattern is different for each frame, noise pattern is generated in the same way so it will be affected by random_algo_grain too.

[turbojet]

madshi: Sorry, I'm having some problems concentrating lately, too much stress/things on my mind. I did some comparing of the 2 algo's and on clean HD sources their is very little difference but when going SD -> HD there's a lot more grain with uniform, could be either negative or positive, here's a photo comparison (http://screenshotcomparison.com/comparison/43201) and video (http://www.sendspace.com/file/n8a6ps). Script's used were f3kdb() and f3kdb(random_algo_ref=2,random_algo_grain=2) Concerning the edge artifacts with gradfun3(smode=0) it's pretty noticeable on SD -> HD but otherwise it's not, smode=3 is just as fast with fewer edge artifacts. A few questions:

1. What was the percentage gain on gpu (from 10 to 40% being 400%) of madvr's debanding on HD4000?
2. Does it require a delay to keep a/v sync like avisynth does? This is an issue with live tv.

SAPikachu: Thanks for the tip on masktools I haven't tried it yet but will soon to see if it's usable in realtime.

[madshi]

I did some comparing of the 2 algo's and on clean HD sources their is very little difference but when going SD -> HD there's a lot more grain with uniform, could be either negative or positive, here's a photo comparison (http://screenshotcomparison.com/comparison/43201) and video (http://www.sendspace.com/file/n8a6ps). Script's used were f3kdb() and f3kdb(random_algo_ref=2,random_algo_grain=2)
Hmmmm... That's quite interesting. Will have a look at this later...

1. What was the percentage gain on gpu (from 10 to 40% being 400%) of madvr's debanding on HD4000?
Gain compared to what? CPU you mean? I haven't done any speed comparisons. My current development PC happens to have a quite fast quad-core CPU but a quite slow GPU. So maybe f3kdb() via AviSynth might even have been even faster on my PC than running it on the GPU. But that's not the point of madVR. GPU speed seems to climb much faster than CPU speed, so it makes sense to run everything on the GPU. And upgrading the GPU to a much faster model is easier than upgrading your CPU to a much faster model. I haven't compared CPU vs GPU. I can only tell you that my HD4000 needs 7ms per 1080p frame for debanding in "low" setting (comparable to f3kdb() default parameters). Of course with a fast GPU the time will be much lower. Probably with e.g. a GeForce 660 (which people often recommend for madVR) it would be under 1ms per 1080p frame. And for SD frames the time will be much lower still.

2. Does it require a delay to keep a/v sync like avisynth does? This is an issue with live tv.
No, of course not. No madVR algorithm requires a delay, or ever will require a delay. That said, I don't know what happens with live TV if you enable the "delay playback start until all queues are full" madVR setting. Might be that playback is delayed a little then, but that's an optional feature, and off by default.

[turbojet]

Hmmmm... That's quite interesting. Will have a look at this later...

On that same sample, range=31 masks a vertical strip of visible macroblocks much more than 16 (default) it's to the right of the picture and above the paper. dynamic_grain=true makes the area above the paper on the right more dynamic which is less distracting to my eyes in this case. Haven't compared these to defaults on other samples but have watched a couple HD and SD shows with them. Other options that I plan to experiment with is seed and random_param_ref / random_param_grain when I get some time. Can someone simply explain these options beyond the txt?

Gain compared to what? CPU you mean? I haven't done any speed comparisons. My current development PC happens to have a quite fast quad-core CPU but a quite slow GPU. So maybe f3kdb() via AviSynth might even have been even faster on my PC than running it on the GPU. But that's not the point of madVR. GPU speed seems to climb much faster than CPU speed, so it makes sense to run everything on the GPU. And upgrading the GPU to a much faster model is easier than upgrading your CPU to a much faster model. I haven't compared CPU vs GPU. I can only tell you that my HD4000 needs 7ms per 1080p frame for debanding in "low" setting (comparable to f3kdb() default parameters). Of course with a fast GPU the time will be much lower. Probably with e.g. a GeForce 660 (which people often recommend for madVR) it would be under 1ms per 1080p frame. And for SD frames the time will be much lower still.

I meant gpu usage with madvr's deband disabled to when it's enabled. Presentation time increase would vary a lot depending on gpu and settings wouldn't it? While relative percentage shouldn't vary much or maybe it would. I agree mostly on gpu but there are situations when offloading option to cpu would be beneficial, 48/50/60/120 fps, igp only htpc, etc. but I'm not asking for it.

No, of course not. No madVR algorithm requires a delay, or ever will require a delay. That said, I don't know what happens with live TV if you enable the "delay playback start until all queues are full" madVR setting. Might be that playback is delayed a little then, but that's an optional feature, and off by default.

That's good to know, 720i/p MPEG2 at 6-7 mbps could really use debanding but that's about the worse case scenario.

[SAPikachu]

Ok, after a lot of user testing and fine tuning, here's the "final" version of the algorithm. There have been a number of changes.

First some definitions:

// orgPixel = original pixel value
// refPixel = one of the 4 reference pixels selected by the algorithm
// refPixelPair = 2 refPixels build a pair; if you connect the two refPixels of a pair with a straight line, the orgPixel lies right in the middle of the line
// avgDif = simple mean average of all 4 refPixels
// maxDif = max dif between the 4 refPixels and the orgPixel
// midDif = dif between the orgPixel value and the average value of a refPixelPair

Now we have 3 thresholds we can check: "avgDif", "maxDif" and "midDif", where "midDif" must be checked twice, once for each "refPixelPair". The logical purpose of the "midDif" is that debanding should really be done mostly for gradients. If you look at a refPixelPair, the orgPixel should have a value which is between the two values of the refPixelPair. If it's outside, it's less likely to be a banded pixel, but instead it's more likely to be e.g. a highlight. Because of this the "midDif" check is a useful check to separate detail from banding.

Instead of doing a simple binary check for each threshold I'm now doing "fuzzy logic". Here's the code I'm using:

float3 avg = (pix1 + pix2 + pix3 + pix4) / 4.0;
float3 avgDif = abs(avg - orgPix);
float3 maxDif = max(abs(pix1 - orgPix), max(abs(pix2 - orgPix), max(abs(pix3 - orgPix), abs(pix4 - orgPix))));
float3 midDif1 = abs(pix1 + pix3 - 2 * orgPix);
float3 midDif2 = abs(pix2 + pix4 - 2 * orgPix);
float3 factor = pow(saturate(3.0 * (1.0 - avgDif / threshAvgDif)) * // "saturate" clips to 0..1 range
saturate(3.0 * (1.0 - maxDif / threshMaxDif)) *
saturate(3.0 * (1.0 - midDif1 / threshMidDif)) *
saturate(3.0 * (1.0 - midDif2 / threshMidDif)), 0.1);
result = orgPix + (avg - orgPix) * factor;

// in pixel shaders one 8bit step is 1.0 / 255.0
And because this is not complicated enough yet, I've added two more tricks:

(1) gradient angle check

This is quite complicated and requires an extra algorithm step - resulting in a noticable performance drop. But it does result in a nice improvement in the "debanding smoothness vs detail loss" ratio. We try to estimate the gradient angle for every pixel. This is done by reading the 8 surround pixels and doing some math on them - but instead of reading the directly surrounding pixels I'm using a read distance of 20 pixels (!!). This is done to give us a better estimate of the real gradient angle. Here's the code:

float3 p00 = tex2Dlod(SourceSampler, float4(Tex.x - xPixSize * 20, Tex.y - yPixSize * 20, 0, 0));
float3 p10 = tex2Dlod(SourceSampler, float4(Tex.x, Tex.y - yPixSize * 20, 0, 0));
float3 p20 = tex2Dlod(SourceSampler, float4(Tex.x + xPixSize * 20, Tex.y - yPixSize * 20, 0, 0));
float3 p01 = tex2Dlod(SourceSampler, float4(Tex.x - xPixSize * 20, Tex.y, 0, 0));
float3 p21 = tex2Dlod(SourceSampler, float4(Tex.x + xPixSize * 20, Tex.y, 0, 0));
float3 p02 = tex2Dlod(SourceSampler, float4(Tex.x - xPixSize * 20, Tex.y + yPixSize * 20, 0, 0));
float3 p12 = tex2Dlod(SourceSampler, float4(Tex.x, Tex.y + yPixSize * 20, 0, 0));
float3 p22 = tex2Dlod(SourceSampler, float4(Tex.x + xPixSize * 20, Tex.y + yPixSize * 20, 0, 0));
float3 gx = (p20 + p21 * 2.0 + p22) - (p00 + p01 * 2.0 + p02);
float3 gy = (p00 + p10 * 2.0 + p20) - (p02 + p12 * 2.0 + p22);
float3 angle = (abs(gx) < 0.01 / 255.0) ? 1.0 : (atan(gy / gx) / PI + 0.5);
The output will be a value between 0.0 and 1.0, telling us the estimated gradient angle of the pixel. Now in the main algorithm step for each "refPixel" we now also read the corresponding gradient angle. Then we compare the estimated gradient angle of each "refPixel" to the estimated gradient angle of the "orgPixel". If the max difference of those 4 "refPixel" angles does not exceed a specific limit (new threshold named "maxAngle") then we apply a boost to the debanding strength. This results in a nice debanding smoothness improvement without harming detail (in most cases). We apply the boost simply by increasing the other 3 thresholds (avgDif, maxDif, midDif) by a specified factor. This factor I'm calling "angleBoost" in madVR.

(2) fade in/out detection

In my experience, banding artifacts are especially annoying when there's a fade to/from white or black, because in such situations the bands start to move around, creating false image edges, which I find extremely distracting. Because of that I've added a detection for when there's a fade from/to white or black. During the fade I'm using a stronger debanding strength (higher thresholds). If you need more detail on how I'm detecting a fade in/out, let me know...

The biggest problem for me is to find good values for all those new thresholds (avgDif, maxDif, midDif, angleBoost and maxAngle). I'm still in the process of researching this, with the help of some madVR users. Here's what I'm currently using, but this is subject to change:

low: avgDif = 0.6 / 255; maxDif = 1.9 / 255; midDif = 1.2 / 255; angleBoost = 1.9; maxAngle = 10;
medium: avgDif = 1.8 / 255; maxDif = 4.0 / 255; midDif = 2.0 / 255; angleBoost = 1.6; maxAngle = 22;
high: avgDif = 3.4 / 255; maxDif = 6.8 / 255; midDif = 3.3 / 255; angleBoost = off; maxAngle = off;
The medium preset is roughly comparable to the current default f3kdb() settings, with maybe slightly higher debanding quality and slightly higher detail preservation. But of course this differs, depending on which exact video you test with. Sometimes the new thresholds not only preserve more detail, but also more banding. The default madVR configuration is currently "low" for normal scenes and "high" when there's a fade in/out. But none of these thresholds are set in stone yet. The algorithm itself is not likely to change, anymore, now, I hope.

Thanks very much madshi. I will look into it as soon as I have time (but this won't happen fast because I have been really busy recently..).

[vood007]

Is there anything i can do to speed up initial flash3kyuu_deband.dll loading? I use it for realtime watching in MPCHC/FFDShow but the additional 2 sec delay when opening a video almost kills the fun. Tried loading the dll from ramdisk but did not help at all. Any suggestions?

[sneaker_ger]

Not trying to keep SAPikachu from answering, but did you try out the current madVR test build (http://madshi.net/madVRanotherTestBuild2.rar)? It features debanding based on flash3kyuu's algorithm. You can find the option in the settings, "processing">"artifact removal".

[turbojet]

Try lowering the buffer, you shouldn't need to keep any past frames and I was able to get away with 3 future frames, any less it would drift out of sync. So 0 3 in ffdshow.

[SAPikachu]

f3kdb itself shouldn't take 2 secs to load (in my test it took less than 1 sec for 1920x1080 video), so I guess you need to tweak some of your ffdshow settings.

[vood007]

Dont know where i could tweak ffdshow in this regard since its perfectly quick when i take out f3kdb from the script. Well looks like i have to live with it, thanks anyway.

[kabal223]

Hi!

Im sorry for my noob question, but I need help. I want to use this in real time with ffdshow but I dont have a clue of how to do that.

Please help, is there a tutorial anywere? thanks!

[ryrynz]

Since you're new to using Avisynth and ffdshow, consider using MadVR's deband feature which is based off of f3kdb if you have a semi decent GPU. At this point I think it's generally accepted that it offers better debanding.

[Sparktank]

debanding algorithm for madVRI will try to implement this when I have time later.

Any update on this update? :o

[SAPikachu]

Any update on this update? :o

I am too busy on my work and other projects recently, so this probably won't happen for some time, sorry.. Though if anyone can send me pull requests or patches, I can merge it into the main repository.

[MonoS]

I'm having problem compiling this plugin at 64bit on Windows using mingw64 and Msys 2.

I start waf using this
./waf configure --static --no-shared --disable-avs --mode=release
Then get this error
Setting top to : /f3kdb
Setting out to : /f3kdb/build
Shared library : no
Static library : yes
Checking for 'g++' (c++ compiler) : /mingw64/bin/g++
Checking for linker flags -Wl,-Bsymbolic : no
Checking for linker flags -Wl,-z,noexecstack : no
- alignas : no

Here's the log
# project f3kdb (2.0pre) configured on Sun Mar 8 22:16:01 2015 by
# waf 1.7.15 (abi 98, python 30303f0 on msys)
# using ./waf configure --static --no-shared --disable-avs --mode=release
#
----------------------------------------
Setting top to
/f3kdb
----------------------------------------
Setting out to
/f3kdb/build
----------------------------------------
Shared library
no
----------------------------------------
Static library
yes
----------------------------------------
Checking for 'g++' (c++ compiler)
find program=['g++', 'c++'] paths=['/mingw64/bin', '/usr/local/bin', '/usr/bin', '/bin', '/mingw32', '/c/ProgramData/Oracle/Java/javapath', '/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common', '/c/Windows/system32', '/c/Windows', '/c/Windows/System32/Wbem', '/c/Windows/System32/WindowsPowerShell/v1.0', '/c/Program Files (x86)/ATI Technologies/ATI.ACE/Core-Static', '/c/Program Files (x86)/Common Files/Roxio Shared/9.0/DLLShared', '/c/Program Files (x86)/QT Lite/QTSystem', '/c/Program Files (x86)/AMD/ATI.ACE/Core-Static', '/c/Program Files (x86)/CMake/bin', '/usr/bin/site_perl', '/usr/bin/vendor_perl', '/usr/bin/core_perl'] var='CXX' -> '/mingw64/bin/g++'
find program=['ar'] paths=['/mingw64/bin', '/usr/local/bin', '/usr/bin', '/bin', '/mingw32', '/c/ProgramData/Oracle/Java/javapath', '/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common', '/c/Windows/system32', '/c/Windows', '/c/Windows/System32/Wbem', '/c/Windows/System32/WindowsPowerShell/v1.0', '/c/Program Files (x86)/ATI Technologies/ATI.ACE/Core-Static', '/c/Program Files (x86)/Common Files/Roxio Shared/9.0/DLLShared', '/c/Program Files (x86)/QT Lite/QTSystem', '/c/Program Files (x86)/AMD/ATI.ACE/Core-Static', '/c/Program Files (x86)/CMake/bin', '/usr/bin/site_perl', '/usr/bin/vendor_perl', '/usr/bin/core_perl'] var='AR' -> '/mingw64/bin/ar'
/mingw64/bin/g++
----------------------------------------
Checking for linker flags -Wl,-Bsymbolic
==>

int main(int argc, char **argv) {
(void)argc; (void)argv;
return 0;
}

<==
[1/2] cxx: build/.conf_check_73cb2ad4dc7ec5ddbb2d8468622d975a/test.cpp -> build/.conf_check_73cb2ad4dc7ec5ddbb2d8468622d975a/testbuild/test.cpp.1.o

['/mingw64/bin/g++', '-fPIC', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fvisibility=hidden', '-fvisibility-inlines-hidden', '-Werror', '-std=c++11', '../test.cpp', '-c', '-o', 'test.cpp.1.o']
err: ../test.cpp:1:0: error: -fPIC ignored for target (all code is position independent) [-Werror]

^
cc1plus.exe: all warnings being treated as errors

from /f3kdb: Test does not build: Traceback (most recent call last):
File "/f3kdb/.waf3-1.7.15-9c6c439a6416a92b3e844736c4ef3c7b/waflib/Tools/c_config.py", line 458, in run_c_code
bld.compile()
File "/f3kdb/.waf3-1.7.15-9c6c439a6416a92b3e844736c4ef3c7b/waflib/Build.py", line 188, in compile
raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
-> task in 'testprog' failed (exit status 1):
{task 7696569521744: cxx test.cpp -> test.cpp.1.o}
['/mingw64/bin/g++', '-fPIC', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fvisibility=hidden', '-fvisibility-inlines-hidden', '-Werror', '-std=c++11', '../test.cpp', '-c', '-o', 'test.cpp.1.o']

no
from /f3kdb: The configuration failed
--------------------------------------------
Checking for linker flags -Wl,-z,noexecstack
==>

int main(int argc, char **argv) {
(void)argc; (void)argv;
return 0;
}

<==
[1/2] cxx: build/.conf_check_1fb68c99b0b8fa44c9dc9af28b04c8de/test.cpp -> build/.conf_check_1fb68c99b0b8fa44c9dc9af28b04c8de/testbuild/test.cpp.1.o

['/mingw64/bin/g++', '-fPIC', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fvisibility=hidden', '-fvisibility-inlines-hidden', '-Werror', '-std=c++11', '../test.cpp', '-c', '-o', 'test.cpp.1.o']
err: ../test.cpp:1:0: error: -fPIC ignored for target (all code is position independent) [-Werror]

^
cc1plus.exe: all warnings being treated as errors

from /f3kdb: Test does not build: Traceback (most recent call last):
File "/f3kdb/.waf3-1.7.15-9c6c439a6416a92b3e844736c4ef3c7b/waflib/Tools/c_config.py", line 458, in run_c_code
bld.compile()
File "/f3kdb/.waf3-1.7.15-9c6c439a6416a92b3e844736c4ef3c7b/waflib/Build.py", line 188, in compile
raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
-> task in 'testprog' failed (exit status 1):
{task 7696569193424: cxx test.cpp -> test.cpp.1.o}
['/mingw64/bin/g++', '-fPIC', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fvisibility=hidden', '-fvisibility-inlines-hidden', '-Werror', '-std=c++11', '../test.cpp', '-c', '-o', 'test.cpp.1.o']

no
from /f3kdb: The configuration failed
--------------------------------------------
- alignas
==>
int main() { alignas(8) int x = 0; return x; }
<==
[1/2] cxx: build/.conf_check_e3129656e5b3600e23d9ba3a04379822/test.cpp -> build/.conf_check_e3129656e5b3600e23d9ba3a04379822/testbuild/test.cpp.1.o

['/mingw64/bin/g++', '-fPIC', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fvisibility=hidden', '-fvisibility-inlines-hidden', '-Werror', '-std=c++11', '-O3', '../test.cpp', '-c', '-o', 'test.cpp.1.o']
err: ../test.cpp:1:0: error: -fPIC ignored for target (all code is position independent) [-Werror]
int main() { alignas(8) int x = 0; return x; }
^
cc1plus.exe: all warnings being treated as errors

from /f3kdb: Test does not build: Traceback (most recent call last):
File "/f3kdb/.waf3-1.7.15-9c6c439a6416a92b3e844736c4ef3c7b/waflib/Tools/c_config.py", line 458, in run_c_code
bld.compile()
File "/f3kdb/.waf3-1.7.15-9c6c439a6416a92b3e844736c4ef3c7b/waflib/Build.py", line 188, in compile
raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
-> task in 'testprog' failed (exit status 1):
{task 7696569195024: cxx test.cpp -> test.cpp.1.o}
['/mingw64/bin/g++', '-fPIC', '-Wall', '-Wextra', '-Wno-unused-parameter', '-fvisibility=hidden', '-fvisibility-inlines-hidden', '-Werror', '-std=c++11', '-O3', '../test.cpp', '-c', '-o', 'test.cpp.1.o']

no
from /f3kdb: The configuration failed

[Myrsloik]

Remove -Werror from the build script. It shouldn't be there.

[Are_]

I think I did try that and was no good. Here is a compile with msvs12 for vapoursynth-flash3kyuu_deband-2.0.20140721 (https://www.amazon.es/clouddrive/share/RmcltkloTt9EyRhks3C7rKAgiETEJTfJaLQgD2xkyvo), and that's the date for the last time code updated on github. It includes 32/64 bit binaries and docs.

[MonoS]

In fact seems that it's not compilable with gcc, i continue to get errors.

Your build works perfectly, thanks Are_

[Magik Mark]

Got an 8bit video.

Executed this command f3kdb(keep_tv_range = true, output_depth = 10)

Rencoded using staxrip x265 2pass 10bit

Got green screen

Any work around?

[~ VEGETA ~]

I am using vapoursynth (portable x64) on windows 7 x64. I have this :


import vapoursynth as vs
import sys
core = vs.get_core()
video = core.lsmas.LWLibavSource(source=r'.......\file.mkv')

db = core.f3kdb.Deband(video)
db.set_output()



I get this:


Failed to evaluate the script:
Python exception: Failed to load C:......\flash3kyuu_deband.dll. GetLastError() returned 126.
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 1491, in vapoursynth.vpy_evaluateScript (src\cython\vapoursynth.c:26905)
File "C:/....../Untitled.vpy", line 4, in <module>
core.std.LoadPlugin(path=r'C:.......\flash3kyuu_deband.dll')
File "src\cython\vapoursynth.pyx", line 1383, in vapoursynth.Function.__call__ (src\cython\vapoursynth.c:25212)
vapoursynth.Error: Failed to load C:\........\flash3kyuu_deband.dll. GetLastError() returned 126.





I also tried loading it manually via LoadPlugin but didn't work. I installed the windows update that is needed for this to work and everything works like lsmash... I guess the problem is it doesn't load the f3kdb plugin.

[trip_let]

I also tried loading it manually via LoadPlugin but didn't work. I installed the windows update that is needed for this to work and everything works like lsmash... I guess the problem is it doesn't load the f3kdb plugin.

You grabbed the 2.0, right?

[~ VEGETA ~]

You grabbed the 2.0, right?

yes that.

[jackoneill]

yes that.

If you got it from the first post in this thread, it's a 32 bit binary. You can find a 64 bit binary a few posts up.

[~ VEGETA ~]

thanks, I already got it after your post. it is better than GrainFactory3 which puts lots of grain and doesn't really work fine with dark area masking.

[Stereodude]

Any possibility for an updated link to download the 2.0pre2 (64-bit and 32-bit)?

[sneaker_ger]

https://mega.nz/#F!Y10U2LiJ!2H29c5mv5npZiTMdtSwveg
("f3kdb-b98d6bc.7z" is newer. Don't remember exactly what is different, I think some crash-fix or something. So I'd use that instead of the older official build.)

[sl1pkn07]

any compilation with avs+ 2.6 API?

[DJATOM]

Sure, http://www.mediafire.com/file/92025c9lsj4546b/flash3kyuu_deband.dll

[sl1pkn07]

32 and 64 bits if is posible

Mani thanks!!

[DJATOM]

http://www.mediafire.com/file/l1mgo50p9k27zb1/f3kdb_latest_AvisynthPluginInit3.7z

[sl1pkn07]

tnx bro!!

[tuanden0]

@DJATOM Could you build x64 lastest version (9f3042e) for vapoursynth?

https://github.com/SAPikachu/flash3kyuu_deband

[DJATOM]

My builds are from latest commit, it should work with Vapoursynth
https://i.imgur.com/IPIRcaL.png

[tuanden0]

My builds are from latest commit, it should work with Vapoursynth
https://i.imgur.com/IPIRcaL.png

:thanks:

[ChaosKing]

https://mega.nz/#F!Y10U2LiJ!2H29c5mv5npZiTMdtSwveg
("f3kdb-b98d6bc.7z" is newer. Don't remember exactly what is different, I think some crash-fix or something. So I'd use that instead of the older official build.)

It should be this one https://github.com/SAPikachu/flash3kyuu_deband/commit/b98d6bc. So it's not the latest commit, but almost ;)

Oh and Myrsloik forked it if no one saw it yet: https://github.com/myrsloik/flash3kyuu_deband

[Stereodude]

It should be this one https://github.com/SAPikachu/flash3kyuu_deband/commit/b98d6bc. So it's not the latest commit, but almost ;)

Oh and Myrsloik forked it if no one saw it yet: https://github.com/myrsloik/flash3kyuu_deband
What unique about his fork? I looked at the commits, but don't see anything elated to algorithm improvements ie: Madshi's suggestions.

What's the state of AVIsynth+ high color depth support with 2.0pre2??

[Myrsloik]

What unique about his fork? I looked at the commits, but don't see anything elated to algorithm improvements ie: Madshi's suggestions.

What's the state of AVIsynth+ high color depth support with 2.0pre2??

I only forked it to bring more sanity to the codebase. Only got halfway on that project though. All the ancient avisynth support and such makes it impossible to compile with clang-cl so I removed it. Not really planning to do much more than clean it up so it's sane to compile and debug. Having the intel compiler tentacles everywhere is just rude!

[Stereodude]

I only forked it to bring more sanity to the codebase. Only got halfway on that project though. All the ancient avisynth support and such makes it impossible to compile with clang-cl so I removed it. Not really planning to do much more than clean it up so it's sane to compile and debug. Having the intel compiler tentacles everywhere is just rude!
So your fork will not support AVIsynth or AVIsynth+, only VapourSynth, or did I misunderstand you?

[foxyshadis]

So your fork will not support AVIsynth or AVIsynth+, only VapourSynth, or did I misunderstand you?

AviSynth+ should work, and maybe 2.6? AviSynth 2.5 won't. A few other plugins have already been through the VS carwash and come out much better on the AVS end.

[Myrsloik]

As I said, not even remotely done yet. But probably VS and Avs+ support. I hade needlessly complicated code like this...

[JKyle]

http://www.mediafire.com/file/l1mgo50p9k27zb1/f3kdb_latest_AvisynthPluginInit3.7z

@DJATOM,

StaxRip ships with your build of flash3kyuu_deband as of version v2.6.0, but it's been pointed out since a while ago that it suffers from a dependency issue.
Without DFTTest and masktools2, it crashes StaxRip or any other AVS script renderer such as AvsPmod preview or VirtualDub2.

https://cdn.discordapp.com/attachments/583884484743790614/850013798499614720/AvsPmod_preview_f3kdb_issue_1_20210603.png

But if DFTTest and masktools2 are loaded, it's OK.

https://i.imgur.com/oPZ34ss.png


One of StaxRip users came up with an independent build (https://gitlab.com/uvz/AviSynthPlus-Plugins-Scripts/-/blob/master/plugins64+/flash3kyuu_deband.dll) by the maintainer of Avisynth+ plugins pack (https://gitlab.com/uvz/AviSynthPlus-Plugins-Scripts), and it does turn out to be dependency free.

https://cdn.discordapp.com/attachments/583884484743790614/850023781208555530/AvsPmod_preview_f3kdb_issue_3_20210603.png

Reading the CHANGELOG (https://gitlab.com/uvz/AviSynthPlus-Plugins-Scripts/-/blob/master/CHANGELOG.md), I could see that he built his binary with Clang but there's no more detailed info on whether and how he modded the official source code. As a matter of fact, it's not even clear if he did build the binary with Clang 'cuz he had built a previous version with an Intel compiler according to the CHANGELOG.

Anyway, do you have any idea about what caused the dependency issue in your build and why a different build does not raise the same issue? :confused:

[Shuichiyagami]

Hello, i got green preview like this when i use this settings
flash3kyuu_deband(range=16, y=48, cb=48, cr=48, grainy=48, grainc=48, dither_algo=3, input_depth=8, output_depth=10)

My point is just to make sure 8-bit video source to became 10-bit after encode.

https://i.ibb.co.com/FhfdTwD/TYUI.png (https://ibb.co.com/FhfdTwD)