flash3kyuu_deband 1.5.1 / 2.0pre2 with native VapourSynth support [2012-12-03]

[SAPikachu]

When I tried ver 1.4 with avisynth 2.6 I got out of memory error in your dll reported by Vdub.
Same conditions, just with older version- no problem.

Maybe there is a bug in my code, can you post your parameter and colorspace?

[SAPikachu]

I get some exception too when a stacked clip (1920x2160) is feeded as YUY2 (not YV16).
Regarding the differences, I found that f3kdb correctly debands zones that GradFun3 doesn't detect, but GradFun3 seems to preserve very dark areas (and edges with similar colors to contiguous zones) better.

To enhance the edges detection, since now we can work on 16bit, isn't it possible to work with curves (increase contrast, get a mask to get where to dither, return to original curve and dither, skipping edges) or to create an edge mask on the source and then copy it (only the edges detected) to the debanded clip?

I will check the YUY2 problem later in this week, but I recommend to use YV16, because YUY2 code is not well-tested nor optimized, so it will be slow and maybe buggy.

Regarding curves and edgemask, they are good ideas, but I have a promise with my friend and need to have some research on a denoise algorithm(maybe a new avs filter), so I won't work on new features of f3kdb for a while. Will work on that after I have some progress on the algorithm. :)

[kolak]

Maybe there is a bug in my code, can you post your parameter and colorspace?

It was yuy2 source, so maybe this is the issue. Standard settings with keep tv range.

[Fullmetal Encoder]

To enhance the edges detection, since now we can work on 16bit, isn't it possible to work with curves (increase contrast, get a mask to get where to dither, return to original curve and dither, skipping edges) or to create an edge mask on the source and then copy it (only the edges detected) to the debanded clip?

This was just the idea I had when I had discovered dither tools. It would seem to me that masktools would do the job quite nicely and could be used with f3kdb inside AviSynth without the need to code any kind of edge mask functionality into f3kdb itself. The only problem I can see though is that masktools would have to be modified/re-written with at least 16-bit internal precision, would it not? A minor detail.

Ideally it would be wonderful if we could get a full 16-bit pipeline established like so: QTGMC(dfttest, nnedi3, masktools) --> f3kdb --> 10-bit x264

I would love to contribute to this goal myself but while I know a little C++ and am studying it it will be a long time before I can do anything meaningful at that level.

[Fullmetal Encoder]

I just recently began experimenting with 10-bit encoding. I'm using f3kdb and I think I understand what it's doing through the encoding stage but I was hoping that someone here could help by confirming if my understanding and approach is correct. At the moment I am working with interlaced Blu-ray material. My goal is to deband the source and then dither down from the 16-bit internal precision to 10-bits using f3kdb before sending the output through to x264-10bit. What I'm thinking is that sending the 16-bit results directly to x264-10bit would truncate the 16 bits to 10 while a much higher quality way to do it is to dither to 10 bits inside f3kdb first. Is this correct? My script is as follows:

LoadPlugin("DGAVCDecode.dll")
AVCSource(dga="00041.dga", deblock=false)
f3kdb(sample_mode=2, dynamic_dither_noise=true, precision_mode=2, output_depth=10, output_mode=2, keep_TV_range=true)

Also, my understanding is that since the source is in YUV 4:2:0 the 16-bit processing in f3kdb is in the same format only with the values in higher decimal precision (internally). When the dithering is done something that goes beyond what simply truncating the values is performed to reduce that precision to 10 bits which are still in the YUV 4:2:0 format. From that point, x264-10bit happily accepts the YUV 4:2:0 10-bit input (from what I've read YUV 4:2:0 can contain 12 bits) to produce the final 10-bit encode. Or is there some kind of format conversion forced upon the values when the 16-bit values are generated? My concern is limiting or if possible eliminating any destructive color conversions in the chain.

[SAPikachu]

Released 1.4.1. Fixed YUY2 support.

[SAPikachu]

This was just the idea I had when I had discovered dither tools. It would seem to me that masktools would do the job quite nicely and could be used with f3kdb inside AviSynth without the need to code any kind of edge mask functionality into f3kdb itself. The only problem I can see though is that masktools would have to be modified/re-written with at least 16-bit internal precision, would it not? A minor detail.

Ideally it would be wonderful if we could get a full 16-bit pipeline established like so: QTGMC(dfttest, nnedi3, masktools) --> f3kdb --> 10-bit x264

I would love to contribute to this goal myself but while I know a little C++ and am studying it it will be a long time before I can do anything meaningful at that level.

Yes, it will better to do masking in masktools since that will be much more flexible. We really need a high bit-depth version of masktools...

I just recently began experimenting with 10-bit encoding. I'm using f3kdb and I think I understand what it's doing through the encoding stage but I was hoping that someone here could help by confirming if my understanding and approach is correct. At the moment I am working with interlaced Blu-ray material. My goal is to deband the source and then dither down from the 16-bit internal precision to 10-bits using f3kdb before sending the output through to x264-10bit. What I'm thinking is that sending the 16-bit results directly to x264-10bit would truncate the 16 bits to 10 while a much higher quality way to do it is to dither to 10 bits inside f3kdb first. Is this correct? My script is as follows:

LoadPlugin("DGAVCDecode.dll")
AVCSource(dga="00041.dga", deblock=false)
f3kdb(sample_mode=2, dynamic_dither_noise=true, precision_mode=2, output_depth=10, output_mode=2, keep_TV_range=true)



If your source is interlaced, I think you need to IVTC/deinterlace first. f3kdb should be put at the end of script.

In theory, dither to 10bit is better than directly feed 16bit to x264, but you need to use a hacked x264 to read non-8/16 bit source. You can use this version: http://j.mp/x264tMod , and specify --input-depth 10 in your command line.




Also, my understanding is that since the source is in YUV 4:2:0 the 16-bit processing in f3kdb is in the same format only with the values in higher decimal precision (internally). When the dithering is done something that goes beyond what simply truncating the values is performed to reduce that precision to 10 bits which are still in the YUV 4:2:0 format. From that point, x264-10bit happily accepts the YUV 4:2:0 10-bit input (from what I've read YUV 4:2:0 can contain 12 bits) to produce the final 10-bit encode. Or is there some kind of format conversion forced upon the values when the 16-bit values are generated? My concern is limiting or if possible eliminating any destructive color conversions in the chain.

From my understanding, 4:2:0 is a format of chroma-subsampling, it is not related to bit depth. You can check it here: http://en.wikipedia.org/wiki/Chroma_subsampling

[sneaker_ger]

I knew that cretindesalpes said that ordered dithering is better for compression, but I didn't expect that much of a difference. I debanded a test clip and then encoded to 8 bit x264 (2106, preset slower, tune animation, 2 pass 2079 kb/s):

source (floyd 8bit):
http://www.abload.de/img/8bit_floydsteinberg_smrulk.png

source (ordered 8bit):
http://www.abload.de/img/8bit_ordered_sourcevku98.png

source (no dither 8bit):
http://www.abload.de/img/8bit_no_dither_sourcet9u2z.png

f3kdb (floyd-steinberg 8bit):
http://www.abload.de/img/8bit_floydsteinberg1pudl.png

f3kdb (ordered 8bit):
http://www.abload.de/img/8bit_orderedyguhy.png

f3kdb (no dither 8bit):
http://www.abload.de/img/8bit_no_dithervquql.png

f3kdb (floyd-steinberg 16 bit, x264 8bit):
http://www.abload.de/img/16bit_floysteinberg_tcouu3.png

My conclusion: directly letting f3kdb dither to 8 bit with ordered dithering looks the best and not dithering looks very similar, followed by using 16 bit output and letting x264cli dither to 8 bit. f3kdb's 8 bit floyd-steinberg dithering gets totally destroyed at the encoding stage.
So for encoding I will from now on directly dither to the target bitdepth (be it 8 bit or 10 bit) using ordered dithering. I didn't see any problem before encoding between ordered and floyd-steinberg - or at least not in a still.
Any comments? Anyone want to share their findings?

Outside the competition:
10 bit x264/10 bit floyd-steinberg (http://www.abload.de/img/10bit_floydsteinberg7yu64.png)
8 bit x264 preset placebo/8-bit ordered (http://www.abload.de/img/8bit_ordered_placebofvuiy.png)


On a side node:
avs2yuv always crashes at the end of a script for me and avs2pipe(26)(mod) doesn't seem to support the high bit depth hack. Any alternatives? I'm using 32 bit AviSynth 2.6.0.

[Keiyakusha]

On a side node:
avs2yuv always crashes at the end of a script for me and avs2pipe(26)(mod) doesn't seem to support the high bit depth hack. Any alternatives? I'm using 32 bit AviSynth 2.6.0.

There was somewhere avs4x264mod_v0.2 on the forum. Only mod supports high bitdepth. It worked well for me

As for your results... it is clear to me that you say that ordered dithering is better, but not sure If i understand 2nd part.
directly letting f3kdb dither to 8 bit with ordered dithering looks the best
looks best compared to what? you seems to be not comparing f3kdb --> ordered 8bit output but encoded as 10bit.
10bit encode is always winner regardless of what your actual input is. Means it is good to use it with pure 8bit input.

[sneaker_ger]

looks best compared to what? you seems to be not comparing f3kdb --> ordered 8bit output but encoded as 10bit.

Except for the single 10 bit screenshot at the bottom of my post, all screenshots were done either directly from the source script (titled "source") or from the final 8 bit x264 encode (titled "f3kdb"). I didn't do any 10 bit x264 tests (again: except the one at the bottom, titled "outside the competition".)
It looks better compared to (1) using floyd-steinberg 8 bit dithering, i.e. precision_mode=3, output_depth=8, (2) using 16 bit floyd-steinberg and letting x264cli dither to 8 bit, i.e. precision_mode=3, output_depth=16 (3) using "no dithering" , i.e. precision_mode=1, output_depth=8

I hope it becomes clearer now. (Maybe I should rename the titles, as they are confusing.)

[sneaker_ger]

There was somewhere avs4x264mod_v0.2 on the forum. Only mod supports high bitdepth. It worked well for me

Thanks, that seems to work. It shows some warning ("avs [info]: Avisynth 2.6+ detected, forcing conversion to YV12"), but I guess that can be ignored in this context.

[Keiyakusha]

I hope it becomes clearer now. (Maybe I should rename the titles, as they are confusing.)
No it is ok. its just a language barrier. I agree, making direct dithering to desired bitdepth before encoding is better.

Thanks, that seems to work. It shows some warning ("avs [info]: Avisynth 2.6+ detected, forcing conversion to YV12"), but I guess that can be ignored in this context.

This is normal, x264 shows this warning for ages when it takes input from avisynth 2.6+. It is not because of avs4x264. Don't really remember now why it does so (because x264 hates possible YV24 as input?). Probably you didn't saw this warning before because avs2yuv makes x264 think it takes raw input and avs4x264 communicates with x264 in a bit "smarter" way so you don't need to specify resolution or framerate for example.

[SAPikachu]

Released 1.4.2, a bug fix build.

[SAPikachu]

On a side node:
avs2yuv always crashes at the end of a script for me and avs2pipe(26)(mod) doesn't seem to support the high bit depth hack. Any alternatives? I'm using 32 bit AviSynth 2.6.0.

You can also try patched x264 build, like http://j.mp/x264tMod , it can directly read interleaved high bit-depth clips without external tools.

[kemuri-_9]

This is normal, x264 shows this warning for ages when it takes input from avisynth 2.6+. It is not because of avs4x264. Don't really remember now why it does so (because x264 hates possible YV24 as input?). Probably you didn't saw this warning before because avs2yuv makes x264 think it takes raw input and avs4x264 communicates with x264 in a bit "smarter" way so you don't need to specify resolution or framerate for example.

That warning no longer occurs in the official x264 builds.
the original issue is because the new 2.6 colorspaces were designed a way that a 2.5-style isYV12() check would always be true.
so before I added 2.6 csp support in the C interface for x264, it needed to force a YV12 conversion because it had no way of telling if it was really a YV12 csp or some new 2.6 csp.

[Fullmetal Encoder]

That warning no longer occurs in the official x264 builds.
the original issue is because the new 2.6 colorspaces were designed a way that a 2.5-style isYV12() check would always be true.
so before I added 2.6 csp support in the C interface for x264, it needed to force a YV12 conversion because it had no way of telling if it was really a YV12 csp or some new 2.6 csp.

When was this warning removing? I have the latest 10-bit x264 and it will still give me this warning.

Also, can I confirm that it isn't making any changes to the input so long as the source starts as YUV 4:2:0? I believe I read this in the original thread where this issue was discussed but I'd like to make sure since it kind of freaks me out.

Thank you.

[Fullmetal Encoder]

Outside the competition:
10 bit x264/10 bit floyd-steinberg (http://www.abload.de/img/10bit_floydsteinberg7yu64.png)
8 bit x264 preset placebo/8-bit ordered (http://www.abload.de/img/8bit_ordered_placebofvuiy.png)


I am curious, for the 10-bit x264 floyd-steinberg image did you first dither to 10 bits with f3kdb and then send the 10-bit output to x264-10bit or did you have x264-10bit cut it down to 10 from 16? I am interested in comparing the difference between the two. I will upload my own comparisons when I get a chance.

[kolak]

Small bit out of topic- ffmpeg gets ProRes encoder with 10bit support :)

[sneaker_ger]

I am curious, for the 10-bit x264 floyd-steinberg image did you first dither to 10 bits with f3kdb and then send the 10-bit output to x264-10bit or did you have x264-10bit cut it down to 10 from 16?

The former.

[Nevilne]

Afaik vanilla x264 doesn't dither 16 to 10. You should passing 10bit avs directly to taro's x264 (which is correct way anyway). I think jeeb's builds support that as well?

[Fullmetal Encoder]

Afaik vanilla x264 doesn't dither 16 to 10.

This is what I thought. I expect x264-10bit to simply truncate the 16 bit values to 10. I would like to see how visible the difference will be.

[kemuri-_9]

When was this warning removing? I have the latest 10-bit x264 and it will still give me this warning.

that warning was removed in this commit (http://git.videolan.org/?p=x264.git;a=commit;h=1a6fd476c477a7870d15d0c7f005ae34813d2f9d), which is actually a long time ago now.

However, this does not necessarily cover any use of avs4x264 or other similar programs that are intended to pipe from 32bit avisynth to 64bit x264.
So if you're still getting the warning now, it's probably originating from that and not x264 itself...

Also, can I confirm that it isn't making any changes to the input so long as the source starts as YUV 4:2:0? I believe I read this in the original thread where this issue was discussed but I'd like to make sure since it kind of freaks me out.

We originally left this to avisynth, where ConvertToYV12() with a YV12 source is a No-Op...
Not sure if you could actually confirm it outside of modifying the source to print the AVS_VideoInfo's pixel_type before and after the ConvertToYV12 call to see that it didn't change.

Afaik vanilla x264 doesn't dither 16 to 10. You should passing 10bit avs directly to taro's x264 (which is correct way anyway). I think jeeb's builds support that as well?

what? if your input to x264 is generally >8 bit depth and it's compiled to output 10-bit files, then there will be a 16->10 dither.

[Nevilne]

Sorry if that was incorrect!

[SAPikachu]

what? if your input to x264 is generally >8 bit depth and it's compiled to output 10-bit files, then there will be a 16->10 dither.

The official x264 build will always do 16->10 dither indeed, but taro's build will skip this dithering when input and output bit-depth are the same.

[killazys]

A question about f3kdb(). If I want to make use of the dithering function only, would I run f3kdb(clip,0,0,0,0,output_mode=2,output_depth=16)? Would that work?

The docs say that "If set to 0, the corresponding plane will be untouched regardless of dither settings." for banding threshold. Does that mean it will not dither either?

[sneaker_ger]

There's f3kdb_dither() (see readme), but dithering is for reducing bitdepth, it does not really make sense to dither to 16 bit output.

[SAPikachu]

A question about f3kdb(). If I want to make use of the dithering function only, would I run f3kdb(clip,0,0,0,0,output_mode=2,output_depth=16)? Would that work?

The docs say that "If set to 0, the corresponding plane will be untouched regardless of dither settings." for banding threshold. Does that mean it will not dither either?

I think what you meant is bit-depth conversion? If input bit-depth != output bit-depth, the plane will always go through the filtering process. I will note that in the readme later. It's OK to set Y/Cb/Cr/ditherY/ditherC to all 0, f3kdb will only convert bit-depth with this parameter set.

BTW: You missed one "0" in your parameter list. ditherC is still the default value. :)

[sneaker_ger]

The official x264 build will always do 16->10 dither indeed, but taro's build will skip this dithering when input and output bit-depth are the same.

Are you sure this is true? In my test that I posted here about a month ago I used 10 bit output and I didn't use taro's build.

Btw, is there any reason for f3kdb_dither() being limited to 8 bit output depth?

[SAPikachu]

Are you sure this is true? In my test that I posted here about a month ago I used 10 bit output and I didn't use taro's build.

Btw, is there any reason for f3kdb_dither() being limited to 8 bit output depth?

Yes, at least at the time of writing, x264 will always convert high bit-depth clips to 16bit first, then dither down to 10bit. Taro's build can skip this process. It can also directly read high bit-depth avs script without avs4x264(mod).

Reason is simple - I was too lazy to add that, since it can now be done entirely in the main f3kdb filter. :P

[killazys]

Thanks SAPikachu! This will help me make quasi-improvements on sources that don't suffer from heavy banding :D

Edit: more like real improvements. Really decreases the whole "purple" issue.

[killazys]

Ah, sneaker_ger. I just read your comment. It actually helps a lot with "purple" color hues related to encoder compression. I can give you screenshot comparisons if you'd like.

Well... in my case, I did use the dither function. So in truth, I am unsure if disabling dither will also disable the purple addressing.

[sneaker_ger]

Yes, at least at the time of writing, x264 will always convert high bit-depth clips to 16bit first, then dither down to 10bit. Taro's build can skip this process. It can also directly read high bit-depth avs script without avs4x264(mod).

Found Taro's builds the other day and his website indeed explicitly mentions this capability - I would have never known. I wonder why the x264 dev's don't at least skip processing when input and output depth are 10 bit.

Ah, sneaker_ger. I just read your comment. It actually helps a lot with "purple" color hues related to encoder compression. I can give you screenshot comparisons if you'd like.

Well... in my case, I did use the dither function. So in truth, I am unsure if disabling dither will also disable the purple addressing.

Yes, I'd be interested in those screenshots, but what are your input and output/encoding bit depths? I don't really understand what f3kdb does when debanding is deactivated and input depth is smaller than output depth. My assumption would have been that it'd just do a skew operation, but from your discussion with SAPikachu this does not seem to be the case. Or do you first skew and then let x264 dither and the result would then be different from the original version?

[SAPikachu]

Yes, I'd be interested in those screenshots, but what are your input and output/encoding bit depths? I don't really understand what f3kdb does when debanding is deactivated and input depth is smaller than output depth. My assumption would have been that it'd just do a skew operation, but from your discussion with SAPikachu this does not seem to be the case. Or do you first skew and then let x264 dither and the result would then be different from the original version?

If output_depth > input_depth and deband-related parameters are all 0, pixels will be simply shifted to output bit-depth.

Thanks SAPikachu! This will help me make quasi-improvements on sources that don't suffer from heavy banding :D

Edit: more like real improvements. Really decreases the whole "purple" issue.

I don't quite understand, what's the "purple" issue?

[killazys]

Here's a screenshot comparison: http://screenshotcomparison.com/comparison/98877

Notice especially the coloring in the top right corner.

[SAPikachu]

I can see the difference. Did you capture the screenshot from 10bit video encoded by official x264 build?

[killazys]

No. I used JEEB's 2106 10-bit x64 build and captured the screenshots by using FFVideoSource(in.mkv) in VirtualDub (copy source to clipboard)

[SAPikachu]

Then I am not sure about the reason. There are too many factors that can cause this problem. Maybe you can try taro's build? I just checked one of my recent encode (8 bit input to taro's 10 bit x264), and didn't find this problem. You can see it here: http://screenshotcomparison.com/comparison/99003 (captured using FFMS2 in AvsP)

[sneaker_ger]

Here's a screenshot comparison: http://screenshotcomparison.com/comparison/98877

Notice especially the coloring in the top right corner.

1. Are these from the same frame?
2. TV or PC range encode?

[sneaker_ger]

Then I am not sure about the reason. There are too many factors that can cause this problem. Maybe you can try taro's build? I just checked one of my recent encode (8 bit input to taro's 10 bit x264), and didn't find this problem. You can see it here: http://screenshotcomparison.com/comparison/99003 (captured using FFMS2 in AvsP)

I think it's very hard to spot anyways, here's a comparison with blue and red colors and I honestly can't see any color skew:
http://screenshotcomparison.com/comparison/99027

It's really only visible with these kinda beige colors, like the one posted by killazys.

[killazys]

Hm. sneaker_ger, I am unsure as to how Coalgirls does its encodes. Those frames are close enough; I assure you the problem can be seen throughout in the grey scenes.

Granted, it's not very noticeable but I'm picky enough to want to dither up and back down to correct it.

And why are you comparing vanilla to JEEB's build?

[sneaker_ger]

And why are you comparing vanilla to JEEB's build?

JEEB's builds supposedly have the corrected TV range 8 bit -> 10 bit conversion, which did not find its way into the vanilla builds yet. I used 8 bit to feed x264cli for my screenshots.
http://x264.fushizen.eu/?p=381
http://forum.doom9.org/showthread.php?p=1517620#post1517620

[killazys]

Oh hm. I also use keep_tv_range in f3kdb if that matters.

[SAPikachu]

Oh hm. I also use keep_tv_range in f3kdb if that matters.

It shouldn't matter if you set ditherC/Cb/Cr to 0, and if your source is TV range (IIRC Madoka is correctly in TV range).

[killazys]

SAPikachu, for that Madoka encode I actually used
f3kdb(mt=true,range=31,Y=96,Cb=96,Cr=96,ditherY=16,ditherC=16,keep_tv_range=true,sample_mode=2,output_mode=2,output_depth=16)

So the color change might be because of the dithering, not the conversion to 16bit. Would that even be possible?

[SAPikachu]

Just thought about that, in theory it shouldn't affect the color in this way, since noises is added randomly. But I don't have time to test it now. Can you try setting ditherC/Cb/Cr to 0 and compare the result?

[killazys]

Sorry.. I don't have that raw anymore. After I finish some more encodes for my group I'll run a short comparison encode with another suitable raw. Likely some other madoka episode.

[Keiyakusha]

Hi, I want to ask something about true 16bit input.
Here (http://o.imgbox.com/aaiHN5lm.png) is how my source looks (yuv444p16le). And here (http://o.imgbox.com/aap666pT.png) is how it looks after I apply the following: f3kdb(input_mode=1,input_depth=16,output_mode=1,output_depth=16,keep_tv_range=true)

My question is... is everything is OK here? I mean, to me it looks like before there was some valuable information in LSB part, but after I applied debanding there is nothing but random noise. Is it not recommended to apply debanding on true 16bit? If so what if my source is mixed with true 16bit parts and 8bit parts that need to be debanded and these parts even can be mixed within the one frame?

[cretindesalpes]

Add ditherY=0 and ditherC=0 to avoid generating noise.

[Keiyakusha]

Thanks, now it seems to be alright. Here (http://o.imgbox.com/aawiEwnv.png) is what I'm getting.
However I'm afraid I have another question. If I want to get 10bit YV24 output I use this:
f3kdb(input_mode=1, input_depth=16, output_mode=1, output_depth=10)
And the output looks like this (http://o.imgbox.com/aasmd7sS.png). While I think the LSB looks really cool, MSB part looks kind of suspicious. Is everything ok with it?

BTW test image that I created you can get here (http://www.mediafire.com/?isvj65dx1qqrpor) (raw yuv444p16le stream)

[cretindesalpes]

This looks right. In 10 bits, the MSB values are confined to 0-3 for all planes, which is out of the 8-bit YUV range fitting in the RGB cube (esp. for chroma). Thus everything will look greenish.