10-bit Processing

[MysteryX]

I'm currently using this script to upscale 288p videos to 720p.

Then I was reading about 10-bit processing, and how it gives more accurate processing while also making encoding 10-20% more effective.
http://avisynth.nl/index.php/High_bit-depth_Support_with_Avisynth
http://x264.nl/x264/10bit_02-ateme-why_does_10bit_save_bandwidth.pdf

So my question is: would I benefit from using 10-bit processing? How could I tweak my script to make the best use of it?

PluginPath=""
LoadPlugin(PluginPath+"ColorMatrix.dll")
LoadPlugin(PluginPath+"KNLMeansCL.dll")
LoadPlugin(PluginPath+"nnedi3.dll")
Import(PluginPath+"edi_rpow2.avsi")
Import(PluginPath+"ResizeX.avsi")
LoadPlugin(PluginPath+"Shader.dll")
Import(PluginPath+"SuperRes.avsi")
LoadPlugin(PluginPath+"svpflow1.dll")
LoadPlugin(PluginPath+"svpflow2.dll")
Import(PluginPath+"InterFrame2.avsi")

SetMTMode(3,8)
AviSource("Preview.avi", audio=true, pixel_type="YV12")
SetMTMode(2)
ColorMatrix(mode="Rec.601->Rec.709")
Crop(0, 0, -8, -0)
SetMTMode(5)
KNLMeansCL(D=2, A=1, h=3, device_type="GPU")
SetMTMode(2)
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="Spline16Resize", Threads=2)""", PluginPath)
InterFrame(Cores=8, Tuning="Smooth", NewNum=60000, NewDen=1001, GPU=true)
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="Spline36Resize", fwidth=944, fheight=724, Threads=2)""", PluginPath)
Spline36Resize(940, 720, 0, 4, -4, -0)

[sneaker_ger]

The paper you linked to is specifically describing 10 bit H.264 encoding, you do not have to change any of your scripts, you only have to use a 10 bit version of x264 for encoding to reap those benefits. The files will be incompatible with every hardware decoder, though.

[MysteryX]

Someone mentioned I could use high bit depth for matrix conversion for better quality. Am I fine with ColorMatrix or should I use an alternative function?

[sneaker_ger]

ColorMatrix() is probably "good enough". If you want high bitdepth you can take a look at the dither tools or fmtconv (VapourSynth).

[foxyshadis]

It all depends on how sensitive your eyes are and whether you have a good monitor/TV. I don't think Interframe or SVP is even capable of high bit depth anyway, so you lose most of the benefit since you can't keep it end-to-end.

Encoding to 10bit is a great idea if you plan to only play it on PC, otherwise you have to use 8bit anyway. (10bit x265 will be hardware-player compatible though.)

[MysteryX]

PC can play 10-bit no problem? What about mobile devices such as Kindle Fire HD?

The other filters in the chain don't support high bit depth (as far as I know). However, the original picture contains limited details which gets doubled and refined several times. Small improvements to the original picture can have noticeable difference once quadrupled and refined. Would running that one specific filter on high bit depth, and getting the result as 8-bit, going to make a difference, or it's useful when using several high-bit-depth filters in a row?

[bxyhxyh]

As for Colormatrix I meant this

Source - https://dl.dropboxusercontent.com/u/58215671/Other/orig.png
Normal Colormatrix - https://dl.dropboxusercontent.com/u/58215671/Other/colormatrix.png
Dithertools - https://dl.dropboxusercontent.com/u/58215671/Other/dithertools.png

Source itself has bandings, but Colormatrix creates more bandings at girl's head.

As for 10bit videos, yeah computers can play it.
I think tablets can decode it if codecs are installed. But I don't know about smartphones. Heard some players can. I haven't tried 10bit videos on mobile device myself.

[MysteryX]

Since I'm making a general video processing tool, I shouldn't look at whether it improves "my" videos, but rather if it improves "some" videos. In this case it does, so it would make sense to use this.

But I'm confused with Dither Tools. I downloaded it... and what command do I have to use? Are there dependencies required to use this other than dither.dll and avstp.dll ?

[Groucho2004]

But I'm confused with Dither Tools. I downloaded it... and what command do I have to use? Are there dependencies required to use this other than dither.dll and avstp.dll ?
Reading documentation seems to be something that you avoid passionately.

[MysteryX]

I'm seeing another high-bit-depth improvement I can add: adding "lsb=true" to edi_rpow2 would cause it to use high-bit-depth resize. Not sure if it makes a noticeable difference but I can compare.

OK I found the sample code in the documentation. No wonder why I couldn't just figure out which function to use, it's not that simple!
Dither_convert_8_to_16 ()
Dither_resize16 (1280, 720)
Dither_convert_yuv_to_rgb (matrix="601", output="rgb48y", lsb_in=true)
r = SelectEvery (3, 0)
g = SelectEvery (3, 1)
b = SelectEvery (3, 2)
Dither_convert_rgb_to_yuv (r, g, b, matrix="709", lsb=false, mode=0)

As for x264 encoding, I could use the 10-bit version... but then I couldn't use the standard FFMPEG library for that. It's a lot simpler when I stick to the standard library.

[MysteryX]

Wow, adding lsb=true to EDI has a HEAVY cost on performance!! And then it corrupts the image; probably a bug in the EDI script where it doesn't convert back from 16-bit.

[Desbreko]

You can have edi_rpow2 use Dither_resize16 by setting lsb=true, but unless SuperRes can work with a stacked 16-bit clip, you'll just have to dither it back to 8-bit, so the difference will be minor.

You should also be aware that by using Dither for the matrix conversion, you're converting to and from RGB, which means upsampling and downsampling the chroma in the process. ColorMatrix, on the other hand, doesn't have to resample the chroma.

Edit:
Wow, adding lsb=true to EDI has a HEAVY cost on performance!! And then it corrupts the image; probably a bug in the EDI script where it doesn't convert back from 16-bit.
The whole point of the lsb option is that it doesn't dither the output back to 8-bit, so that further processing can be done in 16-bit. Call DitherPost after edi_rpow2 if you want the resize to be done with Dither_resize16 but don't want 16-bit output.

[MysteryX]

Trying with high bit depth ColorMatrix. The performance cost is high, but since it's running on the small clip before quadrupling its size, and since the other filters are heavy, it might be OK.

However, applying the pixel ratio is now done at the beginning instead of working with square pixels and applying it at the end. That will have an impact on performance and I'm not sure on the quality impact.

Then... it makes the video literally freeze in the player. Not sure why it's freezing like this. If I let it "sit" on pause for a while, then at some point it starts rendering.

[MysteryX]

What I'm seeing is that the 16-bit processing chain is very expensive. However, it would make sense for AviSynthShader to allow converting back and forth from this 16-bit format, as it would allowing processing whole scripts in 16-bit for those who want to do so.

Also, at first test, it appears that allowing dither tools to apply the pixel ratio causes more harm than first allowing NNEDI3 to upscale pixels and then scaling it back down to the right proportion.

Performance-wise... dither tools are VERY expensive on performance. The conversion happening with AviSynthShader from 8-bit to 16-bit is expensive but really not that much. In theory it shouldn't be more than twice slower, if implemented properly. As far as 8-bit to 16-bit conversion, if converting with bit-shifting 8 pixels at a time with assembly, it can be done very quick. I haven't optimized my own code to that level yet.

If we're talking about converting to RGB in Rec.601 and converting back in Rec.709, I wonder how performance would compare if doing that through AviSynthShader, which would be another option. Or perhaps... bundling that operation when running SuperRes. That would have no performance cost.

[bxyhxyh]

You should also be aware that by using Dither for the matrix conversion, you're converting to and from RGB, which means upsampling and downsampling the chroma in the process. ColorMatrix, on the other hand, doesn't have to resample the chroma.


To that recently I'm using this way.

c=source()
big=c.WhateverUpscaler().Dither_convert_8_to_16()

big.Dither_resize16(1280,720)
Dither_convert_yuv_to_rgb (matrix="601",output="rgb48y",lsb_in=true)
r=SelectEvery(3,0)
g=SelectEvery(3,1)
b=SelectEvery(3,2)
lu=Dither_convert_rgb_to_yuv(r,g,b,matrix="709",lsb=false,mode=6)

YtoUV(big.utoy8().Dither_Resize16(640,360),big.vtoy8().Dither_Resize16(640,360),c.Dither_convert_8_to_16().Dither_resize16(640,360,-0.5))
Dither_convert_yuv_to_rgb (matrix="601",output="rgb48y",lsb_in=true)
r=SelectEvery(3,0)
g=SelectEvery(3,1)
b=SelectEvery(3,2)
c=Dither_convert_rgb_to_yuv(r,g,b,matrix="709",output="YV24",lsb=false,mode=6)

u=c.utoy8()
v=c.vtoy8()
ytouv(u,v,lu)

It's just an example for center shift corrected clips.
Though I don't know it's correct way to change matrix.
I still think there are some mistakes in it.
At least chroma is not blurry as lu's.

It's even more expensive on performance side.

[cretindesalpes]

You can use a larger, sharper kernel for the RGB ↔ YUV conversions. Default for chromak is "bicubic" which is not very sharp.

[bxyhxyh]

You can use a larger, sharper kernel for the RGB ↔ YUV conversions. Default for chromak is "bicubic" which is not very sharp.

What about adding invks argument to RGB ↔ YUV converting functions?
Upscale it with bicubic and downscale back with inverse bicubic when converting back to YV12 videos.
Wouldn't that save the chroma from "resized twice" and save people from hellish script like mine?

[MysteryX]

I tried implementing a high-bit-depth ColorMatrix conversion via HLSL shaders

Here's he result. There's more banding. I'm really not sure what could be causing this.
http://s20.postimg.org/pi9zucixp/Color_Matrix.png

I'm converting to UINT16. If I instead convert to half-float data, there's even more banding.

Edit: it appears OK. If I compare with standard ColorMatrix locally, then the standard function creates more banding.
http://s20.postimg.org/uv3fm7w7x/Color_Matrix2.png

Perhaps the samples above do additional processing, such as debanding? I'm not getting the same results.

The HLSL shader method is a *lot* faster than the dither tools method.

[bxyhxyh]

Perhaps the samples above do additional processing, such as debanding? I'm not getting the same results.

No, no additional band removing methods.
It's because you've downloaded an image file. Image is RGB. If you converted it to YV12 there are banding appears.

Here is a sample video.
http://www.mediafire.com/download/xizai7v4278xo4s/Sample.ts
It's not a vob file because I cut it with tsmuxer.

I faced same problem on a different clip (I forgot what it was.)
Similar videos would be the same I think

[MysteryX]

ah... ok. I convert from YUV to RGB in Rec.601 and then from RGB to YUV in Rec.701

What if the source is already RGB, then how should it be done? I suppose convert from RGB to YUV in Rec.601 and then from YUV to RGB in Rec.709

[MysteryX]

In that case, it does look good
http://s20.postimg.org/8gqcjqiu5/Color_Matrix16.png

I'll release the AviSynthShader version of high bit depth Color Matrix conversion soon.

However... the colors are considerably different than the dither tools version! Which version is good? The dithertools version and mine appear to be doing the inverse of each other.

Here's the result if I do the opposite. Now it looks almost identical to the dithertools version, with a very slight extra banding.
http://s20.postimg.org/jucvuxtct/Color_Matrix16.png

To get this latest result, I called ColorMatrix709to601. Could you take a look at the script I'm using to correct the order? Shader files without "601" use the 709 color space.

# Performs color matrix conversion with 16 bit depth to avoid banding
function ColorMatrix601to709(clip input)
{
folder = default(folder, "")
input
srcYuv = !IsRGB
sourceFormat = IsYV12 ? "YV12" : IsYV24 ? "YV24" : IsRGB24 ? "RGB24" : IsRGB32 ? "RGB32" : ""
Assert(sourceFormat != "", chr(10) + "Source must be YV12, YV24, RGB24 or RGB32" + chr(10))

input = ConvertToFloat()
Shader(srcYuv ? "Yuv601ToGamma.cso" : "GammaToYuv601.cso")
Shader(srcYuv ? "GammaToYuv.cso" : "YuvToGamma.cso")
ExecuteShader(last, input)
ConvertFromFloat(format=sourceFormat)
}

function ColorMatrix709to601(clip input, string "folder")
{
folder = default(folder, "")
input
srcYuv = !IsRGB
sourceFormat = IsYV12 ? "YV12" : IsYV24 ? "YV24" : IsRGB24 ? "RGB24" : IsRGB32 ? "RGB32" : ""
Assert(sourceFormat != "", chr(10) + "Source must be YV12, YV24, RGB24 or RGB32" + chr(10))

input = ConvertToFloat()
Shader(srcYuv ? "YuvToGamma.cso" : "GammaToYuv.cso")
Shader(srcYuv ? "GammaToYuv601.cso" : "Yuv601ToGamma.cso")
ExecuteShader(last, input)
ConvertFromFloat(format=sourceFormat)
}

[cretindesalpes]

What about adding invks argument to RGB ↔ YUV converting functions?
Upscale it with bicubic and downscale back with inverse bicubic when converting back to YV12 videos.
Wouldn't that save the chroma from "resized twice" and save people from hellish script like mine?
invks tries to restore attenuated frequencies but it won’t do miracles and may generate artefacts sometimes. Using a larger kernel with a flatter bandwidth in both passes will be more efficient.

[MysteryX]

As for Colormatrix I meant this

Source - https://dl.dropboxusercontent.com/u/58215671/Other/orig.png
Normal Colormatrix - https://dl.dropboxusercontent.com/u/58215671/Other/colormatrix.png
Dithertools - https://dl.dropboxusercontent.com/u/58215671/Other/dithertools.png

Source itself has bandings, but Colormatrix creates more bandings at girl's head.
I verified my script and it's giving good results with YV12 sources. With your RGB sample, I think you're applying ColorMatrix the wrong way.

[MysteryX]

I have released AviSynthShader v1.1

It includes ColorMatrix for high bit depth color conversion via shaders. SuperRes now also has srcMatrix601 parameter allowing converting color matrix from Rec601 to Rec709 at the same time with no performance cost.

http://forum.doom9.org/showthread.php?p=1747640#post1747640

[bxyhxyh]

With your RGB sample, I think you're applying ColorMatrix the wrong way.
There were no RGB samples. It's just that when taking screenshots application converted them to RGB. Or PNG supports only RGB or something.

[MysteryX]

There were no RGB samples. It's just that when taking screenshots application converted them to RGB. Or PNG supports only RGB or something.
Now that I think of it, ColorMatrix conversion from RGB really doesn't make any sense. I doubt there's a single practical application where that would be useful.

[bxyhxyh]

It includes ColorMatrix for high bit depth color conversion via shaders.
Does it resample YV12 to RGB, or just does calculation on YV12 clip?

[MysteryX]

Does it resample YV12 to RGB, or just does calculation on YV12 clip?
It resamples to RGB in 601 and then converts it back to YUV in 709, which is the same as the dither tools approach.

Since I'm using SuperRes anyway, which already has to do that conversion, I'm applying the matrix conversion in there which has no extra cost on quality or performance.

[bxyhxyh]

Since we talked about changing colormatrix here, I'm posting here.

When upscaling DVDs to 720p which method would you prefer?
Assuming DVD is 720x480, 2x upscaled by nnedi or others and resized to 1280x720.

1. 1440x960 YV12->1440x960 RGB->1280x720 RGB->1280x720 YV12
2. 1440x960 YV12->1280x720 YV12->1280x720 RGB->1280x720 YV12
3. 1440x960 YV12->1280x720 YV24->1280x720 RGB->1280x720 YV12

Second one is no option for me.
Because if we talk about chroma it would be something like

1. 720x480->1440x960->1280x720->640x360
2. 720x480->640x360->1280x720->640x360
3. 720x480->1280x720->1280x720->640x360

So, third one has least artifact?

[MysteryX]

Since we talked about changing colormatrix here, I'm posting here.

When upscaling DVDs to 720p which method would you prefer?
Assuming DVD is 720x480, 2x upscaled by nnedi or others and resized to 1280x720.

1. 1440x960 YV12->1440x960 RGB->1280x720 RGB->1280x720 YV12
2. 1440x960 YV12->1280x720 YV12->1280x720 RGB->1280x720 YV12
3. 1440x960 YV12->1280x720 YV24->1280x720 RGB->1280x720 YV12

Second one is no option for me.
Because if we talk about chroma it would be something like

1. 720x480->1440x960->1280x720->640x360
2. 720x480->640x360->1280x720->640x360
3. 720x480->1280x720->1280x720->640x360

So, third one has least artifact?

This is my answer to upscaling VCDs or DVDs to 720p
http://forum.doom9.org/showthread.php?t=173019

I made further improvement to the script by using SuperResXBR (which needs to be fixed for sub-pixel distortion) instead of NNEDI3+SuperRes, and by using x265 instead of x264. Those changes aren't yet released.

[bxyhxyh]

Well, I know that SuperRes can change colormatrix.

But my point is that I want to minimize those upsize and downsize processes.
Especially YUV is converted to RGB then is converted back to YUV. Those zig zag resized chromas will affect luma not beautifully.

So I think those resizes should be the minimal.

Then for dithertools, it would be something like this.


#2x up
Dither_convert_8_to_16()
Dither_resize16(1280,720,csp="YV24")
Dither_convert_yuv_to_rgb(matrix="601",output="rgb48y",lsb_in=true)
r=selectevery(3,0)
g=selectevery(3,1)
b=selectevery(3,2)
Dither_convert_rgb_to_yuv(r,g,b,matrix="709",output="YV12",chromak="spline16") #or spline36
DitherPost() #or mode=6

[geometer]

in case you can afford the CPU power, "upsampling" principle rules.
higher resolution (1440+), higher bit-depth (e.g. 10bit). there, all corrections can be made with minimal loss (colorspace+adjustments, with correctly recalculated radius and blocksize: sharpening, denoising).
there's a lot about rounding and quantization errors with these colorspaces.
then, downsample to target format, include the (final) dithering in this step. whether dithering helps or makes sense during the upsampled steps of processing, needs to be examined.
the value of dithering lies mainly in the downsampling, where we are actually going to destroy information. thus we try to spread some remainder of that information among a number of surrounding pixels. this makes sense when we have a gradient that will be turned into coarse steps by reducing data. the gradient is modulated into the noise so that it mitigates the "banding". there is a natural trade-off between dithering and sharpening. they compete for the space and for the data bandwidth.

[bxyhxyh]

in case you can afford the CPU power, "upsampling" principle rules.
higher resolution (1440+), higher bit-depth (e.g. 10bit). there, all corrections can be made with minimal loss (colorspace+adjustments, with correctly recalculated radius and blocksize: sharpening, denoising).
there's a lot about rounding and quantization errors with these colorspaces.
then, downsample to target format, include the (final) dithering in this step. whether dithering helps or makes sense during the upsampled steps of processing, needs to be examined.
the value of dithering lies mainly in the downsampling, where we are actually going to destroy information. thus we try to spread some remainder of that information among a number of surrounding pixels. this makes sense when we have a gradient that will be turned into coarse steps by reducing data. the gradient is modulated into the noise so that it mitigates the "banding". there is a natural trade-off between dithering and sharpening. they compete for the space and for the data bandwidth.

Was this reply to original post or my post?
English is not my native language. I only talked about changing the matrix.
And I'm bit confused here.

[geometer]

it was a very general remark about the process as a whole, not a direct answer.
hopefully its details can be approached coming from the bigger picture.

the matrix can have two issues:
a) is it correct at all? some videos come in that don't play by the standards anyway.
b) there are mathematical problems injected, like rounding errors, that would sum up through a number of conversions. upsampling can help this.

I see your target format is 640x360, right? but you are upsizing to 1280x720 intermediately, though the loss of data in the low resolution where it ends up, would likely destroy the benefit of almost every previous improvement.

so, I'm wondering what is the primary flaw you want to correct, that shows up when you do a direct, simple resize with the standard resizer and no hires tools.
if there is nothing beyond some worsened banding, then it takes just a resizer that includes dithering during the process, or in lack of, auxiliary, throw a strong dithering on the source and then just resize with the preferred lanczos or whatever looks sharp enough.

banding in the source can't be cured, unless there is a complex process that can identify the gradients and recalculate a gradient area completely, detecting exactly its borders.
poor man's version would be a gaussian blur plus dithering with wise use of masktools, that leaves the borders untouched. but such is not quite reliable. SmoothGrad() is such an attempt.

then, with the resizing (from 720 to 640 directly) you could try the following (untested)
Dither_resize16 (width=640, height=360, kernelh="impulse 2 -5 12 -20 112", kernelv="impulse 112 -10 12 -4 1", taps=5, fh=-1, fv=-1, kovrspl=1)
but I don't really understand the special taps parameter, plus the center has to be changed too when we use that blind guess kernel instead of the usual splines.

and then there is the bitness reduction, where also some tuning is possible:
DitherPost (mode=0, ampo=3, dyn=true, thr=.5)

so in my understanding of the matter it comes down to
Dither_convert_8_to_16 ... upsampling only bitness, not resolution, no dithering needed here
SmoothGrad ... correction for the banding in the source
Dither_resize16 ... get target resolution and sharpen (use a kernel that corrects special sharpness flaws in the source also)
DitherPost ... back to standard 8bit which is the actual step that requires the dithering in the first place, else the banding will return

[bxyhxyh]

Ok, so you were talking about high bitdepth processing.


I see your target format is 640x360, right? but you are upsizing to 1280x720 intermediately, though the loss of data in the low resolution where it ends up, would likely destroy the benefit of almost every previous improvement.


Please read other posts carefully. I never stated that my target is 640x360.
OP asked about 10 bit processing.
But also in this thread we have talked about changing the colormatrix using dithertools. My earliest post with AVS script was wrong.

Now I'm asking wouldn't this be less artifact way to change color matrix.

That is doing
Resizing to target. But convert to YV24 by upscaling chroma (1 resize on chroma)
Converting to RGB to change matrix (no resize)
Converting to YV12. (2 resizes on chroma)

Instead of
Converting to RGB (1 resize)
Resizing to target (2 resizes)
Converting to YV12. (3 resizes)

[geometer]

bxyhxyh, sorry my bad, I apologize, and for that matter agree to your #31.
(my english is also not as good as I pretend.)

[MysteryX]

Now I'm asking wouldn't this be less artifact way to change color matrix.

That is doing
Resizing to target. But convert to YV24 by upscaling chroma (1 resize on chroma)
Converting to RGB to change matrix (no resize)
Converting to YV12. (2 resizes on chroma)

Instead of
Converting to RGB (1 resize)
Resizing to target (2 resizes)
Converting to YV12. (3 resizes)
Changing color matrix is simple. There are 3 approaches.

1. Standard ColorMatrix, operating in 8-bit color space. It can cause banding due to rounding errors.

2. Changing color matrix with DitherTools, which converts from Rec601 with 16-bit-depth and then back to Rec709. The high-bit-depth prevents banding. This implementation is slow.

3. Changing color matrix with AviSynthShader, using the same approach as with DitherTools but doing it via HLSL shaders on the GPU. It is considerably faster.

If banding isn't an issue, you can get away with 1. If you want to avoid banding, you have to use either 2 or 3.

That's all there is to say about avoiding distortion during color matrix conversion.

[bxyhxyh]

When source is YV12 or YV16, upsampling and downsampling is unavoidable.
More the resizing, more the artifacts.
So you're not taking this as distortion?

[MysteryX]

When source is YV12 or YV16, upsampling and downsampling is unavoidable.
More the resizing, more the artifacts.
So you not taking this as distortion?
oh... DitherTools doesn't need to convert YV12 to YV24, you got a point there.

When comparing NNEDI3 with NNEDI3+SuperRes or SuperResXBR, however, the colors are considerably BETTER and sharper with SuperRes even if it adds color conversions.

I started this thread to do 10-bit encoding with x264. However, according to my tests, x265 preset 'medium' crf 22 does a better job than x264 preset 'veryslow' crf 24: faster processing, smaller file and higher quality. Unlike x264, with x265, 10-bit encoding isn't better than 8-bit. It just makes the image grainier.