YUV to OPP conversion

[MysteryX]

To use BM3D in OPP format, this conversion is super slow. This script alone runs at 10fps on a 5K clip. Before starting the actual work.

clip = clip.resize.Bicubic(format=vs.RGB48, matrix_in=1).bm3d.RGB2OPP(sample=1)
clip = clip.bm3d.OPP2RGB(sample=0).resize.Bicubic(format=vs.YUV444P16, matrix=1)


(it's written in VPY syntax but Avisynth will be the same thing) EDIT: Wait a second, it's BM3DCUDA that's in Avisynth which doesn't include RGB2OPP. How do you even do that conversion in Avisynth if the standard BM3D with that function isn't in Avisynth?

I tried the FMTC library with a custom coefficient and it's even slightly slower at 9fps.

Really, it's converting YUV to RGB using a coefficient and RGB to YUV using another coefficient. No reason that it should be slower than running 4 converts (which runs at over 20fps).

Is there a better way? Is it even necessary to first convert to RGB before converting back to OPP? And what about a GRAY clip, do I convert that at all or just stick to GRAY?

It probably could be done with a LUT table from 16-bit YUV to 32-bit OPP. But the reverse-way, how much memory would a 32-bit LUT table take? I'm also concerned about memory usage because BM3D is HUNGRY!

[Dogway]

Why don't you process it in YUV? All BM3D is asking is to process a single Luma plane, for performance and well due to R G and B correlation.
RGB to OPP is fine if your source is RGB otherwise you can process it in YUV or a better decorrelated model.
I made RGB to OPP functions and they are fast:
155fps (8-bit) 85fps (16-bit)
ConvertToPlanarRGB()
clp=ExtractClip()
RGB_to_OPP(clp[0],clp[1],clp[2],false)
clp=ExtractClip()
OPP_to_RGB(clp[0],clp[1],clp[2],false)

[MysteryX]

I compared the results of YUV vs OPP. There is a considerable quality difference. Confirming what's written in the doc.

[MysteryX]

Search returns nothing in your repository. Got a link to your functions?

[Dogway]

Check here (https://github.com/Dogway/Avisynth-Scripts/blob/master/TransformsPack.v1.0.RC22.avsi). I haven't uploaded OPP yet but I think there are better decorrelated models like YCbCr (CL) or YCgCoR (lossless roundtrip), or OkLab for more advanced. I think in all cases coming from YUV you have to go through RGB.

[MysteryX]

Thanks. I could play around and compare these other formats.

For SMDegrain and KNLMeans, would any of these formats be beneficial over YUV?

[Dogway]

Yes, they should be better but I haven't tested. ConvertFormat() is a long term project so I develop it from time to time without much fanfare. The individual color models should be correct for the most part.
YCbCr (Constant Luminance) is called YcCbcCrc in the functions, it is also used extensively.

My long term goal is to filter in an ideal decorrelated model and in the end encode in H265 LCEVC in IPTPQc2 @ 12-bit with Rec.2020 space.

[MysteryX]

RGB_to_YCgCoR and YCgCoR_to_RGB are pretty simple functions, easy to convert.

What is Y and Y@ here?
G = Expr(Y, Cg, ex_dlut("y range_half - Y@ 2 * x Y - +", bi, fs), optSingleMode=false)

[Dogway]

Y is the luma plane, Y@ is a variable, can store anything, in this case luma plane minus half_range

Currently it doesn't work over 32-bit float (need to standardize the functions similar to IPT and OkLab).
YCgCoR is fast because it doesn't require gamma corrections. As for OPP I can't tell how good the model is since there's practically no literature about it.

[MysteryX]

btw where is mt_adddiff defined? Been searching for it and can't find the definition of it

[Dogway]

Check here (https://tp7.github.io/articles/masktools/) also in the documentation (http://manao4.free.fr/mt_masktools.html#mt_adddiff).

[MysteryX]

oups sorry ex_makeadddiff, I wanted to test the ChromaReconstructor_faster you're using to see whether it could be useful

[Dogway]

ex_makeadddiff() is ex_makediff() then ex_addiff() into one single step so it's faster.

This is a common situation in filters, check below:

#HIGHPASS
HP = ex_makediff(src, src.ex_blur(1)) # Substract lowpass from source, leaving highpass as a result
#SRC + HIGHPASS
ex_adddiff(HP,src) # Add highpass back to original, increasing acutance/sharpness

#HIGHPASS + SRC
ex_makeadddiff(src, src.ex_blur(1)) # The above in a single step



ChromaReconstructor_faster is a feisty2's function so it's slow by nature (he), I tried to make it faster, it's very useful to recover some blurry chroma planes.

[MysteryX]

I tried YCgCoR for BM3D. Performance is great, but the contrasts aren't as well defined as with OPP.

Next will be to convert the OPP conversion functions.

[Dogway]

I ported them already but not sure it's correct since the OPP it's not documented, I simply ported from VS.
# https://github.com/HomeOfVapourSynthEvolution/VapourSynth-BM3D/blob/7b3d8dd32e4aa4d962cf15d75d191b465682ed42/include/Specification.h#L176
function RGB_to_OPP (clip R, clip G, clip B, bool fulls) {

bi = BitsPerComponent(R)
fs = Default (fulls, false)

b32 = bi == 32 ? "" : "range_half +"

O = Expr(R, G, B, ex_dlut(Format("x y z + + 0.333333333 *"), bi, fs), optSingleMode=false)
P1 = Expr(R, B, ex_dlut(Format("x y - 0.5 * "+b32), bi, fs), optSingleMode=false)
P2 = Expr(R, G, B, ex_dlut(Format("x z + 0.25 * y 0.5 * - "+b32), bi, fs), optSingleMode=false)

CombinePlanes(O, P1, P2, planes="YUV") }


# https://github.com/HomeOfVapourSynthEvolution/VapourSynth-BM3D/blob/7b3d8dd32e4aa4d962cf15d75d191b465682ed42/include/Specification.h#L176
function OPP_to_RGB (clip O, clip P1, clip P2, bool fulls) {

bi = BitsPerComponent(O)
fs = Default (fulls, false)

b32 = bi == 32 ? "" : "range_half -"

R = Expr(O, P1, P2, ex_dlut(Format("x y "+b32+" + z "+b32+" 0.666666666 * +"), bi, fs), optSingleMode=false)
G = Expr(O, P2, ex_dlut(Format("x y "+b32+" 1.333333333 * -"), bi, fs), optSingleMode=false)
B = Expr(O, P1, P2, ex_dlut(Format("x z "+b32+" 0.666666666 * + y "+b32+" -"), bi, fs), optSingleMode=false)

CombinePlanes(R, G, B, planes="RGB") }

[MysteryX]

Great. Convert from VapourSynth to Avisynth and then back to VapourSynth. That's what this community is really about!

Same with mClean. Convert from Avisynth to VapourSynth, to xClean and back to Avisynth.

[kedautinh12]

Great. Convert from VapourSynth to Avisynth and then back to VapourSynth. That's what this community is really about!

Same with mClean. Convert from Avisynth to VapourSynth, to xClean and back to Avisynth.

Waiting for your xclean avs ver :D

[MysteryX]

Thanks Godway (not a typo), you saved my life. Great performance! Converts 5K clip back and forth at over 70fps! Like 7-12x the speed of the other methods. Your OPP conversion is working just fine. Compared converting one way with your method and the other way with the built-in method. Same as source.

BM3D performs noticeably better in OPP... KNLMeans performs noticealy better in YCgCoR to avoid slight color bleeching.

[Dogway]

So fast, I don't think I can reach 70fps with my CPU.
Color space probably also affects the denoising, on the Z plane Rec709 is more contrasty and Rec2020 is fainter.
At some point I tried to normalize the chroma planes to the bitdepth dynamic range (as it's the luma plane) but results weren't nice.

[MysteryX]

What about converting straight from YUV to OPP, do you know the math to develop the matrix to do that?

[Dogway]

I investigated something related a month ago, I didn't go too deep but I concluded that while it's possible it involves many more multiplications so probably slower.

[MysteryX]

On my side, I might keep the reference clips in RGB to convert them to whatever formats I need

[tormento]

I ported them already but not sure it's correct since the OPP it's not documented, I simply ported from VS.
As MysteryX stated miracles from it, do you plan to apply the conversion inside your scripts too? :)

[Dogway]

Yes, at some point. That's my plan. But TransformsPack is still in beta (says RC though). I have fixed some/many issues since last update, I think I got the chroma placement equation right still need another night of careful tests, and still need to fix a pending color and clipping issue. Lastly reformat and merge all the color models into ConvertFormat() for easy control.
It's a lot of work to do but this is mid term project. I don't plan to add all color models or spaces, but many in any case.

[MysteryX]

If you see any bug in the code you sent me, let me know so that I update it.

[MysteryX]

Here's an image comparison of xClean with or without conversions to OPP for BM3D and YCgCoR for everything else. (https://slow.pics/c/LyTkrIrF)

[MysteryX]

DogWay did you write this ChromaReconstructor yourself (https://github.com/Dogway/Avisynth-Scripts/blob/master/TransformsPack.v1.0.RC22.avsi#L792)?

It's very different from feisty's version. Might offer both versions, wondering how to call this one.

[Dogway]

Yes, I realized it's not ChromaReconstructor_faster without nnedi3, but an optimized version of ChromaReconstructor() (with wild=false) and relaxed settings.

feisty2 sometimes goes overboard with the settings, hermite is enough for noring.
Supply Y, Uor and Vor, and then Combine Y, U and V.