Frame Convert Not Working

[MysteryX]

I'm trying to convert frame data from one format to another, and then back. The goal is for the result of both conversions to look like the original.

Here's my code so far
https://github.com/mysteryx93/AviSynthShader/blob/master/Convert.cpp

I execute this in AviSynth
ConvertToShader()
ConvertFromShader()

The first problem is that I convert the frame data. The frame width goes from 352 to 1408 (and from YV12 to RGB32 format). In ConvertFromShader, however, I still get a width of 352.

After further debugging, it appears the filters are not running in the order I would expect? First it runs ConvertFromShader, THEN it goes into ConvertToShader. Afterwards, when showing different frames in VirtualDub, it *only* runs ConvertToShader.

Why is this behaving so weirdly?

Also, to convert RGB data back to YUV, for each 2x2 pixel, I get 4 U values and 4 V values. What's the proper way of determining the right U and V value to use? A simple average of the 4?

Of course the code isn't yet working, but at least it doesn't crash! You're free to look into it.

It *should* converts YV12 data into float-precision RGB data between the 0-1 range, which is what HLSL texture shaders expect. So I don't need the standard 16-235 RGB range, I need 0-255 data converted into the 0-1 float range.

This conversion, if done properly with float precision, shouldn't affect quality much when converting back and forth. And if the conversion itself runs in an HLSL shader, it shouldn't cost much on performance either. For now it has to work on the CPU.

[MysteryX]

After closing everything and re-opening it back later, now it calls ConvertToShader and then ConvertFromShader properly.

However, in the 2nd method, vi.Width is still 352 instead of 352*4

Why isn't the output of the first filter being passed to the second filter?

In the 2nd filter's GetFrame method, if I simply return src, it crashes (because float-precision RGB isn't a valid format to display). If I write this, however, it doesn't crash. But the frame dimension isn't what it should be.

PVideoFrame src = child->GetFrame(n, env);
PVideoFrame dst = env->NewVideoFrame(viYV);
return dst;

[StainlessS]

As I cant see the class definition, so I cant see if you have also done a GetVideoInfo to return VideoInfo to following filter.

const VideoInfo& __stdcall GetVideoInfo() { return viRGB; }


Your class is probably returning VideoInfo from your source vi not viRGB, hence the wrong size.

Try to debug one filter at at time.

[MysteryX]

You can see all the files in the project on GitHub.

This is defined in avisynth.h

GenericVideoFilter(PClip _child) : child(_child) { vi = child->GetVideoInfo(); }

Shouldn't vi already contain the incoming clip without me having to request it?

Edit: Oh! I see what you're saying, it's querying a different method to get the information about the frame being returned

Adding that line to the declaration works :) Now "something" displays wide

[MysteryX]

Here you can see all project files. Now it almost returns a valid output. Sometimes it shows the valid image with 1/4 covered by something else, and sometimes the output is corrupt, but the conversion is being done back and forth properly. You can take a look, perhaps you'd see what's wrong with it. The upper-left square is probably just writing at the wrong memory address, but why the output becomes invalid when seeking backwards is beyond me. Any idea?
https://github.com/mysteryx93/AviSynthShader

I also did a performance test for this very unoptimized CPU conversion with AVSMeter.

This script runs at 200fps with 80% CPU usage. Even with this, I won't notice the difference when the whole process runs at 4-6fps! If someone wants to do it for live playback, however, it could be optimized.

SetMTMode(3,8)
LoadPlugin("C:\GitHub\AviSynthShader\Debug\Shader.dll")
AviSource("F:\AVSMeter\Preview.avi", audio=false, pixel_type="YV12")
SetMTMode(2)
ConvertToShader()
ConvertFromShader()

Running the conversion 5 times in a row gives 46fps with 93% CPU usage.

It's hard to test only 1 because I can't preview the first output because it's invalid to display, and I can't run the 2nd one without feeding data from the first one. So I have to run both to test anything. Then afterwards, I can start manipulating the float data with shaders and see the output.

I don't know why the upper-left 1/4 of the image is corrupt, but the 3/4 of it look good. Colors seem OK. I'm not seeing much visible distortion from the conversion. For U and V, I'm simply doing the average of the 4 pixels for now, not sure if it's the right way.

Something is wrong with the synchronization. I don't get the same output for the same position between the source file and the processed script. Source frame 4522 = dest frame 4394

[MysteryX]

Here's something to give a better idea in terms of performance.

Converting to RGB with float data betwen 0 and 1, back and forth 10 times, runs at 21fps with 92% CPU usage.

If I remove the actual conversion and simply store Y, U and V as float data, I get... 23.5fps! The float conversion itself isn't costing much. It seems the unoptimized loops are what cost performance.

[MysteryX]

This script will allow testing the image quality for conversion distortion. Right now it's not working at all because the source and destination frames aren't in sync. StainlessS, could you take a look at my code and see what's wrong with it?

TestQuality()
function TestQuality(clip input)
{
original = input
input = ConvertToShader(input)
input = input.ConvertFromShader()
return mt_makediff(input, original, chroma="process")
}

As a side note, once the conversion back and forth is working, the rest should be simple. Creating and running the DX9 texture shader. Copying the data back and forth will be very simple. Perhaps configuring the shaders will be the only thing that would require work.

[StainlessS]

Where is shader.h

[MysteryX]

You don't need Shader.h. I just added the Shader file with most of the code required to use DirectX HLSL shaders. It copies the data back and forth, creates the DX instance and texture.

All that's left is creating the shader object, which will require first pre-compiling the HLSL code. Then, passing the data it needs. Not much left to do.

[StainlessS]

Doom9 went down in middle of EDIT

Use Avisynth v2.6 VERSION 6 header not OLD version 5 (will have problems in older versions of v2.5, Pre Alpha 4)

Dont think you will need internal.h, not sure.

VS 2008 dont like below

Changed in class definition

// const int bytesPerPixel = 16; // ssS added static
const int bytesPerPixel;


Added in constructor initialization list

bytesPerPixel(16)


Change (YV12 includes IsPlanar)

if (!vi.IsPlanar() || !vi.IsYV12())


To

if (!vi.IsYV12())




#include "Convert.h"

// This page provides a good overview of YUV to RGB conversion.
// http://www.fourcc.org/fccyvrgb.php
// In this case, we want 0-255 RGB values into a 0-1 float, which is what HLSL texture shaders expect.
// There may be color matrix distortion (interpreting Rec709 data in Rec601), but converting back with
// the same formula will cancel most of the distortion. There might be color distortion on the effect of the shader.

ConvertToShader::ConvertToShader(PClip _child, IScriptEnvironment* env) :
GenericVideoFilter(_child), bytesPerPixel(16) { // ssS
// if (!vi.IsPlanar() || !vi.IsYV12())
if (!vi.IsYV12()) // ssS, YV12 includes Planar
env->ThrowError("Source must be YV12");

// Convert from YV12 to float-precision RGB
viRGB = vi;
viRGB.pixel_type = VideoInfo::CS_BGR32;
viRGB.width <<= 2; // RGBA is 4-byte per pixel, float-precision RGB is 16-byte per pixel (4 float). We need to increase width to store all the data.
}

ConvertToShader::~ConvertToShader() {
}

PVideoFrame __stdcall ConvertToShader::GetFrame(int n, IScriptEnvironment* env) {
PVideoFrame src = child->GetFrame(n, env);

// Convert from YV12 to float-precision RGB
PVideoFrame dst = env->NewVideoFrame(viRGB);
conv420toFloatRGB(src->GetReadPtr(PLANAR_Y), src->GetReadPtr(PLANAR_U), src->GetReadPtr(PLANAR_V),
dst->GetWritePtr(),
src->GetPitch(PLANAR_Y), src->GetPitch(PLANAR_U),
dst->GetPitch(),
vi.width, vi.height);
return dst;
}

void ConvertToShader::conv420toFloatRGB(
const byte *py, const byte *pu, const byte *pv, // src
unsigned char *dst, // dst
int pitch1Y, int pitch1UV, // src
int pitch2, // dst
int width, int height) // src
{
width >>= 1;
height >>= 1;
int Y1, Y2, Y3, Y4, U, V;
for (int y = 0; y < height; ++y) {
for (int x = 0; x < width; ++x) {
Y1 = py[x << 1];
U = pu[x];
Y2 = py[(x << 1) + 1];
V = pv[x];
// pixels on 2nd row
Y3 = py[(x << 1) + pitch1Y];
Y4 = py[(x << 1) + pitch1Y + 1];

// ssS (x << 3) to (x << (1 + 4)) ie (x*32) : 32 bytes per pair of Y or 2*bytesPerPixel
convFloat(Y1, U, V, &dst[(x << (1+4))]);
convFloat(Y2, U, V, &dst[(x << (1+4)) + bytesPerPixel]);
convFloat(Y3, U, V, &dst[(x << (1+4))] + pitch2);
convFloat(Y4, U, V, &dst[(x << (1+4))] + pitch2 + bytesPerPixel);
}
py += pitch1Y*2; // ssS, we processed 2*Y vertical already
pu += pitch1UV;
pv += pitch1UV;
dst += pitch2*2; // ssS, we processed 2*Y vertical already
}
}

// Using Rec601 color space. Can be optimized with MMX assembly or by converting on the GPU with a shader.
void ConvertToShader::convFloat(int y, int u, int v, unsigned char* out) {
float r, g, b;
//r = y + (1.370705f * (v - 128));
//g = y - (0.698001f * (v - 128)) - (0.337633f * (u - 128));
//b = y + (1.732446f * (u - 128));
r = y + 1.403f * u;
g = y - 0.344f * u - 0.714f * v;
b = y + 1.770f * u;

if (r > 255) r = 255;
if (g > 255) g = 255;
if (b > 255) b = 255;
if (r < 0) r = 0;
if (g < 0) g = 0;
if (b < 0) b = 0;

// Texture shaders expect data between 0 and 1
r = r / 255 * 1;
g = g / 255 * 1;
b = b / 255 * 1;
/*
// Empty alpha channel
out[0] = out[1] = out[2] = out[3] = 0;
// Store RGB
memcpy(out + 4, &b, sizeof(float));
memcpy(out + 8, &g, sizeof(float));
memcpy(out + 12, &r, sizeof(float));
*/
// ssS, all below changed, B,G,R,A
// Store BGRA
memcpy(out + 0, &b, sizeof(float));
memcpy(out + 4, &g, sizeof(float));
memcpy(out + 8, &r, sizeof(float));
// Empty alpha channel
out[12] = out[13] = out[14] = out[15] = 0;

}


ConvertFromShader::ConvertFromShader(PClip _child, IScriptEnvironment* env) :
GenericVideoFilter(_child), bytesPerPixel(16) { // ssS
// if (vi.IsPlanar() || !vi.IsRGB32())
if (!vi.IsRGB32()) // ssS, RGB32 and Planar are mutually exclusive
env->ThrowError("Source must be float-precision RGB");

// Convert from float-precision RGB to YV12
viYV = vi;
viYV.pixel_type = VideoInfo::CS_YV12;
viYV.width >>= 2; // Float-precision RGB is 16-byte per pixel (4 float), YV12 is 2-byte per pixel
}

ConvertFromShader::~ConvertFromShader() {
}


PVideoFrame __stdcall ConvertFromShader::GetFrame(int n, IScriptEnvironment* env) {
PVideoFrame src = child->GetFrame(n, env);

// Convert from float-precision RGB to YV12
PVideoFrame dst = env->NewVideoFrame(viYV);


// ssS, think below all mixed up, source is actually RGB and not planar, dest is PLANAR
convFloatRGBto420(
src->GetReadPtr(PLANAR_Y), src->GetWritePtr(PLANAR_U), src->GetWritePtr(PLANAR_V),
dst->GetWritePtr(),
src->GetPitch(PLANAR_Y), src->GetPitch(PLANAR_U),
dst->GetPitch(),
viYV.width, viYV.height);
return dst;
}

// END OF MODS SO FAR

void ConvertFromShader::convFloatRGBto420(const byte *src, unsigned char *py, unsigned char *pu, unsigned char *pv,
int pitch1, int pitch2Y, int pitch2UV, int width, int height)
{
width >>= 1;
height >>= 1;
unsigned char U[4], V[4];
for (int y = 0; y < height; ++y) {
for (int x = 0; x < width; ++x) {
convFloat(&src[(x << 3)], &py[x << 1], &U[0], &V[0]);
convFloat(&src[(x << 3) + bytesPerPixel], &py[(x << 1) + 1], &U[1], &V[1]);
convFloat(&src[(x << 3) + pitch1], &py[(x << 1) + 1], &U[2], &V[2]);
convFloat(&src[(x << 3) + pitch1 + bytesPerPixel], &py[(x << 1) + pitch2Y + 1], &U[3], &V[3]);
pu[x] = (U[0] + U[1] + U[2] + U[3]) / 4;
pv[x] = (V[0] + V[1] + V[2] + V[3]) / 4;
}
src += pitch1;
py += pitch2Y;
pu += pitch2UV;
pv += pitch2UV;
}
}

// Using Rec601 color space. Can be optimized with MMX assembly or by converting on the GPU with a shader.
void ConvertFromShader::convFloat(const byte* src, byte* outY, unsigned char* outU, unsigned char* outV) {
float r, g, b;
memcpy(&b, &src[4], sizeof(float));
memcpy(&g, &src[8], sizeof(float));
memcpy(&r, &src[12], sizeof(float));

// rgb are in the 0 to 1 range
r = r / 1 * 255;
b = b / 1 * 255;
g = g / 1 * 255;

float y, u, v;
y = 0.299f * r + 0.587f * g + 0.114f * b;
u = (b - y) * 0.565f;
v = (r - y) * 0.713f;

if (y > 255) y = 255;
if (u > 255) u = 255;
if (v > 255) v = 255;
if (y < 0) y = 0;
if (u < 0) u = 0;
if (v < 0) v = 0;

// Store YUV
outY[0] = unsigned char(y);
outU[0] = unsigned char(u);
outV[0] = unsigned char(v);
}


Time for bed.

zzzzzzzzzzzzZZZZZZZZZZ.

EDIT: Hard to compile without shader.h

[MysteryX]

The project is easier to compile by excluding Shader.h and Shader.cpp as those require the DirectX SDK Framework. I'll look at what you wrote here tomorrow.

[Wilbert]

Also, to convert RGB data back to YUV, for each 2x2 pixel, I get 4 U values and 4 V values. What's the proper way of determining the right U and V value to use? A simple average of the 4?
http://avisynth.nl/index.php/Sampling

[MysteryX]

http://avisynth.nl/index.php/Sampling
Thanks. There will be various tweaks in there to reduce conversion distortion.

StainlessS, I have implemented the changes you suggested. See updated files online. However, right now it works even less than before. Btw I'm not sure I understand the (1+4) part.

We might as well code it right away to either work with half-float or float data with a parameter.

For the byte orders, D3DFORMAT documentation says this. Are we implementing it properly?
All formats are listed from left to right, most-significant bit to least-significant bit. For example, D3DFORMAT_ARGB is ordered from the most-significant bit channel A (alpha), to the least-significant bit channel B (blue). When traversing surface data, the data is stored in memory from least-significant bit to most-significant bit, which means that the channel order in memory is from least-significant bit (blue) to most-significant bit (alpha).

Here's the documentation about shader texture formats. Right now I was looking at the D3DFMT_A32B32G32R32F format, but it can run with any of these formats.
https://msdn.microsoft.com/en-us/library/windows/desktop/bb172558%28v=vs.85%29.aspx

[MysteryX]

One way to test one conversion at a time is to change it to convert to regular RGB for now so that the output can be viewed right away.

It does look like there is still quite a bit of work to get the conversion to work properly and implementing the proper chroma upscaling/downscaling methods. Even if we ignore interlaced videos for now.

StainlessS, would you have time to look into this conversion while I'll work on the DirectX/Shader part?

The conversion could be done in a couple of ways
- Standard RGB to preview output
- Float RGB
- Half-float RGB
- Passing YUL values instead of RGB values to let a shader to the conversion.

Btw, GitHub is really cool for working on projects like this. You can download the updates, make your changes, and submit your changes. It takes a bit of time to learn but once it's running it works well. It's hard to use without a proper interface, but TurtoiseGit(https://code.google.com/p/tortoisegit/) makes it easy.

How to support the various convert formats will be easy: the confFloat function can have a param (precision), when 1 it stores as byte, when 2 as half-float, when 4 as float.

[MysteryX]

I added the Precision parameter which will make it easier to debug.
https://github.com/mysteryx93/AviSynthShader/blob/master/Convert.cpp

Precision can be
0 = YUV, 1 byte per channel (to convert in a shader)
1 = RGB, 1 byte per channel (valid for display)
2 = Half-Float, 2 byte per channel (better performance)
4 = Float, 4 byte per channel (better quality)

When convert with Precision=1, it is still completely corrupt, but at least it is consistent when seeking forward and backwards. And it can be displayed without converting back.

[MysteryX]

OK it's working now :) at least the basic up-side-down version of it. Thanks guys!

[MysteryX]

http://avisynth.nl/index.php/Sampling
I tried going through that document and I really don't understand it.

Could someone summarize in plain simple terms how to convert YV12 to RGB.

For every 2x2 pixels, I have Y1, Y2, Y3, Y4, U, V. I need to know the right U and V for each Y.

When converting back, for every 2x2 pixels, I have Y1, Y2, Y3, Y4, U1, U2, U3, U4, V1, V2, V3, V4. I need to have single U and V values for all 4 Y.

I don't want to bother with YUY2 or interlaced content for now. I'm more concerned about not losing quality when converting back and forth than by performance.

The conversion code is here.
https://github.com/mysteryx93/AviSynthShader/blob/master/Convert.cpp

[Gavino]

The simplest solution for you is to use YV24 as an intermediate format, which will handle all the chroma subsampling issues.
Use ConvertToYV24() /ConvertToYV12() to convert YV12 <-> YV24, either in a script or by using env->Invoke() inside your plugin.
Conversion YV24 <-> RGB is simple, using the formulae you have already discovered.

[MysteryX]

OK that works. When I do mt_makediff between the source and the image converted back and forth, there is still considerable distortion, especially the edge of subtitles and other sharp edges. Is there a way to reduce that? The distortion is less when I convert to float RGB than to byte RGB, but it's still definitely there.

Also when converting from YUV to RGB, sometimes the equations truncate the results between 0-255 so it cannot go back to the same original value. Any thoughts on this?

[MysteryX]

By having the chroma upsampling/downsampling separate from the YUV-RGB conversion, I'm able to see which of these 2 conversions is causing quality loss.

Chroma upsampling/downsampling only causes a difference with subtitles and with the logo: big bright objects that stand out. Besides that, there doesn't seem to be much difference at all in the video itself.

So most of the distortion is in the YUV-RGB conversion itself. Perhaps I need to find a better conversion formula?

I'm really confused by the formulas given on Wikipedia, but I tried the 1st and 2nd formulas on this page.
http://www.fourcc.org/fccyvrgb.php

The 2nd formula wasn't working for me. The 1st one works. But there is quality loss.

As for "correct" formulas, they give results in the 16-235 range while I want to have a full range between 0 and 1. Any inputs as to which formula I should try to minimize distortion? I'm doing this to improve the image quality of my videos; I don't want to scap these benefits in the conversion.

And then there's the issue of Rec709 vs Rec601.

[Groucho2004]

As for "correct" formulas, they give results in the 16-235 range while I want to have a full range between 0 and 1. Any inputs as to which formula I should try to minimize distortion? I'm doing this to improve the image quality of my videos; I don't want to scap these benefits in the conversion.

And then there's the issue of Rec709 vs Rec601.
Have you tried "Dither_convert_yuv_to_rgb" and "Dither_convert_rgb_to_yuv" from cretindesalpes' Dither package? I think he also provides the source code.

[MysteryX]

Why haven't C programmers adopted proper commenting standards? Most of the code I've seen so far has no or few comments, and no structure in the comments.

C# code is generally commented in a standardized way on every classes and methods. C and C++ would need that even more.

But thanks for the link to DitherTools :)

[Groucho2004]

Why haven't C programmers adopted proper commenting standards? Most of the code I've seen so far has no or few comments, and no structure in the comments.

C# code is generally commented in a standardized way on every classes and methods. C and C++ would need that even more.
How is C# code generally commented in a "standardized" way? Is there some magic involved that makes the code comment on itself?

Also, why would C/C++ code need commenting more than other languages?

[MysteryX]

How is C# code generally commented in a "standardized" way? Is there some magic involved that makes the code comment on itself?

Also, why would C/C++ code need commenting more than other languages?
In C#, you type /// above a method and it auto-generates the tags to document the method, each parameter, and the return type. All the information written there automatically shows up in the intellisense. Not everybody uses this all the time, but most tend to do most of the time.

But C# in itself is a higher-level language that tends to be self-explanatory. Especially as the language evolved, functions that would previously require 10 lines of code can now often be written in a single line of code. If the line is complex, then a simple comment above it says what it does.

C, C++ and ASM are low-level languages that require a lot of code to perform operations. It's harder to have an overview of what each function and class does. The functions tend to be related to the implementation of the details rather than to the end feature itself.

Thinking of it, the way this sort of thing would "normally" be done in C# is to have all the conversion code encapsulated into a generic class, and then the AviSynth filter would simply call those generic classes.

But then, integrating the conversion into the logic of AviSynth's structure instead of feeding a generic class with the data does provide greater performance.

I tried copying some code from DitherTools to convert and it's not working at all right now... properly encapsulating the conversion really would make it much simpler to use, where the conversion code would know nothing about the planes.

Oh, I finally understand what "Scale in Linear Light" means! YUL-RGB conversion follows a gamma curve, and Linear Light means not applying that curve in the conversion. Perhaps in this case Linear Light would make more sense than standard conversion.

[Groucho2004]

In C#, you type /// above a method and it auto-generates the tags to document the method, each parameter, and the return type. All the information written there automatically shows up in the intellisense.
Ah, so it's something the IDE provides and has little to do with the language itself, right?

But C# in itself is a higher-level language that tends to be self-explanatory. Especially as the language evolved, functions that would previously require 10 lines of code can now often be written in a single line of code.
C/C++ is high level enough for me, thank you. I'm not programming professionally and can therefore take all the time I need. C/C++ gives me the best performance/convenience ratio.

[MysteryX]

Ah, so it's something the IDE provides and has little to do with the language itself, right?
It's in the way people use it. I'm seeing much more standardized comments in C#, and much more plain comment-less complex code files in C++, where they assume I should just "know what it does". There's nothing preventing C++ programmers from commenting in the same way as C#.

C/C++ is high level enough for me, thank you. I'm not programming professionally and can therefore take all the time I need. C/C++ gives me the best performance/convenience ratio.
C++ is definitely better suited for this kind of stuff: working with video frames, direct memory access and DirectX. That sort of stuff is hard to do in C#, and C++ does have better performance, especially for this, and especially when adding assembly optimization.

Yet, I've seen C# code that grows into a very stable and scaleable platform with surprisingly high performance, and I've seen C++ code become so bloated with unknown and unscaleable code that it degrades performance and can't be maintained or upgraded. The opposite can also be found (badly written C# vs well-written C++). I've heard that AviSynth 2.6 is kind of in that zone of being so bloated with code that it's unmanageable.

Getting back to the YUV-RGB conversion, if the code I have doesn't provide the best quality and the other code can't easily be copied over, I might as well just look for conversion code purely written in HLSL, pass the YUL values to the shader and let the GPU do the conversion.

[Groucho2004]

There's nothing preventing C++ programmers from commenting in the same way as C#.
I was not aware that C# programmers write more/better comments than C/C++ programmers.

[MysteryX]

I was not aware that C# programmers write more/better comments than C/C++ programmers.
Just looking at Matrix2020CLProc.cpp in DitherTools which contains the code that interest me. It has about 10 lines of comments to help interpretation in a file of 1000 lines of code, and these are short comments such as
// RGB -> Y

Not hard to do better than that :)

[cretindesalpes]

I tried copying some code from DitherTools to convert and it's not working at all right now... properly encapsulating the conversion really would make it much simpler to use, where the conversion code would know nothing about the planes.
This was not the goal. If I wanted to make properly documented for others and reusable code, I’d be still writing it (well, I also have a VS macro to generate comment blocks for functions, but if they are not filled with relevant information which takes time, they are absolutely useless). The code is readable and maintainable for me, which is just what I need.

The RGB <-> YUV conversion in dither is a kind of dirty hack using the resizer’s convolution to perform a matrix-vector multiplication. The code is split between dither.avsi and dither.dll and is just a mess. Don’t use it. If you want example code, you’d better check the matrix function in fmtconv for Vapoursynth (supports floating point data with 0–1 / -0.5–+0.5 ranges) or the zimg library. For formula reference, read the ITU-R Rec BT.709.

[MysteryX]

This was not the goal. If I wanted to make properly documented for others and reusable code, I’d be still writing it
I personally find that to save a LOT of time on the long run.

The RGB <-> YUV conversion in dither is a kind of dirty hack using the resizer’s convolution to perform a matrix-vector multiplication. The code is split between dither.avsi and dither.dll and is just a mess. Don’t use it.
Thanks for letting me know :)

Btw, here's what C# code commenting looks like. Sometimes private methods are skipped to save time, but ideally it would all be documented. That code needs to be encapsulated into its own library.
https://github.com/mysteryx93/NaturalGroundingPlayer/blob/master/Business/DownloadBusiness.cs

[MysteryX]

cretindesalpes, have you done a diff before/after converting back and forth to see whether your implementation loses details? How much data is being lost with your code?

Just to know whether it's possible to do so without losing much data.

And do you think I'm better to not have the gamma curve in the conversion (linear light) for doing this back-and-forth conversion?

[cretindesalpes]

No. You’ll have to do the measures yourself and check if it is OK for your requirement (how many back and forth? How is the quality assessed?) It mainly depends on several parameters:

– chroma subsampling or not. You'd better work in 4:4:4 if you do several conversions between RGB and YUV (but this double the data to process in the YUV world). Or use a large kernel for the chroma resampling, like 16-tap Blackman, but this will cause ringing in your intermediate steps.

– data type: float > 16-bit int > 8-bit int

– dithering after each conversion step if your main data type is integer.

For the OETF/EOTF curve, it mostly depends on what you do with the RGB data. Anyway, 8-bit int are not accurate enough for linear light, you’ll need a bigger data type.

[MysteryX]

Shaders work with float. I can either work with 4-byte float, or 2-byte half-float. I don't know which would be better but that can be tested for performance/quality difference.

This is to pass to HLSL shaders. If you run several shaders in a row, then you only need to convert once at the beginning and then back at the end. If you want to do Shader, EEDI3, Shader, however, then you need to convert to float for the shader, back to standard data for EEDI3, and then a 2nd time for the 2nd shader. I don't think this conversion back and forth will happen more than twice in a script.

But within the logic of a shader, there may be several conversions necessary but all with floats.

If you say Linear Light depends on the usage, then I guess the ideal would be to make it configurable depending which code they want to run.

[MysteryX]

huh... it just occurred to me that if I remove cropping in the formula I was already using, almost all the conversion distortion is gone. Not sure how shaders will perform with out-of-range data.