Shader implementation of the NEDI algorithm

[pie1394]

Originally Posted by Shiandow

I used 16 neurons although using 128 neurons doesn't result in a noticeable difference. In this particular image I might actually prefer the way NEDI is a bit softer, it looks somewhat more natural. The extra sharpness of NNEDI3 looks nice at first glance but when fine tuning the NEDI algorithm I've encountered (admittedly rare) cases when what I thought was an artefact of NEDI turned out to be some detail that NNEDI3 decided to erase.

To me the softer result by your NEDI looks like a female with cosmetics on the face -- if compared to BiCubic75AR version.

Yet I don't know if the original contents should look exactly like this, or other sharper algorithms add more artifical enhancements to make it look more pop / stereo...

[Shiandow]

I've discovered that the ChromaNEDI-III shader didn't actually use any edge information so it has been replaced by one that does.

Edit: Quick before/after.

[madshi]

Originally Posted by Shiandow

I used 16 neurons although using 128 neurons doesn't result in a noticeable difference. In this particular image I might actually prefer the way NEDI is a bit softer, it looks somewhat more natural. The extra sharpness of NNEDI3 looks nice at first glance but when fine tuning the NEDI algorithm I've encountered (admittedly rare) cases when what I thought was an artefact of NEDI turned out to be some detail that NNEDI3 decided to erase.

FWIW, if you downscale a sharp image 50% and then upscale it again with NNEDI3, the result is soft compared to the original image.

Originally Posted by Shiandow

What I find somewhat frustrating is that, so far, NEDI has looked good on all video content I've tried. But I'll admit that it has some quirks, I'll try to see if I can iron these out.

Ok, let me know when you're finished with your optimizations, then I'll give it a try with some videos.

Originally Posted by Shiandow

I've discovered that the ChromaNEDI-III shader didn't actually use any edge information so it has been replaced by one that does.

Does that improve image quality visibly?

[Shiandow]

Originally Posted by madshi

Does that improve image quality visibly?

I've added a quick example to my previous post. It seems to have removed some ringing and aliasing.

The previous shader wasn't capable of detecting the direction of an edge so it was basically the same as a box filter except it multiplied the result by some value which depended on whether there was an edge or not. This did actually prevent some 'bleeding' but it was unstable and caused some artefacts. The current shader can detect the direction of an edge and tries to interpolate along that direction.

[Shiandow]

On what channel does MadVR perform nnedi3 resizing in YCbCr mode? Because if I use the chromaNEDI shaders and add a pre-resize shader that converts to RGB the resulting image is quite noticeably better: resized from YCbCr / resized from RGB.

[madshi]

You mean when luma doubling is enabled and chroma doubling is disabled?

[Shiandow]

Yes, I've configured it to use NNEDI3 for luma and Jinc3AR for chroma, except it seems treat the YCbCr image as if it were RGB which causes it to apply NNEDI3 on some other part of the colour space.

[madshi]

madVR first converts everything to RGB. Then if you enable luma doubling but disable chroma doubling, the image is converted to YCbCr just before scaling is done (always using BT.709 for that, IIRC), then NNEDI3 is performed on the Y channel, only, then chroma is upscaled with a different algorithm, then everything is converted back to RGB right after scaling. madVR runs custom pixel shaders always in RGB.

[huhn]

Originally Posted by madshi

madVR first converts everything to RGB. Then if you enable luma doubling but disable chroma doubling, the image is converted to YCbCr just before scaling is done (always using BT.709 for that, IIRC), then NNEDI3 is performed on the Y channel, only, then chroma is upscaled with a different algorithm, then everything is converted back to RGB right after scaling. madVR runs custom pixel shaders always in RGB.

have you think about using YCoCg for this not YCbCr?
it can be used lossless for rgb and it's super fast too. it needs more bits than the source has and i don't have a clue how to use it with 16 bit. the Y channel doesn't change the bit deep in the 10 bit rgb example:

http://wiki.multimedia.cx/index.php?title=YCoCg

[nevcairiel]

Originally Posted by huhn

have you think about using YCoCg for this not YCbCr?
it can be used lossless for rgb and it's super fast too. it needs more bits than the source has and i don't have a clue how to use it with 16 bit. the Y channel doesn't change the bit deep in the 10 bit rgb example:

http://wiki.multimedia.cx/index.php?title=YCoCg

If you're doing this in floating point back and forth, there is no loss in doing this in YCbCr. Its just math.

[huhn]

Originally Posted by nevcairiel

If you're doing this in floating point back and forth, there is no loss in doing this in YCbCr. Its just math.

but the result is "only" stored in 16 bit not float point so this should be faster and more precision.

[nevcairiel]

Originally Posted by huhn

but the result is "only" stored in 16 bit not float point so this should be faster and more precision.

Its 16-bit float to my knowledge.

[madshi]

Lossless conversion between RGB <-> YCbCr is useless for me. I'm calculating in 32bit+ float and storing results in 16bit+ int/float, and final output is only 8bit. So there's gigantic headroom, no need for lossless conversion. It's *MUCH* more important to use a conversion that matches human perception. So BT.709 is a much better choice than YCgCo in this specific situation.

[huhn]

i only ask this to understand this better.

It's *MUCH* more important to use a conversion that matches human perception

if a RGB is lossless transform to YCgCo doesn't that mean it doesn't matter what colorspace RGB was before? it's the same colorspace again after the YCbCr -> RGB as long as the MATH done with it doesn't change the colorspace?
i don't understand this. i'm just missing the knowledge.

[nevcairiel]

His point is that he does processing on the luma plane after the color decorrelation, and for that to be really useful, this luma plane should match what we humans perceive as luma the most closely.
YCgCo was designed with compression efficiency and ease of computation in mind, not to match the human perception.

[huhn]

i never thought there is a different between luma in YCbCr and YCgCo. so luma information aren't simply luma informations good to know.
enough off topic from me i'm sorry.

[madshi]

Originally Posted by nevcairiel

His point is that he does processing on the luma plane after the color decorrelation, and for that to be really useful, this luma plane should match what we humans perceive as luma the most closely.
YCgCo was designed with compression efficiency and ease of computation in mind, not to match the human perception.

Exactly.

I could also scale RGB directly with NNEDI3. I'm not converting to YCbCr because I *have* to. I do that only because I get a better quality/performance ratio if I apply NNEDI3 only on the information which is most important to the human eye (= luma information). Scaling directly in RGB is 3x slower than scaling only the Luma channel.

[Shiandow]

Minor change to chromaNEDI-III: the line "//a /= dot(a,1)" shouldn't be commented out after all, doing so causes too many artefacts. The same line in chromaNEDI-I and -II doesn't seem to cause problems.

[Asmodian]

Originally Posted by madshi

I could also scale RGB directly with NNEDI3. I'm not converting to YCbCr because I *have* to. I do that only because I get a better quality/performance ratio if I apply NNEDI3 only on the information which is most important to the human eye (= luma information). Scaling directly in RGB is 3x slower than scaling only the Luma channel.

To make sure I remember correctly; if one wants NNEDI3 to scale RGB directly, without a RGB->YCbCr->RGB conversion, one simply needs to set both madVR's luma and chroma doubling to the same neuron value? Maybe this would be a better way to compare the algorithms? Though it does make NNEDI3 even slower in comparison.

[madshi]

I always do the RGB->YCbCr->RGB conversion because you usually want to use more neurons for luma than for chroma, even if you enable both luma+chroma doubling.