Doom9's Forum - View Single Post - Shader implementation of the NEDI algorithm

Shiandow · 21st June 2014, 20:06

Oddly enough those seem to contain a typo. In the line "QUAD_REAL g = YUV.x - 0.344*u - 1.403*v" the number 1.403 seems to have been copied from the previous line, it should be somewhere around 0.714. Also the coefficients have been rounded to 3 decimal places which is less accurate than what I was using.

As far as I can tell I'm just using the wrong colour space somehow, I suspect it has something to do with the video using different primaries, which MadVR corrects but the YCbCrtoRGB shader does not. I'll try to see if I can find a quick fix, but I'd rather not spend too much time trying to copy the entire MadVR colour processing chain.

21st June 2014, 20:06	#101 \| Link
Shiandow Registered User Join Date: Dec 2013 Posts: 753	Oddly enough those seem to contain a typo. In the line "QUAD_REAL g = YUV.x - 0.344u - 1.403v" the number 1.403 seems to have been copied from the previous line, it should be somewhere around 0.714. Also the coefficients have been rounded to 3 decimal places which is less accurate than what I was using. As far as I can tell I'm just using the wrong colour space somehow, I suspect it has something to do with the video using different primaries, which MadVR corrects but the YCbCrtoRGB shader does not. I'll try to see if I can find a quick fix, but I'd rather not spend too much time trying to copy the entire MadVR colour processing chain.