Shader implementation of the NEDI algorithm

[Asmodian]

Originally Posted by madshi

I always do the RGB->YCbCr->RGB conversion because you usually want to use more neurons for luma than for chroma, even if you enable both luma+chroma doubling.

Wow, I had badly misread your explanation.

Of course separate luma and chroma doubling (only doubling luma) has always been better in my tests but if it is low effort maybe a special case for the future when we can all run NNEDI3 256 Luma + Chroma doubling? Or does the math workout the same either way?

Originally Posted by Shiandow

On what channel does MadVR perform nnedi3 resizing in YCbCr mode? Because if I use the chromaNEDI shaders and add a pre-resize shader that converts to RGB the resulting image is quite noticeably better: resized from YCbCr / resized from RGB.

From my limited understanding there would be a difference between NNEDI3 or NEDI on R, G, and B vs NNEDI3 or NEDI on Y, U, V converted to RGB. The HVS important luma information is more connected to R, G, and B than it is to U or V and both NNEDI3 and NEDI might be sort of human like as far as edge recognition goes? Also the source usually starts with half resolution U and V. Combining the chroma with the higher resolution luma information (via YUV -> RGB) might result in sharper easier to recognize edges for NNEDI3 or NEDI? Of course there is no longer a luma channel so all channels would have low resolution chroma mixed into them but your example seems to say RGB is better.

[madshi]

Originally Posted by Asmodian

if it is low effort maybe a special case for the future when we can all run NNEDI3 256 Luma + Chroma doubling? Or does the math workout the same either way?

I think the math would either work out the same, or if it does not, YCbCr might be better than RGB. Just a guess, though.

Originally Posted by Shiandow

On what channel does MadVR perform nnedi3 resizing in YCbCr mode? Because if I use the chromaNEDI shaders and add a pre-resize shader that converts to RGB the resulting image is quite noticeably better: resized from YCbCr / resized from RGB.

Hmmmm... I'm not sure what you mean exactly with this. The pre-resize shader should already run in RGB. All of madVR's custom pixel shaders (both pre-resize and post-resize) are always fed with RGB data. So why would you convert to RGB, if the data is already RGB? Can you list the exact processing chain for both of these images?

[Shiandow]

Originally Posted by madshi

I think the math would either work out the same, or if it does not, YCbCr might be better than RGB. Just a guess, though.

Since NNEDI3 is nonlinear the math doesn't exactly work out to be the same. I think all other algorithms in MadVR (except spline) are linear so in those cases it doesn't really matter which colour space you use (unless you change the gamma).

Originally Posted by madshi

Hmmmm... I'm not sure what you mean exactly with this. The pre-resize shader should already run in RGB. All of madVR's custom pixel shaders (both pre-resize and post-resize) are always fed with RGB data. So why would you convert to RGB, if the data is already RGB? Can you list the exact processing chain for both of these images?

I fear I've been slightly unclear. This is happened when I put an empty file called YCbCr in the MadVR folder, and used NNEDI3 for luma doubling and Jinc3AR for image upscaling. It seems that MadVR skips the YCbCr -> RGB conversion but doesn't skip the RGB -> YCbCr conversion after the pre-resize pixel shaders. Which means that it's performing NNEDI3 on the wrong colour space which lowers the picture quality.

It seems that this is also the reason I thought NEDI looked superior, and it may be the reason that improving chroma upscaling seemed to slightly improve the luma upscaling in some of those images.

[madshi]

Originally Posted by Shiandow

Since NNEDI3 is nonlinear the math doesn't exactly work out to be the same.

Yeah, it won't be identical, but probably in the end it will look similar.

Originally Posted by Shiandow

I fear I've been slightly unclear. This is happened when I put an empty file called YCbCr in the MadVR folder, and used NNEDI3 for luma doubling and Jinc3AR for image upscaling. It seems that MadVR skips the YCbCr -> RGB conversion but doesn't skip the RGB -> YCbCr conversion after the pre-resize pixel shaders. Which means that it's performing NNEDI3 on the wrong colour space which lowers the picture quality.

Oooh, I think you're right! Sorry about that. I had implemented the "YCbCr" empty file hack just a long time ago for testing purposes and when implementing the RGB->YCbCr->RGB conversions for NNEDI3 I didn't think of that the "YCbCr" would need special treatment. So it seems NNEDI3 image doubling doesn't work well with the "YCbCr" empty file hack atm. Will put that on my to do list to fix...

[Shiandow]

It seems that the current versions of the ChromaNEDI shaders only use the directional information very weakly, apparently this is still enough to outperform most other algorithms but it causes a loss of detail. I think I've been able to fix this using a few of the same tricks I used in the NEDI shaders (and then some). I'll post these improved versions and a couple of comparisons when I have time, they still require some fine-tuning.

Edit: Later I'll post a more detailed explanation of what I changed but first some comparisons between the new and old method:
(Chroma only)
New method
Old method
Bicubic

With luma

One thing I find worrying is that Bicubic seems to be shifted slightly to the left, I've been unable to confirm if this is correct. For my implementation I assumed that the chroma channel corresponded to the average of each 2x2 block.

[pie1394]

Originally Posted by Shiandow

Edit: Later I'll post a more detailed explanation of what I changed but first some comparisons between the new and old method:
(Chroma only)
New method
Old method
Bicubic

With luma

Personal opinions ...

[High-frequency details]
New method > Bicubic >> Old method

[Low-frequency details]
Bicubic > New method > Old method



This time I use the Sony Fit 11A's 11.6" 1920x1080 to view these samples. This monitor's backlit is QD-LED --- so richer colors than most LCD monitors, and more similar to a good TV set. But it has ridiculous high gamma value by factory default setting. The object / skin-tone colors also look weird (too vivid) if its RGB LUT settings are not adjusted in the display driver.

Anyway it looks impressive with your new method's sample on this monitor. The little dinosaur's head looks more pop/stereo. The girl's hair colors are also distinguishable one by one even on such small-sized FHD panel.

[madshi]

Originally Posted by Shiandow

One thing I find worrying is that Bicubic seems to be shifted slightly to the left, I've been unable to confirm if this is correct. For my implementation I assumed that the chroma channel corresponded to the average of each 2x2 block.

All modern compression algorithms (MPEG2, VC1, h264, h265) have Chroma moved half a pixel to the left for 4:2:0 content:

chroma placement image

[Shiandow]

Originally Posted by madshi

All modern compression algorithms (MPEG2, VC1, h264, h265) have Chroma moved half a pixel to the left for 4:2:0 content:

Then it seems that I have made the current version for the wrong "type" of 4:2:0. Although, on the bright side, that means that the image quality can be improved even further.

Edit: Well luckily that was somewhat easy to fix, and the resulting image is definitely better.

[leeperry]

Originally Posted by Shiandow

Then it seems that I have made the current version for the wrong "type" of 4:2:0

While on that subject, would that be possible to write a PS script that would map the chroma alignment from MPEG2/H264 to MPEG1? There is no way to get properly aligned MPEG1 chroma in mVR when not feeding it RGB so a "hotfix" PS script would be full of win

[Shiandow]

Allright, I think the improved chroma shaders are now more or less ready. The new shaders only use a 2nd order interpolation instead of a 4th order, the linear equation that you needed to solve for a 4th order interpolation has the annoying property that it becomes unstable if you get close to an edge, which means that you either have a lot of artefacts or you can only use the directional information very weakly. The new shaders also use a more stable method for solving these equations which also reduces artefacts. I've also added a smarter way of limiting the resulting coefficients, which forces the chroma intensity to stay the same and prevents it from interpolating too strongly in one direction.

I'll add the new shaders to the first post.

By the way, the only purpose of the third shader is to shift the chroma channel half a pixel upwards; it would probably be better to do this simultaneously with scaling.

[huhn]

Originally Posted by leeperry

While on that subject, would that be possible to write a PS script that would map the chroma alignment from MPEG2/H264 to MPEG1? There is no way to get properly aligned MPEG1 chroma in mVR when not feeding it RGB so a "hotfix" PS script would be full of win

a h264 stream with: chromaloc 1 should provide all infos needed. and mpeg1 input should work too.

[Shiandow]

Originally Posted by leeperry

While on that subject, would that be possible to write a PS script that would map the chroma alignment from MPEG2/H264 to MPEG1? There is no way to get properly aligned MPEG1 chroma in mVR when not feeding it RGB so a "hotfix" PS script would be full of win

A slight adaptation of chromaNEDI-II should do it; it currently shifts things half a pixel vertically, but it could just as well shift them horizontally. Although I would like it better if the shifting was done simultaneously with scaling. Anyway the following code should shift the chroma channels half a pixel to the right. It expects the input to be in YCbCr, and should be used pre-resize. It tries to convert the result to RGB but I can't guarantee that that part will always work; it might be wrong about the colour space.

// $MinimumShaderProfile: ps_3_0
sampler s0 : register(s0);
float4 p0 :  register(c0);

#define width  (p0[0])
#define height (p0[1])

#define px (1.0 / (p0[0]))
#define py (1.0 / (p0[1]))

#define sqr(x) (dot(x,x))
#define I (float2x2(1,0,0,1))

//Conjugate residual 
float2 solve(float2x2 A,float2 b) {
	float2 x = 1/2.0;
	float2 r = b - mul(A,x);
	float2 p = r;
	float2 Ar = mul(A,r);
	float2 Ap = Ar;
	for (int k = 0;k < 3; k++){
		float a = min(100,dot(r,Ar)/dot(Ap,Ap));
		x = x + a*p;
		float2 rk = r; float2 Ark = Ar;
		r = r - a*Ap;
		Ar = mul(A,r);
		float b = dot(r,Ar)/dot(rk,Ark);
		p = r + b*p;
		Ap = Ar + b*Ap;
	}
	return x;
}

float4 toRGB(float4 s1) {
	s1.yz -= 0.5;
	if(width < 1120 && height < 630) return float3(s1.x+1.402*s1.z, dot(s1, float3(1, -.202008/.587, -.419198/.587)), s1.x+1.772*s1.y).rgbb;// SD Y'CbCr to RGB output
	return float3(s1.x+1.5748*s1.z, dot(s1, float3(1, -.1674679/.894, -.4185031/.894)), s1.x+1.8556*s1.y).rgbb;// HD Y'CbCr to RGB output
}

#define Col(xy) (tex2D(s0,tex+float2(px,py)*(xy)).yz-0.5)
#define Get(xy) (tex2D(s0,tex+float2(px,py)*(xy)).x+0.25)
#define Get4(xy) (float2(Get(xy+dir[0]),Get(xy+dir[1])))

float4 main(float2 tex : TEXCOORD0) : COLOR {
	float4 c0 = tex2D(s0,tex);

	float2 dir[2] =  {{-1,0},{1,0}};
	float2 wind[4] = {{-1,0},{1,0},{0,1},{0,-1}};
	float2 pos[2] = {{0,0},{1,0}};
	
	float w = 2;
	float2x2 R = w*mul(float2x1(Get4(0)),float1x2(Get4(0)));
	float2 r = w*Get(0)*Get4(0);
	float4x2 C = {Get4(wind[0]),Get4(wind[1]),Get4(wind[2]),Get4(wind[3])};
	float4 y = {Get(wind[0]),Get(wind[1]),Get(wind[2]),Get(wind[3])};
	R += mul(transpose(C),C);
	r += mul(y,C);

	//Normalize coefficients
	float n = 16;
	R/= n; r /= n;
	
	//Solve equations
	float2 a = solve(R+0.000001*I,r);

	//Limit the coefiicents of a
	float2 b = float2(a[0]+a[1]-1,a[0]-a[1]);
	b[0] = clamp(b[0],-.02,.02); 	//Clamp intensity
	b[1] = clamp(b[1],-1,1); 	//Clamp "directedness"
	a = .5 + float2(.5,.5)*b[0] + float2(.5,-.5)*b[1];

	c0.a = dot(c0.gb,1);
	c0.gb = 0.5+mul(a,float2x2(Col(pos[0]),Col(pos[1])));

	return toRGB(c0);
}

[FireFreak111]

Running the chroma shaders in MPC-HC in pre-resize in order, the first two blur the image, and with the third one the image looks blown out and grainy. With the YbCbCr file, basically the same problem. Chroma set to NN.

Clearly not working here.

GTX 660, newest Madvr, nightly MPC-HC, Windows 8.1.1.

From the samples shown however, this looks like a great middle-ground between Chroma Jinc and NNEDI3, which for DVD and lower quality content I just cant combine with Luma NNEDI3 16N with both double and quad enabled on this 660 and i7 870.

Wish NNEDI3 ran on DirectCompute.

[Shiandow]

Could you post a few screenshots of what happens?

[FireFreak111]

Nearest Neighbour


Chrome-NEDI

[madshi]

Thanks for your efforts so far, Shiandow, I'll definitely play with your shaders when I find some time!

[Shiandow]

Oh it turns out that I copied the wrong shader for chromaNEDI-III, it should work with the current one. Thanks FireFreak111 for noticing that.

[FireFreak111]

Sorry for the full size images, photobuckets linking is messed when trying to use a full size image.

There is a distinct blur when using these shaders, it doesn't appear to be working correctly.

Nearest Neighbour:


Chroma-NEDI

[Shiandow]

Are you sure that the input is in YCbCr? Because it looks like the kind of images you get when the input is RGB.

[FireFreak111]

Thankyou, I had named the YCbCr file wrong. The YCbCr to RGB should be post Chroma-NEDI right? It seems to be working correctly now.