developer discussion: custom pixel shaders -> color space, black/white definition

[madshi]

Hello devs,

now we have multiple renderers (VMR9, EVR-CP, madVR) which support custom pixel shaders. I think it's time to explicitly define a few things which are currently not clearly defined:

(A) Which colorspace should custom pixel shaders run in?

1. R'G'B' (gamma corrected RGB)
2. RGB (linear light RGB)
3. BT.709 Y'CbCr (gamma corrected YCbCr)
4. BT.709 YCbCr (linear light YCbCr)
5. something else

Currently VMR9 and EVR seem to use R'G'B'. Should we stick to that? That would be fine with me. But we should clearly define this.

(B) Where is black and where is white?

1. black 0.0, white 1.0
2. black 16/255, white 235/255
3. black (16 * 256)/65535, white (235 * 256)/65535
4. something else

Currently VMR9 and EVR seem to run pixel shaders with black at 0.0 and white at 1.0. Should we stick to that?

Black at 0.0 has some clear advantages. One key advantage is that the typical RGB <-> YCbCr matrixes simply work as expected. With black at 16/255 the usual matrixes would not work correcty, AFAIK. So if a shader needs to convert RGB to YCbCr, having black at 0.0 makes things much easier. The same applies when a shader wants to convert R'G'B' to RGB because the proper conversion is done with "pow" when black is at 0.0.

However, there's one big disadvantage with having black at 0.0: When using cardinal texture formats for temp storage, BTB and WTW are clipped. Which is what currently actually happens with VMR9 and EVR. Having black at 16/255 and white at 235/255 would preserve BTB and WTW even when using cardinal textures.

-------

I'm not really sure what my preferred solution is. I would vote for either R'G'B' with black at 0.0, or for R'G'B' with black at 16/255. In the first case I'd have to use float textures to preserve BTB/WTW which would probably hurt performance a bit. In the 2nd case I could stick to cardinal textures, however life would be more difficult for shader writers.

Comments / votes, anyone?

[Underground78]

Hello,

How big would be the performance impact? I tend to think that it would be great to avoid adding a new complexity when writing shaders. Also, knowing that basically all shaders currently around are for R'G'B' with black at 0.0, it would require to update all those existing shaders to work with the new definition.

[madshi]

To be honest, I'm not fully sure how big the performance impact would be. Probably it depends on the GPU generation. If the shader math is complex enough, memory performance might not matter at all. However, for simple shaders or for texture intensive shaders probably the memory performance will be the bottleneck. I guess if performance is important, float16 could be used. Personally, I guess if we decide on black = 0.0, I'll probably go float32 for madVR custom pixel shaders.

If we do stay with black = 0.0, then probably VMR9 and EVR should be updated to use at least float16, so BTB and WTW are not lost.

[nevcairiel]

If the VMR/EVR problem with cut-off BTB/WTW is the only reason against black = 0.0, and using R'G'B' is the preferred solution as well, i would vote for simply staying with that. No need to change existing shaders, and if they may be simpler to write, as you say, another reason to stick to it.
If VMR/EVR developers care, they could as well change to float16/float32 textures, but they have been running like this for years without many complaints...

[Ver Greeneyes]

Most shaders probably assume that the format is RGB or R'G'B', and some might even do a conversion to YUV themselves for color difference evaluation. Perceptually linear J'a'b' (CAM02-UCS) is probably too big of a performance hit even for recent cards, and I was never able to solve its problems near black (though I'll get back to messing with it soon if only so I can have something to show for my efforts).

Edit: actually got somewhere with this, though there are still some odd problems to solve. I don't mean to hijack this topic though, just a heads up that getting this working at all isn't necessarily an issue.