madVR linear light scaling, NNEDI3, etc.

[6233638]

I haven't really been following this topic much recently, but there are a few things I wanted to say.

Firstly, if you want accurate downscaling, linear light must be used.
Linear light scaling does a much better job of preserving fine details and representing the original high resolution image.

The problem with linear light scaling - at least in madVR's implementation - is that it can produce very ugly dark ringing around items when used with most of the scaling algorithms.

White text on a red background seems to be a very good illustration of the problem. I made a very quick test based on the text from StinDaWg's image.

Here it is doubled from 720p to 1440p using 256 neuron NNEDI3 and then downscaled using Lanczos 3 with linear light scaling enabled:


As you can see, the results are absolutely terrible - this is why the linear light option should not be used with most downscaling algorithms. (or any upscaling algorithms)

In my testing, the only scaling algorithm which could be used with linear light downscaling to produce acceptable sharpness and image quality was Catmull-Rom with the anti-ringing filter enabled.
You could also use it with Mitchell-Netravali or higher levels of SoftCubic, but those produce quite a soft image.

With real image content, Catmull-Rom with the anti-ringing filter using linear light typically gives you a relatively sharp image, with minimal aliasing and ringing.

With white text on a red background, there does seem to be some residual dark ringing even with this combination though - which had not shown up in my previous tests that focused mostly on images (both photographs and scenes from films) rather than text:




Without using linear light downscaling, it does indeed seem that the anti-ringing filter can also introduce this dark ringing problem.

Lanczos 3 with the anti-ringing filter:


Lanczos 3 or Spline 3 seem like reasonable choices to use without the anti-ringing or linear light options.




But it is important to look at more than one specific example when evaluating scaling algorithms.
While Lanczos 3 without using the anti-ringing filter or linear light scaling might look best in this instance, that does not necessarily mean that it will look best with all images.
I would be hesitant to disable linear light scaling altogether just because this one example shows some residual ringing.

But if you see this sort of thing a lot, then perhaps it is a high enough priority that you want to avoid it.


Edit: it looks like this might specifically be an issue when using NNEDI3 to double the image size, and then scale it back down.

I rendered the text at 1440p instead of 720p and simply downscaled that using Catmull-Rom with the anti-ringing filter and linear light scaling options enabled:



Absolutely no ugly ringing at all.

[turbojet]

Anyhow, is it just me or is the red a bit brighter on the LL pics? I've always seen inaccurate colors with LL which is why I don't use it. I have a feeling it has to do with calibration, my displays aren't professionally calibrated, just set to sRGB and tweaked with images. The responses from the backpack and bug images with LL a few months ago seems to agree with the calibrated vs non-calbrated.

[Asmodian]

This is very interesting. I must say the relationship to NNEDI3 is hard for me to understand. The example Catmull-Rom+AR+LL with and without a NNEDI3 upscale is very obvious but is there any sign of the dark ringing in the NNEDI3 200% image?

I wonder if this is an effect of the 0.5 pixel shifting algorithm used after NNEDI3?

[StinDaWg]

Yup, that's exactly what I'm seeing. AR on any of the downscaling algorithms after NNEDI3 doubling causes jagged text. For now I'm just going to use Spline 3.

By the way, this shouldn't have been moved out of the madVR thread.

[6233638]

Originally Posted by turbojet

Anyhow, is it just me or is the red a bit brighter on the LL pics? I've always seen inaccurate colors with LL which is why I don't use it. I have a feeling it has to do with calibration, my displays aren't professionally calibrated, just set to sRGB and tweaked with images. The responses from the backpack and bug images with LL a few months ago seems to agree with the calibrated vs non-calbrated.

It's the same in all of them. Small details are often darker than they should be when scaling in gamma light though.
If you are used to seeing things darker than they should be, then displaying them correctly (linear light) would look "brighter" - though that "brighter" image is a much closer match for the original before it was downscaled.

Originally Posted by Asmodian

This is very interesting. I must say the relationship to NNEDI3 is hard for me to understand. The example Catmull-Rom+AR+LL with and without a NNEDI3 upscale is very obvious but is there any sign of the dark ringing in the NNEDI3 200% image?

I wonder if this is an effect of the 0.5 pixel shifting algorithm used after NNEDI3?

It's funny, I didn't think to check at 200%.
I suspect you might be right about it being caused by the half-pixel offset.

At 200% size, there shouldn't be any processing done to the image except NNEDI3 - though it looks like the half-pixel offset (I assume) is handled by your currently selected upscaling algorithm.
So when I change upscaling between say Lanczos 3 and Nearest Neighbor, the differences are significant, even though it should only be NNEDI3 scaling in effect.

200% Nearest Neighbor:


200% NNEDI3 with Nearest Neighbor:


200% NNEDI3 with Lanczos 3:


200% NNEDI3 with Lanczos 3 AR:


The softer image resulting from using Lanczos tends to hide this artifact, but it's obvious in the Nearest Neighbor images that NNEDI3 is what's introducing the black ringing, which can look like aliasing once that image is downscaled.

It only seems to be affecting the bottom and right edges of the letters, which is what makes me think it could be the half-pixel offset causing it.

Originally Posted by StinDaWg

Yup, that's exactly what I'm seeing. AR on any of the downscaling algorithms after NNEDI3 doubling causes jagged text. For now I'm just going to use Spline 3.

I think it's just an interaction between anti-ringing/linear light and NNEDI3 image doubling making this much more obvious, rather than being caused by the anti-ringing filter when downscaling.

As you can see in this image, you can use Catmull-Rom with the anti-ringing filter and linear light enabled and get perfectly rendered text with a native 1440p source image.

[Asmodian]

For these images are you using NNEDI3 chroma doubling too? The 200% NNEDI3 with Nearest Neighbor looks like it might be due to chroma being resized differently than luma (with Nearest Neighbor).

I believe madshi stated the 0.5 pixel shifter was a highly optimized (for speed) algorithm used only for the half pixel shift but I have been unable to find the post.

[6233638]

Originally Posted by StinDaWg

I've always wondered why it was recommended here to use Catmull-Rom for downscaling. In various encoding forums where people are obsessed with quality, everyone uses Spline 36 (I guess this is Spline 3 in madVR). Spline is almost as sharp as Lanczos with less ringing/artifacts. Catmull-Rom is too soft IMO. Spline is somewhere in the middle.

You cannot use Spline or Lanczos with linear light scaling without introducing significant artifacts into the image.
If you really care about image quality, you must do any downscaling in linear light.
I suppose any upscaling should probably be done in linear light too, but that generally doesn't look very good.

Catmull-Rom is the sharpest algorithm that can be used with linear light downscaling, and when combined with the anti-ringing filter it results in a sharp image with minimal ringing or aliasing.
I would not have predicted it, but that's why I've spent so much time going back and forth over every scaling combination with a huge range of test images - sometimes the results will surprise you.

While I have not spent much time evaluating the downscaling options in gamma light, in all of my testing I tended to prefer Lanczos over Spline - though I was never a huge fan of either of them.
They both produce very similar results though, the trade-off is that Lanczos is slightly sharper, with slightly more ringing, and Spline is slightly softer with less ringing, but more aliasing. You can't go wrong either way though.

I haven't done much testing for downscaling without linear light being used, but it would not surprise me if Lanczos or Spline were a better choice than Catmull-Rom when scaling in gamma light.

Originally Posted by StinDaWg

If the aliasing is inherent to the algorithm and can't be fixed, I don't think AR should be recommended to use anymore for downscaling (it's still fine for upscaling). One of the greatest benefits from NNEDI3 is near perfect diagonal lines, using AR negates that. I don't see any increased ringing from not using it anyways, at least for 720p->1080p.

If nothing else, this testing has made it clear once again just how good NNEDI3 image doubling looks compared to the regular upscaling algorithms.

[Asmodian]

I tried a quick test starting with your "no-nnedi73b4r.png" (Catmull-Rom downscale) using NNEDI3 256 doubling for both chroma and luma and resizing back down with Lanczos 3 AR+LL, it looks pretty good to me and I don't notice any dark ringing, certainly nothing like your first Lanczos 3 + LL test.

NNEDI3 256 Luma & Chroma, Lanczos 3 AR+LL downscale:

[6233638]

Originally Posted by Asmodian

For these images are you using NNEDI3 chroma doubling too? The 200% NNEDI3 with Nearest Neighbor looks like it might be due to chroma being resized differently than luma (with Nearest Neighbor).

I believe madshi stated the 0.5 pixel shifter was a highly optimized (for speed) algorithm used only for the half pixel shift but I have been unable to find the post.

Since I was using a PNG source image, I did not expect chroma settings to have any effect, though I did also have chroma upscaling set to 256 neuron NNEDI3.
From testing it again, I was correct that the chroma upscaling setting did not affect the results.

But you're right - I had overlooked the chroma doubling setting.
When you enable chroma doubling as well as luma doubling, you get this:



And that's still set to use Nearest Neighbor for luma upscaling.

So it looks like chroma is upscaled using the luma algorithm when there is a mismatch between chroma and luma resolution that has been caused by image doubling - and it makes sense to do that.
It might not be intuitive, but you would want to use the luma algorithm for that rather than chroma, so that's the right way to do it.

This also explains why it only shows up with certain types of images, and why it was obvious with text on a red background. (and likely would with red text on a dark background too)

Unfortunately, that means you're going to have to enable chroma doubling with the same number of neurons as luma to avoid this artifact.
You might be able to get away with using a lower number of neurons for chroma than luma, but while it doesn't result in anything as bad as this, there are still some artifacts around the edges of high contrast areas.

Or I suppose the other option is to simply stop using the anti-ringing filter or linear light options when downscaling, but I'm not sure that's what I would recommend.

[Asmodian]

Originally Posted by 6233638

Or I suppose the other option is to simply stop using the anti-ringing filter or linear light options when downscaling, but I'm not sure that's what I would recommend.

Is this only because the ringing happens to have a similar size compared to these artifacts so it helps blur them? Or are these artifacts causing AR to misbehave? I don't understand why disabling LL would help mask these artifacts either.

The artifacts are obviously worse after AR and/or LL downscaling I just don't understand why.

Edit: After further testing with both luma and chroma doubled: AR and/or LL do not cause or magnify aliasing or dark ringing with this white text on a red background. They simply make the chroma offset artifacts from doubling luma but not chroma more apparent.

[GCRaistlin]

It's interesting to read your discussion but I'm afraid it's beyond my possibilities to use this info as is. Can you please tell me what upscaling / downscaling settings are optimal to use? My system: Core 2 Duo E6400 @ 3.28 MHz, GF 8800 GT, 1920x1080, WinXP SP3.

[Shiandow]

Originally Posted by 6233638

Unfortunately, that means you're going to have to enable chroma doubling with the same number of neurons as luma to avoid this artifact.
You might be able to get away with using a lower number of neurons for chroma than luma, but while it doesn't result in anything as bad as this, there are still some artifacts around the edges of high contrast areas.

A different solution would be to use an adaptation of the chroma upscaling algorithm I created. There's not that much of a difference between upscaling chroma and doubling it. Also the algorithm uses the information from the luma-channel to upscale it so it should (in theory) avoid the problems caused by a mismatched luma and chroma channel.

Unfortunately I didn't have your original image so I had to improvise, but here's the result of using the chroma-doubling algorithm I proposed on the last image of your first post:



It would probably have been sharper if I used the original.

[Asmodian]

Shiandow, I thought of your luma mapped NEDI idea after realizing what was causing this, it would be very interesting to see how close it could get to NNEDI3. A faster alternative to NNEDI3 chroma doubling that could avoid these artifacts would be great.

[Shiandow]

Well I can't think of a reason that it shouldn't work. It may be slightly less efficient but it should definitely be faster than using NNEDI3 again. One nice coincidence is that the algorithm naturally shifts the chroma channel half a pixel so you don't need yet another shader to shift it back again.

I've updated the chromaNEDI shaders to support different chroma channel patterns, including the pattern you need for chroma doubling.

[Frechdachs]

I'm not convinced that those artifacts are a result of the pixel offset, since there are no sort of artifact if AR or LL is disabled. But it is true that this problem only occurs when AR or LL is used in conjunction with NNEDI3.
Also, not only downscaling is affected. Using NNEDI3 doubling followed by upscaling with AR enabled results in artifacts too.

This example is a 480p video that is doubled with NNEDI3 (luma) and then upscaled with Jinc3+AR. (AR is disabled on the second image.)
http://screenshotcomparison.com/comparison/80283
Look at the "P" and the "R".

Originally Posted by 6233638

So it looks like chroma is upscaled using the luma algorithm when there is a mismatch between chroma and luma resolution that has been caused by image doubling - and it makes sense to do that.
It might not be intuitive, but you would want to use the luma algorithm for that rather than chroma, so that's the right way to do it.

I always was under the impression that the algorithm that is set in chroma upscaling is only used to get chroma to the same resolution as luma. So with a 720p video chroma upscaling scales the video only to 720p. If further upscaling is needed (if you watch it on a 1080p display for example) the algorithm secified in image upscaling is used to upscale both chroma and luma. That's why it is called image and not luma upscaling I suppose.
Please correct me if I'm wrong.

[Shiandow]

Originally Posted by Frechdachs

I'm not convinced that those artifacts are a result of the pixel offset

I agree, for the simple reason that the black ringing occurs on all sides of the letters which isn't what you'd expect if it was a wrong pixel offset. My best guess at the moment is that treating chroma and luma differently causes them to drift apart. In this case this seems to cause the letters to become larger in the chroma channel than in the luma channel, which means that there's a small region around each letter where the image is black instead of either red or white. It's possible that the inverse happens if you disable linear light and anti-ringing, but this is incredibly hard to see.

Originally Posted by Frechdachs

This example is a 480p video that is doubled with NNEDI3 (luma) and then upscaled with Jinc3+AR. (AR is disabled on the second image.)
http://screenshotcomparison.com/comparison/80283
Look at the "P" and the "R".

That seems to be an actual bug in the AR algorithm. I don't think it's related to the other issues in this post, which I believe is a consequence of treating chroma and luma separately.

Originally Posted by Frechdachs

I always was under the impression that the algorithm that is set in chroma upscaling is only used to get chroma to the same resolution as luma. [...] If further upscaling is needed (...) the algorithm secified in image upscaling is used to upscale both chroma and luma.

As far as I know that is correct.

[Asmodian]

Originally Posted by Shiandow

That seems to be an actual bug in the AR algorithm. I don't think it's related to the other issues in this post, which I believe is a consequence of treating chroma and luma separately.

It probably is sort of related to the other issues in this post. The artifacts from resizing luma and chroma differently get magnified by the AR algorithm; something about them causes the AR algorithm to misbehave. A similar thing is happening here, the AR algorithm is confused by something so it creates an artifact. It might even be the same thing as I assume this artifact only happens when AR follows NNEDI3 doubling only the luma.

[Shiandow]

Well, it might be related but I think that some of the issues will happen anyway even when AR is behaving "correctly" just because you're not doing the same thing to the chroma and luma channels. Unfortunately I don't know enough about the anti-ringing algorithm to know why using AR seems to be worse than not using AR. I'm also not sure if it is always worse to use AR.

With linear light it is somewhat more obvious why it's causing problems, although I wonder if you could remove those by also using linear light for NNEDI3 and image upscaling.

[StinDaWg]

madshi, can we get a comment on this? Haven't seen you around in awhile.

[madshi]

Can you please create a bug tracker entry with a link to this thread? Ideally with the original PNG used for these tests and maybe also with a video sample. Thanks.