Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Hardware & Software > Software players

Reply
 
Thread Tools Search this Thread Display Modes
Old 18th June 2014, 22:09   #1  |  Link
6233638
Registered User
 
Join Date: Apr 2009
Posts: 1,019
madVR linear light scaling, NNEDI3, etc.

I haven't really been following this topic much recently, but there are a few things I wanted to say.

Firstly, if you want accurate downscaling, linear light must be used.
Linear light scaling does a much better job of preserving fine details and representing the original high resolution image.

The problem with linear light scaling - at least in madVR's implementation - is that it can produce very ugly dark ringing around items when used with most of the scaling algorithms.

White text on a red background seems to be a very good illustration of the problem. I made a very quick test based on the text from StinDaWg's image.

Here it is doubled from 720p to 1440p using 256 neuron NNEDI3 and then downscaled using Lanczos 3 with linear light scaling enabled:


As you can see, the results are absolutely terrible - this is why the linear light option should not be used with most downscaling algorithms. (or any upscaling algorithms)

In my testing, the only scaling algorithm which could be used with linear light downscaling to produce acceptable sharpness and image quality was Catmull-Rom with the anti-ringing filter enabled.
You could also use it with Mitchell-Netravali or higher levels of SoftCubic, but those produce quite a soft image.

With real image content, Catmull-Rom with the anti-ringing filter using linear light typically gives you a relatively sharp image, with minimal aliasing and ringing.

With white text on a red background, there does seem to be some residual dark ringing even with this combination though - which had not shown up in my previous tests that focused mostly on images (both photographs and scenes from films) rather than text:




Without using linear light downscaling, it does indeed seem that the anti-ringing filter can also introduce this dark ringing problem.

Lanczos 3 with the anti-ringing filter:


Lanczos 3 or Spline 3 seem like reasonable choices to use without the anti-ringing or linear light options.




But it is important to look at more than one specific example when evaluating scaling algorithms.
While Lanczos 3 without using the anti-ringing filter or linear light scaling might look best in this instance, that does not necessarily mean that it will look best with all images.
I would be hesitant to disable linear light scaling altogether just because this one example shows some residual ringing.

But if you see this sort of thing a lot, then perhaps it is a high enough priority that you want to avoid it.


Edit: it looks like this might specifically be an issue when using NNEDI3 to double the image size, and then scale it back down.

I rendered the text at 1440p instead of 720p and simply downscaled that using Catmull-Rom with the anti-ringing filter and linear light scaling options enabled:



Absolutely no ugly ringing at all.

Last edited by Guest; 19th June 2014 at 04:47. Reason: moderator split this to its own topic
6233638 is offline   Reply With Quote
Old 19th June 2014, 00:22   #2  |  Link
turbojet
Registered User
 
Join Date: May 2008
Posts: 1,840
Anyhow, is it just me or is the red a bit brighter on the LL pics? I've always seen inaccurate colors with LL which is why I don't use it. I have a feeling it has to do with calibration, my displays aren't professionally calibrated, just set to sRGB and tweaked with images. The responses from the backpack and bug images with LL a few months ago seems to agree with the calibrated vs non-calbrated.
__________________
PC: FX-8320 GTS250 HTPC: G1610 GTX650
PotPlayer/MPC-BE LAVFilters MadVR-Bicubic75AR/Lanczos4AR/Lanczos4AR LumaSharpen -Strength0.9-Pattern3-Clamp0.1-OffsetBias2.0

Last edited by Guest; 19th June 2014 at 01:49. Reason: ot
turbojet is offline   Reply With Quote
Old 19th June 2014, 01:28   #3  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,631
This is very interesting. I must say the relationship to NNEDI3 is hard for me to understand. The example Catmull-Rom+AR+LL with and without a NNEDI3 upscale is very obvious but is there any sign of the dark ringing in the NNEDI3 200% image?

I wonder if this is an effect of the 0.5 pixel shifting algorithm used after NNEDI3?
Asmodian is offline   Reply With Quote
Old 19th June 2014, 02:36   #4  |  Link
StinDaWg
Registered User
 
Join Date: Jan 2014
Posts: 216
Yup, that's exactly what I'm seeing. AR on any of the downscaling algorithms after NNEDI3 doubling causes jagged text. For now I'm just going to use Spline 3.

By the way, this shouldn't have been moved out of the madVR thread.

Last edited by StinDaWg; 19th June 2014 at 02:48.
StinDaWg is offline   Reply With Quote
Old 19th June 2014, 04:11   #5  |  Link
6233638
Registered User
 
Join Date: Apr 2009
Posts: 1,019
Quote:
Originally Posted by turbojet View Post
Anyhow, is it just me or is the red a bit brighter on the LL pics? I've always seen inaccurate colors with LL which is why I don't use it. I have a feeling it has to do with calibration, my displays aren't professionally calibrated, just set to sRGB and tweaked with images. The responses from the backpack and bug images with LL a few months ago seems to agree with the calibrated vs non-calbrated.
It's the same in all of them. Small details are often darker than they should be when scaling in gamma light though.
If you are used to seeing things darker than they should be, then displaying them correctly (linear light) would look "brighter" - though that "brighter" image is a much closer match for the original before it was downscaled.
Quote:
Originally Posted by Asmodian View Post
This is very interesting. I must say the relationship to NNEDI3 is hard for me to understand. The example Catmull-Rom+AR+LL with and without a NNEDI3 upscale is very obvious but is there any sign of the dark ringing in the NNEDI3 200% image?

I wonder if this is an effect of the 0.5 pixel shifting algorithm used after NNEDI3?
It's funny, I didn't think to check at 200%.
I suspect you might be right about it being caused by the half-pixel offset.

At 200% size, there shouldn't be any processing done to the image except NNEDI3 - though it looks like the half-pixel offset (I assume) is handled by your currently selected upscaling algorithm.
So when I change upscaling between say Lanczos 3 and Nearest Neighbor, the differences are significant, even though it should only be NNEDI3 scaling in effect.

200% Nearest Neighbor:


200% NNEDI3 with Nearest Neighbor:


200% NNEDI3 with Lanczos 3:


200% NNEDI3 with Lanczos 3 AR:


The softer image resulting from using Lanczos tends to hide this artifact, but it's obvious in the Nearest Neighbor images that NNEDI3 is what's introducing the black ringing, which can look like aliasing once that image is downscaled.

It only seems to be affecting the bottom and right edges of the letters, which is what makes me think it could be the half-pixel offset causing it.

Quote:
Originally Posted by StinDaWg View Post
Yup, that's exactly what I'm seeing. AR on any of the downscaling algorithms after NNEDI3 doubling causes jagged text. For now I'm just going to use Spline 3.
I think it's just an interaction between anti-ringing/linear light and NNEDI3 image doubling making this much more obvious, rather than being caused by the anti-ringing filter when downscaling.

As you can see in this image, you can use Catmull-Rom with the anti-ringing filter and linear light enabled and get perfectly rendered text with a native 1440p source image.
6233638 is offline   Reply With Quote
Old 19th June 2014, 04:26   #6  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,631
For these images are you using NNEDI3 chroma doubling too? The 200% NNEDI3 with Nearest Neighbor looks like it might be due to chroma being resized differently than luma (with Nearest Neighbor).

I believe madshi stated the 0.5 pixel shifter was a highly optimized (for speed) algorithm used only for the half pixel shift but I have been unable to find the post.
Asmodian is offline   Reply With Quote
Old 19th June 2014, 04:32   #7  |  Link
6233638
Registered User
 
Join Date: Apr 2009
Posts: 1,019
Quote:
Originally Posted by StinDaWg View Post
I've always wondered why it was recommended here to use Catmull-Rom for downscaling. In various encoding forums where people are obsessed with quality, everyone uses Spline 36 (I guess this is Spline 3 in madVR). Spline is almost as sharp as Lanczos with less ringing/artifacts. Catmull-Rom is too soft IMO. Spline is somewhere in the middle.
You cannot use Spline or Lanczos with linear light scaling without introducing significant artifacts into the image.
If you really care about image quality, you must do any downscaling in linear light.
I suppose any upscaling should probably be done in linear light too, but that generally doesn't look very good.

Catmull-Rom is the sharpest algorithm that can be used with linear light downscaling, and when combined with the anti-ringing filter it results in a sharp image with minimal ringing or aliasing.
I would not have predicted it, but that's why I've spent so much time going back and forth over every scaling combination with a huge range of test images - sometimes the results will surprise you.

While I have not spent much time evaluating the downscaling options in gamma light, in all of my testing I tended to prefer Lanczos over Spline - though I was never a huge fan of either of them.
They both produce very similar results though, the trade-off is that Lanczos is slightly sharper, with slightly more ringing, and Spline is slightly softer with less ringing, but more aliasing. You can't go wrong either way though.

I haven't done much testing for downscaling without linear light being used, but it would not surprise me if Lanczos or Spline were a better choice than Catmull-Rom when scaling in gamma light.

Quote:
Originally Posted by StinDaWg View Post
If the aliasing is inherent to the algorithm and can't be fixed, I don't think AR should be recommended to use anymore for downscaling (it's still fine for upscaling). One of the greatest benefits from NNEDI3 is near perfect diagonal lines, using AR negates that. I don't see any increased ringing from not using it anyways, at least for 720p->1080p.
If nothing else, this testing has made it clear once again just how good NNEDI3 image doubling looks compared to the regular upscaling algorithms.

Last edited by 6233638; 19th June 2014 at 04:34.
6233638 is offline   Reply With Quote
Old 19th June 2014, 04:44   #8  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,631
I tried a quick test starting with your "no-nnedi73b4r.png" (Catmull-Rom downscale) using NNEDI3 256 doubling for both chroma and luma and resizing back down with Lanczos 3 AR+LL, it looks pretty good to me and I don't notice any dark ringing, certainly nothing like your first Lanczos 3 + LL test.

NNEDI3 256 Luma & Chroma, Lanczos 3 AR+LL downscale:

Last edited by Asmodian; 19th June 2014 at 04:59. Reason: corrected screenshot
Asmodian is offline   Reply With Quote
Old 19th June 2014, 04:56   #9  |  Link
6233638
Registered User
 
Join Date: Apr 2009
Posts: 1,019
Quote:
Originally Posted by Asmodian View Post
For these images are you using NNEDI3 chroma doubling too? The 200% NNEDI3 with Nearest Neighbor looks like it might be due to chroma being resized differently than luma (with Nearest Neighbor).

I believe madshi stated the 0.5 pixel shifter was a highly optimized (for speed) algorithm used only for the half pixel shift but I have been unable to find the post.
Since I was using a PNG source image, I did not expect chroma settings to have any effect, though I did also have chroma upscaling set to 256 neuron NNEDI3.
From testing it again, I was correct that the chroma upscaling setting did not affect the results.

But you're right - I had overlooked the chroma doubling setting.
When you enable chroma doubling as well as luma doubling, you get this:



And that's still set to use Nearest Neighbor for luma upscaling.

So it looks like chroma is upscaled using the luma algorithm when there is a mismatch between chroma and luma resolution that has been caused by image doubling - and it makes sense to do that.
It might not be intuitive, but you would want to use the luma algorithm for that rather than chroma, so that's the right way to do it.

This also explains why it only shows up with certain types of images, and why it was obvious with text on a red background. (and likely would with red text on a dark background too)

Unfortunately, that means you're going to have to enable chroma doubling with the same number of neurons as luma to avoid this artifact.
You might be able to get away with using a lower number of neurons for chroma than luma, but while it doesn't result in anything as bad as this, there are still some artifacts around the edges of high contrast areas.

Or I suppose the other option is to simply stop using the anti-ringing filter or linear light options when downscaling, but I'm not sure that's what I would recommend.
6233638 is offline   Reply With Quote
Old 19th June 2014, 05:46   #10  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,631
Quote:
Originally Posted by 6233638 View Post
Or I suppose the other option is to simply stop using the anti-ringing filter or linear light options when downscaling, but I'm not sure that's what I would recommend.
Is this only because the ringing happens to have a similar size compared to these artifacts so it helps blur them? Or are these artifacts causing AR to misbehave? I don't understand why disabling LL would help mask these artifacts either.

The artifacts are obviously worse after AR and/or LL downscaling I just don't understand why.

Edit: After further testing with both luma and chroma doubled: AR and/or LL do not cause or magnify aliasing or dark ringing with this white text on a red background. They simply make the chroma offset artifacts from doubling luma but not chroma more apparent.

Last edited by Asmodian; 19th June 2014 at 06:05.
Asmodian is offline   Reply With Quote
Old 19th June 2014, 13:48   #11  |  Link
GCRaistlin
Registered User
 
GCRaistlin's Avatar
 
Join Date: Jun 2006
Posts: 262
It's interesting to read your discussion but I'm afraid it's beyond my possibilities to use this info as is. Can you please tell me what upscaling / downscaling settings are optimal to use? My system: Core 2 Duo E6400 @ 3.28 MHz, GF 8800 GT, 1920x1080, WinXP SP3.
__________________
Magically yours
Raistlin
GCRaistlin is online now   Reply With Quote
Old 19th June 2014, 17:47   #12  |  Link
Shiandow
Registered User
 
Join Date: Dec 2013
Posts: 752
Quote:
Originally Posted by 6233638 View Post
Unfortunately, that means you're going to have to enable chroma doubling with the same number of neurons as luma to avoid this artifact.
You might be able to get away with using a lower number of neurons for chroma than luma, but while it doesn't result in anything as bad as this, there are still some artifacts around the edges of high contrast areas.
A different solution would be to use an adaptation of the chroma upscaling algorithm I created. There's not that much of a difference between upscaling chroma and doubling it. Also the algorithm uses the information from the luma-channel to upscale it so it should (in theory) avoid the problems caused by a mismatched luma and chroma channel.

Unfortunately I didn't have your original image so I had to improvise, but here's the result of using the chroma-doubling algorithm I proposed on the last image of your first post:



It would probably have been sharper if I used the original.

Last edited by Shiandow; 19th June 2014 at 18:22. Reason: grammar
Shiandow is offline   Reply With Quote
Old 19th June 2014, 19:01   #13  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,631
Shiandow, I thought of your luma mapped NEDI idea after realizing what was causing this, it would be very interesting to see how close it could get to NNEDI3. A faster alternative to NNEDI3 chroma doubling that could avoid these artifacts would be great.
Asmodian is offline   Reply With Quote
Old 19th June 2014, 19:47   #14  |  Link
Shiandow
Registered User
 
Join Date: Dec 2013
Posts: 752
Well I can't think of a reason that it shouldn't work. It may be slightly less efficient but it should definitely be faster than using NNEDI3 again. One nice coincidence is that the algorithm naturally shifts the chroma channel half a pixel so you don't need yet another shader to shift it back again.

I've updated the chromaNEDI shaders to support different chroma channel patterns, including the pattern you need for chroma doubling.
Shiandow is offline   Reply With Quote
Old 22nd June 2014, 17:40   #15  |  Link
Frechdachs
Registered User
 
Frechdachs's Avatar
 
Join Date: May 2014
Posts: 20
I'm not convinced that those artifacts are a result of the pixel offset, since there are no sort of artifact if AR or LL is disabled. But it is true that this problem only occurs when AR or LL is used in conjunction with NNEDI3.
Also, not only downscaling is affected. Using NNEDI3 doubling followed by upscaling with AR enabled results in artifacts too.

This example is a 480p video that is doubled with NNEDI3 (luma) and then upscaled with Jinc3+AR. (AR is disabled on the second image.)
http://screenshotcomparison.com/comparison/80283
Look at the "P" and the "R".

Quote:
Originally Posted by 6233638 View Post
So it looks like chroma is upscaled using the luma algorithm when there is a mismatch between chroma and luma resolution that has been caused by image doubling - and it makes sense to do that.
It might not be intuitive, but you would want to use the luma algorithm for that rather than chroma, so that's the right way to do it.
I always was under the impression that the algorithm that is set in chroma upscaling is only used to get chroma to the same resolution as luma. So with a 720p video chroma upscaling scales the video only to 720p. If further upscaling is needed (if you watch it on a 1080p display for example) the algorithm secified in image upscaling is used to upscale both chroma and luma. That's why it is called image and not luma upscaling I suppose.
Please correct me if I'm wrong.

Last edited by Frechdachs; 22nd June 2014 at 17:46.
Frechdachs is offline   Reply With Quote
Old 22nd June 2014, 19:51   #16  |  Link
Shiandow
Registered User
 
Join Date: Dec 2013
Posts: 752
Quote:
Originally Posted by Frechdachs View Post
I'm not convinced that those artifacts are a result of the pixel offset
I agree, for the simple reason that the black ringing occurs on all sides of the letters which isn't what you'd expect if it was a wrong pixel offset. My best guess at the moment is that treating chroma and luma differently causes them to drift apart. In this case this seems to cause the letters to become larger in the chroma channel than in the luma channel, which means that there's a small region around each letter where the image is black instead of either red or white. It's possible that the inverse happens if you disable linear light and anti-ringing, but this is incredibly hard to see.

Quote:
Originally Posted by Frechdachs View Post
This example is a 480p video that is doubled with NNEDI3 (luma) and then upscaled with Jinc3+AR. (AR is disabled on the second image.)
http://screenshotcomparison.com/comparison/80283
Look at the "P" and the "R".
That seems to be an actual bug in the AR algorithm. I don't think it's related to the other issues in this post, which I believe is a consequence of treating chroma and luma separately.

Quote:
Originally Posted by Frechdachs View Post
I always was under the impression that the algorithm that is set in chroma upscaling is only used to get chroma to the same resolution as luma. [...] If further upscaling is needed (...) the algorithm secified in image upscaling is used to upscale both chroma and luma.
As far as I know that is correct.
Shiandow is offline   Reply With Quote
Old 25th June 2014, 02:45   #17  |  Link
Asmodian
Registered User
 
Join Date: Feb 2002
Location: San Jose, California
Posts: 3,631
Quote:
Originally Posted by Shiandow View Post
That seems to be an actual bug in the AR algorithm. I don't think it's related to the other issues in this post, which I believe is a consequence of treating chroma and luma separately.
It probably is sort of related to the other issues in this post. The artifacts from resizing luma and chroma differently get magnified by the AR algorithm; something about them causes the AR algorithm to misbehave. A similar thing is happening here, the AR algorithm is confused by something so it creates an artifact. It might even be the same thing as I assume this artifact only happens when AR follows NNEDI3 doubling only the luma.
Asmodian is offline   Reply With Quote
Old 25th June 2014, 09:48   #18  |  Link
Shiandow
Registered User
 
Join Date: Dec 2013
Posts: 752
Well, it might be related but I think that some of the issues will happen anyway even when AR is behaving "correctly" just because you're not doing the same thing to the chroma and luma channels. Unfortunately I don't know enough about the anti-ringing algorithm to know why using AR seems to be worse than not using AR. I'm also not sure if it is always worse to use AR.

With linear light it is somewhat more obvious why it's causing problems, although I wonder if you could remove those by also using linear light for NNEDI3 and image upscaling.
Shiandow is offline   Reply With Quote
Old 29th June 2014, 04:10   #19  |  Link
StinDaWg
Registered User
 
Join Date: Jan 2014
Posts: 216
madshi, can we get a comment on this? Haven't seen you around in awhile.
StinDaWg is offline   Reply With Quote
Old 8th July 2014, 09:44   #20  |  Link
madshi
Registered Developer
 
Join Date: Sep 2006
Posts: 9,137
Can you please create a bug tracker entry with a link to this thread? Ideally with the original PNG used for these tests and maybe also with a video sample. Thanks.
madshi is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 04:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.