madVR - high quality video renderer (GPU assisted)

[RainyDog]

I've decided to make a big screenshot comparison post to compare the different upscaling and sharpening / post-processing options we have now. In order to make all this more "scientific", I've decided that we need a "ground truth" image to compare to, because that's the only way to properly judge if an algorithm produces good results or not. If we don't have a ground truth to compare to, all our evaluations are somewhat subjective. So here comes.

Let me finish by saying that all of the above is of course only *one* test image. Results could in theory vary with other test images. I would like to encourage anyone who is interested in this kind of comparisons to repeat the same tests I've done, with a different test image. But please make sure you have a ground truth to compare to, because only that allows us to objectively judge. Otherwise all we can say is subjective mumbo-jumbo, and every user will have different preferences.

-------

Comments welcome!

Great post madshi!

A couple of things; can you confirm the strength and number of passes you used for SuperRes and super-xbr strength please? I presume defaults but can't see it mentioned in your post.

Also, is SuperRes only really intended for upscaled content or can it be used as a mild sharpener for HQ 1080p material on a 1080p display to help bring out fine details?

[ryrynz]

Glad to hear you like my testing procedure. I tried to find a way to make things less subjective, to increase the chances that we may end up on the same page.


I'd like to see something similar comparing chroma upscaler accuracy.

[madshi]

can you confirm the strength and number of passes you used for SuperRes and super-xbr strength please? I presume defaults but can't see it mentioned in your post.
super-xbr-100, and highest possible strength (which is 4) for SuperRes. Strength 4 in v0.88.20 equals 8 passes with 1.0 strength in older madVR builds.

Also, is SuperRes only really intended for upscaled content or can it be used as a mild sharpener for HQ 1080p material on a 1080p display to help bring out fine details?
Currently SuperRes only works after upscaling. Maybe we'll find a way to make it work without upscaling, too (Shiandow had a suggestion for that, or alternatively we could use supersampling). But I'm not sure when I'll get around to that.

I'd like to see something similar comparing chroma upscaler accuracy.
That's much more complicated/difficult, I fear. E.g. Bilateral and SuperRes for chroma both look at the luma information, too. And studios might use anything from Nearest Neighbor Downscaling to Lanczos for chroma downscaling.

[Gagorian]

I've not seen Darbee myself yet. But I've been told it does this:

> It's a frame-delayed unsharp mask. So if there's no
> movement in the scene, it's just plain unsharp mask.
> If there's a lot of movement, it's a random artifact
> generator. If there's a little movement, like on a slow
> pan or a helicopter shot, it's an unsharp mask that is
> more aggressive on one side of a detail and less
> aggressive on the other side, adding a little shadow.

So for still images, it should be somewhat similar to AdaptiveSharpen. One big disadvantage of current Darbee implementations is that if you let the display upscale (e.g. 4K display), Darbee is applied before upscaling. That's like applying AdaptiveSharpen in "image enhancements". It's effective, but ugly. You'd get better quality doing it after upscaling. But AFAIK current Darbee implementations don't have the power to do 4K.

Anyway, personally I'm not really a fan of the look that unsharp mask style sharpening produces. To my eyes it looks like fat lines get even fatter. It seems to "bloat up" the image somehow, adding more contrast and pop, but making the image look more artificial. You can see some of that in my screenshot comparison post when comparing AdaptiveSharpen to FineSharp. The look which AdaptiveSharpen produces is typical for unsharp mask style sharpening.



I've been using it (after upscaling if needed) at 1080p on a 133" screen. I believe Darbee looks better in motion than when evaluating still images. I think the effect is ok up until about 30% strength and then it gets progressively more ugly. Until that point that it adds a nice amount of crispness/pop to the image and compensates for the slightly soft image of the projector (Sony HW50). I think the artifacts are the worst problem, especially on subtitles. Using lower strength does reduce the artifacts, but not completely.

I've been following this thread closely to see if any of the new algorithms could allow me to replace the Darbee. SuperRes looks like the best so far but I'm not sure if it works to improve the perceived sharpness of the projector like the Darbee does. Perhaps a touch of AS on top?

[madshi]

I can't really say for sure. Just play with the various options and let your eyes be the judge. And please report your findings here, would love to hear how you like SuperRes, AdaptiveSharpen etc compared to what the Darbee does.

[ryrynz]

SuperRes is near to FineSharp in detail level and sharpness, but at the same time manages to hide all artifacts caused by NNEDI3. So SuperRes is not only a good sharpener, but also manages to repair errors of the upscaling algorithm.

It's good at the standard radius setting of 0.66 but provides a rather soft result. Drop the radius to 0.20 and the image is now godly.
BTW at this radius, strength does very little 4 is a touch sharper than 1 and I prefer it at 4.

Comparison sample upscaled using a 2x pixel resize.

ORIGINAL
http://i.imgur.com/CE4Nuf6.png

NNEDI-256+SR-STR4-R0.66
http://i.imgur.com/pe9q1sJ.png

NNEDI-256+SR-STR4-R0.20
http://i.imgur.com/EbsOORf.png

Much more accurate to the source using SR at a radius of 0.20, lots of extra detail here all over. I have a new high standard...


That's much more complicated/difficult, I fear. E.g. Bilateral and SuperRes for chroma both look at the luma information, too. And studios might use anything from Nearest Neighbor Downscaling to Lanczos for chroma downscaling.

Having something is better than nothing.. might not be 100% accurate but *shrug* work with what you got.

[ikarad]

Bilinear+SuperRes (http://madVR.com/doom9/castle/ScalingAlgosHqSuperRes/CastleBilinearHqSuperRes.png) -|- Jinc+SuperRes (http://madVR.com/doom9/castle/ScalingAlgosHqSuperRes/CastleJincHqSuperRes.png) -|- super-xbr+SuperRes (http://madVR.com/doom9/castle/ScalingAlgosHqSuperRes/CastleSuperXbrHqSuperRes.png) -|- NNEDI3+SuperRes (http://madVR.com/doom9/castle/ScalingAlgosHqSuperRes/CastleNNEDI3HqSuperRes.png) -|- GroundTruth (http://madVR.com/doom9/castle/CastleBig.png)

What we can see here is that SuperRes works well even when using Bilinear upscaling. However, SuperRes does *not* remove aliasing artifacts caused by the upscaling algorithm. E.g. look at the roof edges of the left two towers. Both Bilinear and Jinc have aliasing problems there. super-xbr and NNEDI3 have not. Because of this reason, my recommendation would be to use either super-xbr or NNEDI3, followed by SuperRes, for best image quality. The difference between super-xbr and NNEDI3 is pretty small, if you follow it up with SuperRes with high strength. So using super-xbr should save some precious GPU performance. Using Jinc+SuperRes might be an option, too, but you'll likely get more aliasing problems compared to super-xbr+SuperRes.

Thanks for these comparisons.
I have one question:
super-xbr and nneedi116,32,64,128,256 are only available in image doubling. If I use image doubling, downscaling image is obligatory used for my display (I don't have movie with exactly 50% display resolution).

If I use jinc (jinc is only avalaible in image upscaling) in image upscaling, no need to use donwscaling image.

Even with downscaling, is it better to use superxbr or is it better to use jinc withtout downscaling?

[madshi]

It's good at the standard radius setting of 0.66 but provides a rather soft result. Drop the radius to 0.20 and the image is now godly.
It reproduces some more details, but it also adds a lot of aliasing. This might not be so apparent in the castle image, but I've seen it in other images. E.g. try this:

http://madshi.net/clown.png

I fear we may have to increase the radius to a value *higher* than 0.66 to get rid of aliasing problems. Sad but probably necessary... :(

super-xbr and nneedi116,32,64,128,256 are only available in image doubling. If I use image doubling, downscaling image is obligatory used for my display (I don't have movie with exactly 50% display resolution).

If I use jinc (jinc is only avalaible in image upscaling) in image upscaling, no need to use donwscaling image.

Even with downscaling, is it better to use superxbr or is it better to use jinc withtout downscaling?
I think from a quality point of view, using image doubling + downscaling should be better than straight upscaling. Not sure about GPU power, though. Depending on the exact scaling factor, sometimes doubling + downscaling might be faster or slower than straight upscaling.

[David]

What Didée said about Darbee:

Seems that is not quite comparable. FineSharp, as the name somehow suggets, deals mostly with enhancement of fine structures and edges. Darbee seems to implement something like a wide-range Unsharp-Mask with halo protection.
http://forum.doom9.org/showthread.php?p=1595526#post1595526

A year or some other back, I fiddled with a multi-scale sampled, (quasi-) median enhancer. Those results were more or less similar to what Darbee is achieving.

A (pretty naive and pretty slow) version is to simply use a wide-range median filter for sharpening.

# medianblur.dll
# repairSSE2.dll
# mt_masktools-25.dll

imagesource("D:\darbee_6_0_test.png").converttoyv12()

a = last
b = a.medianblur(12,8,8)
c = a.repair(a.repair(a.repair(a.repair(a.repair(a.repair(a.repair(a.repair(b,1),1),1),1),1),1),1),1)
d = mt_lutxy(a,c,"x x y - abs 1.62 2 ^ / 1 1.62 / ^ 1.62 3 ^ * x y - x y - abs 1.62 + / * +",U=3,V=3)

interleave(a,d)

return(last)


http://thumbnails106.imagebam.com/21481/8fdead214807192.jpg (http://www.imagebam.com/image/8fdead214807192) ==> http://thumbnails106.imagebam.com/21481/8717f9214807176.jpg (http://www.imagebam.com/image/8717f9214807176)

(Open in tabs to flip back-n-forth.)

http://forum.doom9.org/showthread.php?p=1595531#post1595531

Not yet. This is only a torchlight, in practice things shouldn't be done that simple. A particular problem is chroma handling, which must be done different. It would be needed to isolate a saturation layer, and work on that. When done directly like in this little script, nasty things will happen. Against a green background, a grey line will turn into purple, and such.
And for big framesizes like full-HD, a multiscale version is pretty much recommended. Doing a big-range median on full-HD is anything but funny, speed-wise.

http://forum.doom9.org/showthread.php?p=1595542#post1595542

[ryrynz]

It reproduces some more details, but it also adds a lot of aliasing. This might not be so apparent in the castle image, but I've seen it in other images. E.g. try this:

http://madshi.net/clown.png



Does an image that small really fit the bill as far as being comparable to typical use cases?


I fear we may have to increase the radius to a value *higher* than 0.66 to get rid of aliasing problems. Sad but probably necessary... :(


Could that be cheated by an additional anti-aliasing step afterwards? Probably overkill...
I do prefer the edges when using a higher radius but I'd rather have that detail there.
I wonder if Shiandow could focus his attention on this particular area at all, a high radius SR utilized for edges only.. I dunno.

What Didée said about Darbee:


Ugh, that's horrible.

[x7007]

Anyone is able to run CUVID Acceleration from LAV Video Filter with MADVR and not freeze and crash ? None and Write-Back works fine, just the mixed with Madvr + CUVID .

Nvidia 353.49
LavFilters 0.65
PotPlayer 1.6.55124 x32
Windows 10 10240 x64

[ashlar42]

madshi, fantastic service to users with your huge images comparison post. I loved it!

I read what you wrote about subs rendering on black bars. I'll wait patiently. Happy to hear you are aware of the need for it. :)

[RainyDog]

Thanks for these comparisons.
I have one question:
super-xbr and nneedi116,32,64,128,256 are only available in image doubling. If I use image doubling, downscaling image is obligatory used for my display (I don't have movie with exactly 50% display resolution).

If I use jinc (jinc is only avalaible in image upscaling) in image upscaling, no need to use donwscaling image.

Even with downscaling, is it better to use superxbr or is it better to use jinc withtout downscaling?

For scaling 720p>1080p on my 1080p display, I find image-doubling + downscaling to be superior to just upscaling with Jinc3 even with the extra (down)scaling process involved.

Using super-xbr, it's no more resource hungry either.

[omarank]

I've decided to make a big screenshot comparison post to compare the different upscaling and sharpening / post-processing options we have now....

Thank you madshi for taking time to write such a detailed post. It not only shows how good SuperRes is but also the ideal approach to do the testing. The confusion which I had about the absolute best upscaling method in madVR, given there are so many choices now, is cleared up.

[David]

Ugh, that's horrible.

I think so too.

on another topic. Anyone knows if random dithering in madvr 86 was with coloured noise?

With madvr new versions if i untick colour noise i can see banding in the skin of a person in a scene. With madvr 86, evr and wd player that banding is not there.

[huhn]

I think so too.

on another topic. Anyone knows if random dithering in madvr 86 was with coloured noise?

With madvr new versions if i untick colour noise i can see banding in the skin of a person in a scene. With madvr 86, evr and wd player that banding is not there.

can you make screens of this?

[x7007]

Anyone managed to run PotPlayer in windows 10 with CUVID without total player freeze ?

[huhn]

do you get this problem only with madVR?

and why do people still use CUVID? DXVA copyback is just better.

[x7007]

do you get this problem only with madVR?

and why do people still use CUVID? DXVA copyback is just better.



hmm , I don't know, I prefer the CUVID, I am just used to it.

I have the issue when I use madvr+cuvid and only with madvr is the issue , EVR Custom present + Cuvid works fine

[David]

can you make screens of this?

Not right know because i am on holidays, but sure i will even if them are not neccesary for the answer to the question. I have curiouty about this since madvr 87.X

[Thunderbolt8]

which 3dlut files created by which program are currently supported by madvr? I need to create a 3dlut file in which I can modify the input range to modify the black levels and also to change the gamma value slightly

[mbordas]

Anyone is able to run CUVID Acceleration from LAV Video Filter with MADVR and not freeze and crash ? None and Write-Back works fine, just the mixed with Madvr + CUVID .


seems to be a bug in the driver. It only happens on certain content. Use DXVA native or CB in LAVvideo, and uncheck the two DXVA options in madVR under "trade quality for performance".

[huhn]

which 3dlut files created by which program are currently supported by madvr? I need to create a 3dlut file in which I can modify the input range to modify the black levels and also to change the gamma value slightly


you can use madVR to change the "input" range by changing the output range in madVR and use argyllCMS after it. I do it this way too. to better counter my crushed blacks and white clipping on my screen.

of cause you can change the gamma curve in argyllCMS too.

Not right know because i am on holidays, but sure i will even if them are not neccesary for the answer to the question. I have curiouty about this since madvr 87.X
madVR should never show banding when EVR isn't showing it so this could be a serious bug.

if i remember correctly color noise is checked by default. and it just has lower luma noise for more chroma noise it shouldn't be more smooth.

[aufkrawall]

Yeah, I wasn't meaning to say that AdaptiveSharpen should never be used. It could be useful to sharpen overly soft sources, or simply to add some "pop". However, "Upscaling Refinement" was always planned by me to post-process an upscaled image in such a way that it gets nearer to the ground truth. So, I would say that for this specific purpose I would not recommend AdaptiveSharpen.

I have to totally disagree.
It's the only sharpen algorithm that you can set & forget with a low strength, while still having a perceivable effect.
I wouldn't use more than strength 0.2 and probably only us it as an UR.
A SR strength of 4 is very unrealistic, it e.g. looks very bad with my cartoon example (very destructive ringing) and with a low strength, SR often hardly sharpens the image in a perceivable way.
AS 0.2 still looks very nice with the cartoon example when I use NNEDI3 64 to double and super-xbr 50 to quadruple.

For example, SR has a very minimal effect on scaling 720p -> WQHD filmed content, while AS really makes the image look more detailed without being destructive.

I already posted that fence example where AS looks really nice. I'm gonna post another example with extreme scaling to demonstrate that AS is very useful.

[madshi]

What Didée said about Darbee: [...]
Yes, I remember seeing those spider images. And yes, he confirms it's some sort of unsharp mask stuff.

Does an image that small really fit the bill as far as being comparable to typical use cases?
Maybe not. But look at recent posts from Eyldebrandt. He showed screenshots of typical use cases where SuperRes introduced aliasing. He hasn't replied to any of my recent posts, though. So I'm not sure whether increasing the radius setting would help in his case.

I wonder if Shiandow could focus his attention on this particular area at all, a high radius SR utilized for edges only.. I dunno.
Maybe, I'm not sure.

on another topic. Anyone knows if random dithering in madvr 86 was with coloured noise?

With madvr new versions if i untick colour noise i can see banding in the skin of a person in a scene. With madvr 86, evr and wd player that banding is not there.
After a quick look at the source code of v0.86.11 it seems to be colored noise, but I'm not 100% sure right now. No time to analyze that. You could create a unicolored BMP with MS Paint and then take a screenshot of madVR and zoom it up and check if the dithering is colored or not.

which 3dlut files created by which program are currently supported by madvr?
http://madVR.com

I have to totally disagree.
It's the only sharpen algorithm that you can set & forget with a low strength, while still having a perceivable effect.
I wouldn't use more than strength 0.2 and probably only us it as an UR.
A SR strength of 4 is very unrealistic, it e.g. looks very bad with my cartoon example (very destructive ringing) and with a low strength, SR often hardly sharpens the image in a perceivable way.
AS 0.2 still looks very nice with the cartoon example when I use NNEDI3 64 to double and super-xbr 50 to quadruple.

For example, SR has a very minimal effect on scaling 720p -> WQHD filmed content, while AS really makes the image look more detailed without being destructive.

I already posted that fence example where AS looks really nice. I'm gonna post another example with extreme scaling to demonstrate that AS is very useful.
But now we're pretty much back in subjective mumbo-jumbo land. If you take a random image / video frame and upscale it and try to judge which algorithm looks "good" on it, it's all very subjective, matter of taste, and everybody will have different preferences. Which is exactly why I tried to analyze all the algorithms using a more objective scientific approach in this (http://forum.doom9.org/showpost.php?p=1730855&postcount=31986) post. Did you carefully read it and study the image comparisons? Do you disagree with anything in that post?

Again, that post is discussing the best algos to turn a low image resolution picture into the most likely truthful high resolution picture. The purpose of "upscaling refinement" is not to pimp up soft sources, but simply to provide accurate scaling which gets as near to the "ground truth" as possible.

If you want to add some pop to your sources, or fix overly soft sources, then once again it's a matter of taste which algorithm might look good to you. AdaptiveSharpen might be great for that. But that's a different usage area than trying to find a scientifically ideal algo combination for upscaling.

The reason why your cartoon example has destructive ringing with SuperRes is that the cartoon image already has strong ringing in the source. The proper way to handle this is to dering the image before upscaling it (and to improve the SuperRes anti-ringing filter, which is not optimal yet). Algorithms for that to come in a future madVR version. E.g. look here for a work-in-progress:

Cartoon example:
super-xbr-100 (http://madVR.com/temp/CartoonSuperXbr100.png) -|- dering + super-xbr-100 (http://madVR.com/temp/CartoonDeringSuperXbr100.png) -|- dering + super-xbr-100 + SuperRes (http://madVR.com/temp/CartoonDeringSuperXbr100SuperRes.png)

If we had a ground truth for this image, I bet it would be *much* nearer to the SuperRes processed result than to the unprocessed result. There are still artifacts in the processed image, though, I'll give you that. I think the artifacts are mostly coming from the SuperRes AR filter needing some improvements. So there's still work to be done.

[nemoW]

It reproduces some more details, but it also adds a lot of aliasing.

<...>

I fear we may have to increase the radius to a value *higher* than 0.66 to get rid of aliasing problems. Sad but probably necessary... :(
Maybe it's time to add some kind of shader AA filter like FXAA/SMAA?
It would be especially helpful for sources like this: https://yadi.sk/i/hp2iZS68hC554

[aufkrawall]

But now we're pretty much back in subjective mumbo-jumbo land. If you take a random image / video frame and upscale it and try to judge which algorithm looks "good" on it, it's all very subjective, matter of taste, and everybody will have different preferences. Which is exactly why I tried to analyze all the algorithms using a more objective scientific approach in this (http://forum.doom9.org/showpost.php?p=1730855&postcount=31986) post. Did you carefully read it and study the image comparisons? Do you disagree with anything in that post?

Yes. But what do we want to find out?
I think you have a very good point with that naturalistic image which should look as close as possible to the source.
But I also think we need a more comprehensive impression.

Upscaling naturally softens an image, and we're not always upscaling an image with the factor you have used for your example.
So, will SuperRes still "add" enough sharpness if the scaling factor is much higher? madVR offers to run a sharpening pass after each doubling, but you said that SuperRes has less work to do with each pass.
This covers with my following observation:
SuperRes either adds very little sharpness at a reasonable strength or the image looks very odd with higher strength.
I don't think we should be "too scientific" since it's a highly subjective or at least non-objective issue. Maybe if you run a computational analysis which compares each pixel to match the source the most, some algorithms may perform very well, but maybe the result is still not desirable for many or even most users users due to subjective discomfort?
Ok, here comes the example:
Original (yes, there are some weird ghosting artifacts in the source):
http://abload.de/thumb/origzpprq.png (http://abload.de/image.php?img=origzpprq.png)

NNEDI3 64 quadrupling + Jinc AR:
http://abload.de/thumb/nnedi364quadz4oqs.png (http://abload.de/image.php?img=nnedi364quadz4oqs.png)

+ FS 0.8 UR:
http://abload.de/thumb/fs0.8ureirim.png (http://abload.de/image.php?img=fs0.8ureirim.png)

LS default UR:
http://abload.de/thumb/lsdefurl5pbf.png (http://abload.de/image.php?img=lsdefurl5pbf.png)

SuperRes 3 (else default):
http://abload.de/thumb/sr3deftao2h.png (http://abload.de/image.php?img=sr3deftao2h.png)

AS 0.2 UR:
http://abload.de/thumb/as0.2urebp5q.png (http://abload.de/image.php?img=as0.2urebp5q.png)

FS & LS are ring feasts. SuperRes adds some details from the source, which is remarkable, but it fails to really make the image look sharp. It's impossible in this case that the sharpness of SuperRes could be closer to the source than AS. So maybe SuperRes is nearer to the source apart from sharpness in this example, but I doubt many people would describe the result of it better than AS.


The reason why your cartoon example has destructive ringing with SuperRes is that the cartoon image already has strong ringing in the source. The proper way to handle this is to dering the image before upscaling it (and to improve the SuperRes anti-ringing filter, which is not optimal yet). Algorithms for that to come in a future madVR version. E.g. look here for a work-in-progress:

Cartoon example:
super-xbr-100 (http://madVR.com/temp/CartoonSuperXbr100.png) -|- dering + super-xbr-100 (http://madVR.com/temp/CartoonDeringSuperXbr100.png) -|- dering + super-xbr-100 + SuperRes (http://madVR.com/temp/CartoonDeringSuperXbr100SuperRes.png)

If we had a ground truth for this image, I bet it would be *much* nearer to the SuperRes processed result than to the unprocessed result. There are still artifacts in the processed image, though, I'll give you that. I think the artifacts are mostly coming from the SuperRes AR filter needing some improvements. So there's still work to be done.
Here's the original frame (with Jinc AR for chroma):
http://abload.de/thumb/adorigymo4j.png (http://abload.de/image.php?img=adorigymo4j.png)
I think it's not mean to say that super-xbr + SuperRes looks totally cruel regarding ringing in this example. :devil:
Even upscaling the original 720p source to WQHD with NNEDI3 64 doesn't look great with SuperRes with a strength of 1. The ringing in the face of the woman with the red dress gets much more visible than e.g. with AS 0.2.

Of course SuperRes and AS can be combined to achieve more sharpness (which I think is "realistic"). But that ringing really needs to be fixed and the radius of 0.66 is definitely too low to prevent aliasing. Haven't tested yet which value I'd prefer.

[leeperry]

No, it only looks at one frame at a time. It pretty much does what Shiandow explained in that post you quoted earlier. But Shiandow's explanation was for the old SuperRes algo (where we still had number options for anti-aliasing and anti-ringing etc). The new algorithm is based on some newer scientific papers. It works like this:

0) Let e.g. super-xbr upscale the OI (original image). Result: UI (upscaled image).
1) Downscale the UI back to the same size as the OI (using: LQ=Bilinear; HQ=Bicubic).
2) Substract the OI from the downscaled UI, which gives us a sort of "error" image (EI) in the size of the OI.
3) Modify the UI in a clever way, by making use of the EI, to reduce the error.

Steps 1) to 3) is one SuperRes pass. If you want 2 SuperRes passes, you simply perform 1,2,3) twice.

All the magic happens in 3). And this code totally changed from the old to the new SuperRes algorithm. The other parts mostly remained unchanged. The old SuperRes code introduced aliasing and ringing, which is why Shiandow had added extra anti-aliasing and anti-ringing algos. The new SuperRes algorithm doesn't (or shouldn't) produce aliasing, anymore, and not as much ringing as the old code, so that's why there's no anti-ringing and anti-aliasing code in the new algorithm, anymore. I did add my own anti-ringing code in, though.
OK thanks for the detailed explanation, so it's a smarter dithering algorithm but I take it that using infos from the previous frames would be impossible to do in realtime and would require rocket science in order to guess what to keep and what to ignore. I've always been impressed by the effect of SR on 24p motion blur, really great looking :cool:

So the only difference between LQ/HQ is going from bilinear to bicubic, I guess that does put things back into perspective but I would still like to run more fair comparisons with weak LQ in .15 and weaker NNEDI3.

I've done all stuff in windowed mode. I've selected a ground truth image which in 100% view exactly fit in the media player in windowed mode, to make things easier for me. The ground truth is already a downscaled image from an even larger original image. So I could choose the exact size I wanted to have for the ground truth.

Please make sure you downscale using madVR with AR (anti-ringing). Don't downscale using an image/photo editor, unless it also has AR (I'm not aware of any that has).

How about dithering, can I leave monostatic ED2 engaged without tempering with the end results?

I would use another castle picture and a screenshot from a 4K mastered untouched BD.

I guess it would also make sense to use a noisy 1080p BD screenshot, because it would appear that SR likes to repair errors, some smart upscalers fail guessing and we are mostly using all this on low res noisy stuff to begin with.

If working with MPC-HC these procedures will enable you to capture specific frames.
It works that way, but you can also setup your own keybord shortcuts
OK thanks, sounds like a plan!

[ikarad]

Why at 23.976 and 24Hz, in osd there is repeated frames line and at 30,50 or 60Hz, there is no repeated frames line?


24 Hz
http://tof.canardpc.com/preview2/3179f29b-ba0b-4a84-ac72-e6ca35bd9da7.jpg (http://tof.canardpc.com/view/3179f29b-ba0b-4a84-ac72-e6ca35bd9da7.jpg)
60Hz
http://tof.canardpc.com/preview2/540d2a3e-6ce6-44e3-b8b9-d8a7b05bb69e.jpg (http://tof.canardpc.com/view/540d2a3e-6ce6-44e3-b8b9-d8a7b05bb69e.jpg)

[aufkrawall]

When there are obviously too many repeated frames or smooth motion is enabled, the information isn't shown (because it's of no use).

[Q-the-STORM]

I don't think we should be "too scientific" since it's a highly subjective or at least non-objective issue. Maybe if you run a computational analysis which compares each pixel to match the source the most, some algorithms may perform very well, but maybe the result is still not desirable for many or even most users users due to subjective discomfort?

Well that is *exactly* the reason madshi's comparisons were comparing against a "ground truth" image... with that we take away most of the subjectivity...
there is no "I like this image better because.." there is only "This image is closer to the ground truth"..
someone may want the image to be sharper but you can always sharpen... some want the image to be more colorful, you can always add saturation... that's stuff that can always be done to the users preference...
but the point here is a different one...
the point is not to make the image more appealing, the point is for madVR to upscale as accurate as possible to the ground truth...

there is no way to know if an upscale is accurate if we don't have a higher res image, that's why the ground truth image is so important....

if upscale1 is close to the ground truth, if upscale2 is close to the ground truth and upscale3 is close to the ground truth, then we can assume that images that don't have a ground truth image to compare to, are also accurately upscaled...

this is not about "fixing" an image to your viewing prefenrece... if it was shot slightly unsharp, we want madVR to upscale it to an unsharp image... you can always sharp the image afterwards if you want, but people that don't should be given the option not to...

and making the image more sharp because it *might* be more accurate without having a ground truth image to verify, is very subjective...
people will think the image is more accurate because they like it better...



Why at 23.976 and 24Hz, in osd there is repeated frames line and at 30,50 or 60Hz, there is no repeated frames line?

like aufkrawall said, there is no point in having it at 30 or 60Hz..... simply because 24 frames at 60Hz will have 36 repeated frames every second, so the counter would be over 1000 after 30 seconds... there is no value in having it in that case..

[aufkrawall]

if it was shot slightly unsharp, we want madVR to upscale it to an unsharp image
The source posted by me is very small, but if I get close enough, I think the foreground is not unsharp at all and thus the upscaled image shouldn't be blurry either.

[ikarad]

like aufkrawall said, there is no point in having it at 30 or 60Hz..... simply because 24 frames at 60Hz will have 36 repeated frames every second, so the counter would be over 1000 after 30 seconds... there is no value in having it in that case..

When there are obviously too many repeated frames or smooth motion is enabled, the information isn't shown (because it's of no use).
Thanks but I don't understand why this line is not visible at 120hz. 120hz is a multiple of 24.

[Q-the-STORM]

The source posted by me is very small, but if I get close enough, I think the foreground is not unsharp at all and thus the upscaled image shouldn't be blurry either.

I didn't mean your picture specifically, I meant any picture in general...
e.g. if a filmmaker wanted it to be slightly unsharp...

of course in some cases of very downscaled images you will never get an accurate upscale, simply because too much information was lost... but videos at e.g. 480x270 are pretty rare... usually we are talking about at least DVD resolutions, which should have enough information...

Thanks but I don't understand why this line is not visible at 120hz. 120hz is a multiple of 24.

same applies at 120Hz...

24 frames at 120Hz will have 96 repeated frames every second, so the counter would be almost 1000 after 10 seconds...

[huhn]

the repeat line is only there for rendered frames that needs to be repeated for what ever reason. this is not about presentation.

as long as this is not needed you will not see this line.

the refresh rate has nothing to do with this.

[6233638]

Thank you for the in-depth sharpness comparisons everyone.
I had a look over everything posted over the last day or two, and did some testing of my own.

My concern is that most of the test images that people are using as a reference are already over-sharpened, containing pre-existing ringing artifacts and aliasing.
So in order to match the source image, you end up using overly sharp post-processing, which seems to be introducing a lot of artifacts around high-contrast edges.

I would focus more on good Blu-ray sourced images downscaled 50% and trying to match the original sharpness, than using photographs or content from what appears to be low-quality or re-compressed sources, if the goal is restoration rather than enhancement.
I do also wonder if people are checking the results on small PC monitors, or their tv/projection setups.
In what testing I have been able to do, SuperRes seems to be introducing aliaising and particularly ugly high-frequency ringing at all strengths.
Even with a strength of 1 and radius of 1.00 it's still adding aliasing - whether that is real-world images, or test patterns.
Of course if you're combining that with a scaler that already suffers from aliasing (basically anything other than Jinc or image doubling) perhaps the additional aliasing is not that noticeable?

I finally got around to trying out Super-XBR too, which seems to add a lot of low-frequency ringing with sharpness set above 25 - which you may not notice if you're looking at a poor quality source that already has ringing artifacts, rather than a good Blu-ray disc.
Combining Super-XBR with sharpening to achieve similar results to 16-neuron NNEDI3 ended up with higher render times and worse image quality than just using NNEDI3.

Here's one of the images that I was testing with which seemed to show aliasing quite well:
Source (http://abload.de/img/mononoke-source0lo7u.png)
Downscaled (http://abload.de/img/mononoke-ll1er51.png)
I'm not really happy with how that looks using any of the sharpening options, or scaling other than NNEDI3.
But perhaps I haven't spent enough time with the new options, I really haven't been able to do much testing with anything that's been introduced in the last couple of months at all.

[madshi]

Maybe it's time to add some kind of shader AA filter like FXAA/SMAA?
I think FXAA/SMAA works by knowing the vector graphics details (a line goes from one point to another). madVR does not have that kind of information for video. Game rendering and video rendering are *TOTALLY* different. Game rendering is vector graphics. Video rendering is bitmap graphics.

Upscaling naturally softens an image, and we're not always upscaling an image with the factor you have used for your example.
So, will SuperRes still "add" enough sharpness if the scaling factor is much higher? madVR offers to run a sharpening pass after each doubling, but you said that SuperRes has less work to do with each pass.
I've tried quadrupling an image, and it makes no big difference to image sharpness whether I run SuperRes after doubling and quadrupling, or only after quadrupling. Based on that I'd say that SuperRes should work ok, regardless of the upscaling factor. At least in theory. It might make sense to double check that. But please let's do that with a ground truth image to compare to, because otherwise it's all nothing but subjective again.

Ok, here comes the example:
Original (yes, there are some weird ghosting artifacts in the source):

[...]

FS & LS are ring feasts. SuperRes adds some details from the source, which is remarkable, but it fails to really make the image look sharp. It's impossible in this case that the sharpness of SuperRes could be closer to the source than AS. So maybe SuperRes is nearer to the source apart from sharpness in this example, but I doubt many people would describe the result of it better than AS.
Where's the ground truth for this image? Do you *want* our discussion to not be objective? The problem with not having a ground truth is that we can discuss for hours how the upscaled image should look like without ever coming to an agreement. Maybe your very tiny original image was downscaled with a very soft algorithm? In that case the downscaling algorithm was at fault for the very soft result produced by SuperRes. But without having a ground truth, I can't even double check if the downscale was produced with a reasonably good downscaling algorithm or not. I can't check anything, because I've nothing to compare to.

I think you're missing the whole point of my "research": I've no problem if you want to sharpen your video. No problem at all. But artificially sharpening video is not something that should ever be enabled by default in madVR. On the other hand, faithfully upscaling sources, by producing images which are as near to the ground truth as possible, that is something that would make a great default setting in madVR.

Obviously faithful upscaling is useless if it's full of artifacts. So maybe there needs to be more work done on fixing the ringing and aliasing artifacts SuperRes might still produce. But do you understand my motivation here? I'm not trying to tell users what to do. I'm trying to find good default settings for madVR to produce the most accurate image quality. Some users might prefer it sharper, so they can add sharpening on top like e.g. AdaptiveSharpen. Fine with me.

Here's the original frame (with Jinc AR for chroma):
http://abload.de/image.php?img=adorigymo4j.png

I think it's not mean to say that super-xbr + SuperRes looks totally cruel regarding ringing in this example. :devil:
But please have a look at all the lines in these images. Take a step back from your monitor and compare the SuperRes image with the original frame, and with the AdaptiveSharpen image. Here's an extract using super-xbr for upscaling:

http://madVR.com/temp/CartoonComparison.png

You should clearly see that all the lines in the original frame are noticeably thinner than in the AdaptiveSharpen (and unprocessed) image. And you should also see that SuperRes manages to thin most of the lines back to what they should be. YES, there are artifacts. But if you try to ignore them for a second, don't you see that SuperRes brings the upscaled image much nearer to the original image? In contrast to that, AdaptiveSharpen puts more weight on the fat bloated up lines, to make them even more different from the original source. So AdaptiveSharpen moves the image away from what the upscaled image should ideally look like. To you it might look pleasing. To my eyes it looks bad.

I think your whole approach is "wrong". You're trying to find algorithms which produce results which you personally find pleasing. The problem with that is that what is pleasing to you might be totally ugly to the next guy. Instead I'm trying to find algorithms which produce accurate results that are faithful to the original source. If there are artifacts, I have to fix them. But it's still the right approach. If the final results of my tweaks are too soft for your taste, you can always touch them up by adding some sharpen on top. At least you would then sharpen the thin lines instead of the bloated lines.

OK thanks for the detailed explanation, so it's a smarter dithering algorithm
No, it's got nothing to do with dithering.

I take it that using infos from the previous frames would be impossible to do in realtime and would require rocket science in order to guess what to keep and what to ignore.
I've never tried that, I've no idea if that would work and how difficult it would be. It's a *totally* different kind of algorithm than the one developed by Shiandow. Please try not to be confused that there are 2 totally different approaches which share the same "SuperRes" name.

FWIW, some time ago there was a thread in the AviSynth section about the other SuperRes algorithm you're talking about. The AviSynth experts seemed to agree that this kind of stuff (looking at multiple consecutive frames) would only work with heavily aliased sources. But our usual DVD and Blu-Ray sources are typically soft, and not aliased. So for those the multi-frame-SuperRes would probably not work at all.

How about dithering, can I leave monostatic ED2 engaged without tempering with the end results?
Dithering is totally separate, you can use whatever you want. The difference between different dithering methods is extremely small compared to differences between upscaling and sharpening algorithms.

I guess it would also make sense to use a noisy 1080p BD screenshot, because it would appear that SR likes to repair errors, some smart upscalers fail guessing and we are mostly using all this on low res noisy stuff to begin with.
Yes, might be an interesting test.

Well that is *exactly* the reason madshi's comparisons were comparing against a "ground truth" image... with that we take away most of the subjectivity...
there is no "I like this image better because.." there is only "This image is closer to the ground truth"..
someone may want the image to be sharper but you can always sharpen... some want the image to be more colorful, you can always add saturation... that's stuff that can always be done to the users preference...
but the point here is a different one...
the point is not to make the image more appealing, the point is for madVR to upscale as accurate as possible to the ground truth...

there is no way to know if an upscale is accurate if we don't have a higher res image, that's why the ground truth image is so important....

if upscale1 is close to the ground truth, if upscale2 is close to the ground truth and upscale3 is close to the ground truth, then we can assume that images that don't have a ground truth image to compare to, are also accurately upscaled...

this is not about "fixing" an image to your viewing prefenrece... if it was shot slightly unsharp, we want madVR to upscale it to an unsharp image... you can always sharp the image afterwards if you want, but people that don't should be given the option not to...

and making the image more sharp because it *might* be more accurate without having a ground truth image to verify, is very subjective...
people will think the image is more accurate because they like it better...
Yep, exactly.

Thank you for the in-depth sharpness comparisons everyone.
I had a look over everything posted over the last day or two, and did some testing of my own.

My concern is that most of the test images that people are using as a reference are already over-sharpened, containing pre-existing ringing artifacts and aliasing.
So in order to match the source image, you end up using overly sharp post-processing, which seems to be introducing a lot of artifacts around high-contrast edges.

I would focus more on good Blu-ray sourced images downscaled 50% and trying to match the original sharpness, than using photographs or content from what appears to be low-quality or re-compressed sources, if the goal is restoration rather than enhancement.
I do also wonder if people are checking the results on small PC monitors, or their tv/projection setups.
In what testing I have been able to do, SuperRes seems to be introducing aliaising and particularly ugly high-frequency ringing at all strengths.
Even with a strength of 1 and radius of 1.00 it's still adding aliasing - whether that is real-world images, or test patterns.
Of course if you're combining that with a scaler that already suffers from aliasing (basically anything other than Jinc or image doubling) perhaps the additional aliasing is not that noticeable?

I finally got around to trying out Super-XBR too, which seems to add a lot of low-frequency ringing with sharpness set above 25 - which you may not notice if you're looking at a poor quality source that already has ringing artifacts, rather than a good Blu-ray disc.
Combining Super-XBR with sharpening to achieve similar results to 16-neuron NNEDI3 ended up with higher render times and worse image quality than just using NNEDI3.

Here's one of the images that I was testing with which seemed to show aliasing quite well:
Source (http://abload.de/img/mononoke-source0lo7u.png)
Downscaled (http://abload.de/img/mononoke-ll1er51.png)
I'm not really happy with how that looks using any of the sharpening options, or scaling other than NNEDI3.
But perhaps I haven't spent enough time with the new options, I really haven't been able to do much testing with anything that's been introduced in the last couple of months at all.
Thanks, the Mononoke image will be useful. Do you have some samples showing aliasing with filmed content, too? And some samples where that ugly high-frequency detail is especially evident? Unfortunately all the users who complain about aliasing either post no samples at all, or just one image. Would really be useful to have a wider set of samples to work with. PNG images should suffice. (And please always include the original 1080p image, as you've done with Mononoke). Thanks!

[har3inger]

Just FYI, SMAA and FXAA are post-processing shaders that work on the frame buffer by finding edges in the image and smart blending them. Unlike MSAA, you don't need any info regarding the object geometry, and could probably use FXAA or SMAA as photoshop plugins if you wanted to. IMO, they're sort of specific for hard-edge staircase aliasing you get from rasterization and don't really have a place in film anti-aliasing. However, super-xbr was designed for emulators at first, and at least I didn't think it would work at all for film content until I tried it and was amazed by it. Maybe these post-processing filters could be the same.

Keep in mind that because SMAA and FXAA have no access to geometry data, they do jack all for temporal aliasing or crawling edges, which thankfully aren't really that common in filmed content.

[Anime Viewer]

Maybe it's time to add some kind of shader AA filter like FXAA/SMAA?
It would be especially helpful for sources like this: https://yadi.sk/i/hp2iZS68hC554

Just FYI, SMAA and FXAA are post-processing shaders that work on the frame buffer by finding edges in the image and smart blending them. Unlike MSAA, you don't need any info regarding the object geometry, and could probably use FXAA or SMAA as photoshop plugins if you wanted to. IMO, they're sort of specific for hard-edge staircase aliasing you get from rasterization and don't really have a place in film anti-aliasing. However, super-xbr was designed for emulators at first, and at least I didn't think it would work at all for film content until I tried it and was amazed by it. Maybe these post-processing filters could be the same.

Keep in mind that because SMAA and FXAA have no access to geometry data, they do jack all for temporal aliasing or crawling edges, which thankfully aren't really that common in filmed content.

If you have an Nvidia GPU you can force FXAA and other similar antialasing on in Nvidia control panel in the program settings area. You apply it to mpc-hc (or whatever video player you are using with madVR). As the description in Nvidia Control Panel will tell you "FXAA is a fast shader-based post-processing technique that can be applied to any program, including those which do not support other forms of hardware-based antialiasing. FXAA can be used in conjunction with other antialiasing settings to improve overall image quality. Note that enabling this setting globally may affect all programs rendered on the GPU, including video players and the Windows desktop." Since it applies to whatever video player you assign to it in Nvidia by changing it from Use global setting (Off) to "On" madshi shouldn't need to do anything to madVR. You may not see any significant visual difference between having it off or on, and it could actually produce dithering or artifacts. As it says under usage scenarios:
"Turn FXAA off it you notice artifacts or dithering around the edges of objects, particularly around text". "Turn FXAA on to improve image quality with a lesser performance impact than other antialiasing settings"

[ryrynz]

Thanks, the Mononoke image will be useful.

Compared to MPDN the results from SuperRes are quite different. madVR's version is better with line thickness but also adds a boat tonne of aliasing in comparison.
It certainly looks like you're on the right sort of path though with SuperRes, whatever you changed there seems to bring it much closer to the original, just gotta deal with those unwanted artifacts.

[Akeno]

do you get this problem only with madVR?

and why do people still use CUVID? DXVA copyback is just better.

Are we referring to the H/W acceleration in LAV Video? Wouldn't CUVID be the most efficient if the user has an NVIDIA card?

[mrcorbo]

Are we referring to the H/W acceleration in LAV Video? Wouldn't CUVID be the most efficient if the user has an NVIDIA card?

Nope. (http://forum.doom9.org/showpost.php?p=1729879&postcount=19389)

[Akeno]

Nope. (http://forum.doom9.org/showpost.php?p=1729879&postcount=19389)

Well then. This brings up other questions I've never thought about: LAV video forces you to dither videos. How does that work with madVR is also dithering videos? Does the video go through two dithering processes?

[ryrynz]

Well then. This brings up other questions I've never thought about: LAV video forces you to dither videos. How does that work with madVR is also dithering videos? Does the video go through two dithering processes?

LAV defers to madVR for dithering by default, also LAV will only dither when required say when converting from 10-bit to 8-bit.

[Warner306]

@madshi

Can you take a look at rendering stats with Kodi DSPlayer? Upscaled videos (720p -> 1080p) are reporting rendering stats of 1-2ms. This is with v0.88.20.

For anyone interested in giving Kodi DSPlayer a try, a build of Kodi v15 Final is now available.

DSPlayer Download Link: http://forum.kodi.tv/showthread.php?tid=223175

Detailed set-up guide in my signature.

[Telion]

Also reporting unrealistically low rendering stats in 88.20.

[Akeno]

I've made a short comparison along the lines of madshi's post comparing Super-XBR and NNEDI3 in regards to SuperRes.
Source chosen specifically for the details in the eyes. These get extremely soft when upscaling.

GroundTruth (http://i.cubeupload.com/SNi1yK.png) -|- 50% downscaled (http://i.cubeupload.com/LQD0D4.png)
https://i.imgur.com/04QeveP.png

Without Refinement
NNEDI3 (http://i.cubeupload.com/FyLoQ2.png) -|- Super-XBR (http://i.cubeupload.com/dmWWfc.png)

I'm using s-xbr 100 for these comparisons. NNEDI looks a better than s-xbr. You can see that the lines are a bit thicker especially in her eyes and there's small ringing around her chin and on the flash cards she's holding. S-xbr is a bit sharper than NNEDI in this example but NNEDI is closer to the groundtruth here.

SuperRes: Strength 2 / Radius 0.66
NNEDI3 (http://i.cubeupload.com/rD8EEE.png) -|- Super-XBR (http://i.cubeupload.com/APvFPU.png)

With SuperRes, the images look nearly identical. NNEDI benefits a tiny bit from SuperRes. The image is slightly sharpened at the cost of tiny aliasing all around. You can see it on the lines of the flashcards. Super-XBR benefits a lot more from SuperRes as the thickness around the lines is significantly reduced but the ringing is still present.

Conclusions
So here's SuperRes on cartoon and anime content. No horrible directional artifacts to be found unlike the trees in the castle examples. All in all, NNEDI3 and Super-XBR produce nearly identical results when combined with SuperRes. NNEDI3 is still superior though as Super-XBR does produce more artifacts (look at the lower edge where the glass meets the door frame next to her shoulder and on the edges of her uniform) and ringing. If the image could be sharpened just a teeny bit more, we'd be extremely close to the groundtruth. Maybe a change in finesharp's algorithm to decrease artifacts or a denoise filter with finesharp could do the trick.

I'm also going to take back what I said about adaptive sharpen (http://i.cubeupload.com/SdhDo6.png) as an image enhancer. It completely ruins the soft glow she has on her face and makes the image look processed.

[har3inger]

Akeno: Have you tried using higher sharpness (125, 150) Super-XBR to try and match the sharpness you get from nnedi3?

These comparisons are very interesting. It seems like perhaps SuperRes itself is the primary determinant of final image quality. Might be worthwhile to do some tests of jincAR or even bicubic75AR to allow more headroom for more SR passes.

[pureocean]

Dear @madshi,

Firstly, thank you very much for this renderer. It's very useful. Especially on NT6 and NT10. Because of EVR can't support properly seeking, jumping in videos (it's happening shutters when seeking). Why important to me? I can't reading subtitles fast. Therefore I press often back (left) key in media players.

I have black screen problem. My GPU is Radeon HD 3200. I use Potplayer (last version). Catalyst display driver is last version. A sample video (https://sites.google.com/site/pureocean1/madvr_black_screen.mkv?attredirects=0&d=1). When in this video jump to 07, 08, 09, 10 and 18, 19, 20 seconds, it's displaying black screen for an instant. Which settings can fix this problem? I don't know using how "madVR debug.ax".

PS: I'm sorry, my bad English.

[Braum]

I did a short comparison of upscaling refinement options, mainly for myself, needed to know what was best for anime content.

Source : 1280x720
Monitor : 1680x1050

Chroma upscaling : Super-xbr 75
Image downscaling : Lanczos 4 AR LL
Image doubling : Super-xbr 75 (double luma/chroma)
Image upscaling : Jinc AR

Zoom 400%

Original (http://img11.hostingpics.net/pics/705472Original.png)

SuperRes Strengh 2 Radius 0.65 (http://img11.hostingpics.net/pics/115439SR2065.png) (the higher the strengh, the worst the result)

Adaptive Sharpen 0.3 (http://img11.hostingpics.net/pics/177803Ad03.png)

Adaptive Sharpen 0.1 (http://img11.hostingpics.net/pics/607562Ad01.png)

Finesharp 1.0 (http://img11.hostingpics.net/pics/774019FineS1.png)

So SuperRes mostly damage the original content, adaptive sharpen 0.3 is way too much, so I'll go for Finesharp 1.0/1.5 or Adaptive Sharpen 0.1/0.2.

Finesharp seems to be better for movie content (bluray, high quality source) and adaptive sharpen for anime content (relatively new animes), at least that's my preferences.

edit : I decided to go with adaptive sharpen 0.2