View Single Post
Old 28th June 2014, 16:48   #120  |  Link
Shiandow
Registered User
 
Join Date: Dec 2013
Posts: 753
Okay, so I'll now try to explain how the super resolution method works and how I've implemented it. If you only want to know how to configure it just skip to the end.

The general idea behind the super resolution method that Alexey Lukin et al. explained in their paper is to treat upscaling as inverse downscaling. So the aim is to find a high resolution image which, after downscaling, is equal to the low resolution image.

The problem is that this is usually not well defined. For instance let's take the simplest possible example i.e. an image consisting of just 1 pixel, with value X. And say we want to find an image consisting of 2 pixels with values which we'll denote A and B. Now we need to decide on a downsampling algorithm from the image of 2 pixels to the image of 1 pixels. The obvious choice is to just average the two pixels, i.e. X = (A+B)/2. So if we had the values of A and B then we could find X but we're working backwards so we know the value of X and we want to find the values of A and B. However for any value of A there is a value of B such that (A+B)/2 = X, so there's no way to decide upon a solution. This can be solved by requiring the values of A and B to be close to each other, in which case the unique(!) solution is to make A and B equal X.

The SuperRes algorithm works similar to this except with more pixels, it requires the image to be "regular" (pixel close to each other should have "similar" values) and instead of requiring the result to be exactly equal to the original image after downscaling we simply require it to be "faithful". This method has an enormous amount of flexibility since we can choose which downscaling method to use, how to measure "regularity", and how to measure "faithfulness".

My implementation uses a very simple downscaling method which just averages the pixel values over a disk shaped region (with radius sqrt(2)). For measuring "faithfulness" I just square the difference between the downscaled result and the original. The method of measuring "regularity" is somewhat more complicated, since you only want the pixel values to be close when there is no edge, but not when there is. But it basically consists of looking at all pixels that are close and try to minimize some "distance" between the pixel values, I'll explain how I chose this distance later since it's related to how the algorithm works.

To find an image which is both "regular" and is "faithful" to the original image it is simplest to use the gradient descent method, which basically means that we look if it's better to lower or raise a pixel value and change it accordingly. This is similar to viewing the "regularity" and "faithfulness" as some kind of energy and calculate the resulting forces that act on the pixel values, where "regularity" pulls them closer together and "faithfulness" pulls them closer to the original values.

This brings us to how I've chosen to measure regularity, since instead of defining "regularity" directly and try to calculate the forces I just directly define the forces. The force I've chosen looks like this in this plot "x" is the difference between two adjacent pixel values. From the plot you can see that the force increases rapidly when values that were close move away from each other, but remains fairly constant when the difference was large (since that likely means that there's an edge). The corresponding distance is a bit more complicated but looks like this. This distance behaves like the square of the difference close to 0 but behaves more like an absolute value for large differences.

Now that we've defined all of required parts the algorithm consists of the following steps:
  1. Calculate an initial guess
  2. Downscale and calculate differences with original image.
  3. Calculate forces, resulting from "regularity" and "faithfulness".
  4. Apply forces.
  5. Repeat steps 2-4 several times.

The mistake I discovered earlier was that I was trying to combine steps 2 and 3 together, but you can't calculate the forces if you haven't calculated the differences yet. In the end I had to split step 2 in (yet another) shader. This made the algorithm somewhat slower but I think this could be avoided by first doing step 2 and 3 for 1/4 of the pixels and then do step 3 for the other 3/4 of the pixels, this should at least prevent any unnecessary texture calls. Splitting step 2 and 3 also meant that you now need even more shaders to get it all working, which doesn't seem to improve the stability of MadVR (it crashes sometimes when it's using a lot of shaders, usually during start up).

Now there are still some small parts of the algorithm left that I haven't mentioned:

The first is the way I normalise the forces acting on a pixel. I did this by reinterpreting part of the forces as the "weight" and reinterpreting "2x" as the actual force and then use the weighted average instead of just adding them. The idea behind this is that the weight is going to be small when it is close to an edge in which case you want the values to pull together faster to prevent ringing. To make this even effect even more pronounced I actually divide by the square of the total weight. I'm still not 100% sure that this part of the algorithm is actually that beneficial, but changing it would meant that I have to recalibrate it again so I'll just leave it in for now.

The second is the way I store the original values and the differences of the downscaled result with the original. I store these in the alpha channel of the pixels, where for every 2x2 block, the difference is stored in the top-left and the original value is stored in the bottom-right. This results in the lowest possible number of texture calls.

That concludes the description of the algorithm. Which leaves us with the way to use the shaders. Firstly here is a list of parameters and what they do:
  • strength: total strength of the force. If it's larger the algorithm will converge faster but it might go too far, resulting in artefacts.
  • softness: relative strength of the regularizing force. Larger values make the resulting image smoother.
  • radius: effective radius of regularizing force (it uses a Gaussian with that radius as weights). A larger radius should make the image smoother, but only slightly.
  • acuity: controls the threshold for what is considered an edge. If the difference between two pixel values is larger than 1/acuity then it will assume that there's an edge between them.
  • baseline: the baseline of the regularizing force, ensures that even pixels across edges don't get too far from each other. Makes edges softer (putting it to 0 makes edges amazingly crisp but also very rough).
If you want to you can also control the downscaling weights (called "weights"), just make sure that you pick the same weights for each of the 3 shaders (SuperRes, SuperRes-pre, SuperRes-inf).

Secondly, you need to put the shaders in the right order. The chain of shaders needed is getting a bit complicated, but in general it looks like this:

NEDI-pre -> Upscale-I -> Upscale-II -> SuperRes-pre -> SuperRes -> { SuperRes-inf -> SuperRes } -> NEDI-pst

The part between brackets can be repeated as many times as you want. I'd recommend using 3 iterations of the SupeRes shader, but perhaps you can use less if you raise the "strength" parameter. You can replace "Upscale" by "fNEDI", "NEDI" or "Lanczos" (I'd recommend using fNEDI) or just skip those shaders and use NNEDI3 for image upscaling. It you're not using NNEDI3 then you should set image upscaling to nearest. You should also make sure that you're resizing 2x (in both directions).

Finally you can download the needed shaders here. If you want to use them for MPC-BE then just put them in the shader folder, for MPC-HC you need to change the extension to ".hlsl" or add them manually using the shader editor, depending on which version of MPC-HC you have (they recently removed the shader editor, no idea why). For PotPlayer you should change the extension to ".txt" and put them in the shader folder. I'll also change the NEDI shaders in the first post, but I think it no longer makes sense to add all of the code to the first post.

Last edited by Shiandow; 28th June 2014 at 16:50.
Shiandow is offline   Reply With Quote