Lossless restoration of upsized images/videos?

[10L23r]

Is there a general method to restoring upsized images?

For example, if an image is upsized 2x with bilinear interpolation, this

mt_convolution("0, -0.25, 0.75, 0.75, -0.25", "0, -0.25, 0.75, 0.75, -0.25",true,1.0,u=3,v=3).pointresize(width/2,height/2)

can restore the image.

Is there a generalized process that can be used on images upsized by other factors (not as simple as 2x)?

[Gavino]

That's not strictly lossless, because of possible rounding errors.

For odd multiples (3x, 5x, ...), the following function works (for all color spaces):
function RestoreDown(clip c, int scale) {
w = c.width/scale
h = c.height/scale
pixel = (scale-1)/2
c.PointResize(w, h, pixel, IsRGB(c) ? -pixel : pixel)
IsYV12(c) ? MergeChroma(c.PointResize(w, h, 2*pixel, 2*pixel)) : \
IsYUY2(c) ? MergeChroma(c.PointResize(w, h, 2*pixel, pixel)) : last
}

[10L23r]

Yes, rounding errors are always annoying. I don't completely understand how the chroma planes are processed in masktools, but I do find a very slight shift in color.

What about fractional multiples, such as 1.5? It's very tedious to calculate the exact numbers to plug into mt_convolution(). I am particularly interested in bilinear right now.

[Gavino]

I believe the odd upsizing multiples are the only cases where you can exactly recover the original pixels, because only in those cases are the original pixels preserved unchanged in the upsized version. This is because the Avisynth resizers preserve the image centre position.
BTW my function is not limited to bilinear, it is valid for any purely interpolating resizer.

Can you tell me how you calculate the convolution numbers? Does it involve a matrix inversion of the upsizing kernel?
I would have expected only two non-zero coefficients for 2x bilinear upsizing (you have four), since each output pixel is a linear combination of two input pixels (and vice versa).

[10L23r]

To calculate the convolution numbers, you basically need to extrapolate from the interpolated pixels.

Assume that there are three pixels in an image: A (left), B (middle), C (right) each separated by a distance of 1 unit.

Now upsize the image 2x with bilinear so that there are 6 pixels: a1,a2,b1,b2,c1,c2.

a1 is to the left of A by 1/4 units and a2 is to the right by 1/4 units. Same goes for B and C.

Now focus on b2 and c1.
From the definition of bilinear interpolation:

0.75 B + 0.25 C = b2

0.25 B + 0.75 C = c1

Solving this system for B:

B = 1.5 b2 - 0.5 c1


now repeat the process with a2 and b1 and

B = 1.5 b1 - 0.5 a2

average these two to reduce rounding errors (hence four non-zeros):

B = -0.25 a2 + 0.75 b1 + 0.75 b2 - 0.25 c1

I haven't given much thought to matrix inversion, mostly because I forgot mostly all of algebra II ;D
edit: I just checked wikipedia, and it doesn't seem like matrix inversion, especially because you can to inversion in one-dimension.

two non-zeros would also work, but with less accuracy due to rounding.

mt_convolution("0, 0, 0, 1.5, -0.5", "0, 0, 0, 1.5, -0.5",true,1.0,u=3,v=3).pointresize(width/2,height/2)

or

mt_convolution("0, -0.5, 1.5, 0, 0", "0, -0.5, 1.5, 0, 0",true,1.0,u=3,v=3).pointresize(width/2,height/2)

note that all the zeros are not necessary; they are just for comparison.

[Gavino]

OK, thanks for the explanation.
I figured it involved solving simultaneous equations for the original pixels, which is equivalent to matrix inversion.

Extending your method to other resize multiples seems feasible, but (as you say) tedious.
For non-integral multiples, it gets even trickier, because there isn't such a simple repeating pattern.

[10L23r]

If you examine the script in my first post, it kinda looks like the kernel for bicubic resizing, or lanczos with 2 taps. Technically, I could expand the convolution to include d1 and d2 and others. That will look like lanczos with 3 taps. I have this feeling that lanczos with a lot of taps (or the sincresize in avisynth 2.6) would be able to do the same thing as these manual calculations.


That could make thikngs much easier for fractional multiples such as 1.333x or 1.125x

edit: Lanczos w/ 16 taps doesn't restore the image anywhere as well as my convolution :(
maybe sincresize will do the job

[g-force]

I find this all a bit interesting. I have a video that I suspect was resized poorly. If I could re-create the downsized version, I could then upsize with a better method and undo some of the artifacts that resulted from the original upsize.

-G

[Gavino]

It's unlikely you'll be able to do this unless the original upsizing was by a very simple ratio (like 3x).

[g-force]

Maybe not, but a guy can dream.

-G

[10L23r]

The best you can do, other than using convolutions, is using lanczos with a lot of taps.