Perceptually Based Downscaling of Images (needs implementation)

[Sp00kyFox]

hi, there. I wanted to draw some attention to a fairly new developed downscaler by ETH Zurich which get pretty convincing results:
https://graphics.ethz.ch/~cengizo/imageDownscaling.htm

it seems that they didn't publish the code but the algorithm is explained in the paper and presented in pseudo code, it only consists of convolutions and sums.

I developed some shaders for visually enhancing retro games in the past but I don't have any experience with avisynth development. but I'd love to see an avisynth implementation for that, I think it could be a gread addition.

[MysteryX]

If you've done shaders development, one option is to do a shader version and run it through AviSynthShader. If it works well, then someone may decide to implement a native AviSynth version.

Plus perhaps a better HLSL downscaler could improve the result of Shiandow's SuperRes.

[SSH4]

Result is not better than BilinearResize(w/x, h/x) or XSharpen(128,128).Billinearscale(w/x,h/x) or any kind of ImagePreprocess().Billinearscale(w/x,h/x).ImagePostProcess()
And closed source + Patent Pending.

[MysteryX]

Result is not better than BilinearResize(w/x, h/x) or XSharpen(128,128).Billinearscale(w/x,h/x) or any kind of ImagePreprocess().Billinearscale(w/x,h/x).ImagePostProcess()
And closed source + Patent Pending.
The idea is to get a result as close to the source as possible.

Adding any kind of preprocessing or postprocessing can only distort the original image, losing or amplifying details.

There is still nothing like getting just the right amount of details during the downscaling itself.

Whether it's actually 'better' or not isn't an issue. It may be better in some cases and not in other cases. It's always better to have more options of filters to work with so that each can experiment on their own.

If the algorithms are in the paper, that's enough to implement it.

If it's patented, then that's another issue; it's only patent pending.

[SSH4]

Ah, yes. You right. I thought samples on web site is results of this downscaling algorithm but this is just competitors. Inside paper them show good results.

[feisty2]

Bicubicresize(b=-1,c=0) works fairly nice at keeping details when downscaling I think

[Shiandow]

That's true, but 'fairly nice' doesn't exactly compare to defining what 'keeping details' means and making a downscaler that gives the optimal result according to that definition.

[feisty2]

I did my compares of image downscaling like:
1. Scale to the half size with various algorithms
2. Scale back to the original size with nnedi3, and see which one looks closest to the original one
And bicubicresize(b=-1,c=0) won in most cases

[luquinhas0021]

This upscaling algorithm discussed here isn`t good for keeping fine and small edges, although doesn`t add so much artifacts. Just see the paper results and you will see by your own eyes. Downscaling, IMHO, almost never is a good thing, because a large factor, i.e, 4x or more, kill the weak and little edges; and there`s no upscaling method able to recover this details.

[feisty2]

http://i.imgur.com/r5D9F8w.png

bicubicresize(b=-1,c=0) downscaling result list of the samples here, https://graphics.ethz.ch/~cengizo/Files/Sig15HighResNaturalImages.html
and bicubicresize(b=-1,c=0) looks better than "Perceptual" imho...

[feisty2]

certainly not.

[luquinhas0021]

Anyway, I will test. Is notable that Bicubic for upscaling has two extremes: aliasing or blur... A kinda bad :( . I preffer last update of Adobe Detail Preserving. It`s sharper than Bicubic 0,1, keeps fine and small details and has no artifacts, unless those in original images, which it doesn`t smooth.

[Shiandow]

[...]

bicubicresize(b=-1,c=0) downscaling result list of the samples here, https://graphics.ethz.ch/~cengizo/Files/Sig15HighResNaturalImages.html
and bicubicresize(b=-1,c=0) looks better than "Perceptual" imho...

Interesting, could you upload (some of) those files separately, that would make it easier to compare.

By the way, for some incomprehensible reason they have scaled the image on that page from 256x256 to 266x266, which has made them a lot blurrier, you should open them on a separate page if you want to compare quality. In my opinion their perceptual algorithm is better at preserving fine lines and texture, although it might overdo it on some of the images.

[feisty2]

By the way, for some incomprehensible reason they have scaled the image on that page from 256x256 to 266x266, which has made them a lot blurrier, you should open them on a separate page if you want to compare quality. In my opinion their perceptual algorithm is better at preserving fine lines and texture, although it might overdo it on some of the images.

well, I re-checked the downscaled images at 256x256 and it makes sense now, perceptual looks better than bicubicresize(b=-1,c=0):devil:

[raffriff42]

Hmm, gotta test this for myself...
https://www.dropbox.com/s/5ijdq2urf5djia5/downscale2.jpg?raw=1

Note the small images are downsized, then enlarged x2 to exaggerate the artifacts, and to try to look like the sample (https://graphics.ethz.ch/~cengizo/imageDownscaling.htm).

Even so, the sample images don't look right; it looks like they have been softened or something. (EDIT missed the posts above discussing this)

EDIT This is not a good way to test IMHO -- unless you plan to view the final product at x2 scale.

EDIT "Perceptual" is the only one besides "Point" to preserve the eye sparkle, but that can be fixed with gamma-aware resizing (http://forum.doom9.org/showthread.php?t=160038), I think.

Feel free to use this script with any downscale method you want to test.

ImageSource("faceComparisons.png")
O=Crop(0, 0, 636, 0) ## face 1, all sizes
Crop(0, 0, 288, 380) ## face 1, cropped

scale=148.0/364.0 ## per sample image
scale=Min(Max(0.1, scale), 0.5)
## downsize to 1/2 this scale, then upsize x2

wid = mod(2, Round(scale*Width))
hgt = mod(2, Round(scale*Height))

StackHorizontal(
\ BlankClip(Last, color=$ffffff, width=O.Width-(4*wid))
\ , StackVertical(
\ PointResize(wid/2, hgt/2).x2.sub("Point")
\ , Lanczos4Resize(wid/2, hgt/2).x2.sub("Lanczos")
\ , BlankClip(Last, color=$ffffff, width=wid, height=(Height-2*hgt))
\ )
\ , StackVertical(
\ BilinearResize(wid/2, hgt/2).x2.sub("Bilinear")
\ , BicubicResize(wid/2, hgt/2).x2.sub("Bicubic")
\ , BlankClip(Last, color=$ffffff, width=wid, height=(Height-2*hgt))
\ )
\ , StackVertical(
\ BicubicResize(wid/2, hgt/2, b=-1, c=0).x2.sub("Bicubic(-1, 0)")
\ , BicubicResize(wid/2, hgt/2, b=0, c=0.75).x2.sub("Bicubic(0, 0.75)")
\ , BlankClip(Last, color=$ffffff, width=wid, height=(Height-2*hgt))
\ )
\ , StackVertical(
\ Spline16Resize(wid/2, hgt/2).x2.sub("Spline16")
\ , Spline64Resize(wid/2, hgt/2).x2.sub("Spline64")
\ , BlankClip(Last, color=$ffffff, width=wid, height=(Height-2*hgt))
\ )
\ )
return StackVertical(O, Last)

function mod(int m, int i)
{
return i - i % m
}

function x2(clip C)
{
C
return PointResize(2*Width, 2*Height)
}

function sub(clip C, string s)
{
return C.Subtitle(s, size=C.Height/10, align=2)
}

[Shiandow]

well, I re-checked the downscaled images at 256x256 and it makes sense now, perceptual looks better than bicubicresize(b=-1,c=0):devil:

Yeah, still no clue why they felt the need to make their images 10 pixels bigger. They seem to make similar mistakes on other parts of the web site which makes things really confusing. It's almost as if they don't want people to use their algorithm.

By the way, bicubic(-1,0) looks pretty neat, I think I might use that for downscaling from now on. :thanks:

[MysteryX]

By the way, bicubic(-1,0) looks pretty neat, I think I might use that for downscaling from now on. :thanks:
Does that mean your SuperRes algorithm might work better with that?

How does that Bicubic.hlsl need to be updated, changing BC to -1 and 0 or in this implementation it's something else?

https://github.com/mysteryx93/AviSynthShader/blob/master/Shaders/SuperRes/src/SuperRes/Bicubic.hlsl

[Shiandow]

Does that mean your SuperRes algorithm might work better with that?

How does that Bicubic.hlsl need to be updated, changing BC to -1 and 0 or in this implementation it's something else?

https://github.com/mysteryx93/AviSynthShader/blob/master/Shaders/SuperRes/src/SuperRes/Bicubic.hlsl

It might, haven't checked it yet. If you want to try it should indeed work if you just change "B" and "C" to -1 and 0 respectively.

[sqrt(9801)]

Or you can just use Shader with Param2 = "-1,0f"

IIRC, VirtualDub's documentation mentions that using B = -0.6 is mathematically more accurate, so that's something you could try out.

[MysteryX]

Here are some tests with SuperRes using these different downscalers
SuperRes(Passes=2, Strength=.43) with NNEDI3(nns=4) as a prescaler

Current (1/3, 1/3)
http://s20.postimg.org/537l6y861/Lighthouse_Cubic_3.png (http://postimg.org/image/537l6y861/)

-1,0
http://s20.postimg.org/xp0nnc6op/Lighthouse_Cubic_1.png (http://postimg.org/image/xp0nnc6op/)

-.6,0
http://s20.postimg.org/6sgof0nvd/Lighthouse_Cubic_6.png (http://postimg.org/image/6sgof0nvd/)

These clearly make the image softer. Let's try with increased Strength.

-.6,0, Strength=1
http://s20.postimg.org/6tqm8fpp5/Lighthouse_Cubic_6_Str1.png (http://postimg.org/image/6tqm8fpp5/)

My observation: -1,0 is too sort. -6.0 looks more natural than 1/3,1/3 but requires increased strength. It removes some weird darker areas of trees that happen with 1/3,1/3.

Overall, -6,0 with Strength=1 still looks a bit softer but it also looks more natural. I like that one.

What about an ultra-sharp version of it with Passes=3 and Strength=1? It looks pretty decent!
http://s20.postimg.org/oz3kmhp7d/Lighthouse_Cubic_6_Passes3.jpg (http://postimg.org/image/oz3kmhp7d/)

It's worth nothing that the colors are slightly different, especially with the greens. Which one is the most accurate?

The greens are already darker than the NNEDI3 version, so -6,0 is more accurate on the colors.
http://s20.postimg.org/qzamti3x5/Lighthouse_NNEDI3.jpg (http://postimg.org/image/qzamti3x5/)

[raffriff42]

Before and after ResampleHQ (http://forum.doom9.org/showthread.php?t=160038). It's better than internal Resize methods, I think:
for one thing, ResampleHQ seems to preserve specular highlights better.

https://www.dropbox.com/s/fr418uwqxdegqp5/downscale3.png?raw=1

Here are the images at actual size (ie, not 2x size after downscaling, like above)
2x seems to exaggerate my perception of any over-sharpness, putting Lanczos etc at a disadvantage.
https://www.dropbox.com/s/cwazuu47femcug8/downscale3s.png?raw=1

ResampleHQ(wid/2, hgt/2, dither=0>0, kernel="Bicubic", karg1=-1., karg2=0.)
\ .x2.sub("HQ Bicubic(-1, 0)")
# etc...(the rest of the script is here (http://forum.doom9.org/showthread.php?p=1748640#post1748640))

[MysteryX]

Here are some more comparison with Clown.png

1. NNEDI3(nns=4)
2. SuperRes(Passes=2, Strength=.43), Cubic(1/3,1/3)
3. Cubic(-.6,0), Passes=2, Strength=.43
4. Cubic(-.6,0), Passes=2, Strength=1
5. Cubic(-.6,0), Passes=3, Strength=1

http://s20.postimg.org/5yts02f6h/Clown_NNEDI3.jpg (http://postimg.org/image/5yts02f6h/) http://s20.postimg.org/ze1xq1mft/Clown_Super_Res.jpg (http://postimg.org/image/ze1xq1mft/) http://s20.postimg.org/nnvte0tll/Clown_Cubic6_Str43.png (http://postimg.org/image/nnvte0tll/) http://s20.postimg.org/ubsf0md3t/Clown_Cubic6_Str100.png (http://postimg.org/image/ubsf0md3t/) http://s20.postimg.org/5jsstdvx5/Clown_Cubic6_Str100_P3.png (http://postimg.org/image/5jsstdvx5/)

There is still definitely a slight darkening with the current SuperRes implementation and it doesn't happen with Cubic(-.6,0) which definitely looks better and more natural. Passes=3 looks almost identical to Passes=2 but is just very slightly clearer.

Btw, running 3 passes is 21% slower than 2 passes with my implementation.

[MysteryX]

I tried encoding a noisy video with SuperRes and Cubic(-.6,0).

The current implementation better preserves the textures and details. Some details are lost with this tweak, even with Strength=1. Cubic(-1,0) gives an even softer result.

My recommendation: keep it the way it is.

[MysteryX]

Before and after ResampleHQ (http://forum.doom9.org/showthread.php?t=160038). It's better, I think.
You're missing Spline36 which I generally use for downscaling, and Cubic(-.6,0)

These might be the 2 most interesting ones.

[MysteryX]

Here are some downscaling tests with a noisy video that has been upscaled with frame quadrupling and is then being scaled down to 720p.

Cubic(1/3,1/3)
Cubic(-1,0)
Cubic(-.6.0)
Cubic(-.6,0) with DitherTools

http://s20.postimg.org/ve322jcux/Downscale_Cubic.png (http://postimg.org/image/ve322jcux/) http://s20.postimg.org/8bcj3dbdl/Downscale_Cubic1.png (http://postimg.org/image/8bcj3dbdl/) http://s20.postimg.org/vs4e24wyh/Downscale_Cubic6.png (http://postimg.org/image/vs4e24wyh/) http://s20.postimg.org/s9se5qw2h/Downscale_Cubic6_Dither.jpg (http://postimg.org/image/s9se5qw2h/)

I like Cubic(-.6,0), it is more accurate while Cubic(-1,0) is definitely sharper but has more distortion.

I'd love to test this Perceptual algorithm.

As a clarification, here I'm not actually testing the final resize. I'm testing it with NNEDI3's subpixel shifting and resize, which then serves as the input for SuperRes.
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=940, fheight=720, Threads=2).DitherPost()""")

In this case, using DitherTools for downscaling does improve the quality. Currently the output of Dither Resize is converted back to YV12 before serving as input for SuperRes. If I could convert straight from Stack16 format to Shader16 format, it will avoid losing data.

[MysteryX]

In addition to Cubic(-.6,0), in madVR, it is considered that the best downscaling is with Catmull-Rom. In ResizeX.avsi, I see that Catmull-Rom is defined as Cubic(0, .5)

Here's the screenshot with Catmull-Rom and DitherTools
http://s20.postimg.org/feozwq66x/Downscale_Catmull_Rom.jpg (http://postimg.org/image/feozwq66x/)

I prefer Cubic(-.6,0)

[Didée]

There's one thing which "the paper" doesn't tell you, and that is temporal stability. After all we're talking about "video", aren't we. And if some still shots are looking impressive, you don't know if the result on actual video is stable, or if it does introduce some nasty temporal flickering.

Anyway, for a rather typical downscaling scenario, namely resizing 1080p content to 720p, I've found bicubic with ~about~ (-0.5,0.25) fully sufficient. Using FineSharp to re-scale that 720p back to 1080p, the result in many cases is almost indistinguishable from the original source. With carefully chosen settings, I've fooled some friends into believing that the original 1080p vid would be the downscaled-upscaled version, and the (downscaled-upscaled)+FineSharp one would be the original 1080p.

Quite possible that there are better ways to reduce 5000x3000 sources down to 512x384. But then, hey, why would you want to do that.

[Shiandow]

Yeah temporal stability could be a problem, having lots of texture is somewhat meaningless if it becomes more or less random.

That said the method seems to be reasonably robust against small changes to the input image. I don't see anything particularly worrying, though, as far as I can tell a small change of the input should only give a small change to the output. And the similarity measure they're using seems to be reasonably consistent when moving or deforming the image.

[Wilbert]

Removed all non-relevant posts. Please stay on topic guys.

[bxyhxyh]

Or you can just use Shader with Param2 = "-1,0f"

IIRC, VirtualDub's documentation mentions that using B = -0.6 is mathematically more accurate, so that's something you could try out.
Can you point to the part that mentioned it? I can't find it from Virtualdub's help file.

[MysteryX]

I found B=0, C=.75 to give similar and slightly better result than B=-.6, C=0

This goes exactly as described in the documentation
http://avisynth.nl/index.php/Resize

[bxyhxyh]

I found B=0, C=.75 to give similar and slightly better result than B=-.6, C=0

This goes exactly as described in the documentation
http://avisynth.nl/index.php/Resize

Then Lanczos would be better as it written in this page. Eventually we end up choosing Spline36Resize back lol.

Joke aside, I found what I asked.
A=-0.6 gives the most consistent results mathematically, but the other modes may produce more visually pleasing results.

It says a=-0.6 not b=-0.6.
Isn't a=-0.6 is just b=0, c=0.6?

For b=-1, c=0, it changed the shape of the certain objects. So it was no option for me.

Then As Didee recommended I tried b+2c=0 values for downscales.
b=-0.6, c=0.3 had too much ring for me.
So I ended up choosing b=-0.52, c=0.26. Looked nice to me.
It was sharper than Spline36Resize and had less ring than b=-0.6, c=0.3 of course.
(Why 0.52 over 0.5? Well, no real reason at all.)

[StainlessS]

Just reposting this [post #25] to fix changed PostImage.org links
Here are some downscaling tests with a noisy video that has been upscaled with frame quadrupling and is then being scaled down to 720p.

Cubic(1/3,1/3)
Cubic(-1,0)
Cubic(-.6.0)
Cubic(-.6,0) with DitherTools

http://s20.postimg.cc/ve322jcux/Downscale_Cubic.png (http://postimg.cc/image/ve322jcux/) http://s20.postimg.cc/8bcj3dbdl/Downscale_Cubic1.png (http://postimg.cc/image/8bcj3dbdl/) http://s20.postimg.cc/vs4e24wyh/Downscale_Cubic6.png (http://postimg.cc/image/vs4e24wyh/) http://s20.postimg.cc/s9se5qw2h/Downscale_Cubic6_Dither.jpg (http://postimg.cc/image/s9se5qw2h/)

I like Cubic(-.6,0), it is more accurate while Cubic(-1,0) is definitely sharper but has more distortion.

I'd love to test this Perceptual algorithm.

As a clarification, here I'm not actually testing the final resize. I'm testing it with NNEDI3's subpixel shifting and resize, which then serves as the input for SuperRes.
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=940, fheight=720, Threads=2).DitherPost()""")

In this case, using DitherTools for downscaling does improve the quality. Currently the output of Dither Resize is converted back to YV12 before serving as input for SuperRes. If I could convert straight from Stack16 format to Shader16 format, it will avoid losing data.
PostImage.org links were changed to PostImage.cc for some reason,
otherwise links fail.

And from Post #26,
http://s20.postimg.cc/feozwq66x/Downscale_Catmull_Rom.jpg (http://postimg.cc/image/feozwq66x/)

EDIT: and from post #22.
Here are some more comparison with Clown.png

1. NNEDI3(nns=4)
2. SuperRes(Passes=2, Strength=.43), Cubic(1/3,1/3)
3. Cubic(-.6,0), Passes=2, Strength=.43
4. Cubic(-.6,0), Passes=2, Strength=1
5. Cubic(-.6,0), Passes=3, Strength=1

http://s20.postimg.cc/5yts02f6h/Clown_NNEDI3.jpg (http://postimg.cc/image/5yts02f6h/) http://s20.postimg.cc/ze1xq1mft/Clown_Super_Res.jpg (http://postimg.cc/image/ze1xq1mft/) http://s20.postimg.cc/nnvte0tll/Clown_Cubic6_Str43.png (http://postimg.cc/image/nnvte0tll/) http://s20.postimg.cc/ubsf0md3t/Clown_Cubic6_Str100.png (http://postimg.cc/image/ubsf0md3t/) http://s20.postimg.cc/5jsstdvx5/Clown_Cubic6_Str100_P3.png (http://postimg.cc/image/5jsstdvx5/)

There is still definitely a slight darkening with the current SuperRes implementation and it doesn't happen with Cubic(-.6,0) which definitely looks better and more natural. Passes=3 looks almost identical to Passes=2 but is just very slightly clearer.

Btw, running 3 passes is 21% slower than 2 passes with my implementation.

EDIT: And post #20.
Here are some tests with SuperRes using these different downscalers
SuperRes(Passes=2, Strength=.43) with NNEDI3(nns=4) as a prescaler

Current (1/3, 1/3)
http://s20.postimg.cc/537l6y861/Lighthouse_Cubic_3.png (http://postimg.cc/image/537l6y861/)

-1,0
http://s20.postimg.cc/xp0nnc6op/Lighthouse_Cubic_1.png (http://postimg.cc/image/xp0nnc6op/)

-.6,0
http://s20.postimg.cc/6sgof0nvd/Lighthouse_Cubic_6.png (http://postimg.cc/image/6sgof0nvd/)

These clearly make the image softer. Let's try with increased Strength.

-.6,0, Strength=1
http://s20.postimg.cc/6tqm8fpp5/Lighthouse_Cubic_6_Str1.png (http://postimg.cc/image/6tqm8fpp5/)

My observation: -1,0 is too sort. -6.0 looks more natural than 1/3,1/3 but requires increased strength. It removes some weird darker areas of trees that happen with 1/3,1/3.

Overall, -6,0 with Strength=1 still looks a bit softer but it also looks more natural. I like that one.

What about an ultra-sharp version of it with Passes=3 and Strength=1? It looks pretty decent!
http://s20.postimg.cc/oz3kmhp7d/Lighthouse_Cubic_6_Passes3.jpg (http://postimg.cc/image/oz3kmhp7d/)

It's worth nothing that the colors are slightly different, especially with the greens. Which one is the most accurate?

The greens are already darker than the NNEDI3 version, so -6,0 is more accurate on the colors.
http://s20.postimg.cc/qzamti3x5/Lighthouse_NNEDI3.jpg (http://postimg.cc/image/qzamti3x5/)

[kedautinh12]

Thank Stain

[Selur]

btw. I like DPID (https://github.com/WolframRhodium/VapourSynth-dpid/) when downscaling by a factor of 4 or more.

[i990049]

Just reposting this [post #25] to fix changed PostImage.org links

PostImage.org links were changed to PostImage.cc for some reason,
otherwise links fail.

And from Post #26,
http://s20.postimg.cc/feozwq66x/Downscale_Catmull_Rom.jpg (http://postimg.cc/image/feozwq66x/)

EDIT: and from post #22.


EDIT: And post #20.

What about bicubic with (b=-0.5,c=0.25) in post #25#27 and Spline36 in post #24#32，according to this
http://avisynth.nl/index.php/Resize
BicubicResize may be the most visually pleasing of the Resize filters for downsizing to half-size or less.doom9
Try the default setting, (b=0, c=0.75) as above, or (b= -0.5, c=0.25).

[StainlessS]

What about ...
Yeah, I left them for you to re-post [It aint my job to fix everybody elses links] :)

[DTL]

There's one thing which "the paper" doesn't tell you, and that is temporal stability. After all we're talking about "video", aren't we. And if some still shots are looking impressive, you don't know if the result on actual video is stable, or if it does introduce some nasty temporal flickering.


Finally at the 2023 we have built-in 'complementary' pair of scaler kernels:

SinPow and UserDefined2 for downscale and Sinc (SincLin2) or Lanczos and other weighted sinc for upscale.

So for sinc-based displays and hardware players with analog output like DVD/HD players (using sinc pure or Lanczos and many other weighted sincs as upsample) it should produce lowest possible 'temporal nonstability' and controlled ringing while providing adjustments of 'look/makeup' from 'film-soft' to 'video-sharp'.

Also for jinc-based displays the jinc (1d) kernel may be tested as optional for UserDefined2 resize.