Perceptually Based Downscaling of Images (needs implementation)

[raffriff42]

Before and after ResampleHQ. It's better than internal Resize methods, I think:
for one thing, ResampleHQ seems to preserve specular highlights better.



Here are the images at actual size (ie, not 2x size after downscaling, like above)
2x seems to exaggerate my perception of any over-sharpness, putting Lanczos etc at a disadvantage.

ResampleHQ(wid/2, hgt/2, dither=0>0, kernel="Bicubic", karg1=-1., karg2=0.)
\    .x2.sub("HQ Bicubic(-1, 0)")
# etc...

(the rest of the script is here)

[MysteryX]

Here are some more comparison with Clown.png

1. NNEDI3(nns=4)
2. SuperRes(Passes=2, Strength=.43), Cubic(1/3,1/3)
3. Cubic(-.6,0), Passes=2, Strength=.43
4. Cubic(-.6,0), Passes=2, Strength=1
5. Cubic(-.6,0), Passes=3, Strength=1



There is still definitely a slight darkening with the current SuperRes implementation and it doesn't happen with Cubic(-.6,0) which definitely looks better and more natural. Passes=3 looks almost identical to Passes=2 but is just very slightly clearer.

Btw, running 3 passes is 21% slower than 2 passes with my implementation.

[MysteryX]

I tried encoding a noisy video with SuperRes and Cubic(-.6,0).

The current implementation better preserves the textures and details. Some details are lost with this tweak, even with Strength=1. Cubic(-1,0) gives an even softer result.

My recommendation: keep it the way it is.

[MysteryX]

Originally Posted by raffriff42

Before and after ResampleHQ. It's better, I think.

You're missing Spline36 which I generally use for downscaling, and Cubic(-.6,0)

These might be the 2 most interesting ones.

[MysteryX]

Here are some downscaling tests with a noisy video that has been upscaled with frame quadrupling and is then being scaled down to 720p.

Cubic(1/3,1/3)
Cubic(-1,0)
Cubic(-.6.0)
Cubic(-.6,0) with DitherTools



I like Cubic(-.6,0), it is more accurate while Cubic(-1,0) is definitely sharper but has more distortion.

I'd love to test this Perceptual algorithm.

As a clarification, here I'm not actually testing the final resize. I'm testing it with NNEDI3's subpixel shifting and resize, which then serves as the input for SuperRes.
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=940, fheight=720, Threads=2).DitherPost()""")

In this case, using DitherTools for downscaling does improve the quality. Currently the output of Dither Resize is converted back to YV12 before serving as input for SuperRes. If I could convert straight from Stack16 format to Shader16 format, it will avoid losing data.

[MysteryX]

In addition to Cubic(-.6,0), in madVR, it is considered that the best downscaling is with Catmull-Rom. In ResizeX.avsi, I see that Catmull-Rom is defined as Cubic(0, .5)

Here's the screenshot with Catmull-Rom and DitherTools


I prefer Cubic(-.6,0)

[Didée]

There's one thing which "the paper" doesn't tell you, and that is temporal stability. After all we're talking about "video", aren't we. And if some still shots are looking impressive, you don't know if the result on actual video is stable, or if it does introduce some nasty temporal flickering.

Anyway, for a rather typical downscaling scenario, namely resizing 1080p content to 720p, I've found bicubic with ~about~ (-0.5,0.25) fully sufficient. Using FineSharp to re-scale that 720p back to 1080p, the result in many cases is almost indistinguishable from the original source. With carefully chosen settings, I've fooled some friends into believing that the original 1080p vid would be the downscaled-upscaled version, and the (downscaled-upscaled)+FineSharp one would be the original 1080p.

Quite possible that there are better ways to reduce 5000x3000 sources down to 512x384. But then, hey, why would you want to do that.

[Shiandow]

Yeah temporal stability could be a problem, having lots of texture is somewhat meaningless if it becomes more or less random.

That said the method seems to be reasonably robust against small changes to the input image. I don't see anything particularly worrying, though, as far as I can tell a small change of the input should only give a small change to the output. And the similarity measure they're using seems to be reasonably consistent when moving or deforming the image.

[Wilbert]

Removed all non-relevant posts. Please stay on topic guys.

[bxyhxyh]

Originally Posted by sqrt(9801)

Or you can just use Shader with Param2 = "-1,0f"

IIRC, VirtualDub's documentation mentions that using B = -0.6 is mathematically more accurate, so that's something you could try out.

Can you point to the part that mentioned it? I can't find it from Virtualdub's help file.

[MysteryX]

I found B=0, C=.75 to give similar and slightly better result than B=-.6, C=0

This goes exactly as described in the documentation
http://avisynth.nl/index.php/Resize

[bxyhxyh]

Originally Posted by MysteryX

I found B=0, C=.75 to give similar and slightly better result than B=-.6, C=0

This goes exactly as described in the documentation
http://avisynth.nl/index.php/Resize

Then Lanczos would be better as it written in this page. Eventually we end up choosing Spline36Resize back lol.

Joke aside, I found what I asked.

A=-0.6 gives the most consistent results mathematically, but the other modes may produce more visually pleasing results.

It says a=-0.6 not b=-0.6.
Isn't a=-0.6 is just b=0, c=0.6?

For b=-1, c=0, it changed the shape of the certain objects. So it was no option for me.

Then As Didee recommended I tried b+2c=0 values for downscales.
b=-0.6, c=0.3 had too much ring for me.
So I ended up choosing b=-0.52, c=0.26. Looked nice to me.
It was sharper than Spline36Resize and had less ring than b=-0.6, c=0.3 of course.
(Why 0.52 over 0.5? Well, no real reason at all.)

[StainlessS]

Just reposting this [post #25] to fix changed PostImage.org links

Originally Posted by MysteryX

Here are some downscaling tests with a noisy video that has been upscaled with frame quadrupling and is then being scaled down to 720p.

Cubic(1/3,1/3)
Cubic(-1,0)
Cubic(-.6.0)
Cubic(-.6,0) with DitherTools



I like Cubic(-.6,0), it is more accurate while Cubic(-1,0) is definitely sharper but has more distortion.

I'd love to test this Perceptual algorithm.

As a clarification, here I'm not actually testing the final resize. I'm testing it with NNEDI3's subpixel shifting and resize, which then serves as the input for SuperRes.
SuperRes(2, 0.43, 0, """edi_rpow2(2, nns=4, cshift="BicubicResize", a1=-.6, a2=0, lsb=true, fwidth=940, fheight=720, Threads=2).DitherPost()""")

In this case, using DitherTools for downscaling does improve the quality. Currently the output of Dither Resize is converted back to YV12 before serving as input for SuperRes. If I could convert straight from Stack16 format to Shader16 format, it will avoid losing data.

PostImage.org links were changed to PostImage.cc for some reason,
otherwise links fail.

And from Post #26,


EDIT: and from post #22.

Originally Posted by MysteryX

Here are some more comparison with Clown.png

1. NNEDI3(nns=4)
2. SuperRes(Passes=2, Strength=.43), Cubic(1/3,1/3)
3. Cubic(-.6,0), Passes=2, Strength=.43
4. Cubic(-.6,0), Passes=2, Strength=1
5. Cubic(-.6,0), Passes=3, Strength=1



There is still definitely a slight darkening with the current SuperRes implementation and it doesn't happen with Cubic(-.6,0) which definitely looks better and more natural. Passes=3 looks almost identical to Passes=2 but is just very slightly clearer.

Btw, running 3 passes is 21% slower than 2 passes with my implementation.

EDIT: And post #20.

Originally Posted by MysteryX

Here are some tests with SuperRes using these different downscalers
SuperRes(Passes=2, Strength=.43) with NNEDI3(nns=4) as a prescaler

Current (1/3, 1/3)


-1,0


-.6,0


These clearly make the image softer. Let's try with increased Strength.

-.6,0, Strength=1


My observation: -1,0 is too sort. -6.0 looks more natural than 1/3,1/3 but requires increased strength. It removes some weird darker areas of trees that happen with 1/3,1/3.

Overall, -6,0 with Strength=1 still looks a bit softer but it also looks more natural. I like that one.

What about an ultra-sharp version of it with Passes=3 and Strength=1? It looks pretty decent!


It's worth nothing that the colors are slightly different, especially with the greens. Which one is the most accurate?

The greens are already darker than the NNEDI3 version, so -6,0 is more accurate on the colors.

[kedautinh12]

Thank Stain

[Selur]

btw. I like DPID (https://github.com/WolframRhodium/VapourSynth-dpid/) when downscaling by a factor of 4 or more.

[i990049]

Originally Posted by StainlessS

Just reposting this [post #25] to fix changed PostImage.org links

PostImage.org links were changed to PostImage.cc for some reason,
otherwise links fail.

And from Post #26,


EDIT: and from post #22.


EDIT: And post #20.

What about bicubic with (b=-0.5,c=0.25) in post #25#27 and Spline36 in post #24#32，according to this
http://avisynth.nl/index.php/Resize
BicubicResize may be the most visually pleasing of the Resize filters for downsizing to half-size or less.doom9
Try the default setting, (b=0, c=0.75) as above, or (b= -0.5, c=0.25).

[StainlessS]

What about ...

Yeah, I left them for you to re-post [It aint my job to fix everybody elses links]

[DTL]

Originally Posted by Didée

There's one thing which "the paper" doesn't tell you, and that is temporal stability. After all we're talking about "video", aren't we. And if some still shots are looking impressive, you don't know if the result on actual video is stable, or if it does introduce some nasty temporal flickering.

Finally at the 2023 we have built-in 'complementary' pair of scaler kernels:

SinPow and UserDefined2 for downscale and Sinc (SincLin2) or Lanczos and other weighted sinc for upscale.

So for sinc-based displays and hardware players with analog output like DVD/HD players (using sinc pure or Lanczos and many other weighted sincs as upsample) it should produce lowest possible 'temporal nonstability' and controlled ringing while providing adjustments of 'look/makeup' from 'film-soft' to 'video-sharp'.

Also for jinc-based displays the jinc (1d) kernel may be tested as optional for UserDefined2 resize.