ResampleHQ causes weird bands?

[Boulder]

I wanted to compare the output of resamplehq and standard bicubic resizing.

Downloaded resamplehq for VS here:
https://gist.github.com/4re/64642122e359c37543fe

I added parameters a1 and a2 to the function so that I could utilize the b and c values of Bicubic resizing.

First I tested comparing this set:

org = core.fmtc.bitdepth(clp, bits=16)
clp = core.resize.Bicubic(org, width=1280, height=688, filter_param_a=-0.7, filter_param_b=0.35)
clp2 = core.resize.Bicubic(org, width=1280, height=688, filter_param_a=-0.4, filter_param_b=0.2)

planes = [0]
expr = ('{median} x + y -').format(median=1<<(clp.format.bits_per_sample-1))
expr = [(i in planes) * expr for i in range(1)]
result = core.std.Expr([clp, clp2], expr)
result = core.std.Levels(result, min_in=125*256, max_in=132*256, min_out=0, max_out=255*256, planes=0)


It created a nice comparison:
https://preview.ibb.co/neYytG/43462_bicubics.png (https://ibb.co/mtFBDG)

Then I replaced the second clip by clp2 = rhq.resamplehq(org, width=1280, height=688, a1=-0.4, a2=0.2)

This created some strange banding:
https://preview.ibb.co/h1dEYG/43462_rhq2.png (https://ibb.co/cjHmfw)

Is this an issue of fmtconv, which resamplehq uses extensively, or something else? Is there some other alternative to try to do gamma aware resizing?

[Boulder]

The problem seems to disappear if I resize to 1280x720. I probably should report this in the fmtconv thread, although I don't know if cretindesalpes is active anymore.

[feisty2]

try with floating point precision

[Are_]

I already tested with it and bands disappeared, but is it normal for them to not be there just by changing the resolution while maintaining 16 bit integer precision? Does zimg use an internal float precision when dealing with 16bit inputs?

EDIT: And yes, this is a fmtc issue because you can see it when you use it raw too. Gamma aware resizing just makes it more obvious.

[Boulder]

So for now, the performance-wise best option would be to use the internal zimg-based resizer inside resamplehq. The difference is not really visible to the eye as I always compare the original and downsized&reupscaled output and I've not noticed anything weird.

[Are_]

In the past when dealing with 16bit inputs fmtc was noticeable faster, that and being nicer when wanting to deal with many options made me not use zimg. Did you benchmark to make sure it is faster -float vs 16bit?

[Boulder]

The speed difference between the resize methods inside resamplehq seem to be negligible when encoding to x265, which is the bottleneck anyway. The funny thing is that I also tried "core.resize.Bicubic(clp, width=1280, height=688, filter_param_a=-0.6, filter_param_b=0.3)" without anything else and it was just about as fast/slow as using resamplehq.

[Boulder]

orig_format = clip.format.id
tmp_format = vs.RGBH if is_cpu_ivy_bridge_or_later() else vs.RGBS

clip = core.resize.Bicubic(clip, format=tmp_format, transfer='linear')
clip = core.resize.Bicubic(clip, width=w, height=h)
clip = core.resize.Bicubic(clip, format=orig_format, transfer='709')


In fact, it should be faster than fmtconv, because it creates fewer temporary images. According to the link, the fmtconv equivalent uses 8 temporary images.

Trying that one gives an odd error:

in
clp = core.resize.Bicubic(clp, format=tmp_format, transfer='linear')
File "src\cython\vapoursynth.pyx", line 1702, in vapoursynth.Function.__call__ (src\cython\vapoursynth.c:34634)
File "src\cython\vapoursynth.pyx", line 527, in vapoursynth.typedDictToMap (src\cython\vapoursynth.c:12806)
ValueError: invalid literal for int() with base 10: 'linear'

[Are_]

Boulder, if you want to test stuff I updated the gist with a version that makes use of zimg resizer. Not really much tested but it should work.

[Boulder]

Thanks, will give that a go. I see that you have also included a descaler which could also prove to be useful.

[Are_]

Well, I went ahead and tested/fixed it myself. It is roughly 50% faster than fmtc version with higher quality because it uses single float instead 16-bit integer. Overall a nice finding.

[Boulder]

Sorry, forgot to report that I noticed no ill effects. Were there any critical changes in the last version compared to the first fixed one?

[Are_]

Nothing that I can remember, the range_in value was not used at some point, but that would be something you will notice if went wrong in your encode.