RFC: Ultra-fast 2x upscaling filter?

[Dark Shikari]

I've been working a lot on optimizing x264's hpel filter lately with Holger's help and it recently struck me that what we actually have is an incredibly fast 6-tap upscaling filter with coefficients 1,-5,20,-5,1 (approximately Lanczos).

To give an idea of the order of magnitude of the performance here, a measurement on a 64-bit Nehalem system gives speed of about 5 clocks per input pixel, meaning that the filter could handle upscaling a 1920x1080 image to 3840x2160 at 300fps on one core of a 3Ghz system. This is probably overstating things, as memory bandwidth might become a limitation at that point, but the numbers are still quite insane. It also has some limitations; for example, it assumes the image is mod16, but this can probably be trivially worked around.

So, is there interest in something like this--a 2x upscaling filter fast enough that its speed cost is basically negligable regardless of resolution?

And, if so, is there anyone who is experienced enough in Avisynth + nasm to help me do it? ;)

[LoRd_MuldeR]

Wouldn't that be perfect for real-time processing? So it could be a nice addition to ffdshow-tryouts for software scaling :)

[Adub]

Hell yeah there is interest. We are the encoding community, as you well know Dark. We are addicted to speed.

However, I would like to see the quality first. You say "approximately lanczos". Does that mean halo's and everything?

[Who Am I ?]

^

+1 with merlin , also if we can get that much off a speedboost , life's gonna become a LOT easier when encoding HD and upscaled material

g'luck ds , been around reading your posts for long enough on animesuki :)

[tetsuo55]

This would be awesome

Especially if the filter would be able to do ANY(mod16) to TARGET(like 1920x1080)
Maybe this means i can postpone buying a new system!

[Dark Shikari]

This would be awesome

Especially if the filter would be able to do ANY(mod16) to TARGET(like 1920x1080)
Maybe this means i can postpone buying a new system!No, it would only be able to 2x upscale, nothing else. With flexibility you sacrifice an enormous amount of speed.

But of course you can upscale beyond 1080p... and then downscale with the graphics card overlay, which is free.

[tetsuo55]

No, it would only be able to 2x upscale, nothing else. With flexibility you sacrifice an enormous amount of speed.

But of course you can upscale beyond 1080p... and then downscale with the graphics card overlay, which is free.

Okay too bad, still 2x is better than nothing.
SD content would still go from 720x480 to 1440x960 on my 1920x1080 panel

[Fizick]

what minimal assembler streaming instructions this filter uses (if any :) ?

[Dark Shikari]

what minimal assembler streaming instructions this filter uses (if any :) ?All the way through SSSE3.

The filter has somewhere around 600 lines of asm, for both 32-bit and 64-bit, with versions ranging from MMXEXT to SSSE3.

It would require slight tweaking to adapt as currently it outputs the resulting data into 4 separate planes instead of one image, but that's easy to resolve.

[Fizick]

I am not sure that you can do all with nasm only, probably you need in MSVC too.
Use external "c" for assembler function.
see SimpleSample and http://avisynth.org/mediawiki/Filter_SDK/Change_frame_size

[Dark Shikari]

I am not sure that you can do all with nasm only, probably you need in MSVC too.
Use external "c" for assembler function.Well of course, doing it all in nasm would be absurd.

The method on 32-bit would probably go something like:

For one row:
1. Hpel_V
2. Hpel_H
3. Hpel_C
4. Interleave

There would be many gotchas though, like the fact that the filter assumes pixels outside the frame are both available and equal to the edge pixels, so the image would probably have to be increased in size and then padded, or the edge pixels special-cased in a C version.

[*.mp4 guy]

coefficients 1,-5,20,-5,1

For 2x interpolation, Including null positions, a non-windowed lowpass fiter with 3 non-zero coeficients would look like this:

2.54 0 -4.24 0 12.73 20 12.73 0 -4.24 0 2.54

Does your filter actually look like this in mt_convolution style notation?
1 0 -5 0 20 ??? 20 0 -5 0 1

[Dark Shikari]

For 2x interpolation, Including null positions, a non-windowed lowpass fiter with 3 non-zero coeficients would look like this:

2.54 0 -4.24 0 12.73 20 12.73 0 -4.24 0 2.54

Does your filter actually look like this in mt_convolution style notation?
1 0 -5 0 20 ??? 20 0 -5 0 1

an H-position interpolated pixel at position [x+0.5] is equal to:

[x-2] - 5 * [x-1] + 20 * [x] + 20 * [x+1] - 5 * [x+2] + [x+3]

By the way, note that the SSSE3 version of the filter, with some modifications to the H section, might be able to accept arbitrary coefficients. They would have to have a total sum of no more than 128 (to fit in 16-bit signed integer output). Might actually be limited to 64 because of how the C position is calculated, not sure.

The modifications to the H section may come soon as they might improve performance even in the case of the existing x264 filter, as pmaddubsw is basically the most godly instruction ever.

By the way, doesn't this mean that a 1/-5/20/20/-5/1 filter would be less ringy than a Lanczos?

[*.mp4 guy]

huh, I guess my original guess was correct (I thought pixel shifts would be bad for motion estimation). If the filter works as you say, however, it will be "bad".
[edit] the nulls version is correct, being an upscaling filter, not a lowpass filter, the nulls are inherent in its operation, disreguard the other version.

Proof:
lanczos (http://www.imagebam.com/image/6c2f4c29260186)

[edit] Ignore the following, they are incorrect:
Your filter (http://www.imagebam.com/image/c7365b29260188)
Your filter with nulls (http://www.imagebam.com/image/70ce6129260190)

[akupenguin]

I don't know what's wrong with your mt_convolution, but here's what I get with a simple C implementation:
clown h264-hpel (http://img12.imageshack.us/img12/5724/clown4xh264hpel8bit.png)
more aliasing/less ringing than lanczos, but purely better than either version of "your filter".

[*.mp4 guy]

Yes, I figured out the problem a little while ago, mt_convolution runs the convolution with every offset, whereas your algorithm actually only uses the 0.5 offset. This cases mt_convolution to run what is essentially a sharpening kernel on half of the pixels, that should actually be passed through unchanged.

[edit]Here is avisynth code to do X264Hpel interpolation correctly.
function X264Hpel(clip c)
{
source = c.assumeframebased

FIR = string("1 -5 20 20 -5 1")
Filtered = source.MT_Convolution(Horizontal=" 1 ", Vertical=" "+string(FIR)+" ", u=3, v=3)

interleave(source, filtered).assumefieldbased.weave()
}

Here is example usage for 4x interpolation of the clown image.

X264Hpel().turnleft.X264Hpel().turnright
X264Hpel().turnleft.X264Hpel().turnright