16/32-bit Processing, INT vs FLOAT

[MysteryX]

There are some plugins that offer internal 16-bit processing. I'm thinking of MVTools among others which processes data with 32-bit float.

Has anyone experimented with various options to see the performance difference?

32-bit FLOAT is overkill (by how much?)

The CPU isn't good at using 16-bit FLOAT

What about 16-bit or 32-bit INT? What is the performance difference between 16-bit INT, 32-bit INT and 32-bit FLOAT for the same operations?

Then there's the issue that INT loses data due to cropping while FLOAT preserves that data. With AviSynthShader, I did a simple test of converting YUV data to RGB and then back. With FLOAT, the image was the same. With INT, I lost some data.

Another option would be to design a special type of INT16 that allows for 10% overflow. We got 65536 values to represent a 0-255 color. We could reserve the 6553 lower values and 6553 higher values for overflow and that would leave 52430 values to represent a 0-255 color. Would that work that easily?

Would love to hear the opinion of experts on this. I really think that 8-bit isn't enough for internal processing and 16-bit is a LOT better, but up until now it could only be done by hacking left and right. Having a more standardized way of doing it could be useful.

[Mystery Keeper]

*points at VapourSynth*

[MysteryX]

I was thinking of VapourSynth too. How did they decide to implement it?

Edit: just a quick search shows that they have the option of 8-bit, 12bit, 14bit or 16bit INT, and 32bit FLOAT. IMO, FLOAT has the disadvantage of being overkill while INT has the disadvantage of cropping overflow. Working on the GPU allows 16-bit FLOAT which is the best of both worlds but the CPU can't do that.


As Pinterf said, there really isn't much that needs to be changed in the AviSynth core to support 16-bit data, because each filter is responsible for the way they interpret the raw data. Thus, providing native 16-bit support is nothing more than removing artificial barriers, perhaps adding a few helper functions to check types, and providing functions for all the basic operations: resize, convert and dither.

Then as I'm thinking about it, allocating space for upper overflow is easy. The difference between 16-bit float and int is that float is 0-1 while int is 0-65535; only the upper bound changes. Changing that upper bound for regular values to, say, 635535, doesn't change anything the calculations at all besides taking that into account during conversion (and during dithering).

Lower-bound overflow, however, wouldn't be so easy. If the lower regular value is 0, a simple conversion from 8-bit can be done with x = x / 256 * 63535. If space is allocated for lower-bound overflow, that becomes x = x / 256 * 635535 + 1000. It adds an extra addition or substraction to every operation, otherwise operations such as "increase by 10%" would be distorted. A simple extra addition may not be a big deal but it would add complexity, and would be repeated for every single operation on that number. Unless someone creates a new C++ data type that automatically deals with these overflows? Kind of like how BigInteger stores its data its own way.

Do most overflows happen above 255 or below 0? That would need to be experimented.

While supporting this within AVS+ would apparently be very easy, the harder work would be in providing a standard set of tools; it's kind of rewriting DitherTools without using the Stack16 hack.

Pinterf, if you go forward with those plans, instead of the core providing extra functions to manage 16-bit data, perhaps what would be useful is a code file that plugin developers can add into their project to handle basic features. This would require nearly no change to AviSynth itself.

[shekh]

When fitting 16 bit to vdub I decided to not decide on value range (65535 vs 64000 etc). Instead I have per frame and per component dynamic range. In theory it should allow high overflows and to pass lower-depth data around more efficiently. In practice it have not seen any use yet.

[Mystery Keeper]

That's not how VapourSynth works at all. It has INT and FLOAT sample types and arbitrary bitdepths for both of them. There are two properties: bits per sample and bytes per sample. It is up to plugins to check the format and handle them. There are plugins that support 16 bit FLOAT. Implementation is straightforward. Plugins get the plane pointer and read/write bytes_per_sample for every point.

[MysteryX]

When fitting 16 bit to vdub I decided to not decide on value range (65535 vs 64000 etc). Instead I have per frame and per component dynamic range. In theory it should allow high overflows and to pass lower-depth data around more efficiently. In practice it have not seen any use yet.
It seems like you did a good implementation. Nobody used it because there is no formal support of 16-bit in AviSynth and everybody doing it is just hacking around their own way.

Probably nobody knew about what you did to begin with.

Perhaps with better standards established, more plugin developers would do it that way; and that would start with Pinterf doing the ground-work with AviSynth+

That's not how VapourSynth works at all. It has INT and FLOAT sample types and arbitrary bitdepths for both of them. There are two properties: bits per sample and bytes per sample. It is up to plugins to check the format and handle them. There are plugins that support 16 bit FLOAT. Implementation is straightforward. Plugins get the plane pointer and read/write bytes_per_sample for every point.
That the same as with AviSynth. Then it's a matter how what standards are adopted as official data formats to pass between plugins. Stack16 ended up being one such "hacky" standards.

[MysteryX]

I just read this regarding BTB/WTW. I think this addresses the issue of overflows quite well already. At least in the YUV space. It can still become an issue when converting to RGB.


Video signals for NTSC are supposed to be limited to 16-235 to account for possible overshoot and other various reasons. Pretty much the same as how when you record to an audio cassette, you don't just set the gain to where your level is touching 100%, right?

It's not necessary these days because of better cameras and even better video processing but unfortunately the standard is still around. It's kind of how like interlacing stuck around for a long time and still, unfortunately, has not gone the f*** away. Some equipment will even clip off anything out of that range. If it does not clip then it passes BTB (0-15) and/or WTW (236-255)

[TheFluff]

That the same as with AviSynth. Then it's a matter how what standards are adopted as official data formats to pass between plugins. Stack16 ended up being one such "hacky" standards.

It's not. VS has defined sample format metadata and you can register new sample formats programmatically at runtime. A whole lot of functionality supports arbitrary bit depths, for example. It's possible to design plugins that can deal programmatically with you saying "here, I made up a standard for 7-bit YUV" correctly.

[MysteryX]

And the various plugins are written in a way that they can deal with pretty much any of these formats as input and output?

[Myrsloik]

And the various plugins are written in a way that they can deal with pretty much any of these formats as input and output?

The ones where it's easy are.

But there are only 3 formats you should use when filtering anyway. 8 bit, 16 bit and float. The rest just waste memory bandwidth and bits.

In reality you should see all other formats as specialized for input/output adaptation.

[Wilbert]

When fitting 16 bit to vdub I decided to not decide on value range (65535 vs 64000 etc). Instead I have per frame and per component dynamic range. In theory it should allow high overflows and to pass lower-depth data around more efficiently. In practice it have not seen any use yet.
How does this work? Surely most plugins and encoders expect a value range?

[shekh]

How does this work? Surely most plugins and encoders expect a value range?

Of course old plugins cannot work with it. There is only a couple of filters I made myself.
The idea is:
* when output is expecting specific range, the pixel data is normalized right before output.
* many filters don't care and work with arbitrary range. Any geometric filter should pass (flipping, rotating etc) also any convolution (blur).
* some filters involve a matrix multiply at input, so value range may be fused with that matrix.

In worst case it should normalize incoming pixels which is not big deal anyways.

[mawen1250]

The ones where it's easy are.

But there are only 3 formats you should use when filtering anyway. 8 bit, 16 bit and float. The rest just waste memory bandwidth and bits.

In reality you should see all other formats as specialized for input/output adaptation.

There're still exceptions though. For example, CTMF works fine with <=12bit, but will be ridiculously slow when processing 16bit. Other plugins based on histogram or lut may also have this kind of limitations.