Increased colour depth

[mg262]

For the third time while writing a filter, I've found that 8-bit/channel colour depth is a real problem.

- It means you have to try to avoid colour space conversions.

- It means when dealing with n colour spaces we need to implement every conversion separately (to minimise rounding error); with a high colour depth, you only have to provide conversions to and from a reference space (typically XYZ). (Of course direct conversion is faster, where this matters...)

- Most optimised filters which involving averaging will need to use internal 8->16 and 16->8 instructions (i.e. the PACK and UNPCK family).This includes ConvertFPS, and pretty much anything that blurs. Except when memory/HDD throughput is the limiting factor, this means that current filter versions waste time compared to high colour depth versions. (OTOH I'm not sure how often it is memory that limits things.) Also PACK/UNPACKing ties up a register, which is a nuisance.

- Filters like Levels, Gamma might possibly potentially maybe cause banding (although I haven't seen it myself).

- It means you have to worry about rounding in the right direction.

So I was considering hacking the colour format to use a higher depth/channel, simply by using two pixels to encode every one pixel's worth of information. This could be done either in RGB32 (or maybe RGB24) or in YV12 to simulate RGB64 or YV24 spaces. Filters using these colour spaces would be surrounded by a couple of calls like "DoubleDepth" and "HalveDepth". (Actually 15-bit depth might be preferable to 16 because then you can subtract pixel values without worrying -- and more generally not worry too much about signed/unsigned.)

The thing is, this is a hack (which is why I didn't post after the first of the 3 times!) and I know new colour spaces are turning up in V3.0, so I didn't want to take things done the wrong path/tread on anyone's toes without checking first... :confused:

[Manao]

You'll still need the pack / unpack if you switch to YV24. Except of doing unpcklbw/unpckhbw, you'll have to do unpcklwd/unpckhwd, and it'll be even slower. You can limit to 15, or even 14 bits, but that wouldn't be very satisfactory ( what if you've got to sum more than 4 pixels ? What if you've got to make a multiplication ? ). And memory accesses will be doubled, and that's clearly the limiting factor.

Oh, and of course, every filter will be uncompatible and would have to be totally rewritten.

[mg262]

I normally use PMADDWD to do the averaging, and that takes 16-bit arguments and returns 32-bit results. So you can pass values into that instruction without unpacking (and I typically find I have several unpacks for every pack).

every filter will be uncompatible and would have to be totally rewritten.I'm sorry, I don't follow... I meant this:

#normal filters
DoubleDepth
#depth doubled filters
HalveDepth
#normal filters

I really really didn't mean to suggest that I wanted or expected anyone to rewrite their filters... it simply that I have some that I'd want to write in this way, mainly because I need the extra accuracy; this method lets me keep them as separate filters rather than fusing them altogether.

As for memory -- from measurements I thought that I found in some complex filters it wasn't the limiting factor -- but I have played with this kind of stuff a lot less than you, so you're probably right.

[Didée]

For the third time while writing a filter, I've found that 8-bit/channel colour depth is a real problem.
Welcome in the club :)
Whenever bigger color depths will get available, I'll join the party.

[Manao]

mg262 : if you really need the 16 bits depth, then do as you have to. Hacking colorspaces aren't uncommon ( look at MVTools ), and since avs 2.5 doesn't provide a clean way for adding colorspaces, you don't really have the choice.

However, i still maintain that switching from 8 to 16 bits depth won't increase speed, so for me the only valid reason for making the switch is your need for extra accuracy.

[Mug Funky]

i'd settle for 10-bit, though 12 would be cool too (CCDs capture in 12 bits).

one thing that'd be interesting - receiving more than 8 bits from an avi or other source. would AVIsource have to be rewritten/hacked as well? there are BlackMagic codecs available for VfW which support 10-bit 4:2:2, but currently i've ne way of getting them into avisynth (actually, i have no way of getting 10 bits INTO this codec, as all the SDI equipment i have access to is mac/quicktime based).

[mg262]

i still maintain that switching from 8 to 16 bits depth won't increase speed I'm very willing to believe you -- as I said originally, I wasn't sure how often the memory was the limiting factor. (The argument against being that MOVDQA is as fast as MOVQ, and the cache didn't always seem to be the limiting factor, as if it were, moving C++ -> SSE2 shouldn't give substantial speedups. OTOH it did seem very implausible that doubling memory usage would speed things up -- but I couldn't find the hole in that argument. So I wasn't sure.) Even if it would have sped things up I wouldn't have implemented it just for that reason.

Whenever bigger color depths will get available, I'll join the party. Alas, you may be left standing at the door Didée... your scripts use quite a few different filter functions and it would be a lot of work to update them all. :( Which of your scripts would particularly benefit from higher colour depth?

i'd settle for 10-bit, though 12 would be cool too (CCDs capture in 12 bits). 10 or 12 are no easier than 15. Although to my inexperienced eyes 32 (Lustre) seems like overkill ;)one thing that'd be interesting - receiving more than 8 bits from an avi or other source. would AVIsource have to be rewritten/hacked as well? Almost certainly -- unless you could find a way to scale up the information in the file using another application (i.e. multiply by all the values by four); then you could use AVIsource twice and combine the values to reproduce 10-bit.

[Bidoche]

I think AviSource could read 10 bits colorspaces (if updated of course)

After all, it just gets a data block from vfw, and make a frame from it.

[ambrotos]

and what about something like RGBE ?
http://www.graphics.cornell.edu/~bjw/rgbe.html

or any other floating point format ?

[mg262]

and what about something like RGBE ?
http://www.graphics.cornell.edu/~bjw/rgbe.html

or any other floating point format ?Do you mean what about supporting it, or what about importing it?

I can't currently think of any advantages of working with that space over working with a high fixed colour depth? Or more precisely, I can't think of a filter that would benefit from it? (I haven't thought about how much depth FFT/iFFT would require yet...)

[ambrotos]

I was just wondering if using floating point format in filters would be worthy ? (as an internal format, or even as an internal avisynth format. For example between the instruction tofloat() and toyuv12() all the filters could use and return frame expressed in floats...)

[mg262]

I was just wondering if using floating point format in filters would be worthy ? (as an internal format, or even as an internal avisynth format. For example between the instruction tofloat() and toyuv12() all the filters could use and return frame expressed in floats...)I'm nowhere near as bothered about speed as some people (some of the unreleased filters I use are ~1fps, even after a 3-4x SSE2 speedup!), but working in floating point really does make things a lot slower. I'm was thinking of implementing 15-bit depth because I wanted to use some nonlinear colour space transformations (->CIELab or CIELuv) and I don't think it's sensible to try those in 8-bit depth -- but 15 should be enough. Anyway, I think I remember reading that v3 will have floatingpoint format support, at least for the colour spaces you can add yourself. But for most purposes, I don't think floatingpoint would be distinguishable from 16-bit -- FFTs might be an exception, but even there 32-bit might do the job. I'm prepared to be proved wrong though!