Video Encoding: Floating Point or Integer?

[nFury8]

Now, don't laugh yet. :) This is an ignorant question from a non-programmer like me. I would like to know if video encoding is purely a floating-point operation or an integer one; straight from the mouths of babes, from those who've been there and done that. I thought this forum would be the most appropriate place to ask since there's so many programmers here that mostly deal with video encoding. Okay, now you can laugh.

[mf]

Spatial video is just about always in integer form. There are, depending on colorspace, either 3 8-bit ints representing red, green and blue respectively (RGB 24bit), or 4 8-bit ints representing red, green, blue and alpha (transparency) (RGBA 32bit), or 1 8-bit int representing luminance(Y), and 2 subsampled 8-bit ints (which for each "true" pixel represent 4 bits) representing chrominance (color planes, U and V aka Cb and Cr) (YUY2 / 16bit YUV 4:2:2), or 1 8-bit int representing luminance, and 2 subsampled 8-bit ints (representing 2bpp each) representing chrominance (YV12 / 12bit YUV 4:2:0).

When you encode video however, the spatial information is often transformed into discrete frequencies in order to compress in a smart way that the human eye does not notice easily, and depending on the transformation, those can be floating point (DCT for instance, which is used in (M)JPEG, and MPEG variants). Have a read on fourier, DCT, and wavelet transformations. The H.264 standard comes with an integer transform :).

[Nic]

Most of it is integer math. Mainly for speed and for ease of optimisation. floating point math would most likely result in higher quality, but the speed hit wouldn't be worth it.

Even a non-programmer can check if you wish. Get the XviD source and look to see how often int, char, etc are used as variables compared to float & double.

-Nic

ps
appears me and mf were typing at the same time.

[mf]

I also forgot to mention that all (to quote sysKin) "sane" implementations of DCT transforms are integer, though in theory both transform and resulting coefficients should be floating point.

[nFury8]

Thanks for the clarification, sirs. Should shed some light on this matter, it won't be bugging me no more.:)