Quote:
Originally Posted by hank315
I'm already working on it for some time, the general idea is about the same, variance based.
Will be in the next HCenc release.
Just had a look at the implementation in Xvid, well, I'm a C n00b but when I do something like this in Fortran it will overflow on high luminance values:
Code:
for (k = 0; k < 16; k++)
for (l = 0; l < 16; l++)
{
int val = ptr[k*data->current.stride[0] + l];
sum += val;
sum_of_squares += val * val;
}
/* Variance = SSD - SAD^2 / (numpixels) */
int variance = sum_of_squares - sum * sum / 256;
|
The maximum value that sum_of_squares can reach is (255 * 255 * 256), or a bit under 2^24, so it is not a problem.
The maximum value that sum can reach is 255*256... so yes, you are right; sum*sum could overflow a signed integer.
The correction would be to make sum and ssd unsigned integers.