Precision issues with comparing Floats and Integers

[jmac698]

Another bug, this ones crazy!

n1=16777475
n2=float(n1)
errmsg=n1<>n2?"Error: n1<>n2":"n1=n2"
messageclip(string(n1)+" "+string(n2)+" "+errmsg)


Unfortunately, this is preventing me from doing a critical calculation, until I know how to work around it.

[-Vit-]

That's not a bug, but the inherent limitation of 32-bit floats. With only 24 (effective) bits of mantissa + the sign bit, the central block of precisely representable integers is from -16777216 to +16777216. Outside of that range there are 'gaps', integers that cannot be held precisely.

I expect n1 will be held in an int, which can hold that value precisely.

[jmac698]

Ok, that's a reasonable explanation, but I should *never* see two values as equal when they aren't. I'd rather get not equal when it can't be sure. I think that would be safer. It can cause very subtle bugs. Does it make sense that n1=float(n1) yet n1<>int(float(n1)) ?

http://avisynth.org/mediawiki/Known_Issues

[Chikuzen]

Ok, that's a reasonable explanation, but I should *never* see two values as equal when they aren't. I'd rather get not equal when it can't be sure. I think that would be safer. It can cause very subtle bugs. Does it make sense that n1=float(n1) yet n1<>int(float(n1)) ?

http://avisynth.org/mediawiki/Known_Issues

I think this is not an "issue" but a "limitation".
and, you should write a plugin instead of a script, supposing you desire accuracy of double.

[Gavino]

Does it make sense that n1=float(n1) yet n1<>int(float(n1)) ?
Yes, the first comparison is done as floating point and the lhs is first converted to float, so it means float(n1)==float(n1).

http://avisynth.org/mediawiki/Known_Issues
This is not a bug, or even an issue.
The differences between integer and floating point computations are one of the first things you learn in programming (at least, when I was a boy).
And of course here it only comes into play when n1 is >= 2^24.

[-Vit-]

I should *never* see two values as equal when they aren't.
You didn't see two values equal
You converted between types and rounding was required to fit the target type:
float(16777475) = 16777476.0
in the same way as
int(1.2) = 1

Forget suppositions of how floats should work. They they cannot hold an infinity of different values, nor do they have some kind of "not sure" flag. They are simply rounded to the nearest representable value. The rounding occurs for all values, not just integers. The precisely representable 32-bit float values are:
...
Every 0.25 from +/-2097152 to +/-4194304
Every 0.50 from +/-4194304 to +/-8388608
Every 1.00 from +/-8388608 to +/-16777216
Every 2.00 from +/-16777216 to +/-33554432
every 4.00 from +/-33554432 to +/-67108864
...

[jmac698]

I understand your point, but it's one of those things that are obvious to you if you already know it. The fact is, I'm trying to do a computation in script and due to the way this issue interacts, I'm getting a completely wrong answer.

It's not as obvious as you think, because if float were implemented as 32+8, you would get different results. In fact it may even depend on compile switches, the CPU involved, if vectorization is used and so on.

Where in the Avisynth documentation does it state that float is 24+8? If it's not documented, it's impossible to know how to write my code, and you were only able to answer because you were familiar with C and/or the Avisynth code.

If I were to implement a comparison between different precision numbers, I would promote *both* to the largest accuracy needed. So let's see what IEEE754 has to say about this... I can't find a free reference, does anyone know how 754 or C answers this question? And why does Avisynth treat this the way it does? Is it following something C-like?

@Chikuzen
Actually I've been trying to write a plugin, but I can't. It turns out that I'm unable to modify demoaic to work for me, and I had to give up. http://avisynthnew.wikinet.org/wiki/Avisynth_Plugin_Development_in_C#Passing_Parameters_to_your_Plugin

[-Vit-]

If I were to implement a comparison between different precision numbers, I would promote *both* to the largest accuracy needed.
That is exactly what happens, and is part of the "problem" in this case


n1=16777475
n2=float(n1) ==> n2=float(16777475) ==> n2=16777476.0
n1<>n2 ==> 16777475<>16777476.0 ==> 16777476.0<>16777476.0 ==> false

...but the promotion of n1 in the n1<>n2 line doesn't actually change n1
it's just a temporary to perform the comparison so:
n1=16777475
n2=16777476.0
n1<>n2 = false

[Gavino]

Actually I've been trying to write a plugin, but I can't. It turns out that I'm unable to modify demoaic to work for me, and I had to give up.
Don't give up so easily!
If you have specific questions or problems, raise them (in a separate thread) and someone here will help.

[Gavino]

Your original example evaluated in stages

n1=16777475
n2=float(n1) ==> n2=float(16777475) ==> n2=16777476.0
n1<>n2 ==> 16777475<>16777476.0 ==> 16777475.0<>16777476.0 ==> true

Er, no.

n1=16777475
n2=float(n1) ==> n2=float(16777475) ==> n2=16777476.0
n1<>n2 ==> 16777475<>16777476.0 ==> 16777476.0<>16777476.0 ==> false

[-Vit-]

Beat me to it, just fixed that...!
And that's as good a demonstration as any as why you should generally avoid comparing floats for equality/inequality

[jmac698]

Yes, Gavino well illustrated my point, the seemingly illogical conclusion that different numbers are the same.

Anyhow, can we call this a documentation issue, the fact that it's not stated anywhere how float is implemented, and that this is relevant because I can't write my function without knowing this. I'm still curious about what the standards say about the issue as well.

Edit:
Apparently only Fortran implements IEEE754 properly, so I'll change my question to ask what does C++ say about this, as it's pointless memorizing anything to do with IEEE754 for future reference.

http://stackoverflow.com/questions/716311/what-languages-get-ieee-754-right

[-Vit-]

That example is not directly to do with IEEE754. Look at my updated post above (blush) where I illustrate the reason for your seemingly illogical result. It's concerns the fact that the promotion of n1 to a float is only temporary. That's an issue of parsing/evaluation, the exact mechanics or result of the IEEE754 rounding are not relevant, merely the fact that some was needed.

You wouldn't want n1<>n2 to permanently change n1 to a float would you?

[amtm]

Using fortran is not going to change the fact that trying to do floating point comparisons as you tried to do is wrong. You'll still end.up getting erroneous results. You always check if the value is within + or - your error % from the value you expect. This is basic knowledge. Why is it the job of the avisynth documentation to teach you this?

[TheFluff]

Using fortran is not going to change the fact that trying to do floating point comparisons as you tried to do is wrong. You'll still end.up getting erroneous results. You always check if the value is within + or - your error % from the value you expect. This is basic knowledge. Why is it the job of the abisynth documentation to teach you this?

Exactly. If you expect two floats to really be exactly the same (as in the exact same bit pattern, which means you have to differ between positive and negative zero for example) then you should by all means use the equality operator. If not, you should compare in some other way, perhaps using rounding.

As amtm says, this is not something that is unique to Avisynth or to C or to any language in particular. It's just a basic fact of how floating point numbers are implemented on basically all computer hardware, and it is something you pretty much have to know if you ever want to write anything that uses floating point numbers in any capacity whatsoever in a language that isn't so high-level that it has abstracted away hardware maths entirely.

Anyway, that's a digression, since your actual problem is unrelated, as -Vit- pointed out.

[-Vit-]

Just to reinforce the point about not testing floats for equality, try this:

b = 0.11 + 0.11 + 0.11 == 0.33
messageclip(string(b))

[jmac698]

I do understand. I just think we aren't able to communicate to each other. Let me put it this way, someday a new Avisynth comes out which starts using double for it's script float (and why not?). Suddenly my comparison is working. Let's say I'm perfectly knowledgeable about floating point, C, and so forth, yet I still have no way to predict why my script should work in one version and not the other, unless the implmentation of float is documented, further that the casting rules of the arbitrary Avisynth language are documented. Do you agree?

I've updated the wiki to mark it as a documentation issue, not a bug of any kind.
http://avisynth.org/mediawiki/Known_Issues

[amtm]

Let me put it this way, someday a new Avisynth comes out which starts using double for it's script float (and why not?).

Because that's not what a "float" is. Having a different meaning for "float" between script and source code would be incredibly dumb. The correct way would be to introduce the "double" type to not cause needless confusion.

Suddenly my comparison is working. Let's say I'm perfectly knowledgeable about floating point, C, and so forth, yet I still have no way to predict why my script should work in one version and not the other, unless the implmentation of float is documented, further that the casting rules of the arbitrary Avisynth language are documented. Do you agree?

I've updated the wiki to mark it as a documentation issue, not a bug of any kind.
http://avisynth.org/mediawiki/Known_Issues

I would agree with pointing out that float is a single-precision floating number which the docs don't explicitly say now. The docs, though, is not the place to explain to explain the floating point standard or how C++ promotion rules work. That is the point of reference links.

[-Vit-]

OK, you have your double-based avisynth. Will this script give the correct result?

b = 0.1111 + 0.1111 + 0.1111 == 0.3333
messageclip(string(b))

You don't know without trying. It wouldn't.

This would work though:
b = 0.1011 + 0.1011 + 0.1011 == 0.3033

Maybe you think you can test all cases. But calculations are fairly useless without script variables, so how would you test all the possibilities:
b = UserInput1 + UserInput2 + UserInput3 == CalculatedValue

What about:
b = (a * b) / (x + y * v / Sqrt(w)) == p - q

In all but the rarest circumstances it's unsafe to compare floating point values (single or double) for equality or inequality. Edit: I guess it would be useful to make that comment in a tutorial section...

[amtm]

I've updated the wiki to show that.its a single-precision floating-point number with a link to the Wikipedia article on floating point. Is that enough to settle this? Do we also need to point out that int is a 32-bit integer and give explain how values will be truncated if they exceed the maximum allowable number as well?

[jmac698]

Actually I agree with you about pointing to references. Correct, there is no need to explain how floating point works in Avisynth docs, that is out of scope. Simply point out the information necessary to do correct calculations in script (namely the implementation issues).

However, can you think of an appropriate category for alerting beginner programmers, about the pitfalls of trying to compare values of different precision? Perhaps Advanced Scripting Tips?

One more question, what are the promotion rules for Avisynth script? Are you implyiing that they are the same as C++, because the script is written in C++ and somehow the natural way for the interpreter to come out, is with the underlying C++ rules?

I think it could be pointed out that the scripting language follows the underlying C++ promotion rules.

[amtm]

I also added this line to the float function:

Integer values that require more than 24-bits to be represented will have their lower 8-bits truncated yielding unexpected values.

[jmac698]

That's great. Thanks for your help in updating the documentation.

[jmac698]

@Vit
We are miscommunicating. No matter, I am satisfied with the updated documentation.

[amtm]

One more question, what are the promotion rules for Avisynth script? Are you implyiing that they are the same as C++, because the script is written in C++ and somehow the natural way for the interpreter to come out, is with the underlying C++ rules?

I think it could be pointed out that the scripting language follows the underlying C++ promotion rules.

I couldn't say for sure but it would make sense that it would follow how C++ works.

[Gavino]

http://avisynth.org/mediawiki/Operators:
AviSynth operators follow loosely the same rules as C operators, regarding meaning, precedence and associativity. By loosely we mean that there are some exceptions, indicated at the text below.

[amtm]

I assume it also follows the type promotion rules as well?

I believe that was especially causing the misunderstanding since int is lower in the type promotion hierarchy than float whereas jmac698 didnt seem to expect this.

Just so you know jmac the hierarchy from top to bottom goes:

long double
double
float
unsigned long int
long int
unsigned int
int
unsigned short
short
unsigned char
char

[jmac698]

It doesn't say anything about casting, but if I take it at face value, I'd have to assume that the casting is like C. Is this true?

Edit:
antm and I had the same thought :)

Talking about documentation only here, you have to realize that the majority of people writing avisynth scripts are not programmers, or familiar with C. As you can tell, though I'm familiar with C I have hardly used it, so a lot of these assumptions from the experienced programmers are lost to me.

Maybe I'll write up something under Advanced Scripting Tips.

[Gavino]

If a binary operator is applied to an int and a float, the int is first converted to float, as in C.
That's the only case in the Avisynth language where there is any conversion applicable.

[amtm]

Ok, so it is a C-like promotion as I expected.

Obviously most of the types don't apply since avisynth doesn't expose them but I figured I'd show the full hierarchy.

[jmac698]

http://en.wikipedia.org/wiki/Type_conversion
Explains it really well. I was impressed at how neutral it was, recognizing the fact that different languages handle this differently. So it does make a difference, probably, what language Avisynth was written in. If you were to read the docs as your sole reference, there's no way you would know these intricate details. So I've written some articles about it.

http://avisynth.org/mediawiki/Advanced_Scripting_Tips
http://avisynth.org/mediawiki/Known_Issues - I changed this to a brief mention of updated documentation.

[jmac698]

amtm:
Thanks for the list. I didn't see that edit. Very useful. Seems quite intuitive, I would just memorize that as being in order of size. The part that trips me up is promoting long int to float, there is a serious loss there. If I were to make my own lanugage, I'd order it by size of mantissa. Since C doesn't imply the size of long int (IIRC), even a C programmer can't know for sure how to do the calculations I outlined. So it's not so simple after all; not only do you have to know what language you're dealing with, but how it implements it's numerics, and possibly even compiler switches used like -f fast-math

http://gcc.gnu.org/wiki/FloatingPointMath

[amtm]

What's even more fun is that in Windows (whether it's 16-bit, 32-bit or 64-bit) long int is 32-bit. Whereas to get 64-bit you have to use __int64. Whilst on *nix long int is 64-bit on 64-bit systems.

[amtm]

Oh and this:

The comparison operator returns true even though the numbers are different because it only checks within the limits of precision of a float.

should be worded differently. The reason it returns true is because the number was temporarily promoted to float and subsequently truncated before equality was evaluated.

[IanB]

Perhaps a Mod can slice out posts 274 through post 307 (or whatever it is by the time it gets actioned) to new thread "Precision issues with comparing Floats and Integers".

This has nothing to do with 2.6.0 Alpha3 it is a generic issue.

[TheFluff]

there is a reason there is a sizeof() operator in c

[jmac698]

I was thinking that, but it doesn't tell you how many bits of mantissa.

[amtm]

I believe he was referring to my comment about long int sizes.

[Guest]

Perhaps a Mod can slice out posts 274 through post 307 (or whatever it is by the time it gets actioned) to new thread "Precision issues with comparing Floats and Integers".

This has nothing to do with 2.6.0 Alpha3 it is a generic issue. Done.

[amtm]

To do what you want jmac look up frexp in math.h. cmath in C++ has additional overloads over the C version.

[jmac698]

This is all I've got in mingw:

/* 7.12.6.4 Double in C89 */
extern float __cdecl frexpf (float, int*);
#ifndef __NO_INLINE__
__CRT_INLINE float __cdecl frexpf (float x, int* expn)
{return (float) frexp (x, expn);}
#endif
extern long double __cdecl frexpl (long double, int*);

http://www.codecogs.com/reference/c/math.h/frexp.php

Ok, it returns *a* mantissa and *an* exponent, but it's not clear that it's *the* mantissa/exponent of the native format. Sizeof of the int , used to store expn, doesn't tell me the size of the exponent, I'll likely get 4 bytes when the exponent is really 1 byte.

It's not that I need it today, but it would be nice to know if there's a way to write portable code.

[amtm]

You can just hardcode in that float has 24-bit mantissa and double is 53-bit. Those values are defined in the standard so what portability issues do you mean?

[jmac698]

So wait, you mean long int isn't defined in the standard, but you can use sizeof to find it. Yet float and double are defined in the standard, including mantissa size. So therefore switches like -f fast-math make the compiler non-standard, use at your own peril.
And does the C++ standard also conform to IEEE-754 or not?

[amtm]

Huh? I'm talking about the ieee standard for floating point. Secondly long int IS defined in the C standard as guaranteeing that it's at least as large as int but can be larger. Lastly in what way are you asking if it conforms to ieee-754? In what way do you think the standard or some implementation is deficient?

[jmac698]

Ok, there's 3 basic formats, which correspond to what we think of as single, double, and quad.
http://en.wikipedia.org/wiki/IEEE_754-2008

I thought you meant the C standard. Yes, of course the bits are defined in the IEEE standard. But I'm wondering, relevant to C++, if there's a way to know how many bits are in the mantissa programmatically.
If the C++ says, the float and double types conform to the IEEE standard, then the answer is always 24, 53 or 112.
If that's not the case, then is there a way to deduce what it is? I think I've seen math libraries that went through hoops to get the precision right.

There are differences, http://www.akkadia.org/drepper/libm/ shows glibc to be in spec.

[amtm]

Why do you need to know it programmaticly? I told you the sizes. You seem to be inventing issues that don't exist.

[jmac698]

Argh. I wanted to know if there's a way to make it portable. That was the whole question. Either all C++ conformant compilers used IEEE, in which case it's a non-issue, or they don't, in which case the mantissa has to be deduced somehow.

[amtm]

I told you the portable sizes. 24-bit for float, 53-bit for double. That link you pointed to has to do with the error percentage of the floating point calculations not to do with the bit widths of the mantissa/exponent.

[jmac698]

Ok, great thanks.

[amtm]

Also just so you know, "long double" is not necessarily the IEEE quadruple-precision floating point. On x86 and x86_64 it is usually the 80-bit extended format though there are switches in a number of compilers that enable the 128-bit floating point version. As with the size of "long int" with respect to "int", all that the standard requires is that "long double" is at least as precise as "double" but it can be extended.