How can this (very very easy) calculation be done ?

[mbcd]

Hi there,

I have a small problem with some basics in avisynth.
I want to do a very simple calculation, but it ends in wrong values.

Avisynth gives me 720 / 1080 = 0,640741

that is too unprecise, because:
1080 * 0,640741 is 692 and not 720 again

Correct value should be: 0,666666666666666667

How can this be solved?
I have two Integrers as Input and I already casted them to Float by "Float(variable)" but I never get a correct result.

[Gavino]

It should work if the division is done with floats.

:script:

[raffriff42]

Gavino is correct of course (and beat me to it)

q = Float(720) / 1080
return MessageClip(String(q))result = "0.666667" (note display is truncated; actual value has higher precision) (EDIT: not correct - see post #5)

q = Float(720) / 1080
r = q * 1080
return MessageClip(String(r))result = "720.000000"

[mbcd]

I am sitting two hours at this problem and the simplest error was: The value of the variable was wrong, the calculation was correct :stupid:

What a, I checked everything but not this variable ... :mad:

Thanks guys !

[StainlessS]

JFYI


colorbars
q=720.0/1080
RT_Subtitle("%.32f",q) # On-frame
RT_Debug(String(q,"%.32f")) # to debugview window
return last


outputs this (printing 32 digits precision, single float of course less than that [single=7 digs I think, double=15 digs])

00000015 64.56612396 RT_Debug: 0.66666668653488159000000000000000


EDIT: The dross at the end due to fractional binary bits not gracefully converting into decimal (10 is not a power of 2).
EDIT: Above single precision float is actually converted to double before output, and explains 17 significant (but imprecise after 7) digits