Rounding float when converting to string

[hello_hello]

Just curious.... is there a reason why Avisynth doesn't always round from "5" the same way?

Cheers.

Subtitle(string(0.5,"%.0f")) Displays 1

Subtitle(string(0.05,"%.1f")) Displays 0.1

Subtitle(string(0.005,"%.2f")) Displays 0.00

Subtitle(string(0.0005,"%.3f")) Displays 0.001

Subtitle(string(0.00005,"%.4f")) Displays 0.0000

Subtitle(string(0.000005,"%.5f")) Displays 0.00000

Subtitle(string(0.0000005,"%.6f")) Displays 0.000000

[StainlessS]

'Curiouser and curiouser!' cried Alice.

EDIT:
I get same with this [RT_String uses C library routines]

blankclip
Subtitle(RT_string("%.1f",0.05),y=0,lsp=0) # 0.1
Subtitle(RT_string("%.2f",0.005),y=20,lsp=0) # 0.00
Subtitle(RT_string("%.6f",0.0000005),y=40,lsp=0) # 0.000000

I guess you need to ask MicroSoft.

[hello_hello]

It makes me feel a bit less vague though, as over the years I've distinctly remembered it rounds up, then when I've discovered I'd remembered incorrectly I've switched to remembering it rounds down, then sometime later I've discovered it rounds up......

It never occurred to me Avisynth wouldn't be consistent so until now I've just assumed I'm stupid.

[wonkey_monkey]

Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate. It's either slightly above or slightly below the decimal representation so the rounded decimal representation may be 0.0..01 or 0.0...0

[StainlessS]

It never occurred to me Avisynth wouldn't be consistent so until now I've just assumed I'm stupid.
Just because Avisynth can get it wrong, it does not prove conclusively that you are not stupid :)

[DTL]

"Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate."

It may be sad news for lovers of HDR in float32 0.0 .. 1.0 range.
Current HDR PQ 10bit moving pictures systems need about 1 000 000 : 1 absolute linear range with small enough errors. So it must encode about 1/1000000 or better 1/10000000 without visible errors (may be up to 1/10 of least significant value error about acceptable).

[lvqcl]

IIRC float32 can contain all values of uint24, so if 24-bit integer is OK for HDR, then 32-bit float is OK too.

[hello_hello]

Just because Avisynth can get it wrong, it does not prove conclusively that you are not stupid :)

I did briefly consider it might be because the values can't be represented precisely as float. I even suggested it in my opening post originally, but dismissed the idea for some inexplicable reason and deleted that, so I'm back to realising I'm stupid again.

It's a hard life when you're just smart enough to know you're an idiot. If only I was a little bit dumber...

Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate. It's either slightly above or slightly below the decimal representation so the rounded decimal representation may be 0.0..01 or 0.0...0

I have no idea why I didn't check that myself in the first place, apart from the idiot problem.

BlankClip()
Subtitle(string(0.5, "\n0.5 %.30f"), x=20, lsp=10)
Subtitle(string(0.05, "\n\n0.05 %.30f"), x=20, lsp=10)
Subtitle(string(0.005, "\n\n\n0.005 %.30f"), x=20, lsp=10)
Subtitle(string(0.0005, "\n\n\n\n0.0005 %.30f"), x=20, lsp=10)
Subtitle(string(0.00005, "\n\n\n\n\n0.00005 %.30f"), x=20, lsp=10)
Subtitle(string(0.000005, "\n\n\n\n\n\n0.000005 %.30f"), x=20, lsp=10)
Subtitle(string(0.0000005, "\n\n\n\n\n\n\n0.0000005 %.30f"), x=20, lsp=10)

https://i.postimg.cc/3xrbcyXM/clipboard.jpg

[DTL]

It looks float32 precision is still enough for 10bit PQ.

Make test C program:

#include <stdio.h>
#include <math.h>

double fromPQ(double val)
{
double m1 = 2610.0 / 16384.0;
double m2 = (2523.0 / 4096.0) * 128.0;
double c3 = (2392.0 / 4096.0) * 32.0;
double c2 = (2413.0 / 4096.0) * 32.0;
double c1 = 3424.0 / 4096.0;

double rcp_m1 = 1.0 / m1;
double rcp_m2 = 1.0 / m2;

double upper = fmax((pow(val, rcp_m2) - c1), 0);
double lower = c2 - c3 * pow(val, rcp_m2);
double result = pow(upper / lower, rcp_m1);

return result;

}

int main()
{
double val1 = 1.0 / 1023.0;
double val2 = 1.0 / 1022.0;

double fromPQ_v1 = fromPQ(val1);
double fromPQ_v2 = fromPQ(val2);

printf("Linear of %.20f PQ is %.20f \n", val1, fromPQ_v1);
printf("Linear of %.20f PQ in float is %.20f \n", val1, (float)fromPQ_v1);

printf("Linear of %.20f PQ is %.20f \n", val2, fromPQ_v2);
printf("Linear of %.20f PQ in float is %.20f \n", val2, (float)fromPQ_v2);

printf("Linear diff in double is %.20f \n", fromPQ_v2 - fromPQ_v1);
printf("Linear diff in float is %.20f \n", (float)((float)fromPQ_v2 - (float)fromPQ_v1));

}


Output is

Linear of 0.00097751710654936461 PQ is 0.000000004042271_76459
Linear of 0.00097751710654936461 PQ in float is 0.000000004042271_85058
Linear of 0.00097847358121330719 PQ is 0.000000004048986_65264
Linear of 0.00097847358121330719 PQ in float is 0.000000004048986_47943
Linear diff in double is 0.000000000006714_88805
Linear diff in float is 0.000000000006714_62885


So at the lowest 2 levels of linear 10bit PQ (full) the error between double and float difference between 2 lowest encoded levels still lower about 1/1000 of value. Equal digits before _ mark. 4 decimal digits still equal for difference and 7 digits for absolute values. And total dynamic range of 10bit PQ (full) looks about 2.5e8. *if test program is correct*

[hello_hello]

Well the unpredictable rounding probably turned out to be a blessing because it meant I had to tackle the problem a different way and I think it was simpler in the end.

It's just a function for converting float to string with some formatting. I wanted float to display to a specified maximum number of decimal places, but without any unnecessary trailing zeros.

Here it is for anyone who might find it useful. It also removes any negative sign from zero so zero won't display as -0.

Edit: Fixed a silly.

# ===============================================================================
# FormatString
# ===============================================================================
#
# Converts float to string, sets the maximum number of decimal places,
# and removes trailing zeros as well as any negative sign from zero.
#
# Limit:
# The maximum number of decimal places. Range 0 to 6. The default is Limit=3.
#
# Keep (keep float):
# Unless Limit=0, Keep=true ensures integers will still display as float (1.0 rather than 1),
# otherwise they display as integers. The default is Keep=false.
#
# Subtitle(FormatString(281.25, 6)) displays as 281.25 (rather than 281.250000)
# Subtitle(FormatString(281.04, 3)) displays as 281.04
# Subtitle(FormatString(281.04, 1)) displays as 281
# Subtitle(FormatString(281.04, 1, true)) displays as 281.0
#
# ===============================================================================

function FormatString(float F, int "Limit", bool "Keep") {

Limit = abs(default(Limit, 3))
Keep = default(Keep, false)
Limit0 = (Limit == 0)

S = Limit0 ? string(F,"%.0f") : \
(Limit == 1) ? string(F,"%.1f") : \
(Limit == 2) ? string(F,"%.2f") : \
(Limit == 3) ? string(F,"%.3f") : \
(Limit == 4) ? string(F,"%.4f") : \
(Limit == 5) ? string(F,"%.5f") : string(F,"%.6f")

SLength = StrLen(S)
DecPos = Limit0 ? 0 : FindStr(S, ".")
L = Limit0 ? S : LeftStr(S, DecPos - 1)
R = Limit0 ? "" : RightStr(S, SLength - DecPos + 1)
RLength = Limit0 ? 0 : StrLen(R)
Rx = R + "x"

DeleteA = Limit0 ? 0 : \
(FindStr(Rx, "000000x") > 0) ? 6 : \
(FindStr(Rx, "00000x") > 0) ? 5 : \
(FindStr(Rx, "0000x") > 0) ? 4 : \
(FindStr(Rx, "000x") > 0) ? 3 : \
(FindStr(Rx, "00x") > 0) ? 2 : \
(FindStr(Rx, "0x") > 0) ? 1 : 0

DeleteB = Limit0 ? 0 : \
(FindStr(Rx, ".000000x") > 0) ? 7 : \
(FindStr(Rx, ".00000x") > 0) ? 6 : \
(FindStr(Rx, ".0000x") > 0) ? 5 : \
(FindStr(Rx, ".000x") > 0) ? 4 : \
(FindStr(Rx, ".00x") > 0) ? 3 : \
(FindStr(Rx, ".0x") > 0) ? 2 : \
(FindStr(Rx, ".x") > 0) ? 1 : 0

IsDecLast = ((DeleteA + 1) == DeleteB)
R = Limit0 || IsDecLast ? "" : LeftStr(R, RLength - DeleteA)
LR = L + R
IsNegative = (FindStr(LR, "-") > 0)
IsZero = (value(LR) == 0)
End = Limit0 || !Keep || !IsDecLast ? "" : ".0"
LR = IsNegative && IsZero ? RightStr(LR, StrLen(LR) - 1) : LR

return LR + End }

# ===============================================================================