Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
24th December 2021, 12:22 | #1 | Link |
Registered User
Join Date: Mar 2011
Posts: 4,823
|
Rounding float when converting to string
Just curious.... is there a reason why Avisynth doesn't always round from "5" the same way?
Cheers. Subtitle(string(0.5,"%.0f")) Displays 1 Subtitle(string(0.05,"%.1f")) Displays 0.1 Subtitle(string(0.005,"%.2f")) Displays 0.00 Subtitle(string(0.0005,"%.3f")) Displays 0.001 Subtitle(string(0.00005,"%.4f")) Displays 0.0000 Subtitle(string(0.000005,"%.5f")) Displays 0.00000 Subtitle(string(0.0000005,"%.6f")) Displays 0.000000 Last edited by hello_hello; 24th December 2021 at 12:31. |
24th December 2021, 12:52 | #2 | Link |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
'Curiouser and curiouser!' cried Alice.
EDIT: I get same with this [RT_String uses C library routines] Code:
blankclip Subtitle(RT_string("%.1f",0.05),y=0,lsp=0) # 0.1 Subtitle(RT_string("%.2f",0.005),y=20,lsp=0) # 0.00 Subtitle(RT_string("%.6f",0.0000005),y=40,lsp=0) # 0.000000
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? Last edited by StainlessS; 24th December 2021 at 13:14. |
24th December 2021, 13:44 | #3 | Link |
Registered User
Join Date: Mar 2011
Posts: 4,823
|
It makes me feel a bit less vague though, as over the years I've distinctly remembered it rounds up, then when I've discovered I'd remembered incorrectly I've switched to remembering it rounds down, then sometime later I've discovered it rounds up......
It never occurred to me Avisynth wouldn't be consistent so until now I've just assumed I'm stupid. Last edited by hello_hello; 24th December 2021 at 13:46. |
24th December 2021, 14:06 | #4 | Link |
Formerly davidh*****
Join Date: Jan 2004
Posts: 2,493
|
Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate. It's either slightly above or slightly below the decimal representation so the rounded decimal representation may be 0.0..01 or 0.0...0
|
24th December 2021, 14:28 | #5 | Link | |
HeartlessS Usurer
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
|
Quote:
__________________
I sometimes post sober. StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace "Some infinities are bigger than other infinities", but how many of them are infinitely bigger ??? |
|
24th December 2021, 16:11 | #6 | Link |
Registered User
Join Date: Jul 2018
Posts: 1,041
|
"Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate."
It may be sad news for lovers of HDR in float32 0.0 .. 1.0 range. Current HDR PQ 10bit moving pictures systems need about 1 000 000 : 1 absolute linear range with small enough errors. So it must encode about 1/1000000 or better 1/10000000 without visible errors (may be up to 1/10 of least significant value error about acceptable). Last edited by DTL; 24th December 2021 at 16:17. |
24th December 2021, 17:55 | #8 | Link | ||
Registered User
Join Date: Mar 2011
Posts: 4,823
|
Quote:
It's a hard life when you're just smart enough to know you're an idiot. If only I was a little bit dumber... Quote:
Code:
BlankClip() Subtitle(string(0.5, "\n0.5 %.30f"), x=20, lsp=10) Subtitle(string(0.05, "\n\n0.05 %.30f"), x=20, lsp=10) Subtitle(string(0.005, "\n\n\n0.005 %.30f"), x=20, lsp=10) Subtitle(string(0.0005, "\n\n\n\n0.0005 %.30f"), x=20, lsp=10) Subtitle(string(0.00005, "\n\n\n\n\n0.00005 %.30f"), x=20, lsp=10) Subtitle(string(0.000005, "\n\n\n\n\n\n0.000005 %.30f"), x=20, lsp=10) Subtitle(string(0.0000005, "\n\n\n\n\n\n\n0.0000005 %.30f"), x=20, lsp=10) |
||
24th December 2021, 19:54 | #9 | Link |
Registered User
Join Date: Jul 2018
Posts: 1,041
|
It looks float32 precision is still enough for 10bit PQ.
Make test C program: Code:
#include <stdio.h> #include <math.h> double fromPQ(double val) { double m1 = 2610.0 / 16384.0; double m2 = (2523.0 / 4096.0) * 128.0; double c3 = (2392.0 / 4096.0) * 32.0; double c2 = (2413.0 / 4096.0) * 32.0; double c1 = 3424.0 / 4096.0; double rcp_m1 = 1.0 / m1; double rcp_m2 = 1.0 / m2; double upper = fmax((pow(val, rcp_m2) - c1), 0); double lower = c2 - c3 * pow(val, rcp_m2); double result = pow(upper / lower, rcp_m1); return result; } int main() { double val1 = 1.0 / 1023.0; double val2 = 1.0 / 1022.0; double fromPQ_v1 = fromPQ(val1); double fromPQ_v2 = fromPQ(val2); printf("Linear of %.20f PQ is %.20f \n", val1, fromPQ_v1); printf("Linear of %.20f PQ in float is %.20f \n", val1, (float)fromPQ_v1); printf("Linear of %.20f PQ is %.20f \n", val2, fromPQ_v2); printf("Linear of %.20f PQ in float is %.20f \n", val2, (float)fromPQ_v2); printf("Linear diff in double is %.20f \n", fromPQ_v2 - fromPQ_v1); printf("Linear diff in float is %.20f \n", (float)((float)fromPQ_v2 - (float)fromPQ_v1)); } Code:
Linear of 0.00097751710654936461 PQ is 0.000000004042271_76459 Linear of 0.00097751710654936461 PQ in float is 0.000000004042271_85058 Linear of 0.00097847358121330719 PQ is 0.000000004048986_65264 Linear of 0.00097847358121330719 PQ in float is 0.000000004048986_47943 Linear diff in double is 0.000000000006714_88805 Linear diff in float is 0.000000000006714_62885 Last edited by DTL; 24th December 2021 at 20:11. |
30th December 2021, 06:09 | #10 | Link |
Registered User
Join Date: Mar 2011
Posts: 4,823
|
Well the unpredictable rounding probably turned out to be a blessing because it meant I had to tackle the problem a different way and I think it was simpler in the end.
It's just a function for converting float to string with some formatting. I wanted float to display to a specified maximum number of decimal places, but without any unnecessary trailing zeros. Here it is for anyone who might find it useful. It also removes any negative sign from zero so zero won't display as -0. Edit: Fixed a silly. Code:
# =============================================================================== # FormatString # =============================================================================== # # Converts float to string, sets the maximum number of decimal places, # and removes trailing zeros as well as any negative sign from zero. # # Limit: # The maximum number of decimal places. Range 0 to 6. The default is Limit=3. # # Keep (keep float): # Unless Limit=0, Keep=true ensures integers will still display as float (1.0 rather than 1), # otherwise they display as integers. The default is Keep=false. # # Subtitle(FormatString(281.25, 6)) displays as 281.25 (rather than 281.250000) # Subtitle(FormatString(281.04, 3)) displays as 281.04 # Subtitle(FormatString(281.04, 1)) displays as 281 # Subtitle(FormatString(281.04, 1, true)) displays as 281.0 # # =============================================================================== function FormatString(float F, int "Limit", bool "Keep") { Limit = abs(default(Limit, 3)) Keep = default(Keep, false) Limit0 = (Limit == 0) S = Limit0 ? string(F,"%.0f") : \ (Limit == 1) ? string(F,"%.1f") : \ (Limit == 2) ? string(F,"%.2f") : \ (Limit == 3) ? string(F,"%.3f") : \ (Limit == 4) ? string(F,"%.4f") : \ (Limit == 5) ? string(F,"%.5f") : string(F,"%.6f") SLength = StrLen(S) DecPos = Limit0 ? 0 : FindStr(S, ".") L = Limit0 ? S : LeftStr(S, DecPos - 1) R = Limit0 ? "" : RightStr(S, SLength - DecPos + 1) RLength = Limit0 ? 0 : StrLen(R) Rx = R + "x" DeleteA = Limit0 ? 0 : \ (FindStr(Rx, "000000x") > 0) ? 6 : \ (FindStr(Rx, "00000x") > 0) ? 5 : \ (FindStr(Rx, "0000x") > 0) ? 4 : \ (FindStr(Rx, "000x") > 0) ? 3 : \ (FindStr(Rx, "00x") > 0) ? 2 : \ (FindStr(Rx, "0x") > 0) ? 1 : 0 DeleteB = Limit0 ? 0 : \ (FindStr(Rx, ".000000x") > 0) ? 7 : \ (FindStr(Rx, ".00000x") > 0) ? 6 : \ (FindStr(Rx, ".0000x") > 0) ? 5 : \ (FindStr(Rx, ".000x") > 0) ? 4 : \ (FindStr(Rx, ".00x") > 0) ? 3 : \ (FindStr(Rx, ".0x") > 0) ? 2 : \ (FindStr(Rx, ".x") > 0) ? 1 : 0 IsDecLast = ((DeleteA + 1) == DeleteB) R = Limit0 || IsDecLast ? "" : LeftStr(R, RLength - DeleteA) LR = L + R IsNegative = (FindStr(LR, "-") > 0) IsZero = (value(LR) == 0) End = Limit0 || !Keep || !IsDecLast ? "" : ".0" LR = IsNegative && IsZero ? RightStr(LR, StrLen(LR) - 1) : LR return LR + End } # =============================================================================== Last edited by hello_hello; 31st December 2021 at 02:10. |
Thread Tools | Search this Thread |
Display Modes | |
|
|