Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 24th December 2021, 12:22   #1  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
Rounding float when converting to string

Just curious.... is there a reason why Avisynth doesn't always round from "5" the same way?

Cheers.

Subtitle(string(0.5,"%.0f")) Displays 1

Subtitle(string(0.05,"%.1f")) Displays 0.1

Subtitle(string(0.005,"%.2f")) Displays 0.00

Subtitle(string(0.0005,"%.3f")) Displays 0.001

Subtitle(string(0.00005,"%.4f")) Displays 0.0000

Subtitle(string(0.000005,"%.5f")) Displays 0.00000

Subtitle(string(0.0000005,"%.6f")) Displays 0.000000

Last edited by hello_hello; 24th December 2021 at 12:31.
hello_hello is offline   Reply With Quote
Old 24th December 2021, 12:52   #2  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
'Curiouser and curiouser!' cried Alice.

EDIT:
I get same with this [RT_String uses C library routines]
Code:
blankclip
Subtitle(RT_string("%.1f",0.05),y=0,lsp=0)       # 0.1
Subtitle(RT_string("%.2f",0.005),y=20,lsp=0)     # 0.00
Subtitle(RT_string("%.6f",0.0000005),y=40,lsp=0) # 0.000000
I guess you need to ask MicroSoft.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???

Last edited by StainlessS; 24th December 2021 at 13:14.
StainlessS is offline   Reply With Quote
Old 24th December 2021, 13:44   #3  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
It makes me feel a bit less vague though, as over the years I've distinctly remembered it rounds up, then when I've discovered I'd remembered incorrectly I've switched to remembering it rounds down, then sometime later I've discovered it rounds up......

It never occurred to me Avisynth wouldn't be consistent so until now I've just assumed I'm stupid.

Last edited by hello_hello; 24th December 2021 at 13:46.
hello_hello is offline   Reply With Quote
Old 24th December 2021, 14:06   #4  |  Link
wonkey_monkey
Formerly davidh*****
 
wonkey_monkey's Avatar
 
Join Date: Jan 2004
Posts: 2,493
Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate. It's either slightly above or slightly below the decimal representation so the rounded decimal representation may be 0.0..01 or 0.0...0
__________________
My AviSynth filters / I'm the Doctor
wonkey_monkey is offline   Reply With Quote
Old 24th December 2021, 14:28   #5  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Quote:
It never occurred to me Avisynth wouldn't be consistent so until now I've just assumed I'm stupid.
Just because Avisynth can get it wrong, it does not prove conclusively that you are not stupid
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 24th December 2021, 16:11   #6  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
"Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate."

It may be sad news for lovers of HDR in float32 0.0 .. 1.0 range.
Current HDR PQ 10bit moving pictures systems need about 1 000 000 : 1 absolute linear range with small enough errors. So it must encode about 1/1000000 or better 1/10000000 without visible errors (may be up to 1/10 of least significant value error about acceptable).

Last edited by DTL; 24th December 2021 at 16:17.
DTL is offline   Reply With Quote
Old 24th December 2021, 17:45   #7  |  Link
lvqcl
Registered User
 
Join Date: Aug 2015
Posts: 293
IIRC float32 can contain all values of uint24, so if 24-bit integer is OK for HDR, then 32-bit float is OK too.
lvqcl is offline   Reply With Quote
Old 24th December 2021, 17:55   #8  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
Quote:
Originally Posted by StainlessS View Post
Just because Avisynth can get it wrong, it does not prove conclusively that you are not stupid
I did briefly consider it might be because the values can't be represented precisely as float. I even suggested it in my opening post originally, but dismissed the idea for some inexplicable reason and deleted that, so I'm back to realising I'm stupid again.

It's a hard life when you're just smart enough to know you're an idiot. If only I was a little bit dumber...

Quote:
Originally Posted by wonkey_monkey View Post
Apart from 0.5, the internal representation in float format (before formatting) of those numbers is inaccurate. It's either slightly above or slightly below the decimal representation so the rounded decimal representation may be 0.0..01 or 0.0...0
I have no idea why I didn't check that myself in the first place, apart from the idiot problem.

Code:
BlankClip()
Subtitle(string(0.5, "\n0.5     %.30f"), x=20, lsp=10)
Subtitle(string(0.05, "\n\n0.05     %.30f"), x=20, lsp=10)
Subtitle(string(0.005, "\n\n\n0.005     %.30f"), x=20, lsp=10)
Subtitle(string(0.0005, "\n\n\n\n0.0005     %.30f"), x=20, lsp=10)
Subtitle(string(0.00005, "\n\n\n\n\n0.00005     %.30f"), x=20, lsp=10)
Subtitle(string(0.000005, "\n\n\n\n\n\n0.000005     %.30f"), x=20, lsp=10)
Subtitle(string(0.0000005, "\n\n\n\n\n\n\n0.0000005     %.30f"), x=20, lsp=10)
hello_hello is offline   Reply With Quote
Old 24th December 2021, 19:54   #9  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
It looks float32 precision is still enough for 10bit PQ.

Make test C program:
Code:
#include <stdio.h>
#include <math.h>

double fromPQ(double val)
{
	double m1 = 2610.0 / 16384.0;
	double m2 = (2523.0 / 4096.0) * 128.0;
	double c3 = (2392.0 / 4096.0) * 32.0;
	double c2 = (2413.0 / 4096.0) * 32.0;
	double c1 = 3424.0 / 4096.0;

	double rcp_m1 = 1.0 / m1;
	double rcp_m2 = 1.0 / m2;

	double upper = fmax((pow(val, rcp_m2) - c1), 0);
	double lower = c2 - c3 * pow(val, rcp_m2);
	double result = pow(upper / lower, rcp_m1);

	return result;
	
}

int main()
{
	double val1 = 1.0 / 1023.0;
	double val2 = 1.0 / 1022.0;

	double fromPQ_v1 = fromPQ(val1);
	double fromPQ_v2 = fromPQ(val2);

	printf("Linear of %.20f PQ          is %.20f \n", val1, fromPQ_v1);
	printf("Linear of %.20f PQ in float is %.20f \n", val1, (float)fromPQ_v1);

	printf("Linear of %.20f PQ          is %.20f \n", val2, fromPQ_v2);
	printf("Linear of %.20f PQ in float is %.20f \n", val2, (float)fromPQ_v2);

	printf("Linear diff in double is %.20f \n", fromPQ_v2 - fromPQ_v1);
	printf("Linear diff in float  is %.20f \n", (float)((float)fromPQ_v2 - (float)fromPQ_v1));

}
Output is
Code:
Linear of 0.00097751710654936461 PQ          is 0.000000004042271_76459
Linear of 0.00097751710654936461 PQ in float is 0.000000004042271_85058
Linear of 0.00097847358121330719 PQ          is 0.000000004048986_65264
Linear of 0.00097847358121330719 PQ in float is 0.000000004048986_47943
Linear diff in double is 0.000000000006714_88805
Linear diff in float  is 0.000000000006714_62885
So at the lowest 2 levels of linear 10bit PQ (full) the error between double and float difference between 2 lowest encoded levels still lower about 1/1000 of value. Equal digits before _ mark. 4 decimal digits still equal for difference and 7 digits for absolute values. And total dynamic range of 10bit PQ (full) looks about 2.5e8. *if test program is correct*

Last edited by DTL; 24th December 2021 at 20:11.
DTL is offline   Reply With Quote
Old 30th December 2021, 06:09   #10  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
Well the unpredictable rounding probably turned out to be a blessing because it meant I had to tackle the problem a different way and I think it was simpler in the end.

It's just a function for converting float to string with some formatting. I wanted float to display to a specified maximum number of decimal places, but without any unnecessary trailing zeros.

Here it is for anyone who might find it useful. It also removes any negative sign from zero so zero won't display as -0.

Edit: Fixed a silly.

Code:
# ===============================================================================
#            FormatString
# ===============================================================================
#
#  Converts float to string, sets the maximum number of decimal places,
#  and removes trailing zeros as well as any negative sign from zero.
#
#  Limit:
#  The maximum number of decimal places. Range 0 to 6. The default is Limit=3.
#
#  Keep (keep float):
#  Unless Limit=0, Keep=true ensures integers will still display as float (1.0 rather than 1),
#  otherwise they display as integers. The default is Keep=false.
#
#  Subtitle(FormatString(281.25, 6))        displays as 281.25 (rather than 281.250000)
#  Subtitle(FormatString(281.04, 3))        displays as 281.04
#  Subtitle(FormatString(281.04, 1))        displays as 281
#  Subtitle(FormatString(281.04, 1, true))  displays as 281.0
#
# ===============================================================================

function FormatString(float F, int "Limit", bool "Keep")  {

Limit = abs(default(Limit, 3))
Keep = default(Keep, false)
Limit0 = (Limit == 0)

S = Limit0 ? string(F,"%.0f") : \
(Limit == 1) ? string(F,"%.1f") : \
(Limit == 2) ? string(F,"%.2f") : \
(Limit == 3) ? string(F,"%.3f") : \
(Limit == 4) ? string(F,"%.4f") : \
(Limit == 5) ? string(F,"%.5f") : string(F,"%.6f")

SLength = StrLen(S)
DecPos = Limit0 ? 0 : FindStr(S, ".")
L = Limit0 ? S : LeftStr(S, DecPos - 1)
R = Limit0 ? "" : RightStr(S, SLength - DecPos + 1)
RLength = Limit0 ? 0 : StrLen(R)
Rx = R + "x"

DeleteA = Limit0 ? 0 : \
(FindStr(Rx, "000000x") > 0) ? 6 : \
(FindStr(Rx, "00000x") > 0) ? 5 : \
(FindStr(Rx, "0000x") > 0) ? 4 : \
(FindStr(Rx, "000x") > 0) ? 3 : \
(FindStr(Rx, "00x") > 0) ? 2 : \
(FindStr(Rx, "0x") > 0) ? 1 : 0

DeleteB = Limit0 ? 0 : \
(FindStr(Rx, ".000000x") > 0) ? 7 : \
(FindStr(Rx, ".00000x") > 0) ? 6 : \
(FindStr(Rx, ".0000x") > 0) ? 5 : \
(FindStr(Rx, ".000x") > 0) ? 4 : \
(FindStr(Rx, ".00x") > 0) ? 3 : \
(FindStr(Rx, ".0x") > 0) ? 2 : \
(FindStr(Rx, ".x") > 0) ? 1 : 0

IsDecLast = ((DeleteA + 1) == DeleteB)
R = Limit0 || IsDecLast ? "" : LeftStr(R, RLength - DeleteA)
LR = L + R
IsNegative = (FindStr(LR, "-") > 0)
IsZero = (value(LR) == 0)
End = Limit0 || !Keep || !IsDecLast ? "" : ".0"
LR = IsNegative && IsZero ? RightStr(LR, StrLen(LR) - 1) : LR

return LR + End  }

# ===============================================================================

Last edited by hello_hello; 31st December 2021 at 02:10.
hello_hello is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 03:39.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.