The accuracy of floating point

[hello_hello]

The script I'm working on will do sub-pixel cropping using an Avisynth resizer, so in that respect it's not an issue, but it'll also display all it's calculations as subtitles, and when they should be whole numbers but they're not..... I guess it's just a little annoying.

For example, to work out how much extra to crop the width of an 16:9 anamorphic source with a 4:3 pixel aspect ratio (960x720) after resizing the width for square pixels and then adjusting it down for mod 16. I'm using this formula:

Cropping aspect = resized & mod adjust width / cropped height / pixel aspect ratio.

Based on a final output resolution of 1248x688 after cropping the height.
1248 / 688 / (4/3) = 1.3604651162790697674418604651163
so it works out the width needs to be cropped to
688 * 1.3604651162790697674418604651163 = 936 exactly

Then it can be resized to 1248x688 with zero aspect error.

When Avisynth does the same maths:

1248 / 688 / (4/3) = 1.36046504794365230000
688 * 1.36046504794365230000 = 935.99993896484375

As I said, the script doesn't care. It'll pass the extra cropping onto the resizer, and even if a resizer can crop that accurately you'd require bionic eyes to see it, but when the script should be displaying whole numbers for it's cropping and it's not, it's a bit annoying.

I assume the accuracy of floating point maths is just something you live with, but I thought I'd ask in case I'm missing something.

Cheers.

PS I don't want to round the result because the whole point of the script is to crop and resize with zero, or as close to zero aspect error as possible.

[hello_hello]

So far I've worked out the shorter the equation the less inaccurate the result is likely to be, which I suppose is fairly obvious when you think about it. I can't always shorten it because sometimes there's more variables (such as the height also being resized) but for my previous example if I work out the required width after cropping this way:

Resized & mod adjust / pixel aspect ratio
1248 / (4/3)

Avisynth does calculate the result as exactly 936. That's a start.....

[wonkey_monkey]

Rearrange the formula so you do as few divisions as possible, and that you do them last:

688 * (1248 / 688 / (4/3))

688 * (1248 * 3) / (688 * 4)

(1248 * 3) / 4

[hello_hello]

Thanks. I'll try that too.

Edit: Actually, that starts to get harder when you've already defined things earlier in the script, as then the formula might be:

DAR = 16/9
PixAR = DAR*c.height/c.width
1248/PixAR

Maybe I'll have to rethink some of that, or stop defining so much.

Cheers.

[jmartinr]

Why don't you just round in the end? You go for mod 16 anyway.

[TheFluff]

In general if you're using floating point math to compute any kind of fractional number, you should never expect any number involved to be exactly equal to any other number. In double precision, sure, your example appears to be "exactly" 936, but that's really only a coincidence. It just happens to work out in this case that the double precision epsilon is small enough that your calculation rounds to 936.0. If you expect an exact integer result, you should round. If you don't, you shouldn't round.

Also, as davidhorman points out, any time you do a calculation with a fractional result, the result will be rounded and these rounding errors accumulate. Follow his advice regarding divisions to reduce rounding errors.

[hello_hello]

Why don't you just round in the end? You go for mod 16 anyway.

Mod16 was just the example I was working with (I never bother myself), but it was an anamorphic source being resized to a mod16 width of 1248 and the idea is not to worry about mod2 cropping or even whole pixel cropping.

As an example, I've taken a 720x576 source with a 16:9 aspect ratio and told the script to crop like this crop(12,10,-8,-10), and output as close to 16:9 as it can within mod4 dimensions and without resizing the height. If any extra cropping is required, it does the thinking.

CropTest(0, 0, 12, 8, -8, -10, InDAR=16.0/9.0, PicDar=16.0/9.0, Hresize=false, Info=true)

I've only just come back to it but I changed the calculation for this particular "mode" to the method from my last post and so far what it's doing seems to be more accurate than I expected. If it keeps it up I'll try not to think about it.
In the screenshot below, cropping the width to 694.6875 and resizing it to 988 is the correct answer.

This will hopefully replace the previous version (https://forum.videohelp.com/threads/382888-CropResizeBorder-script) of the script to make it more accurate, faster and a little easier to use (eliminating the need to specify a mode). The last one works perfectly, but it's only accurate to within the limitations of mod2 cropping and I was hoping to improve on that. I never want to use a cropping/resizing calculator ever again. I'll keep testing.


https://files.videohelp.com/u/210984/CropTest.jpg

[shekh]

Typically, to display "nice" floating point value you truncate it to some precision. For example "%1.3g" printf-style format means show as much as 3 digits after decimal point, and remove trailing zeroes.
935.9999 will be displayed as 936
Don't know which number formatting is possible in Avisynth.

[raffriff42]

>Don't know which number formatting is possible in Avisynth.
String (http://avisynth.nl/index.php/Internal_functions#String)(688 * 1.36046504794365230000, "%0.3f") = "935.9999"
String(688 * 1.36046504794365230000, "%0.4f") = "936.000"

There's Muldiv (http://avisynth.nl/index.php/Internal_functions#MulDiv) too, which should help precision.
(EDIT if you can rephrase your equation as int x int / int)

[wonkey_monkey]

Surely those format strings are the wrong way round in those examples?

[hello_hello]

Typically, to display "nice" floating point value you truncate it to some precision. For example "%1.3g" printf-style format means show as much as 3 digits after decimal point, and remove trailing zeroes.
935.9999 will be displayed as 936
Don't know which number formatting is possible in Avisynth.

Based on that and what raffriff42 posted, if I limit the number of decimal places to four it looks like I should always round to displaying the correct result (so far).


I'm forcing Avisynth to display lots of decimal places to see what's going on at the moment, but now I've changed the equation a little, so far the only inaccurate calculations I've found in this mode resulted from an NTSC DVD with a 20:11 DAR.

My current method of of using predefined values looks like this in the script:
Additional_Cropping = Cropped_Source_Width - (New_Width / Pixel_Aspect)

which for my NTSC DVD, Avisynth calculated the division as being 673.199951171875

When I broke it back down into it's original form and shuffled the numbers around it became:
Additional_Cropping = 700.0 - (816.0 * 11.0 * float(c.width) / float(c.height) / 20.0)

which Avisynth calculates the multiplication and division as 673.200012207031

The correct answer is 673.2 so no doubt it's more accurate the second way, but for the purposes of the script if the first method is as bad as it gets.....

Once the pixel aspect ratio is 1.0, much of the potential for error is gone and I'm not sure I want to force the user to enter display aspect ratios as numerator and denominator so I can break the equations down more. I know it annoys me. I'll have to think about that one.

raffriff42,
thanks for the Muldiv suggestion! I'd forgotten about that.

It's all a learning experience.....

[jmartinr]

A practical answer is 672. You'll have to round somewhere, before you use the outcome of the calculation for cropping.

[raffriff42]

Surely those format strings are the wrong way round in those examples?Yes :o I was rushing out the door.

[hello_hello]

A practical answer is 672. You'll have to round somewhere, before you use the outcome of the calculation for cropping.

You don't have to round because the Avisynth resizers do sub-pixel cropping.
The script uses the standard crop function to do all the mod2 cropping and the resizer to crop the leftovers.

Even when I'm cropping manually I use resizer cropping a fair bit. Not for sub-pixel cropping, but to crop mod1 if need be for some extra accuracy.

Just as an example, if you wanted to crop four pixels from both the width and height of a 720x576 DVD (with a generic pixel aspect ratio), cropping an extra pixel from one side lets you resize to 16:9 dimensions with zero aspect error.

You can't do this for YV12

Crop(4, 2, -1, -2)
Spline36Resize(960, 540)

but you can do this:

Crop(4, 2, 0, -2)
Spline36Resize(960, 540, 0, 0, -1, 0)

When I first started on the script, the plan was to let the user enter sub-pixel cropping, but it created extra headaches in situations where non-mod2 cropping was entered but the script was in a non-resizing mode etc. Aside from the fact it's probably pointless anyway (when it can resize the script does sub-pixel cropping for you if need be) it hurt my brain too much.

[StainlessS]

String(688 * 1.36046504794365230000, "%0.3f") = "935.9999"
String(688 * 1.36046504794365230000, "%0.4f") = "936.000"

A bit back-to-front above, "%0.3f" produces 3 fractional digits, not 4, and similar typo for "%04f".


Avisynth builtin String() function is a little like C printf() function for formatting floats.

Docs
AviSynth Syntax - Conversion functions
String | v2.07 | String(float / int [, string format_string])
Converts a variable to a string.
If the variable is float or integer, it first converts it to a float and then uses format_string to convert the float to a string. The syntax of format_string is as follows:
%[flags][width][.precision]f
width: the minimum width (the string is never truncated)
precision: the number of digits printed # EDIT: Fractional Digits Printed
flags:
- left align (instead right align)
+ always print the +/- sign
0 padding with leading zeros
' ' print a blank instead of a "+"
# always print the decimal point

You can also put arbitrary text around the format_string as defined above, similar to the C-language printf function.

Subtitle( "Clip height is " + String(last.height) )
Subtitle( String(x, "Value of x is %.3f after AR calc") )
Subtitle( "Value of x is " + String(x, "%.3f") + " after AR calc") ) # same as above
String(1.23, "%f") = '1.23'
String(1.23, "%5.1f") = ' 1.2'
String(1.23, "%1.3f") = '1.230'
String(24, "%05.0f") = '00024'


EDIT: Above, # always print the decimal point does not print decimal point if precision=0.

C printf() on wikipedia
https://en.wikipedia.org/wiki/Printf_format_string#Precision_field

Width field

The Width field specifies a minimum number of characters to output, and is typically used to pad fixed-width fields in tabulated output, where the fields would otherwise be smaller, although it does not cause truncation of oversized fields.

The width field may be omitted, or a numeric integer value, or a dynamic value when passed as another argument when indicated by an asterisk *. For example, printf("%*d", 5, 10) will result in 10 being printed, with a total width of 5 characters.

Though not part of the width field, a leading zero is interpreted as the zero-padding flag mentioned above, and a negative value is treated as the positive value in conjunction with the left-alignment - flag also mentioned above.

Precision field

The Precision field usually specifies a maximum limit on the output, depending on the particular formatting type. For floating point numeric types, it specifies the number of digits to the right of the decimal point that the output should be rounded. For the string type, it limits the number of characters that should be output, after which the string is truncated.

The precision field may be omitted, or a numeric integer value, or a dynamic value when passed as another argument when indicated by an asterisk *. For example, printf("%.*s", 3, "abcdef") will result in abc being printed.

so eg "%7.3f" produces minimum length of 7 characters (incl '.') with 3 decimal places eg 42.998888 = ' 42.999' (SPACE makes it 7 character width).


Fmt="%7.3f" # Width field is 7, Precision field is 3
# Fmt="%07.3f" # with leading zeros
n=42.998888
S=String(n,Fmt)
BlankClip.Subtitle("'" + S + "'") # " 42.999" or "042.999" depending upon Fmt as above.


# From Visual Studio 10, Float.h
# #define FLT_DIG 6 /* Significant decimal digits of precision for single precision float */

EDIT: Use eg Fmt="%7.0f" for width=7 and 0 fractional digits (so minimum of 7 characters, '.' not printed and SPACE padded if fewer than 7 digits)

EDIT: Similar + additional formatting available in StrFmt() plugin, and RT_Stats:- RT_DebugF(), RT_String(), RT_WriteFile() and RT_Subtitle() plug fns.

EDIT: StrFmt:- https://forum.doom9.org/showthread.php?t=174649&highlight=StrFmt
Example with programmable Width and Precision using "*" flag, below Width=10, and Precision=3 (and print leading zeros).

colorbars.killaudio
S = StrFmt("'%0*.*f'",10,3,9999.99) # "%0" = Pad to width with leading zeros instead of SPACE's
Subtitle(S) # print '009999.990', ie total length of digits and '.' is 10, with 3 fractional digits.