Android Question Problem with Float data type

toby

Well-Known Member
Licensed User
Longtime User
I want to store latitude and longitude data as Float data type values. When I moved mouse over flt1 variable, it showed 123.45679 while the Log() yielded another slightly different value.

Could someone tell me what's wrong, please?

Problem with float data type:
Dim flt1 As Float=123.4567890123456
    'hoveover value: 123.45679 (rounded with 5 decimal digits)
    Log(flt1)      'output: 123.456787109375

TIA
 

DonManfred

Expert
Licensed User
Longtime User
Use Numberformat to format a number with any amount of chars you need.
 
Upvote 0

emexes

Expert
Licensed User
A four-byte single-precision float has a 24 bit mantissa = accurate to within 0.85 metres worst case (at equator, opposite prime meridian ie Greenwich London)

An eight-byte double-precision float has an extra 29 bits of mantissa = accurate to within 1.6 nanometres worst case (in Australia aka within a bee's dick)
 
Upvote 0
Top