Numeric conversion limitation?

[agraham]

I'm having trouble displaying large numbers using the built in data types. Decimals and int64 both appear to be limited to 15 bit decimal precision when converted to strings. It looks suspiciously like the internal numeric conversions are limited to this precision. Is this so?

The decimal.dll components (which are a wrapper to system.decimal) work just as I expect, as they have their own string conversion routine, except that I can't put them in arrays.

 

[Erel]

By design, Basic4ppc uses 16 bit for floating point numbers.
You should the use Decimal library for higher precision (it uses 128 bits).
You could create a pseudo array using AddObject and Control keyword.
Something like:

for i = 0 to 49
 AddObject("dec" & i, "Decimal")
 Control("dec" & i).New1
Next

 

[agraham]

By design, Basic4ppc uses 16 bit for floating point numbers.

Sorry Erel but I don't understand your reply. I am obviously getting far higher precision than 16 bit floating point on Decimals - stated to be 96 bits in the help and Int64s - stated to be 64 bits in your help. And I suspect that internally the arithematic is correct but when I display them in a text box or convert them to a string using Format then both are limited to 15 decimal places of precision - as though the string conversion routines cannot cope with more places.

For example the highest number I can get an Int64 or Decimal to display correctly is 999999999999999 (15 places if I've type correctly) and an Int64 isn't a floating point type (nor I thought were Decimals - they may have exponents but should not be susceptible to the rounding problems that afflict floating point types)

 

[Erel]

Sorry but it is of course not 16 bits. Basic4ppc uses 64 bit for all numbers (System.Double).
This is enough for most applications.
If you need higher precision you could use the Decimal library (which uses 128 bits).
This example will show 1/7 with 25 decimal digits.
Don't forget to add the Decimal library.

[SIZE=2][COLOR=#008000]'Add a DecOperator named dec.
[/COLOR][/SIZE][SIZE=2][COLOR=#0000ff]Sub [/COLOR][/SIZE][SIZE=2]Globals
[/SIZE][SIZE=2][COLOR=#0000ff]End Sub
[/COLOR][/SIZE][SIZE=2][COLOR=#0000ff] 
Sub [/COLOR][/SIZE][SIZE=2]App_Start
 dec.New1
[/SIZE][SIZE=2][COLOR=#0000ff] AddObject[/COLOR][/SIZE][SIZE=2]("d1","decNumber")
[/SIZE][SIZE=2][COLOR=#0000ff] AddObject[/COLOR][/SIZE][SIZE=2]("d2","decNumber")
 d1.New1
 d2.New1
 d1.Value = dec.ParseD("1")
 d2.Value = dec.FromDouble(7)
 d1.Value = dec.DivD(d1.Value,d2.Value)
 d1.Value = dec.AddD(d1.Value, d2.Value)
[/SIZE][SIZE=2][COLOR=#0000ff] msgbox[/COLOR][/SIZE][SIZE=2](d1.ToString2("n25"))
[/SIZE][SIZE=2][COLOR=#0000ff]End Sub
[/COLOR][/SIZE]

 

[agraham]

Hi Erel

I was composing the following when your second reply came in. I understand what is happening but I think that your help needs clarification.

Reply was going to be
---------------------

I think there is a misunderstanding here (and a typo in your reply above).

From help on Variables" - "All simple variables are variant variables. Which means that any variable can store any type of number or a string"

My emphasis on "any"

I took this to mean that they were Visual Basic type variants and could take the datatypes specified in help on "Data types"

Name Description Range
· Byte 8-bit unsigned integer 0 - 255
· Int16 16-bit signed integer -32768 - 32767
etc.


However poking around with Reflector it seems that you have only two types internally, strings and doubles (i.e 64 bit floating point and not 16 bit as you replied). Even the arrays of the other datatypes appear to be arrays of double. So I assume that the other data types are produced solely for interface purposes and are never visible internally, which raises the question of what happens if an external library returns a number out of range of a double!

 

[agraham]

If you need higher precision you could use the Decimal library (which uses 128 bits)

Yes I 've already played with that and found it satisfactory