real number decimal precision

I am working with a lot of small probability numbers, which are stored as real variables. It seems that DOORS will only display up to 6 digits of precision past the decimal point. Is there a way for me to extend this to many more digits of precision?
SystemAdmin - Thu Apr 09 14:39:13 EDT 2009

Re: real number decimal precision
tommy3824 - Thu Apr 09 16:03:05 EDT 2009

I don't know what you are doing exactly with these numbers but if you can use strings, you can just move more digits to the left side of the decimal then physically place the period. It is set up to display percentages and is most percise for numbers below 100%.
  

real a = 12345
real b = 67890
real c = 100000000000.0
real d = (a*c)/b
string e = d""
 
int i
int j
int k
 
for(i=0;i<(length e);i++)
{
    if(e[i]"" == ".")
                break
}
 
j = length e[(i+1):] + 11 // Number of 0's in c + 1
k = length e - j
e = e[0:k]"."e[(k+1):(i-1)]""e[(i+1):]"%"
 
 
print e

Re: real number decimal precision
dpechacek - Thu Apr 09 16:35:08 EDT 2009

I don't believe so unless you try to print it as a string instead of a real. But there isn't an option I know of to change decimal precision in DOORS.

AAI Services, Textron
dpechacek@sc-aaicorp.com
David.Pechacek@gmail.com

Re: real number decimal precision
llandale - Wed Apr 22 13:56:40 EDT 2009

Well, if I knew it would take this long I never would have done it. Attached find my new library function 'fStringOfReal()' that does what you want, along with the drivers that prove it. Take the DXL printout and paste into Excel, first changing the cell format to 'text', and making the cells right-justified.

Golly, its ugly but it works. There may be significant string-table waste here, but I doubt you will convert 100s of 1000s of real numbers.

The '17' calibration is accurate but hokey, I cannot think of another way to insure that the printed number does not have any more precision than the original real number. The '50' parameter is arbitrary and hokey also. That seems like a reasonable limit on the number of whole or decimal digits in the final string.

Someone more familiar with 'real' numbers and especially how they are typically implemented on computers may want to provide a better mechanism for doing this.

Not seen in this set of outputs but in earlier versions, was awkward rounding errors introduced by the function. In one case I had "1.113" converted to "1.1129999999999999".

What's next, I suppose, is figuring out how to convert a real number to scientific format, "1234e-3".

>Louie
Attachments

attachment_14243552_Driver_fStringOfReal.dxl

Re: real number decimal precision
Ron_Lewis - Tue Apr 28 15:30:55 EDT 2009

llandale - Wed Apr 22 13:56:40 EDT 2009
Well, if I knew it would take this long I never would have done it. Attached find my new library function 'fStringOfReal()' that does what you want, along with the drivers that prove it. Take the DXL printout and paste into Excel, first changing the cell format to 'text', and making the cells right-justified.

Golly, its ugly but it works. There may be significant string-table waste here, but I doubt you will convert 100s of 1000s of real numbers.

The '17' calibration is accurate but hokey, I cannot think of another way to insure that the printed number does not have any more precision than the original real number. The '50' parameter is arbitrary and hokey also. That seems like a reasonable limit on the number of whole or decimal digits in the final string.

Someone more familiar with 'real' numbers and especially how they are typically implemented on computers may want to provide a better mechanism for doing this.

Not seen in this set of outputs but in earlier versions, was awkward rounding errors introduced by the function. In one case I had "1.113" converted to "1.1129999999999999".

What's next, I suppose, is figuring out how to convert a real number to scientific format, "1234e-3".

>Louie

If you want more percision you can use ole automation to use the precision of:
--excel
or
--vba

Re: real number decimal precision
SystemAdmin - Mon May 04 17:05:33 EDT 2009

That is really disappointing that there is not an easy way to extend the precision of real numbers. Thanks for giving this one so much effort.

Re: real number decimal precision
llandale - Mon May 04 19:33:56 EDT 2009

SystemAdmin - Mon May 04 17:05:33 EDT 2009
That is really disappointing that there is not an easy way to extend the precision of real numbers. Thanks for giving this one so much effort.

My solution doesn't extend their precision, but it converts to a string with far more decimal places than the standard 6.

I suppose another way to do this would be to multiply the number by some huge power of 10, convert to a string, then manipulate the string to put the decimal point in the right place.

>Louie

Re: real number decimal precision
jester76 - Sat Oct 05 23:06:41 EDT 2013

llandale - Wed Apr 22 13:56:40 EDT 2009
Well, if I knew it would take this long I never would have done it. Attached find my new library function 'fStringOfReal()' that does what you want, along with the drivers that prove it. Take the DXL printout and paste into Excel, first changing the cell format to 'text', and making the cells right-justified.

Golly, its ugly but it works. There may be significant string-table waste here, but I doubt you will convert 100s of 1000s of real numbers.

The '17' calibration is accurate but hokey, I cannot think of another way to insure that the printed number does not have any more precision than the original real number. The '50' parameter is arbitrary and hokey also. That seems like a reasonable limit on the number of whole or decimal digits in the final string.

Someone more familiar with 'real' numbers and especially how they are typically implemented on computers may want to provide a better mechanism for doing this.

Not seen in this set of outputs but in earlier versions, was awkward rounding errors introduced by the function. In one case I had "1.113" converted to "1.1129999999999999".

What's next, I suppose, is figuring out how to convert a real number to scientific format, "1234e-3".

>Louie

real rInput = (1.0) * (1.00000000000000014)


print fStringOfReal(rInput,  16, false)

This does not work properly. it displays 1.0000000000000002

Not sure why

Re: real number decimal precision
Adamarla - Mon Oct 07 04:33:40 EDT 2013

jester76 - Sat Oct 05 23:06:41 EDT 2013

real rInput = (1.0) * (1.00000000000000014)


print fStringOfReal(rInput,  16, false)

This does not work properly. it displays 1.0000000000000002

Not sure why

It is not a trivial problem to convert real to a string.

I posted my real formatting library code Here.

Re: real number decimal precision
jester76 - Mon Oct 07 09:12:50 EDT 2013

Adamarla - Mon Oct 07 04:33:40 EDT 2013

It is not a trivial problem to convert real to a string.

I posted my real formatting library code Here.

Hello,

How do I increase the precision to 16 dgits on your script?

thanks

Re: real number decimal precision
Adamarla - Mon Oct 07 10:27:53 EDT 2013

jester76 - Mon Oct 07 09:12:50 EDT 2013

Hello,

How do I increase the precision to 16 dgits on your script?

thanks

It already converts to the maximum precision.

My guess is you are suffering loss due to your calculations.

Real number format does NOT store exact representations of all numbers.

Can you post an example of what you want vs what you get?

-Adam

Re: real number decimal precision
jester76 - Mon Oct 07 11:25:46 EDT 2013

Adamarla - Mon Oct 07 04:33:40 EDT 2013

It is not a trivial problem to convert real to a string.

I posted my real formatting library code Here.

Here is an example of what I am dealing with.

real rInput = (1.0) * (1.00000000000000014)


print Real_ToString(rInput,  false)

This does not work properly. it displays 1.0000000000000002

I should be getting 1.00000000000000014

Thanks for your help

Re: real number decimal precision
jester76 - Mon Oct 07 11:27:48 EDT 2013

Adamarla - Mon Oct 07 10:27:53 EDT 2013

It already converts to the maximum precision.

My guess is you are suffering loss due to your calculations.

Real number format does NOT store exact representations of all numbers.

Can you post an example of what you want vs what you get?

-Adam

real rInput = (1.0) * (1.00000000000000014)


print fStringOfReal(rInput,  16, false)

This does not work properly. it displays 1.0000000000000002

while I should be getting 1.0000000000000001

 

thanks for your help

Re: real number decimal precision
Adamarla - Mon Oct 07 12:59:54 EDT 2013

jester76 - Mon Oct 07 11:27:48 EDT 2013

real rInput = (1.0) * (1.00000000000000014)


print fStringOfReal(rInput,  16, false)

This does not work properly. it displays 1.0000000000000002

while I should be getting 1.0000000000000001

 

thanks for your help

The real format does not support the level of precision you want.

 

print(Real_ToString(1.0000000000000001) "\n")

print(Real_ToString(1.00000000000000014) "\n")

 

1

1.0000000000000002

1.0000000000000001 is stored as 1 and therefore retrieved as 1.

1.00000000000000014 is stored as 1.0000000000000002 and therefore retrieved as 1.0000000000000002.

Calculations may result in precision loss.

Read up on the floating point binary64 (Double precision) format to get a better understanding of what is possible.

-Adam

 

Re: real number decimal precision
llandale - Mon Oct 07 15:46:38 EDT 2013

jester76 - Mon Oct 07 11:25:46 EDT 2013

Here is an example of what I am dealing with.

real rInput = (1.0) * (1.00000000000000014)


print Real_ToString(rInput,  false)

This does not work properly. it displays 1.0000000000000002

I should be getting 1.00000000000000014

Thanks for your help

Your number is stressing the limits of real precision.  Not sure any function can deal with that.

Re: real number decimal precision
jester76 - Tue Oct 08 13:38:21 EDT 2013

Adamarla - Mon Oct 07 12:59:54 EDT 2013

The real format does not support the level of precision you want.

 

print(Real_ToString(1.0000000000000001) "\n")

print(Real_ToString(1.00000000000000014) "\n")

 

1

1.0000000000000002

1.0000000000000001 is stored as 1 and therefore retrieved as 1.

1.00000000000000014 is stored as 1.0000000000000002 and therefore retrieved as 1.0000000000000002.

Calculations may result in precision loss.

Read up on the floating point binary64 (Double precision) format to get a better understanding of what is possible.

-Adam

 

Adam,

 

The article says it can do up 17 digits. Am I missing anything?

 

-Jester

Re: real number decimal precision
Adamarla - Wed Oct 09 04:32:27 EDT 2013

jester76 - Tue Oct 08 13:38:21 EDT 2013

Adam,

 

The article says it can do up 17 digits. Am I missing anything?

 

-Jester

yes.

the total precision is therefore 53 bits (approximately 16 decimal digits

  

1.0000000000000002, the smallest number > 1