I am trying to convert a Float8 to a 17 6 Packed and I am loosing 1/100000th of the value of the decimal portion. The packed result is 1/100000th less than the Float value. How do I stop this from happening? I am calculating taxes.
I had the same problem but going in the other direction. I solved it with the following:
Eval hfloat = decfield * 1.000005
I had to experiment to fing the right number of zeros, but it finally worked.
You don't "solve" it. Binary representations of fractional decimal values often are only approximations because that's how fractional values work when converting from one number base to another with a fixed number of digits.
Simply test it by manually converting .03 decimal to binary and back again. Try with 1-byte, 2-byte, 4-byte and 8-byte binary variables. There will always be a loss of same small fraction. That is exactly why formats such as packed-"decimal" and binary-coded-decimal were invented in the first place. A value such as .03 cannot be accurately represented in binary form with an infinite number of bytes.
Don't use binary data types (e.g., floating point) for fractional decimal calculations.
Alternatively, don't use fractional values -- always convert to integers. For example, don't store dollars as packed (9,2). Instead, store as pennies with packed (9,0). The same digits are used, but there are no fractional portions. Do it all with integer math.
Of course, when you do that, you introduce multiplication by and division by 100, which tends to offset any advantage of using floating-point in the first place.
Tom
I understand the lack of precision using floating point numbers, but why does an 8F number = 1.929000000000E+001convert to 15P5 value of 19.28999? My instruction is 15P5 = 8F.
The data comes to me in an Excel spreadsheet. I have elected to use Scott Klement's XLPARSER4 service program to extract the data, so I am stuck with dealing with floating point numbers.
You say the float field is already messed up. Why does 1.92900000000.E+001 come out as anything but 19.29? I know it does, because a 15P5 field shows it as 19.28999, and a 12P2 field shows it as 19.28.
The Excel column in question is an item cost field with two decimal places. Can I be sure of getting the same number is I use 'eval (h) 15P5 = 8F'?
Assuming that you can't get the XLPARSER4 to pull the field down as text ..
There is something about Half adjust EVAL(H) .. I will have to think on this... got it .. there has to be one more digit (6) in the float then the 15P5
Free Guide: Managing storage for virtual environments
Complete a brief survey to get a complimentary 70-page whitepaper featuring the best methods and solutions for your virtual environment, as well as hypervisor-specific management advice from TechTarget experts. Donâ€™t miss out on this exclusive content!
No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.
Your password has been sent to:cguarrera@techtarget.com
To follow this tag...
Thanks! We'll email you when relevant content is added and updated.
Following
Share this item with your network:
X
We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.
Discuss This Question: 8 Replies
When I use SQL to insert data into FLD05F,
INSERT INTO DECTEST VALUES(1.9290000E+002)
the floating point field contains data that can be off by one digit.
It's not the conversion to 15P5, it's already messed up in the float field.
FLD05F
19290000.E-001
19290000.E-002
19290000.E-003
19290000.E-004
19289999.E-005
19290001.E-006
19290000.E-007
19290000.E-008
19290000.E-003
19290000.E-007
19289999.E-005
19289999.E-005
You say the float field is already messed up. Why does 1.92900000000.E+001 come out as anything but 19.29? I know it does, because a 15P5 field shows it as 19.28999, and a 12P2 field shows it as 19.28.
The Excel column in question is an item cost field with two decimal places. Can I be sure of getting the same number is I use 'eval (h) 15P5 = 8F'?
Assuming that you can't get the XLPARSER4 to pull the field down as text ..
There is something about Half adjust EVAL(H) .. I will have to think on this... got it .. there has to be one more digit (6) in the float then the 15P5
so this 19289999.E-005 goes to 192.89999
works for 19290001.E-006 which goes to 19.29000