Converting Float to packed
0 pts.
0
Q:
Converting Float to packed
RPG
I am trying to convert a Float8 to a 17 6 Packed and I am loosing 1/100000th of the value of the decimal portion. The packed result is 1/100000th less than the Float value. How do I stop this from happening? I am calculating taxes.
ASKED: Jul 18 2005  2:36 PM GMT
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
0
0 pts.
0
A:
 RATE THIS ANSWER
0
Click to Vote:
  •   0
  •  0
  • AddThis Social Bookmark Button
How are you attempting the conversion? Are you using the MOVE opcode, or the %dec BIF? Could this be a truncation matter ... perhaps all you need is an eval(h)?

A code snippet would help diagnose what's happening.

Good luck,
Adam
Last Answered: Jul 19 2005  9:33 AM GMT by aglauser   0 pts.
0
0
Discuss This Answer:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _



_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

RolandT   0 pts.  |   Jul 26 2005  9:25AM GMT

I had the same problem but going in the other direction. I solved it with the following:

Eval hfloat = decfield * 1.000005

I had to experiment to fing the right number of zeros, but it finally worked.

 

TomLiotta   8025 pts.  |   Oct 29 2009  6:44AM GMT

You don’t “solve” it. Binary representations of fractional decimal values often are only approximations because that’s how fractional values work when converting from one number base to another with a fixed number of digits.

Simply test it by manually converting .03 decimal to binary and back again. Try with 1-byte, 2-byte, 4-byte and 8-byte binary variables. There will always be a loss of same small fraction. That is exactly why formats such as packed-”decimal” and binary-coded-decimal were invented in the first place. A value such as .03 cannot be accurately represented in binary form with an infinite number of bytes.

Don’t use binary data types (e.g., floating point) for fractional decimal calculations.

Alternatively, don’t use fractional values — always convert to integers. For example, don’t store dollars as packed (9,2). Instead, store as pennies with packed (9,0). The same digits are used, but there are no fractional portions. Do it all with integer math.

Of course, when you do that, you introduce multiplication by and division by 100, which tends to offset any advantage of using floating-point in the first place.

Tom

 

TomLiotta   8025 pts.  |   Oct 29 2009  6:45AM GMT

Typo — “without an infinite number of bytes…”

Tom

 
0