| mathLIb.round [message #896467] |
Wed, 18 July 2012 13:15  |
 Nathan Reed
Messages: 74
Registered: June 2012 |
Member |
|
|
Windows XP
EDT 0.8.0 (fresh install)
RichUI
I have some RichUI code ported from RBD that had the round function. I had to change the code to use mathLib.round in EDT. I started getting error crrui2074e.
I built a small RichUI part and performed the tests below. Note when the fields are defined as float the negative rounding factor does not throw an error. The float defined fields with a negative rounding factor appear to work as I would expect.
Also note that debug displays the decimal result field after the round as if it is a floating decimal just as i show below (the .0000 are not displayed).
Is this a problem or do I not understand how this function is supposed to be coded?
workField decimal(9,4) = 37272.9502
newField decimal(9,4) = mathLib.round (workField, 1)
results in 37270 (should be 37273.0000 ?)
workField decimal(9,4) = 37278.9502
newField decimal(9,4) = mathLib.round (workField, 1)
results in 37270 (should be 37279.0000 ?)
workField decimal(9,4) = 37272.9502
newField decimal(9,4) = mathLib.round (workField, 2)
results in 37200 (should be 37270.0000 ?)
workField decimal(9,4) = 37272.9502
newField decimal(9,4) = mathLib.round (workField, -1)
results in error crrui2074e (should be 37273.0000 ?)
workField float = 37272.9502
newField float = mathLib.round (workField, 2)
results in 37200 (should be 37270 ?)
workField float = 37272.9502
newField float = mathLib.round (workField, -1)
results in 37278.7 (should be 37278.7)
|
|
|
|
Powered by
FUDForum. Page generated in 0.04533 seconds
] 
Search
Help
Register
Login
Home
