Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » EGL Development Tools » mathLIb.round(Inconsistent with failures?)
mathLIb.round [message #896467] Wed, 18 July 2012 13:15 Go to next message
Nathan Reed is currently offline Nathan ReedFriend
Messages: 74
Registered: June 2012
Member
Windows XP
EDT 0.8.0 (fresh install)
RichUI

I have some RichUI code ported from RBD that had the round function. I had to change the code to use mathLib.round in EDT. I started getting error crrui2074e.

I built a small RichUI part and performed the tests below. Note when the fields are defined as float the negative rounding factor does not throw an error. The float defined fields with a negative rounding factor appear to work as I would expect.

Also note that debug displays the decimal result field after the round as if it is a floating decimal just as i show below (the .0000 are not displayed).

Is this a problem or do I not understand how this function is supposed to be coded?

workField decimal(9,4) = 37272.9502
newField decimal(9,4) = mathLib.round (workField, 1)

results in 37270 (should be 37273.0000 ?)

workField decimal(9,4) = 37278.9502
newField decimal(9,4) = mathLib.round (workField, 1)

results in 37270 (should be 37279.0000 ?)

workField decimal(9,4) = 37272.9502
newField decimal(9,4) = mathLib.round (workField, 2)

results in 37200 (should be 37270.0000 ?)

workField decimal(9,4) = 37272.9502
newField decimal(9,4) = mathLib.round (workField, -1)

results in error crrui2074e (should be 37273.0000 ?)

workField float = 37272.9502
newField float = mathLib.round (workField, 2)

results in 37200 (should be 37270 ?)

workField float = 37272.9502
newField float = mathLib.round (workField, -1)

results in 37278.7 (should be 37278.7)

Re: mathLIb.round [message #896587 is a reply to message #896467] Thu, 19 July 2012 02:35 Go to previous message
fahua jin is currently offline fahua jinFriend
Messages: 58
Registered: July 2011
Member
Nathan,

We did fix several problem of round function in EDT 0.81. I attached a sample in blow for testing the function you mentioned. With the 0.81 build, the result is the same as in RBD. However, it still have one problem - when we print the value by syslib, the zeros of right part of the decimal point is not print (we already have a bug opened for this - https://bugs.eclipse.org/bugs/show_bug.cgi?id=360920).

BTW, you can open the Eclipse help by "Help > Help contents > type 'mathlib' in the search box", and take a look at the description message of round function. Thanks.

Rocky
  • Attachment: H1.egl
    (Size: 1.50KB, Downloaded 285 times)
Previous Topic:MySQL table access
Next Topic:SQLException
Goto Forum:
  


Current Time: Fri Mar 29 10:09:25 GMT 2024

Powered by FUDForum. Page generated in 0.04631 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top