Decimal values incorrectly rounded down instead of up

The product of a decimal equation is incorrectly rounded down when using the DECIMALS option in the definition as well as using the ROUND function to reduce the product's number of decimals.

Jan 6, 2023
Knowledge

Title

URL Name

decimal-values-incorrectly-rounded-down-instead-of-up

Article Number

000227993

Environment

Product: OpenEdge Version: All Supported Versions OS: All Supported Platforms

Question/Problem Description

Explicit decimal rounding using the round function or automatic rounding applied for the DECIMALS attribute is inconsistent. This can be demonstrated using the following ABL code:

DEFINE VARIABLE dec1 AS DECIMAL FORMAT ">,>>9.999".
DEFINE VARIABLE dec2 AS DECIMAL FORMAT ">,>>9.999" DECIMALS 2.
DEFINE VARIABLE dec3 AS DECIMAL FORMAT ">,>>9.999".
DEFINE VARIABLE dec4 AS DECIMAL FORMAT ">,>>9.999".
DEFINE VARIABLE dec5 AS DECIMAL FORMAT ">,>>9.999" DECIMALS 2.
DEFINE VARIABLE dec6 AS DECIMAL FORMAT ">,>>9.999".

// does not round properly
dec1 = 165000 * 3.7 / 36000 * 105.
dec2 = 165000 * 3.7 / 36000 * 105.
dec3 = ROUND(165000 * 3.7 / 36000 * 105, 2).

// rounding works as expected
dec4 = 165000 * 3.9 / 36000 * 105.
dec5 = 165000 * 3.9 / 36000 * 105.
dec6 = ROUND(165000 * 3.9 / 36000 * 105, 2).

CURRENT-WINDOW:WIDTH = 95.
DISPLAY dec1 dec2 dec3 dec4 dec5 dec6 WITH STREAM-IO 3 DOWN WIDTH 95.

The equation (165000 * 3.7 / 36000 * 105) assigned to dec1 results in 1,780.625.
However the same equation when assigned to dec2 (which was defined with DECIMALS 2) results in 1,780.620.
The ROUND function similarly results in 1,780.620.
When changing one of the multipliers from 3.7 to 3.9, the assignments are done as expected to dec4, dec5 and dec6.

Steps to Reproduce

Clarifying Information

Error Message

Defect Number

Enhancement Number

Cause

This is expected behavior.

Let's start with the equation: (165000 * 3.7 / 36000 * 105). If you use a calculator that shows greater than 3 decimals, or anything with the number of decimals longer than 3 you can see that it actually resolves to the following value:

1780.6249999965

When this value is assigned to a variable (like dec1) with 3 decimal places in its format it is automatically rounded up to 1780.625.

dec2 is defined using DECIMALS 2, so when 1780.6249999965 is assigned to it, the value is rounded down to 1780.620 because the part which follows the first two decimals (49999965) is less than 50000000.

Similarly dec3 is rounded down to two decimals which produces the same value as seen in dec2.

The reason dec4-6 appear to be correct is because the product of the elevated multiplier (3.9) is 1876.875 so there is no question of needing to round down.

Resolution

None at this time.

Workaround

Notes

Keyword Phrase

Last Modified Date

1/6/2023 11:53 PM

Decimal values incorrectly rounded down instead of up

Files(0)

Was this article helpful?

Was this article helpful?

Decimal values incorrectly rounded down instead of up

Information

Related Articles