Precision summary

A precision profile is a collection of measurement unit categories and the level of precision associated with each unit category.

For example, temperature is the unit category, and decimal place is the level of precision.

Note: The number of decimal points is for view and report only and does not affect the precision of system storage or calculations.

The highest floating point precision supported by the system is an IEEE 8 byte double precision value. For reporting and displaying purposes, each category of measurement unit has an associated number of decimal places for rounding.

You can modify the data precision for your display. These options affect neither internal data, nor your other data representation.

For calculations where rounding is necessary, the system rounds at predetermined points. The following table lists the different types of calculations and the rounding rules.