Limiting Precision

by Allen Wyatt
(last updated April 25, 2015)

3

There may be times when you want to limit the precision used in computations by Excel. This may come in particularly handy when you are working with set-precision numbers, such as currency amounts. As you learned in other issues of ExcelTips, Excel maintains fifteen digits of internal precision, but that may not be how you have your information displayed.

This dichotomy can lead to some interesting results. For instance, let's say that you have a number stored in Excel, which is 1.44. You want to figure out what 110% of that number is, and then double it. You put the original number (1.44) in cell A1. In cell A2 you place the formula =1.1*A1. In cell A3 you place the formula =2*A2. You then format each of the three cells for Currency formatting. At this point, A1 will contain $1.44, A2 will contain $1.58, and A3 will contain $3.17.

At first glance, it would appear that the result in A3 is wrong. After all, two times $1.58 is $3.16, not $3.17. The difference is due to how rounding is done and how Excel performs internal calculations. It turns out that the value in cell A2 is not really $1.58, but 1.584, and the value in cell A3 is then 3.168. When rounded, it becomes 3.17.

If you want to get around this problem, you can do so by changing how Excel handles precision. Follow these steps:

  1. Choose Options from the Tools menu. You will see the Options dialog box.
  2. Make sure the Calculation tab is selected. (See Figure 1.)
  3. Figure 1. The Calculation tab of the Options dialog box.

  4. Ensure that the Precision As Displayed check box is selected.
  5. Click on OK.

When you do this, Excel only performs calculations using the digits you have displayed in a cell. In the above example, it means that cell A3 now contains $3.16 because the calculation on A2 was done on only $1.58, as displayed.

You should note that when you change this precision setting on Excel, the numbers maintained by Excel are permanently changed. Thus, if you entered some numbers that had five digits to the right of the decimal, and then you displayed only two, changing the precision setting would permanently throw away three of the digits—those farthest to the right.

ExcelTips is your source for cost-effective Microsoft Excel training. This tip (2642) applies to Microsoft Excel 97, 2000, 2002, and 2003.

Author Bio

Allen Wyatt

With more than 50 non-fiction books and numerous magazine articles to his credit, Allen Wyatt is an internationally recognized author. He  is president of Sharon Parq Associates, a computer and publishing services company. ...

MORE FROM ALLEN

Changing the Height of a Font

Scaling the width of a font is easy to do with Word's formatting capabilities. Scaling the height of the fonts is not so ...

Discover More

Converting Paragraphs to Comments

Want to pull text from a bunch of paragraphs and stuff that text into comments? It's easy to do using the macro presented in ...

Discover More

Adding a Little Animation to Your Life

Tired of the same old boring Excel look? You can add some life to your worksheets by introducing some animation. Here's what ...

Discover More

Create Custom Apps with VBA! Discover how to extend the capabilities of Office 2013 (Word, Excel, PowerPoint, Outlook, and Access) with VBA programming, using it for writing macros, automating Office applications, and creating custom applications. Check out Mastering VBA for Office 2013 today!

More ExcelTips (menu)

Viewing Workbook Statistics

Excel keeps track of a range of stats about each workbook you use. If you want to take a look at those stats, it's easy; just ...

Discover More

Inserting the User's Name in a Cell

Need to understand who is using a particular workbook? There are a number of ways you can find out, as discussed in this tip.

Discover More

Disabling Excel's Help System

The Help system built into Excel can be quite a lifesaver when you need to find that quick tidbit that is slipping your mind. ...

Discover More
Subscribe

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

View most recent newsletter.

Comments

If you would like to add an image to your comment (not an avatar, but an image to help in making the point of your comment), include the characters [{fig}] in your comment text. You’ll be prompted to upload your image when you submit the comment. Maximum image size is 8Mpixels. Images larger than 600px wide or 1000px tall will be reduced. Up to three images may be included in a comment. All images are subject to review. Commenting privileges may be curtailed if inappropriate images are posted.

What is 6 - 3?

2015-04-27 07:32:28

Hank Jager

As per comment from Allen, yYou should note that when you change this precision setting on Excel, the numbers maintained by Excel are permanently changed. Thus, if you entered some numbers that had five digits to the right of the decimal, and then you displayed only two, changing the precision setting would permanently throw away three of the digits—those farthest to the right. In order to avoid that you can use the equation =Round(1.10*1.44,2) to get 1.58, and if you multiply that by 2 you get 3.16 as well.


2015-04-25 09:36:33

John Randall

The use of this Tip is great for currency transactions and many other situations.

The ROUND function allows a user to establish the number of digits to which the target cell value will be rounded. This may be advantageous when dealing with very large numbers - say billions or trillions of dollars or the number of miles to the Sun where a string of 9 or 12 numbers is very distracting and scientific notation is not a viable option. For example the US National Debt is now above $18,200,947,645,962. When subjected to Round(value,-9) this is reported as $18,201,000,000,000.

Note that the minus sign is needed to round a number to the left of the decimal place.

Variations of this function include ROUNDUP and ROUNDOWN which automatically establish the direction of rounding rather than following the standard protocol of determining that direction, i.e. up if the pivotal value is 5 or more and down if it is 4 or less.

In the example above, ROUNDUP would provide the same value as shown above. However, ROUNDDOWN would give a result of $18,200,000,000,000.


2015-04-25 09:31:43

John Randall

The use of this Tip is great for currency transactions and many other situations.

The ROUND function allows a user to establish the number of digits to which the target cell value will be rounded. This may be advantageous when dealing with very large numbers - say billions or trillions of dollars or the number of mioles to the Sun where a string of 9 or 12 numbers is very distracting and scientific notation is not a viable option. For example the US National Debt is now above $18,200,947,645,962. When subjected to Round(value,-9) this is reported as $18,201,000,000,000.

Note that the minus sign is needed to round a number to the left of the decimal place.

Variations of this function include ROUNDUP and ROUNDOWN which automatically establish the direction of rounding rather than following the standard protocol of determining that direction, i.e. up if the pivotal value is 5 or more and down if it is 4 or less.

In the example above, ROUNDUP would provide the same value as shown above. However, ROUNDDOWN would give a result of $18,200,000,000,000.


This Site

Got a version of Excel that uses the menu interface (Excel 97, Excel 2000, Excel 2002, or Excel 2003)? This site is for you! If you use a later version of Excel, visit our ExcelTips site focusing on the ribbon interface.

Newest Tips
Subscribe

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

(Your e-mail address is not shared with anyone, ever.)

View the most recent newsletter.