# Precision in Excel

by Allen Wyatt
(last updated August 24, 2013)

When we were in junior high school, each of us learned the meaning of precision from the standpoint of math: it defined a degree of accuracy of a number. Thus, we often saw what at first appeared to be confusing information. Ms. Jameson, the math teacher, would tell us that 4.76 was different than 4.760. The reason, it turns out, is that 4.76 is precise (accurate) to only two decimal places, while 4.760 is precise (accurate) to three decimal places. Thus, 4.760 is more precise than 4.76 because of the number of significant digits each contains. Enter the world of digital calculators and (by extension) spreadsheets, such as Excel. To them, there is no difference between 4.76 and 4.760. Why, you might ask, is this true, particularly when Ms. Jameson made such a big deal about the issue? The reason is because of the way in which calculators and spreadsheets store numbers and the way in which humans store numbers. Humans generally store numbers (in our brains) as they are written. This means that we look at 4.76, and we know it is accurate to only two decimal places because that is all we can see and write. Computer chips, on the other hand, deal in quite a different plane of precision. When you enter a number into Excel, it automatically converts the number to fifteen digits of precision—regardless of what you intended. Thus, 4.76 is automatically padded with zeros at the end out to fifteen digits, even if you thought you were only specifying two decimal places of precision. When you put in the number 4.760, expecting three decimal places of precision, Excel still pads it out to the fifteen places. It is impossible to instruct Excel to pay attention to two decimal places in one instance and three in the other. Humans can do it, but Excel cannot. Note that in the previous paragraph I tried to make a distinction between "digits of precision" and "decimal places of precision." This may sound like a fine hair to split, and some mathematicians may try to brand me a heretic. I make the distinction because many people automatically assume that precision has to do with the number of digits to the right of the decimal point. In Excel, however, precision simply has to do with the number of digits—period. Thus, the fifteen digits of precision maintained by Excel counts digits both before and after the decimal point in the total. If there are four digits to the left of the decimal point, then Excel will only maintain eleven to the right. If there are only two to the left, then Excel maintains thirteen to the right. What does all this mean to the average Excel user? Not a whole lot, other than you may need to look at numbers a little differently than Ms. Jameson did. It doesn't hurt to remember how Excel maintains numbers internally, however, as it can affect the calculations you perform.

ExcelTips is your source for cost-effective Microsoft Excel training. This tip (1981) applies to Microsoft Excel 97, 2000, 2002, and 2003.

##### Author Bio

Allen Wyatt

With more than 50 non-fiction books and numerous magazine articles to his credit, Allen Wyatt is an internationally recognized author. He  is president of Sharon Parq Associates, a computer and publishing services company. ...

##### MORE FROM ALLEN

Determining If a Number is Odd or Even

If you need to know whether a particular value is odd or even, you can use this simple formula. Designed to be used in a ...

Discover More

Slowing Down Mouse Selection

Ever tried to select a range of cells using the mouse, only to have the cells scroll by so quickly you can't make the ...

Discover More

Creating a Custom Power Management Plan

Don't like the built-in power management plans that Windows provides? You can create your own to better fit your power needs.

Discover More

Save Time and Supercharge Excel! Automate virtually any routine task and save yourself hours, days, maybe even weeks. Then, learn how to make Excel do things you thought were simply impossible! Mastering advanced Excel macros has never been easier. Check out Excel 2010 VBA and Macros today!

Precision and Number Display

Do you need to work with very precise numbers in your worksheets? If you do, then it is imperative you understand ...

Discover More

Changing the Color of Worksheet Gridlines

Want the gridlines in your worksheet to be a different color? You aren't limited to stodgy black; Excel lets you make ...

Discover More

Getting Rid of the Startup Spreadsheet

When you start Excel, you normally see a blank worksheet displayed. Here's how to get rid of it.

Discover More
##### Subscribe

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

If you would like to add an image to your comment (not an avatar, but an image to help in making the point of your comment), include the characters [{fig}] in your comment text. Youâ€™ll be prompted to upload your image when you submit the comment. Maximum image size is 6Mpixels. Images larger than 600px wide or 1000px tall will be reduced. Up to three images may be included in a comment. All images are subject to review. Commenting privileges may be curtailed if inappropriate images are posted.

What is 4 + 4?

There are currently no comments for this tip. (Be the first to leave your commentâ€”just use the simple form above!)

##### This Site

Got a version of Excel that uses the menu interface (Excel 97, Excel 2000, Excel 2002, or Excel 2003)? This site is for you! If you use a later version of Excel, visit our ExcelTips site focusing on the ribbon interface.