Precision in Excel

Written by Allen Wyatt (last updated January 23, 2021)
This tip applies to Excel 97, 2000, 2002, and 2003

When we were in junior high school, each of us learned the meaning of precision from the standpoint of math: it defined a degree of accuracy of a number. Thus, we often saw what at first appeared to be confusing information. Ms. Jameson, the math teacher, would tell us that 4.76 was different than 4.760. The reason, it turns out, is that 4.76 is precise (accurate) to only two decimal places, while 4.760 is precise (accurate) to three decimal places. Thus, 4.760 is more precise than 4.76 because of the number of significant digits each contains. Enter the world of digital calculators and (by extension) spreadsheets, such as Excel. To them, there is no difference between 4.76 and 4.760. Why, you might ask, is this true, particularly when Ms. Jameson made such a big deal about the issue? The reason is because of the way in which calculators and spreadsheets store numbers and the way in which humans store numbers. Humans generally store numbers (in our brains) as they are written. This means that we look at 4.76, and we know it is accurate to only two decimal places because that is all we can see and write. Computer chips, on the other hand, deal in quite a different plane of precision. When you enter a number into Excel, it automatically converts the number to fifteen digits of precision—regardless of what you intended. Thus, 4.76 is automatically padded with zeros at the end out to fifteen digits, even if you thought you were only specifying two decimal places of precision. When you put in the number 4.760, expecting three decimal places of precision, Excel still pads it out to the fifteen places. It is impossible to instruct Excel to pay attention to two decimal places in one instance and three in the other. Humans can do it, but Excel cannot. Note that in the previous paragraph I tried to make a distinction between "digits of precision" and "decimal places of precision." This may sound like a fine hair to split, and some mathematicians may try to brand me a heretic. I make the distinction because many people automatically assume that precision has to do with the number of digits to the right of the decimal point. In Excel, however, precision simply has to do with the number of digits—period. Thus, the fifteen digits of precision maintained by Excel counts digits both before and after the decimal point in the total. If there are four digits to the left of the decimal point, then Excel will only maintain eleven to the right. If there are only two to the left, then Excel maintains thirteen to the right. What does all this mean to the average Excel user? Not a whole lot, other than you may need to look at numbers a little differently than Ms. Jameson did. It doesn't hurt to remember how Excel maintains numbers internally, however, as it can affect the calculations you perform.

ExcelTips is your source for cost-effective Microsoft Excel training. This tip (1981) applies to Microsoft Excel 97, 2000, 2002, and 2003.

Author Bio

Allen Wyatt

With more than 50 non-fiction books and numerous magazine articles to his credit, Allen Wyatt is an internationally recognized author. He is president of Sharon Parq Associates, a computer and publishing services company. ...


Specifying Print Quantity in a Cell

When you print a worksheet, you can specify in the Print dialog box how many copies you want printed. If you want the ...

Discover More

Sticking with the Dashes

By default, dashes don't "stick" to the text that follows them. Here's one way around this normal formatting convention.

Discover More

Making a Drop-Down List Indicator Visible

Data validation allows you to create drop-down lists that aid data entry. If you want the indicator for the drop-down ...

Discover More

Create Custom Apps with VBA! Discover how to extend the capabilities of Office 2013 (Word, Excel, PowerPoint, Outlook, and Access) with VBA programming, using it for writing macros, automating Office applications, and creating custom applications. Check out Mastering VBA for Office 2013 today!

More ExcelTips (menu)

Getting Rid of the Startup Spreadsheet

When you start Excel, you normally see a blank worksheet displayed. Here's how to get rid of it.

Discover More

Changing the Color of Worksheet Gridlines

Want the gridlines in your worksheet to be a different color? You aren't limited to stodgy black; Excel lets you make ...

Discover More

Precision and Number Display

Do you need to work with very precise numbers in your worksheets? If you do, then it is imperative you understand ...

Discover More

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

View most recent newsletter.


If you would like to add an image to your comment (not an avatar, but an image to help in making the point of your comment), include the characters [{fig}] (all 7 characters, in the sequence shown) in your comment text. You’ll be prompted to upload your image when you submit the comment. Maximum image size is 6Mpixels. Images larger than 600px wide or 1000px tall will be reduced. Up to three images may be included in a comment. All images are subject to review. Commenting privileges may be curtailed if inappropriate images are posted.

What is 6 + 3?

There are currently no comments for this tip. (Be the first to leave your comment—just use the simple form above!)

This Site

Got a version of Excel that uses the menu interface (Excel 97, Excel 2000, Excel 2002, or Excel 2003)? This site is for you! If you use a later version of Excel, visit our ExcelTips site focusing on the ribbon interface.

Newest Tips

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

(Your e-mail address is not shared with anyone, ever.)

View the most recent newsletter.