Excel.Tips.Net ExcelTips (Menu Interface)

Precision in Excel

When we were in junior high school, each of us learned the meaning of precision from the standpoint of math: it defined a degree of accuracy of a number. Thus, we often saw what at first appeared to be confusing information. Ms. Jameson, the math teacher, would tell us that 4.76 was different than 4.760. The reason, it turns out, is that 4.76 is precise (accurate) to only two decimal places, while 4.760 is precise (accurate) to three decimal places. Thus, 4.760 is more precise than 4.76 because of the number of significant digits each contains. Enter the world of digital calculators and (by extension) spreadsheets, such as Excel. To them, there is no difference between 4.76 and 4.760. Why, you might ask, is this true, particularly when Ms. Jameson made such a big deal about the issue? The reason is because of the way in which calculators and spreadsheets store numbers and the way in which humans store numbers. Humans generally store numbers (in our brains) as they are written. This means that we look at 4.76, and we know it is accurate to only two decimal places because that is all we can see and write. Computer chips, on the other hand, deal in quite a different plane of precision. When you enter a number into Excel, it automatically converts the number to fifteen digits of precision—regardless of what you intended. Thus, 4.76 is automatically padded with zeros at the end out to fifteen digits, even if you thought you were only specifying two decimal places of precision. When you put in the number 4.760, expecting three decimal places of precision, Excel still pads it out to the fifteen places. It is impossible to instruct Excel to pay attention to two decimal places in one instance and three in the other. Humans can do it, but Excel cannot. Note that in the previous paragraph I tried to make a distinction between "digits of precision" and "decimal places of precision." This may sound like a fine hair to split, and some mathematicians may try to brand me a heretic. I make the distinction because many people automatically assume that precision has to do with the number of digits to the right of the decimal point. In Excel, however, precision simply has to do with the number of digits—period. Thus, the fifteen digits of precision maintained by Excel counts digits both before and after the decimal point in the total. If there are four digits to the left of the decimal point, then Excel will only maintain eleven to the right. If there are only two to the left, then Excel maintains thirteen to the right. What does all this mean to the average Excel user? Not a whole lot, other than you may need to look at numbers a little differently than Ms. Jameson did. It doesn't hurt to remember how Excel maintains numbers internally, however, as it can affect the calculations you perform.

ExcelTips is your source for cost-effective Microsoft Excel training. This tip (1981) applies to Microsoft Excel 97, 2000, 2002, and 2003.

Related Tips:

Comprehensive VBA Guide Visual Basic for Applications (VBA) is the language used for writing macros in all Office programs. This complete guide shows both professionals and novices how to master VBA in order to customize the entire Office suite for their needs. Check out Mastering VBA for Office 2010 today!


Leave your own comment:

  Notify me about new comments ONLY FOR THIS TIP
Notify me about new comments ANYWHERE ON THIS SITE
Hide my email address
*What is 5+3 (To prevent automated submissions and spam.)
           Commenting Terms

Comments for this tip:

There are currently no comments for this tip. (Be the first to leave your comment—just use the simple form above!)

Our Company

Sharon Parq Associates, Inc.

About Tips.Net

Contact Us


Advertise with Us

Our Privacy Policy

Our Sites


Beauty and Style




DriveTips (Google Drive)

ExcelTips (Excel 97–2003)

ExcelTips (Excel 2007–2016)



Home Improvement

Money and Finances


Pests and Bugs

Pets and Animals

WindowsTips (Microsoft Windows)

WordTips (Word 97–2003)

WordTips (Word 2007–2016)

Our Products

Helpful E-books

Newsletter Archives


Excel Products

Word Products

Our Authors

Author Index

Write for Tips.Net

Copyright © 2016 Sharon Parq Associates, Inc.