Precision in Excel

When we were in junior high school, each of us learned the meaning of precision from the standpoint of math: it defined a degree of accuracy of a number. Thus, we often saw what at first appeared to be confusing information. Ms. Jameson, the math teacher, would tell us that 4.76 was different than 4.760. The reason, it turns out, is that 4.76 is precise (accurate) to only two decimal places, while 4.760 is precise (accurate) to three decimal places. Thus, 4.760 is more precise than 4.76 because of the number of significant digits each contains. Enter the world of digital calculators and (by extension) spreadsheets, such as Excel. To them, there is no difference between 4.76 and 4.760. Why, you might ask, is this true, particularly when Ms. Jameson made such a big deal about the issue? The reason is because of the way in which calculators and spreadsheets store numbers and the way in which humans store numbers. Humans generally store numbers (in our brains) as they are written. This means that we look at 4.76, and we know it is accurate to only two decimal places because that is all we can see and write. Computer chips, on the other hand, deal in quite a different plane of precision. When you enter a number into Excel, it automatically converts the number to fifteen digits of precision—regardless of what you intended. Thus, 4.76 is automatically padded with zeros at the end out to fifteen digits, even if you thought you were only specifying two decimal places of precision. When you put in the number 4.760, expecting three decimal places of precision, Excel still pads it out to the fifteen places. It is impossible to instruct Excel to pay attention to two decimal places in one instance and three in the other. Humans can do it, but Excel cannot. Note that in the previous paragraph I tried to make a distinction between "digits of precision" and "decimal places of precision." This may sound like a fine hair to split, and some mathematicians may try to brand me a heretic. I make the distinction because many people automatically assume that precision has to do with the number of digits to the right of the decimal point. In Excel, however, precision simply has to do with the number of digits—period. Thus, the fifteen digits of precision maintained by Excel counts digits both before and after the decimal point in the total. If there are four digits to the left of the decimal point, then Excel will only maintain eleven to the right. If there are only two to the left, then Excel maintains thirteen to the right. What does all this mean to the average Excel user? Not a whole lot, other than you may need to look at numbers a little differently than Ms. Jameson did. It doesn't hurt to remember how Excel maintains numbers internally, however, as it can affect the calculations you perform.

ExcelTips is your source for cost-effective Microsoft Excel training. This tip (1981) applies to Microsoft Excel 97, 2000, 2002, and 2003.