Please Note: This article is written for users of the following Microsoft Excel versions: 97, 2000, 2002, and 2003. If you are using a later version (Excel 2007 or later), this tip may not work for you. For a version of this tip written specifically for later versions of Excel, click here: Avoiding Rounding Errors in Formula Results.
Written by Allen Wyatt (last updated June 5, 2021)
This tip applies to Excel 97, 2000, 2002, and 2003
Nick notes that the formula "=0.28*100-INT(0.28*100)=0" returns False even though it is obviously true. He believes the issue has to do with rounding and how the computer uses binary arithmetic, etc. Nick is using the formula as a part of a larger IF statement, and he assumes there are other rounding errors in Excel that can reach out and bite him. He wonders if there is a simple way to avoid formula pitfalls such as this.
Before looking at simple ways to avoid this type of problem, it is important to understand why the problem exists. As Nick notes, it really has to do with binary arithmetic and the fact that Excel rounds information. Behind the scenes, Excel always rounds information to 15 digits. Consider the calculation that Nick is working with:
=0.28*100-INT(0.28*100)=0
When Excel first calculates this, the precedence followed by Excel calculates it in this manner:
=(0.28*100-INT(0.28*100))=0
Note the extra set of parentheses. The result of everything to the left of that final equal sign is 3.55E-15, which means that you end up with this (longhand) formula:
=0.00000000000000355=0
This is obviously not true, which is why you get the value False returned. There are several ways to "fix" this situation. In this case, perhaps the easiest is to simply change your formula to remove the need to compare to zero:
=0.28*100=INT(0.28*100)
This formula returns True, as you would expect. This may not work for all your needs, however. So, a better rule of thumb to avoid problems is to never rely on Excel's rounding. You do this by implementing your own explicit rounding, as shown with these formulas:
=INT(0.28*100)-INT(0.28*100)=0 =ROUND(0.28*100,0)-INT(0.28*100)=0 =ROUND(0.28*100,5)-ROUND(0.28*100,5)=0
Note that you are, in these instances, not allowing Excel to perform calculations to its full precision since that can cause some unexpected results when you are comparing to a specific value, such as zero. Instead, you are forcing Excel to round the values—all the values you are working with—to whatever level of precision you need for your comparison.
Another approach is to not do the comparison to an exact value, like zero. Instead, allow for some "fudge factor" in the comparison, which allows for rounding issues. For instance, you may determine that you only care if the comparison is accurate to one one-hundredth of whatever units you are assuming. In that case, the original formula could be modified in this manner:
=0.28*100-INT(0.28*100)<0.01
This returns True, as one would expect.
ExcelTips is your source for cost-effective Microsoft Excel training. This tip (8143) applies to Microsoft Excel 97, 2000, 2002, and 2003. You can find a version of this tip for the ribbon interface of Excel (Excel 2007 and later) here: Avoiding Rounding Errors in Formula Results.
Create Custom Apps with VBA! Discover how to extend the capabilities of Office 2013 (Word, Excel, PowerPoint, Outlook, and Access) with VBA programming, using it for writing macros, automating Office applications, and creating custom applications. Check out Mastering VBA for Office 2013 today!
Rounding is a fact of life when it comes to using formulas in a worksheet. Sometimes that rounding can be a bit ...
Discover MoreExcel provides a variety of functions you can use to round values in any number of ways. It does not, however, provide a ...
Discover MoreWhen processing data it is not unusual to need to round that data in some way. For instance, you may need to round a ...
Discover MoreFREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."
2023-11-02 16:47:51
Kit
I am having a rounding issue that I'm hoping there is a special way of doing it that will help. I like to set up tables with cross=checking totals. I have tables with # of hours worked for each pay period of the month, then a total column and a calc column to calc the wages. Well, the wages are obviously paid on individual pay periods, but then if you add the monthly hours and calc the wages, rounding issues happen. I've found that I have to calc each pp individually, then add to get the same total.
In my cross-checking totals, if I add the column of each person's hours for the pp, and then multiply by the rate (they all have the same rate), the RoundUp on this doesn't capture each rounding, so I fall short by several cents. Hope this makes sense and someone has an idea, other than adding a bunch of columns. :) Anyone?
THANK YOU!!
Got a version of Excel that uses the menu interface (Excel 97, Excel 2000, Excel 2002, or Excel 2003)? This site is for you! If you use a later version of Excel, visit our ExcelTips site focusing on the ribbon interface.
FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."
Copyright © 2024 Sharon Parq Associates, Inc.
Comments