5 Common Unit Testing Mistakes 

A robust set of automated unit tests are like an advanced security system, protecting your software against code bugs, providing early warning detection about unintended changes, and providing a safety net when making major changes, like large refactors. But any defense system is only as strong as its weakest link. Let’s explore five common unit testing mistakes that can silently compromise your unit test defense network.

Reminder: the goal of unit tests is to identify bugs, not just in the current code base, but future bugs as well. In a sense, a good unit test actually travels through time, finding bugs that don't exist yet. As such, we have to remember to not only test code as it is today, but as it might be some day in the future.

1. Not Securing the Perimeter

Bugs can creep into the crevices of your system, finding gaps in your defense. A favorite entrance point is around the perimeters, i.e. anywhere that deals with limits, ranges, or intervals. Unit tests can help secure your perimeter at these vulnerable locations.

Consider the following pseudo-code for a function that enforces some limit. The perimeter to be secured is some global variable “limit”:

       func isInRange(int index) : Boolean {
           return (index < limit)
        }

To provide complete coverage we know we want to test a true and false case, using a value within the range and a value outside of the range, e.g.:

       test isInRange() {
           setLimit(5)
           assertTrue(isInRange(4))
           assertFalse(isInRange(6))
       }

While the code coverage is complete, the functional testing is not. We have left a wide, untested gap around the edges of our perimeter, specifically when the value matches the index. If the value being tested matches the limit, is that in range or out of range? Find out what the requirement is for the limit (do not trust the code—after all that is what you are validating!) and add the appropriate test, e.g. assertFalse(isInRange(5)).

PRO TIP: Test Limits Exactly
BEWARE: Sloppy testing

Tightening Things Up

When dealing with limits that are not as simple as integers, we have to tighten the perimeter to be even more precise. Consider a typical example like this that determines if a date is expired by comparing it to midnight:

       func isExpired(Time: time) : Boolean {
           return !isBeforeMidnight(time)
       }

Before midnight, it is not expired; after midnight, it is expired. Often, tests are incompletely written by testing to the minute, passing in values like "23:59:00" and "00:00:00" (testing the limit itself - we can be taught!). That leaves the possibility that the method has a massive bug where it reports all times between 23:59:01 and 23:59:59 as expired. That is a big hole in the fence!

If minutes are not granular enough, are seconds? How small do we need to go? It depends on the system and the data types being used, but typically, for time comparison, checking milliseconds is sufficient, e.g. "23:59:59.999" and "00:00:00.000" or "00:00:001" depending on the limit. In high-precision scientific applications, you might have to go down to nanoseconds or further. The point is that tests need to be as precise as the data type to ensure no bugs sneak through holes in the perimeter.

PRO TIP: Test Limits Precisely
BEWARE: Cutting Corners

2. Test Bundling

Now that we have secured the perimeter, it is very tempting to put all of the test cases into a single test, especially for a straightforward method. Using the same pseudo-code:

       func isInRange(int index) : Boolean {
           return (index < limit)
       }

This single test that verifies both true and false cases:

test isInRange() {
           setLimit(5)
           assertTrue(isInRange(4))
           assertTrue(isInRange(5)) // testing the limit - but it will fail!
           assertFalse(isInRange(6))
       }

Congratulations, your test found a bug, and the second test will fail because five should be in range. So what's the problem? The third assert never runs, and the false case never gets executed. 

By bundling tests, any failing test prevents other tests from running, potentially hiding other errors those unrun tests would have exposed. To compound the problem, the later tests will only execute once the bug is fixed, providing a hiding place for other bugs that otherwise would be caught. Keep test methods as discrete as possible, trying to test only one case at a time. Note: this does not mean only one assertion per test case as long as all assertions are related to the same test case.

PRO TIP: One Test Case Per Test
BEWARE: Overly optimizing test code

3. False Positives

We have focused on exposing bugs trying to infiltrate from the outside, but the most dangerous bugs are those from within: bugs in our test code. Possibly the worst of those is the "false positive," that is, a test that passes when it should fail. In our above example of "isInRange()," a simple change of our test case could prove disastrous on multiple levels:

       func isInRange(int index) : Boolean {
           return (index < limit)
        }

       test isInRange() {
           setLimit(5)
           assertFalse(isInRange(5)) // we trusted the code that the limit should be exclusive 
       // rather than verify the requirements
       }

In our scenario, the limit should be inclusive but since the code treated it as exclusive, the tester, believing the code was correct, verified that the code worked precisely as it does, rather than how it should. This test not only masks the bug, but future developers may look at this test, conclude it is correct, and add on to that error, effectively spreading the bug to other areas. A false positive not only creates a false sense of security but also provides a breeding ground for other bugs.

PRO TIP: Verify Your Test
BEWARE: Testing against code instead of requirements

4. Stopping Short

We have learned our lesson. We have tightened up our limit test cases, broken them into separate test cases, and feel secure about our code. Let’s consider the following, where Card has one property, "balance: Double":

       func getTotal(List<Card>: cardList) : Double {
           var Double: total = 0
           for (card in cardList) {
               total = total + card.balance
           }
           return total
       }

Our study of limit testing tells us to test when the card list is empty and when it has cards, so we cleverly come up with the following test cases, in different test methods, of course:

       1. assertEqual(getTotal(new List<Card>()), 0.00)
       2. cardList has 1 card, balance = 10.00, assertEqual(getTotal(cardList), 10.00)
       3. cardList has 2 card, balance = 10.00 on each, assertEqual(getTotal(cardList), 20.00)

That seems thorough, doesn't it? We followed all of the principles of good testing, achieved code coverage, and were even extra thorough by testing lists with one card and with two. In Java-based languages (and maybe others), there is still a bug here that might not be caught, and might not be caught by QA. Doubles have a nasty tendency toward rounding errors, especially when you do math operations on them enough. If you tested this with three cards expecting a result of $30.00, it may result in 29.99998 or 30.000001 or similar. Based on your language’s data typing, this kind of error can easily slip through undetected. Most developers would not think to test for this unless:

• They had good knowledge of behavior like this
• They were working on a bug for receipts showing with values like 10.00001.

While this is easily missed, it points to our need to be curious and imaginative in our testing. Remember, the goal of testing is NOT to prove code works, but to find possible bugs. Take it as a challenge to see if you can break the code. After all, adding extra tests is easy and cheap - we don't need to be efficient in limiting our test code the way we do in the actual application. Get creative, think outside the box, don't limit your imagination to the obvious.

PRO TIP: Tests Are Cheap, Be Extravagant
BEWARE: Trusting code coverage

5. Testing Nothing

This error is common in more complex testing. A typical scenario is that a mock is created to mimic some behavior, and rather than verifying the behavior of the real object, the verification is done on the mock object. An overly simplistic example, we create a mocked object that always returns 10 when getItemCount() is called:

test mockSetupCorrectly() {
           MyViewModel myVM = mock {
               on { getItemCount }.doReturn(10) }
           }

       assertEquals(myVM.getItemCount(), 10) // this will always pass since we mocked it return 10

Of course, it is seldom this obvious, but when there are many objects involved, lots of mocking, data setup, etc. it is easy to accidentally test nothing. On the other hand, sometimes it actually is that simple. I recently heard of a test submitted like this:    

expect(true).toBeTruthy()

I trust that test passed!

PRO TIP: Verify Your Test Setup
BEWARE: Complex test setups

Safeguard Your Defense System 

Kudos to everyone securing their software with a state-of-the-art defense system of unit tests. These tests can serve to protect you from future developer errors and create a safety net for making critical changes and refactoring. But remember: Tests are vulnerable to human error, too, and need to be safeguarded. Being aware of these five common mistakes can strengthen your defenses, both now and in the future.