Thursday, March 06, 2008

TotT: Understanding Your Coverage Data

Code coverage (also called test coverage) measures which lines of source code have been executed by tests. A common misunderstanding of code coverage data is the belief that:

My source code has a high percentage of code coverage; therefore, my code is well-tested.


The above statement is FALSE! High coverage is a necessary, but not sufficient, condition.

Well-tested code =======> High coverage
Well-tested code <===X=== High coverage


The most common type of coverage data collected is statement coverage. It is the least expensive type to collect, and the most intuitive. Statement coverage measures whether a particular line of code is ever reached by tests. Statement coverage does not measure the percentage of unique execution paths exercised.

Limitations of statement coverage:

  • It does not take into account all the possible data inputs. Consider this code:

    int a = b / c;


    This could be covered with b = 18 and c = 6, but never tested where c = 0.

  • Some tools do not provide fractional coverage. For instance, in the following code, when condition a is true, the code already has 100% coverage. Condition b is not evaluated.

    if (a || b) {
      // do something
    }


  • Coverage analysis can only tell you how the code that exists has been exercised. It cannot tell you how code that ought to exist would have been executed. Consider the following:

    error_code = FunctionCall();
    // returns kFatalError, kRecoverableError, or kSuccess
    if (error_code == kFatalError) {
      // handle fatal error, exit
    } else {
      // assume call succeeded
    }



    This code is only handling two out of the three possible return values (a bug!). It is missing code to do error recovery when kRecoverableError is returned. With tests that generate only the values kFatalError and kSuccess, you will see 100% coverage. The test case for kRecoverableError does not increase coverage, and appears “redundant” for coverage purposes, but it exposes the bug!

So the correct way to do coverage analysis is:

  1. Make your tests as comprehensive as you can, without coverage in mind. This means writing as many test case as are necessary, not just the minimum set of test cases to achieve maximum coverage.

  2. Check coverage results from your tests. Find code that's missed in your testing. Also look for unexpected coverage patterns, which usually indicate bugs.

  3. Add additional test cases to address the missed cases you found in step 2.

  4. Repeat step 2-3 until it's no longer cost effective. If it is too difficult to test some of the corner cases, you may want to consider refactoring to improve testability.


Reference for this episode: How to Misuse Code Coverage by Brian Marick from Testing Foundations.

Remember to download this episode of Testing on the Toilet and post it in your office.

7 comments:

  1. nice explanation that code coverage is not the same as path coverage

    ReplyDelete
  2. I would have liked the article to mention object (assembly) level testing. If you define your test cases using breakpoints at every branch (or jump) instruction you can observe every path. I.E. if(a||b) will be at least 2 branch instructions and force you to test both. Testing at object level will also highlight any dead code or logic inserted (or optimized) by the compiler/linker. This is the kind of testing required by DO-178B level A certification. FYI: DO-178B is an FAA specification required for civilian equipment that will fly in US Airspace.

    ReplyDelete
  3. Nice post, Dave; code coverage indeed isn't test coverage.

    Here are a couple of points that I think are important and necessary for to ask about test coverage, where code coverage simply begs the questions:

    1) Will the code in question work reliably?

    2) Do you care about performance in the system under test?

    3) So you've got code coverage--but isn't it testing only your code? It's not testing the code of the operating system, third-party libraries, hardware or firmware, or anything else on which your product depends? That is, code coverage indicates what code you've touched in the product--but we have to test not only the product, but the system--the product and all the things around it.

    4) Does your test coverage tell you about supportability, testability, maintainability, portability, and localizability?

    5) Does your test coverage address the issue of value? Code coverage can tell you if something has been done--but was that something useful?

    Cheers,

    ---Michael B.

    ReplyDelete
  4. Amazing example on code coverage. But looks as if better said than done. Really wish if we lived in a perfect world. Blogging initiatives like the current one keeps the spirits live. Good guys.

    ReplyDelete
  5. great explanation, besides code coverage it also made me many more things to understand.

    Sachin

    ReplyDelete
  6. great explanation, besides code coverage it also made me understand many more things.

    Sachin

    ReplyDelete
  7. great article for code coverage and perfect testing logic

    ReplyDelete

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.