Test from theTop


Test Coverage.

Today I want to talk about Test Coverage.

Consider this conversation between an engineer reporting test results to their manager:

In this example, there's plenty going right, and some things going wrong I'll go over, but let's acknowledge both.


At least they are testing!
Testing is encouraged and supported. Development time is allocated for that purpose.
"Why didn't we catch those bugs?" This is the right question.


Pass/fail scores show percentage of failed tests, but that number could be tens or hundreds, there's no context.
What percentage of the system is covered by tests? We don't know.
Tests don't seem to be effective.

False Positives

When a test breaks, it might have found a bug, or the test code might be in error. Tests in error could represent a bug, other unexpected behavior of the system, or bad test code.

For the purpose of measuring coverage, it doesn't matter why a test is broken. A failed test is not functional, and cannot provide coverage.

This is a topic I can talk about endlessly, but let's stop here.

If you want to read about advanced coverage measurements, I have a dedicated website.

Coverage is a pretty esoteric topic, so I hope I could clarify it for you a bit.