Coverage
Test Coverage.
Today I want to talk about Test Coverage.
Consider this conversation between an engineer reporting test results to their manager:
-
Gigi: "Hey boss, here's the test results for the next release."
She points to the graph on her screen showing 90% passing tests.
Jefa: "Wow that looks good. What are the failed 10%?
Gigi: "Pretty much the same as last release, old bugs in the backlog and a few new ones we reviewed in our standup this morning."
Jefa: "I remember. None of those are show stoppers, let's do it."
==================================
✨ Public Software Release Happens ✨
==================================Jefa: "The client found three new bugs in production! At 2am last night! Why didn't we catch those?"
Gigi does research into the new bugs.
Gigi: "Well it looks like one was something weird on an old module we didn't predict, and the other two were related to the new features. We did write tests for those."
Jefa: "And the tests passed?"
Gigi: "Yes."
In this example, there's plenty going right, and some things going wrong I'll go over, but let's acknowledge both.
Pro:
At least they are testing!
Testing is encouraged and supported. Development time is allocated for that purpose.
"Why didn't we catch those bugs?" This is the right question.
Con:
Pass/fail scores show percentage of failed tests, but that number could be tens or hundreds, there's no context.
What percentage of the system is covered by tests? We don't know.
Tests don't seem to be effective.
-
There are two important types of test Coverage you should know:
Code Coverage - Engineers will calculate exactly which code is covered by automated tests. This type of coverage is only capable of catching a certain class of logic errors.
Specification Coverage - If you define the behavior of your system, you can summarize coverage on a per-specification basis. This measures the real functions of your system from a user's perspective. These tests are comprehensive, but they provide less insight to the bugs they find. Triage and diagnosis will take longer for each bug.
False Positives
When a test breaks, it might have found a bug, or the test code might be in error. Tests in error could represent a bug, other unexpected behavior of the system, or bad test code.
For the purpose of measuring coverage, it doesn't matter why a test is broken. A failed test is not functional, and cannot provide coverage.
This is a topic I can talk about endlessly, but let's stop here.
If you want to read about advanced coverage measurements, I have a dedicated website.
Coverage is a pretty esoteric topic, so I hope I could clarify it for you a bit.
-
What other questions do you have? Just respond to this email, I read every response.
ross@testfromthetop.com