Test from theTop


Dashboards, Graphs and data visualizations.

When testing your software, the result is a lot of data: what worked and what didn't. It makes sense to compile all the results into statistics and graph them out, for ease of understanding.

Let's talk about all those pretty visual graphs of test results. These can be intriguing decorations, with the promise of quick, simplified insight into test results. But the thoughtful manager knows a few caveats about these helpful displays.

Pretty! But does it represent the right data?

Maybe you've encountered that quote before. It's sort of vague, so I'll be clear:

Tests that never break also never reveal bugs.

Is that big green section of pie chart actually a good thing? It could represent a false sense of security. Something to keep in mind.

In fact, once you start using these charts to evaluate test performance, your team might even start to build tests that compliment the charts, but provide little true value.

That's never a good place to be. My experience says that when teams are rushed to produce tests as sort of a last-minute to-do item they are often low quality. The fix for that is a team culture that prioritizes testing, a team who knows the value of automated testing and how it makes their job easier.

Don't get me wrong, charts and data visualizations are vital and useful, you should have them for your test results. I'm pointing out that they can mislead, too. Focus on test quality, or else your coverage metrics aren't worth much.

In a future newsletter we'll dig into Test Coverage, an important, under-looked subject. Understanding Test Coverage is the key to building useful tests, and there are other common mistakes to avoid along that road. Don't worry, I'll help.

One more thing I want to mention about these charts. They can be expensive to build and maintain.

You don't have to code it all from scratch of course. There are tons of applications you can buy or leverage out there to get these nicely color coordinated graphs for your testing data. Even so, they all require you to transport your data to them, that's a highly technical integration task that will cost developer time. Once you commit that time, it's going to be hard to make big changes, just like any other code project. Be careful going in.