Uncharted Waters

Jul 12 2018   10:00AM GMT

The Magic Automation Tool

Matt Heusser Matt Heusser Profile: Matt Heusser


An Automation ToolTake any part of the development process – say testing, my bread and butter. Your team has it all figured out, with a magic automation tool. By magic, I mean exactly that. Push a button and you will get instant test results, clearly articulated.Not fast — instant. I’m talking about magic.

The automation tool never makes a mistake. It magically knows what a change will be and how it should impact the system, so it never reports a false error when the system behavior changes. Because of this, it needs no maintenance. Just change the code, click the button, see all the problems.

Now, will having that accelerate your software delivery at all?

In many cases, the answer is “no.” Even with the organizations that benefit, the benefit might only be a two to four percent increase in throughput.

Here’s why, and how to do better.

Test Execution != All Of Testing

More Automation ToolingThat first thing to come back from that test automation tool is a set of results. Humans will need to examine those results, to understand what went wrong, and to explore to find if the problem is a core bug or the symptom of something larger. Then the bugs need to be defined in a way that can be fixed.

Next we argue about which of the bugs needs fixing.

Then we wait. And wait. And wait some more.

We wait for fixes to be made, unit tested, for a build, and a push to the test server.

Then we can press the button again.

When Tom Wissink was director of Test at Lockheed Martin, he used the phrase “Find->Fix->Retest loop” to the whole process. Fifteen years ago the famed software Michael Bolton, made a similar observation, pointing out that once the first blocking bug is found, it is no longer a “testing” phase, but instead a “fixing” phase.

Bolton’s observation was before Agile Development was popular, but it bears notice. If the primary delay in testing is not testing but instead “fixing, re-testing and re-fixing”, then automating test execution won’t save much time.

Oh, it might save a little. Assuming the tooling was good enough to find most of the important bugs enough of the time, test automation tools could enable release of features to production as soon as they are created, instead of “batching them up” to be deployed every two weeks, two months, or twice a year.

When automation tools won’t help

If the bottleneck on the system is not the task, then automating that task won’t help. The fix->build loop is too slow, and the number of defects is too large, meaning you’d have to press the button more times.

The solution to that often is not faster testing. Instead it is higher first-time quality (less test cycles), faster fixes, better builds, quicker test environments with better test data. Once that’s done, testing might be the bottleneck, and then there will be some work to do.

But there’s another problem.

What Happens When You Assume

Automation tools are not a magic robotEarlier I wrote “Assuming the tooling was good enough to find most of the important bugs enough of the time” and sidestepped if that were really possible. Tom, Michael and I have done plenty of writing on that; you can google our names to find out about test automation tools. Today’s post isn’t about that.

Today’s post is just about finding the bottleneck in your system. Find the thing that if you had a magic button, you could go faster. You don’t need to build a button for that — just find a way to help that activity happen faster. In programming, that could be more powerful programming languages, frameworks like rails, or just bringing in lunch and getting programmers out of useless meetings so they can focus on, well, programming.

Instead of a single silver bullet, look for what Fred Brooks called a collection of bronze bullets, centered around the bottleneck. Once that ceases to be the bottleneck – and yes, some of our customers have done that in testing – figure out where the new bottleneck is.

 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: