Uncharted Waters

Feb 20 2017   10:41AM GMT

Automation Delay – And How To Prevent It

Matt Heusser Matt Heusser Profile: Matt Heusser

Continuous integration
Software testing

Pretend for a moment you are a new development manager or director.delayed

You hired a programmer to lead a test automation effort. Gave the programmer time, resources, and attention. The programmer has no responsibilities outside of the tooling. The automation has had time to mature. It was supposed to speed you up.

Now imagine this conversation, at 10:00AM.

Manager: “I’d like to go to production at noon. Can we do that?”

Programmer: “I don’t know, we need to run the automation.”

Manager: “Okay, sure, get to it.”

Programmer: “First we need to get the exact build we need to run on staging.”

Manager: “Fine, do that, then run the tests.”

Programmer: “Then I need to set up the test data.”

Manager: “Okay, fine, whatever, do it.”

Programmer: “Then I can run the tests.”

Manager: “Fine, so you’ll be done by noon?”

Programmer: “Probably. Then we’ll have failure results to check.”

Manager: “You always have failures?”

Programmer: “Not real failures. Sometimes the tests are flaky or there are environment problems. Sometimes the User Interface has changed but the tests haven’t kept up. So I need to do some maintenance work.”

Manager: “So you’ll know in the early afternoon?”

Programmer: “Sure.”

… time passes …

2:00PM – Automation Delay

Manager: “Okay, what’s up?”

Programmer: “Bugs. Just a couple of showstoppers.”

Manager: “So, no ship today?”

Programmer: “The production programmer said he’ll have it fixed by 4:00PM.”

Manager: “That’s great! We can ship by end of day?”

Programmer: “No, I need to rerun the tests.”

Manager: “Okay, but you can do that overnight, right?”

Programmer: “No, we really need someone to baby-sit them. The screen could lock, that sort of thing. I’ll kick it off in the morning.”

The next morning, at 10:00AM, the new build is certified and can get to production.

Let me be clear:

When the director invested in test automation, that is not what he thought he was buying.

I call this process of fits and starts “automation delay.” It commonly happens when test groups, left to themselves, decide to automate the process, especially though the User Interface.

Interlude: Getting Started

conference_sessionA few years ago I attended training from a major tool vendor. The idea was that some of us independent consultants could learn the tool and support it. After learning how to use the tool, I started to ask questions about a command-line runner and the output the tool produced — so we could kick it off on a schedule, capture the output, then decide if the checks passed, ideally by a computer, that could then choose if the software should be promoted to the next level.

We spent a little time tracking down in the documentation that yes, a command-line runner did exist, and the output was predictable, so we could wrap it in something that produced output for a continuous integration (CI) server to parse.

Once we proved it in theory, the trainer moved on. “After all”, he said “Most of you will be called into companies just getting started. No need for this advanced stuff.”

Just getting started.

So set up the test server by hand, seed the test data by running an import or restore database of some type, then kick off the automation to run at your desk.

Better run it at lunch, or at least find something else to do while the driver runs.


Worse, this scenario sets us up for the problem I mentioned at the beginning – it sets up the customer for automation delay.

The Bottom Line

The Pyramid by Mike Cohn prevents automation delay

Mike Cohn’s Test Automation Pyramid

Too many testers start out automating without support from anyone else. All they see is the GUI, so they begin automating the GUI. That leaves all the problems we mentioned before, and leads to automation delay.

Don’t do that.

Instead, start with the running the tests under continuous integration, which probably means extending CI to actually create a virtual machine and install the latest build, before running the automated checks. Good. Do that. That will probably be hard, and you might need the support of Dev or Operations. That is okay; that is sort of where the term DevOps comes from.

Of course, what I wrote above involves a great deal of sweeping generalizations. You could be anywhere on the spectrum; you might make batch applications that don’t really have a User Interface. For the time being, consider the impact of automation delay, if it could be happening to you — and what to do about it.

 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: