Uncharted Waters

Feb 23 2018   11:01AM GMT

The Meta-Game of Regression Testing Tooling

Matt Heusser Matt Heusser Profile: Matt Heusser

Continuous integration
Test Automation

Selenium for Regression TestingBack in the 1990’s we used to have a long, painful process we called regression testing.  That was the final step before pushing the code to production. It happened after the code was ‘complete’, and pretty much involved re-testing everything. Since the code was typically a big ball of mud without well-separated concerns, any change anywhere could break anything, so we had to re-test all of it. The re-testing was to make sure the quality did not regress, with a new feature breaking an older feature. Hence the term “regression testing.”

Most of the automation strategies I get brought in to help with are plans to do just that: Take the regression testing and have the computer do it, click for click. Inevitably this leads to automation delay, and, more often than not, test tooling project failure.

Today I want to describe a more excellent way.

Regression Testing  –  The Code-First Approach

Regression TestingWhen I sit down on one of these tooling projects, typically bring up an integrated development environment and start working on the code. First, we need selenium-webdriver, capybara, or some other tool to open up a web browser. On top of that there may be another layer, like Cucumber, that allows us to express the tests in high-level business language. Then there’s a glue layer, that implements the code that those english-like cucumber sentences will call. Beyond that we might want to create a reusable code library, perhaps with page objects, that the cucumber implementation will call. This will keep all the id’s and locators and click-type-click statements together in a batch. When the user interface changes, we can change just the code library, and re-run everything.

I have very little interest in any of that.

That is not to say no interest. It’s definitely worth the conversation on how to structure the code, and the strategy above is light years beyond creating yet another “big ball of testing” mud that will itself fall over in three months.

The problem is automation delay.

When the prototype is done, we’ll set up a new test environment, clear out the database, run the prototype, watch the screen flash, pat each other on the back, and build a backlog of new features to test.

Each time the software runs, we need to send a new push to a test environment, and clean the test database. Someone needs to go to their own computer and enter the run command — or else do it from the IDE.

And then we wait.

Then we look at the output to see if it passed or failed. Most of the time, when tests are added to a legacy system, there will be intermittent failures we call “flaky tests.”

This makes running the tool a human, manual step.

Regression Testing  Infrastructure –  Not Code

Regression Testing InfrastructureWhen I talk about a plan for test tooling, I mean a plan for all of it. The code is just one piece.

Another piece of the solution will create the test environment from the command-line, including either clearing the database or setting up a ‘test’ data base with known expected values – for example, known users and permissions so we can predict search results up front.

Then something needs to run the tests, ideally somewhere else, with the capability to extend to run more than one at a time and aggregate the reports. We don’t need to do that at first, but I don’t want to shoot the team in the foot when we need to add it later. A good software craftsman can write the code so that it can be extended, without wasting time on features we won’t need.

I also like having a plan to structure and group the tests, perhaps with tagging, making it possible to run sub-sections of the tests that focus on certain areas, like search, reporting, or path-to-purchase.

By now you have the picture. Setup and runner execute at the end of a continuous integration run, creating a virtual server and running through some scenarios as a user.

The prototype we show at the demo is not an hour of watching the screen fly by and click buttons. Instead, what we show is one test that executes without any humans involved that can be plugged into Continuous Integration today.

The basic game of test tooling is easy.

The meta game? It isn’t much harder.

I just wish more people would show up to play.

 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: