Uncharted Waters

Dec 5 2019   3:44PM GMT

Test Heuristics

Matt Heusser Matt Heusser Profile: Matt Heusser

Software testing

When I was against heuristicsTwelve years ago I was complaining about this term, Heuristics. The term itself is incredibly simple in meaning – the kind of thing  you can learn in thirty seconds. People seemed, to me, to be using it to “sound smart”, to create insiders (who knew the term) and outsiders (who did not).

Today I see the value in it – the term is precise. I want to bring you on the inside.

Let’s start with that thirty second definition.

“Attend to what you eat, how you sleep, and exercise” won’t solve every weight problem. You  could have an insulin problem or other medical condition. Yet it is good enough, often enough, that it is a go-to idea for us. The engineering term for an imperfect way to solve a problem is a heuristic.

Many of us have our heuristics internalized. They are implied in the way we work, and can be as simple as “if you want to get anything done for the day,  don’t start with email. Start by doing the work”, followed by “at the end the the day, park your work to start  it immediately tomorrow.” Those are productivity Heuristics. (By the way, it is prounounced “Hew-Ris-tic”) Today I want to talk about my heuristics in software testing.

Heuristics in Software TestingMaaret on Her Heuristics

Earlier this week I had a great conversation on twitter with Maaret Pyhäjärvi. She ended up tweeting a few of her favorite Heuristics, including “See with asymmetry”,  “Short test first”,  “Amplify annoyance” and “Never be bored.” I expect Maaret could easily unpack each of those into a quick guideline.

It seems fair, since I asked her, to unpack a few of my own.

Overwhelm the Site. I generally start by abusing the site or native app, doing what it does not expect. Sometimes this yields serious security bugs. More likely, I will get a feel for how the programmers handle the unexpected. This allows me to do a quick evaluation. If the software handles exceptions well, the programmers are more likely to handle the general cases well. Combined with a quick happy path test, this can yield a quick evaluation surprisingly quickly.

Think Oppositionally. If the team is using all MacBooks with Safari, I want to test on a windows machine with Edge, and vice versa. If Tablets are supported but no one is using one, you can guess where I will start. This also aids in what I am calling the “hasty evaluation.” It is hasty, in that it could be wrong. Which is heuristic.

Path to Purchase. If you work in Ecommerce, you are familiar with create an account, search, product page, add to cart, checkout. If any of these are broken, the company is losing money. This is the “happy path”, but focused on value for user.

Follow the Money. Similar to path to purchase, I want to find out how the company is compensated.  At one luxury goods company I worked with, we found that the vast majority of money spent online was done in an iPhone with Chrome – not exactly the top of my usual test list. Worse, there was a “stick scroll” problem in that combination. A sorted list of customer spend over a month changed our test priorities.

Look for unanswered combinations. If I see a prose english description with a lot of combinations, I’ll build a decision tree or decision  table and write what I expect. If the combinations are complex enough, some boxes will be blank – what to do when conditions A, B, and C are all false, or perhaps all true, for example. From there, I test the software — then ask what it should do.

Ask Dumb questions. Talk to people about the customer’s intended use. In fact, ask redundant questions. Ask three different people the same question, and look for documents that seem to cover the same ground. Once you have them, look for differences, or area of interpretation. When people interpret the same thing in different ways, or the interpretation is vague — you’ll find bugs.

And, of course, there is throw a little i18n at it. Even if the decision makers swear up and down they do not care and do not need to support international characters, I throw in some Spanish and French-Canadian names, or perhaps Finnish, like Maaret Pyhäjärvi, when I am testing name fields anyway. If the system works, or doesn’t, I learned something. That test is essentially free.

A few of your favorite things

You’ll notice something subtle here. Maaret talked about her heuristics and I wrote about my heuristics. That is, heuristics can be a deeply personal thing. It takes work to take them out of your head, write them down, and compare them. Listing your heuristics is a chance to demonstrate your mastery.

Yes, there are classic heuristics to study, such as Focusing/Defocusing, Dive in and Quit, and the dead bee heuristic. There is a coal mine in the software test literature to study.

Still, at some point, you will develop your own.

Challenge: What are some of your test heuristics?

 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: