software bugs

Proactive testing, or “this app can break”

Before Windows Vista came along and ruined it all, I previously used a bug in Windows Notepad to illustrate a problem testers often face. Vista ruined it by fixing the bug. If you have a version of Windows pre-Vista, you can still try this bug out. To reproduce the issue, open Notepad and type “this app can break”. Then save the file. If you were to close the file and re-open it, you’d find that your data has been corrupted.

Spoiler alert: If you want to research the Notepad problem and see if you can figure out what the issue is, then stop reading. I’ll tell you now it’s not an Easter egg, even though it looks like one.

On Windows XP Notepad calls a method titled IsTextUnicode when it opens a file. You can read about it here. The noteworthy text on this page is the following:

“This function uses various statistical and deterministic methods to make its determination [...] These tests are not foolproof. The statistical tests assume certain amounts of variation between low and high bytes in a string, and some ASCII strings can slip through.”

What that text states is that Notepad uses a heuristic algorithm to open a file. Like any heuristic, it’s a solution to a problem that works most of the time. That’s why you’ve likely never seen that bug before. There are only a finite set of conditions that will cause it to fail.

This bug represents several problems that many testers face everyday:

1. When the development solution is heuristic, or the number of variables involved makes a deterministic solution to the problem impossible to determine manually, testers have to expect that there are cases they will miss that could expose problems. For Notepad, that’s fine. For a heart monitor, it might not be.
2. A method that a developer uses might work perfect for two uses (Word and Wordpad), but might fail when used for a third (perhaps inappropriate) application. We use so many third party languages and frameworks when we develop today, it’s impossible for a developer to keep all of the code they didn’t write straight.
3. As testers, we often need to dig into a problem well past the point of saying: “I noticed what might be problem here.” If we understand why this is a problem, it helps us refine our models of the applications we’re testing. Now your model of Notepad should have changed to imply that Notepad uses a lot of the same code-base that Word uses. That’s interesting to know when testing because it gives you another oracle for what the correct behavior might be. It can also inform your conjectures for application behavior.

I was first given this testing/debug problem by James Bach a number of years ago (pre-Windows XP I think). I think I spent over an hour testing and researching until I came upon the root cause of the problem. It was a valuable lesson for me. Because of this experience, I now look forward to opportunities to help with issue research and isolation.

The effects of not isolating bugs and tips for nailing them down

Recently, fellow SearchSoftwareQuality.com expert David Christiansen shared his post about experiences with testing ruts that he gets into and what he does to stay out of those ruts.

What resonated with me was his description of how he sometimes doesn’t feel like working to isolate bugs:

You did x, y, and z and the app crashed, so you filed a bug report and moved on. Does it crash with just x? Are there variants of y and z that don’t make it crash? How do they work together? If you don’t know and don’t care, you need to power up.”

Dave points out what I believe is an important step for software testers. I’ve seen many testers encounter what could be critical issues, they log a defect ticket in passing with a shallow description of the problem, and they move on. Just to be fair, I’ve done it too. When this happens, I find that often there are two outcomes:

1. The issue isn’t looked at immediately, or even fixed, because the description is vague, looks like an edge case, and doesn’t have clear implications past the immediate problem identified in the ticket.
2. The tester misses out on a deep and rich opportunity to learn more about the application, how it was developed, and what dependencies it has. I find that some of my most insightful observations about the system, how it works, and how that relates to the testing I’m doing comes from isolating defects.

While you don’t need to track down a possible issue to the offending line of code, I think a tester should be able to draw a clear chalk outline around the issue. That means they should be able to say, with some confidence, that under what conditions it does and doesn’t occur, and what minimal set of conditions appear to trigger it. If they can, they should talk about potential impact - but only if it’s data-driven analysis and relevant to getting the issue fixed.

To that end, the following tips might be helpful for when you’re working to isolate an issue:

• Take excellent notes and keep all the evidence. This includes test execution notes in a notepad, screenshots or screen capture utilities, copies of log files, snapshots of disks or virtual images, etc….
• Work to recall what you were doing before you found the problem. Often times, if the cause of the problem isn’t obvious, it was something you did five steps earlier that triggered what you saw. If you can find the deviant step, try variants of that activity to see how else the problem manifests itself.
• If the investigation goes for more than a day, find a place to share information about the problem with the rest of the team (a wiki, SharePoint, or a defect tracking tool). I often find it useful to keep lists of the following information:
  • a list of the symptoms of the problem
  • a list of what variables you’ve looked at already as you try to reproduce the issue
  • a list of the variable you haven’t looked at yet, but you suspect they might be related
  • a list of who you’ve spoke with about the issue (and any details they provide)
  • a list of possible workarounds
  • a list of possible tools (or techniques you may not know or be good at) that might help

At some point, it’s important to recognize that with any difficult problem, you’ll need some stopping heuristics. Of course the one we all want to use is, “I found the problem.” However, sometimes that doesn’t happen. Make sure you have a clear idea of how important the problem is and how much time you have to dedicate to it so at the appropriate time you can drop it or shelve it for later.

For more on this topic, and dealing with other testing ruts, be sure to checkout Dave’s entire post on testing ruts and how he deals with them.

Software testing blog digest: Bug, team woes; memes; Ford Motors

If you don’t read software testing blogs, you’re missing some great advice and thoughtful ramblings on testing philosophies. I tap into those blogs daily, and here I’m sharing the wealth with this reader’s digest of the testing posts I enjoyed this week.

Why bugs are hard to kill

On Maverick Tester, Anne-Marie Charrett describes the mistakes she’s made when doing offsite exploratory testing under tight deadlines. Then, she reveals how she’s stopped making those mistakes in her list of offsite exploratory testing guidelines to bug reporting.

One tidbit of her advice: Write reports right away, even if you are super-busy. She writes that “it takes longer to write them up at the end, when you have to review heaps of cryptic phrases in Session Tester or in your notebook.”

I love the post’s title, “Do your bugs only glow when it’s dark?” It reminds me of the “putting out fires” metaphor. How many times have I gotten emails from co-workers, site experts and others saying they’re late with a response or a deliverable because they’ve been putting out fires? Hey, I’m guilty, too.

In my own work, I see that most of these fires were started when haste made waste. Why is it so hard to take things one step at a time? Oh yeah, there’s a deadline and not enough time to make it.

When familiarity breeds success
Moving on, two posts on Matthew Heusser’s Creative Chaos blog explore thought-provoking topics: team cohesiveness and memes. In his post on Jelled Teams, he ponders the good results of working on a team that’s been together for over a year. How much creativity and productivity is lost, he asks, when companies often shift people from team to team as casually as they do? Too few managers realize that teamwork flourishes when people know each other well enough to feel comfortable sharing their ideas. When a team works well together, it’s an added-value asset in and of itself.

So, project managers, think twice before breaking up good teams!

I see a connection between that post and Heusser’s musings today in The meme’s the thing. Wikipedia calls a meme “a postulated unit or element of cultural ideas, symbols or practices, and is transmitted from one mind to another through speech, gestures, rituals, or other imitable phenomena.” Good grief! I think Heusser’s short definition is better: “It’s an idea - a concept that spreads from person to person. “

Any married person knows that familiarity and mind-melds go hand-in-hand. It stands to reason that team members that’s been together a while will start understanding how each other thinks, and the ideas will start flowing. Community work along the same lines. That’s why, I think, the open source software community has made such great strides so quickly. Another is that open sourcers are so communicative and have created vehicles – sites, projects, message boards and so on – that foster collaboration.

Heusser believes that software testers should be thinking along the same lines and said:

“I believe that the communities I belong to…have ways to test software that are significantly better than the status quo, and we have ways to communicate them and techniques to teach them. Yet if our testing ideas are memes, we need to think about ways to package and present them to win.”

Carrying on with the teamwork theme, there’s a nice exchange on the topic of how to handle unhappy testing teams on Jerry Weinberg’s blog, The Secrets of Consulting. A software test manager at an insurance company wrote to Weinberg, and they –- and others – brainstorm on the subject in an informative message chain.

On the lighter side

Once you’re a software tester, you look at everything from that point of view. So, Software Quality Insights blogger and independent consultant Mike Kelly describes Ford Motor’s web application flaws whe he was trying to spec a new Ford truck. In his entertaining post, he concludes that it’s easier to build and buy a Toyota online. This is something Washington has missed when discussing bailouts and the state of U.S. auto companies, I think.

There were plenty of other good reads in testing posts this week, more than I can cover here. Please comment below if you read something good this week or have a favorite testing blog.