Software Testing archives - Software Quality Insights

Software Quality Insights:

Software testing

Nov 9 2009   3:59PM GMT

How testers can handle switching to Agile’s short iterations



Posted by: Michael Kelly
Software testing, Agile software development

Software testers today are under a lot of deadline pressure due to Agile development’s shorter iterations. I recently hosted a panel on time management for a local software testing group, and attendees asked for advice on how to handle the frustrations associated with moving to short development iterations. While the theme of “making sure you have team buy-in” underlay the entire discussion, there were some particularly insightful nuggets related to making sure people understand why the work is structured the way it is in short iterations.

Robby Slaughter, founder of Slaughter Development, a workflow and productivity consulting company, set up the problem this way: Most software professionals in highly-tactical roles — whether they are authors, developers or marketers — are used to the idea that their work is a representation of professional capacity. So, the pro has this self-concept, and then their deadlines are changed to weeks from the traditional span of three-to-six months. Instead of working on one larger task, the task is broken up into 20 smaller pieces.

“All of a sudden that way in which you represent your own professional capacity, your expertise, is now violated. Now it’s being divided up.”

Slaughter’s main point is that the work is no longer yours. It’s the team’s. The individual parts that you work on or someone else works on are not as important as the overall result. Many times people working in iterations are dealing with that frustration. They no longer have the same level of ownership of the piece that they’re working on.

Brian Ball, software engineer with Software Engineering Professionals, had a slightly different take on how you might illustrate the same point.

“There are principles to why you do things in Scrum,” said Ball. “So if they’re uncomfortable with why we’re only doing a short section of work here, why we’re breaking this up and I’m not doing the entirety of the feature, [or] why I’m only dong this little unimportant subset; then when you start throwing stuff out of the backlog because the customer changed their mind…celebrate that. Say this is work we didn’t do and we no longer have to do. Just throw it out…So they see a concrete result of why the principle was there in the first place. They get the payoff of, ‘Oh yeah, we delayed it and we didn’t have to do it.’ [That's] because it wasn’t important — really important — at the time. It just left us with this uncomfortable and unfinished feeling at the time. “

If buy-in might be part of your problem, the panel offered a question checklist for helping understand what might be blocking buy-in.

“How do you set it up to provide it to them?” asked Francine Carter, founder of Action Coaching & Training. “How do you roll it out? Because the rollout’s the important piece to get people to begin to think about how they’re going to approach it. And if you’re thinking I can’t, then you’re not going to. What about it don’t you feel good about?”

Brian Ball recommends looking at your planning meetings and making sure all the work is being accounted for in the estimates — something that takes practice and measurement. Make sure everyone understands why the work was broken up the way it was, he said. In other words, paint the big picture.

Cory Mausolf, a career coach with Authentic Business and Career Coaching, pointed out that it can be important to make sure people are separating their feelings about the work itself from how the work is being done. Are they uncomfortable about the work? Or are they uncomfortable, he asked, “because that’s not how you did it before?”

You can find the complete audio for the panel members response to this question on the IWST website. There you can also find more information about the speakers along with additional audio from the meeting broken out by topic.

Nov 3 2009   7:52PM GMT

Some tips for one-person software test teams



Posted by: Michael Kelly
Add new tag, Software testing, software testing teams, software, Indianopolis Workshops on Software Testing

Last week I hosted a panel on time management for a local software testing group, IWST (Indianapolis Workshops on Software Testing). One of the questions asked during the meeting went like this: “I just started a new job and I’m at a company where I’m the only tester. Any tips for making that work?”

The entire panel was quick to point out that you’re never alone, and that you always have a team to work with. Brian Ball, a software engineer with Software Engineering Professionals, said that it’s important for you to realize you’re part of a larger team — whether they feel like it or not. He said you need to let the rest of the team know that you’re concerns are really their concerns.

Robby Slaughter, founder of Slaughter Development, a workflow and productivity consulting company, pointed out the following:

“What you really need to do in those situations is identify who are the stakeholders around you who care about the results of your work and the results of what’s happening, and find a way to leverage their expertise, their ability, their commitment and their passion towards what you do. Now that’s a problem that is magnified when you have technical expertise in some area of what you do, and the people around you — the stakeholders — don’t.”

Slaughter also pointed out that one of the most important things you can do is to help people give you information in a way that enables you to do your work better.

“You need to realize that you might have this expertise or this area of technical facility [and they do not]. You need to go back to the stakeholders and identify ways to communicate better. Maybe they need a form to fill out. Maybe they need a tracking system, your ticket tracking for the request. Or maybe you need to find a way to give them information in a more organized fashion, so they can find it more easily so they are not constantly calling you.”

Chris Wingate, principal software engineer for Liberty Mutual, summed up his thoughts on the question the following way:

“To me is seems to come down to two things…and those are trust and education. So you’re the only tester in your organization at the moment, and so it’s very important to build the trust in your competence with the rest of the team — with the developers and the managers. That way when you say, ‘Yes I can do that.’ or ‘No I can’t do that for these technical reasons.’ you don’t have to spend a lot of time explaining all of that and rehashing all of those things that you continually do.

“The other thing is educating the rest of the team about what your capabilities are, what timelines are, and they go together… you have to educate your customer in terminology. Educating your customer is going to help improve the communication and help make you more efficient.”

You can find the complete audio for the panel members’ advise for one-person test teams on the IWST website. There you can also find more information about the speakers along with additional audio from the meeting broken out by topic.


Oct 24 2009   1:41AM GMT

Drilling deep into performance testing at STPCon



Posted by: Matt Heusser
STPcon, software testing and performance test conference, performance testing, Software testing

After yesterday’s Boston SPIN meeting, I got a full night’s sleep, woke up, checked email, and it was time for the full-day workshop on performance testing.

In the morning, Scott Barber presented a half-day workshop on quick and easy performance wins. In the presentation, he split the hard engineering performance testing work — say, simulating and cranking up load –from the “easy stuff.” To do the easy stuff, Scott had four concrete suggestions:

1) Know the mission

Instead of stating with a document — “100% of web pages shall load and display within five seconds” — Scott proposes a model of conversation to collaboratively explore and figure out what the goal is – before testing begins. He calls this “accept / converse / understand.”

2) Am I annoyed?

Scott suggests (and I agree) that it’s actually pretty hard to make decisions on 4.8 seconds verses 6.5 seconds. So instead of working for hard numbers, you can simply have testers write down their annoyance on some scale, perhaps 1-5, at various key points as the test suite runs. (Ideally, do it under simulated load; but if performance stinks for one user, that will tell you a lot in and of yourself.)

3) Watch the trends

Scott suggested that developers can write timers, wrap unit tests in those timers, and simply observe trends to see if the software is getting faster or slower. Likewise, the team can do the same thing to document acceptance and human-run testing. Several people in the room observed performance with a stop watch.

That’s just the tip of the iceberg. Most of Scott’s talk wasn’t on specific technique, but instead on dynamics of projects and general systems thinking skills.

Then, in the afternoon, we had a panel on innovations in performance testing, then a quick mini peer conference where we shared our experiences with perf testing.

In my book, the peer conference idea beats PowerPoint-driven lectures by a mile. Still, I have to say, Scott’s talk was pretty good.


Oct 13 2009   6:25PM GMT

Is your software test team rigorously incompetent?



Posted by: Jan Stafford
James Bach, Software testing, software perils, Software consulting

What happens when a software development or test team’s manager preaches rigor in processes, but doesn’t make sure that team members do processes correctly? “Rigorous incompetence,” says veteran tester and author James Bach. In this video, shot at the recent Software Quality Engineering STARWEST Conference, he shares his views on managers who insist their teams look busy, but don’t ensure that they’re doing their jobs well.



Related Content
Testing rigorously, learning on your own: A chat with James Bach
Author and founder of Satisfice Inc., a software testing and quality assurance company, describes best practices in testing and learning.

James Bach interview: Dispelling software testing myths
The current drive for more rigorous software testing processes and more test documentation is misguided and wastes time and money, says James Bach, author and software testing veteran.


Oct 12 2009   2:24AM GMT

How software test teams’ people skills affect results



Posted by: Michael Kelly
Software testing, software testing teams, Karen Johnson

I wasn’t always the brilliant, charismatic, well-loved tester that I am today. Well, I’m not really any of those things today, but I often like to think I am. That’s my problem. Sometimes my ego gets in the way of my best work. I’m also susceptible to all the bad behaviors that come with tight deadlines, layoffs and work I don’t particularly like. Like everyone else, I can let stress tax my professional relationships.

Karen Johnson will be tackling this problem head on during her talk — titled “Building Alliances” — at the Pacific Northwest Software Quality Conference (PNSQC), which takes place Oct 26-28, 2009 in Portland Oregon. Karen will look at how to create and foster successful professional relationships with teams and people. That means not just fostering a collaborative spirit, but also looking for ways that teams can look out for each other and work together.

“The talk focuses on building healthy relationships at work,” Johnson told me recently. “In theory we have a relationship with everyone we work with, at least at some level. But how many relationships do you have on your team or at your company that you can count on when the going gets tough? The answer is usually a handful of carefully maintained alliances. Those alliances are the solid dependable work relationships my talk is focused on. How do those alliances get built? And how are those relationships maintained?”

Karen Johnson is an independent software test consultant and has been involved in software testing for more than two decades. She has extensive test management experience, and her work often focuses on strategic planning. A frequent speaker at major conferences, she has published numerous articles and recorded webcasts on software testing, many of them here on SearchSoftwareQuality.com. She is also a contributing author to “Beautiful Testing,” a O’Reilly book coming out later this year.

Most recently, Johnson’s focus has been on developing a sense of community for software testers working in the area of regulated software testing. She’s been studying office politics and how they affect people. In her PNSQC session, she’ll reveal her finding on positive practices of healthy, functioning teams and how those practices build good, collaborative relationships.

“I find it interesting that sometimes people in our software testing community recollect with horror painful people events that have taken place from their past work experiences,” she said. “And sometimes people talk about successful alliances at work and people they’ve been able to depend on. But what I haven’t heard people talk about is how to go about creating those healthy relationships. After a number of years of working in a variety of companies, I have a collection of positive and negative experiences. My talk draws from both.

“I don’t like politics. I’m not especially good at them and earlier in my career, I figured I could avoid politics altogether. Once I became a test manager, I realized, I couldn’t avoid politics. Much of what I wanted to say and plan to say, comes from my experiences, my stories and from my heart of what I’ve learned.”

In typical Karen Johnson style, her talk will also be well referenced. Karen told me that in addition to her own experiences, the talk also draws on the work of others. “There is an amazing amount of material on the topic,” said Karen. “These references are especially good reads…”

Clearly, her talk is not focused on the technical aspects of software development. It’s about people. For those who might not feel topics like this are important, Karen offers the following:

“Let me offer this thought: One of the skills I’ve focused on acquiring and sharpening over the years is being observant. When I test software, I’m making observations. When I work with people, I use my observation skills to notice relationships and dynamics. I need to be able to observe changes with people and teams in order to get work done. It’s not just the raw technical skills that get the job done.

“If you want to test software and quietly report defects, I suppose technical skills might be enough. If you want to be someone who has influence to improve a product, you have to be able to work with people. I think it makes sense to have talks at our testing conferences that address people topics.”

Karen will also be teaching a full day tutorial on SQL at PNSQC. The class is a more advanced class focused primarily on joins and queries. For more on the upcoming show, check out the PNSQC website. You can also learn more on Karen Johnson’s website, Testing Reflections blog or by checking out some of her Expert Responses and Tips on SearchSoftwareQuality.com.


Oct 8 2009   7:31PM GMT

Testing rigorously, learning on your own: A chat with James Bach



Posted by: Jan Stafford
Starwest, Software testing

My conversation with James Bach yesterday was certainly the high spot of my StarWest 2009 experience. We talked about topics dear to our hearts, such as critical thinking, self-education, wrong-headed notions about what constitutes good testing practices, Shakespeare and more.

Bach is author and founder of Satisfice Inc., a software testing and quality assurance company, but mostly he’s known as an articulate, passionate writer and speaker. His philosophy of testing is controversial, because he believes that what software testers know from experience and self-education is as valuable, if not more important, than certifications.

He knows, he told me, that certifications and degree help testers get jobs from strangers. Both “pieces of paper” serve as verification that the person has the skills needed for the job. Unfortunately, neither really verifies that the holder can think critically or creatively about what’s needed to do the job in various situations or work on a team.

That said, Bach is much more interested in personal fulfillment and the acquisition of real and not rote learning. He serves up his lifelong self-eduction journey in his new book, Secrets of a Buccaneer-Scholar. It’s taken years to write, he said, because he had to have experiences before he could write about them.

Besides self-education, Bach is currently on a quest to get test organizations to re-examine the concept of rigor in software testing. “Too often, they’re doing their testing rigorously, but they’re also doing their testing incorrectly,” he told me. Rigorously doing something wrongly compounds problems instead of solving them. He sees too many test managers equating success with the large number of tests run and defects found instead of effectiveness. You can read more of his views on test rigor in Mike Kelly’s interview with him, titled James Bach interview: Dispelling software testing myths

As I said, our conversation took off on many tangents, touching on our love for Kenneth Branaugh’s Shakespeare films and the cool things about testers, one of which is their curiosity. This experience was what actually going to conferences is all about, meeting people in person and talking about a little bit of everything.


Oct 5 2009   2:30PM GMT

James Bach interview: Dispelling software testing myths



Posted by: Michael Kelly
James Bach, Software testing

Software consultant, trainer and author James Bach is a bit of a lightning rod in our industry. Most of his notoriety comes from his stand on software testing certifications. He doesn’t like them and isn’t shy about letting people know it. Bach also challenges other widely held beliefs in our industry.

Bach takes his off-the-certification-track testing tutorial, How to teach yourself testing, to StarWest 2009 in Anaheim this week. He’ll also take on some of the myths about rigor in software testing at the Pacific Northwest Software Quality Conference (PNSQC), which takes place Oct 26-28, 2009 in Portland Oregon.

The software myths talk gathers into one place and in succinct form arguments that Bach has been making indirectly for years.

“I keep hearing that more rigor is good and less rigor is bad. Some managers who’ve never studied testing, never done testing, probably have never even seen testing up close, nevertheless insist that it be rigorously planned in advance and fully documented. This is a cancer of ignorance that hobbles our craft.”

Bach’s talk is about clearing up what he calls the “silliness and sloppiness” surrounding the notion of rigorous processes.

“Managers want to say ‘let’s get a lot more rigorous in our processes!’ They may say ‘formal’ or ‘repeatable,’ but it’s all the same sort of thing,” he told me. “But getting rigorous is no panacea. It’s actually a bad idea, in many cases, because rigor can interfere with learning by locking us into bad practice. We need to apply rigor without obsession or compulsion, and at the right level, so that our testing is flexible and inexpensive.”

Bach has been a test manager or consulting tester since Apple lured him from a programming career in 1987. He spent about 10 years in Silicon Valley before going independent and traveling the world teaching rapid software testing skills. He is the author of Lessons Learned in Software Testing, and a new book, Secrets of a Buccaneer-Scholar: How Self-Education and the Pursuit of Passion Can Lead to a Lifetime of Success.

I asked him to offer an example of one of the myths facing our industry. He shared some of the impact of detailed documentation around testing processes and procedures:

“One myth is that a process becomes more rigorous just because it is written down. Well, as anyone ought to know who works with procedures, what is written is not necessarily what is practiced. In fact, it rarely is, in my experience. Moreover, it is not even possible to write down everything that matters about the processes that skilled people use.

“The impact of this myth is that a huge amount of money and time is wasted trying to chase an unhelpful obsession with documentation. It makes good theater - you look busy - but it doesn’t necessarily help you do better work.”

Bach has recently been training to fly a float plane. During PNWSQC he plans to draw some examples from that experience, such as the fact that pilots have to learn many procedures and protocols in order to fly safely. “Rigor is an interesting challenge for a pilot.” he said. “In the talk, I talk about how we use checklists and some of the problems with those checklists.”

A lot of the philosophy behind James’ stance can be found in his latest book:

“Compulsory public education is an example of rigor myths getting drunk and going wild. Many millions of my fellow humans have bought into the idea that education is possible only through schooling. Rigor applied to education is usually presumed to mean that you subordinate your will to that of a schoolmaster, priest, guru or some other authority. I practice a different sort of rigor — education strictly inseparable from life. My life is my education. My life is my own. Education is not some side activity that I do to prepare for life.

“That applies to testing. I am a tester. That is a big part of my life. So, I can’t accept these silly certification programs and bad standards and process guides. Those are examples of rigor, however they represent bad rigor, and my standards are too high for that. I’m trying to raise the standards of the industry so that it will laugh at bad work instead of enshrining it.”

You can learn more about James Bach’s testing practice on Satisfice.com. You can also follow him on Twitter.


Sep 29 2009   9:49PM GMT

At the movies: Exploratory, performance, security testing a kiosk



Posted by: Michael Kelly
Add new tag, Software testing, exploratory testing, software performance testing

Whenever I walk into a movie theater, I remember when I tested a self-service ticket machine. No one was paying me to test the kiosk. I was just killing time, waiting at a theater for someone to join me to watch a movie. The machine looked and functioned similar to an ATM. You select your movie, slide your credit card and print your tickets. What was great about the opportunity is that it allowed me to practice exploratory testing, usability testing, performance testing and security testing all at once.

I discovered that “playing” with the kiosk nicely illustrated what software testers do every day.

The system would allow you to select up to 10 tickets for each type of ticket you could purchase: adult, child and senior. While testing the limits of ticket selection and the proper calculation of the total amount, I noticed that if you max out the number of tickets for senior- and child-priced tickets, the system would beep at you each time you tried to select more then ten tickets. However, when you attempted to select more then ten tickets priced for adults, there was no beep. It made me wonder about the beep. Was it a usability feature?

After I was done doing my functional analysis of the system I had a chance to do some usability testing by watching people interact with the system. I noticed one case in particular that showed what I consider to be a serious defect. A lady using the system selected her movie, entered her credit card information and started waiting as the screen displayed the message: “Please wait while processing your transaction.” I assume that at this point the system was attempting to connect to whatever service it uses to process credit cards.

As luck would have it, at that moment credit card processing for the theater went down. I know this due to the very vocal population of customers at the ticket counter. Unfortunately for the lady making her self-service purchase, the ticket machine seemed to have hung as well. It just sat there saying “Please wait while processing your transaction.” No message saying: “Timed out while connecting to service. Please try again.” No message saying: “Trying your transaction again, please wait.” Nothing. It just sat there.

After about five minutes, the lady finally lost her patience and started pushing the cancel button. She pushed it once. She pushed it a second time - harder. She then pushed it five times in rapid succession. She then put all of her weight into the pushing of the button and kept the button down for several seconds. This processed continued for some time. I counted as she pushed the button over 40 times. Still the screen read: “Please wait while processing your transaction.” So much for the cancel option! She then left the machine and went to the ticket counter for help.

I found other issues while testing, but what stands out for me when reviewing this experience is not the issues I found, but that the process of finding issues “in the wild” is the same that we use “in the lab.” There was setup and configuration for my testing: show times; my credit card; connectivity to the bank; real users I could observe; and my watch to time transaction response times.

There was interaction with the system: myself and others pushing buttons: the system with the bank: the system with the system at the counter that the clerks used; customers swiping cards; and the system printing tickets and receipts.

There was observation of results: noticing beeps and information on the screen; looking at my receipt and tickets; looking at the time on my watch; listening to customer reactions and the conversations at the counter; and seeing the actions the user took under stress.

I was able to draw conclusions based on those observations: the need for better error messaging in the system; the probability of a bug around the beeping for adults; and the fact that the cancel key sticks could be due to multiple people applying fifty pounds of pressure for extended periods of time.

Does that testing process sound familiar?

I like this memory because it illustrates all the basic mechanics of software testing, regardless of the type. It doesn’t matter if it’s functional testing, usability testing, performance testing, security testing, or even automated testing:

  • Testing almost always requires basic setup and system configuration.
  • Testing requires that someone operate the test system or interact with it in some way.
  • Testing requires that someone observe the results of those interactions.
  • Testing requires that someone evaluate those results and draw conclusions.

What’s even better is that I learned something while waiting!


Sep 23 2009   2:31PM GMT

Getting a jump start on software test collaboration



Posted by: Michael Kelly
Software testing, software testing teams, software development

Having tested software during many projects, I’ve seen that the most effective testers are the ones who start early — and I don’t mean the ones who start testing early. I’ll explain what I mean with this step-by-step tour of a pattern I’ve noticed multiple times in software testing projects.

1. A project starts.

2. All the testers work very hard to understand the problem they are trying to solve.

3. Eventually, some small amount of “working” — a very subjective term — software gets delivered to a development environment for the developers to do their unit testing and debugging.

4. One group of testers — often those doing exploratory testing or those working in an agile project context — ask the development team if they can get involved and help test that early code. They don’t care if they can’t log “defects,” because they just want to see the product and provide feedback when and if the developers think it would be helpful.

5. Another group of testers (often those doing scripted testing or those working in more “traditional” corporate testing environments) say they want to wait until unit testing is complete before they start their testing.

6. The group that starts early develops an early collaborative relationship with the developers who let them test their code.

7. Eventually, some small amount of formally “working” software gets delivered to a test environment for all the testers to begin their first test cycle.

8. At this point, both groups of testers start their first test cycle, and both find and log issues.

9. The issues found by the first group — those who worked more closely with the developers — tend to get resolved first, not based on priority or severity, but based on the personal relationship of the tester to the developer.

10. If a defect from that first group — those who worked more closely with the developers — can’t be reproduced, the developer comes over to work with the exploratory tester to isolate the problem.

11. If a defect from the second group — those who waited to start testing — can’t be reproduced, the developer sends it back as “can not reproduce” or the ever famous response, “It works on my machine.”

Before someone points out that “it’s not always practical to start testing early,” and that “there are a lot of good reasons to wait,” I get it. I’m not advocating that in all circumstances you should start testing early. There are plenty of reasons where that might not be practical. However, I’ve never worked in anywhere I would say that’s been the case. And I’ve seen this pattern a lot. It’s not universal, but it’s very common.

The testers who get a jump-start on collaborating with the development often have a closer relationship, and are thus viewed as an asset. It is these testers who are often pressed to help isolate a defects, even if it wasn’t logged by them. They are also the testers who get invited to design review meetings, because their opinions are highly valued.

Does that mean you have to get involved early to be an asset or to have those relationships? Absolutely not. But, I suspect that if you can, you’ll have a better chance of collaborating with people on your project team before pressures are high, stress is up and your “feedback” is viewed as a call for another late night debugging session.


Sep 21 2009   1:48PM GMT

Start with a software test plan template, then throw it away



Posted by: Michael Kelly
software test plans, Software testing, software

There’s nothing more intimidating than a blank sheet of paper. Writers know this to be true, but so do test managers. The easy way out is to pull out a template and to start filling in the various “recommended” sections and details. An even easier approach is to pull out a past test plan and to just start changing project names, diagrams, and technologies. However, these approaches miss the point.

Recently while writing a test plan for a new project, I’ve noticed an odd habit I’ve developed. Ten years ago, when I wrote a test plan I started with a template. Four years ago, if I wrote a test plan I started with a blank sheet of paper. I noticed that when I write a test plan today, I look at templates, decide not to use them, and then end up pulling in pieces of them anyway.

The planning process isn’t about producing a document. Okay, well it shouldn’t be about producing a document. I recognize that in some companies it is. Instead, it’s about thinking about the problem. Software development problems are difficult and solving these problems requires time spent in research, comparing options, and prototyping. Our planning process, in the early stages, is about exploring those options and elaborating on what we think we’ll need to do (and when we’ll need to do it).

I find templates keep me from thinking about the actual problem. Instead they get me thinking about formatting and populating sections that aren’t yet filled in. When I’m using a template, I’m thinking about polish - not content.

However, there’s value to templates. They’re useful checklists for what types of things you should think about. I forget stuff just like anyone else. I’ve gotten a lot of good ideas from templates. So I’ve developed a habit of using a template to “prime the pump.”

I take an initial look at my templates and past test plans and use that to help get me started on problem solving. I’ll then switch over to a blank sheet of paper and start typing out my ideas and thoughts about what we need to test and how we should test it. Later, when I feel I’ve got most of my content, I’ll go back to a template and start pasting the content into the appropriate sections.

This technique keeps me from focusing on polish at the wrong time. There’s nothing wrong with polish, I just don’t want to be thinking about what font to use when I should be thinking about how I’m going to find or generate test data. This technique keeps me free from distractions when what I really need to be doing is focusing on the problem. This helps me deal with some of the intimidation of the blank page, but also allows me to be focused on the difficult topics when that’s what needs to be done.