July, 2009

Software testing blog digest: Bug, team woes; memes; Ford Motors

If you don’t read software testing blogs, you’re missing some great advice and thoughtful ramblings on testing philosophies. I tap into those blogs daily, and here I’m sharing the wealth with this reader’s digest of the testing posts I enjoyed this week.

Why bugs are hard to kill

On Maverick Tester, Anne-Marie Charrett describes the mistakes she’s made when doing offsite exploratory testing under tight deadlines. Then, she reveals how she’s stopped making those mistakes in her list of offsite exploratory testing guidelines to bug reporting.

One tidbit of her advice: Write reports right away, even if you are super-busy. She writes that “it takes longer to write them up at the end, when you have to review heaps of cryptic phrases in Session Tester or in your notebook.”

I love the post’s title, “Do your bugs only glow when it’s dark?” It reminds me of the “putting out fires” metaphor. How many times have I gotten emails from co-workers, site experts and others saying they’re late with a response or a deliverable because they’ve been putting out fires? Hey, I’m guilty, too.

In my own work, I see that most of these fires were started when haste made waste. Why is it so hard to take things one step at a time? Oh yeah, there’s a deadline and not enough time to make it.

When familiarity breeds success
Moving on, two posts on Matthew Heusser’s Creative Chaos blog explore thought-provoking topics: team cohesiveness and memes. In his post on Jelled Teams, he ponders the good results of working on a team that’s been together for over a year. How much creativity and productivity is lost, he asks, when companies often shift people from team to team as casually as they do? Too few managers realize that teamwork flourishes when people know each other well enough to feel comfortable sharing their ideas. When a team works well together, it’s an added-value asset in and of itself.

So, project managers, think twice before breaking up good teams!

I see a connection between that post and Heusser’s musings today in The meme’s the thing. Wikipedia calls a meme “a postulated unit or element of cultural ideas, symbols or practices, and is transmitted from one mind to another through speech, gestures, rituals, or other imitable phenomena.” Good grief! I think Heusser’s short definition is better: “It’s an idea - a concept that spreads from person to person. “

Any married person knows that familiarity and mind-melds go hand-in-hand. It stands to reason that team members that’s been together a while will start understanding how each other thinks, and the ideas will start flowing. Community work along the same lines. That’s why, I think, the open source software community has made such great strides so quickly. Another is that open sourcers are so communicative and have created vehicles – sites, projects, message boards and so on – that foster collaboration.

Heusser believes that software testers should be thinking along the same lines and said:

“I believe that the communities I belong to…have ways to test software that are significantly better than the status quo, and we have ways to communicate them and techniques to teach them. Yet if our testing ideas are memes, we need to think about ways to package and present them to win.”

Carrying on with the teamwork theme, there’s a nice exchange on the topic of how to handle unhappy testing teams on Jerry Weinberg’s blog, The Secrets of Consulting. A software test manager at an insurance company wrote to Weinberg, and they –- and others – brainstorm on the subject in an informative message chain.

On the lighter side

Once you’re a software tester, you look at everything from that point of view. So, Software Quality Insights blogger and independent consultant Mike Kelly describes Ford Motor’s web application flaws whe he was trying to spec a new Ford truck. In his entertaining post, he concludes that it’s easier to build and buy a Toyota online. This is something Washington has missed when discussing bailouts and the state of U.S. auto companies, I think.

There were plenty of other good reads in testing posts this week, more than I can cover here. Please comment below if you read something good this week or have a favorite testing blog.

Tester’s view: IBM buys source code analysis company

In a press release yesterday, IBM announced it would be acquiring Ounce Labs Inc., whose software helps companies reduce the risks and costs associated with security and compliance concerns. IBM will integrate Ounce Labs products into its Rational software business.

For those who might not be familiar, the current lineup of Ounce products include:

• Ounce Core is their security source code analysis engine, used to assess code, enforce rules and policies, and it houses the Ounce security knowledgebase
• Ounce Security Analyst scans, triages and assigns results, and manages security policies allowing you to take action on priority vulnerabilities.
• Ounce Portfolio Manager delivers at-a-glance metrics and information to manage risk enterprise-wide.
• Ounce Automation Server augments Ounce Core by integrating and automating scanning, publishing, and reporting in build environments.
• Ounce Developer Plug-Ins helps pinpoint vulnerabilities and provides remediation advice for rapid fixes.

For those familiar with the latest offerings of IBM Rational, it comes as no surprise that the Ounce Labs products will be offered as part of the IBM Rational AppScan family of Web application security and compliance testing solutions. The current suite of IBM Rational tools (AppScan and Policy Tester) provide some of the basics around security vulnerability scanning, content scanning and compliance testing, but they aren’t as full featured as their competitors products.

When the current Quality Manager suite of tools from Rational came out a year (or so) ago, I was quite happy to see AppScan integrated more closely with the testing products. And over the last several years, Rational has done a better job of integrating their testing and development platforms — moving the tools to a common platform/IDE, etc. Hopefully the addition of the Ounce products will continue that trend of bringing team members together in a common toolset.

For more information on the acquisition, SearchSecurity.com has the full story.

Moving away from NAT for testing

For test and development systems, one practice over the years to allow developers to build test systems or applications has been to use network address translation or NAT. NAT basically puts some device in front of other systems. Development teams can use NAT a number of ways. These include running a virtual machine behind a host’s network, a network appliance or firewall rules.

NAT is bad for testing for a number of reasons. The primary reason is because the test system is behind a (presumed) protected device, there is no pressure to put security as a priority in the test process. Practice points such as weak password, application defaults, unnecessary network configurations and other items leave the test system at risk for propagating poor configuration and practice forward in the lifecycle.

Instead of using NAT, many organizations are using dedicated networks for test and development purposes. There can be firewall rules in and out of the network, yet within the network the test systems are fully present. These dedicated networks can also be configured to be fully isolated or connected upward for important things such as Windows Updates for Microsoft systems.

NAT is a limited in real practice for development cycles. What may not be known is what developers are doing individually with local virtual machines on desktops that may be using NAT.

The governing principle is to treat all levels of test and development with the same network rules that you may subject them to in a live environment.

Using taxonomies to help with test planning

A year ago, I was working on a project where we were doing a failure modes and effects analysis (FMEA) related to failover and recovery. As I was thinking about how to best start my analysis, I recalled that in the past while doing performance testing work I looked at many of the same aspects of the system while planning. As a way to generate ideas, I did some research to identify sources that could help me with my planning. You can take a look at some of the resources I found, or use different taxonomies if you have any that you particularly favor.

Here’s an example of how you might use a resource like this. Let’s take the risks listed in chapter three of Performance Testing Guidance for Web Applications. In the following figure from the book, you’ll see a summary of the risks presented in that chapter.

Figure 1: Performance testing risks, from the book Performance Testing Guidance for Web Applications.

I prefer working with the list of questions the authors have outlined in the chapter, but the graphic does a nice job summarizing things. For each specific risk listed, you want to:

• Ask yourself if you’ve accounted for that risk with your current plan. If you haven’t, figure out if you should. If you think you should, figure out what type of testing would be most appropriate for you. One nice thing about this particular taxonomy is that they give you some guidance there.
• For each risk, move from the generic to the specific. The risk “Can the system be patched or updated without taking it down?” is a great question, and an initial answer might be “yes.” But when I look at the system I currently work with, there are several major systems all working together. I might ask if I can patch all of them. And patch them in what ways; via software, database, run-time dependencies, services, etc.?
• For each risk, ask yourself if there are any slight variations on that risk that might be important to you. Good examples of the practice are the two risks listed in the book: functionality could be compromised under heavy usage; and the application may not stay secure under heavy usage. And you can vary different parts of the same question. In those two risks, they varied the quality criteria — functionality and security — but kept the risks, such as heavy usage, static. You could add other quality criteria or other risks.

The general idea is that you’re using lists like these to help you generate test ideas. In a way, you’re also using them to test the planning work you’ve done so far to make sure you haven’t forgotten or overlooked anything.

New DevSuite 8.0 tools aim to aid multi-project collaboration

TechExcel, a decade-old maker of development tools, released new features to its application lifecycle management software package, DevSuite 8.0. Included are MyWord dashboard engine and wiki tools promise improved team collaboration and status reports on concurrent software projects. Another bow to collaboration support comes in DevSuite 8.0’s new multilingual capabilities and user-definable UI names and values for multiple languages.

When a new product or features are announced, I always wonder what user problems or requests spurred the vendor to invest in developing them. So, when I heard about the DevSuite 8.0 additions, I posed those questions to Paul Unterberg, associate director of product management for Lafayette, Calif.-based TechExcel.

First I asked how users have been getting views and an overview of project status prior to the release of the MyWorld dsashboard engine. Unterberg responded:

“Before we introduced MyWork, the data for an overview was available to a user or a team based on a report. The user had to login, select a project, navigate to the report view, and then run their report. This took a lot of effort. Since the data was already in the system, we simplified the process and put it all in one place.”

My next question: How about the before-and-after picture for integrated wiki tools?

“There was no integrated Wiki before DevSuite 8,” Unterberg said. “This meant that people wishing to collaborate on a requirement or document had few options. They could leave notes to each other, but there was always the risk of someone overwriting another person’s changes. The Wiki simplifies the entire process, and eliminates the risk of user unintentionally erasing another user’s data.”

The overall goal of DevSuite’s integrated set of tools is to marry the strategic and tactical worlds of application development together by creating software that lets management and planning processes co-exist seamlessly with specific task-driven development processes. The team of software tools that enable this relationship provide workflow, process automation, searching, reporting and customization capabilities, among other things.

DevSuite also co-exists with various application development methodologies. For instance, teams using both waterfall and agile processes can live in TechExcel’s ALM framework.

“From our perspective, there should be no relationship between an ALM system and the development methodology a team uses,” Unterberg said. “We’ve heard from many customers the horror stories of their former systems that tried to change the way they worked based on what the system could do.”

It’s better to create processes in the ALM system that change based on how the team works. He described such a situation, saying:

“If a team is agile, for example, they might need less process control and a greater degree of flexibility with how they are able to prioritize work. They might also have the system limit the amount of time they can spend in a certain area; adding a time box to a development iteration, for example. This same functionality might be useless to a non-agile team. A good ALM system should be able to adjust to these needs and give the teams the most flexibility in modeling how work is done.”

Not adding another management layer with ALM is a stringent goal of TechExcel and is played out in DevSuite, Unterberg said. Adding different management when adopting ALM is only necessary if lack of management in a certain area was a driver for the ALM adoption in the first place. “Who is in charge depends greatly on the team and the process they follow,” he concluded. “ALM just enhances, automates and ties that process together.”

CAST 2009: Taking a new look user acceptance testing, an interview with Michael Bolton

User Acceptance Testing (UAT) is a part of most testing plans and projects, but when you ask people what it is, they have a hard time defining it. You quickly find that it isn’t obvious what user acceptance testing means.

I talked to Michael Bolton about his views on UAT this week. He’ll discuss that topic at next week’s Conference for the Association for Software Testing (CAST). When the subject comes to UAT, Bolton said, there’s a lot of miscommunication. “The same words can mean dramatically different things to different people,” he said. He want to help user-acceptance testers “recognize that it’s usually risky to think in terms of what something is, and more helpful to think in terms of what it might be. That helps us to defend ourselves against misunderstanding and being misunderstood.”

Bolton has been teaching software testing on five continents for eight years. He is the co-author of “Rapid Software Testing,” a course that presents a methodology and mindset for testing software expertly in uncertain conditions and under extreme time pressure. He’s also a co-founder of the Toronto Workshops on Software Testing.

Bolton says the idea for his CASTsession first came from a message on a mailing list. In the list, someone suggested the following user acceptance test-driven approach to developing a product:

• Write a bunch of acceptance tests before writing the program
• Write the program
• When the acceptance tests pass, you’re done

“Now to most testers that I know, this sounds crazy. It occurred to me that maybe we should explore alternative interpretations of ‘acceptance tests’ and ‘done,’ but maybe we should also explore the business of exploring alternative interpretations altogether.”

“The language issue has always interested me. Back when I was a project manager for a really successful commercial software company, I noticed that some people weren’t saying what I thought they meant. Others were saying things that I was pretty sure they didn’t mean. I’ve done a paper on user acceptance testing, and I’ve done some classroom work on it, but this is the first time that I’ve taken it to an audience like CAST. “

Bolton wants his talk and his work in this field to trigger discussion. To learn more about his point of view and join that discussion, check out his CAST 2009 presentation. If you can’t make it to Colorado Springs, you can find Bolton’s past articles and conference presentations on his website. You can also follow his work as it unfolds on his blog or via Twitter.

Mike Dwyer at CAST 2009: New simulation helps teams learn Agile

A common criticism of Agile development practices is that they are difficult to scale with large teams. Another common challenge faced, is figuring out where testers can fit in an Agile context. At this year’s Conference for the Association for Software Testing (CAST), July 13-16th in Colorado Springs, Mike Dwyer is presenting a workshop titled “Experiencing Agile Integration.”

I interviewed Dwyer recently about the session, in which he talks about a new simulation that allows participants to experience first-hand what it feels like to apply Agile principles. He said the goal is of the simulation is to provide a simple environment that reflects the dynamics of expanding Agile and enough time for the participants to inspect and adapt how they support the expansion so that the product, team and organization garnish optimum value from going Agile.

Mike Dwyer is a principal Agile coach at BigVisible Solutions working with IT organizations as they adopt Agile and Lean methods. He has extensive experience as a manager, a coach and consultant transforming high growth organizations into hyper-productive Agile organizations. In addition, he is a well-known and respected contributor to the Scrum, Agile, and Lean software community.

A product of BigVisible Solutions team and their shared experiences, the simulation was built after Dwyer and several of his cohorts came off a 280-person, 30-team project. The designers all hold or have held multiple certifications in both PMI and Scrum as well as in IIAB, APICS, and other professional organizations. Many of the BigVisible Solutions team are experienced testers with background in performance, web, application, system, CI, TDD and exploratory work.

“The simulation mirrors what we have seen many organizations do to ‘pilot’ Agile. In order to provide the experience to the attendees and work toward optimal value for them, we use a simple pattern based on Scrum workframes. That is to work at delivery of value for short periods of time, discuss and learn from what we have done and then apply our learning to our next iteration. Value is optimized by the coach/facilitator working with the teams and individuals to inspect and adapt what they are doing to find better ways for the team to reach its goal.”

For more on the upcoming show, check out the CAST conference website. For more about Mike Dwyer and his work, you can checkout his BigVisible blog.

Two experts: Why not to skip some software testing phases

Is software testing really necessary? Do we do it just because everyone else does it? Why is software testing important? While ideas about testing vary, the motive is generally the same: someone believes it has value. We test because someone wants us to test. They may be managers, developers, business executives, regulators, or even customers. But how do testers know if they are doing the right thing or if something is lacking?

I explored these ideas recently in interviews with Neha Thakur, a business technology analyst at Deloitte Consulting, India, and Edgardo Greising, manager of the Performance Testing Laboratory for Centro de Ensayos de Software in Uruguay. Both are speakers at next week’s Conference for the Association for Software Testing (CAST).

Thakur will be exploring this topic in her upcoming talk, “Software Testing - To Be or Not To Be.” She is keenly interested in talking with her peers about identifying and serving testing stakeholders. In her own work, she’s discovered the advantages of identifying and involving stakeholders, and she’ll share methods for stakeholder analysis, gaining stakeholder involvement and making sure stakeholders’ needs are being met by the development team.

Thakur has performed automation testing in a variety of contexts, ranging from medical electronics, to storage and networking, to risk and compliance.

“I have always been curious to know and learn about the various aspects of software testing: the prevalent models, technology, tools etc. This curiosity to learn new things helped me go deep in various management topics and to develop a better understanding of the various stakeholders at each level that might impact the project. It also allows me to be proactive in communicating the risks, issues, and information with the respective stakeholders. I think testing is on an evolutionary path and there are still axioms of test management which need to be improvised.”

“Stakeholders believe in facts and figures; they believe in action and not words merely stated. Thus a subjective way of thinking while testing might not be the correct way of approaching an issue. Thinking objectively always helps. [You need] data, facts, and figures to support [your testing].”

Approaching the problem from another angle, Edgardo Greising plans to look why it can be difficult getting some IT managers to see the value in doing performance testing. According to Greising, this needs to change. In his upcoming CAST talk — titled “Helping Managers To Make Up Their Minds: The ROI of Performance Testing” — Greising plans to explore the risks and costs associated with performance testing.

“I will be talking about the return on investment of a performance test. From my experience, many managers refuse to do performance testing because they think there is a high cost. I always try to illustrate that the cost of not doing performance testing is higher.

Our commercial people have to fight against the cost-myth each time they are visiting potential clients. And, on the other hand, we know that a performance test gives us a lot of information to tune the system and help us avoid system downtime. The objective, then, is to put those things together and show the convenience of performance testing.”

During my interview with Greising, he talked about the ways software performance testing yields great improvements in application health. For one thing, performance testing leads to reducing resources consumed and lowering response times, he said. “With most projects, we are unable to support half the volume expected in production until the tests shows us where the bottlenecks are so we can fix them.”

Unfortunately, Greisling said, performance testing is rarely found as a regular activity in a systems development or migration project. Then, when application deployment approaches at high velocity, nobody has time for even think about it.

Greising is no stranger to keeping cost in mind when testing. He worked as a salesman and pre-sales engineer for 15 years. For him, balancing cost and risk are just a regular part of testing. To talk about cost justification, you need to talk about risk,he said.

For more on the upcoming show, check out the CAST conference website.

CAST 2009 preview: Positioning software testers as service providers

Presenting correct information isn’t just a function of how you write your report at the end of a software project. Instead, it is the result of a complex process that starts with analyzing the needs of your stakeholders, moves on to gathering accurate and timely data from all your multiple project sources, and then results in presenting your findings in the correct format, at the right time and to the proper audience. Joel Montveslisky calls this “Testing Intelligence” and is presenting a talk on that topic at this year’s Conference for the Association for Software Testing (CAST), July 13-16th in Colorado Springs.

“Testing intelligence is a term to describe a slightly different perspective to software testing that places the focus on the needs of our project stakeholders instead of on the application under test. The main idea is to position the testing team as a service organization within the development group, whose purpose is to provide the timely and actionable testing-based visibility needed by the project stakeholders to make their tactical and strategic decisions.”

“In principle this is nothing new, but in practice many testing teams tend to get disconnected from the changing needs of the Organization during the project and end up working for the sake of their own “product coverage needs” or the old information needs of their Project Stakeholders.”

Montvelisky is one of the founders and Product Architect of PractiTest, a company providing an SaaS (Software as a Service) test and quality assurance (QA) management system. He is also a QA consultant specializing in testing processes and a QA Instructor for multiple Israeli Training Centers. A member of the Advisory Board of the Israeli Testing Certification Board (the Israeli chapter of the ISTQB); he publishes articles and a periodic QA blog and is an active speaker in local and international conferences.

According to Montvelisky, the process of gathering testing intelligence starts by correctly identifying your stakeholders, then working with them to understand their needs, and finally providing them with correct and timely information they need. He thinks there are a number of things that make this a hard process, or at least not a trivial process, for testing teams:

“We can start by the fact that many times we are not aware who are all our project stakeholders, we tend to miss some that are not physically close or that enter the project late in the process. Secondly, we testers are not really trained to work with customers [...], so many times we don’t communicate correctly and we assume their needs without consulting with them about what information is important for their decisions, what format they need it, and when. And finally, we don’t take into account the dynamic nature of our projects. We don’t understand that people require specific information at certain times. Nor do we take into account that as the project evolves the information we need to provide changes.”

Montvelisky started developing his CAST talk when he was a QA Manager working for an enterprise software company. There he realized that many stakeholders were frustrated with the existing bureaucracy of the QA work. The stakeholders thought the QA team’s work was dictated by test-planning documents written months beforehand. Montvelisky noticed that those documents were not staying relevant to the current issues affecting the release.

“In this company we made a mind-shift and decided to set aside time during the project for ’specific testing tasks’ that would be given to us by the Development Team in real-time. Soon enough the demand for these tasks increased and we realized that we were providing real value to the process by becoming the eyes and ears of the project. After I left this company and became a consultant I took this approach with me and created a process around it to help organizations make the mind-switch and start working more effectively with their stakeholders throughout the project.”

The chance to get feedback is one reason Montvelisky is excited to be presenting at CAST. “It’s not easy to receive hard criticism,but once you learn to take it in a positive light and use these comments to continue developing your work, it makes it one of the most fruitful encounters for people looking to improve and develop ideas in the field.”

I asked Montvelisky where he thought he might get some pushback on his approach:

“In the past, I’ve heard two main areas of criticism to my approach, both of them fair. First, people explain to me that all their professional lives they’ve worked based on what I describe as testing intelligence, and that this is nothing new to them. To these people I usually come asking for their best practices and asking for their inputs in order to improve my approach.”

“Second, people tell me that our job should limit itself to test, and ‘we should be proud’ of it instead of trying to find big names for what we do, leaving this to the marketing team. To these people, I try to explain that every team in the organization needs to contribute value to the process, and if they think that all their value comes from reporting bugs and coverage percentages then they can continue working like that.”

“Having said that, there is a lot more value that can be provided by the Testing Team, and we don’t need to change what we do in order to provide it we only need to make sure we stay connected with our stakeholders and help them throughout the project and not only at the end of it.”

Montvelisky is currently focusing on a couple of research topics. One of them is related to adding value by correctly utilizing the test management tools in the organization. The other is related to collaboration between testers from different organizations, and different cultures and countries, in order to improve their overall work.

For more on the upcoming show, check out the CAST conference website. For more on Joel Montvelisky and what he’s currently working on, you can follow him on Twitter or his PractiTest QA Blog.

Eight days, 80 testers: Exploratory testing case study at CAST 2009

Software consultant Henrik Andersson implemented an exploratory testing training project in an 80-tester group in only eight days, and he lived to talk about it. Next week, he’ll outline the steps he took to quickly set up a pilot program and train testers in exploratory testing theory and practice during the Conference for the Association for Software Testing (CAST), which takes place July 13-16 in Colorado Springs. In his session, he’ll also cover how he made responsibilities and expectations clear.

I recently interviewed Andersson about his session, titled ”Implementing Exploratory Testing at a Large Organization.” He said his first reaction upon receiving the assignment in question was that it was impossible to implement exploratory testing on this scale in that time frame. To reach out to 80 testers is a challenging thing to do, he said, and it takes time to implement such a different way of testing. Yet, he decided to rise to the challenge. “If I turned it down I would not likely get another chance,” said Andersson, a consultant and founder of House of Test, headquartered in Sweden and China.

Once he accepted the project, he had to figure out how to do the impossible.

“I came up with a little twist on the initial request. I suggested that we should pick one tester from each test team and tutor them to become Exploratory Testing Champions. This gave me an initial group of nine people. This is what we achieved during the 8 days. The Champions would then have the responsibilities to tutor the rest of the testers in their teams. The Champions are now continuously working with this in their test teams, and we have established this new role formally.

Andersson’s case will show what the exploratory testing champions approach achieved. He also will explain in detail how the project was implemented. He’ll describe the workshops on theory and practical exploratory testing that he conducted during theproject. He’ll share observations about tutoring a group of people used to working in a completely different way, the positive feelings and feedback he receive, what surprised him and what approaches did not succeed.

Just so you know that Andersson is no newbie to testing metholodies, here’s some information about his background. As a software tester and consultant, he has worked in a variety of fields, including telecom, medical devices, defense, insurance, SAP and supply chain systems.

For the past 10 years, Andersson has focused on working in a context-driven fashion, mixing exploratory testing with more traditional methods, such as RUP, V-model, TMap and others. However, he has never followed any method by the letter. “I always only took the part that has been useful and invented the parts I was lacking,” said Andersson. “I definitely didn’t do the parts that felt were obstacles or not useful.” Indeed, Andersson enjoys helping organizations transform from the “old school” practices.

For more on the upcoming show, check out the CAST conference website. You can learn more about Henrik Andersson and his company House of Test on their website. You can also follow Henrik on Twitter.