tech conference

CHATing about CAST 2009: Software testing and cultural history

If you’re looking for a cross-discipline topic related to software testing, Rebecca Fiedler’s and Cem Kaner’s upcoming talk at the Conference for the Association for Software Testing (CAST) - July 13-16th in Colorado Springs - might just be for you. Their talk on “Cultural-Historical Activity Theory: Framework, to characterize the activity of software testing” takes a look at one of the most difficult tasks in software testing - discovering and applying contextual information. Cultural-Historical Activity Theory (or CHAT) has been applied widely to software usability analysis, but not so much to testing. Fiedler and Kaner are hoping to change that.

“Cultural Historical Activity Theory provides a clear structure for applying a systems theory-approach to human activities. In particular, it is really useful for looking at change on a human system. Perhaps you’re trying to understand a change that has caused your project to go off the rails or maybe you’ll use it to analyze the introduction of a new tool or technology you’re trying to implement. The Computer-Human Interaction and Computer-Supported Cooperative Work crowds have been using Activity Theory for years. More recently, they’ve begun shifting from user-focused design to context-centered design. It seemed natural, given our advocacy for context-driven testing, to use CHAT to think about the context of testing as well as the context in which the software we’re testing will be used. CHAT helps with that.”

Rebecca Fiedler is an Assistant Professor in the Curriculum, Instruction, and Media Technology Department at Indiana State University. She’s interested in how people learn and how technology can make educational efforts more effective and more accessible to more people. In the testing community, she works with Cem Kaner on the Black Box Software Testing (BBST) courses and AST’s Education SIG. She is also a regular attendee at the Workshop on Teaching Software Testing.

Cem Kaner has pursued a multidisciplinary career centered on the theme of the satisfaction and safety of software customers and software-related workers. With a law degree (practice focused on the law of software quality), a doctorate in Experimental Psychology, and 17 years in the Silicon Valley software industry, Dr. Kaner joined Florida Institute of Technology as Professor of Software Engineering in 2000. Dr. Kaner is senior author of three books: “Testing Computer Software,” “Bad Software,” and “Lessons Learned in Software Testing.”

Fiedler says the idea for the talk started when she was doing her dissertation research a couple of years ago:

“I’m interested in how technology can help people learn and so I spent a lot of time at two different institutions watching college students use a specialized software tool for a high stakes task - high stakes as in their graduation depended on it. Academics love theory so I decided to use CHAT to sharpen the focus of my observations, interviews, and analysis. As Cem and I talked about what I was finding in my research, we started asking, ‘Where was their test group? How could they defer that bug? It does what?’ and other tester-like questions. It wasn’t long before Cem realized this would be a great model for testers to use. We started floating this with some of his testing colleagues and got more and more excited about it.”

Here are a few examples of challenges faced by testers that a CHAT-based analysis might help us better understand and thus more effectively work within:

• Introducing a new metric
• Introducing a new test tool
• Interviewing stakeholders to gather their requirements and to discover the conflicts among stakeholders’ requirements
• Designing tests that are tailored to expose highly important problems
• Describing failures in ways that are intended to motivate specific stakeholders to demand fixes
• Gaining insight into the dynamics of a failing project

Fiedler and Kaner have some concern that some people might find CHAT too complex to master in the short time available. That’s one of the reasons they choose CAST as the venue for their talk.

“I like that the CAST format requires audience and speaker interaction. Attendees get to explore a topic until they’ve heard enough. I also like that the conference isn’t over-scheduled so that you can have lunch or dinner and an extended conversation with speakers and other attendees. I’ve presented CHAT before. On the speaker side, I can tell you that it takes a while to convey the richness of the model. On the listener side, it takes a while to appreciate how it can be used. CAST gives us the time we need to talk about this.

In addition, Fiedler and Kaner indicated they would be willing to take the discussion online after the conference if there were enough interest. “If enough people are interested,” Fiedler said, “we could participate in a discussion forum at AST or TestingClub in which participants apply this to their real examples/situations.”

I asked Fiedler where testers who might not be able to attend the conference could go for more information. She listed off a handful of papers and books that she’s used to help her develop her understanding of the method.

“Yrjo Engestrom (from Helsinki, Finland) developed the CHAT model and first wrote about it in a paper called ‘Learning by Expanding: An Activity - Theoretical Approach to Developmental Research.’ That’s the seminal work but I thought it was a tough read. A few years ago, Sasha Barab, Michael Evans, and Un-Ok Baek wrote a chapter on using CHAT that appeared in the ‘Handbook of Research on Educational Communications and Technology.’ That chapter was very helpful. […] Right now I’m reading two books and I think I’m going to start recommending them to anyone interested in CHAT. They are ‘Activity Centered Design: An Ecological Approach to Designing Smart Tool and Usable Systems‘ by Geri Gay and Helen Hembrooke and ‘Acting with Technology: Activity Theory and Interaction Design‘ by Victor Kaptelinin and Bonnie A. Nardi. Both are grounded in the HCI field, but I think they’ll be helpful to testers, too.”

For more on the upcoming show, check out the CAST conference website. For more on Rebecca Fiedler and online teaching and learning, you can follow her research on her website. For more on Cem Kaner, you can check out his website, or one of his books on software testing.

Good metrics critical to software projects, CAST keynoter says

At this year’s Conference for the Association for Software Testing (CAST), taking place July 13-16th in Colorado Springs, Dr. Jonathan Koomey will be delivering the keynote address on “Real-life lessons for responsible use of data and analysis in decision making.”

In the keynote, Dr. Koomey is planning to present a few recent examples of widely-cited statistics that were grossly misleading or wrong. He’ll then use those examples to summarize real-world lessons for attendees so they can immediately improve their use of data and analysis.

“My presentation will delve into often ignored aspects of the art of problem solving,” said Dr. Koomey. “Including the crucial distinction between facts and values, the dangers of ill-considered assumptions, the need for transparent documentation, and the importance of consistent comparisons.”

Dr. Koomey is a Project Scientist at Lawrence Berkeley National Laboratory and a consulting professor at Stanford University. He is one of the leading international experts on electricity used by computers, office equipment, and data centers, and is the author or co-author of eight books and more than one hundred and fifty articles and reports on energy and environmental economics, technology, forecasting, and policy. He has been quoted in many major media sources, including the New York Times, Wall Street Journal, Barrons, The Financial Times, The Washington Post, Science, Science News, American Scientist and more.

I opened the interview by asking Dr. Koomey about his latest book, Turning Numbers into Knowledge: Mastering the Art of Problem Solving, now out in its second edition.

“The book summarizes what I’ve learned over the years in doing analysis and supervising analysts at Lawrence Berkeley National Laboratory. I’ve hired dozens of analysts over the years but I grew frustrated that I kept having to explain to them basic aspects of the art of analysis, like making good tables and graphs, what constitutes complete documentation, and how to structure a technical report. The book, and this talk, grew out of that frustration. It summarizes the craft of research in a way that is useful for inexperienced analysts, but it’s also a good refresher for those who’ve been in the field for a while. And I tried to make it a fun read, with lots of short chapters, cartoons, and funny graphics.”

When I asked what influenced the talk and the book, Dr. Koomey cited a couple of sources that were influential to him. Those included Edward Tufte’s The Visual Display of Quantitative Information and William Hughes’ 1997 book, Critical Thinking: An Introduction to the Basic Skills. Dr. Koomey also added: “Also surprisingly enough, I was inspired by Zen in the Martial Arts by Joe Hyams. The fluidity and readability of that book influenced the structure of Turning Numbers into Knowledge.”

In addition to his keynote at CAST, Dr. Koomey is also leading a workshop on his new book.

“The workshop will be an interactive exploration of how managers can encourage, prompt, and cajole their employees to give them the numbers they need to make good decisions. It will also help both managers and analysts hone their own analytical skills. Even if you’re a seasoned analyst, the exercises in the workshop will get you to think afresh about the challenges you face at work, and should help you become more effective at your job.”

This will be Dr. Koomey’s first time speaking at CAST, and I asked him if he was looking forward to getting to know the software testing community. “The essence of good software testing is critical thinking, and I’m happy to be in the company of smart people who use their problem-solving skills in new and innovative ways. I always learn something when I attend conferences like this.”

For more on the upcoming show, check out the CAST conference website.

CAST 2009: The challenges of regulation, an interview with Jean Ann Harrison

Veteran software testing and quality assurance pro Jean Ann Harrison will be presenting a software testing case study based on her experiences at a medical device study at this year’s Conference for the Association for Software Testing (CAST), slated for July 13-16th in Colorado Springs.

In her session — titled “A Balancing Act: Satisfying Regulators, End Users, Business Leaders and Development” — Harrison plans to provide guidance to testers who have to deal with conflicting priorities between developers, project managers, customers/patients, and regulators.

“Priorities clash and inevitably software testers are in the middle of a battlefield between developers trying to get their work done and delivered while project managers are trying to make a deadline, customers/patients want to make sure the product works as expected and the regulators demand the proper documentation delivered in a sequential timeframe.” Harrison went on to share some of the questions she hopes to answer in the talk. “How do testers balance all these shareholder’s priorities? How can testers decide which direction to take as the project matures? Which shareholders’ take precedence over another?”

With 10 years of experience in software quality assurance and testing and three years testing embedded software on portable devices, Jean Ann Harrison has gained broad experience in various software testing processes and has worked in varied contexts including large multi-million dollar corporations, venture capital, and start-up companies. Harrison is currently works for CardioNet, Inc. where her primary role is testing software embedded in medical devices that provide diagnostic data for physicians to determine their patient’s heart condition.

“I developed the talk through my own learning process as a software tester working in a regulated environment for the first time. What to do, what not to do, what one can expect, and how to handle the demands of a regulated company helped formulate my subject. And most companies producing software usually have some sort of description of what is wanted, needed, or expected when the project is completed. Most companies usually have some method of traceability of software requirements, software design, and product information. In a regulated environment, the role of documentation is the centerpiece of any project.”

When asked to expand a bit on the challenges of regulation, Harrison continued:

“First, a single source location for documentation must be identified, implemented and then monitored. Then documentation is distinct in a regulated environment by the level of detail provided, the sequence of submittal of documentation, identifying appropriate reviewers to approve the documentation, and a historical record is maintained for traceability purposes. This process is extremely formalized and dates of submittals are critical to the project. Non-regulated environments tend to be more relaxed and even the most formal processes have allowable slips. In regulated environments, slips are not acceptable, and contingency plans must be implemented to explain deviations. If regulated environments do not meet regulators demands, the certifications are rescinded.”

One of the things I found most interesting about my interview with Jean Ann Harrison was her biggest influence for the talk, which came not from the field of testing, but instead from political science. Harrison majored in Political Science 25 years ago. She’s found that the analytical thinking skills her professors emphasized play a large part in her success.

“In the four years and loads of courses, exercises were given to force students to practice analytical thinking. Software testers are constantly required to analyze how to do something, how to improve, what is the data telling you, analyze different perspectives, create. Over the years, my analytical skills have evolved but certainly were given a solid foundation because two professors teaching the subject of Political Science felt the skill was critical to the coursework. One exercise that was given to me in a course called Research Methods, I use today to train and mentor software testers. It is simplistic in nature but very difficult to implement. The exercise requires them to generate a new hypothesis that they personally have not read about, been trained in, or been given any sort of research material on. Then they are required to describe and prove the hypothesis using empirical means.”

When asked why she chose CAST as the venue for her talk, Harrison shared that this year’s theme for the conference, “Serving our stakeholders,” is directly relevant to some of the lessons her current company is learning, as it’s experiencing growth. “Each department is learning who the customers are,” Harrison says, “How we can better be of service, and what can we learn from our mistakes.”

For more on the upcoming show, check out the CAST conference website.