October, 2009

Oct 25 2009   12:24AM GMT

STPCon wrap-up: SpeedGeeking, testing Web 2.0 applications

Posted by: Matt Heusser
STPcon, software, testing, Conference, Fall 2009, Cambidge

It’s been an amazing week here in Boston while attending the Software Test&Performance Conference.  Friday morning, we ran SpeedGeeking; in the afternoon, I gave my testing Web 2.0 talk, then it was time for dinner and some amazing conversation, and, eventually, sleep.

Similar to lightning talks, SpeedGeeking is a series of seven-minute talks, in rapid fire, one after the other, with one big “BUT”.  Instead of having the whole audience of 100 some-odd people, sitting through the talks in order, we split the room into seven different tables and had the speakers walk around. They chose not to use powerpoint but instead using old-fashioned flip charts and markers.

Justin Hunter, David Gilbert, James Bach, Michael Bolton, Jon Bach, and Scott Barber all lined up, waited for the bell, and … pandemonium ensured. I was stuck between Jon ad James Bach - yes - “inside the Bachs.”

My talk was on productivity on test teams, how teams measure the time they spend testing, as compared to support activities like documentation, status, meetings. Some were surprised to find that as little as 10-30% of time is spent actually testing. Thus, to improve throughput, many test teams don’t need to adopt a fancy methodology or purchase a tool. Instead, they could simply spend more time testing. James Bach covered his “Huh? Really? So?” approach to testing certain claims, while Michael Bolton talked about Testing Vs. Checking. David Gilbert talked about testing as a social, humanistic activity. Aside from some issues with the acoustics, I was really happy with how SpeedGeeking turned out.

Ironically after being stuck “Betwen the Bachs” at SpeedGeeking, the next session I attended was a keynote called “Testing outside the Bachs”, where Jon and James explained exploratory testing

In my “Testing Web 2.0 Applications” talk, I discussed the user-centric web including, ironicaly, blogging, where the buisness model is to have the user’s themselves develop the value, with the tools acting as a enabler.  I covered techniques such as testing with a dirty environment, segmenting the tests, and testing race conditions.  I’ve also placed the slides for the talk online here.

Then I went out to dinner with a few attendees, where I asked for a blueberry beer and got regular beer with physical blueberries in it.  (I was expecting a blueberry flavored beer.) I guess I didn’t specific enough with the requirements, eh?

Oct 24 2009   2:07AM GMT

Astronaut’s STPCon advice: Teamwork delivers “The Right Stuff”

Posted by: Matt Heusser
Mike Mullane, Software Test and Performance Conference, Software project management, Challenger shuttle

I’m in Cambridge, Mass., for the the Software Test and Performance Conference this week.  After Wednesday’s performance test workshop, Thursday was the first regular conference day.

U.S. Air Force Colonel Mike Mullane — now retired after also serving as a space-shuttle astronaut — gave the opening keynote. Mike’s talk was on fundamentals of teamwork. He began by taking us through what happens on the day of a shuttle launch: Suiting up, strapping in, countdown and liftoff.

After explaining the tremendous pressure and risk of a shuttle mission, Mike asked us one question: If you were on a project like that, what kind of team do you want to be working with?

The best, right?

Mike talked about these three fundamentals of teamwork:

• Normalization of deviance;
• Responsibility; and
• Courageous self-leadership

I’ll tell you about one of them, “normalization of deviance.” Yes, Mike has a way with words.

To start with, Colonel Mullane reviewed the tragedy of the Challenger Accident, when O-ring failure causes the destruction of the shuttle. O-rings are basically rubber rings; they create a seal or barrier. You’ve probably seen them on your faucet. In the case of your faucet, they keep the water from leaking, but in the case of the Space Shuttle, they prevent the heat of the engines from leaking out from where the engines connect to the rest of the shuttle, instead forcing the heat down the hole in the bottom.

In the case of the Challenger accident, the o-ring failed, the heat leaked out, the section of the booster melted, this changed the pressure situation and the rockets, essentially, vaporized. This directly resulted in the death of the entire crew, including four astronauts Mullane had trained with directly.

How did this happen? Mullane used a fancy term: normalization of deviance. It basically means taking shortcuts, often under intense schedule or budget pressure. The first time you do this, you probably have no negative consequences. This leads to what Mike called “false feedback,” or that the absence of something bad happening means it was safe to do so.

Eventually the shortcut becomes the norm.

Now the original safeguard was put in place for some reason. Ignore it long enough, and eventually, you’ll have a “predictable surprise.” In almost all cases, the predictable surprise is not good for the team. Colonel Mullane refers to the Challenger accident not as an accident, but as a predictable surprise.

Then he pulls out documents. It turns out that in 14 of the 24 missions before Challenger, the contractor who recovered the solid fuel boosters from the water inspected the boosters and found critical or urgent problems in the o-rings. Colonel Mullane read multiple lettters from the contractor to NASA urging immediate action. One of those letters predicted the failure of the shuttle to the mission number.

The problem? Mission One was a success. So was Mission Two. For that matter, NASA had promised congress 26 missions a year, massively decreasing the cost per mission.

It’s really hard to ground a shuttle fleet when you promised the American people to put up a shuttle every two weeks.

Instead of grounding the fleet, NASA tested a damaged o-ring under pressure, and it worked, so NASA granted a waiver.

This subtle change meant that something intolerable had become expected. To avoid normalization of deviance, Mike recommended that we:

• Recognize we are vulnerable;
• Yes, plan the work and work the plan, but not blindly; and
• Maintain situational awareness.

By the time Mike finished, I found myself tearing up a little.  And the talk only got better.

We software testers may not be flying to the moon, but I submit there are still a few lessons we could stand to learn from the astronauts.

Oct 24 2009   1:41AM GMT

Drilling deep into performance testing at STPCon

Posted by: Matt Heusser
STPcon, software testing and performance test conference, performance testing, Software testing

After yesterday’s Boston SPIN meeting, I got a full night’s sleep, woke up, checked email, and it was time for the full-day workshop on performance testing.

In the morning, Scott Barber presented a half-day workshop on quick and easy performance wins. In the presentation, he split the hard engineering performance testing work — say, simulating and cranking up load –from the “easy stuff.” To do the easy stuff, Scott had four concrete suggestions:

1) Know the mission

Instead of stating with a document — “100% of web pages shall load and display within five seconds” — Scott proposes a model of conversation to collaboratively explore and figure out what the goal is – before testing begins. He calls this “accept / converse / understand.”

2) Am I annoyed?

Scott suggests (and I agree) that it’s actually pretty hard to make decisions on 4.8 seconds verses 6.5 seconds. So instead of working for hard numbers, you can simply have testers write down their annoyance on some scale, perhaps 1-5, at various key points as the test suite runs. (Ideally, do it under simulated load; but if performance stinks for one user, that will tell you a lot in and of yourself.)

3) Watch the trends

Scott suggested that developers can write timers, wrap unit tests in those timers, and simply observe trends to see if the software is getting faster or slower. Likewise, the team can do the same thing to document acceptance and human-run testing. Several people in the room observed performance with a stop watch.

That’s just the tip of the iceberg. Most of Scott’s talk wasn’t on specific technique, but instead on dynamics of projects and general systems thinking skills.

Then, in the afternoon, we had a panel on innovations in performance testing, then a quick mini peer conference where we shared our experiences with perf testing.

In my book, the peer conference idea beats PowerPoint-driven lectures by a mile. Still, I have to say, Scott’s talk was pretty good.

Oct 23 2009   9:11PM GMT

STPCon: How SocialText uses Agile, wikis and remote developers

Posted by: Dan Mondello
STPcon, agile, SocialText, iterations, software testing and development

At the Software Test and Performance Conference (STPCon) this week, I had the pleasure of spending time with Matt Heusser, a software tester for a social networking software development company, SocialText, he is also a SearchSoftwareQuality.com contributor and spoke in a few STPCon sessions this year.

Heusser focused on his development work at SocialText in one session. He told me he had been preparing for months on his presentation on developing software that helps remote workers stay in touch with co-workers and up to date on their businesses, work agendas and projects. Also, he had the challenge of describing how SocialText uses the agile development model.

In the session, he described SocialText’s model step by step, showing such things as how the client’s work request is handed down linearly to the appropriate team chapters. He explained what happened in client-SocialText developer kick-off meetings, noting that all team members attend online using their networking and communication tools. The kickoff is an opportunity for client questions can be asked and requirements clarified, he said. After that initial meeting, the business model becomes a staircase of iteration constructing events. But the whole team has the availability to follow the project, even at phases they are not directly involved with.

SocialText’s programming and products are based on a wiki, making them easy to modify and fairly easy to write the first place. “Anyone dedicated enough, could write something similar,” said Heusser, but SocialText got a head start in this area.

In SocialText’s agile model, Iterations are normally completed bi-weekly. Once approved, the next functional addition is begun on the approved code. This process recycles through all of the testing. Testing is done throughout the entire development process, so that when “crunch time” come, the team doesn’t have to run the full train of tests. Most are already completed, and the few that remain can be held off until a more appropriate time can selected.

Heusser admitted that their process probably wouldn’t work every software development company. “One of the rarest things about working SocialText, is that we use our software to create new software,” he said. “This helps us in the usability testing. If a function doesn’t work properly, we notice it very quickly, as we use every feature numerous times in the development of every product.”

Another rare thing about SocialText is its hiring of many remote developers. Generally, quality of applicant aced location everytime. “In the 21st century, it is ridiculous to rely on 18th century hiring models. Those who worked in mines and on railways had to be geographically placed in the vicinity of their work, of course. That just isn’t and shouldn’t have to be the case these days. We have the technology, why not use it?”

Oct 22 2009   5:35PM GMT

Ways Agile beats waterfall development, boosts software quality

Posted by: Dan Mondello
agile, Agile software development, waterfall, Damon Poole

“How many of you are agile-ists?” Agile expert and author Damon Poole asked attendees this at the SPIN local user group meeting near Boston this week. Then he said: “Agile-ists look like normal people… but if you don’t believe in agile, let me assure you, there is nothing wrong with you.”

Poole gave various examples and analogies of how agile development methodology works, noting that group involvement was a key of agile success. Agile developers benefit from receiving more frequent feedback from customers and having short-term, reachable deadlines, Poole said.

“Agile versus waterfall or any other methodology can be compared to touchdowns versus yardage,” he said. “Where is the credit given? Clearly you can’t win the game based on ridiculous yardage without scoring.”

This analogy points to the major downfall of waterfall, a process in which the team scrambles in all different directions and gains some ground, but often fails to complete the project on time. Agile, he said, is different; you set reachable goals for short phases, or iterations, of development.

This process of “continuous integration” involves completing one set of functional code, getting feedback and then adding new coding to it and retesting. Doing projects in phases assures a higher quality because of multiple tests and retests carried out during the SDLC (Software Development Life Cycle).

“With agile, there is an absence of confusion,” Poole said. “Do you guys remember the childhood game ‘telephone’?” As nearly every hand in the room went skyward, Poole smiled: “How did that game go?” The audience chuckled, and appeared follow his point. If one person tells a group a direction or a goal and the group collaborates there is less chance for project digression. The project is more likely to be what the customer is looking for which is the equivalent of success.

Agile doesn’t only benefit the industry or a company alone, Poole concluded. It also can benefit the individual. He listed these benefits for the agile team member

1. Less cancelled and or shelved work
2. Less pressure or stress over project duration, especially during crunch time as testing is done throughout SDLC
3. More individual ownership and feeling of involvement in a project
4. Increased project success

Poole frequently repeated the phrase, “the simplest thing that could possibly work.” Agile is the simplification of complex tasks, he said, satisfying the customer today and tomorrow, allowing for full software development team collaboration and good returns on investments.

<
Boston SPIN presenter, Damon Poole (on right) speaking with SearchSoftwareQuality.com contributing writer Matt Heusser

Oct 21 2009   4:00AM GMT

First takes on Boston SPIN with Damon Poole and STPCon

Posted by: Matt Heusser

As I write this entry, I’m in the air above Boston, just about to land for the Software Test and Performance Conference. The trip is actually a twofer; I’ll be at the conference tomorrow, but tonight I’ll be out at the Boston SPIN (Software Process Improvement Network), a local user group.

… time passes …

After touchdown at Logan International Airport (Boston), there is no airport shuttle to the conference. Believe it or not, one of the editors at SearchSoftwareQuality.com works and lives in town. Not only is he going to the conference, but he also offered to pick me up at the airport.

And, true to his word, there he is. Also, true to his word, his modified black T-Bird is the loudest car in the parking lot.  I suppose I can’t complain about a favor from a friend.  And the FireBird has some serious horse power.

After I check in at the hotel, it’s time for The SPIN meeting on “Agile Software Development: Does it make a difference?”

Nearly as importantly, there will be pizza.

The Boston SPIN

Yes, there was pizza. So much pizza, in fact, that I missed a little meeting in the corner: a roundtable discussion of Reliability Metrics. Instead, I sit through an informal discussion with a few people who choose to sit at our table; deep enough that Dan Mondello does his own blog post on it.

I do get to see the main event, which was an introduction to Agile Software Development by Damon Poole. Damon is also the author of Do it Yourself Agile, which is nice - and free - e-book on the concept.

The slides from Damon’s talk are online. You can view them here.

A few of the ideas in the talk caught my ear enough to want to share here, in my words:

* We tend to like to think of our work as “done” when we have written code, or run tests, or completed requirements documents. The only “done” that matters is running tested features.

* Development with Legacy Software is just hard to do in general. Yet if you take it a piece at a time and slowly begin to apply agile techniques, over time, the software will become more malleable.

* There is no business value in a complex architecture. Instead of designing for extensibility, just build what the customer asks for and design a system that can be easily changed in the future. (Matt Heusser adds: In my entire career, every time we designed the software to be extensible in a specific direction, that darn customer asked to have it extended in a direction we hadn’t anticipated, and we had to start over at square one.)

* Agile development enables the team to change priorities every iteration.  In this age where a new product like the iPod can radically change the business landscape in a matter of months, it might be better to not be locked into a requirements document with a multi-year release plan.  Just a thought.

The big reveal


At the end, Damon didn’t see a business case for ‘Agile’ development. Instead, he layed out 14 different practices – such as continuous integration, daily standups, refactoring, user stories, and one piece flow. According to Damon, each of these practices can stand on their own, with perhaps a 5% performance improvement.  Because the practices build on each other, doing several of them have a compound-interest effect.

I don’t think Damon meant us to take this literally or scientifically, but merely metaphorically, as a thought experiment. If every Agile practice stands on it’s own, and the lot of them, taken together, provide even more benefit – then why not try an Agile approach?

Oct 20 2009   6:40AM GMT

Boston SPIN: A small group’s big ideas about agile development

Posted by: Dan Mondello

I was invited to attend a SPIN (Software Process Improvement) meeting near Boston. SPIN has been in running for 17+ years and has over 1,100 registered members.

At the meeting, I joined a group gathered around a circular table to finish off my pizza slice and drink my apple juice. The group was already deep in conversation over the topic of our assembly: What exactly is agile software development, and where and how can it be practiced?

For Tru Hong, a QA engineer and analyst, “agile describes the process of telling small stories, small stories that would help in the telling of larger stories.” “Stories” are used commonly in requirements-based in agile, and storytelling is defining long-term goals and then setting shorter goals in which to accomplish the final vision.

Everyone seated at the table had a reasonable working knowledge of agile and its derivatives. While all agreed that agile is their preferred methodology, they also admitted that it’s difficult to define and sometimes to implement.

Two testers from Pembroke, Mass., discussed their experience with agile, which seemed to be similar to that or others at the table. Their company, which will remain anonymous in this post, was recently bought by a rival vendor that uses agile. As a result, the company had begun a restructuring process which included a move to agile. Outside a general knowledge of the term “agile,” his company’s management had only a vague estimation of what the buzzword referenced. So, management didn’t really understand the scope of the move, and it was left to the development team to implement.

Keith Willett, software product director for American Science and Engineering, Inc. (AS&E) located in neighboring Billerica, Mass., spoke highly of his company’s success with Scrum. His team was an early adopter of the methodology, and he’s Scrum-certified. well. Willett’s background, much like the others at the table, was in traditional waterfall approaches. He called waterfall the “the most medieval of software development methodologies,” and others agreed.

Complete adoption of agile is AS&E’s goal, Willett said. His team is making everything agile, everything from approaching a software project to tool usage to the way in which revenue is gathered and distributed amongst the project.

Matt Heusser — a test pro, agile advocate and frequent SearchSoftwareQuality.com contributor — joined the discussion and shared his work experiences with agile. Company-enforced agile had led to his development teams turning out iterations bi-weekly. Sometimes achieving that goal meant “a modest amount of overtime,” Heusser said. Though the deadlines are challenging, he said, he’s found the work completed in agile is more rewarding, because it leaves you with a sense of ownership over a project.

Oct 19 2009   8:29PM GMT

Pulling compliance audits, procedures and practice together

Posted by: Rick Vanover
Rick Vanover

In my last post I mentioned that it is worth the effort when drafting work instructions that match actual practice. Now, I want to take that one step further and hit on a point that can many of us that go through compliance audits.

In a perfect world, compliance audits are a quick check under the hood and a handshake. Unfortunately, they are much more involved than that. One practice point that I want to share that can help the compliance audit practice is related to structuring work instructions and established procedures to meet compliance requirements. For things like software updates and critical vulnerability scans, you can make your life simpler by a little forward thought. Take for instance PCI audit requirements for software update frequency, if your work procedures are written to the requirements of PCI you are one step ahead of the game. Further, if you address your authoritative system inventory with a procedure that is used to perform periodic security scans; the compliance requirements, documentation requirements and actual practice go full-circle without any additional work.

Saving additional work is critical. Having disparate compliance efforts, documented procedures, and actual practices are just asking for things to get out of sync. The investment in making the procedures match compliance requirements is not a new one, but common sense for those with any regulatory footprint. This can assist staff that are new to compliance requirements as well as make business plans clear in time-critical situations such as a company acquisition.

Do you write your procedural inventory to the compliance requirements? If so, share your efficiency tips below.

Oct 13 2009   8:44PM GMT

Project management consultant/author extends savings to SearchSoftwareQuality readers

Posted by: Dan Mondello
Add new tag, Barbee Davis

Barbee Davis is an experienced Project management consultant and author who routinely writes for a semi-monthly international publication, Community Post on project manager concerns as well as ways to successfully negotiate projects.

She is offering our readers this 30% savings coupon on her latest project management book published through O’Reilly: 97 Things Every Project Manager Should Know.

To receive this 30% title savings, apply this promotional code ABF09 through the O’Reilly website

For more information on Davis click here.

---

Related Content
Software expert on Agile’s rise, avoiding project management mistakes
Software project management consultant opines on importance of agile development, common errors in PM, PM career preparation and more.

Oct 13 2009   6:25PM GMT

Is your software test team rigorously incompetent?

Posted by: Jan Stafford
James Bach, Software testing, software perils, Software consulting

What happens when a software development or test team’s manager preaches rigor in processes, but doesn’t make sure that team members do processes correctly? “Rigorous incompetence,” says veteran tester and author James Bach. In this video, shot at the recent Software Quality Engineering STARWEST Conference, he shares his views on managers who insist their teams look busy, but don’t ensure that they’re doing their jobs well.

---

Related Content
Testing rigorously, learning on your own: A chat with James Bach
Author and founder of Satisfice Inc., a software testing and quality assurance company, describes best practices in testing and learning.

James Bach interview: Dispelling software testing myths
The current drive for more rigorous software testing processes and more test documentation is misguided and wastes time and money, says James Bach, author and software testing veteran.