software quality assurance

Side effect of Openbravo on Ubuntu: Easier testing, QA

Openbravo just released its ERP 2.50 Profesional Subscription for Ubuntu, an integrated open source ERP software stack packaged with the Ubuntu operating system. cost-effective commercial open source product that is cloud-deployable via virtual appliance and also available on various platforms.

The new package offers a software testing and quality assurance (QA) benefit, too, according to John Fandl, Director of Product Strategy for Openbravo.

“The QA angle centers on the general benefit of standardization in aiding QA efficiency,” Fandl said.”When you can execute a hands-off installation that runs in an hour, and automatically creates and pre-configures a full ERP stack — including database, web server, application server — that makes it really easy for enterprises to do a proper QA cycle, with separate development, QA, user acceptance and production environments.”

I asked Fandl to carry through that thought to the testing side of things. He said that installing proprietary ERP stracks is difficult, so the QA function often has to compromise and forgo proper testing.” Considering the complexity of ERP, it’s hard to match QA environments actually to production environment. “For example, they may be testing on a different version of the application server or database than is in production, which can cause surprises when the code is promoted to production. Being able to rely on an efficient, automated “full-stack installation” that can be effectively “pulled from the cloud on-demand” is a godsend for QA.”

Fandl has a point, Medicity QA director and SearchSoftwareQuality.com site expert John Overbaugh told me.

“There is definitely value in a clean installation of the entire stack of applications. Anytime a technology is difficult to employ, teams will find a way around it, by either mimicking a clean install or by doing a small amount of machine clean-up before starting in again on testing. This often results in an unreliable environment.”

Fandl elaborated, saying: “Inexperienced testers and especially developers doing unit testing–since QA is not their major focus — may not be as rigorous in regards to testing on a proven-clean environment. Mimicking a clean environment sounds like a time savings, and it does work most of the time…until it doesn’t. The problem comes from subtle environment differences that arise over time between QA and production environments; differences that “shouldn’t matter” until they do! And the way you find out is with a production problem that you can’t duplicate in the QA environment. Ouch.”

So, I asked Fandl, how does the Openbravo-Ubuntu package help testers get clean installs and avoid these ills?

Fandl told me that fully automating the installation, including the entire stack and all of its dependencies, gives the same result everytime, regardless of the starting-point state of the target machine.

“For example, if Tomcat 5.5 is on the machine, the installation package which knows that Tomcat 6.0 is required for Openbravo ERP 2.50) will automatically retrieve it from the Ubuntu repository and will upgrade your server from Tomcat 5.5 to 6.0, before continuing with the Openbravo application installation. So, Ubuntu’s Debian-based package management system transparently takes care of these details, so that the QA person does not have to be an expert in the underlying stack. This is a great help, especially for business-centric QA staff testing ERP, who may not know how to determine what version of a system component like Tomcat is actually running on the server.”

Openbravo did its initial testing –installing the package from the public repository — from a clean instance of Ubuntu 9.04 set up inside of a virtual machine.

With this release, Openbravo is following in the footsteps of other open source ERP and software vendors who are creating easy-to-install stacks. For more information on this trend, check the blogosphere, where Matthew MacKenzie writes about Openbravo and SMB ERP. Also, on the blog, How Software Is Built, Scott Swigart and Sean Campbell interview OpenBravo CTO Paolo Juvara, who oversees product development. Juvara notes that Openbravo provides a foundation upon which developers and users can customize their software, making components proprietary if they wish.

CAST 2009: Challenging one of the classic ideas around testing, an interview with Doug Hoffman

At next month’s Conference of the Association for Software Testing (CAST) in Colorado Springs Doug Hoffman will call to question one of the most fundamental ideas in software testing: Do tests really pass or fail? I had the opportunity to talk with Hoffman about his conference session titled “Why tests don’t pass.”

Doug Hoffman has over thirty years experience in software quality assurance and has earned degrees in Computer Science, Electrical Engineering, and an MBA. He is currently working as an independent with Software Quality Methods, LLC. Hoffman is involved in just about every organization having to do with software quality; he’s an ASQ Fellow, a member in the ACM and IEEE, and is a Founding Member and a Director of the Association for Software Testing.

When asked to summarize his talk, Hoffman got straight to the point, “The results of running a test aren’t really pass or fail. I think this message will resonate with part of the audience and may inspire others to challenge the idea. CAST is a venue where such discussion is encouraged.”

The idea is expanded on in the summary for his talk:

Most testers think of tests passing or failing. Either they found a bug or they didn’t. Unfortunately, experience repeatedly shows us that passing a test doesn’t really mean there is no bug. It is possible for bugs to exist in the feature being tested in spite of passing the test of that capability. It is also quite possible for a test to surface an error but it not be detected at the time. Passing really only means that we didn’t notice anything interesting.

Likewise, failing a test is no guarantee that a bug is present. There could be a bug in the test itself, a configuration problem, corrupted data, or a host of other explainable reasons that do not mean that there is anything wrong with the software being tested. Failing really only means that something that was noticed warrants further investigation.

“I think all we can really conclude from a test is whether or not further work is appropriate.” Hoffman said. “The talk goes into why I think this, and some of the implications of thinking this way.”

When I asked Hoffman what inspired him to question the binary nature of a test, he said: “I was discussing the value (or lack of value) of pass/fail metrics when it occurred to me how bogus the numbers were, and some of the reasons. That led me to think through what ‘pass’ and ‘fail’ mean.”

So where does this leave teams who use pass/fail metrics? What does Hoffman see as a better alternative? Instead of a world of pass/fail, which doesn’t inspire additional work or thinking about the problem, he sees a system where a result might lead you down the road to additional investigation or bug reporting. With each result you have to ask additional questions before you move on. It challenges the tester to evaluate when they are really done with something, or if they’ve gotten all the value they can from an activity.

“Even with exploratory sessions, we conclude whether or not there are problems to report now and further avenues where we think we’ve detected problems, or not. For discrete test cases it is much clearer whether or not further work is indicated. In any case, most people refer to the software as failing or passing based on these indications.”

“The idea of a test passing/failing, indeed the idea of discrete tests, may be foreign to some people who have only known exploratory testing. In those contexts there may be audience members who may challenge that tests don’t pass or fail because the concepts aren’t applicable.”

So for Hoffman, testers doing exploratory testing face this issue all the time and already have methods for dealing with it. “There also could be criticism that I look at test results as being binary,” said Hoffman. “Others may consider there to be more than two outcomes. Again, I think it depends on how pass and fail are defined.”

In the past, Hoffman has done extensive work around test oracles. An oracle is the principle or mechanism by which we recognize a problem (that is, it’s how you can tell the good behavior from the bad). When asked how this work relates to his work on test oracles, Hoffman replied: “This is one conclusion I’ve drawn from that oracle work. Over the years I stopped talking about passing and failing, but had never consciously realized it.”

For more on the upcoming show, check out the CAST conference website. I also recommend, if you haven’t already, familiarizing yourself with Doug Hoffman’s work, which is available at Software Quality Methods.

Ways to drive adoption of test-driven development

At a recent Indianapolis Workshop on Software Testing, I joined a group of software testing pros in examining the practice of test-driven development (TDD). We brainstormed ahout how to increase adoption by helping teams new to the practice. While there are general principles for introducing change into an organization that come into play, we came up with the following practices that would make the TDD practice sticky in most organizations.

One topic that came up often was making sure the team understands why implementing test-driven development is needed. Managers need explain the benefits in terms of how teams will benefit. A key to success here is finding out what the team’s objections and concerns are before you begin implementation. Without that information, objections can’t be addressed in advance, and they’ll show up later in ways that will slow down the project.

Once people understand the goal, the next step will likely be to pilot the practice. Some ideas from the group included starting with the strongest advocates in the group, targeting the programmers with most influence first, or pulling in external resources that already know how to do it and can share their experience with the team. By starting with a smaller team and growing from there, you’re (hopefully) building a success story. You want to create a sub-culture where people want it to succeed.

To help make the tests a more visible and tangible part of the development practice, make sure they are included in code reviews and that when people pair they are still working test-first.  You want to create a culture where programmers don’t just challenge each other on code and design, but also on tests. You can also make unit testing tasks a visible part of the development process; include them in stories, on your kanban board, or as part of task-out.

In addition to making sure people understand the goals and getting the practice integrated into your development practices, there are specific tools and metrics that can help. Here are some good practices:

• Get people looking at static analysis metrics — complexity, magic numbers, etc. — for the code before and after test-driven development.
• Get people looking at code coverage for unit tests and notice how it changes as adoption picks up.
• When you start, it can be helpful to have an endpoint in mind and trace how test-driven development can assist you in getting there.
• Try to find and measure other metrics to support the end goal; like time to fix with test-driven development versus without.

Some of the tools that came up during the workshop that can help included Heckle, Flog, and Blame. There are tons of tools out there to help teams manage unit tests and provide code and testing metrics. Figure out what works for you, based on your programming language, IDE of choice, and how your team wants to work. As your team starts down the road of implementing test-driven development, recognize that they’ll need support and reinforcement. Finally, understand that the change likely won’t happen overnight.

These are just a few of the many good ideas offered in the March meeting of theIndianapolis Workshop on Software Testing. Besides yours turly, participants of the workshop included Chris Achard, Patrick Bailey, Anthony Bye, Jason Gladish, Matthew Gordon, Tim Harvey, Frank Jaloma, Courtney Jones, Baher Malek, Joel Meador, Elijah Miller, Steve Pollak, Russell Scheerer, Elizabeth M. Shaw, Dustin Sparks, Miles Z. Sterrett, Jon Strayer, Nick Voll, and Jeff White.