Software Quality Insights

January 18, 2009  5:22 PM

Testing on mobile devices

MichaelDKelly Michael Kelly Profile: MichaelDKelly

Over the last few months, I’ve been asked a lot about testing on mobile devices. At my own company and at other companies in the Midwest (which isn’t exactly known as a hotbed of technology). I’ll plead mostly ignorance, most of what I know is from Julian Harty, whose work I follow both in print and at conferences. I think the topic is important enough that in a series of workshops I co-host, we’ve added a full day to the topic.

Yesterday, I read this story on new handsets and the issues that are encountered in the marketplace as vendors work to keep up with the cutting edge and market demand. What I really like about this article, aside from it being an interesting news story, is that I think it provides a nice snapshot of some of the common challenges of testing on mobile devices.

  • Meeting launch targets: I’m not sure how unique this is to mobile devices, but it certainly seems like deadlines often coincide with hardware deadlines. For example, if you’re writing software for the iPhone, then to some extent your releases are tied to theirs, if for no other reason than support (see software updates below). If a new “iClone” comes out, there’s a chance you get to port your iPhone app over to that device.
  • Handsets are getting more complex: My first Palm had a calendar, took notes, and had a calculator. My first cell phone only made phone calls. They claimed there were other applications on it, but no one was really using the early cell phone calculator. I mean, really? Now, I can read a spreadsheet on a phone, take a picture, stream media, look up directions in real-time, and likely just about anything else I can think of. And I can do that using a keypad, stylus, keyboard, gestures, or the soon to be invented chip implant. I’m sure it’s only two or three years out … really.
  • Consumer expectation: Expectations for mobile devices are high. From performance and reliability to recoverability of data, the quality criteria used when testing on mobile devices is subtly different from those of traditional Web and desktop applications. The software I use on my phone, while it’s typically functionally simpler, operations in a much more hostile environment: there’s less processing power, less memory, less disk space, connectivity comes and goes, and my attention span is shorter because I can’t really multitask like I can on my desktop.
  • Software updates: While this isn’t a new testing problem, for many programmers this problem has largely been solved. Think of Web browsers: If you want to hit the largest markets, you need to support two to four browsers. For each of those, you’ll make some sort of decision on what versions you’ll update, which service packs, and depending on your software, what hotfixes you’ll need to test with (and on what schedule). In addition, regression test automation for applications that run in a Web browser is largely trivial (compared to even five years ago). On mobile applications, there’s way more than three main platforms, each one offering regular updates (often paired with hardware updates), and your ability to effectively automate much of that testing is largely not trivial given that it’s some mix of emulators and actual physical devices.
  • Touch-screen sparks new problems: A subset of the complexity problem above, touch-screens (like graffiti before them) introduce a relatively new method of interaction with the product. This likely means increased usability testing and changes in thinking about design (as the article points out). It’s most likely just the latest step in the evolution of that platform.

January 16, 2009  9:17 PM

How to maintain software quality, complete projects in a recession

Jan Stafford Jan Stafford Profile: Jan Stafford

Software consultants, vendors and project managers are already seeing software project failures and slowdowns resulting from the new recession.

That finding and the advice for software developers and project managers offered in this post comes from interviews I’ve recently conducted with 13 software industry experts and the blogosphere.

“Companies are panicking due to the economy. They’re compressing projects and schedules. As a result, key projects are failing,” Lawrence Oliva — senior consultant/program manager with CH2M HILL, an Englewood, Colo.-based engineering and program management firm –- told me in a recent conversation. “In part, this is the result of trying to get things done faster with less resources.”

Consultants are seeing reductions in software testing being used to cut costs. That tactic is a recipe for project failure, said consultant Karen Johnson in my recent post about why software project fail.)

My sources also spoke of seeing several large, current software projects in crisis mode because senior developers were laid off and the remaining, less-experienced developers didn’t have the know-how needed. Unfortunately, many development groups have been running lean for a long time and running leaner pretty much means stopping development.

“Lean computing is nothing new,” Oliva said. “Companies have been operating at skeletal levels since the economic downturn of 2000, when that downturn caused intense development staffing cuts. It really is sad to think about having to be even leaner.”

That’s the situation, but it isn’t hopeless. Gleaned from my interviews, here are some tips for maintaining software quality and project success during this recession:

Analysts and others advise development teams that lean computing and the agile model can help them do more with less.

Some companies have already made this move and feel ready for tough times. For example, Des Moines, Iowa-based Principal Financial Group –- which provides software products for the financial industry — uses the agile model and has standardized project management, testing and quality assurance processes and tools used by all its development groups.

“Standardizing has reduced redundancies in tools, processes, documentation and more, running as lean as possible,” Principal’s senior IT system analyst Mark Ford told me. Principal uses HP Quality Center to manage development.

Project managers whose companies have not already made cuts to development staffing or budgets need to advise decision-making executives about the business benefits of their projects, industry insiders advised. Also, when cutbacks have taken place or are proposed, PMs need to explain the business impact and guide where the cuts will be made. IBM’s director of Rational offerings, David Locke, offers this advice:

“Always talk in the context of how the project will help the business. If I take this 10% off, I will have the least impact on the business. Give real information, not data. Remind managers that the company creates software to make our businesses more streamlined, more competitive and to deliver better ROI. Your company may need immediate cost savings, but eliminating software that could give the business the most cost-effective competitive edge is probably not the way to go. Talk about business impact, always.”

This is certainly an appropriate time for PMs to push for investments in software testing automation tools, like automated code checkers, experts told me. However, Oliva added:

“Don’t trust in those automated systems absolutely. You could get the wrong results from the use of automated systems.At some point human beings have to be involved. Delivering quality software is not a machine-to-machine process. Not having quality monitored and managed by humans is dangerous.”

Another suggested labor- and cost-saving move is adopting streamlined models of programming, like Extreme Programming or Structured Programming.

The most important strategy for weathering the recession is becoming more realistic about software needs and reducing complexity of software. “If companies do this, it could be the one good thing that comes from the recession,” Oliva said.

Continue the conversation: Please tell me what adjustments you and your development team are making to deal with the tight economy. You can respond by commenting below or writing to me at

Here are some resources for more advice and information:

    1 Comment     RSS Feed     Email a friend

January 13, 2009  11:49 PM

Value of SANS’ list of top software errors rests on project managers

Jan Stafford Jan Stafford Profile: Jan Stafford

Whether the CWE/SANS list of the 25 most dangerous programming errors will contribute to the creation of better software depends on whether managers, rather than developers, read it and take action, according to Jack Danahy, chief technology officer and co-founder of source code vulnerability analysis firm Ounce Labs Inc. I talked with Danahy today about the follow-up and follow-through that could make the list a valuable turning point in development, rather than a partially-remembered checklist.

“It’s one thing to come up with a list of 25 things that developers should consider, but we haven’t arrived at a point where anyone is meaningfully asking or requiring developers to consider these things,” Danahy said.

Project managers should support developers spending time to research these issues, in Danahy’s view. The best-case scenario would be that software development managers -– the program manager, business unit manager, software auditor, etc. -– would use this list in specifications, asking for metrics to make sure that those errors have been looked for and eliminated before the software rolls into production.

“Developers won’t remember this list off the top of their heads, but if it becomes codifed as a requirement they will remember,” Danahy said. “A team could come up with distilled list of 5to 12 key design criteria that would provide the essence of keeping these errors from happening.”

To read more opinions about the CWE/SANS research, check out software development pro Mike Kelly’s post on using the list and’s news report.

What will your organization do with this list? Will it have an impact or be quickly forgotten? Sound off in our comments below or by writing to

January 13, 2009  10:46 PM

Using SANS’ 25 most dangerous programming errors list

MichaelDKelly Michael Kelly Profile: MichaelDKelly

In a small cry of victory today, someone on the team found this article from the BBC detailing the “top 25 most dangerous programming errors.” I say small cry of victory, because he had recently logged a ticket in JIRA detailing one of those errors, but when the ticket came up for review it was ignored. It was acknowledged as an issue, but pushed to a later sprint for work.

While I agree with the reasons we used when we prioritized the ticket, for me this incident demonstrated a common pattern I see in the teams I’ve worked with. First, there seems to be an expectation that the testing team shouldn’t be looking for errors like this — that is, unless you’re a high-priced security tester. Second, that when issues like this are found they take a backseat to the more traditional functional defects.

I like research like this (both SANS and OWASP do great work in this area), because it gives me a way to structure the conversations that take place when these issues come up. I find that programmers typically respond well to links to catalogs of errors with descriptions. They are unrelated to the software they are working on. It makes it less personal I think — less close to home.

That said, these issues aren’t always burning, top-priority issues. Like I said, in the context of our current project where we found one of these, we have time to fix it. Given the current list of commitments, the issues we know we need to work, and the relative risk of this causing a problem — this specific issue can be sidelined for a couple of weeks until we get to it. That perspective is important as well, and it’s one that I sometimes forget. There’s always a business story to tell as well with issues like this — context is important.

For those interested in the more details on the list of programming errors, you can find the full list of errors from SAN here.

January 12, 2009  1:06 PM

Why software projects fail and more will fail in 2009

Jan Stafford Jan Stafford Profile: Jan Stafford

Why do software projects fail? There are many reasons –- and they’re spelled out below –- and 2009 may bring more failures than usual as budget cuts spur project managers to make cuts in the wrong places. So said software quality consultants Lawrence Oliva and Karen Johnson when I talked to them about 2009’s software quality landscape.

Human error causes most software project failures, said Oliva — senior consultant/program manager with CH2M HILL, an Englewood, Colo.-based engineering and program management firm — so most are avoidable. Here is his list of the mistakes he sees most often and his comments:

1. Unclear requirements: “Most people don’t know what to build because they’ve never defined it. When they build the software, it fails because it doesn’t meet people’s needs.”

2. Overly optimistic and/or unrealistic schedules. “People rush or skip things if the schedule isn’t realistic. Also, companies are panicking due to the economy. They’re compressing projects and schedules.”

3. Lack of user input: This links back to requirements mistake #1. “Developers don’t talk to people who are going to use the software.”

4. Lack of executive sponsorship and support: “When management doesn’t support and protect the project, it can often be undermined by internal politics and budget cuts.”

5. Turnover and layoffs: “Projects often fail when key people leave the project early in its lifetime.” Companies’ modern habit of laying off senior and, thus, higher-paid workers -– such as senior developers –- in favor of less experienced, lower-paid workers puzzles Oliva and me. “It doesn’t make sense, because the more experienced people take a humongous amount stability, experience with them that usually isn’t available any other place,” he said. “That hurts a project and hurts a company.”

I agreed, noting that the time lost due to less experienced workers’ mistakes and learning curve probably negates the savings in salaries paid. Oliva and I discussed that companies could do total-cost-of-layoff analyses; but Oliva said companies probably wouldn’t take the time to do that. “In a poor economy, companies often make hasty and project-wrecking decisions,” Oliva said. 

Besides layoffs, the recession is leading to other foolhardy cutbacks. It seems obvious that skipping testing is a path to software project failures, but software testing consultant Karen Johnson is seeing companies do just that. Johnson told me that companies are cutting down on or even skipping software testing altogether as a recessionary cost-saving method. If this trend continues, look for more embarrassing outages caused by admitted software failures or for “undisclosed reasons.”

While researching software failures, I read a Code Diesel post on software failures by developer Sameer Bora. To Oliva’s reasons why projects fail, Bora adds these common mistakes: sloppy development practices and poor reporting. Bora also created a handy chart on why software projects fail. Printed out or used as a screensaver, even, it could provide a visual reminder of project pitfalls.

Now it’s your turn: Do these reasons for failure ring a bell? Can you think of others? Let me know by commenting below or writing to me at

December 19, 2008  1:35 AM

Security flaws and Agile boom top software quality news in 2008

Jan Stafford Jan Stafford Profile: Jan Stafford

Security vulnerabilities and the boom in Agile development adoption topped the news charts in 2008. Here’s a rundown of the five most-read news articles and their significance.

Three of the top five articles focus on Agile development. In the #1 story, Predicting software quality trends for 2008, software quality experts predicted that Agile adoption will increase. Two other articles in the top five were about Agile. In fact about one-third of the top 20 news stories were about Agile.

Agile development: Not just for ‘agilists’ anymore (#4) discussed Agile’s move outside of its early adopter niche and the impact of its wider usage. Next came Software development groups take many routes to Agile (#5) which revealed results of’s 2008 Agile Trends Survey. Most Agile technique users, according to that survey, use Scrum (41%). Next in popularity was Extreme Programming (XP) at 15%. Others use hybrid Agile methodologies, while about three percent use Crystal and Dynamic Systems Development Method (DSMD) each.

The SSQ 2008 Agile Trends Survey showed that 45% of software pros follow Agile methodologies, while 44% use waterfall. Other development methodologies cited were test-driven development (19%) and RUP (15%).

Security flaws in leading open source software platforms drew a lot of attention, and articles on security issues in the Spring Framework and open source Java projects ranked in second and third place, respectively.

While open source development projects typically fix flaws quickly, as happened in these cases, the emergence of serious vulnerabilities may have taken some IT pros by surprise. People expect problems with Microsoft products, but not with open source products, said Kevin Beaver, CISSP and Principle Logic LLC consultant, commenting on the fact that these two stories got so many clicks.

“We’re seeing more and more that open source has its own security woes,” Beaver said.

Don’t blame the open source community and its developers, Beaver advised.

“The fact is that as long as human beings are developing applications on the complex and extensible OS (operating system) architectures we have, there will be security problems.”

Most importantly, he said, the emergence of vulnerabilities should not scare anyone away from using open source software like open source Java or the Spring Application Framework.

It is important to point out that just because a static analysis tool vendor finds flaws in open source code, that doesn’t mean the vulnerabilities can/will ever be exploited.

Keep using open source software, Beaver concluded, and “take this marketing tactic with a grain of salt.”

Now that you’ve checked out these stories, please let me know your choices for the top software quality news stories of 2008. You can write to me at

December 18, 2008  7:29 PM

Top five software quality tips of the year

Beth Pariseau Beth Pariseau Profile: Beth Pariseau

Check out’s most popular tips of 2008 — these top five tips for developers, QA testers, project managers and other software pros include indispensable expert advice on performance testing, software testing estimates and more. If you missed these great tips the first time around, don’t miss them now.

1. Cracking passwords the Web application way: Do you know how well your Web applications can stand up to authentication attacks? In this application security tip, Kevin Beaver, CISSP, lists some common Web app vulnerabilities and explains how to perform password cracking tests.

2. The ABC’s of software testing models: There are as many models for testing software as for developing software. Learn the basics of the common software testing models including waterfall-style, iterative-style and agile-style testing.

3. Testing for performance: Assess the problem space: Part one in a three-part series on performance testing, this tip explains how to assess the initial problem space before the first round of testing, including developing an understanding of your goals, how the systems will be used and other strategies.

4. What to include in a performance test plan: Find out what features a solid performance testing plan needs to include at a minimum, and how to reduce the likelihood of miscommunication.

5. How to estimate for testing on a new software project: How do you estimate for software testing on a brand-new project when you have no historical data for reference? Learn some methods in this tip.

December 17, 2008  5:02 PM

Open source, agile help move to lean software development

Jan Stafford Jan Stafford Profile: Jan Stafford

Bloated applications, platforms and architectures slow application development and make quality control and everyday usage time-consuming and nonproductive, said Forrester Research principal analyst John R. Rymer in a phone conversation yesterday. In this post, I’ll share Rymer’s thoughts on why software pros should join the lean software movement and his advice on how to create appropriately sized and efficient software. 

Forrester’s newly-published report, Lean Software is Agile, Fit-to-Purpose, and Efficient, lays out how software got so fat, costly and inefficient; the evidence that IT organizations are moving to lean software; the challenges involved in lightweight software development; and strategies for joining the movement. Rymer told me that the demand for lean or lightweight software is coming from conventional business application users, the ones who first signed up for mainstream — and now bloated — apps from IBM, Microsoft, Oracle and other major software vendors. Many took the path of least resistance, as in following the safe IT path that spawned the saying: “Nobody ever gets fired for choosing IBM.”  

While major software vendors piled on features that add complexity and can foster customer lock-in, in came the lean software approach of Linux and open source developers. 

“Open source is the driving factor for lean computing,” said Rymer. “Now people can replace conventional application servers, for example, with open source application servers and get lower cost and more innovative features. People are comfortable with open source software, which is now in its second wave of adoption.”  

Agile development is another popular path away from traditional “big-bang” software development.  

“Agile development is independent of any technical platform or development approach,” said Rymer. “It’s a method. What’s neat about this is that people are delivering application features sometimes every two weeks or each month. Rather than deliver the big-bang project after years of work, they’re delivering applications in increments. They’re working in an incremental fashion to deliver features over time, providing value quickly and continue to add value. That’s a way to modulate your costs, to spread your costs and investment over a period of time.” 

This Forrester report’s recommendations advise software pros to update their application platform and tools strategies. How do your tools and platforms fit with the lean software approach? Right now, many organizations are working with platforms that are too bloated and nonproductive, Rymer said. 

“A lot of shops adopted J2EE, and they’re really struggling now to keep up with the demand for new applications. It’s not a real productive environment. Just coding things up in Java takes a lot of time.” 

The era of one-development-platform shops should be over, Rymer said. 

“If you have a variety of application scenarios, don’t assume you have to adopt one platform to do all of them. There are a variety of tools now. People who choose to use Spring (Application Framework), for example, are oftentimes using it alongside their J2EE. They can run the Spring on an extra app server. So, it’s not like there are these hairy choices that force you to throw away what you’ve got. Pick the right tool for the job, and if you’re smart about it you can integrate these things.”  

Rymer suggested that the PHP framework is built for speedy development. It’s now “a real framework and not a collection of modules,” he said. “You can build certain Web applications very quickly, much more quickly than you can with either conventional .NET or Java development.”  

Don’t think that lean computing is a movement to oust established vendors, Rymer noted before we signed off. Remember that even Microsoft is involved in the second wave of open source development tools adoption. “If you want to use Ruby or Python in the .NET world, you can,” he said. When change is driven by developer and business IT pros, big vendors like IBM, Microsoft and Oracle will join in. 

December 10, 2008  4:03 PM

Security boost for LAMP stack

Colin Smith Colin Smith Profile: Colin Smith

LAMP, an open-source Web development platform based on Linux, Apache, MySQL, and PHP, is getting some added protection from attacks thanks to Metaforic.

Metaforic, a provider of anti-tamper solutions, announced that upon request it will provide free versions of secured Apache and MySQL to enterprises. Utilizing a lightweight version of MetaFortress, Metaforic will provide anti-tamper protection and continuous integrity checking for critical parts of the LAMP stack to help defend against multi-vector attacks designed to discover and exploit the weakest point of an organization’s server infrastructure.

The move is important because as more enterprises deploy open-source technology, cybercriminals will target the security vulnerabilities within that infrastructure. And those criminals are looking for any weak spot, whether it’s the operating system, Web server, database system, or the application layer.

MetaFortress Open is specifically targeted at network infrastructure applications. Like the flagship MetaFortress, Open is an anti-tamper solution that inserts security and integrity checks into an application’s source code to prevent against hacking and unauthorized usage.

December 9, 2008  7:13 PM

Recession causing software developers to rethink processes

Colin Smith Colin Smith Profile: Colin Smith

With the recession weighing down on all of us, I’ve heard a few people talk about not letting this crisis go to waste. Ryan Martens in his column last week said now is the time for companies to take a close look at their development processes and make changes that will reduce costs now as well as in the future.

HP Software is also talking about taking advantage of the crisis. Mark Sarbiewski, director of product marketing at HP Software, said the company recommends leveraging the crisis to do three things:

  1. Get control of IT spending — Determine priorities and eliminate low-priority things.

  2. Put solutions in place now that allow you to centralize, eliminate redundancy, and maximize your experts

  3. Drive through process change and automation — Standardize on best practices and automate, focusing on the development process and operations.

To help companies accomplish those things, HP Software today announced two significant products: HP Quality Center 10.0 and HP Universal Configuration Management Database (UCMDB) 8.0.

“We look at Quality Center 10.0 as the heart of how you can change the software development life cycle,” Sarbiewski said. “It includes requirements management, test management, and defect management all in one place.”

At any time you can see how you’re doing against the software requirements. Additionally, HP has improved the ability to share things between projects.

“In Quality Center 10.0 we’ve expanded beyond the simple project model and can promote processes to all projects and share things with all projects throughout the entire cycle,” Sarbiewski said.

Quality Center 10.0 also integrates with HP’s other testing solutions.

To help on the operations side, HP Universal Configuration Management Database (UCMDB) 8.0 can help organizations continually track how everything connects and manage change across all the tiers.

“We’ve now integrated this dependency map into all those systems. I’m monitoring the parts of the biz service and all the pieces support it. I can automatically notify if I see something going wrong in any piece,” Sarbiewski said. “We’re moving from being reactive to being predictive.”

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: