Software testing tools

Using source code analysis tools

I found a great article earlier this week on static analysis tools by Mary Brandel. In the article, “How to choose and use source code analysis tools,” she cites some statistics on the static analysis market, including:

• “The entire software security market was worth about US $300 million in 2007″
• “The tools portion of that market doubled from 2006 to 2007 to about $180 million”
• “About half of that is attributable to static analysis tools, which amounted to about $91.9 million”

In the article, Brandel also offers some evaluation criteria for when you start looking at source code analysis tools. These include language support and integration, assessment accuracy, customization, and knowledge base. She also provides some dos and don’ts for source code analysis. I think the most valuable tidbits from that list include:

• DO consider using more than one tool: The article provides a good story about Lint vs. Coverity, and I’ve found that static analysis tools will find different issues as well. Each vendor will have its own specific focus on vulnerabilities and warnings.
• DO retain the human element: While I’ve yet to work with a team that thinks adding automated tools like this will allow you to remove people, there’s certainly the feeling from the marketing materials that the results are intuitive. That’s typically not the case. You often need to know what you’re looking at or you’ll miss the subtleties in the data. I agree with the “truly an art form” quote. This stuff is hard, and while tools make it easier, it’s still brain-engaged work.
• DO consider reporting flexibility: At some companies this is a big deal. When working with smaller software development organizations, it doesn’t matter what the reports look like. The only people looking at them are the people working in the code. However at a larger company, Fortune 500 for example, information like this normally needs to be summarized and reported up.

What are the top software tools of 2008?

As the year starts to wind down, we at SearchSoftwareQuality.com are looking back at what took place during 2008. One thing that we’re focusing on is the tools and solutions that were released. In an effort to help our readers understand what tools are available to help them, we are creating a guide to tools released in 2008 to be published in January.

In order for us to do that, we need your help identifying tools that were released. The tool categories we’re focusing on:

• Software testing
• Test management
• Code quality
• Application security
• Software requirements
• Agile development
• Project management
• Application lifecycle management
• Application performance monitoring & management

Please send us information about tools released between Jan. 1, 2008, and Oct. 31, 2008, that you’d like us to consider for the guide. The tools must be new products or significant upgrades. And you must include the following information:

• Product name and version/model number
• Company name
• URL for the product
• Product or company logo
• Date product was released
• Tool category (see above)
• Product description
• If it’s an upgrade, features that were added
• What makes it innovative?
• Details about how it performs
• Details about its ease of use and manageability
• Pricing

Send your product submissions to Editor@SearchSoftwareQuality.com by Friday, Dec. 12.

Lowered cost for security testing suite

In response to economic issues and as a way to encourage more companies to test applications for security, Ounce Labs has reduced the cost of its Static Application Security Testing suite.

The shift in the pricing and licensing models will lower costs and complexity for Ounce 6 customers, the company said.

In a prepared statement Ounce CEO Gary Jackson said:

We intend to accelerate enterprise adoption and make source code analysis more accessible for every company concerned with application security, from the smallest shops to the largest enterprises. The new pricing schedule will speed time to value and ensure that every organization can afford, deploy, and capitalize on source code scanning to protect their critical data.

The fee for the defined organization will allow unlimited users or seats, and unlimited product installations. Third party contractors working for the organization will also have access to the products, at no additional fee.

“Site” is a single legal organization where all users are within a 3 mile radius. An organization with fewer than 200 employees, and operating revenue of $50 million or less is a Level A Site. This may be an independent company or the department of a larger company.

“Business Unit” is a geographically dispersed, single legal organization. An organization with operating revenue of $500 Million or less is a Level C Business Unit.

The importance — and challenges — of debugging

Debugging is often challenging for programmers, so much so that many books have been written about it. One of the latest to hit the shelves is The Art of Debugging with GDB, DDD, and Eclipse written by Norman Matloff and Peter Jay Salzman and published by No Starch Press.

According to Matloff, debugging is challenging because “you’re looking for a needle in a haystack. And in many cases, the bug turns out to be in the portion of the program you’re most sure is correct!”

And that, he said, is one of the most common pitfalls programmers encounter. They’re sure that some part of a code is correct when in actuality that is where the bug is hiding. Additionally, Matloff said, some programmers will not use a debugging tool.

What can programmers do to avoid those pitfalls?

“In our book, we put forth a key guideline that we call the Principle of Confirmation, which says that debugging is basically a process of confirming that what you think really is true. Eventually you find an assertion that fails to confirm, and then you’ve likely pinpointed the location of the bug,” Matloff said.

Of the debugging tools out there, Matloff says the benefits open-source tools includes the ability for them to be used on any of the major platforms — Linux, Mac, and Windows.

“Thus you can have the same environment available to you even if you work on several different kinds of systems,” he said.

When asked if there are issues that make debugging more critical these days, Matloff said there is much debate about this.

“For example, some people think that strongly typed languages are “safer,” i.e. less bug-prone, while others believe that such languages complicate the code so much that they generate MORE bugs,” he said.

What do you think? Are some languages safer than others? Do you have any tips for making the debugging process easier?