This week, Coverity, a company that provides a tool that performs static analysis of code, announced findings of their annual report on the state of open source software integrity. Details of the report include findings from the analysis of 291 popular open source projects and over 61 million lines of code. Included were tests of an Android kernel from the popular HTC Droid Incredible.
The report shows that almost half the defects found were considered high risk with the possibility of causing security vulnerabilities or system crashes.
I spoke with Coverity Chief Scientist Andy Chou about the report and about the quality of open source code in general.
When asked whether proprietary software was higher quality than open source, Chou notes that many commercial products are a mix of proprietary and open source.
We often get asked the question, how do these compare? Our thinking has evolved over time. The boundary between the two is quite blurry. If you look at a lot of proprietary commercial software it often contains open source software so it’s very difficult to separate the open source and proprietary components these days. If you look at typical mobile phones operating system, for example, Android is a good example, the whole operating system is open source, but OEMs can add proprietary software on top of it for custom applications or custom devices. So when you put the whole system together, it’s a hybrid of the two. The fact that it’s such a mixture makes it very difficult to separate measurements.
Still, I persisted, aren’t vendors responsible for ensuring the quality of their overall product? Open source code is visible, so shouldn’t it be tested if it’s packaged as a commercial product?
Often commercial software vendors and OEMs have limited visibility into the quality of the software they’re using and the accountability is quite fragmented. It’s not easy to pinpoint exactly who has a handle on all of the software.
Despite the blurred boundaries, I wanted to know if the studies were showing that software offered by vendors for a cost was better quality than open source. My assumption had always been that vendors would have more resources than open source providers to hire the testers and purchase the necessary tools to ensure high quality. Chou answered that there was a wide range of quality in both the open source market as well as the commercial market.
There’s no simple pat answer. The differences between the best and worst are very broad. There’s a spectrum. Same thing with commercial software. Some industries may choose to release early, knowing there are going to be defects and that’s a business trade off they’re willing to take.