Recently, SSQ created a quality metrics guide which includes a series of articles, tips and stories related to measuring software quality. It’s a complicated and controversial topic with no easy answers. We asked our readers to weigh in, and I wanted to share a couple of insightful responses we received.
Darek Malinowski, an ALM Solutions Architect at Hewlett-Packard, writes:
In my opinion we should start from business requirements. First we should estimate the weight (criticality) or business impact of each requirement. Business impact is related to loss of money, loss of confidence, loss of market etc. So the best to measure quality is requirements coverage by tests vs. requirements criticality. This can tell if the product we deliver to the business provides all needed functionality as required.
Haya Rubinstein works as the Quality and Delivery team lead in SAP in the IMS NetWeaver MDM group. After reading Crystal Bedell’s article, What upper management should know about managing software testing processes, she gave a very detailed answer to our question about how her organization measured software quality.
In my experience, management is interested in the test KPI’s even before the release is decided on. There is usually a quality standard that they wish to adhere to.
Following is an example of KPI’s for a software release:
- 0% defects on the top ten priority functionalities.
- Less than 5% degradation in performance.
- All new features have been tested.
- All new “hard” software requirements were met.
- All quality standards were met or mitigations were agreed on with quality standard owners.
- Over 90% of tests on new features have passed.
- Over 95% of all unit tests have passed.
- Over 95% of all regression tests have passed.
- Over 95% of all quality standard tests have passed.
- No Very High or High priority defects remain open.
- All medium priority defects have been examined by development to ensure they are not a symptom of a more severe issue.
Rubinstein goes on to describe the detailed and disciplined process that is used throughout the development life-cycle with test milestones starting with the planning of the feature through it’s delivery. She says that at the end of the test cycle, management receives weekly reports aimed at showing compliance to the KPIs including:
- If there are any risks to KPI compliance they are presented together with the mitigations proposed.
- The compliance to the KPI’s above with the current status (number or % and color coded)
- Details of open Very High/High priority defects –Defect #, Summary of issue, Created by, Opened on date, Assigned to, Current status
- Approval status for the relevant features according to hard software requirements and details of remaining defects that are still open on each of the features.
- Link to full defect report.
- Link to full test plan report.
- Notes exist for all open defects.
An overall status (green, yellow, or red).
What about your organization? What metrics do you use to measure quality?