Posted by: Shilpa Venkateshwaran
Metrics, Test Metrics
Metric Goof Ups
Companies depend on metrics to manage their businesses. Metrics can measure failures or successes, it can help with showing trends or progress. It helps create baselines upon which changes can based on. Since the companies depend so much on metrics its important to get metrics right. Metric goofs ups are common and can be very easily avoided. These are some goof ups or common mistakes that can occur when putting metrics together or when the data is misinterpreted by the end user of the metrics. Its not that people put wrong numbers or bad metrics on purpose, these are things that crop up without us knowing. Or these can be errors due to user mis-interpretation. Both these errors can be reduced if we take the time to avoid these mistakes when creating the metrics.
Few most common mistakes are:
Cascade Errors – This is when one error knowingly or unknowingly causes several more in other places. So everything based of that initial number is now not valid. Formula Errors – If you plain just use the wrong formula or wrong values to calculate a number. These errors can skew your data.
Apples to Oranges – This happens when you are not comparing apples to apples. When you say you have executed 10 tests day one and 20 tests day two and 10 each rest of the week. So in total they are 60 tests. Are these similar tests? If not then you have to break it down by complexity or features to really present what that 100 means. When you group things together they have to be similar ones.
Drill based on numbers not meaning – Sometimes we focus on the numbers and not what it means in the big picture view. So for example if we hear there are 200 new defects in one month, we cant just panic without understanding the context of it. Are these new issues in an area that is new and not test before? Are these defects may be that were known (maybe minor) that were logged together when the testers had some downtime? So its really important to understand the context before we create panics.
These errors are very common and there are several ways to reduce them
- Get someone to review your data on a regular basis.
- Run error reports on a regular basis. If you run a report to gather defects created per build, run one by date and see if the numbers add up in the end.
- Always double check your data and numbers.
- Even if you use calculators or excel formulas, once in a while calculate some by hand and see they still add up.
- If something doesn’t feel right make sure you go back. If you see a higher defect density percentage, make sure its really high and not just a goof up. Also note these are sections that you will get questions on and so be ready to answer what caused these spikes.
Sign off note: Have you made any goof ups? What did you learn from it?