Test Automation Metrics: Tracking Progress

Did you ever have one of those days/weeks/months when your boss walks nervously into your office and says in that ‘I-just-came-out-of-a-very-long-and-gruesome-meeting’ tone of voice, “Well, , how’s that automation project coming all?” You figure that just answering “Fine” probably is not enough in this situation, since the fate of the company — and certainly your job — hangs in the balance. Oh, if only you had been tracking your progress all along!

Well, have no fear, the answer is here!

In a previous blog entry, I stated that “automated testing is the programmatic execution of test cases.” That gives us the starting point for our metrics: manual test case count. Generally, these counts fall into two categories: strategically Planned Test Cases, and actually written Manual Test Cases.

Most automation effort tends to be focused on coverting written manual test cases to automated test scripts, so that gives us our third metric, Automated Test Cases.

Progress can be gauged by the ratio of actual to expected. Let’s asemble our metrics into a table and see how this plays out:

As shown above, the progress of automation is expressed as the ratio between (Automated Test Cases)/(Manual Test Cases). This implies a typical dependency of automation on the creation of separate, detailed manual tests created by subject matter experts. If your organization follows a different sort of QA process, where the person doing the automation is also writing the manual tests and converting them directly into automated test scripts, the formula could be altered to express the ratio of (Automated Test Cases)/(Planned Test Cases).