“When Joe the Helpdesk guy goes to IT leadership, he has statistics. Same day call resolution, percentage of calls that go through to a person versus to voicemail, and number of tickets resolved per day. When the CIO goes to board meetings, he can represent that. Accounting has cost numbers; Sales has actually dollars. What does IT dev have? Expenses. The only way to show improvement is to spend less. We need metrics! It doesn’t matter what they are; we’ll just through out a bunch of them and see what sticks. What ideas do you have?”
I’m not kidding; that was the actual conversation.
We came up with a number of measurements, things like the number of projects completed in a month, or the number of requests on the small projects team divided by the number completed in a month, which would estimate the size of the backlog. As you can probably guess, I was less than excited – “we’re really going to base our performance on our customers ability to think of new things for us to do?”
So when I heard that Eric Ries, author of The Lean Startup had planned to editing a second book in the series called Lean Analytics, I was excited. My copy arrived two weeks ago, just in time to read on airplanes and in hotels; here’s what I found.
My concern at the IT shop was over the validity of the measure. If we started to measure number of projects complete and reward number of projects complete, then we’d certainly see more projects complete. They just might be pushed out early or left with critical features complete. (One of the measures was change controls per week. The day after the announcement, Jeff, one of the team leads, filed twenty change controls, all for essentially the same change. I can’t make this stuff up.)
So while a metric may be interesting as an inquiry, to figure out what is going on, as soon as it becomes a control, you’ve got problems.
Croll and Yoskvitz, the authors of the book, are talking about measurements for a different purpose – to validate the market, to see if customers will buy the product, and then, later, to tweak the product and marketing to best reach the market you have. The first worry they have is “vanity metrics”, that make you feel good but don’t inform you about the state of business or aid in making a decision. Their list of “vanity metrics” includes “hits” (add ten images to increase your hits!”), page views, unique visitors, and number of registered users (there’s a stat that is guaranteed to go up and to the right!)
From Metrics to Lean Startup
If you measure one thing (“revenue generated per month”) that doesn’t really give you a sense of an experiment. Early in the book Cross suggests five additional ways to slice users: By segment (business vs. entertainment users), cohorts (people who signed up, grouped into why they signed up), A/B Testing (vary the user experience and track who is more likely to buy), and multivariate analysis, where you conduct multiple A/B splits simultaneously and use statistics to isolate results.
That’s just chapter 2.
I’m going to need to do more blog posts.
Chapter 3 and 4 are about the long funnel – the process of moving a potential customer from hearing about your business to an actual purchase decision. That process can take months. A strategy that just measures sales will capture only a trailing indicator; it will tell you about the people that signed up months ago, not currently, so Croll and Yoskovitz devote two chapters to measuring the pipeline. With additional chapters on SaaS, mobile applications, media sites, and user-created content, I slowly beging to realize that Lean Analytics is poised at web-based startups, and might not be too much help in measuring performance in IT. There are some applications you might use the chapters to, say, measure the success of a mobile app rollout. Plus there is plenty of material about Key Performance Indicators, or KPIs. More about that when I have time.
But … Back in the IT Shop
There is one chapter that is worth its weight in gold, that has broad general applicability. It is advice on how to define and position your business, regardless of if you are a startup, an IT shop, or a lawn mowing service.
It’s called the lean canvas, and it’s something I would like to talk about next.