A Business Application Summation

Nov 12 2012   9:58AM GMT

Big data, Karl Rove and DrunkNateSilver prove a data quality point

Barney Beal Barney Beal Profile: Barney Beal

If you follow the same bloggers, tweeters and publications that I do, after last week you could be forgiven for looking forward to the next four years of the Nate Silver administration No one has probably done more for nerd-dom since Lewis Skolnick and Gilbert Lowell. Although the Harvard Business Review’s decision to nominate the data scientist as the sexiest job of the 21st century certainly did its part.

Still, is there a clearer sign of instant celebrity in the modern era than inspiring your own parody Twitter handle? @DrunkNate Silver is pretty funny (though a warning, it’s a bit bawdy if that bothers some folks).

It’s a good story, and a good headline, tied to one of the world’s biggest events, but I have to say I find much of it over the top. I saw one story claiming Nate Silver’s victory over punditry. I guess that makes sense if you think people watch pundits for actual election predictions. Karl Rove’s meltdown on election night over Ohio would indicate anyone looking for accurate predictions from these folks aren’t getting the measured analysis they hoped.

I’d argue that people watch pundits for entertainment, not measured analysis. Look, statistical analysis has penetrated the sports world pretty pervasively — far quicker than in a lot of business segments. After all, Billy Beane, the General Manager of the Oakland A’s, with no other business background is a member of the NetSuite board of directors. Silver himself started out by developing the PECOTA system, to predict baseball players’ hitting and pitching performance. But people are still listening to sports talk radio and they’re not discussing statistics. I live in Boston, believe me I know.

Another story in Slate threw a little cold water on the idea of Silver as a towering genius, and it makes a good point. Silver didn’t win the election, he just predicted it with incredible accuracy. And he did that using polling numbers that have been around for years. Obama won the election. With a great deal of help from his own number crunchers as this Time piece points out.

Political campaigns have long been at the forefront of data mining and predictive analysis. By most accounts, the Obama campaign has been particularly good at it. Stories about couples who voted early only to later get a phone call asking if their son had voted yet are a pretty good indication Obama’s campaign was well organized. That is something for business to strive for. I wish I could get through an IVR system without having to enter my account information three times.That makes headlines indicating that Silver provided some ‘big data’ lessons none too surprising. The fact that Silver got the election right and a bunch of right-leaning pundits got it wrong may be a vindication of big data for the enterprise, but I think it points to another lesson and that’s one of data quality.Silver’s trick wasn’t collecting election data and running it through a supercomputer. It was finding the right polls and weighting them according to accuracy, which seems to me much more about the quality of his data than a dive into big data analytics. Data quality has traditionally been given short shrift when popular technology trends like CRM and BI emerge only to rear its ugly head after the project has been rolled out. That is a lesson businesses would be wise to take from this election.

 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: