Data quality is still the bane of the business intelligence (BI) environment. Or so I was told by two consultants I interviewed for an upcoming BI ezine. Even when everyone agrees the data is not up to snuff, the question remains whether it is worth fixing, they said.
But is concern about data quality misplaced? There are at least two competing theories about this: what I’ll call the old school view and the big data movement. The old school view is that data quality matters: garbage in, garbage out. Thus, time is spent on data cleansing, extracting, transforming and so forth, and the strong belief is that this is time well-spent.
The big data movement has spawned a different worldview on data quality: the bigger, the better. The central idea here is that data-crunching in itself is cleansing. Things that don’t fit into the data model are like flotsam and jetsam — an insignificant, superficial layer on your deep ocean of insight.
In the real world, I’m finding that CIOs understand the big data quality perspective — and some would like to embrace it — but the old school wins out. And that’s not because these CIOs are risk-averse. Case in point is Greg Taffet, CIO at U.S. Gas & Electric, a Miami-based reseller of gas and electricity. The fast-growing company has doubled in size in every one of the last four years that Taffet has been there. Not that he’s complaining. He’s one of those CIOs who like to be where the action is. “I was previously the CIO and employee No. 4 at MXenergy, and left when it hit $1 billion in revenue. I was recruited here to do the same,” he said.
But the ever-changing business fundamentals make building “a real BI environment,” as he puts it, particularly challenging. When it comes to data quality, Taffet is definitely old-school: “The tools are really not that distinguishable. We have to know our business. We have to get into the minds of the executives and the operational people, so we can set up the tools to do their job.” For him, data quality is the bedrock of a real BI environment.
So, amid a whirlwind of growth, Taffet and his BI team meet weekly with people from finance, operations and sales to make sure there’s no disagreement about the quality of the data IT is collecting. “When we see something that is not expected, we drill down into the details and see if it is a variation on something that was not accounted for but should be, or something that should be taken out,” he said. It’s time well spent, he says, toward building that real BI environment.
But then Taffet’s not dealing with the volumes of data that qualify as big data — yet. “I still see that we have several years before we get hit with what we call big data,” he said. And until the tsunami hits, he’s sticking with old school. You?