Posted by: Michael Tidmarsh
Big Data, Hadoop
Big data image via Shutterstock
By James Kobielus (@jameskobielus)
Big data has its discontents. The discontent is taking the form of a backlash against the incessant hype of big data’s most noteworthy rock star approach: Hadoop. The backlash is a necessary reality-check in an otherwise fast-growing space. Often in this industry, when a technology is super-hot, the hype can get in the way of rational decision making, both among users and among solution providers.
Backlashes against hype are inevitable. At times, it almost feels like people discuss big data with the assumption that bigger is necessarily better and that throwing more data at your problems will automatically produce insights. You should be searching for those special problems, often of a scientific nature, that can be solved best through petabyte-scale analytics. You don’t need a data center full of maxed-out storage arrays to derive powerful insights. Gut feel is free, and it often thrives on the scantiest information.
This viewpoint, which is now in the mainstream, is often referred to as “small data,” or as the need for traditional lower-scale analytics platforms to balance the hype for all things “big.” Of course, keeping your BI, performance management, and statistical modeling deployments small and simple can be a challenge. This is especially true as your user base grows; the range of data sources, structured reports, metrics, dashboards, models, visualizations, and downstream applications grows. Throughout the life of your “small data” environment, it will probably continue to grow into “big data” territory on the volume, velocity, and/or variety dimensions.
As your needs scale from small to big, you should have a data platform that you can grow modularly and cost-effectively to keep pace. Big data and small data should be joined at the hip in your strategy–simply a seamless spectrum of analytical platforms, tools, and approaches serving your business.
Big data, small data. Big insights, small epiphanies. It’s all important. And none of it is mutually exclusive.