Posted by: Alex Barrett
Big Data, data center infrastructure, Pete Warden
Big Data will transform your data center. Then again, maybe it won’t.
I went to GigaOm’s Structure Big Data show in New York City this week to see what’s new in the world of data analysis, and there I found that forward-looking data analysts are using open source software and commodity scale-out x86 servers to solve problems usually reserved for Big Iron.
Indeed, not only was I struck by the ubiquity of open source, x86, scale-out clusters, and the like, but also the marked absense of everything else: Oracle Exadata, the big Unix vendors, etc. At least as far as the avant-garde is concerned, those are the tools of the past.
More to the point, some of the presenters managed to make having any servers at all seem somewhat quaint. In a talk enticingly titled ‘Supercomputing on a minimum wage,” Pete Warden, founder of OpenHeatMap and a former Apple programmer, told a crowd of well-heeled venture capitalists how he crawled and derived meaningful analyses of 210 million public Facebook profiles (for which he was subsequently sued) for about $100 in Amazon EC2 machine time.
“Did you know you can hire 100 servers from Amazon for $10 an hour?” he told the crowd.
Let me repeat that; A 100-node supercomputer for $10/hour.
What does this mean to data center managers? On the one hand, it means that data analysts and business intelligence folks are definitely impacting the architecture and design of the future data centers. But on the other, unless you work for a large cloud provider, the likelihood that you’ll have to design and manage those systems is relatively small.