MapReduce

Overheard - MapReduce is a screwdriver

“Just because you’ve got the best hammer in the entire world doesn’t make everything a nail. If you’ve got a screw, even a cheap, old, rusty screwdriver is going to do a better job. And MapReduce is a lot better than a cheap, old, rusty screwdriver.” Mark C. Chu-Carroll, Databases are hammers; MapReduce is a screwdriver

RDBs are absolutely brilliant things. They’re amazing tools, which can be used to build amazing software. I’ve done a lot of work using RDBs, and without them, I wouldn’t have been able to do some of the work that I’m proudest of. I don’t want to cut down RDBs at all: they’re truly great. But not everything is a relational database, and not everything is naturally suited towards being treated as if it were relational. The criticisms of MapReduce all come down to: “But it’s not the way relational databases would do it!” - without every realizing that that’s the point. RDBs don’t parallelize very well: how many RDBs do you know that can efficiently split a task among 1,000 cheap computers? RDBs don’t handle non-tabular data well: RDBs are notorious for doing a poor job on recursive data structures. MapReduce isn’t intended to replace relational databases: it’s intended to provide a lightweight way of programming things so that they can run fast by running in parallel on a lot of machines. That’s all it was intended to do.

Overheard: What’s inside IBM’s Blue Cloud

“Blue Cloud is based on an open-source project called Hadoop that manages computing resources across large clusters of computers. Hadoop includes an open-source version of MapReduce, the same software Google uses to efficiently distribute its computing chores across its servers around the world.” Erick Schonfeld, IBM’s Blue Cloud is Web Computing By Another Name