Talking Data Podcast

Feb 25 2018   3:43AM GMT

LinkedIn open sources Dynamometer load and stress tester for HDFS

Jack Vaughan Jack Vaughan Profile: Jack Vaughan


The scale of HDFS continues to soar upward. For large social media and cloud providers, the size of Hadoop clusters is such that it is hard to test out this basic component of classic Hadoop at scale before roll outs. That is another one of those niggling issues that slows Hadoop adoption.

At LinkedIn, the challenges of successfully making even small configuration changes across broad arrays of HDFS led a team to create Dynamometer. This load and stress test suite uses actual NameNodes, combined with simulated DataNodes to prove out changes to settings across large the Hadoop data farms that help LinkedIn link folks together.

Today, many issues are impossible to test out without running a cluster that is similar in size to what were is used in production, according to Carl Steinbach, senior staff software engineer at LinkedIn. He said that one of the goals of the project is to have positive “upstream effect” on the work of Apache community members’ releases, to, in effect, make testing at scale foremost in the effort.

Steinbach’s colleague, engineer Erik Krogen, adds that HDFS developers are looking toward the day when they can find bugs before new versions are committed, rather than six -months later when new software comes to cover very large scale clusters.

In this edition of the Talking Data Podcast, the crew speaks with analyst Mike Matchett, of the SmallWorldBigData consultancy, to get a better view into Dynamometer. Testing tools of this kind will only gain in importance going forward, he suggested.

“Whatever big data we have today, it’s going to be bigger tomorrow,” said Matchett.

Also on tap in this podcast is a discussion of TensorFlow from Google. The company has supported work on this machine learning framework on CPUs, GPUs, and, most recently TPUs – these being Tensor Processing Units build especially to accomplish highly iterative neural network computations. Word is that Google is preparing to open up TPU processing to outsiders that use its Google Cloud Platform. – Jack Vaughan


 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: