Posted by: CarlBrooks
cloud benchmarking, cloud performance, cloudclimate.com, neat-o, network monitoring in the cloud, Paessler
The site consists of Paessler’s PRTG Network Monitor software running in various public cloud environments around the world. Each PRTG ‘sensor’ reports back information on perfomance, in real time, and displays each stream on the site for all the world to see. Founder Dirk Paessler said the idea struck him after he began testing his Windows-only software kit on newly available EC2 Windows images.
“We began to create a network of PRTG installations” he said, to see how each cloud would stack up in terms of performance. “My initial interest was to find a way to compare these clouds to each other,” he said. Public clouds were technologically diverse, and he wanted a way to visualize that in a rudimentary fashion for users.
The results, especially over time, are fascinating. Best performers for the money?
“Newservers has the highest performance.” Paessler said. “They’re an interesting case because they give you bare metal,” he said, despite selling capacity and time the same way Amazon does. That may give virtualization boosters pause for thought.
Does that mean everyone else is a poor relation? No, said Paessler. Amazon had excellent overall performance, and the major indicator for performance may be in how you consume, rather than who you consume.
“If you compare, on Amazon, a small CPU[instance] with a c1.medium, the c1.medium is giving you a lot more bang for the buck,” he said, something value-conscious cloud consumers may not realize. Different applications on different platforms may simply be better suited for one flavor of cloud over another, too. Paessler noted that his software was written with Intel processors in mind; using a provider who based their cloud on AMD CPUs showed a steep and inaccurate performance disadvantage.
Yet another twist was that cloud performance varied in a very great degree as tests moved from one cloud to another, Paessler said. The monitoring software tests short-term CPU capacity, data transfer, network response and similar metrics, and how a test turned out really depended on where you were watching from.
“We found out the connection from EC2-EU to EC2-US was very very fast, very reliable, but from EC2-EU to NewServers” it dropped off sharply, he said. Similar variations in response and performance were seen moving data to and from other clouds as well. Analysis is proving more complex than he had imagined. “We are talking about a [world-wide] web here,” he said.
Paessler doesn’t make too much of his new toy, however. He said that the site shows only the most basic and rough kind of information, and doesn’t take into account any number of factors. That will come with time as cloud matures, and as he finds ways to improve his little experiment.
“As with all benchmark testing, this is only a clue,” he said. “If you do consider cloud hosting, try two, three, four [providers] and try them with your application.” Don’t assume Paessler’s results hold true across the board.
In the meantime, the site will prove a fascinating time sink for statisticians, analysts and cloud watchers. Paessler said he put the site up partly to softsoap the cloud community, partly as a public service, and partly, just because he could.
With a few dozen reporting nodes, co-locating in so many locations would have cost a pretty penny, but in the cloud, the site doesn’t cost Paessler more than a few hundred dollars to run, and the results for observers are proving well worth it.