Starting in April 2009, Nasuni put 16 cloud storage providers through stress tests  to determine how they handled performance, availability and scalability in real-world cloud operations.

Only six of the initial 16 showed they are ready to handle the demands of the cloud, Nasuni claims, while some of the others failed a basic APIs functionality test. Amazon Simple Storage Service (S3) and Microsoft Azure were the leaders, with Nirvanix, Rackspace, AT&T Synaptic Storage as a Service and Peer 1 Hosting also putting up passing grades.

“You won’t believe what is out there,” Nasuni CEO Andres Rodriguez said. “Some had awful APIs that made them unworkable. Some had some crazy SOAP-based APIs that were terrible.”

Nasuni did not identify the providers that received failing grades, preferring to focus on those found worthy. Amazon and Microsoft Azure came out as the strongest across the board.

Amazon S3 had the highest availability with only 1.43 outages per month – deemed insignificant in duration – for a 100% uptime score. Azure, Peer 1 and Rackspace all had 99.9% availability.

Rodriguez described availability as the cloud providers’ ability to continue operations and receive reads and writes, even through upgrades. “If you can’t be up 99.9 percent, you shouldn’t be in this business,” he said.

For performance testing, Nasuni looked at how fast providers can write and read files. Their systems were put through multiple, simultaneous threads, varying object sizes and workload types. They were tested on their read and write speed of large (1 MB), medium (128 KB) and small (1 KB) files.

The tests found S3 provided the most consistently fast service for all file types, although Nirvanix was fastest at reading large files and Microsoft Azure wrote all size files fastest.

Nasuni tested scalability by continuously writing small files with many, concurrent threads for several weeks or until it hit 100 million objects. Amazon S3 and Microsoft Azure were also the top performers in these tests. Amazon had zero error rates for reads and writes. Microsoft Azure had a small error rate (0.07%) while reading objects.

The reported stated: “Though Nirvanx was faster than Amazon S3 for large files and Microsoft Azure was slightly faster when it comes to writing files, no other vendor posted the kind of consistently fast service level across all file types as did Amazon S3. It had the fewest outages and best uptime and was the only CSP to post a 0.0 percent error rate in both reading and writing objects.”