Cloud storage

Microsoft says it has recovered Sidekick data

The Sidekick data-loss debacle may be drawing to a close.

According to a post on Microsoft’s website by corporate vice president Roz Ho,

We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

Ho also went on to provide some further details as to what caused the outage and how it was handled:

We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data…we have made changes to improve the overall stability of the Sidekick service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained.

All’s well that ends well, but I do wonder if this will make people more conscientious about making local copies of important data sent to a public cloud.

EMC denies blog claim that its SAN was involved in Sidekick outage

EMC officials are saying today that a new blog post, which cites an anonymous source as saying an EMC SAN was involved in the recent Sidekick outage, is inaccurate.

According to the blog post, which appeared at RoughlyDrafted Magazine:

To the engineers familiar with Microsoft’s internal operations who spoke with us, that suggests two possible scenarios. First, that Microsoft decided to suddenly replace Danger’s existing infrastructure with its own, and simply failed to carry this out. Danger’s existing system to support Sidekick users was built using an Oracle Real Application Cluster, storing its data in a SAN (storage area network) so that the information would be available to a cluster of high availability servers. This approach is expressly designed to be resilient to hardware failure.

[..]

Danger’s Sidekick data center had ”been running on autopilot for some time, so I don’t understand why they would be spending any time upgrading stuff unless there was a hardware failure of some kind,“ wrote the insider. Given Microsoft’s penchant for ”for running the latest and greatest,“ however, ”I wouldn’t be surprised if they found out that [storage vendor] EMC had some new SAN firmware and they just had to put it on the main production servers right away.“

Reached for comment today, an EMC spokesperson said no EMC products were involved.

Another blog yesterday also cited an anonymous source in saying that a SAN upgrade project allegedly involved in the outage was outsourced to Hitachi, but did not identify the brand of SAN involved. Multiple HDS spokespeople have not returned phone calls and emails seeking comment since yesterday.

A Microsoft spokesperson made the following comment for Storage Soup:

I can clarify that the Sidekick runs on Danger’s proprietary service that Microsoft inherited when it acquired Danger in 2008. The Danger service is built on a mix of Danger created technologies and 3rd party technologies. However, other than that we do not have anything else to share right now.  

It actually may not matter at the end of the day whose SAN it was — it seems it was human error (or, as the RoughlyDrafted blog goes on to speculate, possible sabotage) responsible for the outage. The RoughlyDrafted blog goes on to claim:

A variety of ”dogfooding“ or aggressive upgrades could have resulted in data failure, the source explained, ”especially when the right precautions haven’t been taken and the people you hired to do the work are contractors who might not know what they’re doing.“ The Oracle database Danger was using was ”definitely one of the more confusing and troublesome to administer, from my limited experience. It’s entirely possible that they weren’t backing up the ’single copy’ of the database properly, despite the redundant SAN and redundant servers.“

“Just because there may have been an error during a SAN upgrade doesn’t mean the guy’s an idiot or that the storage vendor’s stuff doesn’t work. The fundamental question here is where are the backups?” said backup expert W. Curtis Preston.

This remains an open question as of this hour, as a new statement issued by T-Mobile suggests there may be some data that’s recoverable– “We…remain hopeful that for the majority of our customers, personal content can be recovered.”

A New York Times report released this week cited a T-Mobile official as saying data on the Sidekick server and its backup server were corrupted.

But it also can’t be assumed that thorough secondary copies of data were made by the cloud service. Slightly higher-end online PC backup services like Carbonite and SpiderOak, previously questioned about geographic redundancy available for their services should their primary data centers fail (this following a high-profile outage and lawsuit for Carbonite–where users experienced data loss), have cited costs and pricing pressures as reasons for not offering that level of redundancy for consumer customers.

Another important point in all this is that users might not be losing data if they synced data to their PCs as well as the cloud. T-Mobile offers an IntelliSync service for a fee to sync data between the Sidekick and the PC; there are also free synchronization clients available online. Users would’ve had to have those services in place prior to the outage, however.

“The bottom line is that a free cloud service shouldn’t be your only copy of data,” Preston said.

Hitachi implicated in Sidekick outage

News broke this morning of an outage for users of the Sidekick mobile smartphone,  in which T-Mobile warned users of the device not to power down their phones, or personal data would be irretrievably lost thanks to a server outage at Danger, a Microsoft subsidiary that supports the Sidekick.

Meanwhile, Engadget has blogged that the storage and backup infrastructure at Danger was to blame for the outage:

Alleged details on the events leading up to Danger’s doomsday scenario are starting to come out of the woodwork, and it all paints a truly embarrassing picture: Microsoft, possibly trying to compensate for lost and / or laid-off Danger employees, outsources an upgrade of its Sidekick SAN to Hitachi, which — for reasons unknown — fails to make a backup before starting. Long story short, the upgrade runs into complications, data is lost, and without a backup to revert to, untold thousands of Sidekick users get shafted in an epic way rarely seen in an age of well-defined, well-understood IT strategies. 

If confirmed, it would be the second high-profile outage Hitachi has been associated with in the last six months. An HDS SAN was also implicated when Barclay’s ATMs in the UK stopped working in June.

Regardless of the source of the failure, outages like this usually draw attention to the fundamental risk of cloud computing — the things that can happen when all of users’ data “eggs” are put in one service provider’s “basket.”

Requests for comment are in to Microsoft and HDS and have not yet been returned. Stay tuned.

Zmanda spruces up Windows cloud backup

Open-source data backup software company Zmanda Inc. is releasing version 2.0 of its Zmanda Cloud Backup (ZCB) for Windows today.

New features include:

• Geography control - customers can tag data so that it’s backed up to a cloud data center in a certain region. For ecample, users in Europe can specify data that has to stay in Europe per European Union regulations. Customers can also choose to send data to data centers closest to their location for better performance of data migrations and retrieval over the network.
• Selective restore - the ability to restore one file from a data set; not new for Zmanda’s main backup product, but new for ZCB.
• Windows Security Certificate Encryption - Previously data sent to the cloud through ZCB was encrypted using standard AES encryption; support for the Windows certificate “is the highest level of encryption for Windows systems,” said Zmanda CEO Chander Kant. “It means they can use the same certificate they’re used to if they encrypt files on their Windows server and can make bare-metal restores for DR easier.”

Zmanda Cloud Backup 1.0 was first released last December. Kant said there are currently about 100 customers using it to backup systems to the cloud.

Storage blogosphere rumbles with rumors of EMC Atmos Compute

Reports from storage industry blogs and an a EMC Corp. customer event suggest EMC is getting ready to launch a new cloud computing service based on its Atmos onLine storage as a service offering.

Jeff Darcy, a software engineer and author of a blog dubbed Canned Platypus, reported after a meetup event in the Boston area August 14 that an EMC presentation on Atmos had divulged plans to offer a compute service like Amazon’s Elastic Compute Cloud (EC2):

They’ll be rolling out a compute service “by the end of the year” to support in-data-center access to data. It looks like it’ll be roughly comparable to EC2 plus S3/EBS [Elastic Block Storage]; there was no mention of supporting other features like SDB [Amazon Simple DataBase]/SQS [Simple Queue Service], and of course EMC pricing is likely to keep people on Amazon.

Meanwhile, Nirvanix evangelist Stephen Foskett turned up a link to a copy of the EMC presentation described by Darcy, which contains the following slide:

Unlike Darcy, Foskett believes this service will be “a real EC2 challenger,” pointing out some differentiators (bold in the original):

• They (probably) use VMware ESX, which is more common and familiar than Xen. Atmos Compute Service might even be able to handle existing ESX instances migrated in from private servers!
• Atmos onLine storage supports NFS in addition to the Atmos API, unlike Amazon’s own S3 which is API-only.
• They offer VLANs for enhanced network security, which Amazon lacks.
• They seem to offer per-instance internal persistent IP addresses, another area of frustration for EC2 users.

This seems like a likely candidate to be announced at VMWorld in less than two weeks because it ties in with VMware’s persistent cloud messaging. Despite VMware’s marketing efforts, some public cloud service providers such as RackSpace say they find the open-source XenServer more customizable. It would make sense for VMware’s parent company to throw its weight behind VMware as a public cloud offering at this year’s show.

Amazon fleshes out S3 sneakernet

Amazon now supports data export from its S3 storage cloud onto customers’ removable hard drives.

Amazon first opened up this “sneakernet” for import/upload to the Amazon cloud earlier this spring, allowing customers with large data sets to send the data to Amazon on removable media rather than trying to migrate the data over an Internet connection. This most recent announcement means users can extract data from the cloud using this method, too.

At the time of the first announcement, Amazon bloggers referenced the quote that immediately jumped to my mind reading about the export feature: “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.”

Amazon is far from the first or only cloud storage vendor to use seeding devices to get large data sets into the cloud rather than trying to squish terabytes through the average broadband Internet connection. Indeed, this network bottleneck is considered one of the biggest barriers to cloud computing adoption to date, and cloud backup vendors including EMC’s Mozy already send out seeding devices to upload or restore terabytes of data.

Companies such as NetEx are also offering software that promises to cut down on bandwidth between service providers and consumers downloading large, say, video files from centralized data centers. Others, including Cleversafe, are proposing to split data into chunks and among multiple sites to cut down on bandwidth and preserve data security.

So far, however, for the largest data sets — as this Amazon announcement demonstrates — nobody’s quite beaten the highway.

Zmanda opens up cloud backup API

Open source backup software vendor Zmanda Inc., among the first to offer customers a direct link between its backup software and cloud storage, is opening up its API for connecting backup software with cloud storage service providers to other vendors, including competitors.

CEO Chander Kant says Zmanda is sponsoring a new open source project called ZCloud, which offers an API to show how backup sofware connects to storage cloud service providers to avoid duplicating work as more backup vendors offer the option. “Today every backup softrware has to talk a different language,” Kant said. “This will get rid of those idiosyncrasies and make interoperability easier.”

The subject of standard cloud APIs and interoperability is a hot one in the still-nascent cloud computing market. While some are already calling for or developing standardized cloud interfaces, others say it’s too early to establish industry-wide standards without quashing differentiation.

Kant points out that an API specifically for backup tools isn’t the same as industry-wide, homogenizing standardization. “This is how standardization is actually going to happen — not an overall set of specs, but specific ones based on specific use cases,” he said.

He added that while the API specifies common aspects of connecting backup to the cloud, there’s no reason a backup software vendor can’t add its own differentiators to the integration. “The API allows for discovery of the underlying storage clouds, keeping developers from having to repeat basic stuff,” he said. At this early stage, he says “most people are first trying to get to a basic level of functionality.”

SGI releases first post-acquisition product

SGI has launched its first enterprise data storage product since Rackable bought Silcon Graphics Inc. for $42.5 million and changed its name to SGI following the acquisition.

The new product, CloudRack X2, is a scaled-down version of its CloudRack C2, which is shipped already racked. CloudRack X2 holds up to nine TR2000 trays, which contain CPU as well as up to eight 3.5-inch SATA hard disk drives. C2, by contrast, holds up to 20 drives. X2 can be deployed as a cabinet or slotted into an existing rack.

SGI says it can customize the hardware it ships in either product, offering a range of Intel Xeon and AMD processors and user-configurable CPU/ memory/capacity combination. The goal is to sell this product into Internet service providers as well as HPC and digital graphics shops such as post-production editing. SGI is the latest in a list of HPC scale-out vendors to bring their high-end products downmarket, although unlike Isilon and BlueArc, SGI is not targeting mainstream enterprises with new software.

“The hardware design has been tailored for cloud computing,” SGI senior product marketing director Geoffrey Noer said. “Most of those service providres have their own custom software — one of our big value propositions is that we can tune our hardware to their specific apps.”

Ah, there’s that word again: “cloud.” We reported Thursday on some new plans brewing behind the scenes at NetApp and Emulex in the cloud storage space, as well as the commentary from several industry analysts about how, well, cloudy this buzzword has become. Just look at these three companies alone — one, NetApp, is focused on delivering IT as a service within enterprise data centers through a virtual infrastructure; another, Emulex, is focusing on connectivity between internal enterprise infrastructure and external service provider data centers; a third, SGI is offering little in the way of virtualization or software integration and cites the hardware design in defining its system as suitable for the cloud.

“It’s like trying to bag smoke right now,” Illuminata analyst John Webster said of pinning down a technical definition for the “cloud” buzzword.

However, one thing is abundantly clear: whatever the cloud turns out to be, vendors are gung-ho about it leading into fall. Expect further developments, particularly from NetApp. Then we’ll have to see whether end users follow.

Storage vendors still forecasting heavy cloud cover

If you think the hype about cloud computing has peaked, think again.

Vendors who build their hardware and software around cloud computing say it has a solid grasp on the service provider market and is getting ready to cover the enterprise.

Sajai Krishnan, CEO of cloud storage startup ParaScale, says service providers are heavyily committed to the cloud and others are coming around as they grasp its value.

“The cloud is still a fuzzy concept, despite all this religion around it,” Krishnan said “You need to spend a couple of hours talking about it, then the light bulb goes off. But for people who only read the occasional article, it still raises more questions than it answers.”

He thinks enterprises are ready to turn to clouds, initially with the help of service providers.

“It’s not either/or,” he said. “For certain large applications, we’ve seen folks roll out an internal cloud and for other things they go external. When you have an application that’s running inside an enterprise four or five years, take it out and put a VMware wrapper on it, test it and put it back into production, an IT shop doesn’t have the cycles to do that. If a service provider has the expertise to handle something like that, it drives up the value of cloud storage inside these companies very quickly.”

During his company’s earnings call with analysts Monday night, 3PAR CEO Dave Scott said the storage systems vendor is anticipating a shift toward the cloud. 3PAR has long billed itself as a utility storage company and counts service providers as large customers.

“We believe that we are in the midst of major secular trend with cloud computing as an ultimate replacement of much of the information technology that is currently owned and operated by enterprises,” Scott said.

For now, much of that trend is driven by service providers.

Tier1 Research pegs the cloud service market at around $300 million this year with cloud storage making up about 40 percent to 60 percent of that. Tier1 analyst Antonio Piraino says he considers storage “low-hanging fruit” among cloud services.

Krishnan says the service provider market is starting to split into three areas. One is the mass-market cloud service providers such as Amazon S3, Google, and Rackspace. The second area is smaller providers who combine virtualization, multi-tenant storage clouds and private hosted clouds. The third segment consists of the large telcos such as AT&T, etsVerizon Business, and Deutsche Telecom.

“Service providers are beyond the confusion and in the fourth quarter you’ll see a whole slew of announcements around cloud services,” Krishnan said. “They know the technology. On the enterprise side, there’s still a lot of confusion. When you talk about cloud, Amazon comes to mind first.”

SunGard exec says storage will be ‘final domino’ in cloud interoperability

SunGard’s technical officer for cloud computing, Don Norbeck, talked with Storage Soup this afternoon on the service provider’s participation in the Distributed Management Task Force (DMTF) Open Cloud Standards Incubator and the “physics problem” that currently stands between IT and true application portability.

Storage Soup: Tell me about the standards body you joined and why…
Norbeck: DMTF has a good track record with previous initiatives. They brought VMware, Microsoft and Citrix to the table and got them to agree to include metadata to allow a base level of interoperability between them for the Virtualization Management Initiative (VMAN). The Open Virtualization Format (OVF) is similarly impressive to us.

SS: Did you just join the group this week? Is it a new initiative?
Norbeck: It’s relatively new - the group formed this April, and SunGard was part of that initial discussion. The news today is that we petitioned to be included in the leadership board and were just approved.

SS: Who else is participating in this standards effort?
Norbeck: Other members of the initiative include Cisco, EMC, VMware, Microsoft, HP, AMD, Rackspace, Savvio and Sun. Right now it’s an incubator discussion to define basic components of the cloud and how they should be administered. We don’t often participate in standards efforts, but we see extreme value as a service provider in being involved in this conversation early on.

SS: What kinds of things will the incubator be defining? What does it have to do with SunGard’s disaster recovery business?
Norbeck: Our first hypothesis is that there are going to be hundreds of different clouds out there with different characteristics - some optimized for speed, some for cost and some for availability. The cloud will serve two purposes: avoiding downtime and the expansion of infrastructure for peak demand. How much capacity you can spin up and how quickly you can fail over to a cloud data center depends on an up-front information exchange between the end user and the provider to tell how much and what to spin up for true application portability.

SS: I always thought cloud standards had more to do with interoperability between service providers - I thought the way users send data to service providers is already relatively well understood.
Norbeck: Before you can float workloads between service provider infrastructures, you have to figure out first how users move the workload beyond their firewall. That’s the first step. If we can all agree on application portability standards within that framework, we may be able to set something up where you can follow the sun from an electrical power perspective.

SS: Will the standard address how to move data over distance? Seems like that’s a hurdle VMware is trying to overcome right now, for example.
Norbeck: We’re still limited by distance. Network data transmission capacity is still a scarce resource. I’m not sure what the solution is - maybe enabling content delivery networks for branch offices so there are small bits of critical data everywhere, or leveraging some WAN acceleration technology in between. Storage is going to be the final domino to fall for the model of computing platform cloud aspires to be.

SS: How would you answer those who say it’s too early at this stage of the cloud to start imposing standards?
Norbeck: With any standards effort, the proof is in the actual utilization of the standard. This effort is more at the discussion stage, in which we’re looking to agree to language that will enable our customers to utilize us better. It’s too early to impose World Wide Web (WWW) type standards on cloud computing, but it’s not too early for the conversation.