Google’s vanguard platform service AppEngine has fallen down , hot on the heels of Rackspace’s storied hot mess on Monday
Follow the real-time nerd outrage on, what else, Twitter.
A Google spokesman, presumably Brett Slatkin, posted to the AppEngine Google Group about the outage saying the problems had started at approximately 6:30AM PST .
Very little is known about the infrastructure that underpins AppEngine, but entering “unplanned maintenance mode” and knocking your entire userbase off their data, especially when they have no alternative and no way to mitigate risk by using other platforms is certain to be a very low-level problem.
Rackspace recouped some of it’s sorely besmirched street cred by promptly and thoroughly communicating information about the outage, via Twitter, blogs and direct communication. It remains to been seen if Google will do the same.
More than 24 hours after users began reporting Rackspace hosted services were unresponsive, and the main site went dark, Rackspace has possibly set a new record for transparency and accountability, if not customer satisfaction, by tirelessly tweeting the entire episode.
They also ran to update the company blog (how droll, so Web 1.0, right?) and blamed power outages in their Dallas data center.
For additional amusment, see the
vulturescompetitors flock to #rackspacefail
An official statement has not been made and a request for comment has gone unanswered to date; so the root of the problem is still to be determined. Amazon’s recent calamity was exacerbated by lightning unaccountably penetrating a supposedly world-class data center- it’ll be interesting to see if Rackspace’s facilities have similar flaws.
UPDATE: Rackspace HAS NOT released their incident report, but it’s out in the wild. According to the report, which I won’t post, but will summarize from, since the content is fair game at this point: a mains breaker flipped and one line of generator backups had “excitation failure” which means they didn’t start up properly. Subsequently 3 banks of UPS batteries bled out and slammed a bunch of racks — which means they weren’t charging properly or worse, underdesigned for the load.
What this means in the simplest possible terms: “Heads Will Roll”. Between this and Amazon’s air-to-ground static electricity adventure, data center types are wagging their grimy, highly redundant fingers as hard as possible at these incidents.
UPDATE: the incident report is now public
HP plans to release its own open source cloud computing platform, according to the Director of HP’s Service Automation and Integration Labs, Chris Whitney. The Open Cirrus project, which HP Labs sponsors along with Yahoo! and Intel, was designed to put together “the ultimate stack of software people can use to build a cloud,” he said.
The project is a collaboration between the IT giants to pool computing resources at different sites into a “testbed” cloud and open it to researchers. Whitney said that Open Cirrus has about 300 researchers onboard since announcing partnerships with far-flung computer science labs in Russia, Malaysia and South Korea. He said HP has contributed about 10,000 computing nodes(virtual servers), Yahoo! about 3-4,000, and the other partners are kicking in at least 1,000 nodes each.
“We’re definitely envisioning an open-source software stack under the GPL” he said, similar to the eponymous LAMP software stack. He said they are experimenting with existing free cloud technologies (like EUCALYPTUS) and also gathering data from HP Labs’ own hardware installation, which runs on commodity HP data center servers.
Whitney said his team is also experimenting on a low level with using optical fiber communications on server backplanes and heating and cooling techniques in HP’s installation.
At a higher level, researchers are focusing on different applications for Hadoop, like data mining applications and “wide-area Hadoop” — data processing over distant geographical locations. There is a short list of current projects, but more are expected from the new partner sites.
Several open source cloud projects exist already, like Spanish Abiquo and UC-Berkeley’s EUCALYPTUS (released on Ubuntu 9.10), Canadian Enomaly, the Globus Nimbus project and others. Cloud leaders like Amazon and Rackspace run their clouds on open source technology but do not release their technology publicly. IBM is facilitating an EU-funded project called RESERVOIR, but it’s goals appear to be stretgic rather than practical.
HP’s entry, when and if it arrives, will mark the first open source cloud platform released by a major commercial vendor; certainly something to watch.
In hindsight, the lightning-strikes-Amazon-data center story is a tidy little example of a nu-media bubble. Someone should make a graph of the coverage indexed by hysteria, outrage, maniacal prophesy and supposition and tweet it or something.
Having now had a nice talk with real live Amazon people, it seems they are treating it mostly as a public relations problem, and the real issue is transparency.
You see, for Amazon watchers, the Holy Grail is to find out exactly what and where Amazon’s servers are. But Amazon isn’t keen on handing out details, likely because the reality is messy and because they might be making it up as they go along. Those Amazon watchers might want to relax. Sure, Amazon is a going concern, but it doesn’t have the kind of scratch or incentive to re-invent the
wheel server like Google or Microsoft do.
Further hurting Amazon’s cause is that most hosting companies are more than happy to tell you what they run. Verizon, for instance, recently boasted about its new “CaaS” hardware. Pricing also starts at $250/month, and that’s before you fire up a single server.
Amazon is trying to run away from that game and focuses on delivery. But after a certain point, people do really care about the nuts and bolts, since unlike semi-durable consumer goods, an EC2 instance is an ongoing concern, and users want to understand how their application is staying up (I know — so last century, right?).
I did have a chance to ask about Amazon’s hush-hush data center facilities. I didn’t get much more than a general admission that “Availability Zones” are usually located in different data centers, and that there are four in the US as of June 9. Amazon was also apparently startled to discover that one facility had electrical exposure to the Great Outdoors. That’s still progress. Hopefully, there’ll be more. I’m waiting, and I know lots of others are as well.
Salesforce claims 110,000 business applications are currently running on top of Force.com which competes against Amazon AWS, Google App Engine and Microsoft Windows Azure in the cloud computing application development market.
Offering a limited, free version is a typical ploy by software companies to get users hooked, and then when they need more functionality they are forced to buy a subscription. Salesforce.com began this way with a free version of its SaaS CRM product. They’d get the VP of sales to sidestep IT and buy it for themselves, then they’re in, along with all the company’s data and nobody can get them out.
Force.com Free Edition offers:
- one custom app
- one website with up to 250,000 page views per month
- up to 10 custom objects (custom database tables) per user
- a sandbox development environment to test the app or site before deploying it
- basic online training
- a library of sample apps
Companies that need more than one app, or to support more than 100 users, or more than one website must upgrade to a commericial subscription to Force.com that begins at $25 per user per month.
For more on CRM check out Voices of CRM
In a succint one-day recap of the mouth-watering prospects of using cloud computing and the legless terror it can engender, website gdgt.com autoscaled gigantic traffic, and Amazon’s flagshp EC2 service went dark for hours in the night after an electrical storm.
That’s right- according to Amazon (mouse over the teeny little ‘i’s), “A lightning storm caused damage to a single Power Distribution Unit (PDU) in a single Availability Zone.” That means that one of its data centers in the US popped a cork and shut down an unspecified number of racks after a lightning strike.
The outage lasted from detection at 6:39PM PT on 6/10 to full availbility at 1:20AM PT on 6/11. That’s five hours of unexpected downtime, kids. Anyone running real-time applications or large batch jobs when their server got slammed? Any lost revenue/time/work? Lets check the SLA shall we? Graciously, Amazon will not be charging affected customers for services that went dark.
Amazon’s public stance on this so far? A pop-up window on their status page (see above).
On the other side of the coin, currently-quite minimalist gdgt streamed the keynote from Apple’s legendary dog-and-pony show, the Worldwide Developers’ Conference and incurred the expected Japanese-monster sized traffic spike. Reportedly traffic averaged 656 page views per second throughout the event, serving up something like 4.7 million total veiws by the end.
gdgt did this on Rackspace’s “Cloud Sites”, a scalable webhosting platform that starts small and charges users for extra capacity as needed. No word so far on exactly HOW much money they spent or saved, but it is presumably significantly less than if they had planned ahead and bought/rented the capacity that they needed. “Significant” in this context means “statistically observable”, by the way. Webhosting isn’t exactly a super-premium market at this point, this stunt probably didn’t ring up a staggering total.
We did try for comment; look for updates if the gdgt guys come through. Anyway, collective “good job!” for the penny-pinching whiz-bang, boys.
UPDATE: it was a few hundred bucks total according to Rackspace spokespeople.
Yet despite this cheerful little story of a moveable feast of web delivery, Amazon’s memento mori still sits at the head of table, an uncomfortable reminder that real-world “cloud delivery” means lightning, damaged data centers, and unexpected, unpreventable downtime with no recompense.
Turning a funny little relational database project into one of the largest commercial open source companies ever seen, Mårten Mickos oversaw the growth of MySQL from 2001 all the way through its acquisition of MySQL by Sun in 2008. In February 2009, Mickos dusted Sun off his feet and presumably took a long vacation. Now he’s back in the news, joining the Board of Directors of three year-old RightScale.
There are parallels between RightScale and MySQL. Based on open source software? Check. Sitting pretty at the front of a burgeoning market? Check. Strong and contributing community of users? Check.
MySQL and RightScale also have a venture capital firm in common: Benchmark Capital. Michael Crandall, RightScale CEO said he met Mickos through that connection, and that Mickos was an informal advisor and friend.
RightScale is almost out of the nest now, with a high profile in cloud computing. RightScale provides management and tools for cloud consumers across all the major cloud providers. It boasts buzz-worthy new-school customers like Animoto and Sling Media and old-school giants like Capgemini. But, competitors are springing up like weeds, armed with funding and new ideas, and customers already paying for computing power can be hard to sell on additional spending.
Mickos was contacted for this article but declined to be interviewed because he didn’t wish undue publicity on the move. Crandall said that Mickos will act in an advisory role on strategy and business development.
Crandall sees other parallels between cloud computing and MySQL. In its early days, developers adopted the free database without approval from higher ups because of the way it was so well suited to the new class of Web-based, interactive applications they were writing. “In a similar fashion, cloud adoption is being done through ‘Shadow IT,’” said Crandall.
Leveraging Amazon’s cheap computers instead of free software, Crandall hopes to take RightScale down the same road, fostering users of its free suite of cloud management tools and selling sophisticated extras to those who don’t have the time or the expertise to do it themselves.
Whether or not RightScale and Crandall will be successful remains to be seen. Cloud computing and the marketplace growing around it is in its formative years, and the road to maturity is littered with cast-off toys that have been outgrown.
What follows is a semi-regular exposition on all the products announcements we can’t cover in longer form, kids. They’re all interesting technology, really neat in some cases, strategically interesting in others, but SearchCloudComputing.com really needs to show how real people are using said awesome technologies, and what’s really driving that use. That takes time and reporting, so stuff gets left in the mailbag and interviews get left on the floor. Hopefully we can push all the interesting stuff that won’t be a story into this kind of post from time to time.
For instance , I spoke with newly minted VP of cloud computing at CSC, Brian Boruff. CSC is a big ($16.7 billion, 92,000 employees) consulting firm, and is opening a cloud computing division. The only real thing they can offer you so far is ‘cloud orchestration consultants’ who will come in and take care of the nuts and bolts using dozens, if not hundreds of other peoples’ technologies in your business — making sure they all meet whatever regulatory needs you have, auditing and compliance and so forth. CSC is rustling up an “alliance partner” to resell a standard package of cloud services and IaaS later on this year. Boruff commented on the rapidly evolving cloud market, saying “we are the only large player that’s technologically independent — we don’t sell [hardware], we don’t sell software”, so he feels CSC will have some influence on what becomes “cloud standard.” Unless it guesses wrong, of course.
SOA software maker TIBCO announced a management-minded suite for developers who really want to play in the cloud but have pesky, grumpy IT managers with governance needs. TIBCO Silver will make sure the “operations guy understands everything that” that developers do in the cloud, even after the fact, say spokesman Phillip Tree. It does this by automating a slew of governance functions, like performance monitoring, version tracking, logging, etc. It allows formalized test environments to be set up, so developers can play in TIBCO Silver/Amazon EC2, and then take their work to the SOA boss, who can in turn start an official dev cycle with a minimum of shouting and headaches.
Managed services firm IP Services is using application virtualization from InstallFree to provide regulatory compliance to applications in the cloud. Given that you’re paying IP Services good money to hang on to your apps and data, one assumes that they are using InstallFree to ensure compliance in their own virtualized, multihomed environment, not farming your goods out to EC2 or something.
Open source Java Virtual Machine scalers Terracotta announced a partnership with VMware. Customers can virtualize everything on VMware, develop JVMs within Terracotta and presumably hold a raffle for all the servers they don’t need any more, as customers, one presumes, port their VMware images to compatible public clouds. While that’s neat, what this really is another arrow in proprietary VMware’s quiver against a cloud market dominated by open source.
Two gentle ribbings:
Cloud software vendor ParaScale released a “TCO calculator,” but it doesn’t seem to work, and there are no instructions. So, that wasn’t well thought out.
Hosting firm REDPLAID (subsidiary of Connectria) has decided that having a shopping cart on its website and offering VMware machines for rent constitutes a public cloud. Honestly? Maybe the on-demand billing, self-service portal, repository of machine images and scalable resources are on the roadmap, but this might constitute band-wagon jumping to more cynical observers.
Open source ringleader Mårten Mickos and former windmill-tilter-in-chief at developer darling MySQL, apparently couldn’t bear the gilded cage of Sun’s “Senior Vice President” offices. He fled Sun in February and now he’s signed on the Board of Directors at RightScale, the burgeoning cloud management start-up that is, akin to MySQL, built on open source software and that already owns a tidy piece of the Amazon middleware market.
Written statements from CEO Micheal Crandell and Mickos have a fair share of the usual blather:
“I am extremely pleased that Mårten will be contributing to the vision and direction of RightScale at a time of explosive growth for the company and the industry,” said Michael Crandell, RightScale CEO. “Mårten has proven to be one of the industry’s most innovative leaders, and we hope to tap his experience as we aggressively expand RightScale’s cloud management offerings to new markets.”
“RightScale has established itself as a leading cloud management platform, and the company continues to demonstrate tremendous momentum as its customer base and ecosystem expand with strategic partners like Sun and Canonical,” noted Mickos. “I look forward to offering my experience as a member of the RightScale board to help the company maintain and expand its leadership in a rapidly growing market.”
Mickos is a top gun in the IT world, but RightScale is only a venture-capital funded company at this point. Sun is, well, part of Oracle now, but it makes RightScale look like small fry.
What’s behind Mickos’ move? Does he really want a new challenge? Did he know about Oracle eating Sun? Inquiring minds want to know.
It’s not entirely clear what Mickos’ new role will be, but it’s always safe to assume that if they announced the news, they want you to know, whatever their reasons. Stayed tuned.
The following post,
Enomaly owns CloudCamp™ – has it jumped the shark?, by Sam Johnston, working group secretary at Open Grid Forum, is definitely ‘inside baseball’ but an interesting look at how proprietary interests are battling open standards in the cloud.
The conversation reminds me of The People’s Front of Judea sketch in Monty Python’s The Life of Brian. Check it out.