Amazon EC2

Amazon EC2 zap smash: everyone’s cool with it

In hindsight, the lightning-strikes-Amazon-data center story is a tidy little example of a nu-media bubble. Someone should make a graph of the coverage indexed by hysteria, outrage, maniacal prophesy and supposition and tweet it or something.

Having now had a nice talk with real live Amazon people, it seems they are treating it mostly as a public relations problem, and the real issue is transparency.

You see, for Amazon watchers, the Holy Grail is to find out exactly what and where Amazon’s servers are. But Amazon isn’t keen on handing out details, likely because the reality is messy and because they might be making it up as they go along. Those Amazon watchers might want to relax. Sure, Amazon is a going concern, but it doesn’t have the kind of scratch or incentive to re-invent the wheel server like Google or Microsoft do.

Further hurting Amazon’s cause is that most hosting companies are more than happy to tell you what they run. Verizon, for instance, recently boasted about its new “CaaS” hardware. Pricing also starts at $250/month, and that’s before you fire up a single server.

Amazon is trying to run away from that game and focuses on delivery. But after a certain point, people do really care about the nuts and bolts, since unlike semi-durable consumer goods, an EC2 instance is an ongoing concern, and users want to understand how their application is staying up (I know — so last century, right?).

I did have a chance to ask about Amazon’s hush-hush data center facilities. I didn’t get much more than a general admission that “Availability Zones” are usually located in different data centers, and that there are four in the US as of June 9. Amazon was also apparently startled to discover that one facility had electrical exposure to the Great Outdoors. That’s still progress. Hopefully, there’ll be more. I’m waiting, and I know lots of others are as well.

Light and dark in the Cloud: Amazon fails, Rackspace punts

In a succint one-day recap of the mouth-watering prospects of using cloud computing and the legless terror it can engender, website gdgt.com autoscaled gigantic traffic, and Amazon’s flagshp EC2 service went dark for hours in the night after an electrical storm.

That’s right- according to Amazon (mouse over the teeny little ‘i’s), “A lightning storm caused damage to a single Power Distribution Unit (PDU) in a single Availability Zone.” That means that one of its data centers in the US popped a cork and shut down an unspecified number of racks after a lightning strike.

The outage lasted from detection at 6:39PM PT on 6/10 to full availbility at 1:20AM PT on 6/11. That’s five hours of unexpected downtime, kids. Anyone running real-time applications or large batch jobs when their server got slammed? Any lost revenue/time/work? Lets check the SLA shall we? Graciously, Amazon will not be charging affected customers for services that went dark.

Amazon’s public stance on this so far? A pop-up window on their status page (see above).

On the other side of the coin, currently-quite minimalist gdgt streamed the keynote from Apple’s legendary dog-and-pony show, the Worldwide Developers’ Conference and incurred the expected Japanese-monster sized traffic spike. Reportedly traffic averaged 656 page views per second throughout the event, serving up something like 4.7 million total veiws by the end.

gdgt did this on Rackspace’s “Cloud Sites”, a scalable webhosting platform that starts small and charges users for extra capacity as needed. No word so far on exactly HOW much money they spent or saved, but it is presumably significantly less than if they had planned ahead and bought/rented the capacity that they needed. “Significant” in this context means “statistically observable”, by the way. Webhosting isn’t exactly a super-premium market at this point, this stunt probably didn’t ring up a staggering total.

We did try for comment; look for updates if the gdgt guys come through. Anyway, collective “good job!” for the penny-pinching whiz-bang, boys.

UPDATE: it was a few hundred bucks total according to Rackspace spokespeople.

Yet despite this cheerful little story of a moveable feast of web delivery, Amazon’s memento mori still sits at the head of table, an uncomfortable reminder that real-world “cloud delivery” means lightning, damaged data centers, and unexpected, unpreventable downtime with no recompense.

Citrix and Amazon hop into mindbending infinite-loop bed together, offer virtualizing of virtual servers in virtualized cloud

We’re sensing someone had an office pool going on “alliteration they could get away with” for this slug line: “Citrix Announces Citrix C3 Lab Built on Amazon Web Services to Connect Cloud Computing to the Corporate Datacenter”.

Anyways, what the mess of ess’ and clatter of consonants means is that Citrix, as part of its big show this week is announcing their Citrix’ XenApp software is now available for rent, from AWS, to run Citrix servers on Amazon’s cloud.

C3 stands for Citrix Cloud Center, their management suite for IaaS providers.

That’s the Citrix Citrix Cloud Center Lab, for those who collect poorly thought out product names.

The brain-pain part comes from the fact that Amazon itself runs its cloud on Xen, so a customer of Citrix Citrix would, technically, be using XenApps to use Xen virtual machines to manage and deploy public cloud resources to their customers as a customer using public cloud resources running on Xen virtual machines.

Hey, that may be great and just what C3 users want and it may work just fine, but it’s a hell of a rabbit hole to send your data down, ontologically speaking. We’re not prejudiced, we’re just trying to keep up.

Also, according to the article published today by SearchEnterpriseDesktop.com News Director Alex Barrett, you’ll be able to do it all from your iPhone. I think they should call the app, “DRINK ME”, but no-one reads anymore, they’ll never get it.

This is doubtless targeted at developers who want to play with Citrix distributed computing offerings, and the odd IT shop that wants to offer Citrix AND call itself “cloud” and can’t afford its own data center.

Really, though, it’s just another way of proving that Amazon’s cloud model is still the top of heap. Oracle, IBM, now Citrix, and others are essentially opening Amazon storefronts, only it’s buzzword-friendly grid computing instead of baby clothes and lawn sprinklers. Amazon doesn’t care- they have a two and a half year leap on everyone else wanting to offer public cloud instances, the best distribution channel extant and plenty of headroom. All they have to do is make space, keep the lights on, let their little playfriends from the middleware/manageware classroom play in their pool, and rake in the dough.

I’ll scrape together some time soon and poke around, see how good a job Citrix has done making this work and post some screenshots and an update in a day or so. Just a head’s up, ladies and gentlemen: I may never been seen again.

IBM pushes cloud plans, partnerships

If anyone can solve the problems associated with selling cloud computing to enterprises, it should be IBM. Today the company announced a step in that direction with a slew of new services and partnerships to build cloud services for businesses.

The key challenges to overcome are:

1) Identity management - who sees my application and data in the cloud? Security and regulatory requirements are crucial.

2) Which workloads and applications are appropriate for cloud computing?

3) How does my application in the cloud get access to my data which is still stored onsite?

IBM doesn’t have all these answers yet but says it is working with the following organizations to understand these issues.

Elizabeth Arden, Nexxera, The United States Golf Association, and Indigo Bio Systems sign on as new IBM cloud computing customers

IBM Global Services will offer data protection software “as a service” through the cloud, in addition to a new IBM cloud environment for businesses to safely test applications

First live demonstration of a global “overflow cloud” – IBM and Juniper Networks to install hybrid cloud capabilities across IBM’s worldwide Cloud Labs for customer engagements. This is to let users bridge between private clouds and IBM’s public cloud offerings to turn up resources as needed.

At 13 worldwide cloud centers, IBM offers server capacity on demand, online data protection, and Lotus e-mail and collaboration software.

IBM Rational AppScan 7.8 lets users continuously monitor the Web services they publish into the cloud to check that they are secure, compliant and meet business policies.

Service Management Center for Cloud Computing contains a set of offerings including Tivoli Provisioning Manager 7.1 and the new Tivoli Service Automation Manager, to automate the deployment and management of private clouds.

Finally, IBM said it will launch a Tivoli Storage as a Service offering through its Business Continuity & Resiliency Services cloud. Not available until late in 2009, users will be able to consume Tivoli data protection technologies via a cloud and pay for only what they use. EMC and Symantec are already offering these kinds of services.

From these announcements it looks like IBM will be able to help businesses figure out which workloads to shift into the cloud, but there are no details yet on how it will ensure identity management, security and compliance.

Amazon claims 400,000 web services users

An industry insider close to Amazon’s Web Services (AWS) business unit told us the company claims to have 400,000 customers using its web services offering.

AWS includes EC2, the compute-on-demand offering, S3, the hosted storage service, SimpleDB for hosted databases, Simple Queue Service (SQS) a communication channel for developers to store messages and CloudFront, which is a content delivery network.

Amazon has not publicly discussed much detail about its customers and how they are using AWS. For instance, of these 400,000 users, how many are using EC2 and S3, just S3 or just EC2? Is anyone using SimpleDB or CloudFront yet? How many of these users were one-time customers? My hunch is that 400,000 number includes any customer that has touched AWS regardless of whether they are still using it.

In conversations with IT users, it’s clear they are interested in these services, but need more reference cases on how to use it. A great success story goes a long way.

During a webinar on cloud computing today, James Staten, principal analyst at Forrester Research said enterprises need more transparency from EC2 to show that it can meet SLAs. “The predictability [of the service] is not good enough for business,” he said, noting that EC2 had two lengthy outages in 2008.  Small businesses and gaming and entertainment companies are the biggest adopters of EC2, he said. The former can’t afford to build their own datacenters, while gaming and movie companies require extra infrastructure around the release of new games and movies, which can be setup and torn down as needed.

Staten said enterprises are using cloud services like EC2 for R&D projects, quick promotions, partner integration and colloboration and new ventures.  He called for more companies to share how they are using these services and recommended that IT shops begin to experiment with it.  Staten suggested endorsing one to two clouds as “IT approved” and establishing an internal policy for using these services. He urged IT organizations to let cloud providers know what you want and what’s more important to you? Secure enterprise links, standards, SLA expectations, levels of support (24/7 phone support, for example)? My guess would be all of the above. If you’d rather, I can hammer on the vendors, so let me know.

VMware touts benefits of private cloud computing, VDC-OS

VMware, Inc. is on a mission to show companies that they can get the benefits of cloud computing without handing their mission critical applications over to an outside provider; with the upcoming Virtual Data Center-Operating System (VDC-OS), IT will be able to create secure, private cloud environments.

The yet to be released VDC-OS represents the evolution of the VMware Infrastructure; the platform, which is due for release sometime this year,  will transform traditional data centers into internal cloud environments. The business case for creating an private cloud is less complexity in the data center; software like VDC-OS will virtualize and automate systems to the point that there is less ‘knob turning’ and more time spent on tasks that improve business, said VMware Sr. Director of Product Marketing, Bogomil Balkansky.

“Too much of IT budgets are spent on management tasks and keeping the lights on, instead of on tasks that actually improve business,” Balkansky said. “Infrastructure complexities should not get in the way of this, but they do.”

While external clouds like Amazon EC2 offer the same benefits of internal clouds, VMware is betting that large enterprises won’t send their mission critical applications outside the four walls of their data centers to these providers. Instead, they will want to create private cloud compute infrastructures using software like VDC-OS.

“There are security challenges with public clouds; enterprises don’t trust [outsiders] with their customer and financial data,” Balkansky said.  “We want to transfer the notion of cloud computing to internal data center operations.”

VMware is also hosting a webinar on January 29 about Internal Cloud Computing, if you want to hear more on this.

Balkansky said private cloud computing environments will gain traction in large data centers, but that could just be a self-serving prophecy. After all, most public cloud providers won’t pay for VMware software and use free and open source Xen instead; hence, VMware has no place to go but within the enterprises that already know and love VMware.

While VMware is on an private cloud advocacy mission, as the largest virtualization provider on the planet, it can’t ignore the need to play well with public clouds. That’s where VMware’s vCloud initiative comes into play; it will eventually allow VMware users to move their virtual machines on demand between their datacenters and cloud service providers, and over 200 partners have signed up to support vCloud so far, Balkansky said.

Gartner VP predicts thousands of clouds

Gartner’s Vice President and distinguished analyst Tom Bittman spoke with us about the IT industry evolution led by virtualization and cloud computing, and why big players like VMware won’t be the virtualization software of choice.

Since virtualization is the foundation of cloud computing, clouds are the next logical step for virtualization vendors like VMware and Citrix Systems, but Bittman said if these vendors don’t make pricing changes, cloud platform providers like Google and Amazon won’t use them.

“Cloud computing is a wide open market, dominated by open source Xen. It is a market that is there for the taking, and for VMware that would require a significantly different pricing model,” said Bittman, who also blogs about virtualization and cloud computing. “Sun and Citrix could get a major foothold in the cloud market as well, if they get their act together.”

VMware has taken steps towards becoming cloud friendly with its Vcloud initiative, but Vcloud is limiting because the provider has to use VMware, Bittman said. Microsoft also has its own cloud service, Azure, supported by Hyper-V.

Microsoft will probably try to turn Azure into a platform for ISV’s to build software as a service, so “in a lot of ways, they are trying to build a platform for a cloud,” Bittman said. But, “there is no reason Windows will be a prominent player in the cloud…[because providers] like Amazon EC2 don’t care what the OS is; all they care about is what is being provided.”

The future of clouds; more providers, fewer OSes

Today, cloud computing is dominated by a small number of large providers, but in the years ahead there will probably be ecosystems built around those islands; Software as a Service (SAAS) built upon the existing clouds, and the sharing of resources between cloud providers, Bittman said. He also expects fragmentation from the few general cloud platforms of today into many specialty cloud providers with applications and infrastructures that cater to specific industries, like healthcare, which have specific compliance requirements, Bittman said.

“We will see a growth to thousands of cloud providers and they won’t want to write their own software using Xen; they will want to buy software and that is where companies like Sun could make a play,” Bittman said.

Cloud computing is also changing the game when it comes to operating systems; the concept of the Meta-OS (like VMware’s Virtual Data Center OS) is changing the paradigm of using one OS per physical server, Bittman said.”The old idea is you build one platform to manage one box, but if I have 10,000 boxes, I don’t want 10,000 OSEs managing everything independently,” Bittman said. “If I turn an OS into a dumb container, it can work in a much more distributed way, like Microsoft’s Azure, which is essentially Windows 2008 sprinkled all throughout the data center. This is changing the way we look at OSes going forward.”

Cloud computing has the power to change things in the IT industry because of what it offers companies; flexibility and agility, Bittman said.

“Most infrastructures today focus on cost, but we are beginning to see a focus shift towards agility. People are using [cloud environments] not because of the cost savings, but because it is flexible. The ability to make changes according to demand qickly is becoming a more important factor for data centers,” Bittman said.

Should Amazon EC2 follow Moore’s Law?

According to the economical side of Moore’s Law, processing power gets cheaper every year due to vendors being able to pack more of it in the same amount of space. Should cloud computing follow Moore’s Law?

Let’s take a look at Amazon’s computing offering in the cloud — the Elastic Compute Cloud (EC2). EC2 is a web service that lets customers rent Amazon servers on which they can host their own applications. There are different price levels called “instances.” The basic one is 10 cents an hour and, according to Amazon, is equivalent to a 32-bit system with 1.7 GB of memory,  1 EC2 Compute Unit, and 160 GB of instance storage.

So what is an EC2 Compute Unit? According to Amazon, it is equal to a 1.0-1.2 GHz 2007 Opteron or Xeon processor. When EC2 first came out in 2006, one EC2 Compute Unit was equivalent to an “early-2006 1.7 GHz Xeon processor” according to Amazon documentation on EC2 Instance Types.

(Two odd things about this: There’s a 20% difference between a 1.0 GHz processor and a 1.2 GHz processor, so what gives? And if a 1.0-1.2 GHz processor in 2007 is equivalent to a 1.7 GHz processor in 2006, why change the definition at all?)

The price has stayed the same since 2006 at 10 cents an hour for that basic instance, but you are getting the same amount of processing power now that you were in 2008. So it is not following the economic portion of Moore’s Law.

“You can now get a quad-core server for the same price you could get a single-core server in 2006. But cloud computing is not taking advantage of Moore’s Law,” said Raj Dutt, the CEO of Voxel Dot Net, a New York-based hosting company. “It’s the same price for the same amount of processing power.”

The question is whether it should. Clay Ryder, president of analyst firm Sageza Group, doesn’t necessarily think so. He sees EC2 and other cloud computing products to be based on a different pricing scheme. Whereas servers are based on a sales model that includes the cost of time, materials and markup, cloud computing is more of a values-based pricing model, and the two are not the same.

Ryder likened it to owning a car compared to renting or leasing. When you buy a car, you’re paying for the cost of materials to build it, the cost of labor it took to build (time), and any markup to make profit. When you rent one, you pay for a service.

“There is a lot of value in the Amazon approach,” he said. “You can turn it off and turn it on, and there’s no long-term costto you, and that is an intangible value.”

Ryder hits on something here. When you pay for EC2, you’re not just paying for the server hardware. You’re also paying for the data center infrastructure around it (land acquisition, building costs, power generation, chillers, racks, etc.) and the cost of labor it takes to maintain that infrastructure and the servers. If you have your own data center and your own people, you don’t pay for that when you go buy a dozen servers from Dell.

Some might still argue that at least a portion of the cost should be subject to Moore’s Law. After all, Amazon is charging the same price for the same amount of processing power, even though that processing power is getting cheaper for them to buy.

Then again, maybe Amazon has already factored in Moore’s Law, but has also factored in the increasing cost of labor, electricity, and materials to build a data center to run those servers. So in the end, it’s all a wash, and the price stays the same.

Cloud computing, are you in or out?

While the cloud is a new dimension of technology that IT managers and administrators will bake into the technology landscape, we have to make one fundamental decision: will the technology be in our out of our traditional data centers? This is a loaded question in regards to policy, security, compliance and a myriad of other categories that are very line-of-business specific.

One element than help give perspective for making this type of decision can be applied from basic management challenges. One important thing that I learned from working with various project managers is that in any engagement it is important to determine what you can manage and what you can (or cannot) control. When back-end components of the cloud reside outside of traditional internal data centers, we can manage the cloud — but not entirely control it. Part of this – identified in Lauren Horowitz’ post on this site – concerns the topics of transparency, service issues and cloud standards. When it comes down to it, if the back-end components of the cloud are outside the traditional data center, it cannot be fully controlled internally.

For offerings such as the Amazon EC2, Microsoft’s Azure, and Rackspace, these offerings are off site from internal data centers. With this parameter, decisions have to be made about what lies inside and outside traditional data centers. The alternative to the cloud back end being external, however, may not be as attractive and the time to market compared to a provided solution is inferior.

Building a cloud internally may be a daunting task for some organizations, especially when some of the more primary components may not already be in place. One mechanism that can truly enable an internal cloud is a virtualized server environment. In quantifying the virtual environment, it is not necessarily how many virtual machines or hosts are in use but the percentage of systems that are virtual machines. Along with that, another building block of a cloud is a storage grid for ultimate flexibility on data protection. Lastly, network capabilities are a pillar that defines the internal cloud. This can include the use of load-balancing and traffic-managing switches. With all of that, it becomes pretty clear that the costs and growing pains could be significant.

Make no mistake, there will be cloud computing success stories. But in the case of your own implementation — determining where the back-end cloud components reside will be a critical question that will need answering sooner than later.

Cloud computing allowed you to read an 1851 New York Times article online

Nicholas Carr recounts the story of the New York Times trying to get its archives all online, and using cloud computing to do it.

The short version: NYT scanned all their articles in, resulting in four terabytes worth of TIFF files. They wanted to convert them all the PDFs but weren’t capable of doing it inhouse. So a software programmer at NYT sent them all to Amazon’s Simple Storage System, created some code with Amazon’s Elastic Compute Cloud to convert them to PDFs, and voila, a day later it was done.

The total cost for the computing job? Gottfrid told me that the entire EC2 bill came to $240. (That’s 10 cents per computer-hour times 100 computers times 24 hours; there were no bandwidth charges since all the data transfers took place within Amazon’s system - from S3 to EC2 and back.)

If it wasn’t for the cloud, Gottfrid told me, the Times may well have abandoned the effort. Doing the conversion would have either taken a whole lot of time or a whole lot of money, and it would have been a big pain in the ass. With the cloud, though, it was fast, easy, and cheap, and it only required a single employee to pull it off. “The self-service nature of EC2 is incredibly powerful,” says Gottfrid. “It is often taken for granted but it is a real democratizing force in lowering the barriers.”

Which brings Carr to his main point: Cloud computing will be important for what it will be able to do that already can’t be done. Up to now, most people are focusing on how to transfer their current IT infrastructure into the cloud. But cloud computing will make its mark by opening up avenues that were previously closed, or not even built yet.

But as one commenter stated, moving existing infrastructure is going to naturally be the first focus, as enterprises are worried about their current infrastructure, and not necessarily new tasks that the cloud could tackle. It will take a more long-term visionary within the company (such as the chief technology officer) to figure out which new trenches to build.