Azure infrastructure

The Azure cloud(s) is (are) built on Microsoft’s definition of commodity infrastructure. It’s “Microsoft Blades,” that is, bespoke OEM blade servers from several manufacturers. It’s probably Dell or HP, just saying, in dense racks. Microsoft containerizes its data centers now and pictures abound; this is only interesting to data center nerds anyway.

For systems managements nerds, here’s a 2006 presentation from Microsoft on the rudiments of shared I/O and blade design.

Azure considers each rack a ‘node’ of compute power and puts a switch on top of it. Each node — servers+top rack switch — is considered a ‘fault domain’ (see glossary, below), i.e., a possible point of failure. An aggregator and load balancers manage groups of nodes, and all feed back to the Fabric Controller (FC), the operational heart of Azure.

The FC gets it’s marching orders from the “Red Dog Front End” (RDFE). RDFE takes its name from nomenclature left over from Dave Cutler’s original Red Dog project that became Azure. The RDFE acts as kind of router for request and traffic to and from the load balancers and Fabric Controller.

Russinovich said that the development team passed an establishment called the “Pink Poodle” while driving one day. Red Dog was deemed more suitable, and Russinovich claims not to know what sort of establishment the Pink Poodle is.

How Azure works
Azure works like this:

You/Azure portal

|___RDFE
• |___Aggregators and Load Balancers
  • |___Fabric Controller
    • |__Nodes

The Fabric Controller

The Fabric Controller does all the heavy lifting for Azure. It provisions, stores, delivers, monitors and commands the virtual machines (VMs) that make up Azure. It is a “distributed stateful application distributed across data center nodes and fault domains.”

In English, this means there are a number of Fabric Controller instances running in various racks. One is elected to act as the primary controller. If it fails, another picks up the slack. If the entire FC fails, all of the operations it started, including the nodes, keep running, albeit without much governance until it comes back online. If you start a service on Azure, the FC can fall over entirely and your service is not shut down.

The Fabric Controller automates pretty much everything, including new hardware installs. New blades are configured for PXE and the FC has a PXE boot server in it. It boots a ‘maintenance image,’ which downloads a host operating system (OS) that includes all the parts necessary to make it an Azure host machine. Sysprep is run, the system is rebooted as a unique machine and the FC sucks it into the fold.

The Fabric Controller is a modified Windows Server 2008 OS, as are the host OS and the standard pre-configured Web and Worker Role instances.

What happens when you ask for a Role

The FC has two primary objectives: to satisfy user requests and policies and to optimize and simplify deployment. It does all of this automatically, “learning as it goes” about the state of the data center, Russinovich said.

Log into Azure and ask for a new “Web Role” instance and what happens? The portal takes your request to the RDFE. The RDFE asks the Fabric Controller for the same, based on the parameters you set and your location, proximity, etc. The Fabric Controller scans the available nodes and looks for (in the standard case) two nodes that do not share a Fault Domain, and are thus fault-tolerant.

This could be two racks right next to each other. Russinovich said that FC considers network proximity and available connectivity as factors in optimizing performance. Azure is unlikely to pick nodes in two different facilities unless necessary or specified.

Fabric Controller, having found its juicy young nodes bursting with unused capacity, then puts the role-defining files at the host. The host OS creates the requested virtual machines and three Virtual Hard Drives (VHDs) (count ’em, three!): a stock ‘differencing’ VHD (D:\) for the OS image, a ‘resource’ VHD (C:\) for user temporary files and a Role VHD (next available drive letter), for role specific files. The host agent starts the VM and away we go.

The load balancers, interestingly, do nothing until the instance receives its first external HTTP communication (GET); only then is the instance routed to an external endpoint and live to the network.

The Platform as a Service part

Why so complicated? Well, it’s a) Windows and b) the point is to automate maintenance and stuff. The regular updates that Windows Azure systems undergoes — same as (within the specifications of what is running) the rest of the Windows world — happen typically about once a month and require restarting the VMs.

Now for the fun part: Azure requires two instances running to enjoy its 99.9% uptime service-level agreement (SLA), and that’s one reason why. Microsoft essentially enforces a high-availability, uninterrupted fault tolerance fire drill every time the instances are updated. Minor updates and changes to configuration do not require restarts, but what Russinovich called ‘VIP swaps’ do.

Obviously, this needs to be done in such a way that the user doesn’t skip a beat. A complicated hopscotch takes place as updates are installed to the resource VHD. One instance is shut down and the resource VHD updated, then the other one. The differencing VHDa makes sure new data that comes into the Azure service is retained and synced as each VM reboots.

Virtualization and security

What is it running on, we asked? Head scratching ensued for many moons as Microsoft pushed Hyper-V to customers but claimed Azure was not compatible or interoperable with Hyper-V.

It is, in fact, a fork of Hyper-V. Russinovich said it was basically tailored from the ground up for the hardware layout that Microsoft uses, same as the Azure OSes.

Russinovich said that the virtual machine is the security boundary for Azure. At the hypervisor level, the host agents on each physical machine are trusted. The Fabric Controller OSes are trusted. The guest agent- the part the user controls—is not trusted. The VMs communicate only through the load balancers and the public (user’s endpoint) IP and back down again.

Some clever security person may now appear and make fun of this scheme, but that’s not my job.

The Fabric Controller handles network security and Hyper-V uses machine state registries (MSRs) to verify basic machine integrity. That’s not incredibly rich detail, but its more than you knew five minutes ago and I guarantee its more than you know about how Amazon secures Xen. Here’s a little more on Hyper-V security.

New additions to Azure, like full admin rights on VMs (aka elevated privileges) justify this approach, Russinovich said. “We know for a fact we have to rely on this [model] for security,” he said.

Everyone feel safe and cozy? New user-built VM Roles are implemented a little differently

Azure now offers users the ability to craft their own Windows images and run them on Microsoft’s cloud. These VM Roles are built by you (sysprep recommended) and uploaded to your blob storage. When you create a service around your custom VMs and start the instances, Fabric Controller takes pains to redundantly ensure redundancy. It makes a shadow copy of your file, caches that shadow copy (in the VHD cacher, of course) and then creates the three VHDs seen above for each VM needed. From there, you’re on your own; Microsoft does not consider having to perform your own patches an asset in Azure.

A healthy host is a happy host

Azure uses heartbeats to measure instance health: It simply pings the Fabric Controller every few seconds and that’s that. Here again, fault tolerance is in play. You have two instances running (if you’re doing it right. Azure will let you run one, but then you don’t get the SLA). If one fails, the heartbeat times out, the differencing VHD on the other VM starts ticking over and Azure restarts the faulty VM, or recreates the configuration somewhere else. Then changes are synced and you’re back in business.

Do not end these processes

Now that we have the ability to RDP into our Azure Roles and monkey around, Russinovich helpfully explains that the processes Azure runs within the VM are WaAppHost.exe (Worker Role), WaWebHost.exe (Web Role), clouddrivesvc.exe (All Roles) and a handful of others, a special w3wp.exe for IIS configuration and so forth. All of these were previously restricted from user access but can be accessed via the new admin privileges.

Many of the features set out here are in development and beta but are promised to the end user soon. Russinovich noted that the operations outlined here still could change significantly. At any rate, his PDC session provided a fascinating look into how a cloud can operate, and it’s approximately eleventy bajillion percent more than I (or anyone else, for that matter) know about how Amazon Web Services or Google App Engine works.

Glossary:

Azure : Microsoft’s cloud infrastructure platform

Fabric Controller: A set of modified virtual Windows Server 2008 images running across Azure that control provisioning and management

Fault Domain: A set of resources within an Azure data center that are considered non-fault tolerant and a discrete unit, like a single rack of servers. A Service by default splits virtual instances across at least two Fault Domains.

Role: Microsoft’s name for a specific configuration of Azure virtual machine. The terminology is from Hyper-V.

Service: Azure lets users run Services, which then run virtual machine instances in a few pre-configured types, like Web or Worker Roles. A Service is a batch of instances that are all governed by the Service parameters and policy.

Web Role: An instance pre-configured to run Microsoft’s Web server technology Internet Information Services (IIS)

Worker Role: An instance configured not to run IIS but instead to run applications developed and/or uploaded to the VM by the end user

VM Role: User-created, unsupported Windows Server 2008 virtual machine images that are uploaded by the user and controlled through the user portal. Unlike Web and Worker Roles, these are not updated and maintained automatically by Azure.

While Redwood is still vapor as far as the public is concerned (and the basic VMware cloud technology, vCloud is still in pre-release at ver. 09) – it’s clear that VMware thinks it can capitalize on its position as the default virtualization platform for the enterprise and swoop in to become the private cloud platform of choice as enterprises increasing retool their data centers to look, and work, more like services like Rackspace and Amazon Web Services.

Some people are grumpy about the term private cloud, saying it’s just a data center modernized and automated to the hilt – let’s get that out of the way by noting that “private cloud” is a lot easier to say than “highly automated and fully managed self-provisioning server infrastructure data center system with integrated billing”. It’s also less annoying than “Infrastructure 3.0″, a term that can make normally calm operators scream like enraged pterodactyls. Private cloud it is.

Project Redwood, now known as the VMware Service Director, will lay over a VMware vSphere installation and allow users governed self-service usage via a web portal and an API, effectively obscuring both the data center hardware and the virtualization software VMware customers are used to operating. The goal is to automate resource management so that admins don’t have to and make distributing computing resources as easy and flexible as possible, while maintaining full control.

According to the presentation, vCloud Service Director will support three modes of resource management: “Allocation pools“, where users are given a ‘container’ of resources and allowed to create and use VMs anyway they like up to the limits of the CPU and storage they paid for; “Reservation pools“, which give users a set of resources they can increase or decrease by themselves and “Pay-per-VM” for single-instance purchasing.

–From the article

That’s the IT side taken care of- the other really significant concept is vApps- users can build, save and move application stacks en suite, and will be able to flow out of their private cloud into VMware-approved public cloud services– vCloud Express hosters like BlueLock and Terremark. So admins get control and visibility, and users get true scalability and self-service. That means there’s something for everyone in the enterprise.

Other tidbits from the document-VMware’s concept of cloud:

Cloud Computing according to VMware
Lightweight entry/exit service acquisition model
Consumption based pricing
Accessible using standard internet protocols
Elastic
Improved economics due to shared infrastructure
Massively more efficient to manage

And how Redwood is the answer:

Project Redwood Strategy
High-Level: Enable broad deployment of
compute clouds by:
• Delivering a software solution enabling self-service
access to compute infrastructure
• Establishing the most compelling platform for
internal and external clouds
Approach
• Allow enterprises to create fully-functional internal
cloud infrastructure
• Create a broad ecosystem of cloud providers to
give enterprises choice
• Provide identical interfaces between internal and
external clouds to allow toolsets to operate
identically with either
• Enable developers on the cloud platform to create
new applications within a cloud framework

Of course, there are products that can already do this and already well on the way to maturity- Abiquo springs to mind. You can do everything Redwood is shooting for today, if you’re so inclined. A titillating report says an audience that reportedly contained VMware engineers cheered during an Abiquo demo. The problem is you have to bring your own hypervisor- few want their YAVS(Yet Another Vendor Syndrome)infection complicated.

Oracle, on the other hand, has reinvented itself as a “complete stack” of private cloud products, from the Sun iron on up, and IBM is happy to sell you iron that behaves like cloud, and so on.

But VMware is betting brand loyalty, severe antipathy towards non-commodity hardware and inertia will catapult it past the upstarts and comfortably ahead of Microsoft, its real competition here, which is shooting for the same goal with Hyper-V and the Dynamic Data Center but is at least a year behind VMware here.

Enterprises running clouds are inevitable, goes the thinking; virtualization is ideally suited to both cloud computing and the commoditized hardware market–provide the entire software stack needed to turn those servers and switches into compute clouds, and you’ll make out like a bandit, especially when the only serious competition to try and offer the same thing right now is Canonical on one extreme, and Oracle on the other.

If you are running an enterprise data center, want drop-in, one-stop cloud computing, and your options are “free–from hippies” or “bend over“, VMware, who already makes your preferred hypervisor, will be a favored alternative. All they have to do is execute.

“We are deeply troubled by the weakness of the technical and financial support behind this decision, and fear the state is potentially making a $300 million mistake,” Carlyle said in a letter to Governor Christine Gregoire published on Carlyle’s website. Co-written with Representative Hans Dunshee, the letter was first picked up by Pacific Northwest regional news site Crosscut.com

In a nutshell, the letter calls for a halt to a bond sale to fund the project and a review of existing cloud services, like “Google, Microsoft, Amazon or others as many companies and governments are doing today.” Further, it argues that the trend in outsourcing data and services is a fait accompli and a better use of taxpayer dollars.

Unfortunately, Carlyle’s letter sometimes reads like it was written by a jingo-happy IT vendor. To wit: “How best to efficiently and effectively move away from hardware-centric, expensive, proprietary, silos of data trapped in old databases to open, transparent, flexible, accessible, customer-oriented applications available via the Internet?” he asks.

(I think we’ve all snoozed through that PowerPoint talk, no?)

This is understandable. Carlyle comes fresh from the communications industry, where silos are not filled with grain and budgets are fine-tuned with an axe, as opposed to government, where silos are more than likely filled with grain and budgets are fed like foie gras geese.

Dunshee appears to be a more traditional politician; interestingly, he lists many unions as backers, groups likely to want state construction dollars.

It’s unclear why Carlyle and Dunshee believe the new IT infrastructure would go to waste. What’s notable, however, is that cloud is now commonplace enough that a politician will throw it out there and hold traditional IT up as the poorer model. That’s a long step in discourse from “cutting edge.”