Today’s guest post comes to us from Matt Gowarty.

Infoblox’s Cricket Liu, author of DNS and BIND, will speak to audiences across the globe on December 7th in a complementary, one-hour live session on how to increase network uptime and efficiency.

Cricket will cover a variety of topics related to IP Address Management for Microsoft environments including:

• Enterprise requirements for IP address management
• Sample architectures for introducing IP address management to your network
• A walkthrough of Infoblox’s approach to Microsoft IP address management

You will be able to watch the presentation live over the internet, or if you are near one of eleven select cities, you may attend a sponsored event party, where, in addition to the live broadcast, a breakfast, luncheon, or evening reception will be held.

Click the image below to learn more and to register.

Today’s guest post from Matt Gowarty.

Yesterday, I discussed Microsoft’s very public and damaging experience of “human error” in routine network change. But the real challenge for Microsoft and virtually every other organization is the sheer number and complexity of managing the number of changes and the configurations across the entire network. If every organization had unlimited staff, resources and time, it could be done manually, but as we all know, no organization has that luxury. So organizations must enhance their existing change processes with more automation and intelligence to reduce the risk of vulnerabilities which just hammered Microsoft.

There are several best practices to help organization manage change better and reduce the risk of human error. The best practices include:

• Have a change process and follow it
• Use a “trust but verify model”—you trust everyone follows the process, but verify by tracking every change on the network, both planned and unplanned
• Implement your best practices/standards/compliance policies but not just during

installation, use ongoing management to detect violations

• Proactively monitor change and configuration 24×7 to find problems and hard to find issues

As I talk to organizations across the world, virtually every IT organization has the first best practice implemented with a change process. But then the day to day use of the next three drops radically. Many organizations assume no changes occur outside of maintenance window or everything is documented perfectly, but in reality, we all know their our outliers all of the time. Again, virtually every organization has best practices or gold standards, but only think about them during the initial install. They are too busy to go out and look device by device to find any violations and “configuration creep” is bound to cause inconsistency. And finally, the proactive management has the lowest percentage today for more IT organizations because they are so busy doing everything else, they wait for a problem to occur and then go into the troubleshooting and firefighting mode.

The good news is these challenges can be addressed through automation, intelligence and control solutions. NetMRI is a leading solution for helping organization take more control and automate network configuration, change and compliance management across the entire network. Automation, control and intelligence can help with aspects such as:

• Limiting human error by automating changes
• Detecting planned and unplanned modifications by identifying every change
• Ensuring changes do not violate compliance standards or internal best practices
• Leveraging intelligence to find suboptimal configurations before end users are impacted or vulnerabilities exploited

While in an ideal world, it’s impossible to eliminate every unintended consequence that could damage an enterprise, the potential risk and exposure can be greatly reduced through managing change, configuration and compliance better. Organizations spend huge sums on redundancy and back up plans, but over and over again, the vast majority of issues are caused by change. Organizations should step up more and start to invest the time, people and resources to eliminate then number one cause of issues today.

Who knows, if the Microsoft staff saw an unplanned change right away or received an alert of a suboptimal configuration or identified a security best practice was violated, could this problem have been fixed well before a hacker found a way in? No one knows for sure, but a betting man would have just eliminated three potential risks in a matter of minutes.

Today’s guest post by Matt Gowarty.

Last week, Robert McMillan of IDG News wrote about a recent incident where Microsoft suffered critical issues when a human error during a change left Microsoft vulnerable to unwanted consequences. According to Microsoft, “We have completed our investigation and found that two misconfigured network hardware devices in a testing lab were compromised due to human error. Those devices have been removed.” The change(s) that caused the misconfigurations opened up vulnerabilities that led to spam being sent through the Microsoft equipment—quite embarrassing for the company who has been stepping up security the past several years with all of the negative publicity tied to hacks of the popular Internet Explorer Web browser.

The take-away, to this blogger, is pretty loud and clear: Companies—even those of Microsoft’s ilk—are incredibly vulnerable if they’re behind the times if they are relying on spreadsheets and change logs to manage change within the IT department. (Admittedly, not all companies will be the target of malicious attacks on the scale that Microsoft faces, either.)

With the “human error” comment in Microsoft’s statement, it appears this vulnerability was an unintended consequence of an authorized network administrator making a mistake. The statement does not say whether the modification occurred during an approved change process time window or was an unplanned changed made by an authorized user.

The simple truth: With the hundreds or thousands of changes occurring each and every month, it’s extremely difficult to keep up with all of the individual changes, and more importantly, manage the changes and ensure they follow best practices and stay within compliance mandates or gold standard templates.

A huge mistake made by organizations today is assuming a change process is all you need to ensure safe and correct changes throughout your organization. A normal change process would include steps like:

• A device is determined to need a change for any number of reasons
• A change request is submitted
• A person or panel reviews all requests and makes edits or accepts
• The planned change is placed in a ticketing system and assigned a time during an approved maintenance window
• The change is implemented and documented
• The ticket is closed

While the change management process is critical to reduce the risk, it is assuming three major aspects that have huge potential impacts for the organization and the network infrastructure. And we all know what our mothers taught us about assuming.

1. No change is ever made that doesn’t follow the process
2. The review person or panel has the expertise to catch every potential suboptimal configuration (such as how a change in one device could potentially impact a network neighbor along a service path)
3. The actual change will be implemented correctly—i.e. no human error such as “fat fingered” or incorrect copy and paste

In the above Microsoft example, the vulnerability could have been caused by any one of the risks above or any combination of two or more. This is assuming the issue was an inadvertent mistake. Now if the vulnerability was caused during a change but was intended to cause harm, all of the sudden the risk and challenge in finding it grows rapidly because there would be no documentation or the configuration changed on purpose and would be hidden.

After the fact forensics is never the ideal driver of instilling a change management process and policy. In our next post, we’ll detail the best-practices of change management—that need to be implemented before the breach.

Today’s guest post from Lori MacVittie.

Deploying a virtual network appliance is the easy part, it’s the operational management that’s hard.

The buzz and excitement over VMware’s announcement of its new products at VMworld was high and for a brief moment there was a return to  focusing on the network. You know, the large portion of the data center that provides connectivity and enables collaboration; the part that delivers applications to users (which really is the point of all architectures). Unfortunately the buzz reared up and overtook that focus with yet another round of double rainbow guy commentary regarding how cool and great it’s going to be when the network is virtualized and is “flexible” and “rapidly provisioned” and “cheap.”

Two of out three ain’t bad, I guess.

As noted by open-source management provider Xenoss in a forthcoming survey a lot of the folks (more than 70 percent) actually doing the work of managing a virtualized environment “prefer tools that manage their entire infrastructure as opposed to a virtualization-specific solution”. Interestingly the author of the aforementioned article echoes the belief that the “killer” application for cloud computing is tooling, i.e. management.

So let’s get our head out of the clouds for a minute and think about this realistically. There are, after all, two different sets of concerns regarding network and application network infrastructure in the data center and only one of them is addressed by the vision of a completely virtualized data center. The other requires a deeper management strategy and dynamic infrastructure components.

THE TWO FACES of CLOUD and DYNAMIC DATA CENTERS

There are two parts to a dynamic data center:

1. Deployment
2. Execution

What virtualization of the infrastructure makes easy is the tasks associated with number one: deployment. The “flexibility” touted by proponents of virtual network appliance-comprised architectures speak only to the deployment flexibility of traditional hardware-based network components. There’s almost no discussion of the flexibility of the network infrastructure component itself (which is just as if not more important to a dynamic data center) and of the way in which components (virtual or iron) will be integrated with the rest of the infrastructure.

No, no they don’t. And “integration” with a management or orchestration system for the purposes of provisioning and initial configuration is (again) only half (or less) of the picture. The use of something like VMware’s vCloud API will certainly get a virtual network appliance deployed (a.k.a. rapidly provisioned) and if you need to move it or launch more there’s no better way to integrate the operational procedures associated with those tasks.

But if you need to deploy a new web application firewall policy, that’s a way different story. The vCloud API is generalized, it’s not going to necessarily have the specific means by which you can deploy – and subsequently codify the appropriate application of that policy – on any given network component. And even if it did, at this point it’s going to be very general, as in the policy will still have to be specific to the component. Managing a network infrastructure component is not the same as managing a virtual machine. Moving around a VM is easy, moving around what’s in that VM is not. If you move across network partitions (VLANs, subnets, networks) you’re going to have to reconfigure the component. Period. It is the management of that aspect of a network infrastructure component – physical or virtual – that is problematic and which is not addressed by virtualization.

ADAPTATION at RUN-TIME

The second face of a dynamic data center is the adaptability and the capability to collaborate (share context) of the infrastructure itself. Without the ability of the infrastructure to essentially “reconfigure” itself during execution, what you end up with is the means to rapidly deploy and migrate a static infrastructure. If the behavior of each of the networking components deployed as virtual network appliances is codified in a rigid manner and does not automatically adapt based on the context of the environment and applications it is delivering, it is static. It is the adaptability of the infrastructure that makes it dynamic, not the way in which it is deployed.

Example: You deployed a Load balancer as a virtual network appliance. It can be migrated and even scaled out using virtual data center management technology. It can be deployed on-demand. It is now flexible. But while it’s running how do you add new resources to a pool? Remove resources from a pool? How do you ensure that users accessing applications in that environment doing so via a high-latency WAN are experiencing the best possible response time? How do you virtually patch a platform level vulnerability to prevent exploitation while the defect is addressed? How do you marry the very different delivery requirements for a mobile device with a LAN-attached desktop browser? How do you integrate it with the management platform so you can manage it and not just its virtual container?

More importantly, perhaps, to the operational (devops) folks is how do you adapt to the changing application environment? As applications are being launched and decommissioned, how does one instruct the infrastructure to modify its configuration to the “new” environment? Is this this adaptation, this automation, that provides the greatest value in a highly virtualized or cloud computing environment because this is where the rubber meets the road. The rapid provisioning of components requires the rapid adaption of supporting network and application network infrastructure as a means to eliminate the cost in dollars and time of manually adapting infrastructure configuration to the “real-time” configuration and needs of applications.

CLOUD is ABOUT APPS and OPS

Rich Miller summed it up well when he said “cloud is all about ops and apps.” It is not about any single technology. It’s a means to deliver and scale applications in a way that is efficient and more affordable than it’s ever been. But in order to achieve that efficiency and that reduction in costs the focus is necessarily on ops and the infrastructure they are tasked with deploying and subsequently managing. While virtualization certainly addresses many of the challenges associated with the former, it does almost nothing to ease the costs and effort required for the latter.

Consider the additional layer of networking abstraction introduced by virtualization. That has to be managed. For every IP address you add in the virtualization layer (that’s in addition to the IP addresses already used/required by the <insert network component here> the cost of managing every other IP address already in service also increases. The cost of IP address management is linear function of the number of IP addresses in use. And if you’re going to be managing that virtual machine via a management system, it’s going to have at least one IP address itself.

Increasing the cost of IP address management is exactly the opposite of what the new network and a dynamic infrastructure, a.k.a. infrastructure 2.0, is supposed to be producing. This is not solving the diseconomy of scale problem introduced by virtualization and cloud computing so often referenced by Greg Ness , it is the problem. Virtualization is making it easier to deploy and even scale applications and lowering CAPEX, but in doing so it is introducing additional complexity that can only be addressed by a solid, holistic management strategy – one that embraces integration across the entire infrastructure. That does not, by the way, yet exist. But they’re coming.

In a services based infrastructure, which is what a dynamic infrastructure strategy is trying to achieve, the “platform” is less important than the services (and how they are integrated) are provided. It is not virtualization that makes a network fluid instead of brittle, it is the services and the way in which they adapt to the environment to ensure availability, security, and a high-performing delivery system. Virtualization is a means to an end, it is not the end itself. It is not addressing the operational needs of a highly fluid and volatile environment.

Virtualization is not making it any easier to manage the actual components or behavior of the network, it’s just making it easier to deploy them.

Today’s guest post brought to us by Matt Gowarty.

Facebook going down for a few hours may have caused a spike in employee productivity.

But the culprit in this event shines a bright light on the biggest problem for IT teams worldwide—change. A Facebook blog post references “Today we made a change to the persistent copy of a configuration value that was interpreted as invalid” which caused the performance impact for over 2.5 hours.

When dealing with network change and configurations, organizations must be more proactive in the testing, validating and monitoring the ongoing status of both the critical network infrastructure (routers, switches, firewalls, etc.) as well as the Web services (applications, databases and servers). In fact, analyst reports show approximately 2/3rds of network performance issues are tied to change.

Too often, organizations spend the majority of time and resources focusing on the “outside attack” or employees with malicious intent to harm the organization. The Facebook example highlights by far the biggest issue for enterprises worldwide—a change made with the best intentions can often yield unintended and damaging consequences. For most IT organizations, simple changes are often the hardest to find and troubleshoot. Even with some of the most advanced network and application development capabilities and IT staff, the entire Facebook community was brought to its knees for more than two hours.

Tony Bradley from PCWorld highlighted the impact. “The Facebook outage was caused by implementing a configuration value on the live Web site without proper testing and validation. Had Facebook tested the new configuration value in a lab environment designed to mirror the real-world database cluster, it should have identified the problem with the new configuration value, and the error loop that caused this problem before allowing it to take the entire Facebook site offline.”

While this problem was extremely painful for Facebook and its users, it appears to have been an easy problem to detect because the change caused an immediate problem. The IT staff probably identified the culprit relatively quickly. The lengthy time and effort troubleshooting the issue was due to the extent of the issue. For many organizations, the potential change and configuration errors that lurk in the network for days, weeks or months are the issues that typically cause the biggest headaches for most IT shops.

These issues are the most complex because it takes another event, change or configuration issue to cause enough of a reaction for end-users to experience pain or monitoring tools to trip a threshold. The time and effort to find and solve these problems are exponentially longer because the initial culprit was hiding for days or weeks and it took another event to cause the pain to show up. It’s times like these that force IT and networking teams to spend days or weeks trying the siphon though all of the changes that may have occurred over the past days or weeks to trigger the failing event.

Using Facebook’s issue as the example, how long do you think it would have taken to find the problem if the change occurred last month and was a suboptimal setting that didn’t cause the problem right away? The odds are it would have been much longer than a few hours. I’d estimate it would have taken days or weeks to find the change related issue within the network infrastructure of a company the size of Facebook.

The silver lining—if Facebook should go down again in the next few months and you can’t update your status every few hours, you’ll have more time to fine tune your fantasy football team—oops, I mean work on that critical project for your boss.

Today’s guest post from Lori MacVittie.

Infrastructure 2.0 ? cloud computing ? IT as a Service. There is a difference between Infrastructure 2.0 and cloud. There is also a difference between cloud and IT as a Service. But they do go together, like a parfait. And everybody likes a parfait…

The introduction of the newest member of the cloud computing buzzword family is “IT as a Service.” It is understandably causing some confusion because, after all, isn’t that just another way to describe “private cloud”?  No, actually it isn’t. There’s a lot more to it than that, and it’s very applicable to both private and public models. Furthermore, equating “cloud computing” to “IT as a Service” does both a big a disservice as making synonyms of “Infrastructure 2.0” and “cloud computing.” These three [ concepts | models | technologies ] are highly intertwined and in some cases even interdependent, but they are not the same.

In the simplest explanation possible: infrastructure 2.0 enables cloud computing which enables IT as a service.

Now that we’ve got that out of the way, let’s dig in.

ENABLE DOES NOT MEAN EQUAL TO

One of the core issues seems to be the rush to equate “enable” with “equal”. There is a relationship between these three technological concepts but they are in no wise equivalent nor should be they be treated as such. Like SOA, the differences between them revolve primarily around the level of abstraction and the layers at which they operate. Not the layers of the OSI model or the technology stack, but the layers of a data center architecture.

Let’s start at the bottom, shall we?

INFRASTRUCTURE 2.0

At the very lowest layer of the architecture is Infrastructure 2.0. Infrastructure 2.0 is focused on enabling dynamism and collaboration across the network and application delivery network infrastructure. It is the way in which traditionally disconnected (from a communication and management point of view) data center foundational components are imbued with the ability to connect and collaborate. This is primarily accomplished via open, standards-based APIs that provide a granular set of operational functions that can be invoked from a variety of programmatic methods such as orchestration systems, custom applications, and via integration with traditional data center management solutions. Infrastructure 2.0 is about making the network smarter both from a management and a run-time (execution) point of view, but in the case of its relationship to cloud and IT as a Service the view is primarily focused on management.

Infrastructure 2.0 includes the service-enablement of everything from routers to switches, from load balancers to application acceleration, from firewalls to web application security components to server (physical and virtual) infrastructure. It is, distilled to its core essence, API-enabled components.

CLOUD COMPUTING

Cloud computing is the closest to SOA in that it is about enabling operational services in much the same way as SOA was about enabling business services. Cloud computing takes the infrastructure layer services and orchestrates them together to codify an operational process that provides a more efficient means by which compute, network, storage, and security resources can be provisioned and managed. This, like Infrastructure 2.0, is an enabling technology. Alone, these operational services are generally discrete and are packaged up specifically as the means to an end – on-demand provisioning of IT services.

Cloud computing is the service-enablement of operational services and also carries along the notion of an API. In the case of cloud computing, this API serves as a framework through which specific operations can be accomplished in a push-button like manner.

IT as a SERVICE

At the top of our technology pyramid, as it is likely obvious at this point we are building up to the “pinnacle” of IT by laying more aggressively focused layers atop one another, we have IT as a Service. IT as a Service, unlike cloud computing, is designed not only to be consumed by other IT-minded folks, but also by (allegedly) business folks. IT as a Service broadens the provisioning and management of resources and begins to include not only operational services but those services that are more, well, businessy, such as identity management and access to resources.

IT as a Service builds on the services provided by cloud computing, which is often called a “cloud framework” or a “cloud API” and provides the means by which resources can be provisioned and managed. Now that sounds an awful lot like “cloud computing” but the abstraction is a bit higher than what we expect with cloud. Even in a cloud computing API we are steal interacting more directly with operational and compute-type resources. We’re provisioning, primarily, infrastructure services but we are doing so at a much higher layer and in a way that makes it easy for both business and application developers and analysts to do so.

An example is probably in order at this point.

THE THREE LAYERS in the ARCHITECTURAL PARFAIT

Let us imagine a simple “application” which itself requires only one server and which must be available at all times.

That’s the “service” IT is going to provide to the business.

In order to accomplish this seemingly simple task, there’s a lot that actually has to go on under the hood, within the bowels of IT.

LAYER ONE

Consider, if you will, what fulfilling that request means. You need at least two servers and a Load balancer, you need a server and some storage, and you need – albeit unknown to the business user – firewall rules to ensure the application is only accessible to those whom you designate. So at the bottom layer of the stack (Infrastructure 2.0) you need a set of components that match these functions and they must be all be enabled with an API (or at a minimum by able to be automated via traditional scripting methods). Now the actual task of configuring a load balancer is not just a single API call. Ask RackSpace, or GoGrid, or Terremark, or any other cloud provider. It takes multiple steps to authenticate and configure – in the right order – that component. The same is true of many components at the infrastructure layer: the APIs are necessarily granular enough to provide the flexibility necessary to be combined in a way as to be customizable for each unique environment in which they may be deployed. So what you end up with is a set of infrastructure services that comprise the appropriate API calls for each component based on the specific operational policies in place.

LAYER TWO

At the next layer up you’re providing even more abstract frameworks. The “cloud API” at this layer may provide services such as “auto-scaling” that require a great deal of configuration and registration of components with other components. There’s automation and orchestration occurring at this layer of the IT Service Stack, as it were, that is much more complex but narrowly focused than at the previous infrastructure layer. It is at this layer that the services become more customized and able to provide business and customer specific options. It is also at this layer where things become more operationally focused, with the provisioning of “application resources” comprising perhaps the provisioning of both compute and storage resources. This layer also lays the foundation for metering and monitoring (cause you want to provide visibility, right?) which essentially overlays, i.e. makes a service of, multiple infrastructure resource monitoring services.

LAYER THREE

At the top layer is IT as a Service, and this is where systems become very abstracted and get turned into the IT King “A La Carte” Menu that is the ultimate goal according to everyone who’s anyone (and a few people who aren’t). This layer offers an interface to the cloud in such a way as to make self-service possible. It may not be Infrabook or even very pretty, but as long as it gets the job done cosmetics are just enhancing the value of what exists in the first place. IT as a Service is the culmination of all the work done at the previous layers to fine-tune services until they are at the point where they are consumable – in the sense that they are easy to understand and require no real technical understanding of what’s actually going on. After all, a business user or application developer doesn’t really need to know how the server and storage resources are provisioned, just in what sizes and how much it’s going to cost.

IT as a Service ultimately enables the end-user – whomever that may be – to easily “order” IT services to fulfill the application specific requirements associated with an application deployment. That means availability, scalability, security, monitoring, and performance.

A DYNAMIC DATA CENTER ARCHITECTURE

One of the first questions that should come to mind is: why does it matter? After all, one could cut out the “cloud computing” layer and go straight from infrastructure services to IT as a Service. While that’s technically true it eliminates one of the biggest benefits of a layered and highly abstracted architecture : agility. By presenting each layer to the layer above as services, we are effectively employing the principles of a service-oriented architecture and separating the implementation from the interface. This provides the ability to modify the implementation without impacting the interface, which means less down-time and very little – if any – modification in layers above the layer being modified. This translates into, at the lowest level, vender agnosticism and the ability to avoid vendor-lock in. If two components, say a Juniper switch and a Cisco switch, are enabled with the means by which they can be enabled as services, then it becomes possible to switch the two at the implementation layer without requiring the changes to trickle upward through the interface and into the higher layers of the architecture.

It’s polymorphism applied to an data center operation rather than a single object’s operations, to put it in developer’s terms. It’s SOA applied to a data center rather than an application, to put it in an architect’s terms.

It’s an architectural parfait and, as we all know, everybody loves a parfait, right?