After two straight years of wages remaining nearly flat, tech professionals on average garnered salary increases of more than two percent, boosting their average annual wage to $81,327 from $79,384 in 2010.

Virtualization as a skill also saw jumps in pay, up six percent to $86,669.

Within that, however, though VMware-related software skills saw a 4.4% increase compared with 2010 to 2011 salaries, up to $82,688, these salaries had taken a dip between 2009-2010 and 2010-2011, from $82,273 down to $79,199. Similarly, Xen skills increased 4.7% compared with 2010-2011’s $81,713, but had fallen compared with 2009-2010, when this skill paid $85,924.

Hyper-V skills, which pulled down an average salary of $81,701, were not tracked the previous year.

Meanwhile, pure virtualization appeared nowhere on the list of the skills with commanded salaries in the six-figure range. The highest paid among cloud and virtualization qualifications was Windows Azure, which raked in $102,510 – this was the first year salaries were tracked for this skill, so there’s no comparison with prior years.

And Azure was beaten out by salaries for other six-figure skills like Advanced Business Application Programming, which jumped 3% to $109,157, or Services Oriented Architecture (SOA), which jumped 6% to $108, 210.

The Dice Salary Survey was administered online with 18,325 employed technology professionals responding between September 19 and November 21, 2011.

RAID, or Random Array of Independent Disks, refers to the way data is striped and / or mirrored across disks to achieve redundancy. Higher RAID levels afford better data protection, but can eat up more disk capacity. Previously, the VSA required RAID 10, which offers high levels of data protection, but contributed to a 75% capacity overhead for the overall VSA.

Now, the VSA supports RAID 5 or RAID 6, which use fewer disks for data protection, resulting in more available capacity for users.

Under RAID 10, hosts in a three-node VSA cluster running the maximum eight disks at the maximum 2 terabytes (TB) capacity each would have a raw capacity of 16 TB, but a usable capacity of just 4 TB. Across the entire cluster, this means that 48 TB raw capacity would be needed to yield 12 TB of usable space.

VMware’s VSA documentation now contains a capacity calculator showing the new capacities available with the new RAID levels. Long story short, the same maximum configuration of eight 2 TB disks per host in a three node cluster now yields 21 TB in a RAID 5 configuration and 18 TB in a RAID 6 configuration.

High storage overhead was among the chief complaints about the VSA after its launch last August. The relaxed RAID requirements do not address other areas of criticism about the product, such as the fact that users must choose between a two and a three-node cluster before provisioning; the VSA cannot start as a two-node configuration and scale up to a three-node configuration.

“While this makes the VSA more functional by increasing the capacity of the storage, I still see it as a ‘niche’ player,” wrote Tim Antonowicz, senior architect at Salem, NH-based VAR Mosaic Technology, in an email. “There aren’t many scenarios where the VSA will be a better play than a dedicated SAN…as for our customers that were looking at deploying the VSA, we have been able to move them into a low-cost SAN solution for similar pricing and they have gone that route instead of using the VSA.”

The new program, called Virtual Customer Labs, will use the same underlying technologies as the Hands-on Labs at VMworld, according to a lunch session presented by Josh Liebster, a systems and sales engineer for VMware. It will allow users to test various VMware products “without having to worry about any Active Directory mishaps or storage being misconfigured.”

Technologies available on Virtual Customer Labs will include:

vCenter Operations Enterprise 1.0

• 2-3 hosts with 2-3 vApps
• custom application monitoring, dashboards, the ability to configure widgets and troubleshoot SmartAlerts

vCloud Director 1.5

• One new vCloud Director installation, vCenter Orchestrator, vCenter Chargeback and vCloud Connector

SRM 5.0

• 3 virtual data centers
• A-B site — array-based replication, demos of failover and reprotect functions
• A-0C site — vSphere Replication –demonstrates planned migrations

Users can only access the site through a service engineer request.

Auto Deploy can be used to deliver host and VM configuration information across the network, while an Auto Deploy server manages state information for each host. It stands to be most appealing in large environments where quick deployment of new VMs is a must.

In an environment where vCenter, the vCenter database and the Auto Deploy server are all virtualized in the same vCenter datacenter, and all hosts and VMs are totally reliant on Auto Deploy for their state information, Auto Deploy cannot set up vSphere Distributed Switches (vDS) if vCenter Server is unavailable. If the host can’t connect to vCenter Server, it remains in maintenance mode and virtual machines cannot start.

In other words, according to a recent blog post by Forbes Guthrie, a vExpert, and infrastructure architect specializing in virtualization,

…here’s the scenario. Everything powers off, all at once. You hit the power button on the servers. The hosts boot up, but stay in Maintenance Mode because they can’t hit the vCenter VM or Auto Deploy VM for their Host Profile. In Maintenance Mode the VMs won’t power on. The vDS switch cannot be created. You can’t power on your vCenter VM. You can’t power on your Auto Deploy VM.

There are a lot of “ifs” necessary to create this scenario. However, Guthrie wrote in a separate email, “Auto Deploy is best suited to large and rapidly changing environments, and it is just those sorts of progressive designs that are likely to virtualize their management servers such as a vCenter, a vCenter database and an Auto Deploy server.”

One way to design around the potential availability issue is to have a separate management cluster, which could be restarted prior to the rest of the environment to ensure the availability of vCenter. “Currently, most VMware customers who are considering Auto Deploy are likely to be large enough that they can absorb the overhead associated with an additional single-purpose cluster,” Guthrie wrote in the email. A separate management cluster running on dedicated hardware can also protect vCenter from performance issues caused by contention from other workloads.

The downside of running a separate management cluster comes in the form of increased costs, for separate hardware as well as ongoing management of multiple vSphere environments.

A remote secondary Auto Deploy instance, or a remote vCenter instance connected with vCenter Heartbeat are other potential design approaches that can mitigate a circular dependency scenario.

Ephemeral ports on the vDS can ensure vCenter always has connectivity to the network, Webster suggested. “[Ephemeral ports are] an option that you can use with a distributed switch when you’ve only got two 10-Gig NICs on the host, and that’s really where most of these problems potentially come in,” he said. “If you’ve got lots of NICs on your hosts, you’re probably going to have a few more vSwitches around, and you might still have your vCenter server connected on a standard switch port group, so you’re not going to run into the dependencies that exist with the VMware distributed switches.

“You’ve also got the option, of course, of using the Cisco Nexus 1000V, which doesn’t have any of these dependencies either,” Webster added.

VXLAN, or Virtual eXtensible LAN, submitted to IETF last fall and talked up at last year’s VMWorld, is a protocol for routing Layer 2 traffic over Layer 3 networks, with the goal of either expanding the available VLAN address space, or supporting inter-data center VM mobility, depending on who you ask. It’s also important to note that VXLAN, for now, is still a proposed standard that remains largely theoretical.

Recently, a blog post by Scott Lowe caught my interest by pointing out some key differences between VXLAN and Cisco’s Overlay Transport Virtualization (OTV) protocol, which also encapsulates layer 2 frames in layer 3 packets.

Under VXLAN, this encapsulation is performed at what are called VXLAN Tunnel End Points (VTEPs), which operate at the host level; under OTV, these endpoints lie in Nexus 7000 switches.

Lowe’s post illustrates some important hurdles VXLAN has to cross before it can perform inter-data-center VM migrations without a “traffic trombone” issue. Namely, vShield Edge, which acts as the default gateway in VXLAN environments, currently cannot be made redundant between sites.

VMware’s Duncan Epping appears in the comments on Lowe’s blog saying VMware is working on addressing this, but for now, this would mean that in theoretical VXLAN-land, traffic meant for a VM live-migrated to Site B would still have to pass through a vShield Edge device at Site A. An OTV deployment that follows Cisco’s recommended practices does not have this particular problem.

So, let’s assume that by the time VXLAN sees the light of day, the vShield Edge redundancy issue has been resolved, and VXLAN also has some equivalent to OTV’s use of HSRP and identical default gateway addresses at each location, so that traffic no longer has to ‘trombone’ between data centers when workloads are moved. Hasn’t it then just reinvented the OTV wheel?

Some might say that the advantage VXLAN has here vs. OTV is that VXLAN is on the standards track while OTV is not. But this is also where other experts, like blogger Denton Gentry, point out that tunneling VXLAN-style still comes with tradeoffs – namely, some trickiness when it comes to traversing firewalls.

the way VXLAN uses its UDP header can make firewall traversal a bit more challenging. The inner packet headers can hash to a well known UDP port number like 53, making it look like a DNS response, but a firewall attempting to inspect the contents of the frame will not find a valid DNS packet. It would be important to disable any deep packet inspection for packets traveling between VTEP endpoints. If VXLAN is used to extend an L2 network all the way across a WAN the firewall question becomes more interesting.

Some experts have also pointed out in previous discussions about VXLAN that creating tunnels with endpoints at the host level can also block visibility into the network for other kinds of third-party management and security tools which use packet scanning and analysis.

A way around some of these problems might be for VXLAN to be integrated into physical networking equipment, rather than encapsulating and decapsulating packets at host-based endpoints. But is integration into physical networking equipment even a likely scenario for VXLAN? At least right now, the fact that VXLAN doesn’t rely on physical networking equipment appears to be a significant difference between it and other protocols like OTV. Is this intentional? In other words, is VXLAN meant to be a kind of shortcut that bypasses the need for retooled physical networking equipment to create inter-data-center tunnels? If so, to what end? And how would visibility issues then be addressed?

After venturing down these various rabbit holes, you could just revert to characterizing VXLAN as an attempt to enlarge the VLAN address space – since it adds a 24-bit VXLAN Network Identifier (VNI) to each packet header, there can be 2^24, or 16 million, VXLAN segments, a substantial increase over the current VLAN limit of 4096 segments.

But to me, this would beg the question, Why all the talk about VM mobility with VXLAN in the first place?

As server virtualization becomes mainstream, vendors like VMware have to come up with a new return on investment (ROI) proposition to sell to customers, and experts say to be on the lookout for a new sales pitch around automation and operational savings.

“The purpose of automation is to create a new hard-dollar ROI for virtualization, basically opex savings by streamlining operations, as opposed to the old ROI which was capex savings that came from server consolidation,” said Bernd Harzog, analyst with The Virtualization Practice.

Consolidation ratios won’t be as high as mission-critical apps are virtualized, and so “in order for the virtualization freight train to keep rolling down the tracks, VMware needs a second, new incremental hard dollar ROI,” he said. “That’s going to come from this automation-driven opex savings delivered by vCenter Operations. And that’s going to shake up the management software industry in an extremely profound way.”

You can already see this in the aggressive stance taken by Quest subsidiary VKernel around the launch of its vOps 4.5 product, which features automation of certain tasks like the deletion of Zombie VMs.

“In the long term it will be a race between management suites, and with VKernel now part of Quest they certainly have a great shot at becoming one of the winning management suites,” Harzog said, if only because there are already a lot of vFoglight customers monitoring production VMware environments. .

But this begs the question – are IT pros ready to hand over the virtualization management reins to automation tools?

The answer seems to be a qualified yes – virtualization pros are beginning to explore automation tools, but in baby steps. “Interested? Absolutely. Ready for it? No,” said Maish Saidel-Keesing, a virtualization administrator for an Israeli technology company. “But we were also not ready once upon a time for using DRS to vMotion a VM from one place to another automatically. Show me one virtualization admin that does not use this feature today.”

“It will come,” said Saidel-Keesing. “But will take longer than 2012 for people to trust technology to remediate problems automatically.”

Virtualization management tasks like capacity and configuration management [AB1] will be the first beachhead for automation to establish itself within enterprises, users predict, while more critical tasks like provisioning will follow later.

“I’m not sure I’d be comfortable with [automatically] creating new environments at this point, just because everything we seem to deploy has a lot of very intricate configuration,” said Philip Sellers, systems administrator for South Carolina-based Horry Telephone Cooperative (HTC), “But from the management side, I see that as the opportunity today.”

In Sellers’ shop, automation has begun to creep in in the form of vCenter Update Manager and vCenter Orchestrator. In fact, so attached is HTC to VUM that it’s delaying an upgrade to vSphere 5 because VUM will no longer support automated guest patching, Sellers said.

“On the Orchestrator side of things, we’re very new in that,” he added. “We’ve only had it in our environment for a month, two months…it’s doing some very general tasks like cloning on a weekly basis.”

Still, “we’ve been playing with some of the automation stuff, just trying to get a handle on it, because we see that’s the general direction we’re going as an industry.”

“[Paul] Maritz from VMware has been discussing automation for the better part of a year,” said Alex Rosemblat, product marketing manager for VKernel. With vOps 4.5, Rosemblat claims, “we’re really starting to deliver on the future vision that VMware has laid out.”

An example of VKernel’s first foray into capacity management automation in vOps 4.5 is ‘Zombie VM’ deletion – the automatic removal of VMDK files deemed waste by the software. By default, vOps marks a VMDK as waste if it has no connection to anything in vCenter and has been sitting in storage for more than 90 days. Users can also customize these criteria.

“What users have had to do in the past, after they receive this list of waste files, is go through and manually find these waste files and delete them,” Rosemblat said. “If you have a couple hundred files, that can turn into several hours’ worth of work. We can do that basically with the click of a button.” In the next release, users will be able to schedule Zombie VM deletions.

Other new automation features introduced with 4.5 include snapshot auto-merge, and more “one-click issue remediation” similar to waste file deletion, but for memory limit sizing. Previously, vOps could address memory allocated to particular VMs, but not memory limits, another setting within vSphere which restricts the amount of allocated memory a VM actually uses. This release adds visibility and automated remediation for memory limits to existing support for memory allocations. vCPU sizing is also supported with this release, in addition to physical CPU resource sizing.

Also new with this release: a new way of doing reports designed to shave down the amount of time admins spend on this task. Now, an extended custom URL generation process allows admins to send management a link that shows a real-time view of the environment, rather than repeatedly generating and sending reports.
Finally, vOps 4.5 also includes the ability to forecast how much hardware will be needed to support an environment based on the current growth rate; common trend alarm warnings delivered by default; application type tags and resource sizing configuration groups; plus vSphere 5 and raw device mapping (RDM) support.

“VMware’s strategy for operations management is to combine performance management, capacity management, [and] configuration management…with the self-learning analytics that came with Integrien for the purpose of automating IT operations,” said Bernd Harzog, analyst with The Virtualization Practice. “Quest is coming out of the gate, in terms of a response [to VMware], better and faster than anybody else in the ecosystem.”

But it’s still too early to pick a winner in this race, Harzog said. “Right now we’re on chapter one of a twenty-chapter book.”

vCenter Chargeback has been renamed Chargeback Manager and folded into the vCenter Operations suite. According to the release notes issued Nov. 30, version 2.0 contains more than a dozen new features, including support for vSphere 5.0 and vCloud Director 1.5.

The new version is also more flexible, industry observers say, in that it allows for thick- and thin-provisioned VMs to be charged differently, supports raw device mapping, and can be configured to apply costs to virtual machines only when they are powered on. The new version also allows users to define storage tiers, include VM storage profiles in a tier, and automatically group datastores according to their storage profiles.

The new version also resolves nine issues from earlier releases, such as reports displaying usage and cost for shared virtual machines multiple times, and both scheduled and archived reports failing when users were deleted.

But the “Known Issues” section, which contains about 40 items, includes some real UI bugs, observers say. “A lot of the UI bugs look like the kind that would be very distracting to a user, cause load on the IT guys when the product malfunctioned, and make the product look and feel poorly constructed,” said one VMware pro.

“The first known issue, ‘Connection times out when browsing hierarchy,’ is a perfect example of this. The connection might time out when browsing a large vCenter setup. Aren’t places with large setups the people you want to sell this to? Why would they buy something that won’t work right?”

“It looks like VMware has a big opportunity here to improve this product,.” the expert added.

It’s unclear, however, just how many users will be affected by these issues, given that interest in the product has been low from the beginning. In general, virtualization chargeback tools face an uphill battle against existing corporate accounting practices.

Though some users say vCenter Chargeback looks good in theory, and can envision use cases, little appears to have changed on the adoption front as of late.

VMware was one of the investors that contributed to an $8.5 million round of funding for the IT systems automation software maker, bringing Puppet’s total funding to $15.75 million since its founding in 2005.

Puppet’s software, available as both an open source offering and in a commercial version called Puppet Enterprise, automates provisioning of VMware and Amazon EC2 instances in enterprise IT environments. IT shops have also used the open-source version of Puppet to increase server-to-admin ratios. Demand for experience with Puppet’s software has grown rapidly in the last year, according to a Wall Street Journal report.

VMware and Cisco “have hands-on, in-production-at-scale experience with Puppet – in some cases, going back several years,” wrote Puppet founder Luke Kanies in a blog post on the funding round.

The issue affected vSphere 5 virtual machines connected through software-based iSCSI initiators, and occurred, according to VMware’s Knowledge Base, because “ESXi 5.0 attempts to connect to all configured or known targets from all configured software iSCSI portals. If a connection fails, ESXi 5.0 retries the connection 9 times.”

According to the Knowledge Base article, “VMware is delivering an ISO file for this patch release due to the nature of this issue. This is not common practice and is only done in special circumstances.”

In some cases, this bug led to boot times of up to 90 minutes. Bill Hill, infrastructure IT lead for a Portland-based logistics company, said it took some of his servers that long to boot with the buggy version of ESXi 5.0.

This would be an open-and-shut case of a simple bug fix, Hill said, but it remains a mystery to him, as well as to some VMware insiders, why the original buggy ISO file remains available for download on VMware’s website as of today.

“It’s a frustrating situation,” said Hill. “Why leave a time bomb out there?”

VMware’s PR representatives did not have an official response as of this post.