An issue with some versions of EMC’s Engenuity Symmetrix VMax disk array microcode and VMware vSphere’s vStorage APIs for Array Integration (VAAI) has EMC warning users of potential hits to performance and data integrity.

An unspecified but relatively small number of users have experienced performance slowdowns as a result of using EMC’s VMax arrays at certain microcode levels with VAAI, according to a blog post written yesterday by Chad Sakac, EMC’s vice president for the VMware Technical Alliance.

“When using VAAI for Storage vMotion and Cloning operations, performance degradation can result due to the the operation(s) taking longer to complete than when using Software (Host) Data Copy,” says an EMC knowledgebase article referred to by Sakac in his blog.

This is the opposite of what the APIs are intended to do: offload processing from the virtual host to the storage array to improve performance. There is also the potential for VM data to be inconsistent after a reboot following Storage vMotion, according to the article.

These problems have “popped up in enough places…that everyone at VMware and EMC went into furious high gear on this over the last few days,” Sakac wrote. There is now a patch called an Engenuity Pack — Engenuity is the name EMC gives to the microcode on its Symmetrix arrays — to prevent these issues on the VMax side, but customers must also contact VMware support for a hot patch to resolve the performance degradation issue with Storage vMotion.

Right now, it seems these are not widespread problems. Checks with VAAI users of other storage products this week haven’t yielded any reports of problems, and market reaction to VAAI in general has been tepid so far. The issue is a bit of a head-scratcher, though, given reports that EMC was among the VMware design partners for VAAI prior to the release of vSphere 4.1 last year.

The introduction of VPlex Geo means it’s now technically possible to perform live migrations of VMs and their associated data across thousands of miles. A VPlex Global model is also still waiting in the wings, which EMC says will support cross-continental live migrations.

These products could at least theoretically change disaster recovery and automated high availability in virtual environments, if they take off. The ability to federate storage with transactionally consistent caching over distance – meaning data centers that are miles apart can be pooled and used as one resource, a step beyond replication-driven HA and failover between sites — also has potential uses for data mobility (“cloud bursting”) and multi-site collaboration.

Devil’s in the details

However, three factors hamper long-distance vMotion: its low tolerance for network latency; networking bandwidth requirements; and cost.

Scott Lowe, VMware-Cisco solutions principal for EMC, emphasized in Tweets on Tuesday that the existence of VPlex Geo means does not mean immediate availability of fully supported, production-ready vMotion over asynchronous distances, “Not yet.” Lowe said. “Need to watch vMotion [Round Trip Time] limits.” On Wednesday, he added, “see VMware’s support statements regarding requirements for long-distance vMotion.”

According to the document, “Virtual Machine Mobility with VMware VMotion and Cisco Data Center Interconnect Technologies,” the maximum latency between the two VMware vSphere servers cannot exceed 5 milliseconds (ms) in any case. This could change, of course, but has yet to do so. No official support statement for vMotion over asynchronous distances has been issued by either VMware or EMC.

Meanwhile, the VMware / Cisco support document for distance vMotion also includes the requirement that “the IP subnet on which the virtual machine resides must be accessible from both the source and destination VMware ESX servers.” This is also known as Layer 2 adjacency, and the current support document calls for an IP network with a minimum bandwidth of 622 Mbps between sites to accomodate it.  Such a configuration over distance also involves logically “stretching” the Layer 2 domain, through technologies such as Cisco’s Overlay Transport Virtualization (OTV).

If vendors build it, will users pay?

With time, the technical and support barriers to performing live migrations with local data access can easily be broken down. But that’s where Wikibon analyst David Floyer points out that high costs, especially for network bandwidth, will figure heavily into any business justification for the technology.

VPlex Local and Metro give users a healthy return on investment, Floyer pointed out. For example, VPlex Metro used with Oracle’s Real Application Clustering can allow effective stretched clustering well over the application’s native tolerance for 1 kilometer’s worth of latency, without requiring 622 Mbps of bandwidth between sites. But when it comes to mobility over longer distances, especially for high availability and disaster recovery, the cost of the bandwidth required to overcome the low tolerance for latency in vMotion today is out of the reach of most users, Floyer said.

Moreover, he asked: what is the use case? “Why would you do [live migration over distance]?” he asked. If it’s load-balancing between data centers more than 100 km apart, simply adding processing capacity where it’s needed would still be cheaper and less complex than migrating over big network links. Similarly, existing replication-based disaster recovery technologies will probably work well for the majority of enterprises, he said.

“It’s better to keep data where it is, and move the workload to it when you have to,” according to Floyer. Long-distance vMotion “is a lovely theory, but the cost of doing it is just not going to be practical.”

Update 2 pm ET 5-12-11: This post has been changed to correct information about stretched clustering that appeared in the original version.

At first glance, VMware’s Mozy acquisition seems like another one of its cloud pick-ups — see: SpringSource, Zimbra, etc. — that won’t do much for virtualization customers. But Mozy may bring some new capabilities to VMware’s core business after all. From VMware CTO Stephen Herrod’s announcement:

I’m also excited about some of the core data-handling technologies developed by the Mozy team. The Mozy future roadmap is going to excite consumers and businesses alike, and we also see the opportunity to leverage Mozy’s data compression, synchronization, client integration, and analytic tools to extend several existing and not-yet-announced VMware products.

On the cloud computing side, the Mozy acquisition will do a lot for VMware, if for nothing other than Mozy’s infrastructure; the service’s data centers store more than 70 petabytes, or 70 million gigabytes. And although Mozy is a popular consumer service, it does boast 70,000 business customers.

Because EMC owns VMware, you might think this move doesn’t make a ton of sense. The companies work very closely together these days, you might ask, so why does it matter who owns what? But they are clearly trying to position VMware as the cloud infrastructure and management company. Last year’s Ionix acquisition showed it, and this Mozy acquisition only reinforces this strategy.

VMware’s master plan is starting to bear fruit.

When VMware introduced Virtual Infrastructure 3 (VI3), the company started to allow third parties to introduce their own plug-ins to the client. Initially, what we saw were folks in the VMware community building their own plug-ins to extend the functionality of the VI3 Client.

Perhaps the most well known was Andrew Kutz’s plug-in that added a GUI front-end to Storage vMotion before VMware developed its own. VMware also got in on the act, using the plug-in architecture to add functionality to the base vCenter install with VMware Update Manager and VMware Site Recovery Manager.

Now VMware’s partners — especially the OEMs and the storage vendors — are getting in on the act. When VMware first unveiled this plug-in approach, it seemed clear to me that the company was trying to build a core management platform (vCenter), whilst allowing others to extend its functionality. It was my prediction that VMware wanted vCenter to become the main management tool from where most system administrators would do their work. If successful, vCenter might even usurp vendor-specific management tools — especially if you can do 90% of your daily administration tasks more efficiently with a plug-in.

Over the last couple of weeks I’ve been checking out the new raft of VMware vCenter plug-ins. I started off with the Dell Management Plug-in for vCenter (MPV), then worked with the EMC Virtual Storage Integrator (VSI) and NetApp Virtual Storage Console (VSC), and I will be soon doing a review of Dell EqualLogic’s Host Integration Tools for VMware.

Dell Management Plug-in for vCenter

The Dell Management Plug-in allows you to control the day-to-day management of Dell PowerEdge servers directly from vCenter. The plug-in adds a whole new series of Dell-focused alarms and alerts to the standard ones provided by VMware. And it enhances the standard monitoring of a physical host with views of the hardware, storage, warranty status and firmware.

Previously you might have used a special utility or management system from Dell to carry out firmware updates, but now these can be triggered directly from the vCenter plug-in. The Dell MPV also offers links to the Dell Remote Access Card (DRAC) interface and the OpenManage Server Administrator system, and it enables a way of deploying ESX hosts to brand-new hardware directly from inside vCenter.

Interestingly, rather than using PXE boot to transfer the install, the MPV communicates directly with DRAC on the Dell server. Even more useful are its abilities to configure the local RAID controller and join the installed ESX host to vCenter, with the option to apply a VMware host profile.

I think Dell’s approach gives a good steer on how all OEMs are going to focus on VMware as the primary management platform in the datacenter and integrate more closely with vCenter then they have ever dared to before. It’s recognition that VMware is the de facto hypervisor in the corporate space, and none of the vendors feel the need to swim against the prevailing tide.

VCenter plug-ins from storage vendors

On the storage side of the fence, all three of the big vendors — Dell, EMC and NetApp — have been busily developing new plug-ins, vying to be the one that most closely integrates with VMware. This competition will ultimately benefit all VMware admins, regardless of which storage vendor they happen to use. (I doubt very much whether the decision to go with one storage vendor over another rests so heavily on how well they integrate with VMware. Prices and existing relationships seem to have a greater an influence.)

I love what these storage vendors are doing. They are all offering very similar functionality with their plug-ins, and they are all capable of efficiently carving up storage from the disks that make up the array. Additionally, these storage plug-ins can also provision new virtual desktops and enhance the views of your storage within vCenter. They save you massive amounts of time and greatly simplify the process of providing new storage to your ESX hosts.

In fact, where once I prided myself on the ability to navigate around EMC Unisphere, NetApp System Builder and Dell EqualLogic Group Manager, I’m now seriously considering doing all main storage tasks via the vendors’ respective vCenter plug-ins. It’s so much quicker, and you’re less prone to make mistakes. (I don’t know about you, but really I’m more of server guy than a storage guy, so I don’t get to practice my storage skills as much as I should. So I will admit I’m the first to make errors in my administration at the storage layer. Anything that reduces my capacity to screw up on that front is heartily welcomed.)

VCenter plug-ins in the cloud

The other big deal about these storage plug-ins is their potential effect on cloud computing. Right now the “cloud administrator” has to work with the storage layer, slicing it up into various tiers and presenting it to the end user. This method is quite crude; it is very easy to provide more storage than is strictly necessary, and the creator-owner of the VMs in the cloud has little control over the placing of the virtual disks.

But imagine a situation where end users could create volumes/LUNs independently from a central storage pool, using vCenter plug-ins. If they create VMs for a project, they could create their own storage as they needed it. When they were done with their VMs, they would delete the VMs and their associated volumes/LUNs, handing that free space back to the storage pool.

Of course, this vision is rather grand. (The cloud has the tendency to make all those who consider it become starry-eyed.) But there are some obvious barriers. In the large corporate space, my customers who are VMware admins have said they love the idea of these storage plug-ins, but the sad reality at the moment is their storage teams wouldn’t let them use them; perhaps the storage teams fear that their importance to the whole of corporate IT would be diminished if VMware admins could provision their own storage.

Looking deeper into the issue, you recognize that that this self-provisioning model undermines years of careful, centralized storage management. The job of the storage guy is to make sure performance is great and that storage space is not wasted. If the storage team hands over the reins to the VMware admin, you could argue that not only would you have VM sprawl on your hands, but volume/LUN sprawl to boot. It seems we want two things that are in constant conflict with each other: We want self-service, demand-driven resource allocation, but we want to be control freaks, too. I’m wondering if there maybe more benefits if we learn to lose control.

Back in 2003, when the acquisition happened, VMware was only four years old, and server virtualization was still a fringe technology. Just check out this CNET story on EMC’s VMware acquisition: It describes VMware as “a start-up that sells software to make servers more flexible” and doesn’t even mention the word “virtualization” until the fifth paragraph.

Now virtualization is one of the hottest IT markets, and VMware is its leader, with nearly $3 billion in annual revenue and a yearly conference that drew 14,000 people. So why would EMC want to get rid of VMware? Or why would VMware want to get rid of its core virtualization business?

That’s what two observers have suggested in recent blog posts.

Simon Wardley, researcher at the Leading Edge Forum, predicts that the the division between VMware’s virtualization and platform businesses will grow wider and wider, and that the company will eventually sell off the virtualization part altogether. Why? Because VMware’s virtualization business is under attack by “open source systems such as KVM” and the growth of cloud computing.

(Wardley, a former executive at open source vendor Canonical, also writes that “a dominant position in the enterprise is no guarantee for future success” and cites Novell’s demise as an example. But everybody was talking about that when Hyper-V came out three years ago, and it hasn’t exactly proven to be true in the virtualization market.)

Sure, KVM has drawn some interest from the open source community, but VMware users aren’t ditching vSphere and flocking to Red Hat Enterprise Virtualization en masse. And virtualization is an underlying component of cloud infrastructures, among organizations that build private clouds and providers that host public clouds. Wardley says it’s a “misconceived” notion that virtualization is necessary for cloud computing, and while he’s technically right, the cloud market is by and large relying on virtualization.

Even if you agree with Wardley and think open source virtualization and non-virtualized cloud computing will take down VMware’s core business, there’s still the issue of timing. As GigaOm’s Derrick Harris wrote in response to Wardley’s blog post:

VMware itself acknowledges that platforms are the future — thus its heavy activity building out its SpringSource business — but the last time I asked someone from VMware what the timeline for such a transition is, I heard it would be about 10 years. … And with estimates that server virtualization isn’t really close to peaking in terms of adoption, there’s still a lot of money for VMware to make.

So let’s just say VMware is holding on to its virtualization business for the foreseeable future. There’s still the question of whether EMC will hold on to VMware. One person who sees the possibility of a sale is Software Advice consultant Christopher Baum, who has identified VMware as a potential Hewlett-Packard acquisition target. He writes:

In the grey area between glass rooms and cloud computing is virtualization. VMWare supplies software that makes a single physical server appear to be several virtual servers. Virtualization can increase security and ease management for individual applications or user classes. The firm would be a good candidate for HP this year.

Nothing Baum says is incorrect, but it’s definitely not complete. VMware would be a good candidate for lots of vendors to acquire — if it were for sale, which it’s not. Any server, storage or infrastructure vendor would love to own the most dominant company in the virtualization market.

I’m not saying things can’t eventually change. The Roman Empire fell, General Motors filed for bankruptcy and Foreigner went from selling out arenas to playing at VMworld. One day a time will come when virtualization isn’t a hot market and VMware isn’t the top dog. But until that time comes, EMC isn’t selling VMware, and VMware isn’t selling its virtualization business.

For example, yesterday, the coalition between VMware, Cisco and EMC (VCE) launched new Vblocks targeting VDI and SAP deployments, but VCE’s senior vice president of solutions Todd Pavone declined to comment on how many customers Vblocks have garnered since they officially began shipping last November.

“There is definitely momentum for standard architectures,” said Gartner Inc. analyst Chris Wolf. Some cloud service providers have found that enterprise customers are willing to pay a premium for a trusted architecture from vendors they know, he said. Stacks are also garnering interest among financial institutions that want to deploy trusted infrastructure quickly.

Still, while the turnkey concept may fly, there’s no guarantee these particular offerings from big vendors will burn up the market. Among the potential use cases for Vblocks for companies that can afford them is a quick-setup test / dev environment. But there are also startups like Kubisys, launched earlier this year, looking to offer turnkey test / dev appliances for about $80,000 MSRP – far lower than the million-dollar price tags on some Vblock bundles. The same is true in the VDI space, though Vblocks may have greater allure for MySAP users struggling with that resource-intensive application.

Since the early days of the VCE alliance, however, there has also been concern among some enterprises that preconfigured bundles will lock them in and constrain their choice of technologies and vendors. And reports from the sales field are lukewarm, indicating growing but moderate interest.

“We are seeing significant interest in VCE and Vblocks, but most of the time it ends up getting broken up in to individual parts (i.e.  an EMC array or a Cisco UCS), unless it is a greenfield opportunity, like a new data center,” wrote one systems integrator in New England. “We do expect to sell two real Vblocks this quarter, though.”

It’s not just VMware’s stack offerings having mixed success in the market, either (at least, from what information we can gather about them) – many of Oracle‘s users, for example, have flatly rejected the company’s attempts to get them to run Oracle apps only on Oracle VM and stacks of the company’s server and storage hardware acquired with Sun.

Yet big vendors, like HP, continue to rack up acquisitions in an effort to build these turnkey stacks, and continue to insist that customers are asking — nay, demanding — that they deliver them. If this is really true, and there’s a huge groundswell of IT managers begging for proprietary turnkey stacks, I haven’t caught sight of it myself yet — nor have I been given any specific revenue or market share numbers that reflect it.

Immediately, the talk was of how Dell partner (and VMware parent company) EMC would react. Dell’s acquisition of EqualLogic in late 2007 made things rocky between the companies behind the scenes, at least for a little while. The companies revised their OEM/reseller agreements, and Dell stopped referring prospects to EMC for high-end array sales. All along, EMC and Dell continued to insist that their relationship was as strong as ever. And they’re saying the same thing now, even though Dell has acquired a company that clearly competes with EMC.

But then again, as an analyst put it when we were discussing the deal yesterday, “there are ways to remain neutral publicly, but behind the scenes what can happen is very different.”

Some users, for example, already report having experienced contention between EMC and Dell following the EqualLogic deal, with VMware caught in the middle. One user in the public sector on the West Coast told me recently he stopped participating in his local VMUG after he said competitive strife between Dell / EqualLogic and EMC “made life difficult”.  The group had had a big Dell / EqualLogic membership, and wanted to bring in reps from those companies to speak, but the user said EMC was “moaning about bringing in someone seen as as a competitor — it was all very passive-aggressive.”

By now, Dell / EqualLogic seems fully entrenched in the VMware fold as the original shock of the EqualLogic deal has worn off. Dell / EqualLogic was one of the ‘design partners’ VMware said it sent a vStorage API SDK prior to other players in the storage market. But despite the fact that the SDK reportedly contained a ‘fourth primitive’ for integration of thin provisioning, and 3PAR was among the first in the storage market to popularize the feature, 3PAR was not a part of that early-access group. This may ultimately make little difference to end users, as most storage vendors (including 3PAR) by now have received the code. But early access for ‘design partners’ has given rise to the perception in some corners of the industry that VMware is capable of playing favorites in subtle ways.

Meanwhile, some industry-watchers have predicted that the 3PAR deal will make the rift between Dell and EMC widen even further, until there is a public split in addition to underground animosity. “Slowly but surely, Dell will wean its way off the EMC storage machine,” wrote Gary Orenstein at GigaOm. ”Not immediately, but eventually.”

Of course, similar predictions were made after the EqualLogic buy, and it has yet to come to pass. But it’s also increasingly clear that the “Balkanization” of IT vendors is continuing, even accelerating, and that server virtualization vendors have a fine line to walk when it comes to navigating these choppy waters.

Take, for example, the announcement by HP of a $250 million investment in technology development with Microsoft, just a few weeks after VMware CEO Paul Maritz participated in a cozy webcast announcing a new joint venture with EMC CEO Joe Tucci and Cisco CEO John Chambers.  HP denied it, but its subsequent alignment with Microsoft was widely seen as a direct response to VCE. Another big player, HDS, has also aligned itself with Microsoft, even OEMing Microsoft’s System Center Operations Manager (SCOM) as the basis for a forthcoming cloud computing “stack” offering.

Meanwhile, Oracle has already shown it expects users to run applications on only Oracle’s own Xen-based virtual machine (Oracle’s suggestion for users who don’t want to run more than one server virtualization environment to manage different applications? Move everything to Oracle VM). What’s to stop similar lines in the sand from being drawn around VMware and Hyper-V, as the infrastructure players open new competitive battlefronts as they build their ‘stacks’, and begin stepping on toes they haven’t stepped on before?

Nothing is for certain, but the issue warranting attention from virtual server users as these deals unfold is is the question of whether the zeal for soup-to-nuts “stacks” among large IT vendors — most of which are marketed as a stepping stone to the private cloud – will have an effect on their choices about which hypervisor to use with their preferred underlying hardware. It’s not that EMC will cease supporting Hyper-V if users require it, or that HP will kick out VMware completely. But I wouldn’t be surprised if infrastructure vendors are already beginning to steer prospective buyers toward the advantages of their special integrations with one hypervisor or the other, and it’s easy to imagine the subtle ways this could ‘make life difficult’ for virtualization users caught between shifting alliances.

Now, Oracle is banking on an appliance to do the same for its lagging virtualization market share.

Our sister site SearchITChannel.com reports that a so-called “Oracle VM machine” (perhaps developed by Oracle’s Department of Redundancy Department?) is in the works. Oracle President Charles Phillips disclosed the news during the company’s quarterly earnings call last week.

The Oracle VM machine will bundle the Oracle VM hypervisor with Oracle VM Manager software on a server with integrated network switches and storage arrays. Phillips didn’t offer many details (or a timeline), but it’s basically the same approach that Cisco has taken with UCS and rival Hewlett-Packard is now taking with its converged infrastructure push.

At best, converged infrastructure is only a good option for certain organizations — usually those that are totally new to virtualization or don’t have large virtual infrastructures already in place. In this respect, Oracle is making a smart move: In theory, it will be easier to gain virtualization market share by going after greenfield opportunities than by trying to convert VMware shops.

But a lot of potential customers are wary of these kinds of appliances because they fear vendor lock-in. Sure, Cisco relies on VMware virtualization and EMC storage for the UCS, and HP has agreements in place with both VMware and Microsoft, but you still limit your options by going this route.

And the Oracle VM machine will presumably be even worse, because Oracle doesn’t need to rely on any other vendor’s equipment. The company already has the virtualization and management software, and all the hardware is there too, thanks to the Sun acquisition.

Between Oracle’s very late push into the virtualization market and the overall lukewarm reception to these kinds of appliances, the Oracle VM machine better offer one heck of a kick-start if it’s going to change the company’s fortunes.

At this week’s EMC World, the annual conference from VMware’s parent company, the theme was “Journey to the Private Cloud.” It looks like VMworld 2010 will be even more focused on the cloud. Of the eight session tracks that the show will offer, four will be about cloud computing: hybrid and public cloud, private cloud management, private cloud business continuity and private cloud security. (By contrast, only two tracks even have the word “virtualization” in their titles: desktop virtualization and virtualization 101.)

Clearly, VMworld 2010 will mark the show’s transformation from a virtualization conference to a cloud conference.

It doesn’t mean that there won’t be lots of virtualization talk at the show. A quick glance at the presentations up for consideration in the cloud tracks shows plenty of familiar topics: “10 Best Free Tools for vSphere Management,” “Virtualizing Mission-Critical Applications,” “How to Reduce Your Desktop Attack Surface” and “Creating Your Virtual Lab in a Few Easy Steps,” just to name a handful.

In past years, these presentations and other like them would have stood on their own virtualization merits. But they’ll likely have more cloud context this year. How can free vSphere management tools turn your data center into a private cloud? What extra steps do you have to take to virtualize mission-critical apps in a private cloud? Things like that.

Cloud computing and private clouds in particular are all the rage these days, so this approach makes sense for VMware. Does it make sense for potential VMworld attendees? Probably most, but there are some who don’t like this shift. Consultant Tom Howarth, who runs the VMPlanet blog, wrote on Twitter this morning: “Seems to me that VMworld should be renamed Cloudworld – way too much focus on that for my liking.”

What do you think about all the clouds that are on the VMworld 2010 horizon? Let me know in the comments or reply to @ServerVirt_TT on Twitter.

Even though EMC has owned VMware since 2003, the two companies haven’t always been so buddy-buddy. Yesterday, an EMC executive said what many have suspected for a while: that EMC and VMware get along so well now because VMware co-founder Diane Greene isn’t in the picture anymore.

“It’s completely due to the change in leadership,” EMC senior vice president Dennis Hoffman told me in an interview.

EMC CEO (and VMware chairman) Joe Tucci fired Greene as VMware CEO in July 2008, citing her lack of “operational experience.” We learned a few months later that, according to the New York Times, it was an ugly scene: Tucci fired Greene in front of her husband, VMware co-founder Mendel Rosenblum, then turned and offered Greene’s seat on the board to Rosenblum (who declined).

Hoffman works with VMware a lot these days, as he’s EMC’s lead representative in the Virtual Computing Environment coalition with VMware and Cisco. He didn’t mention Greene — or her successor, Paul Maritz — by name during the interview. But he told me that before said leadership change, VMware had an “anti-EMC” attitude.

“There were folks around EMC that felt it was a very disadvantageous situation,” he said.

Maritz, a former longtime Microsoft exec, was running EMC’s cloud computing business before Tucci tapped him to replace Greene. Since then, “we’re getting closer, and we all certainly feel that,” Hoffman said. “It’s easy to be close when you share the same vision.”