VMware

Storage vendors put together ESX iSCSI cookbook

Just came across a pretty interesting resource on EMC’er Chad Sakac’s Virtual Geek blog (first brought to my attention by Stephen Foskett). It’s a guide to ESX and iSCSI co-developed by, among others, Andy Banta of VMware, Vaughn Stewart of NetApp, Eric Schott of Dell/EqualLogic, Adam Carter of HP/Lefthand, and David Black of EMC.

The post gets into nitty-gritty details and even includes what look like scanned-in napkin drawings to illustrate some of the complexities of performance management using ESX 3.x server with iSCSI. There are multiple links to futher resources on everything from the fundamentals of link aggregation to the full iSCSI spec.

But the bottom line for storage users is that “the ESX 3.x software initiator only supports a single iSCSI session with a single TCP connection for each iSCSI target…So, no matter what MPIO setup you have in ESX, it doesn’t matter how many paths show up in the storage multipathing GUI for multipathing to a single iSCSI Target, because there’s only one iSCSI initiator port.”

There are ways around it–in short, the post states, “Use the ESX iSCSI software initiator. Use multiple iSCSI targets. Use MPIO at the ESX layer. Add Ethernet links and iSCSI targets to increase overall throughput. Ser your expectation for no more than ~160MBps for a single iSCSI target.”

There’s also a workaround for single LUNs needing more than 160 MBps, using an iSCSI initiator in the guest along with MPIO, though the post acknowledges, “It has a big downside…you need to manually configure the storage inside each guest, which doesn’t scale particularly well from a configuration standpoint – so for most customers [say] they stick with the ‘keep it simple’ method.”

The best news out of this post for VMware and iSCSI users, though, is probably the pre-announcement that this behavior will be changing in future ESX releases.

ESX and SATA: So happy together?

I’m a big fan of SAS. I’ve professed my undying love and devotion to it (at least until solid-state disk becomes just a little more affordable). So why on earth would I be writing about putting VMware’s ESX Server on SATA disks?

I was poking around on the Internet a few weeks back and came across a deal I couldn’t possibly refuse: a refurbished dual Opteron server with 4 GB of RAM and four hard drive bays (one with a 400 GB drive) with caddies and a decent warranty for $229. No, that’s not a typo!

The downside is that the server is all SATA, and ESX won’t install on some generic SATA controllers attached to motherboards, or even some add-in SATA cards, without some serious cajoling and at least hunting around on VMware’s support forum. These hacks are not officially (or even unofficially) supported. In fact, VMFS is explicitly NOT supported on local SATA disk(scroll to the bottom of page 22). There are exceptions to this and there is work on a SATA FAQ where you can get more info.

When my server arrived, I installed an Areca SATA RAID card in it, downloaded the beta Areca driver for ESX 3.5 update 1 and went about the install. Areca is my favorite SATA/SAS add-in card manufacturer. Their cards are stable and have proven to be the fastest cards in their class for the money.

Why not use LSI, you may ask, especially considering that their driver is in the ESX kernel? Well, I like speed, and I usually don’t make compromises when it comes to the performance of a storage subsystem unless it is absolutely, positively required due to some other dependency. (As a side note, I have LSI  installed in my Vista x64 desktop due to a lack of x64 driver availability for my Areca card at the time of install. See my Windows on SAS blog post for more info.)

The Areca driver install went smoothly, and I currently have Exchange 2007, a Linux email server, a Windows 2003 domain controller, a Windows 2008 domain controller, a Windows XP desktop, and an OpenSolaris installation on the 1.2 TB of local SATA storage. So far, things look strong and stable with the driver in place. Normally, I try to stay away from needing a driver disk at install time to get to storage, as it makes recovery a nightmare if you are forced to use a live CD to repair your system. But this process was so stable that I may need to add exceptions to my prejudices. . . .

I’ve had acceptable performance (according to my subjective non-scientific benchmarking) with the relatively small number of VMs I have deployed on this server, so all in all, the storage subsystem is performing admirably.

This leads me to the question: Is SATA in ESX’s future? I wonder, seeing that Xen has native support for a myriad of storage subsystems, and Windows Hyper-V will install on SATA. I can’t see VMware being able to hold the line of local VMFS on SCSI/SAS only for much longer.

Couple this with storage Vmotion and it leads to more questions about the storage subsystems used for VMware and how it affects VMware’s already strained relationship with storage vendors. On the other side of that coin, why isn’t Microsoft or Citrix taking any heat for allowing the use of onboard SATA attached to cheap integrated controllers? Have the generic onboard storage controllers (Marvell and Silicon Image come to mind) reached the point where they are deemed capable of handling heavy server I/O loads? I’m sure they’ve advanced but I’m not convinced they can handle the storage I/O loads that a server with 20 VMs would generate.

But I have a bigger dilemma than all of this. How would I free the terabytes of storage I’d have trapped in these servers if I used eight 1 TB SATA drives? ESX isn’t exactly the platform of choice for a roll-your-own NAS. While it is technically possible for one to install an NFS server daemon on an ESX host, doing so may not be such a good idea. I’ve seen posts on how to compile and install the NFS module on an ESX host, but I haven’t seen any hard documentation as to the overall performance of the system when serving up VMFS via NFS and host/guest VMs simultaneously. You could also create a guest and turn that guest into an iSCSI or NFS target and point your ESX server to it, but that is also a bit kludgy.

In other words, just because I can run ESX on SATA disks local to the host doesn’t mean I should. I haven’t found a compelling business reason to have local VMFS on SATA in these times of inexpensive NAS. However, the pleasant side effect of this dilemma is that until I come up with a business case for using SATA as primary storage for an ESX server, I have a super cheap host that I can test various tweaks and changes to ESX before I roll them up into a QA or production environment.  Maybe that’s reason enough to have one or two ESX deployments on SATA.

Symantec question marks: email archiving SaaS and VMware competition

There are some potential inferences that can be made from two moves Symantec Corp. made last week: the acquisition of MessageLabs and the launch of Veritas Cluster Server One. However, for now clear answers as to whether or not those inferences are correct are not forthcoming.

MessageLabs is partnered with Fortiva to offer email archiving SaaS, so I wondered if the acquisition might mean that Symantec will get into that kind of offering as well.  SearchSecurity.com reported that MessageLabs CEO Adrian Chamberlain will be heading up a new SaaS group at Symantec, though Symantec officials also told SearchSecurity they won’t be SaaS-enabling all products. This remains an open question for now, as a Symantec spokesperson told me that product roadmaps will be decided after the acquisition closes, which might not happen until year end.

Symantec also launched VCS One, with the goal of allowing organizations to keep active farms of virtual servers running at a disaster recovery site, as well as recover tiered applications with dependencies intact.  ”Right now, this process is dependent on a lot of tribal knowledge in the heads of individuals who know the right order and design scripts to run this kind of recovery,”  said Mark Lohmeyer, vice president and general manager of the VCS product group at Symantec.

If any of that sounds familiar, it might be beacause this past May, VMware and its storage partners launched VMware Site Recovery Manager,  allowing VMware’s VirtualCenter to execute commands against storage arrays at primary and secondary sites during recoveries and enable VirtualCenter-generated metadata about virtual machines to be replicated, along with system and application data. In part, SRM is designed to help server virtualization customers automate their disaster recovery checklists, which many of them keep on paper and check off manually.

Meanwhile, Symantec has been among the most outspoken of storage vendors about friction with the server virtualization giant, and at Vision this year took VMware rival Citrix XenServer under its wing and into its product line, claiming the resultant Veritas Virtual Infrastructure product will be a better approach than VMware’s Virtual Machine File System (VMFS)  for server virtualization in large environments.

However, Symantec positions VCS One as complementary to SRM, rather than competitive with it. A Symantec spokesperson emailed me the following statement when I asked about it late last week:

VCS One is a complementary solution for VMware environments that can help improve overall availability of the environment in production, for mission-critical apps, by taking an application-centric approach to HA/DR.  And, we work closely with VMware to integrate with, and leverage VMware technologies such as Vmotion (for reducing planned downtime) and DRS today, and we’re looking at how we can also integrate with SRM in the future.  Finally, our solution is ideal for heterogeneous physical and virtual environments, that includes VMware as well as other platforms (which is the case in virtually every data center).

It’s important to note, though, that VCS One only supports VMware virtual machines at present, which might make the kind of competitive statements made earlier this year a bit awkward at this stage. Lohmeyer says VCS One was under development before Xen came on the scene. “[Support for Xen] will be in our very next release,” he said. Once that happens, I wonder if Symantec’s messaging might change somewhat.

IBM virtual desktop storage update - sort of

Last week I wrote about some confusion I had regarding IBM’s virtual storage optimizer (VSO) for VMware Desktop Infrastructure (VDI), especially after I was told by a VMware official that the IBM product, credited to an internally-developed algorithm, was based on VMware’s Linked Clone API.

I wrote to one of the researchers involved and got a response through IBM’s PR spokesperson that:

• The IBM-developed algorithm is based on VMware API available in Virtual Infrastructure version 3, not the VMWare LinkedClone API. Specifically, the algorithm uses VMware Infrastructure SDK 2.5.0 as documented at https://secure.techtarget.com/exchweb/bin/redir.asp?URL=http://www.vmware.com/support/developer/vc-sdk/ and file system level access on ESX servers.
• We developed the algorithm based on the API that was publicly available and supported at the time that we began development efforts
• VMware can provide detail on the differences between the APIs in Virtual Infrastructure version 3 and VMware LinkedClone API

So far no response from VMware.

Regardless of what API was or was not used, what I am trying to get at is the functional difference between these two products, if any. If there is one, it’s important for users to know about. If there isn’t one, it speaks to the growing convergence between VMware’s virtual infrastructure and storage vendors’ value-add software.

the bottom line right now seems to be that IBM’s product is for existing IBM customers, since it requires professional services through IGS. There are some shops that need the IBM label before they buy, and so VSO could at least be a fit for them.

Appreciate weigh-ins from IBM, VDI, and / or VMware experts.

Much ado about virtual desktop data storage

Last week at VMWorld, IBM announced the Virtual Storage Optimizer (VSO), ESX-based software that reduces virtual desktop storage by creating a “golden image” of the desktop’s operating system and other static files, while saving the changes users might make to that image.  It’s a concept akin to NetApp’s space-efficient snapshots, but because it’s delivered in software at the ESX level, IBM said, it can be applied to any storage system.

The next day, at a keynote, VMware officials demonstrated a new concept they’re rolling out in the next version of VMware Infrastructure called LinkedClones. LinkedClones create a “golden image” of virtual desktop files, as well as incremental changes;. The golden images, VMware demonstrated, can be updated with patches that automatically proliferate to all virtual machines based on the image to simplify rollouts and updates.  

From the briefing I’d had with IBM the day before and this keynote demonstration, these products seemed similar. Since the VMware demo last Wednesday, I’ve been trying to assess what might be different about them. VMware officials I spoke with Wednesday and Thursday said they didn’t know enough about IBM’s product to comment on its differentiation and IBM spokespeople were unavailable.  IBM’s press release about VSO had stated that it was “based on an algorithm developed by IBM Research,” but VMware said LinkedClones  wasn’t based on anything from IBM.

Today I spoke with VMware director of enterprise desktop Jerry Chen, who told IBM’s VSO is based on the LinkedClones API. I asked about the algorithm developed by IBM. “There are things our partners can do to further optimize LinkedClone,” Chen said. “For example, there are different settings for the number of LinkedClones each master virtual machine can copy.”

Chen said he wasn’t familiar enough with the IBM product to say what VSO adds on top of LinkedClones.  Meanwhile, IBM has been coy on this one. Since last week, I’ve put in numerous requests for comment by phone and email, including a fresh round of requests today after speaking with Chen. So far, no comments have been forthcoming.

Vmworld: Hope Springs Eternal!

I’ve been on an emotional roller coaster recently. I’ve had a slim chance of making it out to Vmworld in Las Vegas this week and I got really excited. Then … crushing blow I couldn’t go, then the sky’s parted birds chirped and a harpist showed up from out of nowhere I could go again, then disaster … alas it wasn’t meant to be this year. I won’t be able to make the trek out to the show for the product I’m sorely waiting to be released in my toaster so I can ramp up total resource usage making my breakfast (the harpist gave me a dirty look before she packed up and left)(the harpist gave me a dirty look before she packed up and left).

So instead, I decided to write “what I want from Vmworld” here.

1) Non-Windows support for the Virtual Center/Infrastructure stack

Really,  why does it HAVE to be run in Windows? MySQL and friends run on just about everything so what about the server stack is so tied to the Windows code base that couldn’t be run in some other OS or even their OWN OS ala ESX. I’ve been running into more and more folks on my client list that don’t want to manage Windows in order to manage their virtual infrastructure. I’m looking forward to them announcing an alternative to running on Windows.

2) Windows 2008 support

For the folks who are running a Windows centric shop Windows  2008 is a reality, I have a client who runs it now exclusively and if an app can’t be qualified on 2008 it can’t run in their shop, period, no exceptions. They use 2008, they love it and they aren’t looking back. Funny thing, even though 2008 has been available to the public since early this year Virtual Infrastructure stack does NOT run in 2008 without unsupported shoehorning. PLEASE PLEASE PLEASE release (not announce, but release) Windows 2008 support at Vmworld, the harpist is counting on you!

3) A new license server mode/technology

I’m not sure if it has any fans, but I can tell you one of my biggest and in my opinion the most glaring issues with the entire VI (Virtual Infrastructure) stack is the licensing server technology. FlexLM can’t be clustered, it can be made HA but only with knowledge of how to make Windows 2k3 HA. Now Vmware isn’t the only one to use this technology, Citrix does as well, take a browse through Citrix support forums to see how many friends Citrix made when they started using it. Vmware had a similar number of friends, me included. The technology stinks, ditch it, please! Announce a new alternative that can be clustered and is OS agnostic, an application server based license model comes to mind like Tomcat etc.

4) Updated time keeping

The current ESX server technology is pretty good at dealing with time drift of virtual machines on the same host, but across multiple hosts there is some work to be done. Yes, one can use NTP but when things like time sensitive audit data can’t stand even a second drift NTP becomes an unworkable solution. Vmware, you listening? Help us out, let the hosts synch themselves to each other so vm’s on separate hosts have precisely the same time.

5) Physical and Virtual conversion

I’ll kinda give Vvmware a pass on this one because there are apps from companies like Platespin andVizioncore, but … physical to virtual and back again is a weak spot. If they announce better conversion tools, or hey a takeover of one of those companies I’d be a happier admin.

6) Capacity planning

There are services surrounding the Vwmare Capacity Planner that third party vendors offer similar to IBM’s CDAT study, I understand the ecosystem it feeds, but I think Vmware would be better served if the full suite of measurement tools and methodology available to the consultants conducting a capacity exercise were available to the broader public. I’d be willing to bet that the mjaority of people will still make use of experienced third parties to conduct the exercises, however those who can’t or have shops so small they are not on the radar of service providers would be able to take advantage of a great resource. It would really be nice to have the opportunity to even do ongoing internal audits using these tools and methodologies. Make me wish I could make it, and the harpists dirty look even that much more meaningful, announce opening all capacity tools up to the public.

EMC enters no-spin zone on VMware CEO’s departure

A day later after the fact, is has become clear that VMware didn’t dump CEO Diane Greene because of personal differences between her and EMC executives. The final straw was a business dispute with EMC brass.

Sources close to EMC say the issue that cost Greene her job was she wanted EMC to spin off VMware into an independent company while EMC CEO Joe Tucci and his team want to keep the majority stake EMC holds. EMC spun out around 15% of VMware’s shares last August in an IPO that raised $1.1 billion.

“Diane wanted a spinout of VMware, and EMC said no we won’t spin out VMware at this time because it is our golden goose,” said a financial analyst who covers EMC.

A full spinoff of VMware has been an issue ever since the first IPO. Analysts have asked Tucci on EMC’s last two earnings calls if a spinoff was in the works for next year (a spinoff would be tax-free if EMC waits until 2009). Tucci gave almost the same answer both times: He said EMC had no plans to spin off VMware “right now” because both companies were performing well, and that creates shareholder value. He added both times that management and the board are focused on creating maximum shareholder value for both sets of shareholders.

Try telling that to investors who have watched VMware’s stock price drop from $125.25 to $40.19 in eight months while EMC shares fell 11.6% Tuesday to a 52-week low of $13.39. Now some analysts are wondering if investors will demand a spinoff.

EMC shed little light on the situation since yesterday morning’s news release revealing that Greene would be replaced by former Microsoft and EMC exec Paul Maritz. Neither EMC nor VMware held conference calls to discuss the move, although Tucci and Maritz did speak to selected media outlets. Tucci told Reuters and the Wall Street Journal that Greene lacked experience to run such a large company as VMware. Maritz told BusinessWeek he’s not underestimating VMware competitor Microsoft, nor is he awed by his old company.

A few of EMC’s bloggers have weighed in, wondering what all the fuss is about. EMC’s VP of Technology Alliances Chuck Hollis wrote that some people are reacting irrationally to Green’s departure.

The real surprise for me is that it was such a surprise for so many people. If you think about it, successful companies often go through different phases, and it’s not unusual for different management skills and styles to be required during the journey.

True. But I know one company that has kept the same CEO for seven years while morphing from a storage systems company to a hardware/software/security/services/server virtualization/wannabe network management behemoth. So EMC actually makes the case against change for change’s sake.

In a blog titled VMware Greene’s contract not renewed, EMC’s Mark “Zilla” Twomey points out that Greene’s contract expires in late July. That explains the timing of the move, but not the reason for it.

VMware CEO Greene gets pink slip

Last week we ran a story on SearchStorage.com by Beth Pariseau about friction between VMware and storage vendors as more storage gets connected to virtual machines. Because the story focused on technology and not the internal workings of VMware, EMC was not among the storage vendors listed as banging heads with VMware.

But we found out today just how much friction there was between VMware and EMC, which still owns 86% of VMware after spinning out the rest in an IPO 11 months ago. The friction came to a head this morning when VMware’s board, chaired by EMC CEO Joe Tucci, replaced CEO Diane Greene with EMC executive Paul Maritz. Maritz joined EMC in February when EMC acquired his Pi Corp. and installed Maritz as head of its Cloud Division. Before he started Pi, Maritz spent 14 years with Microsoft and was on Microsoft’s executive committee.

EMC did not say Greene was fired, but its press release did not say she left on her own. Nor did it include any comment from Greene. On behalf of the board, Tucci thanked Greene “for her considerable contributions to VMware” and then predicted that VMware would increase its market share lead in the server virtualization market.

Execs from VMware and EMC have never had a warm relationship, but Greene and her team were allowed to run the company as they wanted as long as it kept growing revenue every year at astronomical numbers. Today’s release also mentioned that VMware was “modestly” adjusting its 2008 revenue guidance below the 50% year-over-year growth it forecast. But a slight miss is hardly enough reason to fire a CEO who has had nothing but success since EMC bought the company in early 2004. Perhaps it was an opening, though, and an excuse to make a move that the board wanted to make for awhile.

One financial analyst who follows both companies attributed the move to “a fundamental difference between VMware and EMC management” more than any failures on Greene’s part. There shouldn’t be much difference now — Greene was the lone VMware executive among VMware’s eight directors. Maritz becomes the sixth EMC rep on the board.

Maritz’s Microsoft background could be another reason for the move. Microsoft’s Hyper-V is now on the market as a competitor to VMware, which may be making some investors nervous. VMware’s stock price has also tumbled from $152.25 last October to $53.19 at today’s opening, but that was largely attributed to the overall market. Today’s news only accelerated the fall — shares fell more than 24% to $40.19 at the market’s close.

Forrester Research analyst Galen Schreck said the drop before today was likely due to market conditions more than increased competition. “I don’t think the entrance of Hyper-V has made much of an impact in this short period of time,” he said.

Going forward, Schreck said Maritz might want to take more of an industry leader role than Greene did. “Diane had been characterized as a humble person, not really flashy, not a big evangelist.,” he said. “Paul could take on a role with more public presence, be more of a keynote-speaker type.”

EMC hasn’t yet replaced Maritz as the head of its Cloud Division but an EMC spokesman said he expects a replacement soon.

Recovering data from a crashed drive using VMware

I was talking with a friend the other day about the prospect of multi-terabyte hard drives and how painful it would be to lose that much data. My friend — being my friend of course — countered that it’s not the amount of data, but where it resides and what the data is that’s important.

For instance, he went on, the EEPROM on your desktop motherboard isn’t more than 2MB worth of data. Yet without it the bazillion hours of work you have stored on your desktop hard drive, while safe and sound, is still useless to you because you can’t access it because your computer won’t boot.

After conceding the point, I rephrased the statement to emphasize the loss of multiple terabytes of data residing on a platter-based spinning medium, located in a computer or computer-like device providing data storage services to said computer, group of computers, or computer-like devices (whew!).

Without blinking an eye, he said he’d started a hard drive data recovery company. He built a clean room and had been perfecting his recovery skills on hard drives purchased on, all of places, eBay. As an aside, use a hammer and nail, or Sawzall, to properly delete all data from unwanted hard drives you dispose of.

A while back, I got a frantic call from a family member whose laptop hard drive had crashed. She was beside herself because on her hard drive were all the digital photos she’d ever taken. . .ALL of them. She’d meant to back up her stuff to a disk but never got around to it. She wanted to know was there anything I could do to help her.

That is when it hit me full force, I have brilliant and baleful friends.

My friend recovered almost all the data from her hard drive for me (at a very reasonable price) and now she has the first pictures of her child, some of her wedding photos and other very important moments in her life back, and on DVD this time. The whole saga got me thinking: Am I really protected from a hard drive crash? How about the executives I support? What would I do if my array at home failed where I have all of my photos!

Seeing the look on my relative’s face when I presented her with all of her photos was priceless. But it got me thinking about all the other people out there in the SMB world with the 0.5 person IT shop who don’t even know these services exist, much less who can afford the super-high cost of traditional data recovery. I don’t think today’s data protection schemes are going to be able to handle the eventuality of these super-sized drives making their way to the same SMB shops.

Do the math. A decent 100Mb pipe can push about 3TB an hour (this takes into account -25% for packet and transmission overhead). If you had three people with a terabyte drive, you’d saturate a 100MB uplink should they decide to back up to a device on the network. How are we going to back that up? The storage SaaS startups making their way to market aren’t going to be able to keep up either. Imagine backing up 400-700GB over your home Internet link where your upstream bandwidth is only 768Kbps.

I saw this coming a bit back when I got my grubby hands on the Hitachi Terabyte drive and have begun using a combination of VMware Player and VMware Workstation to mitigate my issues with capacious storage at home. I essentially virtualize the machine I want to use and deploy that on top of a generic OS install, replete with a pretty icon (in my case, Debian Linux), instructing the user to launch the player as their “desktop.” I’ll eventually get to a point where I will move upward from Player to Workstation for all my machines (right now cost is limiting me to using player for most of my machines), then run snapshots and back up the snaps to the same location as the original VMDK using RSync.

It sounds like a lot of work, but try explaining to your wife that she’s lost all her projects she’s been working on and you don’t have a recent backup because her drive is too big to back up quickly. You’ll appreciate the effort that much more when you can say, “I’ve got you covered, hon!!”

Here’s the visual I use when I explain this concept.

1) Fold a piece of paper four times (or use a folded napkin)

1a) Imagine the paper (napkin) as your physical hard drive

2) Tear off two or three 1-inch pieces of that napkin. Put them on the table next to the napkin.

2a) Imagine those pieces as virtual hard drives or volumes.

3) Reorder those 1-inch pieces of the napkin. Easy, isn’t it?

4) Peel apart the layers of those 1-inch pieces, 4x as much stuff to manipulate, making it take a little longer to move things around the table, no?

4a) Imagine those layers as individual files.

Take this one step further. Blow a soft puff of air at the three 1-inch pieces before you peel them apart (this works best with the napkin as they are slightly “stuck” together). Think of that puff of air as a failure or some sort of issue with storage. Do the same when you’ve peeled apart the pieces.

Now you have a great way to envision how your task of managing individual files (family photos) on a gargantuan hard drive (look how much napkin you have left!!). Multiply that out by a couple of napkins and you see why all of a sudden this problem of failed drives and how to protect against it becomes really hard in the TB-drive world. This can open eyes at the management level. It puts a real and appropriate understanding of why we as storage admins freak out at times when they refuse to allocate budget.

I started out talking about the advent of huge drives and what are you going to do to get the data back should they fail? I’ve developed my own solution to protect myself using some free and not-so-free tools from VMware, but I’m not sure it would scale well, or be easily manageable. Maybe a small challenge to the hardcore virtualizers out there may be in order. . . .