In this blog entry, I passed on system administrators’ complaints about the difficulty in tracking virtual machines in their large companies. The fact that this is a problem surprised me and also surprises others.
For example, on his blog, Tarry Singh asks:
“What kind of a manager are you anyways to not have a track of the machines (Virtual or Physical) in your environment?”
You’re not an unusual manager, it seems. Tarry was responding to an article in The Register titled “How many VMs are on your LAN — and how sure are you?”
Tarry thinks the Register story is a sales pitch to sell yet another auditing software. I don’t agree. I think that virtual machines are so easy to deploy that IT-savvy employees are creating VMs for their departments.
Somewhat in jest, blogger Dirk Elmendorf wrote that with virtualization:
“Now I can set up my own pet network independent from the watchful eye of IT.”
Kenny Scott responds to that blog, saying:
“Clear policies in the workplace are all that is needed to combat workers installing new Windows boxes on virtual instances, because it’s not any harder to install Windows on a virtual instance than it is on an old desktop that you want to use to do a bit of testing.”
Way back in 2006, Gartner analyst Tom Bittman told us that tracking VMs would be a big problem. In that article, he said:
“It’s a different beast with physical servers. Although server sprawl is always hard, at least you can point to a physical server and know it is there. With a VM, it is a lot easier for it to get lost.”
So, it seems surprising that IT managers didn’t anticipate this problem, but, obviously, some capitulated to the demand for VMs and deployed first without planning. I’m sure that a bunch of IT managers didn’t fall into this trap. However, I bet quite a few are grappling with rogue VMs in their organizations.
I’d like to hear from managers who have a solid VM-tracking plan. How did you do it? Got any VM-detecting tips up your sleeve? I’d also be interested in hearing from those who are having problems. Please share your experiences in a comment to this post or via my email, firstname.lastname@example.org.
The editors of SearchDataCenter.com, SearchEnterpriseLinux.com and SearchServerVirtualization.com would like to know more about your server and virtualization decisions in order to better serve your needs. Take this brief survey to help us understand your plans for choosing and implementing various types of servers and how you’ll use virtualization.
Take our survey now and let us know, and you’ll also be entered to win a $200 Amazon.com gift certificate! Simply include your email address at the end, and we will select a winner from among the pool of respondents.
With your help, we’ll find out more about your server decisions, so we can provide information on our sites and at our events that better fits your needs. Thanks in advance for taking the time to help us.
Unless you live under a rock, by now you’re probably heard about VI3. But have you seen it in action? This “short” (ha) 20-minute long video I found on YouTube shows you exactly what it does.
It features a VMware “guru” and a virtualization “newbie” who asks every possible question you could think of. It’s actually a pretty decent video. Check it out here: VMware Infrastructure 3 demo. (I was going to try to embed the video, but this blog won’t let me… yet.)
While we’re on the subject of videos, I found another good VI3 video-this time about upgrading. Why should you upgrade? Find out here: VMware Virtual Infrastructure 3 Upgrading. The speaker is a little dry, though.
One of the overarching questions I’ve had since I started covering virtualization is how will it influence the kinds of server purchases IT managers make? Is it better to buy several small, slim servers, e.g., blades, or a single large and beefy one? Now we know. Virtualization is prompting IT managers to buy fewer larger boxes, richly configured with multi-core chips and oodles of RAM. So much so that yesterday, the venerable market research firm IDC did something it seemingly never does: changed its server sales forecast in a downward direction. An article on ZDnet states:
IDC on Tuesday lopped 4.5 million units off its forecast for the number of x86 servers to ship in the second half of the decade after concluding that virtualization and multicore processors are cutting into purchases.
That 4.5 million number is a major change–about 10 percent of the servers the market analysis firm had expected would be sold from 2006 to 2010. In addition, the firm trimmed its spending forecast by $2.4 billion.
But at the same time, I’ve had countless conversations with executives from the first- and second-tier server vendors that virtualization remains a key area of focus for them, that sure, what they lose in quantity of servers sold, they’ll make up for in quality of servers sold, blah blah blah. Now, I’m no MBA, but ten bucks says that the IBMs, Suns, Dells and HPs of the world are going to find ways to offset their losses. Need lots of memory? Great — but don’t expect any huge price reductions on 4GB DIMMs. Need more I/O? Don’t just add another Gigabit NIC, why don’t you upgrade the whole kit-and-kaboodle to 10Gig Ethernet?! You get my drift…
I’m a big fan of free… free as in beer and free as in speech. Sometimes that even means free as in ad-supported. NOT Adware-supported, mind you, but ad-supported free software runs second in my book to truly free open-source software. Anyone remember Pointcast? Yeah, it was a bandwidth hog in an analog age, but I LOVED it. Knowing that, you can imagine how many of the F/OSS systems management products I’ve tried. The answer is: Enough to speak on the subject at Data Center Decisions, if not speak well (hey, first time on that end of the podium… but thats another story). To get back on track, I even liked a lot of them too, Nagios, Zenoss, Hyperic, and even Groundworks new version (though Andrew Kutz knows how much I loathed their previous version from our talks at Data Center Decisions, I’ve since changed my tune) made my short list on the OSS side.
What I was looking for:
- The ability to scan the network at set intervals, creating and maintaining a detailed scan-based inventory.
- WMI-based tools to get detailed software, services, and other information from desktops and Windows servers.
- An SMTP- and/or SMS-aware alerting system that would email and/or text my phone when the poo hit the fan.
- Rudimentary ticketing so that when one of those alerts come, I have a system to manage them by.
- The ability to monitor VMware virtual machines, and manage sprawl.
Eventually I settled on using Spiceworks. It’s free, but not Open Source. There’s no Linux version, which would normally kill me because I don’t like paying Microsoft for an operating system when I’m trying to use something free, but the resource useage is low enough that after testing I put it on an already existing file server. They all did this, but the simplest to set-up and use was another application, Spiceworks. It was quick, simple, and does everything I want. The helpdesk system in 1.5 is simple enough that I may migrate from our current software over to it. Jury’s still out on that, since the helpdesk Web portal piece trusts user input (by typing in your email address) about identity, rather than authenticating, and I’m not sure about HIPAA implications. It’s not a medical system, but it could be misused to put in fake tickets about medical systems, etc. etc. Anyway, I looked over what the ad-supported system sends out, what the pricacy policy is, and decided it was worth using since it doesn’t compromise any private data, and the ads are inobtrusive. Ok, long story short… it does a nice job identifying hardware, including virtual machines. Some short screen caps follow:
This is a virtual machine sitting on VMware Server 1.0.2. I use VS on desktops for some of our legacy apps that need (gasp) Win98, so keeping tabs on who’s making more VMs and sucking up their resources (not to mention adding to sprawl) is key. People like to play, and it’s not always as easy to lock them down as you would like. VMware-based hardware shows up like real hardware if you click the configuration tab (I won’t post the image here, at least until I edit out some serial numbers and other proprietary stuff), and the details go much further into the machine’s info. It also manages linux boxes (granted, without WMI, not as much info is gleaned, but there’s still lots of useful data.
Here’s the really useful part – regular scans, plus the ability to pick up virtual machines like they were any other machine. IOW – the ability to control virtual machine sprawl and manage documentation for vms.
Next up, once I’m done playing with Virtual Iron and have some nice Xen VMs, is to try Spiceworks out and see how it detects and documents Xen-based virtual machines. Should be a nice synergy of tests.
“Whoever comes up with a solid virtual machine documentation process will make a killing this year.”
Those words were spoken — off-the-record — by a senior systems administrator for a major utility company. I ran into him at the Red Hat RHEL5 release party this week. The subject of virtualization was in the air, and — spontaneously in casual conversations and without any prompting from me — four separate sys admins (who asked to remain anonymous) complained about their virtual machine documentation problems.
A seasoned admin — a mainframe expert — with a major financial institution said that many VMs were being deployed in her company’s data centers and departments and no workable tracing mechanisms were in place. A sys admin for a telecommunications company said it took a team of three people doing nothing else two weeks to track down all the VMs.
When two qlever, xany technologies meet, who knows what can happen? On Monday, Qlusters, which makes the open source OpenQRM server provisioning and monitoring software, will add official support for Xen, the open-source virtual machine monitor used by XenSource, Red Hat, Novell and others. Qlusters’ OpenQRM, which founder and CEO Ofer Shoshan likens to Cassatt and IBM Tivoli Orchestrator, but without the configuration management capabilities of an Opsware or BladeLogic, has been shipping since 2002, and has been in production since 2004. Customers number “in the tens,” Shoshan said, but what the Palo Alto, Calif.-based startup lacks in quantity, they make up for in quality: Network Appliance, Morgan Stanley, Credit Suisse First Boston (CSFB), and TradeWave.
As far as Xen is concerned, Shoshan said the company was supporting it to address some of customers’ current concerns around virtualization leader VMware: price, but also performance. “What we see in the market is virtualization is still mainly used in test/dev and QA [Ed. note: really?], not so much in production,” Shoshen said. Having more choices should “make people more comfortable.”
It looks like the ERP business is finally starting to catch on to the fact that their customers want to use server virtualization to cut costs, reduce downtime, and do all of those nifty things that come with SV. Novell and SAP seem to have sat down and sorted this out. It’s almost too bad it was Novell (well, the gossip-monger in me things it was Novell’s SuSe folks in Germany talking to SAP folks in Germany), because that means somehow this huge opportunity will get mis-marketed and then mis-sold, and some other company (VMware probably) will make the real bank off of it. Sorry Novell, I love ya dearly, but it’s a Fact of Life – you’re the Ted McGinley of technology companies. Anyway… this comes from a press release on Novell’s website:
“WALTHAM, Mass.—13 Mar 2007—Novell today announced that SUSE® Linux Enterprise Server 10 from Novell® with integrated Xen* virtualization technology is now available for SAP* NetWeaver* and mySAP* Business Suite. Jointly tested by Novell and SAP, SUSE Linux Enterprise Server with Xen met or exceeded SAP’s stringent performance requirements for SAP applications in a virtualized environment. Virtualization of the IT infrastructure for SAP deployments can result in enormous advantages for businesses, such as consolidation of workloads onto fewer servers for reduced capital and management costs. With this new validated solution, customers can confidently deploy their SAP applications in a virtualized environment using SUSE Linux Enterprise Server, resulting in a more reliable, flexible and cost-effective platform for mission-critical computing.”
It looks to me like VMware is behind the 8-ball on this. Looking over this 2006 IBM Techdoc, I found it wholly comprised of this rather disheartening statement:
“1. Does SAP support VMWARE for non-production?
As documented in SAP Notes # 674851, SAP currently supports VMWare in a non-production environment. This is now supported because of the improvement in storage and performance issues.
2. Does SAP support VMWARE for production?
As further documented in SAP Notes # 674851, SAP has issued a conditional statement of support. You may need to provide SAP access to a system on which VMWare is not running, but on which the error can be reproduced. This is only necessary if the error appears to occur in the layer between the operating system and the virtualizing software.”
Yet there’s more to it than that… it seems like SAP is saying the same thing MS said for years – “We will need a box that isn’t virtual in the event we can’t figure it out, or else we’ll just give up and blame it on the virtual environment”. Digging a bit deeper I found this post in a listserv relating to VMware and SAP:
“I found SAP note 171380-Linux Released IBM hardware :
The basic models listed below were successfully tested for use with SAP software in the LinuxLab and were released for practical operation: …
– eserver xSeries x445 VMware ESX Server 2.1″
And then there’s this from the actual note that a SAP-customer friend of mine was kind enough to snip for me:
“SAP does NOT support the production operation of SAP systems based on the Windows platform. The reason for this, among other things, is that Microsoft itself does NOT support the use of VMWare for MS products. If you still want to use VMWare in production operation, and you require some support, you must give SAP employees access to a system on which VMWare is not operated, but on which the error can nevertheless be reproduced.”
Here’s the rub – the references are to SAP Notes I can’t access because I’m no longer one of their customers, so I don’t know if they’ve been updated. They reference the use of Windows, meaning that the product line they’re issuing the note for is the GSX (now Server) line, not the ESX line. They also date from 2006 and 2004, prior to VI3. Back in the days of working for a Fortune 500 I could verify the currency of that Note and whether it applies to VI3 or just ESX 2.x and lower, but these days… no can do. So, lets ask the readers – is this still true? If it is, somebody at VMware needs to get a move on and push back on the “conditional support” BS. That 2004 SAP Note that’s referenced might not even really apply – its entirely possible the testing was done on the service console, without a thought that VMware isn’t Linux, just the console is. All in all, quite confusing, quite annoying, and quite difficult for SAP customers who are also VMware customers. I bet that’s an awful lot of them.
To the matter at hand of the press release – it’s about darn time that ERP vendors get on board the bandwagon that Microsoft started (by supporting their own ERP on their own virtualization platform). With all the work MS has put into their acquired Great Plains, Navision, and Solomon products, they’re clearly moving in the direction of taking on the big players (all two of them), and part of that strategy has to be the price point as well as the integration “ease” of an all Windows/AD environment. Part of that price point is going to include facts and figures on power consumption, license costs, reduced hardware, and all of the other virtualization-related benefits (namely the incredible ease of BC/DR when working with virtual machines).
Why is this important? Well, I can point to a real-world example. I happen to have it from an eminently reliable source that a mega-gigantic consumer goods company (which will remain nameless because I have a lot of money invested in their stock) has been migrating from Oracle to MS SQL to run their SAP environments as part of a move to cut licensing costs. I also happen to have it from an equally eminently reliable source that there are serious concerns about BC/DR with the SQL servers supporting SAP, and whether or not they can be brought back online at all in the event of a true site-wide disaster. Now add in that most of the IT staff at this large consumer goods company are in the process of being outsourced to one of the big computer company’s consulting arm, and you can see the disaster waiting to happen. This is exactly where such a large company should be embracing the built-in DR capabilities of VMware, Virtual Iron, or just plain old Xen. Well, ok, I wouldn’t trust a company worth tens and tens of billions to a small company, so VMware it is. If SAP won’t support VMware in production, a company like the one I’m speaking about can make them. They have the huge amount of clout required to get it done, so they should get it done.
Here’s my advice, even if you (and I’m speaking to the mega-consumer goods company here) have to use traditional DR for the SAP boxes – keep moving from Oracle to MS SQL. Save money on licenses. But for Pete’s sake (and you know who you are, Peter), don’t halt VMware project. Keep it alive. Move the MS SQL boxes to VMware. Replicate between SANs that spread between multiple sites, at the block level. Keep DR boxes there running VMware, even if you have to keep them cold. If you lose a box in the data center, VMotion the virtual machine to another box. If you lose the site, bring the systems back online from that replicated SAN storage with mere minutes (maybe even mere seconds) of downtime. Save billions of dollars in lost revenue that would otherwise result from SAP being offline. Imagine not producing any soap, or delicious beverages, or any of your other wonderful products that can be bought so affordably and yet make so much money. Now imagine that happens, and that there are documented concerns about the DR reliability of the existing systems, as well as a proposed solution to that problem.
So VMware, whatchya gonna do? If I were Diane Green, I’d be hight-tailing it over to SAP and Oracle to get virtual hardware certified just like IBM, HP, and other vendors do for their physical hardware.
We’re trying to build our reader base, so I added us to Technorati. Fun times.
(Editor’s Note: scrosby is Simon Crosby, CTO of XenSource)
As the enterprise Linux Distros go to market with integrated Xen virtualization, the jockeying for position based on the promise of integrated virtualization is going to heat up.
You may recall that last year, when Novell shipped SLES 10 with Xen 3.0.2 in Q306, they were criticized by Red Hat, who claimed that Xen was nowhere near Enterprise ready. In fact, SLES 10 delivers what it promises: a basic implementation of Xen in Linux, that allows SLES 10 to virtualize itself. Most of the flak that Novell received had nothing to do with Xen – for example, their use of the Linux loopback driver (which is known to be flaky) for guest block I/O caused some users a lot of pain, and their storage architecture is awkward – having all disks owned by the host makes it difficult to manage guest storage effectively.
The launch of Red Hat’s RHEL 5 with Xen is a very big deal. Why? Red Hat dominates the Linux market and RHEL 5 will deliver the Xen hypervisor to millions of servers world-wide, with a seven year support commitment. The implementation choices Red Hat has made for Xen will have a tremendous impact not only on the RHEL footprint itself, but also on the RHEL derivatives, such as Asianux, Red Flag, CentOS and (it must be said) Oracle “Unthinkable” Linux.
Fortunately, it appears from the beta that I’ve been running, that Red Hat has taken time to understand some of the key subtleties of virtualization. For example, RHEL 5 uses the blktap driver for guest block I/O that XenSource added last summer – a zero-copy, high performance alternative to loopback that I expect will improve performance and avoid the stability issues seen with Linux loopback. Also promising is the Xen-ready RHEL 4.5, which allows RHEL 5 to virtualize some of the Red Hat installed base, and Red Hat’s plans to integrate the cluster technology and file system GFS to deliver infrastructure components for high availability based on Xen.
It’s disappointing that Red Hat has elected not to packaage the Xen project’s implementation of the DMTF CIM standard for virtualization management. Every other virtualization vendor will support the DMTF API, so the virtualization management ecosystem vendors will have to re-jig their products to work with RHEL’s libvirt. Overall, the package still feels very much like Linux that virtualizes more Linux, and Red Hat has been wise not to over promote what this release will allow users to achieve.
RHEL 5 ships Xen 3.0.3, which I find amusing, because at about the same time as VMware and Red Hat announced a deal, in which (amongst other things) Red Hat will certify RHEL on ESX, VMware published a performance benchmark of Xen 3.0.3 vs ESX, in which they claimed that Xen was nowhere near enterprise ready. Admittedly the VMware “study” used Windows as the guest, but for Linux, Xen 3.0.3 easily outperforms ESX. (We also debunked the VMware study, showing XenEnterprise typically equals or beats ESX for Windows and Linux).
You won’t find many users virtualizing Windows using RHEL 5, but Novell is clearly planning a Windows future for SLES 10 SP1. Living up to its chameleon logo, the SUSE team is surely hoping to get some of the new SLES 10 licensees delivered by Microsoft to use SLES 10 to virtualize Linux and Windows. SLES 10 SP1 will package Xen 3.0.4, which has good support for Windows, and Novell recently announced an arrangement with Intel for help to develop Windows drivers for Xen. Hopefully SUSE will improve their storage management architecture for SP1, and overcome the need to emulate the 16 bit graphical SLES installer on Intel VT, which makes installation annoyingly slow.
Ultimately, the jury is still out on whether the Windows IT Pro wants to use Linux to virtualize Windows, even if the Linux comes for free from Microsoft. My guess is that the answer is “no” – and that the platform virtualization model of ESX and XenEnterprise will continue to dominate at least until the arrival of Microsoft’s Windows Hypervisor. <sales plug> XenEnterprise, through its upcoming VHD support and enhanced Windows feature set, allows Windows admins to achieve high performance consolidation today, and migrate to the Windows Hypervisor whenever it arrives. </sales plug> I’m similarly doubtful that having Windows support will cause RHEL customers to switch to SLES in large numbers, no matter how aggressive the Novell pricing. But the topic of Licensing is worth another blog entry on another day…
Bottom line: the Xen hypervisor is an engine, and not a car. A powerful engine that needs a great gearbox, shocks, tires and the body to go with it. ESX Server is a Bentley, XenEnterprise a Lexus. To paraphrase Diane Greene, OS vendors are not virtualization experts. So we shouldn’t be surprised that the first OSV implementations of Xen felt more like a Ford Fiesta, and moreover the Xen feature set has been improving rapidly. Don’t count the distros out just yet – they are very focussed on building value around enterprise ready Xen virtualization, and RHEL 5 marks a great achievement by Red Hat and the Xen community.