I am blogging from the first annual Technosium 2008 Global Conference and Expo in Santa Clara, Calif., where I have gotten some clarification on regulatory compliance, a topic that perplexes me and other virtualization managers and administrators I know.
Have we bent the rules of regulatory compliance to get virtualized systems online? With the configuration involved in putting systems on DMZ networks, Internet-facing networks, and customer or vendor networks that handle regulatory-sensitive material (like HIPAA, Sarbanes-Oxley and others) on virtualized systems, we may have created a compliance issue.
After talking to consultants and vendors, I came to the conclusion that it may be time to check and double check the regulatory compliance aspects of any work on our virtualized implementations. There’s the possibility that, from a compliance perspective, we may not have segmented all of our regulatory-protected systems adequately.
Naturally, vendors have seen the problem rearing its head and are offering automated tools. At the show today, for instance, I met with Joao Ambra, security product manager for Modulo, which specializes in IT governance, risk and compliance management. Modulo’s Risk Manager Software product assesses regulatory compliance as well as risk assessment and audit services for organizations.
Besides describing Modulo’s product, Ambra gave advice on four key areas for determining if regulatory compliance is met:
Technology: This includes the infrastructure components as the network environment, databases, servers, computers and other physical elements.
Processes: Procedures such as backup, restore, disaster recovery, password policies and internal change control management make up a processes assessment.
People: Staff training levels on the technologies used and regulations applicable to the organization are important parts of the employee inventory.
Environment: The environment consists of physical access, facilities and risks associated with physical presence of computing resources (and protected data).
According to Ambra, the key strategy to collecting data for compliance-measurable items includes identifying where virtualization fits into the components. When a virtualized system hosts critical elements or regulatory-sensitive material such as databases, access to protected healthcare data, or fixed asset systems, the virtual host and all of its elements are subject to the same scrutiny as the underlying systems. This includes the hardware, database and security configurations for the virtual environment.
Virtualization, in principle, protects from server systems running too many roles while accessing protected data. However, this is all contingent upon the implementation of the virtual environment.
Today at the Technosium Global Conference and Expo in Santa Clara, Calif., I saw a cool demonstration of FastScale Composer at work deploying virtual (and physical) servers.
FastScale Technologies Inc., maker of Composer, is a VMware alliance partner. FastScale Composer is a tool that facilitiates building physical and virtual servers from bare metal with a configurable inventory of operating systems, applications and updates. FastScale Composer is suited for data centers with 250 servers or more.
I met with FastScale CEO Lynn LeBlanc and Richard Offer, vice president of engineering, who discussed FastScale Composer’s key feature: a software component repository that contains operating system binaries, software packages, updates and user-configurable material available for systems.
Within the Composer interface, systems are allocated with your configured inventory to be used when they boot. The underlying technology for arriving new systems is a pre-boot execution (PXE) environment that will have the configuration for the system delivered. Composer excels at this step because the package that arrives to the new system is just what’s needed. For example, in a demo I saw, a base Linux install for a Red Hat system arrived to the system as only an 8 MB image via PXE. While that is not an entire installation, the full inventory is made available to the servers via the respository.
What impressed me is this: Should any element of the system need something from the repository, it is automatically retrieved. Also, servers can be built without the need to retrieve from the repository if you want everything available locally or the repository not be available.
FastScale also has an interface into VMware. While you can perform traditional PXE builds on virtual systems as you would on physical systems, FastScale Composer’s Virtual Manager plug-in will populate new servers directly to VMware ESX. The Virtual Manager option to Composer will allow a virtual machine to be created as VMDK files and imported to ESX or VMware server. A small agent is required on at least one ESX server to receive the VMDK from Composer.
LeBlanc and Offer told me that a new version of FastScale Composer, coming soon, will incorporate Microsft Windows version support and an improved interface. For more information or to arrange a demo, visit the FastScale web site.
I am blogging from the inaugural Technosium Global Conference and Expo at the Santa Clara Convention Center. I’ll be signing on intermittently to provide you with everything I consider beneficial to the virtualization space.
First off is storage business continuity. I had an opportunity to attend a breakout hosted by Eric Herzog, the vice president of operations for Asempra Technologies. While business continuity is a topic we all are familiar with, attendees had the chance to look at the building blocks of a successful strategy for continuity, and how it applies to storage for virtualized systems. What I took away from this breakout was that there needs to be clearly defined goals that the business requirements define the following within an organization’s service level agreement (SLA):
- Ability to measure availability and uptime
- Data loss tolerance for your business needs (financial and operational)
- Disaster recovery time frames made specific to your environment
- Solutions that reduce costs with combination of technologies with reduced complexity
- Ensuring that the SLA leverages the existing infrastructure (hardware, software, networks) as much as possible
- Ensuring that there will be usable data on first recovery
Traditional approaches to data continuity to virtualization systems tend to respond with multiple technologies, various products, limited manageability and control, increased costs and expense, and a cumbersome process that limits successful execution of the SLA. While each storage business continuity strategy has positives and negatives, the right solution will depend on your virtualization availability requirements. Among the newer storage technologies are drive snapshots, data deduplication, and remote replication. Some solutions address actual virtual machines, where some address the shared storage systems that host virtual environments. Remote replication provides the fastest recovery time for a virtualized storage system, but also at the highest expense.
In summary, the business continuity strategy for virtualized systems needs to resolve primarily around the technology behind the storage systems in use. The other challenge to virtualization management is to define the goals of virtualization continuity via an SLA.
Despite VMware Inc.’s having missed analysts’ estimates for the fourth quarter of 2007 — which sent shares into an after-hours free-fall on Monday (trading was at $61 a share at 6:00 p.m. Eastern Standard Time, down from $83 at the market’s close) — the company’s CEO Diane Greene maintained that the company will not lower product pricing in 2008. “We do think we can maintain our pricing,” Greene said on an earnings call with investors. “We anticipate continuing our pricing over the course of the year.”
Greene also said that while customers (of which there are now reportedly more than 100,000) are investigating Hyper-V, thus far the company hasn’t seen an impact from Microsoft’s forthcoming virtualization platform. “Our customers have evaluated it and told us they saw no reason to switch,” Greene said, pointing to an InfoWorld report that compared Hyper-V to VMware Server 1.0, “but not as polished.”
Of course, Wall Street is rarely satisfied. During Q407, VMware increased its revenue 80% to $412 million, profits more than doubled, and the company is on a run rate of $1.6 billion. But the ongoing shift in revenue from licensing to services (support, subscription and professional services), as well slower U.S. growth relative to international markets has worried investors, who responded by depressing the stock price to levels not seen since mid-August, days after the IPO.
Looking forward, Greene told investors that in 2008 the company plans to unveil new products for security and virtual desktops and will “preview a major new Virtual Infrastructure suite.”
Test-and-development environments that want to see how software runs on Citrix Systems Inc.’s XenServer virtual machines can now do so, thanks to VMLogix, which has added Citrix XenServer to the list of platforms supported by its LabManager offering.
Citrix XenServer joins a comprehensive list of virtualization platforms supported by VMLogix, including VMware ESX Server, VMware Server, and Microsoft Virtual Server; support for Oracle VM and Sun xVM is also forthcoming, said CEO Sameer Dholokia. For the time being, Dholokia said, the company has seen “a fair bit of interest in testing VMLogix on Citrix XenServer.”
The VMLogix offering competes directly with VMware’s Lab Manager and conceivably with the newly announced Stage Manager. In fact, Dholokia claims that VMLogix’s offering already includes much of the functionality included in Stage Manager and said that VMware customers may not understand the distinction between the Lab and Stage Manager products. “It will be interesting to see how they manage the confusion factor: ‘When do I use [VMware] Lab Manager, when do I use Stage Manager?'” Dholokia said.
Pricing for VMLogix LabManager is $25,000, plus a $2,500 agent fee per two-CPU server.
ivi (pronounced eve-e) stands for Java Virtual Interface, and it is a project that aims to create a single, graphical, management interface for all the major virtualization products.
Implemented in Java+Swing, ivi is truly portable. Currently ivi uses the VMware Virtual Infrastructure 3 (VI3) software development kit (SDK) to communicate with VI servers and the XenApi to talk to Xen servers. Future plans include adding support for libvirt to allow communication with KVM and OpenVZ and, eventually, support for the Common Information Model (CIM) as a way to talk with VMware, Xen, and Microsoft all through one interface.
The number one barrier to a properly utilized datacenter is the lack of a single management tool to control a heterogeneous environment. The idea behind ivi is to create a single management application that can be used to control all of your datacenter’s virtualization solutions. With so many virtualization products available, using so many different types of architectures, it is more important than ever to possess a management tool that can provide IT professionals a single point of management.
You can read more about ivi at http://www.lostcreations.com/code/wiki/ivi.
Since 2.0 is almost out, I imagine this will need a follow-up at some point. (Good…it’ll keep me focussed on blogging.) In the meantime, I decided to give the 64-bit world a whirl, as we’re evaluating moving to Exchange 2k7.
With my box racked up and plugged in, I grabbed the ISO for CentOS 5.1 and gave it a whirl. On a side note, it installed perfectly on a Dell PowerEdge 1950 w/ SAS disks in a RAID 5 array on a PERC5 card. In spite of some people having problems related to the RAID setup, mine went through flawlessly. (Apparently there is a known issue with multiple arrays on a single card and GRUB’s placement on the wrong array.)
After getting the OS up and running, I gave it a whirl using a FAQ I found at Nixcraft to install VMware server.
I did it once on a fully-updated install, complete with the updated kernel packages, and it bombed out. Going back, and using an updated kernel worked flawlessly. To sum up the process, here’s what to do:
- Install CentOS. Make sure you have your gcc packages installed (under the Development trees during setup).
- Grab the latest rpm from VMware and install:
# rpm -ivh VMware-server-<version>.rpm
- Install some more needed libraries:
yum install libXtst-devel libXrender-devel
- Install xinetd:
# yum install xinetd
- Run your config:
Considering the recent acquisition of Thinstall by VMware, I have to wonder: Is PortableApps next on the list of to-be-acquired companies? The two companies have one thing in common: They both have products designed to take entire applications and put them into a single container for portability and reduction in complexity. I’ve kept a number of PortableApps applications on my USB stick for a long time — it’s nice to have a quick set of tools to use without having to use somebody else’s settings, leave traces of my work on their systems, etc. Throw in VMware player and a stripped-down guest OS with PA’s software on it and you have a real winner. Take it to the next step and put PortableApps applications onto a server that distributes software via a thin approach (Citrix, 2X, etc.) and you have a hit in application virtualization. One big beef I, and many otherwise-fans, have about Citrix is the all-too-real potential for winding up DLL Hell. Applications to be served via server-based computing solutions like Citrix (or 2X, TS, etc.) often need to be isolated if they are mission critical (would you run your Peoplesoft app on the same Citrix server with your Office 2007 suite?), which usually means adding more Citrix servers. This, in turn, means a heavier workload for staff and host servers (if you virtualize Citrix).
Enter application virtualization. There are a lot of good brands out there, notably Softricity, which was swallowed up by Microsoft already. Thinstall is another application virtualizer (albeit via an entirely different process). Then there is PortableApps, which does much of what Thinstall does, just not as much of it. Thinstall 3.3’s product description reads (in part) as follows:
Thinstall is an Application Virtualization Platform that enables complex software to be delivered as self-contained EXE files which can run instantly with zero installation from any data source. The core of Thinstall VS is the Virtual Operating System, a small light-weight component which is embedded with each “Thinstalled” application.
PortableApp’s reads like this:
A portable app is a computer program that you can carry around with you on a portable device and use on any Windows computer. When your USB flash drive, portable hard drive, iPod or other portable device is plugged in, you have access to your software and personal data just as you would on your own PC. And when you unplug the device, none of your personal data is left behind.
There are differences, of course, but the overall business models are very similar. PA is an open-source outfit, which makes it a bit more transparent than Thinstall, which has a mix of OSS and proprietary products. Since PA is purely open source, and relies a lot on the community to deliver portable-ized apps, its list of of programs is smaller and limited to open-source and other freely licensed software. Still, with the recent focus on OSS in the enterprise, one can’t help but see the value of a PortableApps version of OpenOffice sitting on a Citrix Server for thin-client and virtual desktop users to access.
Since Thinstall and PortableApps both provide OpenOffice (Thinstall does so as a demo), I took them for a spin. In my “everyman” test, which was certainly not scientific, I ran them both from a network share on the same NAS box over SMB to a Parallels virtualized XP machine with 512 MB of memory. The Thinstall demo downloads as a zip file that you extract to the desired location. The PA app downloads and installs (using an NSIS installer) to the desired location. Once extracted, I ran them three times each — the total time listed is until the app was usable. The Thinstall application (which uses v2.4 of OpenOffice) took 18 seconds on the first test to load up, 13 seconds on the second pass, and 16 seconds on the third. The PortableApps version, which uses v2.3 of OpenOffice, required setup information (your name, if you want to register, etc.). I decided to discount this from my scoring, but it took 17 seconds, in case anyone is interested, from launch to the registration screen. Once that was done, loading time until a usable screen appeared took 18, 22, and 17 seconds. The splash screens appeared at 10, six and eight seconds, respectively, for Thinstall; and six, eight and eight seconds for PortableApps (excluding the PA-promo splash, which took one second each time and is a separate splash from the OOO splash).
I moved them to the local drive and things got livelier. Thinstall loaded in six, three and three seconds (with splash at two, one and one seconds). PA loaded in six, two and two seconds (with splashes at one second each, but the PA-specific splash obscured them two of three times).
There are a lot of other interesting applications out there just ripe for the virtualization space; browser-based desktops like that offered in alpha by g.ho.st comes to mind. Expect a post on that sometime in the near future.
Todays episode is … screwing up your test environment and fixing it. As many know, I test management tools like some people switch socks. I have a simple theory on it: If what I’m using could be better, and there’s no business impact in switching, I’m going to switch. I went from Groundwork Open Source to Spiceworks 1.0 (see my prior post here) to Hyperic. Spiceworks 2.0 is out (and has been for some time) and it’s even spicier than the last version, although the Chili Peppers were missing for a while. So time to put it into the demo lab, right? Absolutely! A little short on time? Of course! In so being, I hosed that Spiceworks 2.0 demo box from sheer stupidity. And I mean sheer stupidity. I didn’t follow any best practices. The worst thing I did was putting the app on a default XP build image. It didn’t need to be on a server OS for the initial testing phase, and I had a blank XP image ready to go. To channel my Inner Yoda: Saved myself some setup time I did. Hosed myself in the long run I had.
The default image for an XP Virtual Desktop is 8 GB in size. This is just enough space for the applications to be installed and to have room for updates, patches and whatnot. It makes maximum use of disk space on our NAS box without leaving the XP machine’s disk too stripped for future expansion. Or so I thought.
As it turns out, Spiceworks eats up disk space, about 4 GB for the one subnet I have it scanning. I didn’t have the helpdesk module in use except for my own testing, so that space count doesn’t include tickets with attachments. Considering that this machine had just under 4 GB of free space, I soon found myself with a problem, one that caused my Spiceworks install to fail, and the box to slow to a crawl.
What to do … should I rebuild? I could, but then I’d lose the data this thing had accumulated over the few days’ time it took to fill the disk, and I liked the results of the test. It was performing on par with Hyperic in terms of general scanning and reporting, although the drill-down detail was somewhat less, and the ability to customize access to a given resource was less (Spiceworks supports configuring only one Windows domain account and one *nix account, and uses SSH and WMI, as opposed to local agents reporting back in the way that Hyperic does). The alerting system was solid. The inventories were solid. Everything about the software was flying high and showing happy, except, of course, the one problem I was having – keeping it from eating the “entire” disk drive. Also, it was running great on XP, and with VMware, there’s no real difference in managing disaster recovery or backup for an operating system. I’m still ticked that there’s no version to install on a Linux box … but that’s for a different forum than this.
Should I recover as much disk space as I can and hope for the best? Sure, why not give it a try? I’ve already hosed myself here, in spite of being addicted to documentation and following templated procedures whenever possible. A short while later, no real luck. As much as I remove, Spiceworks continues to grow as it records scanning results and archives historical data.
Time to fix the problem at the disk level. I did this not too long ago on my XP machine in Parallels, based on this blog post, so I just modified the process to fit the VMware Server 1.04 environment. First off, to expand this disk size, I went to the terminal on my Macbook, remoted to my VMware Server box via SSH, and executed the following command from within my virtual machine’s directory (on my boxes, this is /vmdir rather than the default) :
vmware-vdiskmanager -x 14GB “My Disk Name Redacted.vmdk”
After a short time the disk was successfully expanded. Now, of course, just because the disk was expanded doesn’t mean the space is usable, which is where GParted comes in. Grab a copy from the wonderful folks who host it (SourceForge). GParted makes you go through some configuration when you boot from it, just so that you get the best display resolution and so that it boots properly, but most anyone with experience in IT can follow the process without me outlining it here (especially since it only involved pressing enter twice, and has been outlined elsewhere ad infinitum). I switched the removable drive over to my GParted iso file, booted and configured. At this point, as you can see from the images, I had a full view of my disks and could manipulate to my heart’s content:
I simply right-clicked, selected Resize/Move, and slid the slider from the current position to the end of the drive. The steps, graphically, are as follows:
Strangely enough, it errored out on me (and I didn’t get a cap … dangit). Turns out it wasn’t a show-stopper, but it was strange. When I rebooted, it loaded into Windows normally, which threw me for a loop — it is *supposed* to do a disk check. When the check completes, Windows should see the expanded drive size. As this pic shows, there was a decided difference in opinion between what the drive’s capacity was and what was available in reality.
Not deterred, I manually ran the chkdsk for the next reboot, whereupon nothing changed. Since I had an error before, I decided to try again. I resized the drive to add 1 MB of unallocated space at the end, applied (this time with no errors), and rebooted.
Windows came up, ran a quick scan of the disk, and voila! It found the extra space just fine.
Was the error just an anomaly? I checked Google but nothing really popped. I checked the GParted forums — nothing really popped. Chalk it up to a slight hiccup at the wrong time, perhaps a bug as yet unseen. I posted the info and moved on.
So, as you can see, GParted helped me pull my test bacon out of the test fire before it test-burned. This is definitely a utility I will be keeping in my toolbag from now on.
VMware’s flagship products ESX 3.5 and Virtual Center 2.5 have been available for a little over a month now. When the upgrades were made available, there was much excitement on the newly touted features. So, many IT professionals quickly hurried off and downloaded their product updates and then came to a collective stopping point. How do we upgrade ESX while in use? Sure we upgraded from ESX 3.01 to 3.02 with very little impact. But the change from 3.0x to 3.5 may seem worthy of more preparation because the scope of the change is larger with some of the new features, like Storage VMotion. With the release, here is a simple upgrade strategy that many are adapting:
- Allocate two ESX 3.0x systems as 3.5 candidates (not everyone will be able to do this, I realize).
- Carve these two systems into their own cluster or data center.
- Make sure all existing VMware DRS rules would be okay with two systems removed.
- Upgrade or fresh install one of the systems to ESX 3.5.
- Test migration from ESX 3.0x to the new ESX 3.5 system.
- Test VMware tools versioning and test any upgrade virtual machine tasks.
This strategy will replicate what you will likely face in a real upgrade situation, as you may not be able to. Because you may only be able to have a limited number of systems available for maintenance at any given time, it is good to be able to replicate that in a test datacenter or cluster. In smaller implementations, this could be repeated with one host where the migration test would be from a live ESX host. Evaluation software may also be a consideration to make available the correct number of hosts to simulate the co-existence of ESX 3.0x and 3.5.
Keep it moving
Should your configuration allow seamless migration between your ESX 3.0x and 3.5 hosts – that should not be a crutch for undefined periods of mixed versions. A good practice would be to have all hosts on the same version of ESX within a cluster. Larger environments may have difficulty moving all systems to the new version, but strategize within your Virtual Center configuration to determine the best configuration for temporary mixed versioning. The goal should be to get all systems on the same version enterprise wide – but only after you are completely comfortable with 3.5 in your infrastructure.
The horse’s mouth
VMware provides many quality resources online, I’ve saved some work for you and collected some of the highlighted pieces here for review in relation to ESX 3.5 upgrades:
These resources are a good strategy in being well informed for the what your plan for ESX 3.5 will entail. Simply installing without preparation is surely a recipe for mis-configuration or incorrectly applying your configurations as intendend. And the test upgrade procedure to become familiar with a mixed environment will allow you to clear the way for an end-state configuration of a single version of ESX.