Some time back, before I was invited on as a blogger for SSV, I was interviewed by the always-fun-to-work-with Adam Trujillo about Virtualization in the Data Center, and, like all good writers, Adam left the best question for last:
“What about hardware decisions — should data center managers be considering scale-up instead of scale-out?”
My response was:
“I personally prefer a scaled-up approach because there is a reduction in ongoing costs, such as power, space, cooling, and physical maintenance. Also, the complexity factor is reduced when there is less hardware to manage. An exception to that would be data centers without existing centralized storage — the initial acquisition becomes more expensive in scale-up operations if a SAN infrastructure is not already in place.”
I’m guilty of being one of those people that says “Durnit, why didn’t I say this or that?” or “Dangit, why didn’t I quantify that a little more?” even well after the fact, making me perhaps my own worst critic. In this case, I really felt I left some stuff unsaid. One item that irks me about that answer is that I should have made more mention of blades. I hate blades in their current incarnation. I think they’re the worst idea in IT – they’re hot, cramped, delicate, with slower components and limited expansion ports – if you name something about a blade, I can find a reason to hate it. That said, I shouldn’t have left them out of my line of thought – a good IT Manager needs to consider uncomfortable things, difficult things, even distasteful things, when looking at something impactful. Or so says the wisdom of Frank Hayes, to whose articles I often find myself nodding to the affirmative while reading. So, here goes.
Blades are hot – they have limited cooling options built-in. That’s often a “value-add” (choke) of specialized rack systems and chassis systems provided by third-party vendors. Here’s a few links to illustrate the point:
- Power and cooling woes undercut blade server benefits
- IBM feels the heat
- Heat relief for data centers using blades
- Concerns heat up over keeping blades cool
A rack of big-honkin’ boxes will make you feel toasty on the parts next to their fans. A rack of blades will cook you medium-well given enough time. To prevent the data equivalent of multiple mini-supernovas you need to install the correct cooling – the correct tonnage of AC, hot and cold rack aisles, proper ventilation, air temperature monitors, system heat monitors, etc. In many data centers, the cost of new construction (or re-construction) may very well exceed even long-term cost savings from server consolidation, and even if you can afford the construction and still come out with positive ROI, that cooling comes at a monthly utility cost – you must increase your power consumption to keep things cool.
That said, this is where virtualization has been proven out over the last decade as a way decrease the number of servers and offload them to blades. That may mean that you can remove enough servers to use your existing heat management systems in a more focussed way and not have to break the bank. Even if it’s a five-to-one ratio of servers removed to virtualization-equipped blades added, you’re coming out ahead. Add in centralized storage systems to connect to the blades and the scales may well tip back in favor of Mr. Heat Miser again, but probably not. Getting a ten-to-one ratio means blades are a winner. This is assuming a large server consolidation via virtualization project. If it’s not a big percentage of your boxes being affected, you’ll be back in the hot seat, quite literally.
Ever need five or more NICs for a virtualization host? I have. If I had blades, I’d be using three blades to get that done, assuming dual nics, and five or more on single-nic blades. That means more blades, more virtualization software licenses I don’t need, more hardware to fail, and more physical boxes when what I want to do is REDUCE the number of physical boxes. Right now server blades are still too young – many vendor’s products have all the components are included on the blade, and not modular enough. PC blade systems have it a little better – some limited peripheral connectivity at the user-site (see this link for one manufacturer’s solution), but still, it’s an entire box in a chassis with all the difficulties of expanding that micro-sized PCs and laptops have.
So, I think it’s safe to say that I still hate traditional blades. But I think they’ll be the saviour of the data center soon, and then I will love them. Why? Because here’s my ideal blade system: a truly modular system that will change everything about blades. The best part, it’s available now from several of the larger vendors. The changes are part of a new design “paradigm” (please note my bias against that word) – the end-result is a blade system where the blades can be NICs or other devices, as needed and plugged into the chassis, connected in either a physical layer with ye olde jumper or a software layer (in the chassis management software, perhaps). Lets say I get a blade and I need to put ESX on it, but I need six NICs because of guest system network i/o requirements… ok, I get another blade with a quad-NIC on it, plug it into the chassis, and configure it – voila, a single computer with five or six NICs in two blade slots, using one license. Or perhaps I need ten USB connectors for some virtualized CAD desktops, which require USB key fobs in order to use the CAD software – I plug in a server blade and a USB blade, configure it, and voila, one server, ten USB ports, one license. Expand that out far enough, and you can have whatever you need in terms of peripherals in a blade chassis. If you go to IBM’s website, you get a whole panopoly of choices – switchblades (that one always give me a chuckle) and NIC blades are readily available for expanding your blade chassis out to do more than just host some servers. HP upstages them a bit and has a great product out now that provides PCI-X and PCI-e ports. This is from their website:
“Provides PCI-X or PCI-e expansion slots for c-Class blade server in an adjacent enclosure bay.
- Each PCI Expansion Blade can hold one or two PCI-X cards( 3.3V or universal) ; or one or two PCI-e cards(x1, x4, or x8)
- Installed PCI-X cards must use less than 25 watts per card. Installed PCIe cards must use less than 75 watts per PCIe slot, or a single PCIe card can use up to 150 watts, with a special power connector enabled on the PCI Expansion blade.
- Supports typical third-party (non-HP) PCI cards, such as SSL or XML accelerator cards, VOIP cards, special purpose telecommunications cards, and some graphic acceleration cards.”
This is interesting – a couple of PCI-e quad-NICs in one of an expansion unit and my NIC requirements are set. Or perhaps a couple of PCI-e USB add-in cards. Or a high-end PCI-X or PCI-e video card. Ok that gets troublesome when you need a lot of them – you can wind up with one blade and a chassis full of expansion slits containing video cards – the cost might not be worth it.
In any case, this dramatically changes my view on scaling up or out. Right now, I still stand for scaling up because blades don’t work in my enviornment – I have heat problems. I have space problems too, which blades could solve, but not with my heat problems. I prefer to buy larger-sized servers with lots of expandability (DL300 and 500 series, PowerEdge 2000 and 6000 series, etc.) and add in NICs as needed rather than buy blades or 1U boxes because I can do more with these larger-sized machines even though they take up more room. I fully expect that to change in the future – at some point I see myself stopping with the scaling up and starting with the scaling out – only I expect the “out” part of that will involve a lot less real estate and more options than currently available.
SearchServerVirtualization.com is now soliciting nominations for its Products of the Year awards. We invite you to nominate your favorite product or your company’s product by using the form at the entry page. Winning products will be featured in January 2008 on SearchServerVirtualization.com.
SearchServerVirtualization.com staff and other industry experts will judge the entries. Your product(s) qualify for submission if they have shipped (or have been significantly upgraded) between October 31, 2006 and before November 1, 2007.
If you are submitting more than one product, you must fill out a separate form for each product.
Note: Products entered for Best of VMworld awards are eligible for entry. This is an entirely separate award.
The deadline for all submissions is November 2, 2007.
Products are limited to one category. They must fit into one of the following categories for consideration:
-Data protection (Including backup, replication, HA and FT products)
-Systems management: Monitoring and reporting
-Hardware for virtualization (Including, but not limited to: Servers, storage, I/O components and client devices).
-Virtualization platforms (e.g. VMware ESX, VI3, Microsoft Virtual Server, Virtuozzo, XenEnterprise, etc.)
We’ve identified the following criteria as being most important, and will judge accordingly:
* New or upgraded features and capabilities
* If the product is an upgrade, how the upgrade has affected sales and user adoption, and
* User reviews.
Bloggers, feel free to mention this in your own blog to spread the word!
The need to hire qualified staff to design, implement and manage virtualized environments is growing, and that means hiring managers are having to shift focus towards this distruptive technology and be ready with good interview questions for their prospective hires.
1. Do you have experience in (VMware/Xen/Virtual Iron/Virtuozzo) implementations?
This is the no-brainer question, and the lead in to the others. If the prospective hire’s answer is no, stop right here, do not proceed past go, do not collect $200. Even a certified candidate may not have any experience, and an inexperienced candidate isn’t one you want for the job, since you probably have staff who would like to learn on the job or be trained, and already have the internal processes and procedure knowledge to edge out the competition from outside.
2. When implementing a virtualization environment, what do you consider the most important feature of the product to ensure overall success of the implementation?
This question is good for sorting out who sees the strategic value of virtualization and who is focussed on the technical aspects. A good answer will cover either failover functions or the ability to reduce costs, and relate how they will benefit the business in technical terms. Neither a techie or a managerial answer is right or wrong, but rather will help you sort the crowd of interviewees into the categories you are looking for.
3 . When you were at WidgetMakers, Inc. you list in your resume that you used VirtualBlahBlahBlah to aid your company in meeting the goal of DoingThisOrThat. Can you share with me what challenges you experienced and how you overcame them?
This is a typical interview question surrounding any product, and it needs to be asked for any product you are hiring somebody to work with.
4. How deep is your understanding of storage systems, and can you share an example of how you used this knowledge in a virtualized environment at WidgetMakers, Inc.?
5. How deep is your understanding of network switching, and can you tell me how you would use virtual switches in a broad virtualization implementation?
Cross-disciplinary skills are crucial for virtualization, particularly around storage. Many larger companies have storage administration teams, server administration teams, network administration teams. Being able to work with these groups doesn’t mean that the candidate can work with the technology, and its important that, if the position is technical, that they can do both.
6. Tell me about how you would configure a virtual environment to best take advantage of its features in a backup and disaster recovery framework?
Being able to understand how to use DR-friendly features like VMware’s vmotion and backup-friendly features like snapshots can make all the difference in candidate selection. It’s important for a candidate to know how to keep the business running, even if they don’t know the business itself yet.
7. Tell me about VirtualizationProductFeature, and what you think makes it valuable or not valuable.
This gets into the technical understanding of the product, a crucial point in both technical and managerial interviews. If a technical candidate blows this one, they need to go home. If a managerial candidate doesn’t provide a business-oriented answer, they need to go home or consider a technical position.
8. BadThing happens. Tell me how you would troubleshoot the situation and get it resolved.
A typical technical question, and one that should always be asked to both technical and managerial candidates. Managerial candidates may get some leeway in technical minutia, but absolutely must speak about their role as the manager and how they would deal with their technical staff to get the problem resolved. This is also a rinse-and-repeat question that should be asked a couple of times, using different BadThings.
There are also consultant-specific questions to ask, if that’s what you’re looking for. Things like:
1. How many VCPs do you have on your staff?
Until the other companies start with their own certs, the VCP is where the game is at.
2. How many virtualization projects has your company undertaken in the last year?
The default no-brainer.
3. Do you eat your own dog food? By that I mean does your company use the product internally as well as support it?
Also a no-brainer
4. What was your company’s most spectacular failure?
Everyone is going to tall you about their company’s great success. Make them squirm a bit and tell you about how they failed, then then ask:
5. What did you do to correct the situation?
This will tell you what kind of consulting firm you are dealing with. If they can be upfront with these two questions, if the failure wasn’t a show-stopper for your environment, and if they dealt with it right, they get high kudos.
Obviously there are many, many more questions to be asked of potential staff, managers, and consultants, so many that I’d like to encourage people to comment about questions you like to ask, would like to be asked, or think would be important – I’m looking forward to some audience participation!
Following the launch of the article “VMware dispels virtualization myths (sort of),” VMware emailed me to correct some issues about virtual machine security.
According to VMware, an “incorrect statement” was made by Burton Group Analyst Chris Wolf, who, like all of the engineers at VMware he’s spoke with, he thought to be correct.
In the article, Wolf said, “one significant issue with virtual machine security is with virtual switch isolation. The current all-or-nothing approach to making a virtual switch ‘promiscuous’ in order to connect it to an IDS/IPS is not favorable to security.”
For example, “if you connect an IDS appliance to a virtual switch in promiscuous mode,” Wolf said, “not only can the IDS capture all of the traffic traversing the switch, but every other VM on the same virtual switch in promiscuous mode could capture each other’s traffic as well.”
This statement ruffled some feathers at VMware, and they quickly emailed me and Burton to “educate us” and the VMware community that in fact, VMware allows (and encourages) users to configure only the ports they need to be promiscuous as such. This is not a per vswitch setting, but rather a per portgroup setting. The way to configure a vswitch for IDS/IPS is to create a separate portgroup from those used for normal VMs and configure it for “Promiscuous Allowed,” a VMware spokesperson said.
After testing this out in his own lab, Wolf said it is really an easy solution, because the architecture is already there.
“At the switch level, promiscuous mode is an all or nothing configuration. VMware doesn’t argue this. However, a way around this issue is by configuring a separate port group on a virtual switch just for the IDS and making the port group promiscuous. That allows the IDS to monitor the vswitch traffic and still keep all other traffic isolated,” Wolf learned from VMware.
“So, with the port group feature it isn’t all or nothing, it can be granular,” Wolf said. That said, “Vmware’s own team wasn’t even aware of this,” therefore it’s unlikely many VMware administrators are either, he said.
So the record stands corrected. “The option of making a virtual switch ‘promiscuous’ in order to connect it to an IDS/IPS is not favorable to security and should never be used,” Wolf said. Instead, administrators should create a dedicated port group on the switch for the IDS and only make the IDS port group promiscuous. This would allow the IDS to monitor all unicast traffic on the switch while preventing all other VMs on the virtual switch from seeing each other’s unicast traffic.”
Developments at VMworld 2007 show that virtualization 2.0 has arrived, says Burton Group analyst Andrew Kutz. But can virtualization stay sexy when it is mainstream?
VMware CEO Diane Greene says VMworld 2007 wowed her with innovation and enormous vendor and user participation.
Analyst Barb Goldworm explains why VMworld 2007 ushered in a new era in virtualization.
Good-bye to pesky print drivers, hello to virtual printing. ThinPrint’s VDI-focused printing approach won recognition in the SearchServerVirtualization.com VMworld Awards’ Utilities category.
InovaWave CEO Chris Ostertag describes virtualization’s I/O problems and how InovaWave VirtualOctane for ESX Server could solve them. The soon-to-be release product won a SearchServerVirtualization.com VMworld Award for Best New Technology.
Akorri founder and CTO Richard Corley describes the virtualization management problems solved by Akorri BalancePoint 1.7, which won SearchServerVirtualization.com’s VMworld Award in the performance monitoring and optimization category.