Today’s guest post is from Graeme Elliott, a Sydney-based Storage Architect for a large financial firm and leader of the Sydney Tivoli Storage Users Group. Elliott will be starting his own blog on IT Knowledge Exchange shortly, to be titled The Art of Storage.
Virtual Storage is the “art” of moving the “smarts” (Mirroring, Snapshots, Replication etc.) from a standard storage array’s controller and placing them into an appliance. This appliance is placed in the data path between the Host and the Storage Array. Even though these virtual appliances are in path there is generally only a minimal performance hit and in some cases, a performance boost due to the appliance’s caching algorithms.
The storage from the backend storage array is now presented to the virtual appliance just like any other host. The virtual appliance can further carve up or merge this storage as desired into “virtual disks” that can be allocated to your hosts. From the Hosts’ perspective, these virtual disks are just like any other disk.
The advantages of doing this are tremendous.
- All that you need to worry about when acquiring storage arrays is storage and performance (no smarts required)
- Span “virtual disks” across multiple arrays or Raid Groups
- Only need to the use the host drivers for the Virtual Appliance, not for the specific storage array vendor
- Storage from arrays can be placed in pools on the Virtual Appliance allowing for a tiered approach to allocations
Another major benefit of these “virtual appliances” -and one that in my experience usually drives the initial purchase-is the data migration features offered. Being able to migrate data between storage arrays while the host is online is highly beneficial in most organizations when it comes time to lifecycle a storage array.
In my “ideal” storage environment, I would have a virtual storage appliance between all hosts and their storage. This provides a consistent and fast method for storage administrators to provision storage while also providing a consistent presentation of this storage to the hosts no matter what backend storage vendor or storage array model is used.
There are always downsides for technologies like this and virtual storage is no different. Troubleshooting performance is more complex as host data can now reside across multiple backend storage array Raid Groups and LUNs, spread across multiple storage arrays or be shared with other hosts on a backend LUN.
A week ago, after coming across some rather graphic server room pictures (SFW unless your office believes in the sanctity of server equipment), I asked the community for its favorite server room DIY or horror stories, promising the winner fame and fortune (or at least some free swag). Well, we have a winner: IT Knowledge Exchange member Batye’s tale of a Russian server room that might’ve stumped the Hardy Boys. As thanks, we shipped Batye free copies of:
This went out without any apparent cause: Total breakdown in communication, only to return again later. It wasn’t a virus, and all the hardware seemed to be fully functional, until it completely disappeared.
The facility was quite modern: Secure parking, video security, armed guards with AKM’s and Saigas. The office even required passing a facial recognition scan to access the facilities. After having a specialized name badge made up, complete with his photo and a unique barcode, he was given access to the server room, the epicenter of the irregular crashes.
The scene made our leaked server room pics look like a networking Taj Mahal: The server room was a refinished washroom/utility closet. Nobody had bothered to put in server racks, leaving the high-end equipment and air conditioner to sit directly on the floor amid moldy walls and leaky pipes.
The problem quickly became apparent: Anytime someone flushed the toilet, water (one prays) from the leaking pipes got on the UPS. The UPS’ surge protection would kick in, automatically powering itself down along with the servers connected to it, hence the disappearing pings.
A few hours later, the air conditioner would dry up the flushed water and the UPS would reset itself, and all would be more or less right with the world again – until someone else needed to go.
Blame the setup on new fangled IT theories: The server room was managed by a 17-year-old kid, the son of one of the company’s executives.
If you want some 140-character goodness all about virtualization, check out these accounts to keep track of what’s going on.
1. @virtnews: Get the latest news on virtualization software from the top virtualization blogs across the internet including VMware, SearchServerVirtualization, and Virtualization.com, conveniently organized on their webpartner site.
2. @VMware: Receive updates on events, products, and troubleshooting.
3. @vmwarecares: Ask your questions about VMware and get answers from the community and experts. Their profile boasts “human assistance”; can’t beat that!
4. @SMBVirtualization: Everything you could possibly need aside from the hardware, this feed is chockfull of white papers, blog posts, videos and instructional material geared specifically toward small- to medium-sized businesses.
5. @ServerVirt_TT: Get the best server virtualization info from the site editor of SearchServerVirtualization.com, Colin Steele.
6. @brianmadden: He’s a virtualization blogger providing updates on virtualization software and services as well as links to his latest blog posts.
7. @govirtual: Virtualization.info’s official Twitter account.
8. @scott_lowe: He’s an IT professional who specializes in virtualization and VMWare who’s very active with his followers. Send him a question!
9. @ThatFridgeGuy: Rod Gabriel sends out bite-sized insights on the IT world.
Then of course, there’s always us: Follow ITKE on Twitter for a new IT relevant topic each month. Who are some of your favorite virtualization pros to follow on Twitter? Let us know in the comments section!
While VMWare’s slimmed down, bare-metal server virtualization product ESXi has caused some amount of consternation among certain fans of the Linux console that its older, slightly heftier brother ESX offers, the lightened requirements have allowed some pretty cool virtualization tricks, as SearchServerVirtualization’s Alex Barrett reported:
IT professionals on virtualization’s bleeding edge have begun to experiment with a new method of deploying a hypervisor to a server: installing a slimmed-down hypervisor such as VMware ESXi on a USB flash drive or secure digital (SD) card.
“People will need to get used to a more ESXi way of doing things,” said Simon Seagrave, the author of the virtualization TechHead blog. He uses the slimmed-down ESXi in his home lab and has grown fond of it.
Virtual-environment-on-a-stick! If you want to play along at home, VMInfo offers a handy, full-color PDF guide (Warning: PDF) while the popular virtualization blog Yellow-Bricks offers a handy guide that breaks down putting VMWare onto a USB drive into 13 (relatively) easy steps.
Aside from simply sounding cool, virtualization is the most efficient way to, well, be efficient: Server virtualization cuts down on how many servers you have laying around, desktop virtualization can cut helpdesk overhead and network virtualization helps keep it all organized without the messy wires getting in the way. Interested in having your cake and eating it too? Then check out these helpful blogs. Care to share your own virtualization goodies? Let us know in the IT Knowledge Exchange forums and hear where other IT professionals get their news, or e-mail me directly and we can add your suggestions to our list.
From the Inside: Editorial & User Blogs
- Search Server Virtualization: Search Server Virtualization editors outline how industry changes and announcements affect how your company uses virtualization.
- Virtualization Pro: SearchVMWare editors provide the grittier bits of the business side along with resources you need to check out.
- Irregular Expressions: User Dan O’Connor reveals vulnerabilities and exploits while sharing virtualization tips and tricks.
From the Outside
- VMBlog: David Marshall keeps track of all things virtualization so you don’t have to.
- Virtualization.Info: Virtualization news headlines compiled and explained in one place.
- AMD’s Virtualization Blog: Best practices for products and services offered by AMD.
- Scott Lowe’s Blog: Contributor to SearchVMware and SearchServerVirtualization, here he blogs about his personal and professional IT experiences, focusing on virtualization, storage and servers.
- Virtual Lifestyle: Joep Piscaer has the virtual cred to back him up; he’s received the highest of recognitions from VMware for his work.
- Virtualize: Martijn Lohmeijer takes us through the steps of implementing VMWare Virtual Infrastructure. Learn what to expect and avoid from his firsthand experience.
This came out in April, but I missed it until it came through over the NAISG mailing list: CBS did a nice investigative piece on how much confidential, legally protected information was set free via unwiped copy machines, many of which keep copies of all the copies they’ve ever made, easily accessible by secondary-market buyers with the right technical know-how.
[kml_flashembed movie="http://www.youtube.com/v/iC38D5am7go" width="425" height="350" wmode="transparent" /]
Peter from FDCServers just left a voicemail response to the IT Watch Blog’s questions about the alleged pictures circulating of FDCServers’ server environment, saying that yes, the pictures were of FDCServers’ servers, but most of them weren’t from the colocated data center they host, but rather the NOC and repair centers where FDCServers fixes up whatever ails your favorite boxen. Furthermore, he said the pictures were taken by a competitor, which would obviously imply some ulterior motives.
Don’t believe him? Go see for yourself: He invited interested customer or potential customers to tour the data center, an invitation that has generally been extended by FDCServers. Interested in taking him up on it? Try asking via their contact page.
The transcription of his full voicemail:
Hi Michael, this is Peter at FDCServers. You left a voicemail asking about the pictures that are circulating the Internet that are supposed to represent our data center. Some of them, some of them are not. Some of them are, some of them are not. Basically does pictures were taken by person that works for a competitor. Most of the pictures that he snapped are actually parts and machines that are sitting in our work benches or workshops in the NOC area where our guys basically repair servers. All of our data centers are actually accessible for tours, so obviously if that was the true state of our facilities then nobody would be hosting with us. On the contrary, the Chicago warehouse over 6,000 servers and our Chicago facility is sold out due to demand. As you can imagine just for the fact that those pictures are taken by somebody who works for a competitor, they were taken out of context and they do not represent you know the state of our facility. There was a picture of some ducked tape drives was a machine that a customer sent over and those drives were sitting on a workbench, and those pictures were not of working environment of FDCServers. As I said, anybody is welcome to come take a tour to see for themselves to see if those pictures represent the true state of our facility or not. If you have any other questions feel free to give me a call at [redacted]. Thanks a lot. Have a nice day.
Update: FDCServers responds, saying a) yes, those pictures are of servers in our building; b) they are of our NOC and our repair center, and are not in production environment; c) they were taken by a competitor. Read the full response here.
I went to the FDC Datacenter around the end of October after my friend who has a colocated server there asked me to help bring it back online. Not a dedicated server, a colocated server. It had been down for roughly 10 hours, and FDC’s technicians had no idea what was wrong with it. The machine wouldn’t post, and the fdc techs insisted my friend (who lives several states away) would need to supply replacement hardware to get the machine to post. After talking to him for several hours, and assuming fdc’s techs themselves were right, my friend insisted that he would pay me to go to the datacenter and troubleshoot the issue.
The problem turned out to be a busted PCI-E bus, but the colocation facility hosted servers holding the power supply in with ducktape, ports completely askew and some downright bushy wiring. I called FDCServers to see if they’d confirm it was, indeed, their location, but they haven’t returned my message yet.
They do, however, host their own gallery of data center pictures, which show a generally more organized side of things:
So are cardboard cases standard operating procedure for colocation facilities? Have you seen your own data center hacks that trump these? Shouldn’t someone call the fire marshal? I’d love to hear your thoughts on these pressing questions at Michael@ITKnowledgeExchange.com: I’ll happily keep your information private if requested, and we’ve got a free T-Shirt for the best data center hack we come across.
If doing it right is not a popular option since many organizations are hell-bent on doing it wrong, let’s try to determine “What is the least wrong way which will actually get used in real-life?”
•WP1: Skip need determination step altogether – just buy something
–“My boss said that we need a correlation engine” (more about this mistake)
–“I know this guy who sells log management tools …”
•WP2: Define the need for SIEM in general
–“We need, you know, ‘do SIEM’ and stuff”
These situations are actually quite common and most unquestionably wrong; and many a SIEM project has been slaughtered as a result.
BTW, what partially inspired this post was a lot of Google queries for “which siem system is right for security in my company?” that landed on my blog. Think about this! Folks think that Google actually knows what SIEM is best for their organizations An additional inspiration was provided by a discussion I had with a colleague who said that many SIEM purchases also had a hidden “opportunity cost.” Namely, the money spent on a SIEM were thus not spent on something that could have contributed a lot more to risk reduction at this particular organizations. The final inspiration came from all the “MARS tossing” that is going on now; the organizations who acquired a SIEM product a few years ago and never managed to apply it to anything useful are now on the market for – you guessed it! – a new SIEM. These same folks then google for “SIEM justification” since they literally cannot say why they wasted $280,000 of perfectly good dollars…
In any case, what IS the least wrong way? How about this flow (drastic oversimplification alert!):
- Do you really need a SIEM? Or do you want a SIEM? Figure this one out please….
- If you need a SIEM to solve a particular problem, what would it cost (time, staff time, money) to solve it with SIEM and without SIEM? Which is cheaper, better, faster?
- What problems won’t you solve due to engaging in a multi-month SIEM project? Is this acceptable?
- Next, will a simpler – and cheaper!- log management tool do the trick?
- Are existing SIEM solutions actually capable to solving that problem you have? At a cost you can afford to pay?
- Will existing SIEM solutions work in your organizations: politically, culturally, geographically, etc?
- Are you prepared to WORK (yes, w-o-r-k!) to make SIEM solve your problem? What exactly is your expectation, SOC-in-a-box, perchance?
- How about open source SIEM combined with other tools and integration services?
- Only here you can start planning the deployment, phased approach, log source integrations, correlation rules, dashboards, etc.
(we can call it an “almost right” approach)
And by all means, study vendor stuff on “how to choose a SIEM?” [some of it will in fact be written by the same party as this post :-)], but don’t take it as gospel. The above list should get you going at least.
Here are some example of “SIEM gone wild” from recent experience.
In one case, a company called a consultant and said that they needed assistance with SIEM implementation. He asks: do you have business requirements defined? No. Do you have a product selected? No. But you want to implement already? Yes. *painful pause*
In another case, a company picked a SIEM that was [supposedly] the easiest to deploy. While undoubtfully an important criteria, wouldn’t an enlightened reader of my blog agree that this requirement comes a close SECOND right after the “Will it solve my security problem?!!!” This particular organization just focused on ease of deployment… and FAIL didn’t have to wait too long
BTW, lately I’ve been puzzled about the whole concept of “co-managed SIEM” (subject of one of the future blog posts). I think it is gaining popularity (example) for that very reason mentioned in this post: folks don’t want to figure that stuff out, the want the crack team of mercenaries to parachute in, deploy and operationalize a SIEM for them – and then continue running it for some time…or forever. I was told that sometimes it is cheaper than signing up for an MSSP – and you retain more control while learning from the experts on how to do it. But more on this in the future post.
Finally, just have to mention it: I am available for SIEM and log management consulting projects.
TANSTAAFL, or “There ain’t so such thing as a free lunch,” was once the rally cry of both science fiction great Robert Heinlein and economics giant Milton Friedman, so it should come as no surprise that even as Microsoft promises 90% energy savings, as it did in a recent campaign on sister site SearchSQLServer.com (see above), there’s a catch. In this case, as spelled out by SearchSQLServer’s Alex Barrett (also pictured above), that catch is Microsoft cracking down on SQL Server licensing fee loopholes, and not everyone is pleased:
IT organizations that virtualize Microsoft SQL Server may have to rethink their strategy with the upcoming SQL Server 2008 R2, which dramatically increases the price of running virtualized instances of the database in some configurations.
But whereas Enterprise Edition used to offer unlimited virtualization rights if all of the processors in the system were licensed, Enterprise Edition for R2 only supports up to four virtual instances. To gain unlimited virtualization rights, customers will need to purchase the new Datacenter Edition, which costs twice as much as Enterprise Edition: $54,990 per processor (without Software Assurance), compared to $27,495.
Scott Cochran, network engineer at a large life insurance company in Baltimore, Md., said his company’s plans to virtualize SQL Server Enterprise Edition were probably “off the table.”
“We were having a hard time getting management to sign off on two processors at the old pricing,” Cochran said.
Microsoft responded that the changes will affect only “a very few customers,” but Brent Ozar, a noted SQL Server professional and avid blogger, called the license update “an ugly change for shops who use virtualization,” recommending SQL Server administrators buy now if they can avoid the potential licensing headaches and costs later on.
Our resident SQL Server expert MrDenny has not yet weighed in (Update: He has, however, commented extensively on SQL Server 2008 R2), but in a recent talk on SQL Server virtualization, he highlighted cut hardware costs and free server redundancy as drivers for SQL virtualization in a recent talk for SQL Saturday. While Enterprise edition users will still see both of these benefits, the scope of what they can do has now been significantly limited.