A week ago, after coming across some rather graphic server room pictures (SFW unless your office believes in the sanctity of server equipment), I asked the community for its favorite server room DIY or horror stories, promising the winner fame and fortune (or at least some free swag). Well, we have a winner: IT Knowledge Exchange member Batye’s tale of a Russian server room that might’ve stumped the Hardy Boys. As thanks, we shipped Batye free copies of:
This went out without any apparent cause: Total breakdown in communication, only to return again later. It wasn’t a virus, and all the hardware seemed to be fully functional, until it completely disappeared.
The facility was quite modern: Secure parking, video security, armed guards with AKM’s and Saigas. The office even required passing a facial recognition scan to access the facilities. After having a specialized name badge made up, complete with his photo and a unique barcode, he was given access to the server room, the epicenter of the irregular crashes.
The scene made our leaked server room pics look like a networking Taj Mahal: The server room was a refinished washroom/utility closet. Nobody had bothered to put in server racks, leaving the high-end equipment and air conditioner to sit directly on the floor amid moldy walls and leaky pipes.
The problem quickly became apparent: Anytime someone flushed the toilet, water (one prays) from the leaking pipes got on the UPS. The UPS’ surge protection would kick in, automatically powering itself down along with the servers connected to it, hence the disappearing pings.
A few hours later, the air conditioner would dry up the flushed water and the UPS would reset itself, and all would be more or less right with the world again – until someone else needed to go.
Blame the setup on new fangled IT theories: The server room was managed by a 17-year-old kid, the son of one of the company’s executives.
If you want some 140-character goodness all about virtualization, check out these accounts to keep track of what’s going on.
1. @virtnews: Get the latest news on virtualization software from the top virtualization blogs across the internet including VMware, SearchServerVirtualization, and Virtualization.com, conveniently organized on their webpartner site.
2. @VMware: Receive updates on events, products, and troubleshooting.
3. @vmwarecares: Ask your questions about VMware and get answers from the community and experts. Their profile boasts “human assistance”; can’t beat that!
4. @SMBVirtualization: Everything you could possibly need aside from the hardware, this feed is chockfull of white papers, blog posts, videos and instructional material geared specifically toward small- to medium-sized businesses.
5. @ServerVirt_TT: Get the best server virtualization info from the site editor of SearchServerVirtualization.com, Colin Steele.
6. @brianmadden: He’s a virtualization blogger providing updates on virtualization software and services as well as links to his latest blog posts.
7. @govirtual: Virtualization.info’s official Twitter account.
8. @scott_lowe: He’s an IT professional who specializes in virtualization and VMWare who’s very active with his followers. Send him a question!
9. @ThatFridgeGuy: Rod Gabriel sends out bite-sized insights on the IT world.
Then of course, there’s always us: Follow ITKE on Twitter for a new IT relevant topic each month. Who are some of your favorite virtualization pros to follow on Twitter? Let us know in the comments section!
While VMWare’s slimmed down, bare-metal server virtualization product ESXi has caused some amount of consternation among certain fans of the Linux console that its older, slightly heftier brother ESX offers, the lightened requirements have allowed some pretty cool virtualization tricks, as SearchServerVirtualization’s Alex Barrett reported:
IT professionals on virtualization’s bleeding edge have begun to experiment with a new method of deploying a hypervisor to a server: installing a slimmed-down hypervisor such as VMware ESXi on a USB flash drive or secure digital (SD) card.
“People will need to get used to a more ESXi way of doing things,” said Simon Seagrave, the author of the virtualization TechHead blog. He uses the slimmed-down ESXi in his home lab and has grown fond of it.
Virtual-environment-on-a-stick! If you want to play along at home, VMInfo offers a handy, full-color PDF guide (Warning: PDF) while the popular virtualization blog Yellow-Bricks offers a handy guide that breaks down putting VMWare onto a USB drive into 13 (relatively) easy steps.
Aside from simply sounding cool, virtualization is the most efficient way to, well, be efficient: Server virtualization cuts down on how many servers you have laying around, desktop virtualization can cut helpdesk overhead and network virtualization helps keep it all organized without the messy wires getting in the way. Interested in having your cake and eating it too? Then check out these helpful blogs. Care to share your own virtualization goodies? Let us know in the IT Knowledge Exchange forums and hear where other IT professionals get their news, or e-mail me directly and we can add your suggestions to our list.
From the Inside: Editorial & User Blogs
- Search Server Virtualization: Search Server Virtualization editors outline how industry changes and announcements affect how your company uses virtualization.
- Virtualization Pro: SearchVMWare editors provide the grittier bits of the business side along with resources you need to check out.
- Irregular Expressions: User Dan O’Connor reveals vulnerabilities and exploits while sharing virtualization tips and tricks.
From the Outside
- VMBlog: David Marshall keeps track of all things virtualization so you don’t have to.
- Virtualization.Info: Virtualization news headlines compiled and explained in one place.
- AMD’s Virtualization Blog: Best practices for products and services offered by AMD.
- Scott Lowe’s Blog: Contributor to SearchVMware and SearchServerVirtualization, here he blogs about his personal and professional IT experiences, focusing on virtualization, storage and servers.
- Virtual Lifestyle: Joep Piscaer has the virtual cred to back him up; he’s received the highest of recognitions from VMware for his work.
- Virtualize: Martijn Lohmeijer takes us through the steps of implementing VMWare Virtual Infrastructure. Learn what to expect and avoid from his firsthand experience.
This came out in April, but I missed it until it came through over the NAISG mailing list: CBS did a nice investigative piece on how much confidential, legally protected information was set free via unwiped copy machines, many of which keep copies of all the copies they’ve ever made, easily accessible by secondary-market buyers with the right technical know-how.
[kml_flashembed movie="http://www.youtube.com/v/iC38D5am7go" width="425" height="350" wmode="transparent" /]
Peter from FDCServers just left a voicemail response to the IT Watch Blog’s questions about the alleged pictures circulating of FDCServers’ server environment, saying that yes, the pictures were of FDCServers’ servers, but most of them weren’t from the colocated data center they host, but rather the NOC and repair centers where FDCServers fixes up whatever ails your favorite boxen. Furthermore, he said the pictures were taken by a competitor, which would obviously imply some ulterior motives.
Don’t believe him? Go see for yourself: He invited interested customer or potential customers to tour the data center, an invitation that has generally been extended by FDCServers. Interested in taking him up on it? Try asking via their contact page.
The transcription of his full voicemail:
Hi Michael, this is Peter at FDCServers. You left a voicemail asking about the pictures that are circulating the Internet that are supposed to represent our data center. Some of them, some of them are not. Some of them are, some of them are not. Basically does pictures were taken by person that works for a competitor. Most of the pictures that he snapped are actually parts and machines that are sitting in our work benches or workshops in the NOC area where our guys basically repair servers. All of our data centers are actually accessible for tours, so obviously if that was the true state of our facilities then nobody would be hosting with us. On the contrary, the Chicago warehouse over 6,000 servers and our Chicago facility is sold out due to demand. As you can imagine just for the fact that those pictures are taken by somebody who works for a competitor, they were taken out of context and they do not represent you know the state of our facility. There was a picture of some ducked tape drives was a machine that a customer sent over and those drives were sitting on a workbench, and those pictures were not of working environment of FDCServers. As I said, anybody is welcome to come take a tour to see for themselves to see if those pictures represent the true state of our facility or not. If you have any other questions feel free to give me a call at [redacted]. Thanks a lot. Have a nice day.
Update: FDCServers responds, saying a) yes, those pictures are of servers in our building; b) they are of our NOC and our repair center, and are not in production environment; c) they were taken by a competitor. Read the full response here.
I went to the FDC Datacenter around the end of October after my friend who has a colocated server there asked me to help bring it back online. Not a dedicated server, a colocated server. It had been down for roughly 10 hours, and FDC’s technicians had no idea what was wrong with it. The machine wouldn’t post, and the fdc techs insisted my friend (who lives several states away) would need to supply replacement hardware to get the machine to post. After talking to him for several hours, and assuming fdc’s techs themselves were right, my friend insisted that he would pay me to go to the datacenter and troubleshoot the issue.
The problem turned out to be a busted PCI-E bus, but the colocation facility hosted servers holding the power supply in with ducktape, ports completely askew and some downright bushy wiring. I called FDCServers to see if they’d confirm it was, indeed, their location, but they haven’t returned my message yet.
They do, however, host their own gallery of data center pictures, which show a generally more organized side of things:
So are cardboard cases standard operating procedure for colocation facilities? Have you seen your own data center hacks that trump these? Shouldn’t someone call the fire marshal? I’d love to hear your thoughts on these pressing questions at Michael@ITKnowledgeExchange.com: I’ll happily keep your information private if requested, and we’ve got a free T-Shirt for the best data center hack we come across.
If doing it right is not a popular option since many organizations are hell-bent on doing it wrong, let’s try to determine “What is the least wrong way which will actually get used in real-life?”
•WP1: Skip need determination step altogether – just buy something
–“My boss said that we need a correlation engine” (more about this mistake)
–“I know this guy who sells log management tools …”
•WP2: Define the need for SIEM in general
–“We need, you know, ‘do SIEM’ and stuff” 🙂
These situations are actually quite common and most unquestionably wrong; and many a SIEM project has been slaughtered as a result.
BTW, what partially inspired this post was a lot of Google queries for “which siem system is right for security in my company?” that landed on my blog. Think about this! Folks think that Google actually knows what SIEM is best for their organizations 🙂 An additional inspiration was provided by a discussion I had with a colleague who said that many SIEM purchases also had a hidden “opportunity cost.” Namely, the money spent on a SIEM were thus not spent on something that could have contributed a lot more to risk reduction at this particular organizations. The final inspiration came from all the “MARS tossing” that is going on now; the organizations who acquired a SIEM product a few years ago and never managed to apply it to anything useful are now on the market for – you guessed it! – a new SIEM. These same folks then google for “SIEM justification” since they literally cannot say why they wasted $280,000 of perfectly good dollars…
In any case, what IS the least wrong way? How about this flow (drastic oversimplification alert!):
- Do you really need a SIEM? Or do you want a SIEM? Figure this one out please….
- If you need a SIEM to solve a particular problem, what would it cost (time, staff time, money) to solve it with SIEM and without SIEM? Which is cheaper, better, faster?
- What problems won’t you solve due to engaging in a multi-month SIEM project? Is this acceptable?
- Next, will a simpler – and cheaper!- log management tool do the trick?
- Are existing SIEM solutions actually capable to solving that problem you have? At a cost you can afford to pay?
- Will existing SIEM solutions work in your organizations: politically, culturally, geographically, etc?
- Are you prepared to WORK (yes, w-o-r-k!) to make SIEM solve your problem? What exactly is your expectation, SOC-in-a-box, perchance?
- How about open source SIEM combined with other tools and integration services?
- Only here you can start planning the deployment, phased approach, log source integrations, correlation rules, dashboards, etc.
(we can call it an “almost right” approach)
And by all means, study vendor stuff on “how to choose a SIEM?” [some of it will in fact be written by the same party as this post :-)], but don’t take it as gospel. The above list should get you going at least.
Here are some example of “SIEM gone wild” from recent experience.
In one case, a company called a consultant and said that they needed assistance with SIEM implementation. He asks: do you have business requirements defined? No. Do you have a product selected? No. But you want to implement already? Yes. *painful pause*
In another case, a company picked a SIEM that was [supposedly] the easiest to deploy. While undoubtfully an important criteria, wouldn’t an enlightened reader of my blog agree that this requirement comes a close SECOND right after the “Will it solve my security problem?!!!” This particular organization just focused on ease of deployment… and FAIL didn’t have to wait too long 🙂
BTW, lately I’ve been puzzled about the whole concept of “co-managed SIEM” (subject of one of the future blog posts). I think it is gaining popularity (example) for that very reason mentioned in this post: folks don’t want to figure that stuff out, the want the crack team of mercenaries to parachute in, deploy and operationalize a SIEM for them – and then continue running it for some time…or forever. I was told that sometimes it is cheaper than signing up for an MSSP – and you retain more control while learning from the experts on how to do it. But more on this in the future post.
Finally, just have to mention it: I am available for SIEM and log management consulting projects.
TANSTAAFL, or “There ain’t so such thing as a free lunch,” was once the rally cry of both science fiction great Robert Heinlein and economics giant Milton Friedman, so it should come as no surprise that even as Microsoft promises 90% energy savings, as it did in a recent campaign on sister site SearchSQLServer.com (see above), there’s a catch. In this case, as spelled out by SearchSQLServer’s Alex Barrett (also pictured above), that catch is Microsoft cracking down on SQL Server licensing fee loopholes, and not everyone is pleased:
IT organizations that virtualize Microsoft SQL Server may have to rethink their strategy with the upcoming SQL Server 2008 R2, which dramatically increases the price of running virtualized instances of the database in some configurations.
But whereas Enterprise Edition used to offer unlimited virtualization rights if all of the processors in the system were licensed, Enterprise Edition for R2 only supports up to four virtual instances. To gain unlimited virtualization rights, customers will need to purchase the new Datacenter Edition, which costs twice as much as Enterprise Edition: $54,990 per processor (without Software Assurance), compared to $27,495.
Scott Cochran, network engineer at a large life insurance company in Baltimore, Md., said his company’s plans to virtualize SQL Server Enterprise Edition were probably “off the table.”
“We were having a hard time getting management to sign off on two processors at the old pricing,” Cochran said.
Microsoft responded that the changes will affect only “a very few customers,” but Brent Ozar, a noted SQL Server professional and avid blogger, called the license update “an ugly change for shops who use virtualization,” recommending SQL Server administrators buy now if they can avoid the potential licensing headaches and costs later on.
Our resident SQL Server expert MrDenny has not yet weighed in (Update: He has, however, commented extensively on SQL Server 2008 R2), but in a recent talk on SQL Server virtualization, he highlighted cut hardware costs and free server redundancy as drivers for SQL virtualization in a recent talk for SQL Saturday. While Enterprise edition users will still see both of these benefits, the scope of what they can do has now been significantly limited.
Too many desktop virtualization platform options, deployment choices and licensing paths might stimmying desktop virtualization. As CIO’s Kevin Fogarty reports:
Desktop virtualization now comes in so many varieties that even vendors confuse terms referring to the flavors.
Market leader Citrix Systems, now working hard to expand virtual desktops into roles that the company hasn’t traditionally filled, rolled out a version of its Xen Desktop solution last fall that allowed customers to choose any of six major delivery methods.
Competitor VMware is close behind, followed by Microsoft and a host of add-on vendors and open-source integrators offering similar approaches, bolstered from the other end of the client-hardware spectrum by thin- or zero-client virtualization products such as Pano Logic or NComputing.
Add to that the potential to stream apps to end users from external SaaS providers, access all or part of a virtual desktop from the cloud via platform-as-a-service, nestle a secure VM within an otherwise insecure personally owned iPad, smartphone or other gadget—and the choice gets very complicated, according to Chris Wolf, infrastructure and virtualization analyst at The Burton Group.
And while the rush into virtualized desktop and device OS’s isn’t fully on yet (some of Fogarty’s aforementioned combinations aren’t even available or projected to be available), desktop virtualization was gathering its fair share of buzz at Interop, the somewhat stodgy IT conference geared towards networking professionals I headed out to last week. The conference even had a whole track day devoted to desktop virtualization. Just sample the topic list and you can tell this is a serious technology:
- Methodology for evaluating your user requirements and determining which technologies and combined approaches are best suited for your users’ needs
- A deeper understanding of the various approaches and architectures of the different desktop virtualization and application virtualization and streaming technologies
- An independent look at the top desktop and application virtualization solutions