Oct 2 2007 12:59PM GMT
Posted by: admin
disaster recovery,
Backup & recovery,
Hardware issues,
Linux Done Right,
Administration, interoperability and integration
Consider this the second in an occasional, meandering series of articles on Linux done right. These aren’t meant to boost the sales of any particular vendor, but instead are meant to show other end users, IT managers and decision makers what to look for when vetting applications and operating system migrations. It can be support, migrations strategies, execution or anything and everything in between. If it’s Linux done right, then you’ll find it here.
Matthew Porter, the CEO of
Contegix, is an anomaly as far as I’m concerned–and I don’t mean that in a negative way whatsoever.You see, Contegix, a managed hosting provider based in St. Louis, Mo., is a 100% Linux shop. Every server they run internally has Red Hat Enterprise Linux 3, 4 or 5 installed (although they’re not using Xen just yet), and all their applications, save a financial/payroll application that just
has to run on Windows as a virtual instance in VMware, runs on Linux.OK, so that makes them a 99% Linux shop with a vestigial Microsoft Windows appendix, and I apologize. In an industry that holds sacred the
“five nine’s,” I think you can give me some slack on this one.
Anyway, outside of European universities and some HPC instances, 100% Linux shops are a rare breed in this heterogeneous operating system mishmash of a world we live in today. But that still hasn’t stopped Contegix. In a call last week, Porter told me that business is going well and growing fast. So fast, in fact, that Porter called what’s happened over the past few months “explosive.””We’ve grown 10% every month over the past couple of years,” he said. “Today it’s more like 14%.”
I called Contegix an anomaly, but their story isn’t all the surprising when you look at Linux growth over the same period of time. Everyone from Gartner to IDC to our friends at Saugatuck have pegged 2009-2011 or thereabouts as the magic year where Linux takes an approximate 50% share of all mission critical operations in the enterprise. That’s not edge of enterprise stuff in addition to mission critical, either–it’s bare bones “if this messes up then our business suffers” stuff.
But that’s all in the amorphous soup of the far future. Contegix was an all Linux shop now, and with all of that growth over the past few quarters, it was starting to experience what can only be described as growing pains. Legacy software and a surging pile of user data that grew every month were taxing the system and tying up resources for days at a time, Porter said.
Their old backup solution, Arkeia, worked well for about a year, Porter said, but couldn’t scale and Contegix was spending 40+ hours per week managing backups and recoveries.
“The problem we were dealing with was that we were working around the limitations of our previous software,” Porter said. “It often took 24 hours to backup the index that the software was using.” Sometimes that 24-hour estimate was being generous, and the backup took longer (some recovery or file system-related efforts were eating up 42 or more hours a clip). “When a customer needed some stored, even if it was just a 65 meg file or a database or whatever, it may have taken and hour just to restore that. And we were storing about 50 terabytes a month,” he said.
As Contegix continued to grow, speeding up the backup and recovery time would become a top priority going forward.
Looking for options, thinking of Linux
A Linux shop should expect a certain degree of Linux respect and understanding, right? Contegix’s case was no exception. From the onset, Porter and his team sought out vendors who could provide recovery and back up peace of mind with a Linux twist, no questions asked. They had to, because Porter wasn’t about to spend even more money to retrain his staff on Windows or SQL Server.
“We have a lot of Postgres and MySQL, so it was critical to have hot backup plug-ins for those databases … [and] we had literally no technical staff that used Windows as a desktop. We didn’t want to learn SQL Server,” he said.
Those strict specifications hurt the first candidate, Oceanport, N.J.-based CommVault, right out of the gate. With CommVault’s offering, called Simpana, Porter said his staff was asked to learn SQL Server. “Given the ownership costs, CommVault had higher costs of ownership,” Porter said.
Nor did CommVault offer support for MySQL or PostgreSQL. Contegix was also unable to test the application because CommVault wanted a signed PO first. No deal.
The next solution came from Symantec, which Porter and some of the Contegix team had had some experience with at a previous company. From what Porter told me, things didn’t go well even with the prior encounter serving as a foot in the door. Again, the hangup arrived because of how Contegix viewed the vendor’s approach to Linux, Linux support and testing.
“[Symantec weren't as nimble in evaluation process as they could have been. It took two months to get a quote, but there was still no demo unit. The installation process was too costly. The there was the Linux dynamic. The reseller we went through basically said 'we only sell for Windows, but we can do Linux after we get approval for Linux.'," Porter said. "It kind of felt like they fully supported [Linux], but not fully at all.”
Symantec’s application, NetBackup, was also out of Contegix’s price range, and they were worried about the potential management hours they would have to spend on NetBackup.
Cue the Price is Right “you lose” gong sound.
Finding some Linux spine
Rounding out a trio of back up and recovery options was BakBone Software, a backup and recovery vendor based in San Diego. Interestingly enough, the trait that immediately stuck out in Porter’s mind about his experiences with BakBone wasn’t technical, it was support and sales-related.
“The same sales rep we dealt with in the beginning was there a year later. Sometimes when you see a lot of turnover the reps don’t really believe in the product, or it’s not selling, but that obviously wasn’t the case,” he said.
The came the point on which many Linux and open source software relationships are made or broken: support. How does it fare? Is it what you’ve become accustomed to over the years? Is it better? Is it completely different? Is it professional?
In Porter’s case, he asks similar questions, but he also has a test of his own that’s been generated from Contegix’s own support practices. “[As a managed hosting provider] we always have support staff on hand at all times 24/7/365, and we answer every ticket in five minutes. We assign an engineer to that ticket, not some sales rep or whatever. When an organization like ours is built around support as the number one feature, then vendors must have that same mentality,” he said.
Long story short, BakBone did support MySQL and Postgres, and the handful of other applications on hand like Ruby on Rails and Java, and it allowed testing and the price point was right, so Porter bought into NetVault: Backup 8.0.
The server implementation took less than a day, and today Contegix has migrated about 98% of its Arkeia servers over to NetVault. In twenty more days, Porter expects the migration to be complete.
“The consolidation was was a huge benefit for us. They can do full consolidation or a synthetic one. The second big draw for us is the not just the consolidation is that there, it is the fact that we have great independent restore time, that’s fast and a great way to back up our catalogue and index,” Porter said. “We do a lot of back up to a fiber channel SAN. With NetVault, we could mount our SAN in drivesafe just like Oracle does, so that the load can be shared among back end servers and multiple backups and clients. Literally, we have three or four servers that just perform backup.”
For Contegix, the ability to share media and have those multiple backup servers is “ubelieveably smart,” Porter said. “We were spending so much time writing custom scripts to work with the ODL system before and many of those were already features in BakBone,” Porter said.
Indeed, before the third party backup and recovery app was introduced to the Contegix back end environment, the IT staff was wasting a good 100-150 hours per month on those customer scripts. But not anymore.
Like I wrote earlier, the migration off legacy is about 98% done. Something could still go wrong, I suppose, but that’s not the feeling I got when talking with Porter. From the sounds of things this shop will remain a Linux-only club for the indefinite future.
Have a Linux Done Right success story you’d like to share? Send it to me at Jack Loftus, News Writer and I guarantee I’ll get you the 15 minutes of IT fame you so richly deserve.
0 Comments
RSS Feed
Email a friend
Good job to his systems administrators for getting LT back up and running so quickly.
