Storage

Storm Preparations

As Hurricane Fay approaches the Florida today, we’re rapidly making plans to insure our Sarasota office is prepared. This is something of a new twist for us, because Upstate NY companies typically don’t have to worry too much about hurricanes. One thing our preparations have highlighted is our data needs, and once again how insatiable our demand for storage space can be in Advertising.

To give you a better idea of what this means, here are some actual examples. We have a client folder in our Sarasota office which contains the current advertising files which are currently under production. This folder is about 85GB. While this might not seem like a huge volume if you’re a large company, I would argue that it’s a good size folder for an office of less than 20 people. We have one office which currently has no Creative staff, and the client folders in that office are less than 10 GB in size. Again, these examples are from our smaller offices. The file size dilemma becomes larger when you look at us as a whole and include our larger offices in the equation. I’ve had vendors tell me that we’re more like a 3,000 employee company when you consider our storage needs, and we currently have about 275 employees.

Copying 85GB across our WAN in order to have a live copy of our Sarasota clients folders is not an option for today. However, we are looking at tools for replication and data deduplication which might allow us do just that. For now, we’ll have to settle on backing things up the old-fashioned way, using tapes and off-site storage. It works, but it would certainly be nice if I had an up-to-date copy of my Sarasota client folders sitting here in Syracuse, ready to be brought to life if necessary. It’s one of our goals to have such a system in place in the next year or so.

Archiving Creative Materials

One of the more difficult challenges we face is that of archiving our creative materials. We’re currently taking a fresh look at this process, reviewing both the process in terms of who is doing what, but also taking a fresh look at the storage media we’re using.

Our current process involves burning all of our closed jobs to DVDs. In our main office alone, we routinely archive up to 20 DVDs (approximately 100 GB) of material every month, and we have 6 other offices all facing the same issue. Burning the DVDs is actually only part of the process, although it may be the most man-hour and time intensive part.

The first step involves making a final determination about when a job is complete, and this is a step which has to be done by the Creative Department. They are ultimately the owners of this data, and once a job has been tagged as final, it is moved to a read-only network location where it remains until it gets burned to DVD. How long it remains on the network depends on our current remaining disk space, but we can usually maintain 2-3 months on our network. This is very helpful, because most requests for archived jobs occur in this time span. Having he materials readily available on the network saves us from having to retrieve from an offline DVD.

Once we need to reclaim disk space, we then begin the process of burning this material to DVDs. First, we have to break the data up into DVD-size chunks. Then, we burn it onto DVD, verifying against the original. Finally, we duplicate the DVD, catalog the DVD using DiskTracker, and send the duplicate DVD to one of our other locations for permanent offsite storage. The DiskTracker catalog is shared on our network for all users to search. This part of the process requires both time and manpower, up to 2 days per month. It’s also contentious in terms of who actually does it – in some of our offices we utilize IT personnel while in others it’s done by Creative or Studio.

Over the years we’ve constantly looked for ways to streamline and improve the process, including jukebox-based systems, but we’ve usually rejected most systems because of the expense of adding them to all of our locations.

At the moment we’re actually contemplating replacing our DVDs with hard drive based storage. We could store at least 6 months of archive materials on a 500 GB hard drive. The amount of time we could save using a hard drive based system would save us a tremendous amount of time, but it also raises a lot of issues – including costs, reliability, susceptibility to viruses, and the potential for losing more data in a single shot. Personally, it seems like most of those issues can be resolved, and we’re going to test a hard drive based solution over the coming months.

We’ll see how it goes, but it will be very interesting to see how we’re archiving and what we’re using 5-10 years from now.

Paying Attention to Pricing Models

Over the last several days, I’ve gotten pretty worked up over some vendor pricing models. I understand that vendors are attempting to recoup the investment they have in developing their products, but unfortunately the pricing model vendors may choose to adopt can hit you particularly hard. It’s opened my eyes to this particular issue, and I’m going to pay a whole lot more attention to it going forward.

How would we like it if Cisco or HP suddenly decided to adopt a new pricing model for their network switches? What if this new pricing model required that we had to pay more based on the volume of data moving through the switch? This model would be prohibitively expensive for those of us who routinely move big Photoshop files around. It also doesn’t make a lot of sense, because it’s the same switch whether you’re moving around Word documents or Photoshop files. Thankfully, things don’t work that way in the switch world. You purchase the switch you need, and pricing varies depending on the switch’s speed and the number of ports. The pricing model makes sense.

This brings me to a couple of vendors who apply a different model in order to recoup their development costs. For the past several months I’ve been in discussions with Bakbone about their backup products. Bakbone uses a pricing model where you’re asked to pay based on the volume of data you’re backing up. We’ve also purchased network acceleration devices from Expand Networks in the past 12 months. In Expand’s case, the pricing model is based on the amount of bandwidth being accelerated. In both cases, the smaller company which has large storage and bandwidth needs is penalized. The pricing model works a whole lot better if you’re saving and moving Word documents than it does for Photoshop documents.

In the case of backup software, if I’ve already spent the money for the drive space to store my backups, why on earth should I have to pay a software vendor based on how much data I’m backing up? Does the software have to work any harder in order to back up more stuff? In the case of network acceleration, if I’ve already purchased a piece of hardware which is sufficient for my bandwidth needs, why do I have to pay more to accelerate 20MBps than I do to accelerate 10MBps? The hardware hasn’t changed. I’m already paying my ISP for the bandwidth. Isn’t the whole idea of a network acceleration device that I get more bandwidth for the buck? It feels to me like I’m paying for that extra bandwidth either way.

For a company such as ours, where our data and bandwidth needs tend to be excessive, and out of proportion to other companies of our size, these pricing models are prohibitively expensive. A pricing model based on users or connections makes much more sense for us. It’s been a good lesson for me. In the case of Expand we’ve been very happy with the product itself. Their network acceleration appliances work well, and we’ve managed to work out the pricing issues so far. However, this could be an ongoing battle as our company grows, and I don’t particularly relish the thought of revisiting these pricing issues every time we add an office or renew our support contracts.

It’s been a valuable lesson learned for me, and it’s one I’ll pay a lot more attention to in the future. Because our storage and bandwidth needs are out of proportion to our size, basing a pricing model on these elements just doesn’t work for us.

Our EqualLogic Experience So Far

We’re currently in the process of configuring our very first EqualLogic storage unit. It’s a model from their PS Series, and it gives us a total of 6.5 terabytes of storage in a RAID 50 configuration. We’ve only had it live for about a week now, but we love what we’ve seen so far.

The ease of setup and configuration are light years ahead of what we had on our old fiber channel SAN. While our old SAN provided plenty of storage, configuration and management always seemed to involve a call to our vendor, and even then things never seemed to go smoothly. Based on a few days with the EqualLogic unit, I’m reasonably certain that the calls to the vendor for hand-holding are going to be few and far between.

Besides the ease of use, some of the things we love include the ability to utilize what EqualLogic calls thin provisioning. This enables us to create a 500 GB volume while only using 50 GB of actual drive space. The host OS sees a 500 GB volume, but we’re only using 50 GB on the SAN. As the space begins to fill up, the space used on the SAN also begins to approach the 500 GB allocation, but it gives you a way to actually allocate more space than you have initially. Obviously, this is a feature that you want to be careful with, because overuse could result in the SAN actually filling up on you if several of these thin provisioned volumes filled up at the same time. The unit provides plenty of warnings to alert you as these volumes approach a level of your choosing.

The snapshot feature is another feature which we are learning to love. When you provision a new volume, you can simply choose to save snapshots of the volume at whatever interval you choose. Want a daily snapshot for backup purposes? No problem. Want to snapshot a volume prior to an update? No problem. The snapshots are differential copies, so they take up very little space. Any one of these snapshots can be mounted as a physical volume used to quickly restore individual files or an entire volume if necessary. Here’s a real life example- currently we have a thin provisioned volume which is taking up 96 GB of actual disk space (the host OS sees it as a 500 GB volume), and we have taken 10 daily differential snapshots of this volume which are currently utilizing 1.36 GB of space.

Obviously, the unit also provides all kinds of built-in redundancy, with dual controllers, power supplies, fans, and 14 total disks. Short of lengthy power outages or losing the entire server room, not much is going to take this product down. We’re just scratching the surface of what the box will do, and we have plans to do a lot more with it. In the coming months we’ll migrate our VMWare and Exchange storage to this unit, and we also plan on utilizing the snapshots and replication features into our evolving disaster recovery strategy.