1. Copy the backups to a USB drive and ship it – aka. Sneaker Net
2. Upload the files across the VPN using RoboCopy
3. FTP the files up to the new site

While I love using #1 as it’s the easiest and usually the quickest way to move huge amounts of information it’s the most expensive, and it isn’t very repeatable.  If there’s a problem with the data transfer or you need to redo the data transfer then you need to purchase another USB drive from the first provider and have them ship it up to the new site.  As this all needs to be done pretty quickly that means that every time you do it you need to pay for overnight shipping which gets expensive, fast.  Not to mention that either you need to be at the destination site or you have to pay for remote hands to open the box and connect the hard drive.  In this case that means paying a few hundred bucks to have a guy at the remote site unbox the drive and connect it as the data center is in New Jersey and I’m in California a short 5.5 hour plane flight away.

Option #2 that I give here is a decent option as well, except that it only single threaded unless you do some really interesting stuff to spin up multiple copies of robocopy.  The reason that you want multiple threads running is because most Managed Service Providers has some sort of Quality of Service settings configured on their routers so that one connection isn’t able to take all the bandwidth available.  In this case each connection is limited to about 500kb of bandwidth so if I run several threads I get more throughput than if I run just a single thread.

Which leads me to option #3.  As I’m moving lots of database files it’s easy enough for me to do multi-threaded FTP as I can sent each file separately getting getter bandwidth (currently I’m pushing 1,662.90 kb per second).  I do this not with the native command line FTP or with the web browser, but by using a little FTP application which has long been abandoned by the developer called LeechFTP.  While it hasn’t been upgraded in years, and some of the error messages aren’t in English, it’s a great app for moving massive amounts of data in a multi-threaded process.

Now because FTP does totally suck when it comes to resuming from a failed upload process I add an additional step into the process, I take whatever data I’m planning on transferring and use either winrar or 7zip to break the files into smaller chunks.  Typically I’ll just take the entire folder that has all the backups and make one massive upload set out of it.  I usually break the files into 100 Meg segments as those will usually be able to be uploaded without any sort of interruption, and if there is a problem reuploading 100 Megs worth of data usually won’t take all that long.  Now I don’t bother to compress the data, I just put it into a single large rar or 7z fileset.  The reason that I don’t bother trying to compress the data is that it’ll take hours to compress and the time saved usually is pretty small if any (especially as these backups are already compressed).  Both winrar and 7z have store only options which usually run pretty quickly.  The example 581 Gigs of data that I’m working with here was able to be stored by 7z in about an hour and a half.

I’ve begun using 7z instead of winrar for these projects as I’ve found something very annoying about winrar when using it to copy files up to a network share (like say the shared network drive that the SQL Server is going to restore from).  When winrar decompressed all the data it wants to put it into the local %temp% folder which ends up filling the C drive of what ever server you are doing the work on, while 7z doesn’t have this annoying “feature”.

Once the full backups are copied up (~4 days in this case) I just unpack them and get the restore process started (keep in mind that I’ve got log backups being copied across the same network link as well.  I’ll take about how I get them across later on.

Denny

Our current environment has served us well, but we have outgrown the services that Rackspace can offer us, and we have purchased our own production environment.  This isn’t any rinky dink environment either.  We are starting out with a fully redundant, highly available environment which can be scaled by simply deploying more VMs, and in the event that the VMware hardware is over tasked by simply plugging another VMware server into the mix, and shifting the VMs from one node of the cluster to another.

We are very proud of our new environment, so I figured that I’d give you some of the tech specs of it (yeah, I’m totally bragging here).

On the storage side of things we’ve got an EMC CX4-240 with ~35TB of storage laid out in three tiers.  This is connected via multiple 4 Gig fibre cables to a pair of Cisco fibre switches.  Each fibre switch is connected to each of the SAN attached servers.

We went with Dell servers (I would have preferred HP servers, but I was overruled).

The SQL Servers and the VMware servers are identical.  Quad chip, quad core servers each with 64 Gigs of RAM.  Each pair will be clustered for High Availability.  The VMware servers will look a little like they puked cables out of the back.  Because of all the various subnets and to ensure that each subnet is connected to each of the redundant network switches each of the VMware ESX servers will have 11 Ethernet cables, and 2 fibre cables coming out of the back.

The VMware vCenter services are running on a little single chip quad core server.  This is the only part of the system which isn’t redundant, but ESX can run fine for up to 14 days without the License server running, and since this machine has a 4 hour turn around on parts we’ll be fine if the machine dies.

The file servers which host the screenshots, emails, etc which have been captured by our application and will be served to the website upon request are a pair of dual chip, quad core servers also clustered for high availability.

All the servers are SAN attached via the fibre and all data will be saved on the SAN.

Our current environment is much smaller.  A single SQL Server, three web servers, and two file servers.  The only redundant pieces are the fibre cables from the SQL Server to the SAN, and the fact that we have three web servers.  However if the newer web server goes out in the middle of the day, the other two will choke at this point.

Rackspace has been pretty good to us over the years.  It just wasn’t cost effective for us to purchase this level of hardware before now, and Rackspace was able to provide us with a good service level for a reasonable price.  But at this point, because of the amount of hardware we were looking to move into, and the amount of bandwidth we are going to be using it simply became more cost effective for us to host the systems at a local CoLo.

The main reason that I’m telling everyone this is that if you have been trying to find me for the last two weeks or so this is why I can’t be found.  I’ve been spending pretty much every waking moment this together and getting it all setup so that we can migrate over to it.

Needless to say its an awesome project.  How many people get the chance to build a new data center and design it the way they want to from scratch.  Pretty much no one.  Data centers usually grow from a small config of a server or two in a sporadic way, and they are inherited from one person to the next.  But this time I get to design everything they way I want to from the grown up.  It’s going to be a blast.

Denny