April, 2008

Exchange 2007 Migration

I’m sleeping better at night. We’re in the middle of a major project, part of which involves consolidating way too many e-mail servers onto a single Exchange 2007 server. We never really planned to end up with as many e-mail servers as we did, but through a combination of growth and acquisitions, we somehow ended up with 6 of them servicing about 200 people. It was way too many servers for the number of users, even in a multi-office environment. We’re in the process of fixing that now, moving all of our users onto Exchange 2007. Prior to making this move, our primary server was running Exchange 2003, and the hardware was becoming shaky to the point we held our breath every time we restarted it.

I’m a big fan of Exchange. We’ve been running it since Exchange 2000 was released, and I’ve watched it become stronger with each subsequent release. This thing is a workhorse, and it’s been incredibly reliable over the years. When we have had issues with it, we’ve found Microsoft’s Exchange support to be more than equal to the task. As a result, we’ve had a solid track record of providing e-mail service to our employees.

While it’s probably a bit early for me to be singing the praises of Exchange 2007, what I’ve seen so far has proven to be incredibly useful. First, we had to move our storage databases off local storage and onto our shared iSCSI storage after we rolled out the server. (Our EqualLogic box arrived after the Exchange box.) Moving the databases was something which would have terrified me with prior releases. In fact, I’ve never even considered doing it. With Exchange 2007, we simply utilized the wizard and moved all of our databases on a Friday evening. We finished the DB moves in about an hour, and most users never even noticed an interruption in service.

Currently, we’re in the process of migrating our user’s mailboxes onto Exchange 2007. We’ve completed the move for 3 of our offices, and we’re in the process of moving 4 others. In the past, this type of migration meant a Saturday in the office moving mailboxes. With Exchange 2007, I literally do sleep through the moves. We schedule the moves overnight, and when we start the following morning everything is finished. Out of approximately 120 mailboxes moved so far, we’ve only encountered errors on two moves, and neither prevented the mailbox from moving. When they arrive in the morning, our users don’t even realize they’ve been moved.

There’s much more to Exchange 2007 than moving databases and migrating mailboxes, but so far I’ve been impressed. One of the other reasons we moved onto Exchange 2007 was the addition of Unified Messaging. That’s the next step in our implementation, and since that service is new to Exchange 2007 I’m prepared for some growing pains. However, when I consider the pain we’ve experienced with telecomm voicemail systems over the years, I’m actually looking forward to bringing this service into Exchange. Regardless of how that implementation goes, I’m sleeping better these days knowing that the foundation of my e-mail services is in good shape.

Virtualizing the Ad Agency

Virtualization seems to have taken the IT world by storm, and those of us in the Advertising business are along for the ride. I’ve been playing around with virtualization myself for over a year now, and I only see this becoming a bigger part of our business in the future.

It starts at the desktop level, where personally I’ve been running VMware’s Workstation product on my personal laptop. I’ve been running Vista for over a year now, and during that time Workstation has provided me with a convenient copy of XP for those apps which didn’t behave on Vista. The reason I ultimately chose VMware’s product over Microsoft’s desktop virtualization was that it included support for USB devices.

Even more important for those of us in the Agency business at a desktop level, are the possibilities for desktop virtualization the Mac side. Once again, we’ve opted to go with VMware’s Fusion product on our Intel-based Macs. We’re currently struggling with two problems on our Macs which have proven difficult to solve. We’re not huge fans of Microsoft Entourage as an e-mail client, and I could probably devote an entire rant to Microsoft’s decision to drop support for Outlook on the Mac side. We’re also experiencing problems with Mac access to our SharePoint sites. SharePoint works with Safari and Firefox, but as one would expect with a Microsoft app, it works much better with Internet Explorer. We’re hoping that both problems will be solved by actually providing our Mac users with access to Outlook and IE through VMware’s Fusion and a local copy of Windows on their machines.

On the server side, we took the plunge into VMware last year, purchasing ESX Server. We’ve gone fairly slowly in terms of virtualizing our infrastructure, but we’re currently running a SharePoint server, two utility servers, and two development servers as VMs, and we’ve become totally sold on the technology. As a result, we’ve added a second box in order to utilize Virtual Center to aid in the management and to provide load balancing. We’ve also added shared storage via our EqualLogic iSCSI SAN. It all works great, and the possibilities it provides us are endless. If there is a downside, this stuff is expensive, and VMware’s product line and licensing are pretty confusing, and that could give them a problem down the road as Microsoft’s Hyper-V product continues to mature.

We’ve got big plans for virtualization in our Agency. In addition to what we’re doing already, we’re considering virtualization for both high availability and disaster recovery. While we’ve been warned not to virtualize some things, such as domain controllers, Exchange server, and SQL server, we do feel that we can employ a virtual copy in a high-availability or disaster scenario, especially in cases where we maintain the data on a separate platform. We’re also going to explore the possibility of creating a “remote office in a box”, providing us with a quick solution which we could use in acquisitions or the opening of new offices. Our remote offices require a fairly basic setup, and it’s one which we think could be completely virtualized.

We’re going to continue on the path to virtualizing both our servers and desktops. We’ll also be taking a long look at Microsoft’s Hyper-V product. I’ll let you know how it goes.

IT Discussion Boards

You would think this one would be a no-brainer for those of us in Advertising. So much of Web 2.0 seems to be geared towards advertising, it just seems like everybody in this business should be experts and well-versed in everything which makes up Web 2.0. On the contrary, I am often amazed at how slowly we seem to adopt some of this stuff. Case in point, a committee consisting of IT personnel from various Agencies engaged in a discussion last week regarding Mac file problems on Windows-based file servers. How do you think this discussion took place? On an internet discussion board? Nope - All the questions and suggestions showed up in my e-mail inbox.

The biggest problem in holding a discussion using e-mail is that the messages eventually get deleted. All of the good suggestions/answers are lost forever. The reason the group uses e-mail is because discussion boards have failed miserably. Questions often go unanswered completely, or solicit very little feedback. When the same question is posed via e-mail to the group, 13 replies show up within a few short hours. E-mail works for us. Previous attempts at a discussion board have failed, so we continue to use the method that works for us.

I’m not sure exactly what the problem is. Perhaps we simply don’t have critical mass in terms of users. This is a fairly tightly defined niche in the IT world, and we may just not be large enough to get an active discussion board off the ground. Maybe we’re not casting a wide enough net in terms of attracting membership to such a board. The two attempts I’m aware of involve the AAAAs, and the Advantage (an Advertising specific accounting package) User’s Group. Both are fairly large groups, but neither is close to covering everybody in the Advertising business, which results in a membership base that may not be quite large enough. Is there a critical mass required of message boards before they become truly active and useful?

Perhaps we haven’t utilized the correct tools for building our discussion board. I know that one of the keys for driving traffic to our own internal SharePoint site is the user alerts. If we encouraged participants to setup and manage e-mail alerts on an Ad Agency IT Forum, then new posts (or a daily summary) would still show up in our e-mail inbox. A board capable of providing RSS feeds could also help for people choosing to keep updated via that mechanism.

If anyone has an answer to this one, I’d love to hear it. Hey, at least I’m not sending you an e-mail asking for the answer.

Vista and the Death of Windows

Ok, who hasn’t read about this one? Since Gartner issued their report last week on the current state of Windows, it seems like everybody has had something to say about this topic. Since this one has a little bit of that “Mac vs PC” edge to it, I thought I might as well chime in with my own Vista experience. I’m one of those who haven’t really experienced the downside of Vista, but I’m also not beating the drums to update all the PCs in my company either.

For the past several years, my company has been fairly aggressive in moving to the “latest and greatest” versions of just about anything. For the most part, our experiences have been good. I think the first thing we jumped on, installing almost on the release date was Exchange 2003. We got caught in a situation where we had to upgrade a server, we knew we’d be upgrading to 2003 eventually, so we decided to bite the bullet and dive in without waiting for Service Pack 1. We didn’t encounter any major issues, and it seems like ever since we’ve been fairly aggressive about upgrading, on both the PC and Mac side of things. We ran into a similar convergence of upgrades on the Mac side of things with the launch of OS-X and our conversion to Adobe Creative Suite. We survived that also.

I’ve read all the horrible reviews of Vista, and it always seems like they are accompanied with a slew of comments by people telling their Vista horror stories. The groundswell of opinion against this OS has been impressive to say the least. But what’s actually concerned me more has been Microsoft’s seeming inability, or lack of desire, to fight back. The anecdotal stories attached to many of these reviews always leave me shaking my head – how did so many people manage to have such a difficult time with Vista?

I’ve been running Vista since the late beta stages. Crashes have been non-existent, and any incompatibilities I’ve discovered have been few and far between. UAC was a nuisance during the initial setup of machines, but once I’ve had PCs setup, it really stopped being a bother and frankly, I’d like to know about it when something (including myself) requests admin access. I had a horrible initial experience with my current Dell laptop which was running Vista out-of-the-box, but as soon as I disabled Google desktop the speed increased significantly. I’ve been happy with it ever since.

Frankly, the biggest anecdotal experience I can relate with regard to Vista has been with my daughter’s laptop. I took a chance and loaded her up with Vista over a year ago when she returned to school for the spring semester. She’s had that machine on a college campus for over a year now with everything that entails in terms of computing danger. I got my hands on it about 2 weeks ago expecting to find a real mess, but instead it was running flawlessly. I’m not sure that would experience would have been as trouble-free with any previous version of Windows.

Just like the comments I read from IT people detailing their horror stories, my stories represent a small sample, and they don’t necessarily prove a thing. To be honest, because of the bad press, the fact that I haven’t been overwhelmed with the features of Vista, and the rumors we’re hearing about Vista’s successor, I’m not racing to upgrade my company’s PCs to Vista. I’m also not buying the arguments that Windows is dead.

MidMarket CIOs (and IT Directors)

I ran across the following article at eWeek today, regarding MidMarket CIOs, and it struck me as especially relative to my own situation. I wrote about Growing the IT Department last week, and I guess this is just another aspect of that discussion. However, the eWeek article struck a chord today, because I can genuinely relate to my job becoming more strategic in nature, while at the same time I’m becoming less and less involved in some of the technical details.

While I talked at length last week about what this means to my staff, and the growing need for specialized skills, this part of the equation is more about me. I’m not sure exactly how MidMarket is defined, and especially in the advertising business, but I’m pretty sure that I’m squarely in it. I’ve been trying to pinpoint the exact point in time where my own job started to change, but it’s not easy, especially given some of the ups and downs of the industry in general, and my company in particular.

One part of it has definitely been adding employees to the IT staff, but they haven’t always been here in our local office. While I’ve had people working for me for almost 10 years, the only time that I’ve really been able to focus on the strategic issues has been when I’ve had somebody else in the home office who takes some of the desktop support role off my plate. When I’ve had nobody in that role, it was never an issue of completely neglecting the strategic aspect of things, but rather a case of not being able to focus on those issues during the normal work day. Any time spent working on strategic issues was usually on evenings or weekends.

The other ingredient in moving toward a more strategic role has simply been time and experience. It’s only been in the past 2-3 years that management has fully grasped the difference between the strategic role of IT and that of day-to-day desktop support. For me that was the defining moment where I finally felt more like an IT Director, and less like a support person. Some of this was also about people gaining trust and respect for my administrators, because at first they keep coming back to you because you’re perceived as the “expert on everything”.

I find the numbers in the eWeek article interesting, but they also don’t tell me anything I didn’t already know. We usually don’t quite fit into any of the “niches”, but at the same time there’s some comfort in seeing the facts, and realizing that there are others in the same boat. However, sometimes it’s difficult to relate to folks who are in the same boat, because many times they are still in that desktop support role, or at the very least remain very close to the technical details. You can sense the surprise at times when they find out you’re not completely up to speed on the technical aspect of some issue. It wasn’t so long ago that I was wondering that same thing myself.

Personally, I’m glad I ended up in this business and in this company.

Paying Attention to Pricing Models

Over the last several days, I’ve gotten pretty worked up over some vendor pricing models. I understand that vendors are attempting to recoup the investment they have in developing their products, but unfortunately the pricing model vendors may choose to adopt can hit you particularly hard. It’s opened my eyes to this particular issue, and I’m going to pay a whole lot more attention to it going forward.

How would we like it if Cisco or HP suddenly decided to adopt a new pricing model for their network switches? What if this new pricing model required that we had to pay more based on the volume of data moving through the switch? This model would be prohibitively expensive for those of us who routinely move big Photoshop files around. It also doesn’t make a lot of sense, because it’s the same switch whether you’re moving around Word documents or Photoshop files. Thankfully, things don’t work that way in the switch world. You purchase the switch you need, and pricing varies depending on the switch’s speed and the number of ports. The pricing model makes sense.

This brings me to a couple of vendors who apply a different model in order to recoup their development costs. For the past several months I’ve been in discussions with Bakbone about their backup products. Bakbone uses a pricing model where you’re asked to pay based on the volume of data you’re backing up. We’ve also purchased network acceleration devices from Expand Networks in the past 12 months. In Expand’s case, the pricing model is based on the amount of bandwidth being accelerated. In both cases, the smaller company which has large storage and bandwidth needs is penalized. The pricing model works a whole lot better if you’re saving and moving Word documents than it does for Photoshop documents.

In the case of backup software, if I’ve already spent the money for the drive space to store my backups, why on earth should I have to pay a software vendor based on how much data I’m backing up? Does the software have to work any harder in order to back up more stuff? In the case of network acceleration, if I’ve already purchased a piece of hardware which is sufficient for my bandwidth needs, why do I have to pay more to accelerate 20MBps than I do to accelerate 10MBps? The hardware hasn’t changed. I’m already paying my ISP for the bandwidth. Isn’t the whole idea of a network acceleration device that I get more bandwidth for the buck? It feels to me like I’m paying for that extra bandwidth either way.

For a company such as ours, where our data and bandwidth needs tend to be excessive, and out of proportion to other companies of our size, these pricing models are prohibitively expensive. A pricing model based on users or connections makes much more sense for us. It’s been a good lesson for me. In the case of Expand we’ve been very happy with the product itself. Their network acceleration appliances work well, and we’ve managed to work out the pricing issues so far. However, this could be an ongoing battle as our company grows, and I don’t particularly relish the thought of revisiting these pricing issues every time we add an office or renew our support contracts.

It’s been a valuable lesson learned for me, and it’s one I’ll pay a lot more attention to in the future. Because our storage and bandwidth needs are out of proportion to our size, basing a pricing model on these elements just doesn’t work for us.

Growing the IT Department (or Managing Expansion)

There. I always wanted to use “Growing the IT Department” for a title. A few years back, I presented at an Ad Agency CIO Conference, and my topic at the time was “Managing Contraction”. I’m here to tell you that growing the department is a lot more enjoyable (but not necessarily any easier).

The gist of my previous talk was that we found new and unique ways to “do more with less”. Not exactly a revelation there, but it was the reality of what happened to us. At the time, we had lost about 50% of our IT staff, while the company itself probably only got 25% smaller, and of course the number of servers and other system devices we supported didn’t shrink at all. That’s probably a story familiar to many of you, and it certainly wasn’t without pain.

But let’s ignore the pain for now, and talk about a growing IT Department. One of the real challenges in a company which is becoming larger is making sure that your IT staff is not only sized correctly, but also making sure that you have the correct specialties on hand. That is what makes this tough, especially when the company and the IT Department grow as the result of acquisitions. I guess an IT Department can be like a box of chocolates too (apologies to Forrest Gump), because you truly don’t know what you’re going to get when you “acquire” a new employee. In my case I’ve been pretty lucky, because although my staff is split between those I’ve hired (3) and those I’ve acquired (2), all are outstanding IT guys in their own right.

What makes it difficult for us, especially in this business, is the broad range of IT specialties which we have to cover. When you start out, as I did 15 years ago as the only IT support person on staff, you necessarily have to be that jack-of-all-trades, and that’s typically what we get when we acquire a new company. However, it really has become a lot more complicated in the past 15 years, as we’ve had things like digital AV equipment and telecomm added to our plate. We’ve also seen products such as SharePoint and SQL server arrive, which can add significantly to productivity, but which require significant and specialized support themselves. From time to time, you have to sit back and ask yourself how you would staff the company today if you were doing it from scratch. I’m doing a bit of that now, but I’m working closely with my staff to begin to slot people into more specialized IT roles. For the most part it seems to be working, but it’s not necessarily easy.

I also tell my staff that we may never get completely away from the jack-of-all trade roll, but I don’t think that’s a bad thing either. I think it helps an IT Director or CIO to “get their hands dirty” once in a while, and I don’t think any of should ever make a complete break with everything. On the other hand, the specialties give us a great way to give people more opportunities for growth.

The other interesting thing I’ve noticed is that I’m still doing many of the things I discussed in that earlier Managing Contraction talk. We’ve never stopped looking for ways to become more efficient and to do more with less. Sometimes, you also have to work to convince your staff that the latest “efficiency” doesn’t necessarily mean the end of their job, but once again that’s where the new specialties come into play. Just because you’re implementing a new system which means we might spend less time visiting desktops, doesn’t mean we can’t find productive work for everybody.

A lot of that stems with going through the pain of layoffs, and it tends to keep us from growing too fast. Even now, my department tends to be on the lean side, and I think that’s a good thing. It makes us less of a target during economic downturns. Hopefully we can win that argument when the time comes.

Supporting Creative

This is another of those interesting issues I mentioned in my first post, and one that I argued makes our industry unique from an IT perspective. Today, I’d like to discuss some of the issues which surround the support of a large Creative contingent.

While I’d like to keep the discussion centered mostly on the people and work, I do think it is worth mentioning that part of the discussion also relates once again to the PC vs. Mac thing. Since the Creative users are all on the Macintosh platforms, they sometimes perceive that IT support is more geared towards the PC side of the house. Because the size of our company dictates that our IT personnel be “jack-of-all-trades”, we’ve never really had the luxury of hiring pure Mac support specialists. We also haven’t ignored the Mac in our hiring of support personnel. Of the 5 people working for me today, 2 of them came from publishing backgrounds where they actually performed Mac support. We also made a point of sending one of our most recent hires off to the remote office he now supports with a Mac PowerBook under his arm. Even though he wasn’t a Mac guy, the users in that office took it as a very good sign that he was willing and able to support the Mac platform. It took them a while to realize that he was spending a lot of his time on a Vista virtual machine running on Parallels, and in the meantime he actually did learn to support the Macs. Gaining the trust of the Creative users is paramount in supporting them, and actually having Macs on our support people’s desk goes a long ways in building that support. One of the big problems in supporting Macs is finding qualified people. Windows support people are a dime a dozen, and it can be difficult finding good Mac support people who are also willing and able to support the PC.

Another side of this issue which I’ve become more sensitive lately is the perception that IT hates the Mac. I’ll be the first to admit that we have some very entertaining discussions/debates amongst our IT staff regarding the PC vs. Mac battle, but unfortunately some of these debates have spilled over into our discussions with the Creative employees. When I started hearing from Creative that “IT hates the Mac”, I decided it was time to talk to my personnel, and to ask them to be careful what they said. That’s not necessarily an easy thing to ask – it’s a little too much thought control for my liking, but I felt that we needed to be careful about the perception we were creating. The funny thing about this is what the Creative staff isn’t hearing. Those internal IT discussions are just as frequently gripe sessions about Microsoft as they are about Apple, but our Mac users never hear us having those discussions. One of the nice things about having a larger staff of IT people, is that we get an opportunity to discuss these things amongst ourselves. This is one of the things mentioned most frequently when we’ve acquired new offices with existing IT personnel. Typically, these IT people were the only one in their office, and they never had anyone else who truly understood some of the issues we face. Finally being able to discuss some of these things with their peers is a big deal to them.

Obviously there’s more to supporting Creative than just the Mac vs. PC issue. The potential clash between the typical IT personality and the typical Creative personality is probably the single biggest issue, but I’ll leave that one for another day.

The Mac is a Wonderful (and Secure) Platform

I’ve already received one e-mail from a reader pointing me to a blog post refuting the claims of security issues with the Mac . I suppose I should take some consolation in the fact that I’m not getting slammed harder for my first Mac vs. PC post, but I also think the above post helps point out the bigger issue. The argument has become so heated, that sometimes even the good points tend to get lost in the extreme spirit of the debate.

First of all, let me begin by saying that I’m not a security expert. Please refer to my first post for a better understanding of the realities of what I do. At this level of IT, we have to be something of a jack-of-all-trades, and I probably know just enough about security to be dangerous, but the same can probably be said about a lot of specialized IT subject matters. We know enough to get by. In my case, that knowledge has been good enough to get me through 15 years in this job without suffering from any major system outages or downtime to virus outbreaks. I think we’ve done a pretty good job so far, but I’m also not naïve enough to think that any of my systems are invulnerable, and I would never make that claim.

I had misgivings about going “negative” in last week’s post about Mac vs. PC, and in hindsight perhaps the article about the Macbook Air hack deserves greater scrutiny. I’m going to try to avoid this trap in the future. I think there are plenty of positive reasons to choose both platforms. The Mac is undoubtedly a very secure platform, and it’s become even more so in recent versions of OS-X. The fact of the matter is I don’t lose a lot of sleep worrying about my Macs getting hacked. However, is there really anything wrong with pointing out flaws where they do exist, and perhaps letting Apple know that for those of out here in the trenches – security does matter?

Should the negatives come into play in this discussion? Certainly they should. Any time we make a big decision in IT we try to weigh the pros and cons of several options, so the negative factors are something we must consider. The problem with being negative in this discussion is that you’re immediately perceived by the other side to be a “basher”. I can assure you that is not the case with me.

My bigger point in terms of the Mac vs. PC debate is that the argument itself has become too bitter. Case in point being the article at the top of this post – the writer is obviously a big fan of the Mac platform, and I certainly have no issue with anyone having that opinion. However, when you start writing about the “prejudices of an idiot public” and “Microsoft’s criminal actions against its customers”, you run the risk of losing me. I’m simply not willing to go that far in supporting either platform, and personally I think it’s going too far. I also think it’s too bad, because the article itself makes some genuinely good arguments, but when the author goes over that edge into what I’d call extremism, sometimes those good arguments tend to get lost on me.

There are some things I would like to see both Microsoft and Apple do differently. For now, out of fear of inciting either group I’ll avoid those topics. I will save them for a rainy day, because I do think those topics are worthy of discussion.

I’ll say it again. I have plenty of good reasons for purchasing Macs. I have plenty of good reasons for purchasing PCs. Perhaps outlining those reasons would make good topics for my next couple of posts.