Here We Grow Again

We’re getting bigger again. Eric Mower and Associates has just combined with another agency here in Syracuse. We’ll be adding 43 people to our current staff of 80 in this office, and they’re moving into our offices around July 1^st. Good news for the agency – significant challenges for IT.

Space is going to be a huge issue. Squeezing another 40 people into our existing space is not going to be easy, and we’re going to be using some spaces which have never been used for offices. Wireless seems like a convenient solution, but we’ve never felt that wireless was a good option for our Creative staff, and that limits our options in terms of who can occupy any wireless offices.

The space issue also presents another possibility – telecommuting. We’ve discussed telecommuting in the past, but we haven’t really done much of it. On the other hand, many of us in the company can and do work extremely efficiently from home. Working from home has been a way of life for those of us in IT for several years. Many of our other users have also become quite adept at working from home, and we’ve even introduces a handful of IP phones in the past couple years. All of this makes me wonder if perhaps we shouldn’t step this up a bit as we’re squeezed for space. Telecommuting is tough for our Creative and Account Service staffs which tend to need more face-to-face interaction, but we have others, including myself, HR, and Accounting Directors who might actually be able to share an office and work from home on the days we’re not in.

Communications is always a challenge, and what’s going to make this really interesting is our ongoing implementation of Microsoft’s Unified Communications system. Having this happen in the middle of that implementation is going to be challenging, however it also presents opportunities to utilize the new systems to actually speed up the integration. Prior to the move in date, we may utilize the Communicator client to provide chat and voice connectivity with our new co-workers.

Like I said – the integration will offer significant challenges, but that’s nothing new in this business. We’ll figure it all out, and I’ll fill you in on the highlights.

Is That a Blog, a Wiki, or an E-Mail?

One of the things which continues to be a barrier to our adoption of Web 2.0 tools, particularly our SharePoint Blogs and Wiki’s is confusion over where to post something. I really think this fear of being wrong is preventing some of our users from using these tools. I’d like to suggest that we stop worrying about it, and just post. Stop worrying about whether or not you’re correct. It’s not going to be the end of the world is something which is more appropriate for a Wiki entry is posted as a Blog or vice versa. The important thing is that useful information gets posted someplace where it will be preserved.

Unfortunately, we continue to rely on e-mail to publish information which is better served being posted on our SharePoint site, where it can be easily found weeks or months later. The problem is that e-mail is the best tool to use to notify people of a new post. It’s possible to set up alerts, but we don’t necessarily want to have every Wiki entry sent to everyone in the Agency.

There are a couple fairly simple solutions to this problem. First, we simply need to teach our users how to create and send a link via e-mail. When an item of interest is posted in a SharePoint Blog or Wiki, the person posting it needs to copy a shortcut to the item, and then paste the URL into an e-mail telling the appropriate audience about the item. Yes, it’s an extra step, and it’s easier to just send an e-mail, but having the item preserved for posterity is worth the effort.

Another option isn’t in place right now, but we could “mail enable” certain SharePoint items, including Blogs. We also could begin implementing some of the Audience targeting features of SharePoint which would help with Wikis. This would allow people to add an address to an e-mail which would cause the contents of the e-mail to be automatically added as a Blog entry. Audience targeting would allow people to target Wiki entries (and other SharePoint items) to specific audiences (i.e. Agency Wide, Account Service, etc.). This option, however, would require quite a bit of setup and testing prior to implementation.

For now, we’ll focus on education and training. Hopefully, If we can get people to begin posting to our Blogs and Wikis, perhaps they’ll overcome some of those fears and this will become less of an issue.

Archiving Creative Materials

One of the more difficult challenges we face is that of archiving our creative materials. We’re currently taking a fresh look at this process, reviewing both the process in terms of who is doing what, but also taking a fresh look at the storage media we’re using.

Our current process involves burning all of our closed jobs to DVDs. In our main office alone, we routinely archive up to 20 DVDs (approximately 100 GB) of material every month, and we have 6 other offices all facing the same issue. Burning the DVDs is actually only part of the process, although it may be the most man-hour and time intensive part.

The first step involves making a final determination about when a job is complete, and this is a step which has to be done by the Creative Department. They are ultimately the owners of this data, and once a job has been tagged as final, it is moved to a read-only network location where it remains until it gets burned to DVD. How long it remains on the network depends on our current remaining disk space, but we can usually maintain 2-3 months on our network. This is very helpful, because most requests for archived jobs occur in this time span. Having he materials readily available on the network saves us from having to retrieve from an offline DVD.

Once we need to reclaim disk space, we then begin the process of burning this material to DVDs. First, we have to break the data up into DVD-size chunks. Then, we burn it onto DVD, verifying against the original. Finally, we duplicate the DVD, catalog the DVD using DiskTracker, and send the duplicate DVD to one of our other locations for permanent offsite storage. The DiskTracker catalog is shared on our network for all users to search. This part of the process requires both time and manpower, up to 2 days per month. It’s also contentious in terms of who actually does it – in some of our offices we utilize IT personnel while in others it’s done by Creative or Studio.

Over the years we’ve constantly looked for ways to streamline and improve the process, including jukebox-based systems, but we’ve usually rejected most systems because of the expense of adding them to all of our locations.

At the moment we’re actually contemplating replacing our DVDs with hard drive based storage. We could store at least 6 months of archive materials on a 500 GB hard drive. The amount of time we could save using a hard drive based system would save us a tremendous amount of time, but it also raises a lot of issues – including costs, reliability, susceptibility to viruses, and the potential for losing more data in a single shot. Personally, it seems like most of those issues can be resolved, and we’re going to test a hard drive based solution over the coming months.

We’ll see how it goes, but it will be very interesting to see how we’re archiving and what we’re using 5-10 years from now.

Exchange 2007 Migration

I’m sleeping better at night. We’re in the middle of a major project, part of which involves consolidating way too many e-mail servers onto a single Exchange 2007 server. We never really planned to end up with as many e-mail servers as we did, but through a combination of growth and acquisitions, we somehow ended up with 6 of them servicing about 200 people. It was way too many servers for the number of users, even in a multi-office environment. We’re in the process of fixing that now, moving all of our users onto Exchange 2007. Prior to making this move, our primary server was running Exchange 2003, and the hardware was becoming shaky to the point we held our breath every time we restarted it.

I’m a big fan of Exchange. We’ve been running it since Exchange 2000 was released, and I’ve watched it become stronger with each subsequent release. This thing is a workhorse, and it’s been incredibly reliable over the years. When we have had issues with it, we’ve found Microsoft’s Exchange support to be more than equal to the task. As a result, we’ve had a solid track record of providing e-mail service to our employees.

While it’s probably a bit early for me to be singing the praises of Exchange 2007, what I’ve seen so far has proven to be incredibly useful. First, we had to move our storage databases off local storage and onto our shared iSCSI storage after we rolled out the server. (Our EqualLogic box arrived after the Exchange box.) Moving the databases was something which would have terrified me with prior releases. In fact, I’ve never even considered doing it. With Exchange 2007, we simply utilized the wizard and moved all of our databases on a Friday evening. We finished the DB moves in about an hour, and most users never even noticed an interruption in service.

Currently, we’re in the process of migrating our user’s mailboxes onto Exchange 2007. We’ve completed the move for 3 of our offices, and we’re in the process of moving 4 others. In the past, this type of migration meant a Saturday in the office moving mailboxes. With Exchange 2007, I literally do sleep through the moves. We schedule the moves overnight, and when we start the following morning everything is finished. Out of approximately 120 mailboxes moved so far, we’ve only encountered errors on two moves, and neither prevented the mailbox from moving. When they arrive in the morning, our users don’t even realize they’ve been moved.

There’s much more to Exchange 2007 than moving databases and migrating mailboxes, but so far I’ve been impressed. One of the other reasons we moved onto Exchange 2007 was the addition of Unified Messaging. That’s the next step in our implementation, and since that service is new to Exchange 2007 I’m prepared for some growing pains. However, when I consider the pain we’ve experienced with telecomm voicemail systems over the years, I’m actually looking forward to bringing this service into Exchange. Regardless of how that implementation goes, I’m sleeping better these days knowing that the foundation of my e-mail services is in good shape.

Virtualizing the Ad Agency

Virtualization seems to have taken the IT world by storm, and those of us in the Advertising business are along for the ride. I’ve been playing around with virtualization myself for over a year now, and I only see this becoming a bigger part of our business in the future.

It starts at the desktop level, where personally I’ve been running VMware’s Workstation product on my personal laptop. I’ve been running Vista for over a year now, and during that time Workstation has provided me with a convenient copy of XP for those apps which didn’t behave on Vista. The reason I ultimately chose VMware’s product over Microsoft’s desktop virtualization was that it included support for USB devices.

Even more important for those of us in the Agency business at a desktop level, are the possibilities for desktop virtualization the Mac side. Once again, we’ve opted to go with VMware’s Fusion product on our Intel-based Macs. We’re currently struggling with two problems on our Macs which have proven difficult to solve. We’re not huge fans of Microsoft Entourage as an e-mail client, and I could probably devote an entire rant to Microsoft’s decision to drop support for Outlook on the Mac side. We’re also experiencing problems with Mac access to our SharePoint sites. SharePoint works with Safari and Firefox, but as one would expect with a Microsoft app, it works much better with Internet Explorer. We’re hoping that both problems will be solved by actually providing our Mac users with access to Outlook and IE through VMware’s Fusion and a local copy of Windows on their machines.

On the server side, we took the plunge into VMware last year, purchasing ESX Server. We’ve gone fairly slowly in terms of virtualizing our infrastructure, but we’re currently running a SharePoint server, two utility servers, and two development servers as VMs, and we’ve become totally sold on the technology. As a result, we’ve added a second box in order to utilize Virtual Center to aid in the management and to provide load balancing. We’ve also added shared storage via our EqualLogic iSCSI SAN. It all works great, and the possibilities it provides us are endless. If there is a downside, this stuff is expensive, and VMware’s product line and licensing are pretty confusing, and that could give them a problem down the road as Microsoft’s Hyper-V product continues to mature.

We’ve got big plans for virtualization in our Agency. In addition to what we’re doing already, we’re considering virtualization for both high availability and disaster recovery. While we’ve been warned not to virtualize some things, such as domain controllers, Exchange server, and SQL server, we do feel that we can employ a virtual copy in a high-availability or disaster scenario, especially in cases where we maintain the data on a separate platform. We’re also going to explore the possibility of creating a “remote office in a box”, providing us with a quick solution which we could use in acquisitions or the opening of new offices. Our remote offices require a fairly basic setup, and it’s one which we think could be completely virtualized.

We’re going to continue on the path to virtualizing both our servers and desktops. We’ll also be taking a long look at Microsoft’s Hyper-V product. I’ll let you know how it goes.

IT Discussion Boards

You would think this one would be a no-brainer for those of us in Advertising. So much of Web 2.0 seems to be geared towards advertising, it just seems like everybody in this business should be experts and well-versed in everything which makes up Web 2.0. On the contrary, I am often amazed at how slowly we seem to adopt some of this stuff. Case in point, a committee consisting of IT personnel from various Agencies engaged in a discussion last week regarding Mac file problems on Windows-based file servers. How do you think this discussion took place? On an internet discussion board? Nope - All the questions and suggestions showed up in my e-mail inbox.

The biggest problem in holding a discussion using e-mail is that the messages eventually get deleted. All of the good suggestions/answers are lost forever. The reason the group uses e-mail is because discussion boards have failed miserably. Questions often go unanswered completely, or solicit very little feedback. When the same question is posed via e-mail to the group, 13 replies show up within a few short hours. E-mail works for us. Previous attempts at a discussion board have failed, so we continue to use the method that works for us.

I’m not sure exactly what the problem is. Perhaps we simply don’t have critical mass in terms of users. This is a fairly tightly defined niche in the IT world, and we may just not be large enough to get an active discussion board off the ground. Maybe we’re not casting a wide enough net in terms of attracting membership to such a board. The two attempts I’m aware of involve the AAAAs, and the Advantage (an Advertising specific accounting package) User’s Group. Both are fairly large groups, but neither is close to covering everybody in the Advertising business, which results in a membership base that may not be quite large enough. Is there a critical mass required of message boards before they become truly active and useful?

Perhaps we haven’t utilized the correct tools for building our discussion board. I know that one of the keys for driving traffic to our own internal SharePoint site is the user alerts. If we encouraged participants to setup and manage e-mail alerts on an Ad Agency IT Forum, then new posts (or a daily summary) would still show up in our e-mail inbox. A board capable of providing RSS feeds could also help for people choosing to keep updated via that mechanism.

If anyone has an answer to this one, I’d love to hear it. Hey, at least I’m not sending you an e-mail asking for the answer.

Vista and the Death of Windows

Ok, who hasn’t read about this one? Since Gartner issued their report last week on the current state of Windows, it seems like everybody has had something to say about this topic. Since this one has a little bit of that “Mac vs PC” edge to it, I thought I might as well chime in with my own Vista experience. I’m one of those who haven’t really experienced the downside of Vista, but I’m also not beating the drums to update all the PCs in my company either.

For the past several years, my company has been fairly aggressive in moving to the “latest and greatest” versions of just about anything. For the most part, our experiences have been good. I think the first thing we jumped on, installing almost on the release date was Exchange 2003. We got caught in a situation where we had to upgrade a server, we knew we’d be upgrading to 2003 eventually, so we decided to bite the bullet and dive in without waiting for Service Pack 1. We didn’t encounter any major issues, and it seems like ever since we’ve been fairly aggressive about upgrading, on both the PC and Mac side of things. We ran into a similar convergence of upgrades on the Mac side of things with the launch of OS-X and our conversion to Adobe Creative Suite. We survived that also.

I’ve read all the horrible reviews of Vista, and it always seems like they are accompanied with a slew of comments by people telling their Vista horror stories. The groundswell of opinion against this OS has been impressive to say the least. But what’s actually concerned me more has been Microsoft’s seeming inability, or lack of desire, to fight back. The anecdotal stories attached to many of these reviews always leave me shaking my head – how did so many people manage to have such a difficult time with Vista?

I’ve been running Vista since the late beta stages. Crashes have been non-existent, and any incompatibilities I’ve discovered have been few and far between. UAC was a nuisance during the initial setup of machines, but once I’ve had PCs setup, it really stopped being a bother and frankly, I’d like to know about it when something (including myself) requests admin access. I had a horrible initial experience with my current Dell laptop which was running Vista out-of-the-box, but as soon as I disabled Google desktop the speed increased significantly. I’ve been happy with it ever since.

Frankly, the biggest anecdotal experience I can relate with regard to Vista has been with my daughter’s laptop. I took a chance and loaded her up with Vista over a year ago when she returned to school for the spring semester. She’s had that machine on a college campus for over a year now with everything that entails in terms of computing danger. I got my hands on it about 2 weeks ago expecting to find a real mess, but instead it was running flawlessly. I’m not sure that would experience would have been as trouble-free with any previous version of Windows.

Just like the comments I read from IT people detailing their horror stories, my stories represent a small sample, and they don’t necessarily prove a thing. To be honest, because of the bad press, the fact that I haven’t been overwhelmed with the features of Vista, and the rumors we’re hearing about Vista’s successor, I’m not racing to upgrade my company’s PCs to Vista. I’m also not buying the arguments that Windows is dead.

MidMarket CIOs (and IT Directors)

I ran across the following article at eWeek today, regarding MidMarket CIOs, and it struck me as especially relative to my own situation. I wrote about Growing the IT Department last week, and I guess this is just another aspect of that discussion. However, the eWeek article struck a chord today, because I can genuinely relate to my job becoming more strategic in nature, while at the same time I’m becoming less and less involved in some of the technical details.

While I talked at length last week about what this means to my staff, and the growing need for specialized skills, this part of the equation is more about me. I’m not sure exactly how MidMarket is defined, and especially in the advertising business, but I’m pretty sure that I’m squarely in it. I’ve been trying to pinpoint the exact point in time where my own job started to change, but it’s not easy, especially given some of the ups and downs of the industry in general, and my company in particular.

One part of it has definitely been adding employees to the IT staff, but they haven’t always been here in our local office. While I’ve had people working for me for almost 10 years, the only time that I’ve really been able to focus on the strategic issues has been when I’ve had somebody else in the home office who takes some of the desktop support role off my plate. When I’ve had nobody in that role, it was never an issue of completely neglecting the strategic aspect of things, but rather a case of not being able to focus on those issues during the normal work day. Any time spent working on strategic issues was usually on evenings or weekends.

The other ingredient in moving toward a more strategic role has simply been time and experience. It’s only been in the past 2-3 years that management has fully grasped the difference between the strategic role of IT and that of day-to-day desktop support. For me that was the defining moment where I finally felt more like an IT Director, and less like a support person. Some of this was also about people gaining trust and respect for my administrators, because at first they keep coming back to you because you’re perceived as the “expert on everything”.

I find the numbers in the eWeek article interesting, but they also don’t tell me anything I didn’t already know. We usually don’t quite fit into any of the “niches”, but at the same time there’s some comfort in seeing the facts, and realizing that there are others in the same boat. However, sometimes it’s difficult to relate to folks who are in the same boat, because many times they are still in that desktop support role, or at the very least remain very close to the technical details. You can sense the surprise at times when they find out you’re not completely up to speed on the technical aspect of some issue. It wasn’t so long ago that I was wondering that same thing myself.

Personally, I’m glad I ended up in this business and in this company.

Paying Attention to Pricing Models

Over the last several days, I’ve gotten pretty worked up over some vendor pricing models. I understand that vendors are attempting to recoup the investment they have in developing their products, but unfortunately the pricing model vendors may choose to adopt can hit you particularly hard. It’s opened my eyes to this particular issue, and I’m going to pay a whole lot more attention to it going forward.

How would we like it if Cisco or HP suddenly decided to adopt a new pricing model for their network switches? What if this new pricing model required that we had to pay more based on the volume of data moving through the switch? This model would be prohibitively expensive for those of us who routinely move big Photoshop files around. It also doesn’t make a lot of sense, because it’s the same switch whether you’re moving around Word documents or Photoshop files. Thankfully, things don’t work that way in the switch world. You purchase the switch you need, and pricing varies depending on the switch’s speed and the number of ports. The pricing model makes sense.

This brings me to a couple of vendors who apply a different model in order to recoup their development costs. For the past several months I’ve been in discussions with Bakbone about their backup products. Bakbone uses a pricing model where you’re asked to pay based on the volume of data you’re backing up. We’ve also purchased network acceleration devices from Expand Networks in the past 12 months. In Expand’s case, the pricing model is based on the amount of bandwidth being accelerated. In both cases, the smaller company which has large storage and bandwidth needs is penalized. The pricing model works a whole lot better if you’re saving and moving Word documents than it does for Photoshop documents.

In the case of backup software, if I’ve already spent the money for the drive space to store my backups, why on earth should I have to pay a software vendor based on how much data I’m backing up? Does the software have to work any harder in order to back up more stuff? In the case of network acceleration, if I’ve already purchased a piece of hardware which is sufficient for my bandwidth needs, why do I have to pay more to accelerate 20MBps than I do to accelerate 10MBps? The hardware hasn’t changed. I’m already paying my ISP for the bandwidth. Isn’t the whole idea of a network acceleration device that I get more bandwidth for the buck? It feels to me like I’m paying for that extra bandwidth either way.

For a company such as ours, where our data and bandwidth needs tend to be excessive, and out of proportion to other companies of our size, these pricing models are prohibitively expensive. A pricing model based on users or connections makes much more sense for us. It’s been a good lesson for me. In the case of Expand we’ve been very happy with the product itself. Their network acceleration appliances work well, and we’ve managed to work out the pricing issues so far. However, this could be an ongoing battle as our company grows, and I don’t particularly relish the thought of revisiting these pricing issues every time we add an office or renew our support contracts.

It’s been a valuable lesson learned for me, and it’s one I’ll pay a lot more attention to in the future. Because our storage and bandwidth needs are out of proportion to our size, basing a pricing model on these elements just doesn’t work for us.

Growing the IT Department (or Managing Expansion)

There. I always wanted to use “Growing the IT Department” for a title. A few years back, I presented at an Ad Agency CIO Conference, and my topic at the time was “Managing Contraction”. I’m here to tell you that growing the department is a lot more enjoyable (but not necessarily any easier).

The gist of my previous talk was that we found new and unique ways to “do more with less”. Not exactly a revelation there, but it was the reality of what happened to us. At the time, we had lost about 50% of our IT staff, while the company itself probably only got 25% smaller, and of course the number of servers and other system devices we supported didn’t shrink at all. That’s probably a story familiar to many of you, and it certainly wasn’t without pain.

But let’s ignore the pain for now, and talk about a growing IT Department. One of the real challenges in a company which is becoming larger is making sure that your IT staff is not only sized correctly, but also making sure that you have the correct specialties on hand. That is what makes this tough, especially when the company and the IT Department grow as the result of acquisitions. I guess an IT Department can be like a box of chocolates too (apologies to Forrest Gump), because you truly don’t know what you’re going to get when you “acquire” a new employee. In my case I’ve been pretty lucky, because although my staff is split between those I’ve hired (3) and those I’ve acquired (2), all are outstanding IT guys in their own right.

What makes it difficult for us, especially in this business, is the broad range of IT specialties which we have to cover. When you start out, as I did 15 years ago as the only IT support person on staff, you necessarily have to be that jack-of-all-trades, and that’s typically what we get when we acquire a new company. However, it really has become a lot more complicated in the past 15 years, as we’ve had things like digital AV equipment and telecomm added to our plate. We’ve also seen products such as SharePoint and SQL server arrive, which can add significantly to productivity, but which require significant and specialized support themselves. From time to time, you have to sit back and ask yourself how you would staff the company today if you were doing it from scratch. I’m doing a bit of that now, but I’m working closely with my staff to begin to slot people into more specialized IT roles. For the most part it seems to be working, but it’s not necessarily easy.

I also tell my staff that we may never get completely away from the jack-of-all trade roll, but I don’t think that’s a bad thing either. I think it helps an IT Director or CIO to “get their hands dirty” once in a while, and I don’t think any of should ever make a complete break with everything. On the other hand, the specialties give us a great way to give people more opportunities for growth.

The other interesting thing I’ve noticed is that I’m still doing many of the things I discussed in that earlier Managing Contraction talk. We’ve never stopped looking for ways to become more efficient and to do more with less. Sometimes, you also have to work to convince your staff that the latest “efficiency” doesn’t necessarily mean the end of their job, but once again that’s where the new specialties come into play. Just because you’re implementing a new system which means we might spend less time visiting desktops, doesn’t mean we can’t find productive work for everybody.

A lot of that stems with going through the pain of layoffs, and it tends to keep us from growing too fast. Even now, my department tends to be on the lean side, and I think that’s a good thing. It makes us less of a target during economic downturns. Hopefully we can win that argument when the time comes.