I’m writing this in a moving car on the way to my son’s lacrosse game. It’s about a 1-hour drive, and I’ve already communicated several items to my IT staff via e-mail, activated 3 new users on our Office Communications Server, and begun writing this post. Needless to say, my wife (and driver) is getting very little of my attention, and that’s probably not fair.
I think it’s fantastic that we have the capability to do what we do from remote locations, but I think we have to be very careful not to go overboard. I spent last Thursday evening in the Charlotte airport, communicating with co-workers who were literally on the other side of the globe. It was a scenario in which 5-years ago, these questions would have gone unanswered for days – we actually accomplished something in about 5 minutes.
We have to do what we have to do sometimes. In my current situation, which is overloaded with time-critical projects, it seems like these tools are invaluable. On the one hand, we can walk out of the office confident that we can still be productive in a moving vehicle. On the other hand, we can’t lose touch with the people sitting next to us in the car.
That’s why I’m making this a short post.
In addition to our current acquisition-related projects, and our implementation of Microsoft Unified Communications, we’re also in the midst of another project which began late last year – implementing a paperless workflow system into the heart of our day-to-day operations. While paperless workflow isn’t really an IT project, it definitely touches upon several IT systems. It’s really not even fair to try to discuss the scope of this project in a blog entry, but I’ll try to summarize it.
At the heart of our paperless workflow system is Adobe Acrobat, accompanied by WebDAV servers at each of our office locations, allowing our users to collaborate electronically on a job from start to finish. Since we’re using the WebDAV servers for storage of the PDF files, we’re not cluttering up our user’s mailboxes with multiple versions of a job.
We’re using some Microsoft software – Exchange (e-mail) and SharePoint to traffic and schedule the job through the Agency. The two systems work very well together to provide notification of status changes on a project. SharePoint was perhaps the biggest surprise we encountered during the early stages of the project. I briefed our Studio Department on SharePoint one day late last year, expecting a lukewarm reception to a Microsoft product from some of our most hardcore Mac users. It turns out they had been looking for a way to schedule jobs electronically, and the next day I was shocked to find them tearing down their scheduling whiteboards. Just like that, SharePoint had become a mission critical application. They loved the way that status changes in the SharePoint schedule automatically kicked off e-mail alerts to the entire team.
Other aspects of the project got more into some IT hardware. Gigabit Ethernet capabilities finally made it possible for our Mac users to effectively work off a file server rather than constantly moving files from the network to their local hard drives. This provided a huge benefit to collaboration for our Creative teams. It did change things significantly from an IT perspective, as we begin to move away from large hard drives on our local machines and need more storage on the Creative servers.
We also added more LCD monitors to our inventory in order to facilitate working electronically. Not having a physical piece of paper to review and markup meant that more of this work was being done on screen. A small investment in monitors has added up to huge gains in productivity.
It’s certainly not an insignificant project from an IT perspective. I find myself spending quite a bit of time explaining the complexities involved in making all these systems work reliably, but it does seem to be working. We’ll probably be spending most of the rest of this year getting this implemented in all of our offices. In light of everything else we have going on; it’s just another part of a very busy year for IT.
How do you handle growth? This is a question that’s really been troubling me over the past couple of weeks. For many of us, the day-to-day issues of running IT are pretty straight forward. We keep our existing “stuff” running, and occasionally we introduce some new “stuff”. For years, that’s what I did also. Occasionally, we’d get a little larger, adding an office every once in a while. That presented different challenges, but it was always manageable.
Our current growth feels different. Despite adding some locations over the past 10 years, we’ve hovered in the 150-200 employee range for much of that time. We’ve now jumped to about 260 employees, and we may be heading to 500 employees much fast than I ever envisioned. Furthermore, if that happens, it will probably mean a doubling of our existing locations – from 7 offices to somewhere between 12-15 offices.
What does all of this do in terms of IT and Telecomm support? The typical small IT Agency, which is large enough to require IT support, comes with an IT “Jack of all trades”, and there’s a limit to how many of those types an Agency needs.
At some point, you’ve got to start thinking about what a larger IT Department should look like. I think we’ve reached that point, so I’m thinking about it a lot. How many IT people would be required to support 500 people in 12-15 different locations, keeping in mind that we support 2 different platforms (Mac and PC), and that we also support telecomm. I’ve always felt like 35 was the point where an Agency needs in-house support, I think that number is a pretty good starting point for determining a support ratio in this business. I also think you can increase that number as you get larger, gaining some efficiency of scale. So let’s say that a 500 person company could get by with a 50-1 ratio – that means 10 IT people.
The next question is how to apportion those 10. Personally, I think an organization of this size should move to a formal help desk environment, which could be centralized in one location, or even split between several locations if geography dictates it. How many help desk personnel? What about the rest of the IT positions? An organization of 500 probably justifies a few specialists of some type – perhaps a telecomm manager, a network manager, a DB specialist, a graphics systems manager, etc., I guess those jacks-of-all trades might come in handy after all.
We’re making excellent progress on our Microsoft UC implementation. To date, we’ve rolled out Microsoft Office Communications Server and in our main office, and we’ve begun testing with the Communicator and LiveMeeting clients. In addition, we’ve nearly completed the migration to Exchange 2007 and we’re in the process of removing Exchange servers from our remote sites. This week we’re rolling out OCS Mediation servers to our main office and two remote locations. These servers, along with soon to be installed gateway servers, will be the connection between our Exchange server and our phone switches. We’re probably about 2 weeks away from tying it all into our existing phone system.
I’m currently in the process of presenting an overview to our users in order to prep them for what’s ahead. We’ll be doing formal training for the system around the middle of June, but for now I’m just laying some of the groundwork and giving employees an idea of what’s ahead. One of the big reasons we’re deploying Microsoft UC is to facilitate the connection of new phone systems into our existing system. Secondary to that is our plan to once again implement Unified Messaging – this time through Exchange 2007, and also to provide presence information via the Communicator client.
One of my great fears is that I’m introducing too much too fast, and some of the items I didn’t mention above could also take off on us. In addition to those features mentioned above, we’ll also be providing chat capability, the ability to easily redirect phone calls to other numbers, and additional conferencing capability through LiveMeeting. I’d like to roll these items out following the initial implementation of UC, but I know that some of my users will begin to play with them on day one. Other than LiveMeeting, which is a separate client I can choose to leave uninstalled for now, many of the features mentioned above are built right into Communicator.
I’m also a bit nervous about Microsoft’s Mac support. We’ve only recently received an update to Messenger for the Mac which works with OCS 2007, and we’re currently searching for a LiveMeeting client. I’m concerned that LiveMeeting on OCS requires a client and does not provide any web-based conferencing capability. That being the case, it’s not necessarily the end of the world, because out intent all along was to use this primarily as an internal conferencing system. We use Acrobat Connect as our conferencing system to the outside world. However, it would be serious limitation if LiveMeeting didn’t provide a way to include our own Macs.That would hurt.
We’re getting bigger again. Eric Mower and Associates has just combined with another agency here in Syracuse. We’ll be adding 43 people to our current staff of 80 in this office, and they’re moving into our offices around July 1st. Good news for the agency – significant challenges for IT.
Space is going to be a huge issue. Squeezing another 40 people into our existing space is not going to be easy, and we’re going to be using some spaces which have never been used for offices. Wireless seems like a convenient solution, but we’ve never felt that wireless was a good option for our Creative staff, and that limits our options in terms of who can occupy any wireless offices.
The space issue also presents another possibility – telecommuting. We’ve discussed telecommuting in the past, but we haven’t really done much of it. On the other hand, many of us in the company can and do work extremely efficiently from home. Working from home has been a way of life for those of us in IT for several years. Many of our other users have also become quite adept at working from home, and we’ve even introduces a handful of IP phones in the past couple years. All of this makes me wonder if perhaps we shouldn’t step this up a bit as we’re squeezed for space. Telecommuting is tough for our Creative and Account Service staffs which tend to need more face-to-face interaction, but we have others, including myself, HR, and Accounting Directors who might actually be able to share an office and work from home on the days we’re not in.
Communications is always a challenge, and what’s going to make this really interesting is our ongoing implementation of Microsoft’s Unified Communications system. Having this happen in the middle of that implementation is going to be challenging, however it also presents opportunities to utilize the new systems to actually speed up the integration. Prior to the move in date, we may utilize the Communicator client to provide chat and voice connectivity with our new co-workers.
Like I said – the integration will offer significant challenges, but that’s nothing new in this business. We’ll figure it all out, and I’ll fill you in on the highlights.
One of the things which continues to be a barrier to our adoption of Web 2.0 tools, particularly our SharePoint Blogs and Wiki’s is confusion over where to post something. I really think this fear of being wrong is preventing some of our users from using these tools. I’d like to suggest that we stop worrying about it, and just post. Stop worrying about whether or not you’re correct. It’s not going to be the end of the world is something which is more appropriate for a Wiki entry is posted as a Blog or vice versa. The important thing is that useful information gets posted someplace where it will be preserved.
Unfortunately, we continue to rely on e-mail to publish information which is better served being posted on our SharePoint site, where it can be easily found weeks or months later. The problem is that e-mail is the best tool to use to notify people of a new post. It’s possible to set up alerts, but we don’t necessarily want to have every Wiki entry sent to everyone in the Agency.
There are a couple fairly simple solutions to this problem. First, we simply need to teach our users how to create and send a link via e-mail. When an item of interest is posted in a SharePoint Blog or Wiki, the person posting it needs to copy a shortcut to the item, and then paste the URL into an e-mail telling the appropriate audience about the item. Yes, it’s an extra step, and it’s easier to just send an e-mail, but having the item preserved for posterity is worth the effort.
Another option isn’t in place right now, but we could “mail enable” certain SharePoint items, including Blogs. We also could begin implementing some of the Audience targeting features of SharePoint which would help with Wikis. This would allow people to add an address to an e-mail which would cause the contents of the e-mail to be automatically added as a Blog entry. Audience targeting would allow people to target Wiki entries (and other SharePoint items) to specific audiences (i.e. Agency Wide, Account Service, etc.). This option, however, would require quite a bit of setup and testing prior to implementation.
For now, we’ll focus on education and training. Hopefully, If we can get people to begin posting to our Blogs and Wikis, perhaps they’ll overcome some of those fears and this will become less of an issue.
One of the more difficult challenges we face is that of archiving our creative materials. We’re currently taking a fresh look at this process, reviewing both the process in terms of who is doing what, but also taking a fresh look at the storage media we’re using.
Our current process involves burning all of our closed jobs to DVDs. In our main office alone, we routinely archive up to 20 DVDs (approximately 100 GB) of material every month, and we have 6 other offices all facing the same issue. Burning the DVDs is actually only part of the process, although it may be the most man-hour and time intensive part.
The first step involves making a final determination about when a job is complete, and this is a step which has to be done by the Creative Department. They are ultimately the owners of this data, and once a job has been tagged as final, it is moved to a read-only network location where it remains until it gets burned to DVD. How long it remains on the network depends on our current remaining disk space, but we can usually maintain 2-3 months on our network. This is very helpful, because most requests for archived jobs occur in this time span. Having he materials readily available on the network saves us from having to retrieve from an offline DVD.
Once we need to reclaim disk space, we then begin the process of burning this material to DVDs. First, we have to break the data up into DVD-size chunks. Then, we burn it onto DVD, verifying against the original. Finally, we duplicate the DVD, catalog the DVD using DiskTracker, and send the duplicate DVD to one of our other locations for permanent offsite storage. The DiskTracker catalog is shared on our network for all users to search. This part of the process requires both time and manpower, up to 2 days per month. It’s also contentious in terms of who actually does it – in some of our offices we utilize IT personnel while in others it’s done by Creative or Studio.
Over the years we’ve constantly looked for ways to streamline and improve the process, including jukebox-based systems, but we’ve usually rejected most systems because of the expense of adding them to all of our locations.
At the moment we’re actually contemplating replacing our DVDs with hard drive based storage. We could store at least 6 months of archive materials on a 500 GB hard drive. The amount of time we could save using a hard drive based system would save us a tremendous amount of time, but it also raises a lot of issues – including costs, reliability, susceptibility to viruses, and the potential for losing more data in a single shot. Personally, it seems like most of those issues can be resolved, and we’re going to test a hard drive based solution over the coming months.
We’ll see how it goes, but it will be very interesting to see how we’re archiving and what we’re using 5-10 years from now.
I’m sleeping better at night. We’re in the middle of a major project, part of which involves consolidating way too many e-mail servers onto a single Exchange 2007 server. We never really planned to end up with as many e-mail servers as we did, but through a combination of growth and acquisitions, we somehow ended up with 6 of them servicing about 200 people. It was way too many servers for the number of users, even in a multi-office environment. We’re in the process of fixing that now, moving all of our users onto Exchange 2007. Prior to making this move, our primary server was running Exchange 2003, and the hardware was becoming shaky to the point we held our breath every time we restarted it.
I’m a big fan of Exchange. We’ve been running it since Exchange 2000 was released, and I’ve watched it become stronger with each subsequent release. This thing is a workhorse, and it’s been incredibly reliable over the years. When we have had issues with it, we’ve found Microsoft’s Exchange support to be more than equal to the task. As a result, we’ve had a solid track record of providing e-mail service to our employees.
While it’s probably a bit early for me to be singing the praises of Exchange 2007, what I’ve seen so far has proven to be incredibly useful. First, we had to move our storage databases off local storage and onto our shared iSCSI storage after we rolled out the server. (Our EqualLogic box arrived after the Exchange box.) Moving the databases was something which would have terrified me with prior releases. In fact, I’ve never even considered doing it. With Exchange 2007, we simply utilized the wizard and moved all of our databases on a Friday evening. We finished the DB moves in about an hour, and most users never even noticed an interruption in service.
Currently, we’re in the process of migrating our user’s mailboxes onto Exchange 2007. We’ve completed the move for 3 of our offices, and we’re in the process of moving 4 others. In the past, this type of migration meant a Saturday in the office moving mailboxes. With Exchange 2007, I literally do sleep through the moves. We schedule the moves overnight, and when we start the following morning everything is finished. Out of approximately 120 mailboxes moved so far, we’ve only encountered errors on two moves, and neither prevented the mailbox from moving. When they arrive in the morning, our users don’t even realize they’ve been moved.
There’s much more to Exchange 2007 than moving databases and migrating mailboxes, but so far I’ve been impressed. One of the other reasons we moved onto Exchange 2007 was the addition of Unified Messaging. That’s the next step in our implementation, and since that service is new to Exchange 2007 I’m prepared for some growing pains. However, when I consider the pain we’ve experienced with telecomm voicemail systems over the years, I’m actually looking forward to bringing this service into Exchange. Regardless of how that implementation goes, I’m sleeping better these days knowing that the foundation of my e-mail services is in good shape.
Virtualization seems to have taken the IT world by storm, and those of us in the Advertising business are along for the ride. I’ve been playing around with virtualization myself for over a year now, and I only see this becoming a bigger part of our business in the future.
It starts at the desktop level, where personally I’ve been running VMware’s Workstation product on my personal laptop. I’ve been running Vista for over a year now, and during that time Workstation has provided me with a convenient copy of XP for those apps which didn’t behave on Vista. The reason I ultimately chose VMware’s product over Microsoft’s desktop virtualization was that it included support for USB devices.
Even more important for those of us in the Agency business at a desktop level, are the possibilities for desktop virtualization the Mac side. Once again, we’ve opted to go with VMware’s Fusion product on our Intel-based Macs. We’re currently struggling with two problems on our Macs which have proven difficult to solve. We’re not huge fans of Microsoft Entourage as an e-mail client, and I could probably devote an entire rant to Microsoft’s decision to drop support for Outlook on the Mac side. We’re also experiencing problems with Mac access to our SharePoint sites. SharePoint works with Safari and Firefox, but as one would expect with a Microsoft app, it works much better with Internet Explorer. We’re hoping that both problems will be solved by actually providing our Mac users with access to Outlook and IE through VMware’s Fusion and a local copy of Windows on their machines.
On the server side, we took the plunge into VMware last year, purchasing ESX Server. We’ve gone fairly slowly in terms of virtualizing our infrastructure, but we’re currently running a SharePoint server, two utility servers, and two development servers as VMs, and we’ve become totally sold on the technology. As a result, we’ve added a second box in order to utilize Virtual Center to aid in the management and to provide load balancing. We’ve also added shared storage via our EqualLogic iSCSI SAN. It all works great, and the possibilities it provides us are endless. If there is a downside, this stuff is expensive, and VMware’s product line and licensing are pretty confusing, and that could give them a problem down the road as Microsoft’s Hyper-V product continues to mature.
We’ve got big plans for virtualization in our Agency. In addition to what we’re doing already, we’re considering virtualization for both high availability and disaster recovery. While we’ve been warned not to virtualize some things, such as domain controllers, Exchange server, and SQL server, we do feel that we can employ a virtual copy in a high-availability or disaster scenario, especially in cases where we maintain the data on a separate platform. We’re also going to explore the possibility of creating a “remote office in a box”, providing us with a quick solution which we could use in acquisitions or the opening of new offices. Our remote offices require a fairly basic setup, and it’s one which we think could be completely virtualized.
We’re going to continue on the path to virtualizing both our servers and desktops. We’ll also be taking a long look at Microsoft’s Hyper-V product. I’ll let you know how it goes.
You would think this one would be a no-brainer for those of us in Advertising. So much of Web 2.0 seems to be geared towards advertising, it just seems like everybody in this business should be experts and well-versed in everything which makes up Web 2.0. On the contrary, I am often amazed at how slowly we seem to adopt some of this stuff. Case in point, a committee consisting of IT personnel from various Agencies engaged in a discussion last week regarding Mac file problems on Windows-based file servers. How do you think this discussion took place? On an internet discussion board? Nope – All the questions and suggestions showed up in my e-mail inbox.
The biggest problem in holding a discussion using e-mail is that the messages eventually get deleted. All of the good suggestions/answers are lost forever. The reason the group uses e-mail is because discussion boards have failed miserably. Questions often go unanswered completely, or solicit very little feedback. When the same question is posed via e-mail to the group, 13 replies show up within a few short hours. E-mail works for us. Previous attempts at a discussion board have failed, so we continue to use the method that works for us.
Perhaps we haven’t utilized the correct tools for building our discussion board. I know that one of the keys for driving traffic to our own internal SharePoint site is the user alerts. If we encouraged participants to setup and manage e-mail alerts on an Ad Agency IT Forum, then new posts (or a daily summary) would still show up in our e-mail inbox. A board capable of providing RSS feeds could also help for people choosing to keep updated via that mechanism.
If anyone has an answer to this one, I’d love to hear it. Hey, at least I’m not sending you an e-mail asking for the answer.