As a business grows there is often a rebellion in IT departments.  On the one hand, IT technicians become overworked.  What I call IT cowboys enjoy the opportunity to wander the network arbitrarily fixing problems and playing the “lone wolf” on the network.  As the business grows though more and more “lone wolves” wander the network they begin stepping all over each other’s work.  In other words without understanding the changes or discoveries another lone wolf … err… technician has made, there is a risk.  The risk is that the change will break the fix of another technician made earlier. 

In a previous posting I discussed how IT people need to be Team players.  In another I discussed the supporting operational roles to the Incident and Problem management roles.  In this article I’d like to discuss a problem I recently ran into on a client site.  As an IT Consultant, I run into these problems all the time.  The simplest solution is to re-organize the department.

Two common symptoms of poor IT business behavior:  

No Root Cause analysis -

Without a problem management team, the Incident team will end up troubleshooting and fixing problems over and over again.  If the company pays $10 to resolve a ticket, each time the incident reoccurs the company pays another $10 for the ticket resolution.  It’s not unusual for tickets to re-occur hundreds of times in a year’s period.  This increases the cost of incident support per problem from $10 to $10 times the hundreds of times the problem goes through the incident process again and again.

Symptom: runaway costs fixing the same problem over and over again.

No Change Management -

A common problem I see is a technician coming in to fix a problem.  After the technician leaves, the customer finds a new problem or even worse an old problem has come back.  The incident team is called and a new ticket is open for each error.  Each technician continues to work on the same problems over and over again.

Symptom: runaway cost fixing the same problem over and over again.  This becomes compounded because while technicians fix one problem, they are breaking an old fix or creating new problems.  Incident problems on the network begin to snowball.

The most important thing to understand is that accountability.  As in any business department there must be a way to track and verify problems.  Managers are taught to first assign tasks, then second follow-up on those tasks.  This follow-up maintains the integrity of the business system.  Too often IT is seen as black magic.  Therefore the second management task does not occur.  Without follow-up IT experts setup their own priorities.

For example making the CPA and the controller the same person is asking for trouble.  Who follows up on the CPA if the CPA is the controller?  The same is true for when making the Incident manager and the problem manager the same person.  Making them the same person reduces accountability of the owner of the two roles.  By building in accountability, technical problems are reduced in the IT department in the same way financial errors and bankruptcy is reduced when a controller role is added to the accounting department.

This accountability is something the old “Lone Wolf” type of technician never had to deal with.  Yet as the business grows, this type of behavior actually damages the real value of the organization.  As a Seattle IT Consultant I work with IT departments and business owners to reorganize departments to avoid these types of conflicts between technicians. 

I am getting very tired of Systems Administrators that blame their users.  Once again I found myself between an IT Services vendor and their client.  As a Seattle IT Consultant I am often called in to help an owner fix their network.  Most IT Services Vendors are anxious when I walk in the door, thinking I may be trying to steal their clients.  The good ones find themselves with more work because I hand out a lot of referrals. I’m not an IT Services vendor in the traditional sense.  I work with clients to sit on their side of the table.  I translate the technical speak so common in our industry, into a language business owners understand.

Statistically thirty Percent of network system errors are caused by hardware failure.  If you’ve been in IT you know that most hardware errors occur in the first couple months the system is setup.  Once the system is running and doing what it’s supposed to, hardware failures are very rare.

Statistically we know that 80% of businesses report consistently IT failures

Once the system is up and running whose fault is it?  What I’ve found is that it’s usually not the user.  Yet when I’m called in, I find that more often than not the users in the organization are being blamed.  I’ve seen one extreme example after another.  Where users are being blamed, but the real problem is the system technician, system administrator or the systems architect.  It’s very easy to blame the system failure on technology.  The technology is never offended.  The technology vendor is always willing to take the blame, if it means you buy more hardware.  The problem is that once the hardware is replaced, if the system continues to fail, you know it isn’t the hardware.  So eventually technology vendors begin blaming the users.  After all who else is there that could be causing the problem.

Statistically 70% of IT failures is due to human error. 

If so, human error is always user error isn’t it?  The reality is that hardware and software vendors have made technology so bullet proof from the user, that users really can’t bring down a network any longer.  Not when that network is setup in its default configuration from the manufacturer.  The reality is that most network failures are not cause by users.  They are caused by the poor planning of Systems experts. 

There are a lot of advantages to the cloud.  There is the advantage gained by removing the ongoing cost of server replacement from the organization’s capital expenditures budgets.  In a Break/fix network environment most networks function between 85 and 95% availability.  By improving Availability from 95 to 99.9% availability organizations improve workplace productivity.  Workplace productivity losses average around $7,000 / hour for small businesses and $50,000 / hour for larger businesses and can be millions of dollars a minute for the largest enterprise organizations.  Recovering 4.9% or more of workplace productivity can mean the difference between red and black accounting ink.

There’s another unspoken reason for the cloud though.  That’s to take technology out of the hands of the average system administrator.  Today’s small business system administrator is a “Jack of all trades” master of none.  Small businesses can’t afford an entire IT department.  Instead they hire someone who is good at everything.  In the cloud, teams of technical experts are hired with deep technical knowledge in every technology.  These teams are working in a highly organized network.  Most IT Support vendors are still tracking tickets in their head or on paper.  There are managed services organizations that manage 1000′s of clients.  Then there are cloud providers who manage literally Millions of clients at a per user cost that is much less and provide 3, 4 and even more 9′s of availability to clients.

The reality is that the cloud is possible because todays small business systems administrators are statistically the real reason for most IT Failures.  These administrators don’t even admit it and charge their customers for fixing their failures.  These system’s people think they are fooling their clients.  They do know and are looking for some way to protect themselves.

Want to stay in business?  My recommendation is never blame the user… even if it is their fault.  Design systems that are bulletproof, highly available and highly scalable.  It’s probably too late though because the cloud is coming.