I had a recent client frustrated by reboots. It seemed that a couple times a day, the individual computers in a custom cabinetry shop somehow lost connection to the internet. Often a reboot of the system would fix the problem. Then an hour later or maybe a day later, the problem would reoccur. This didn’t happen on just one computer, but on every computer in the organization. Not at the same time, but infrequently and unpredictably to computers across the company. In any given 15 minute period someone had rebooted their system somewhere in the organization.
The accounting department was most frustrated, especially during payroll. Payroll is always a hectic time of the month. Imagine rebooting your computer 5 or six times a day while going through the payroll process. The manager of the accounting group couldn’t predict when any accounting process would be done. The owner was accepting, the employees were troopers, but the laws around payroll are not so forgiving. Payroll checks need to be done on time.
Each time the IT Support vendor was called, the company was billed… but the problem was never truly solved. His strategy was to reboot the computer, get the problem computer online, then walk out the door. Then when the next failure occurred, he would be called again.
After a two hour audit of the systems, I discovered that the DNS had been setup incorrectly. That weekend we fixed the DNS problem and nobody had to reboot their computers again. It’s interesting to note that after the problem was finally resolved things quickly changed. The next month the company made an extra $30,000. (A 33% increase in monthly income.) That $30,000 was pure profit with no additional overhead expense. IT costs had gone down and overtime paid to work staff went down of course. The change was that the daily disruption to the workstations was no longer occurring.
As a Seattle IT Consultant, I’ve seen this type of thing happen over and over again. The DNS failure wasn’t a technical failure but rather ignorance on the part of the technician who setup the DNS. This ignorance shows up everywhere. As the technology becomes more and more complicated this ignorance shows up more frequently. I use the term technical glass ceiling to describe the ignorance. In the early days of technology there used to be one technical expert who knew everything about the network. As the network grow and become more complicated it is now impossible to know everything. When a technician reaches their technical glass ceiling they’ve reached the limit of their full understanding and begin putting in “work-a-rounds”. These work-a-rounds are seldom scalable so are like ticking time bombs. Eventually when the business outgrows the work-a-round the time bomb goes off and the network fails.
If the network had remained small, the network would have been fine. The problem is that companies grow. This means the network must grow. So too must the knowledge exchange about the network and the technology.