One of the junior members on my audit team likes to rag me about how often I harp on patching at various client sites. He started out by calling me “Captain Patch,” but I pointed out that I like “Kernel” much better. Why have just a nickname when you can make a really good pun with it too?
It’s easy to say, “Patch your servers.” Beware of the auditor that lays that one on you and walks off; it’s NOT a one step process. Patches can break things and breaking things in production, especially if you’re running custom software on those servers, can be a disaster of large proportions.
Some years ago at a bank I was working at, our outsourced network team started patching servers over the weekend. Unbeknownst to them, the patch updated a driver for the disk array on our Compaq servers with one that didn’t work. Install patch….no server. A server won’t boot if it can’t find the disk to load the software!
About 25 servers were “patched” before they realized the problem. Since they were in Texas, a whole lot of us got out of bed in Boston and headed to the Bank to try and discover the cause of massive server failure. That was a very long weekend.
You gotta have a plan! And it needs to be a fast one, because bad guys start reverse-engineering the code the minute the patch is released. And the plan has to test those patches to make sure everything works, before deploying to production.
“We don’t have test boxes.” Of course not. I used development boxes, (announcing it first), then after 24 hours, if nothing had broken, it went to the backup production boxes across the continent. If all went well, changes could go into production for highly critical patches in less than 48 hours. Use secondary Domain controllers, file servers, database test servers, etc. The most critical server gets patched last, but fast.
If you’re running a critical application with outsourced software, write it into the contract with the vendor that they will test patches quickly and update you so that your servers can be patched. If you sign a contract without this requirement, shame on you!
Decided not to apply a patch for Media Windows v.exty xx? OK, but who made the decision to bypass certain critical ones? You’ve got to document what didn’t get patched and why. Otherwise, you could be the one called on your vacation by a furious boss with a broken server. Or, God forbid, a hacked one.
No excuses. Figure out a plan, draw up a procedure, and save yourself major headaches.