May, 2008

Steps to an Easy Audit: Standardizing Patch Management

Many of my clients ask me what is the best way to deal with applications and operating systems that need to be patched frequently (like Microsoft’s monthly “Patch Tuesday”). Industry best practices have emerged in some simple steps that can work in almost any size organization:

1. Create a Patch Management Policy
Policies come from the top down and demonstrate that the executive level is invested in managing and securing the enterprise. Policies dictate the scope of standards (“Once a month, we will review patches and document risk, etc, etc”) and procedures (“the patches will be tested on server A, deployed on production servers C,D,E…”) to deploy the patches. This Policy should be part of your overall Vulnerability Management. It’s part of managing risk.

2. Identify What Needs to Be Patched
Don’t just think Microsoft, or even just think servers. Every device on your network, at some point or another will need to be updated. Some devices may require multiple types of updates, such as a server that holds a database; both need to be patched. You need to know your entire inventory, and categorize those devices in terms of risk, so that you can prioritize patching. Some devices, such as your workstations, will need to have software patched, such as Microsoft Office. It will be worth your while to have a complete software and hardware inventory. Think of the SQL slammer worm: it paralyzed networks not only because of SQL Server, but because of all the default installations of MSDE (Microsoft Desktop Database Edition) on workstations.

3. Classify and Rank
Different devices on your network may have different levels of risk, so that patching them takes higher priority than say, patching a printer (yes, these must be updated, too!). Take the time to assess the devices according to their risk levels so that you know what must be patched first.

Different patches have different levels of risk. Those that Microsoft, for example, identifies as “critical,” should always be assessed and tested for deployment first. All patches should be reviewed for criticality and ranked. That allows testing and deployment to proceed efficiently. Lower priority patches can be deployed on a more moderate timeline.

4. Test, Test, Test
It certainly is true that Microsoft, in particular, has vastly improved its process for creating solid software updates that don’t, generally, break things. However, good process requires that you not make changes on a production system without ensuring that the patch will not break some critical procedure. Use your development environment as a test bed, or even your less critical production servers to rollout the patches to. You’ll be glad you did.

5. Document, Document, Document
What got patched when? How will you remember the reason you didn’t patch a particular server in six months? Develop a documentation system, or better yet, use your Help Desk system to document who, what, when and why. You can track compliance very effectively with those reports and not an extra cent.

6. Trust, but Verify
On more than one audit, I have discovered that the great piece of software the client set up to patch everything stopped working, or didn’t patch certain other systems. Once a quarter, run a check using MBSA (Microsoft Baseline Security Analyzer, a FREE tool from Microsoft) for Microsoft products, at the very least. It will also give you a good baseline on the security of those servers, and a better night’s sleep!

Five Myths About Compliance

Compliance: The state of conformity of a regulated party (including a corporation, institution, individual or other legal entity) with a legislative or regulatory requirement or a recognized standard.

1. If we’re compliant, that means we’re secure.

Would that this were so! Unfortunately, the letter of the law is usually far less than the spirit of the law. If Management isn’t invested in securing the entire network, they often settle for compliance with existing laws and regulations. Having a secure environment takes a lot of resources, time, and investment in infrastructure processes. Compliance is only the beginning.

2. We have {enter name of software here} so we’re compliant.

I’m sorry, but there is no such software. There’s plenty that give out some really pretty reports and provide some oversight, but that’s very different from securing an enterprise. Usually, multiple layers of software and good people are required.

3. Our vendors tell us they’re compliant, so our information is secure. If they get broken into, it won’t impact us.

Ultimately, it’s YOUR data, whether it resides on a third-party’s web site or inside your network. If they are broken into and your data is stolen, that is what the newspaper will report. Your customers will blame YOU for not monitoring the vendor more closely. You can outsource functions, but not responsibility.

4. Everybody in my organization is trustworthy.

The “fraud triangle” consists of the following elements: Incentive, opportunity and capability. Anyone can find incentive if the opportunity and capability exists. Insider attacks are far more effective than external hackers. Organizations of any size that don’t do internal monitoring are asking for trouble - they’ve provided opportunity, incentive is plentiful and often the hack takes very little capability. If people know they are being monitored, they will think twice.

5. Compliance only happens once a year.

If you have strong security controls, policies and procedures in place, your auditors won’t spend half as much time on site, the audit will not require hours and hours of extra work and you can be quite pleased with yourself at the end of the day. Isn’t it great when you give the auditors what they’re looking for and they go away? If all that stuff is working, you have a secure environment and an audit is a non-event.

Tips for Admins: How (NOT) to Have an Good IT Audit

Over the years, I’ve gotten used to the people I “visit” trying really hard not to make faces when I’m introduced. Nobody likes to see an auditor roll in the door. I try to make it as easy as possible, and whatever I can to fit into the schedules of busy engineers and managers. But I’ve also gotten used to some tell-tale signs that the audit is not going to go well:

Don’t prepare any information in advance and tell me you’re very busy
We send out requests for information a month in advance and offer custom scripts to help get the information easily. It’s usually information you should have at your fingertips - user lists, MBSA scans, router configurations, etc. Database queries take a little more time. I know you’re really busy - what admin isn’t? When I don’t get any information, it doesn’t make you look busy, it makes you look incompetent.

Don’t answer my emails or phone calls
If your manager has told you to do this, route my requests directly to him, and cc me. Then you’re off the hook and your manager can look bad. That’s what they’re there for. If you’re just avoiding me, well, see the note above.

Be condescending about technical issues
Yes, I know IT Auditors don’t know your systems as well as you do, and we never will. We have to ask for dumb things. Be patient and tolerant, and we’re much more likely to be helpful.

Don’t allow my laptop on your network because “it’s a security issue.”
Please don’t embarrass yourself this way. A competent engineer can route us directly out the firewall without ever touching the network. This statement means you’re either incompetent, lazy, or hiding something. Not to mention the fact that I’ve been vetted, ’scoped and checked across multiple continents AND my company has a boatload of liability insurance. I break it, I own it. Smile, your network is safe due to your competence, isn’t it? Make it look easy.

Stonewall giving me access to critical systems because “you might break something.”
Other than questioning my technical competence, (thanks!) it tells me that you’re afraid I’m going to find something you don’t want me to see. Truly secure and resilient systems can recover from almost anything an admin can do to them. If your systems aren’t that secure or able to fail over, acknowledge it upfront. We’ll work out what I need to see together.

Lie outright.
I can count on the fingers of one hand (and not use all the fingers) the systems I’ve seen where the engineers and managers have been proud to walk me through and show me what they are doing. I love being “wowed.” I don’t get that very often, and I really enjoy seeing a well run network.

I’ve also had engineers take me aside and reveal security issues they were concerned about that weren’t being addressed, and I keep those sources as confidential as I can. If you tell me where the problems are, then I know you are not the problem. If you are losing sleep over some issue, share the pain - I can lose sleep, too. You can use an auditor’s report to get management to pay attention to security issues.

Make the most of my visit. Ask lots of questions. Understand why I’m asking what I’m asking for. It will make your job easier, and I’ll be out of your hair sooner. And who knows, you might want to be an IT Auditor someday. You’d probably be really good at it, because you would know where to look.