February, 2008

It Makes Me Tear My Hair Out #1

Visa, in conjunction with the US Chamber of Commerce, has published an alert that identifies the leading causes of data breaches. Full details can be found at the Chamber’s website. The five leading causes of card-related breaches are:

1) Storage of mag stripe data
2) Missing or outdated security patches
3) Use of vendor supplied default settings and passwords
4) SQL injection
5) Unnecessary and vulnerable services on servers

Why tear my hair out? Numbers 2, 3, and 5 shouldn’t be on this list. We’ve had how many warnings and regulations and requirements about patches, default settings and unnecessary services? And business wonders why it needs regulatory requirements. Because these bad business practices happen routinely. Because too many business owners don’t want to spend the money to secure their systems.

Just four months ago I did an audit of two online MSSQL databases, only to discover their administrative SA IDs had been left in the default configuration of “no password.” Why do we keep dropping the ball? Crooks are dumb, but they’re not that dumb.

Last year I interviewed a VP development of an online marketing software for a clothing retailer. When I asked him what steps he was taking to address SQL Injection, he replied, “What’s SQL Injection?”

Well, I’ve used up my italics for the day. Sigh.

But the Chamber website has some really nice papers and templates for those looking to get started with security policies and procedures. Good for them!

Call me “Kernel” Patch

One of the junior members on my audit team likes to rag me about how often I harp on patching at various client sites. He started out by calling me “Captain Patch,” but I pointed out that I like “Kernel” much better. Why have just a nickname when you can make a really good pun with it too?

It’s easy to say, “Patch your servers.” Beware of the auditor that lays that one on you and walks off; it’s NOT a one step process. Patches can break things and breaking things in production, especially if you’re running custom software on those servers, can be a disaster of large proportions.

Some years ago at a bank I was working at, our outsourced network team started patching servers over the weekend. Unbeknownst to them, the patch updated a driver for the disk array on our Compaq servers with one that didn’t work. Install patch….no server. A server won’t boot if it can’t find the disk to load the software!

About 25 servers were “patched” before they realized the problem. Since they were in Texas, a whole lot of us got out of bed in Boston and headed to the Bank to try and discover the cause of massive server failure. That was a very long weekend.

You gotta have a plan! And it needs to be a fast one, because bad guys start reverse-engineering the code the minute the patch is released. And the plan has to test those patches to make sure everything works, before deploying to production.

“We don’t have test boxes.” Of course not. I used development boxes, (announcing it first), then after 24 hours, if nothing had broken, it went to the backup production boxes across the continent. If all went well, changes could go into production for highly critical patches in less than 48 hours. Use secondary Domain controllers, file servers, database test servers, etc. The most critical server gets patched last, but fast.

If you’re running a critical application with outsourced software, write it into the contract with the vendor that they will test patches quickly and update you so that your servers can be patched. If you sign a contract without this requirement, shame on you!

Decided not to apply a patch for Media Windows v.exty xx? OK, but who made the decision to bypass certain critical ones? You’ve got to document what didn’t get patched and why. Otherwise, you could be the one called on your vacation by a furious boss with a broken server. Or, God forbid, a hacked one.

No excuses. Figure out a plan, draw up a procedure, and save yourself major headaches.

Security by Auditor: Don’t Make Me Do It

When I go out on exams to client sites, I am often amazed that I find things in bad shape - terminated users on systems, unpatched servers, holes in firewalls, secret 5 on Cisco routers…..Why? Because it’s not rocket science. Whether it’s SOX, SAS 70 or PCI, auditors will be checking pretty much the same things.

First - Policies and procedures? Got em? Don’t make ME do it. If I write your policies (by requiring you to do it to become compliant), you have no investment or interest in implementing them or maintaining them. Write your own, and you own the procedure. I recently got an “Incident Response Policy” which had been flagrantly plagiarized from a university. The author, an admin in a service bureau, didn’t bother to read the complete document and at least Find|Replace the appropriate nouns and names. He had actually set himself up to fail with a policy that would be impossible for him to implement.

Second - Access Controls? Who is in charge of requesting adds/changes/deletes, and who completes the request? DON’T make me do it. Don’t tell me you track it by emails, because those get lost and deleted. Email is NOT an access control. Who wanted it, who approved it, and who did it? When? I’ll be asking…..

Third - System Security? Are your servers patched? Don’t make me do it. If you give me a bunch of MBSA reports, you’d better READ THEM FIRST. That way, when I start asking you why there are so many local administrators on your database server, you’ll have an answer ready….especially if those IDs belong to users who have left the company. Do you REALLY want to explain why you’re missing 37 patches? While your manager and CSO grind their teeth behind you? I understand why you haven’t applied IE7 or the Windows Malicious Software Removal Tool (which Micro$oft says is a “critical” item), but security updates labeled “critical” and are over two months old WILL require an explanation. Count on it, and better yet, get it patched.

Any good auditor will give you the test AND the answers you need to pass. If you flunk, it’s not me. Do you really want to have to implement security the way an auditor (who does NOT know your network) recommends? Please don’t make me do it. If you know what I’m going to ask for, build it in, own it, make it yours, not mine.

And, I’m sorry to say, all of the examples are based on my experience in the real world of IT.

What Makes a Good IT Auditor?

I had a great discussion today with the head of auditing for a regional bank. He talked about the need for IT Auditors to understand the systems they were auditing. But how much knowledge of technical environments should an IT Auditor have?

Quick answer: As much as possible.

I have met numerous CISAs and CISSPs that cannot audit systems without a checklist or a software tool. I’m not saying we have to know everything about each operating system; we don’t have to be engineers. (But it sure helps!) If we’re only as good as our tool, we’re doomed.

We have to know how users, groups, access controls to files and directories are created, maintained and changed. If an engineer says he can’t get something from a system for me (like a list of etc/password) I need to be able to show him what I want. Otherwise I don’t know what I’m talking about, and he can get rid of me by playing dumb.

I worked with an Oracle DBA not long ago who didn’t know about being able to turn off the TNS Listener if it wasn’t password-protected. He didn’t believe me when I said I could turn it off. So we agreed to run a test, and I turned off his TNS Listener service remotely, with no password. (Needless to say, we did this on the DEVELOPMENT server).

His attitude towards me did a total 180 degree turn. After a pregnant moment of silence over the phone, in a whole different tone of voice, he asked where I had gotten my information and what I did. From then on we had a very good relationship - he no longer treated me like a pain in the anatomy. (PS, this only works on Oracle 9i and below).

If you don’t know about this exploit, and why it’s important, read the whitepaper from here.

There’s nothing like a demonstration to clarify the point. If I hadn’t done that, he would have dismissed my finding. I frequently have to make my geek bones with engineers when I go on an audit, but I don’t mind because we can then operate from a position of mutual respect.

So many audit departments don’t have the depth to truly partner with their IT departments. It should be a match made in heaven, but it doesn’t work if one group has more knowledge than the other.

When I did a PCI exam for a Tier 1 merchant who had outsourced their IT functions, I ran an MBSA scan to determine if the systems were patched. Their outsourced vendor claimed that the servers were all patched, and he had the change control documents to prove it. I went on a sample of twenty servers and confirmed that the servers were not patched by looking in the file system, and I returned the results to him. Unfortunately, somebody got fired, but now the servers are patched and they check to confirm.

What would the response have been if I couldn’t prove the finding? That’s why IT audit needs to know systems.