Steps to an Easy Audit

When a Control is NOT a Control or, “It’s Good Enough”

I run into an awful lot of engineers who hate paperwork (I feel the same way.) They are busy fixing problems, building new application support and dealing with upper managers who have no idea what they’re asking for, clueless users and now I come along to top it off asking for a bunch of documentation.

Been there, done that.

I gently explain, after I have corrected their misapprehension that auditors know nothing about IT, that if it’s not written down, it doesn’t exist. I know some engineers who believe in job security that way, but the fact is it just makes it harder for the next person to step into that role. That role will always exist. So why make it easier for the next person? Sooner or later, that next person will be you.

Why write down how a server should be built? Why write down how the servers get patched? Why bother changing the administrator password on all the servers and a different one on all the workstations? Why check to make sure that the anti virus server is actually updating all those machines? Why test to confirm that the group policy for downloading patches is actually working, and how to do that?

It’s part of being a professional engineer. It’s part of all the certifications we have signed off on; that pesky ethical paragraph that asks us to be responsible, dedicated and at the top of our game whether the job asks for that, or more commonly, does not.

It’s also a really great way of showing just how much work you do.

“Good Enough” is short for “Good Enough to Get Hacked.”

Bottom line? When you are sitting in front of a judge testifying as to what steps were taken to secure your organization, you WILL be asked what policies, standards and procedures you were following. If you have none to give the judge, you will be roasted by the jury, and your company will lose its case.

We can blame the company for not “making” us do it, but that’s not the real deal, is it?

Six Questions to Ask A New Software Vendor

During the “sales romance,” when software vendors are showing off the bells and whistles of their product to the ooohs and aahhhs of management, it’s a challenge sometimes to be the “wet blanket” of security reality.

All too often, executives make software purchases without any regard as to regulatory requirements, security best practices or implementation costs. It’s my job (and yours) to educate and ask the hard questions, preferrably BEFORE they shell out the big bucks, then bang their heads against cost overruns due to requirements, implementation of add-ons and customized product.

Once you’ve signed the software contract, it’s way too late to ask questions. You need the sort of answers that will give you some assurance that the software vendor is committed to writing secure code, cares about writing secure code, and invests in the time and training of their staff to ensure their product can provide the right information about what their software does “under the hood.”

This should be a part of the “due diligence” organizations perform before making software purchases. It’s more important to your company, in the long run, than the financial solvency of the software company. A great company writing bad code is a higher risk than a shaky company writing excellent code.

It’s also a great time to see how mature the software company is in terms of software development practices. “Real” software companies have code storage, written documentation of process and practices, as well as defined QA testing. Beware of a company that may say they have all these things, but have nothing in writing.

So:

1. Where is security in your software development life cycle? (If they don’t know, or have to go look, be careful)
2. Do you test your software for security vulnerabilities? What do you use to test your software? Can I see the latest report? (kudos to them if you can see it. If they don’t test, you’re buying a pig in a poke)
3. If there is a database, do you have a security design for it? (Try to find out if they’ve basically opened up the database with one application ID that can do everything - cheap and easy coding. Cheap and easy to hack, too)
4. If this is a web-based application, do you provide documentation on how to install your application securely? (Watch for the ones that use a “default installation” of a web server to load their application on. It’s likely that if you secure the web server, you’ll break their application.)
5. What kind of logging and reporting does your application do? (You’re looking for good reporting on WHO, WHAT and WHEN. You’ll need this for your auditors, too)
6. How quickly will you test Microsoft patches so that we can patch? (This is critical; you don’t want to have a vulnerable server hacked because they don’t want to test, or be put off with promises of the next upgrade. Make sure they test for IIS, SQL and/or any other specialty the application uses.)

These are the kinds of questions that software vendors need to keep hearing, so that they know the customer cares about security and won’t buy a product that isn’t secure. Plus you’ll weed out hundreds of hours of pain and aggravation. It’s a “win-win.” For you.

Auditing MS SQL - Roles, and Why They Matter

SQL “Server” runs on top of MS Windows, and it has groups inside of it that are not seen on the Windows server or even the Windows Domain. That’s why we have to check and make sure that inappropriate users don’t have complete access to everything inside the database. Not everyone should be looking at those payroll files!

So, to confuse us a little more, instead of calling them groups (like Users, Administrators, Power Users, etc) SQL calls them “roles.” There are roles based on SQL as a whole (the SQL “Server”) and roles based on individual databases. Kind of like Domain Admins vs Local Admins.

SQL Server roles are equal to Domain Admins inside SQL Server. They give rights to many different core functions of SQL, and the sysadmins role has rights to everything inside SQL. By default, sa is here, but you should look very carefully at any other user in this group. DBAs like to give themselves a “backdoor” into the server this way, but that should be removed. Best practices recommend always having the DBA use their Domain ID for insertion into this role. This way you can log and monitor their access, and when they leave the company and their ID is disabled, they won’t be able to access the SQL databases. It also let’s them know that you are watching, and people tend to behave better that way.

I have found that software developers LOVE to put their names in this group, and not their Domain IDs, either. If it’s a production database, they have no business having anything other than SELECT rights anywhere. Kick ‘em out!

Here’s the SQL Server roles default permissions:

Fixed Server Roles
Sysadmin – can perform any activity (and Builtin\Administrators are part of this group by default)
Serveradmin – can set SQL-server config options and shut down SQL Server.
Securityadmin – manage logins and CREATE DATABASE permissions, read error logs and change passwords (within SQL, not Windows)
Setupadmin - manages linked servers and startup procedures
Processadmin – can manage processes running in SQL server
Dbcreator- can create alter and drop databases
Diskadmin - can manage disk files
Bulkadmin - can execute BULK INSERT statements

How do you find out who is in these roles? Use the results of a stored procedure: sp_helpsrvrolemember with the results taken from the Master database.

For individual databases, there are database roles. People in these roles can be all powerful, but only within that individual database, not all of SQL. How do you know how many databases are inside one SQL Server installation? Again, a stored procedure: sp_helpdb

You’d be surprised how DBAs like to create their own little mini databases (on production boxes!) just to “do things.” They may be great “things,” but those databases need to go somewhere else.

In order to see the members of all the database roles, run another stored procedure: sp_helprolemember run from each production database in SQL Server. It’s a little tedious, but you will get the information you need.

I commonly monitor the membership in the db_owner role (equivalent to a local server Administrator). By default, the dbo, or db_owner is the only one in this role. If the DBAs are already sysadmins, they don’t need to be in this role, unless you want to be very granular in your controls (NOT a bad idea). Developers shouldn’t be in this role, or just as bad, the application ID of an application accessing data.

This is where a lot of software development falls off the security bandwagon. If the application ID is db_owner, it means that the ID has access to everything in that database. If I can acquire the application ID’s password, (and some of them don’t have one) I can get into all the data. I wouldn’t use the application, just connect directly via ODBC or even Excel.

It’s easier to write applications using the application ID as db_owner, but unless you are using middleware to vet everyone’s login via the Windows Domain, you take the risk of losing all that confidential data out the back door.

If you can acquire this information on a quarterly basis, you will go a long way towards having an easy audit and a better night’s sleep!

FREE Tools for Auditing MS SQL Server

There’s a lot of really nice application tools to audit SQL databases out there. They have lots of bells and whistles and write out a really nice report with professional formatting. If you’ve got one of those, LUCKY YOU. But most of us Admins and Auditors have to scrounge for what we can find with the budget we’ve got (read $0).

So I always like to start out with SQLPing A nice tool from SQLSecurity.com that scans the network both actively and passively, testing SQL’s default listening port 1433 and it reports on what it finds (nothing fancy, just text). This includes those desktop versions of SQL Server: MSDE, that are often configured with SA-no password. This tool will tell you what version SQL is running, and will even do a test for SA-no password. Since SA cannot be locked out, you won’t damage the server with one attempt.

NOTE: Check with your network administrator before running this test. It’s also a great test of your intrusion detection system, because, if the IDS is configured properly, it should catch it and alert for it. Make sure management knows you are using this, if that’s what you’re going to use it for. No IDS? No worries.

I also recommend SQLSecurity. com for a lot of great information and scripts for DBAs. They know all about SQL Injection (Unlike a CIO I recently interviewed three years ago) and they have lots of MS SQL information well worth your visit.

The other free tools I use are found inside SQL Server. Yes, inside the SQL database, and they are called “Stored Procedures.” This is a fancy name for pre-written sequenced query language batch files. The folks at Microsoft have done us all a great favor by writing hundreds of them. Inside the Master database are the stored procedures you want to run, or have a DBA run for you and output to .csv format files. (Each database also has stored procedures, but the Master database SPs are the ones you want.) There is a table full of them, and here are the ones I use:

sp_helpdb Names and file locations on the Windows server of all SQL databases. (And you should find out who or what has access to those files)

sp_helpuser Review usernames, groups the user belongs to, and their default login database.

sp_helplogins a. Identify and review any external users and groups; b. Look for mappings of login name to UserOrAlias or as DB Owner. c. Check AUser and ARemote columns to identify who has remote access to what database.

sp_helpsrvrolemember Users and groups assigned to each server role (More on this later)

sp_helprotect Permissions in the database. Examine this list carefully for what rights are granted and denied to whom.

Get the results of these queries, results from the last post, and SQLPing. You’ll have some very interesting items to review. and remember, Google is your friend.

Eigen’s 2008 InfoSecurity “Rules of Thumb”

Rule #1 - You can pay now, or you can pay later, but if you choose to pay later, you will pay MORE.

Rule #2 - You can outsource function, but you cannot outsource responsibility.

Rule #3 - A classic, shamelessly plagiarized: “Faster, Better, Cheaper. Pick TWO.”

Rule #4 - Make NICE with your auditors, no matter how dumb they are.

Rule # 5 - The volume of company executives screaming about the “cost” of information security is the direct inverse of how little money they’ve put into it in the past.

Rule # 6 - Don’t expect the best audit from the cheapest bidder. You get exactly what you pay for. Unless, of course, that’s exactly what you want. See Rule #1.

Rule # 7 - Compliance with regulations is a Gentleman’s C.

Rule # 8 - If you have “checkbox security,” you will have a box full of checks. Paid to other people.

Rule # 9 - The skills of your IT people directly relate to the training they receive. See Rule #1.

Rule #10 - No more acronyms! PCMCIA.

Firewalls Part IV - Quis custodiet ipsos custodes?

Who guards the guardians? Good IT governance mandates oversight of all IT functions. The firewall tends to be neglected, because it appears to be such a back-office function that only engineers or admins actually see and work on.

However, it is one of the most critical pieces of the IT infrastructure. As a result, the following steps ought to be put in place:

1.) Administrative access - Too many shops use one ID (”admin”) and password. Each person who accesses the firewall should have a unique ID and very strong password that should be changed more frequently than the standard policy. All administrative access to the firewall should be logged, and the logs stored separately for review.

Access should be reviewed at least annually to confirm no inappropriate users have been allowed access and signed off as reviewed by management.

2.) Changes - Too often firewall rules get changed “on the fly” or as the result of a phone call from a panicked person in the board room who can’t get to his demo site or custom application. Tough nookies. All changes to the firewall rulebase should go through change controls. Emergency change controls should go further up the management tree for approval, not just the network manager. This way it’s documented as to who asked for what, who did it, and who approved. It will make people think ahead and think twice before saying, “just open it up.”

Many firewalls can be enabled to send an email every time a rule is changed, added or deleted. This should be enabled immediately so that a management person (ideally, the ISO) is notified. This will also help eliminate admins giving themselves a convenient back door into the network.

A rulebase review should be conducted quarterly with management sign-off. It helps them be aware and start to understand what a firewall is really doing for the business.

3.) Monitoring the Logs - Firewall logs are a goldmine of information. Ideally you have a system for collecting the logs for analysis and storage with a third-party application. Someone should be looking at them every day. If the organization is smart, they have a product that can store, analyze and correlate logs from servers, routers, IDSes and firewalls. One stop shopping that everyone can review - including management. (Well, you can offer!) Make sure the designated person is logged or signs off on such reviews - then your audit will be a snap.

It’s Not Your Mother’s Firewall Anymore - Part III

When all is said and done, a configuring a firewall comes down to creating a set of rules. Firewalls are bi-directional - they control traffic going out (outbound) to the Internet (or the DMZ) and they control traffic coming in (inbound) to the network or the DMZ. You are configuring for WHO, WHAT, WHERE and WHY.

WHO - in firewalls, everything is identified by IP address, whether it’s a single server or internal subnets. Firewalls have a category called “ANY,” which is a “red flag to the bull” for an IT Auditor. “ANY” is short for ANYBODY. Do you want anybody on the Internet to have access to something on your internal network? It’s sloppy work on the part of anyone who is configuring the firewall to use “ANY.” It means they didn’t take the time to specifically identify parties who need to access the network. The absolute only time the use of “ANY” is justified is when you have a web server you want customers to come to (think amazon.com, or ebay.com). And you can bet those folks don’t have their web servers inside with their corporate databases - those web servers are sitting alone in a DMZ.

“ANY” doesn’t work for outbound traffic either - you want to identify your internal subnets so that you can LOG who is going outbound and to WHERE. This is how Hannaford might have caught their hackers - the hackers were sending an outbound text file to somewhere in Europe from a store server - that could have set off an alarm. (Apologies to Hannaford; hindsight is, of course, 20-20 vision.)

WHAT - Applications on servers and other network devices respond by port number. There are 65,535 ports for UDP connections and 65,535 TCP connections. (I know this number by heart from the days when I did tech support for a firewall company.) Commonly, web servers use port 80/TCP. Theoretically, a server could be listening on all ports, but it will only listen on the ports that are running services on the server. If you are not running IIS or Apache, your server will not be listening on port 80 for connection requests.

So, here’s another place not to use ANY. Don’t allow hackers to bang away on every conceivable port. If you are only using FTP, disable web services, and allow traffic only to port 21/22. (Applications can use more than one port). Be specific about what services people can get to from the Internet.

Outbound, you may want to use the firewall to limit things like Peer-2-Peer networking, instant messaging, IRQ, video streaming, etc. There are also certain blocks of IP addresses you might want to bar access to, such as the entire netblock in Russia that hosts so many hacker applications.

WHERE - If you know WHAT, you also know WHERE. Be specific about what IP addresses can get to what on your DMZ and your internal network. If your email server in the DMZ is delivering to an internal email server, make it a rule from one IP address to the other. If clients want to connect or deliver data, request the specific IP addresses the client will be using. Will it be more work? Yes. But you will know exactly who came in, where they can go and what they are using.

Can hacker applications use standard ports? Absolutely. They often use port 80 since it is so commonly allowed in and out. Applications can be configured or coded to use non-standard ports. A good application firewall should catch some of this, and an IDS can catch the rest.

Finally WHY. I see too many configurations with rules labeled “test” or “demo,” with nothing in the comments section. Every rule should have a business owner AND a description. What are you testing? When does the test end? Who is testing what? If the cost of managing the firewall goes up because of a business application, this is your justification for getting extra funds from the business, AND a great opportunity to educate them.

Opening the firewall to let in “just the printer?” Is the printer connected to the rest of your network? Then you’ve just opened a hacker rest stop. Printers have hard drives and usually a default web server and FTP as well.

Next: Management Oversight of Firewalls

Steps to an Easy Audit (3) - Compensating Controls

These two magic words should be in every network manager and system engineer’s lexicon. It’s your get-out-of-jail (not necessarily free) card with an IT Auditor.

Every IT shop has an application, a device, a configuration that breaks good security rules and usually corporate policy, as well. Every one. Stuff gets bought and installed by another back office group that turns out to have a boatload of security holes. A legacy router in a critical location that can’t have its IOS upgraded. A bunch of wireless registers using WEP. A database that gives every user db_owner access. A firewall rule into the corporate network when everything should be going into a DMZ. (You don’t let anything into your network from the Internet, DO YOU??? Please say no.) Anyway, these issues will all come forward in an IT Audit. If they don’t, somebody should be looking into better IT Auditors.

So, if that has to happen, what controls can you put in place, that compensate for the risk of the vulnerability? Set up a firewall in front of that pesky application that Finance set up? Make that old router log connections AND remove telnet? Lock that firewall rule from IP address to IP address and require 2 factor authentication? All these things are good, but you need one more thing:

Documentation. Document the following:
1.) Your organization acknowledges the risk (identify the issue in the application, router, or firewall)
2.) Management acceptance of a break in policy (covers your assets)
3.) How those compensating controls specifically address the risk
4.) A plan and time line for removal of the vulnerability.

After this, you can hand it to the Auditor, and it will go in their report, but it can be considered reasonable.

Now, a large caveat here: Please resist the temptation to try and make a silk purse out of a sow’s ear. I’ve heard a boatload of excuses called “compensating controls” usually because someone higher up doesn’t want to spend the money. If they won’t listen to you, be honest with your IT Auditor. Management will often listen to an auditor because the report goes to the Board. Don’t take the heat for a bad management decision by backing it up with one of your own!

The second caveat is make sure you are actually doing the compensating control, whether it’s log monitoring, or firewalling, or whatever. Be prepared for the auditor wanting to review that control, just like all the others.

Some auditors might say I’m giving admins an easy out - but I don’t think so. It just means that auditors should look a little harder and know a little more about the systems they are auditing. If we all get smarter, we all get better.

Steps to an Easy Audit (2) - Where’s the Beef, ah, I mean, Data?

Remember that commercial (I’m dating myself, I know) where the little old lady lifts the top of the burger bun and says, “Where’s the beef?” All things considered, we have to ask the same sorts of questions about data.

Usually we’re looking at a nice fat application wrapped around data. It looks great, manipulates the data into all sorts of interesting reports, and adds a lot of value to the data. But ultimately, without the hamburger, the bun is useless.

Many IT Auditors and business managers tend to approach security by testing the application controls - synonymous with testing the hamburger by sampling the bun. The bun looks great, controls are all set, no one can see what they shouldn’t, right?

BIG Wrong. This is why the PCAOB is now requiring SOX auditors to examine the configuration of databases. I’m delighted to see that this is finally a requirement, because that’s where the beef is, and always will be. Inside the databases.

First question to ask: How does the application talk to the database?
Every connection to the database requires a username and password (even if the password is blank). EVERY CONNECTION. So, what is the application using? I’ve found IDs and passwords hard-coded inside applications (well, you won’t be changing the password on that one!), inside ASP code on the web server that serves up the application (hack the web server, get the database, too!) and sniffed it online using port 1433 and/or ODBC connections (IDs and passwords run in clear text). Another fun one is to examine a user’s workstation and find the ID in the ODBC configuration (pushed out via script) so that the user can use their Excel application or nice Access database code. (Of course, I’m talking Microsoft SQL server here, but these items are applicable across the variety of databases.)

Second question to ask: What rights does the application ID have?
It’s ironic that so many software companies cut expenses by enabling an application ID that has db-owner rights to the database. That way everything works, right? I’ve seen it dozens of times, and it’s always painful. Often there is no DBA at the company, or it was installed and the DBA is stuck with it. If it’s in production, removing that access will probably break the application, and no one wants that. So everyone crosses their fingers and toes that no one discovers this easy in. And, PS, hopefully there is a password to that ID.

Third question to ask: Is logging enabled to critical database tables?
Don’t believe anyone who says, “Logging can’t be turned on due to performance issues!” Sure, if you turn EVERYTHING on to be logged, the server will tank. But setting up triggers that send reports and logging access to half-a-dozen tables is not going to impact the server (unless, of course, the application is already a hog). It means more work for the DBA, but if she is skilled, it shouldn’t be a problem. A good DBA can do that in his sleep.

This way you can watch who is getting to the beef. Why does the application ID matter so much? Because if I have that ID and password, I don’t need that application to get into the database. The application is just a pretty face - I can connect via Excel, Access, or any other database-connecting application as long as I have a username and password. Just to see what I can get. And if that application ID has dbowner access, or even better, sysadmin access, I can get everything in the database.

The application may have good controls, but outside the application, or using MY application, those controls won’t exist. Beef, lots of it, sans bun. In short, hacker hamburger.

So, OK, how does this make a good audit, Eigen, you ask? Sounds like a pain in the neck, right? Two magic words for you: compensating controls.

Steps to an Easy Audit: Standardizing Patch Management

Many of my clients ask me what is the best way to deal with applications and operating systems that need to be patched frequently (like Microsoft’s monthly “Patch Tuesday”). Industry best practices have emerged in some simple steps that can work in almost any size organization:

1. Create a Patch Management Policy
Policies come from the top down and demonstrate that the executive level is invested in managing and securing the enterprise. Policies dictate the scope of standards (“Once a month, we will review patches and document risk, etc, etc”) and procedures (“the patches will be tested on server A, deployed on production servers C,D,E…”) to deploy the patches. This Policy should be part of your overall Vulnerability Management. It’s part of managing risk.

2. Identify What Needs to Be Patched
Don’t just think Microsoft, or even just think servers. Every device on your network, at some point or another will need to be updated. Some devices may require multiple types of updates, such as a server that holds a database; both need to be patched. You need to know your entire inventory, and categorize those devices in terms of risk, so that you can prioritize patching. Some devices, such as your workstations, will need to have software patched, such as Microsoft Office. It will be worth your while to have a complete software and hardware inventory. Think of the SQL slammer worm: it paralyzed networks not only because of SQL Server, but because of all the default installations of MSDE (Microsoft Desktop Database Edition) on workstations.

3. Classify and Rank
Different devices on your network may have different levels of risk, so that patching them takes higher priority than say, patching a printer (yes, these must be updated, too!). Take the time to assess the devices according to their risk levels so that you know what must be patched first.

Different patches have different levels of risk. Those that Microsoft, for example, identifies as “critical,” should always be assessed and tested for deployment first. All patches should be reviewed for criticality and ranked. That allows testing and deployment to proceed efficiently. Lower priority patches can be deployed on a more moderate timeline.

4. Test, Test, Test
It certainly is true that Microsoft, in particular, has vastly improved its process for creating solid software updates that don’t, generally, break things. However, good process requires that you not make changes on a production system without ensuring that the patch will not break some critical procedure. Use your development environment as a test bed, or even your less critical production servers to rollout the patches to. You’ll be glad you did.

5. Document, Document, Document
What got patched when? How will you remember the reason you didn’t patch a particular server in six months? Develop a documentation system, or better yet, use your Help Desk system to document who, what, when and why. You can track compliance very effectively with those reports and not an extra cent.

6. Trust, but Verify
On more than one audit, I have discovered that the great piece of software the client set up to patch everything stopped working, or didn’t patch certain other systems. Once a quarter, run a check using MBSA (Microsoft Baseline Security Analyzer, a FREE tool from Microsoft) for Microsoft products, at the very least. It will also give you a good baseline on the security of those servers, and a better night’s sleep!