Whenever the subject comes up of logging activity in a database, immediately the complaints of “Too much overhead!” can be heard. Everybody thinks it’s a good idea in theory, but from a practical standpoint, it adds a lot of burdens to the database.

From a security standpoint, it’s really difficult to make sure that DBAs or Administrators are accurately logged AND denied access to the logs. On the database server itself, it’s next to impossible.

This isn’t really a new idea, but it has recently gained a lot of adherents: database monitoring. Quest Software has had some good products around for monitoring performance, but recently the focus (because of compliance, big surprise) has turned to access controls, logging, and monitoring activity.

For example, someone might have noticed a little sooner at Countrywide that someone was accessing a lot of customer data if a Database Activity Monitoring device had been installed.

There are two versions of this type of device. First, is the Network-based DAM, which can monitor all traffic going to and from the database server, and puts no load on the server itself. This is a great idea, unless, of course, your traffic is encrypted. Another issue is that this type of monitoring will miss activity that is local to the server itself.

Second is the host-based DAM, which is really the most effective of the two, because it can see everything you want to see via an agent installed on the server that reports back to the monitoring device elsewhere on the network. The overhead of an agent will not be as high as trying to enable auditing within the database itself, and, as much as I am not fond of agent software, in this case I would make an exception, after careful testing.

The drawback to this system is that the agent could be disabled, but the DAM should immediately alert personnel to that fact. If you are able to size your server appropriately, an agent’s overhead could be minimized. I’d love to hear from anyone using this type of configuration, and how they like it.

So, to confuse us a little more, instead of calling them groups (like Users, Administrators, Power Users, etc) SQL calls them “roles.” There are roles based on SQL as a whole (the SQL “Server”) and roles based on individual databases. Kind of like Domain Admins vs Local Admins.

SQL Server roles are equal to Domain Admins inside SQL Server. They give rights to many different core functions of SQL, and the sysadmins role has rights to everything inside SQL. By default, sa is here, but you should look very carefully at any other user in this group. DBAs like to give themselves a “backdoor” into the server this way, but that should be removed. Best practices recommend always having the DBA use their Domain ID for insertion into this role. This way you can log and monitor their access, and when they leave the company and their ID is disabled, they won’t be able to access the SQL databases. It also let’s them know that you are watching, and people tend to behave better that way.

I have found that software developers LOVE to put their names in this group, and not their Domain IDs, either. If it’s a production database, they have no business having anything other than SELECT rights anywhere. Kick ‘em out!

Here’s the SQL Server roles default permissions:

Fixed Server Roles
Sysadmin – can perform any activity (and Builtin\Administrators are part of this group by default)
Serveradmin – can set SQL-server config options and shut down SQL Server.
Securityadmin – manage logins and CREATE DATABASE permissions, read error logs and change passwords (within SQL, not Windows)
Setupadmin – manages linked servers and startup procedures
Processadmin – can manage processes running in SQL server
Dbcreator- can create alter and drop databases
Diskadmin – can manage disk files
Bulkadmin – can execute BULK INSERT statements

How do you find out who is in these roles? Use the results of a stored procedure: sp_helpsrvrolemember with the results taken from the Master database.

For individual databases, there are database roles. People in these roles can be all powerful, but only within that individual database, not all of SQL. How do you know how many databases are inside one SQL Server installation? Again, a stored procedure: sp_helpdb

You’d be surprised how DBAs like to create their own little mini databases (on production boxes!) just to “do things.” They may be great “things,” but those databases need to go somewhere else.

In order to see the members of all the database roles, run another stored procedure: sp_helprolemember run from each production database in SQL Server. It’s a little tedious, but you will get the information you need.

I commonly monitor the membership in the db_owner role (equivalent to a local server Administrator). By default, the dbo, or db_owner is the only one in this role. If the DBAs are already sysadmins, they don’t need to be in this role, unless you want to be very granular in your controls (NOT a bad idea). Developers shouldn’t be in this role, or just as bad, the application ID of an application accessing data.

This is where a lot of software development falls off the security bandwagon. If the application ID is db_owner, it means that the ID has access to everything in that database. If I can acquire the application ID’s password, (and some of them don’t have one) I can get into all the data. I wouldn’t use the application, just connect directly via ODBC or even Excel.

It’s easier to write applications using the application ID as db_owner, but unless you are using middleware to vet everyone’s login via the Windows Domain, you take the risk of losing all that confidential data out the back door.

If you can acquire this information on a quarterly basis, you will go a long way towards having an easy audit and a better night’s sleep!

So I always like to start out with SQLPing A nice tool from SQLSecurity.com that scans the network both actively and passively, testing SQL’s default listening port 1433 and it reports on what it finds (nothing fancy, just text). This includes those desktop versions of SQL Server: MSDE, that are often configured with SA-no password. This tool will tell you what version SQL is running, and will even do a test for SA-no password. Since SA cannot be locked out, you won’t damage the server with one attempt.

NOTE: Check with your network administrator before running this test. It’s also a great test of your intrusion detection system, because, if the IDS is configured properly, it should catch it and alert for it. Make sure management knows you are using this, if that’s what you’re going to use it for. No IDS? No worries.

I also recommend SQLSecurity. com for a lot of great information and scripts for DBAs. They know all about SQL Injection (Unlike a CIO I recently interviewed three years ago) and they have lots of MS SQL information well worth your visit.

The other free tools I use are found inside SQL Server. Yes, inside the SQL database, and they are called “Stored Procedures.” This is a fancy name for pre-written sequenced query language batch files. The folks at Microsoft have done us all a great favor by writing hundreds of them. Inside the Master database are the stored procedures you want to run, or have a DBA run for you and output to .csv format files. (Each database also has stored procedures, but the Master database SPs are the ones you want.) There is a table full of them, and here are the ones I use:

sp_helpdb Names and file locations on the Windows server of all SQL databases. (And you should find out who or what has access to those files)

sp_helpuser Review usernames, groups the user belongs to, and their default login database.

sp_helplogins a. Identify and review any external users and groups; b. Look for mappings of login name to UserOrAlias or as DB Owner. c. Check AUser and ARemote columns to identify who has remote access to what database.

sp_helpsrvrolemember Users and groups assigned to each server role (More on this later)

sp_helprotect Permissions in the database. Examine this list carefully for what rights are granted and denied to whom.

Get the results of these queries, results from the last post, and SQLPing. You’ll have some very interesting items to review. and remember, Google is your friend.

MS SQL Server is one of the easiest systems for an auditor to review. I have a script I use (when they let me ON the server) or hand off to the DBA/Network Administrators and watch them run it. The most difficult part is sorting all the data into something readable. I always ask for the output in .csv format, which I can then port to Excel.

SQL Server always has a core of databases installed by default. A “tempdb” database used by SQL during its operations, a “sample” database, Northwind (which you can delete, by the way), a “model” database, and the critical one (to auditors and IT Security) the Master database. I do a thorough audit of this database before I move to any other databases installed by applications.

The Master database holds the core functions and controls of MS SQL. SQL has its own users and groups, that are created outside of the Windows server and any Windows Domain the server might belong to. You won’t see them in the local server administration; only the “user” SQL installs to run the SQL services. You can add Windows users and groups to SQL users and groups, and this causes a lot of confusion amongst auditors and non-DBAs.

For auditing purposes, we can keep it simple. But I do want to add one important note first: None of the queries and stored procedures I’m going to discuss with you do ANYTHING in SQL other than READ. All of the procedures use the SELECT command, which is equivalent to READ in SQL.

This is really important in auditing, as the first rule is always: Do No Harm to the System. Always! And it means I don’t accept excuses from DBAs who are afraid of “adding more stress” to the server. Reading a table takes a nanosecond.

Tables in the Master database hold information about the management of SQL as a whole. There is a table in the Master database called syslogins. In SQL 2005 it’s called sysxlogins. This contains information about all users that have a login ID to SQL, regardless of what database they access. (And you want to make sure that very few can log into the Master database).

This is where you will find out about the “sa” ID, which is the equivalent to “administrator” inside MS SQL. You won’t see this ID anywhere in the Windows operating system, but it is critical to examine inside SQL. It can be disabled in SQL 2005, but NOT SQL 2000.

So, here’s what we use: SELECT sid,name,password,dbname FROM syslogins If you can, grab it off your SQL system (even a development box) to get a sense of the structure.

I look for several things in the results of the query, but the two most important are: Who has access and do all the users have a password?

Yes, a password. I am STILL running across production systems that use the sa ID with no password. You might as well write HACK ME across your forehead, poke yourself with a pointed stick and resign. Developers still do this in their coding because it’s easy and fast. You get what you pay for.

Under the Password column in the results, you should see gobbledygook or the word NULL. If there are users and groups from a Windows Domain, they will be identified by the domain name or server name and then the ID, thus: Server\Name (for a local account) or Domain\Name. These IDs always have NULL in the password column, because SQL expects them to be authenticated by Windows.

I look for a user ID with NULL in the password column. And then I test to confirm the ID is blank by using ODBC to connect to the SQL server. I am regularly horrified, I am sorry to say.

Examine the list of users for IDs that have been created inside SQL. DBAs and developers are notorious for creating “backdoor” IDs that are not part of the Windows domain and only exist inside SQL. The problem is, when they leave, those IDs don’t get deleted or disabled when you disable or delete their Domain ID. They just sit there, waiting to be used and abused. Again, I’m regularly horrified, I’m sorry to say.

Finally, the query results will show you each user ID’s default database. Make sure IDs only have access to the database they need.

Next: MS SQL internal groups

Eight million people stayed at 1,312 locations from 2007. Is this “Identity Theft?” It’s a darned nice start. Only the Social Security number is missing. Certainly the names, addresses, business information, details of employment, credit card numbers and expiration dates could be used for synthetic identity theft.

According to the Herald:

“The Sunday Herald understands that a hacker from India – new to the world of cyber-crime – succeeded in bypassing the system’s security software and placing a Trojan virus on one of the Best Western Hotel machines used for reservations. The next time a member of staff logged in, her username and password were collected and stored.”

One of the first things I learned doing penetration testing was that you don’t have to have some fancy piece of coding to break in. It can be the simplest thing – finding a set of keys in someone’s desk – that gets you into the server room. In fact, it usually IS the simplest thing. Their web site may have great security, but that was easily bypassed by a user login.

Best Western evidently had not noticed all the activity that account was generating – sucking all the data out of their databases. Which takes us back to auditing databases, doesn’t it?

Best Western’s response? Tim Wade, head of marketing for Best Western GB, said it was “unlikely” that whoever was responsible got hold of the details of “every booking at every hotel” in Europe because of the way their system worked. Has anyone mentioned to Best Western that letting a marketing guy handle communications for a data breach is not always the best choice? “Unlikely” is not a word that I find comforting. What are the facts? Why don’t they know exactly how much was taken? Because they probably don’t have any security logging in the right place. It’s why they didn’t notice the breach in the first place.

Let’s hope they didn’t get all the way into the American side of the company. Or maybe they have. How would we know?

They are the motherlode of the organization and the last line of defense in a hack. It’s critical that DBAs have the tools at their disposal to monitor and provide reporting. If your database isn’t secure, the hacker won’t care how well indexed it is.

And there are a lot of ways in. If I have administrative access to the server, I can copy all the database files, take them away and reload them on my own database server. If I have unencrypted backups of those files, I can do the same thing.

So the first step in auditing the database is to examine the server the database is running on. This gets confusing to non-DBAs and auditors because many of the terms used inside the database are similiar to server terms. It’s important to keep them separate, and to make sure that access to the database files on the server is monitored. Server administrators do not need to have access to those files, but they may have to, in order to manage/backup the server. So, set up logging.
Make sure everyone who has a need to access that server administratively has a unique ID. Remove access to root(*NIX) or Administrator (Windows). They can have administrative rights, just make sure you can identify them by ID and IP connection.

Finally, what about the backup tapes? If they are not encrypted, you can join the “breach list” of companies that have lost their data when tapes were misplaced, stolen, or “disappeared.”

NEXT: Inside the Database “Server”

Applications aren’t something you just run on one PC anymore (I know I’m preaching to the choir, here). Financial applications, especially, are all networked, and the storage is usually a relational database like Oracle, MS SQL, Sybase, DB2 or MySQL. Relational databases are wonderful for business because you can correlate so many different facts.

So why are they so scary to me? Because they are rarely audited.

I need a network ID to log in, so the database is safe, right? No.

The application has security controls, so my database is safe, right? No.

DBAs (Database Administrators) know exactly what I am talking about here. All those items are just the outer edge of security. If I have a network jack and a database ID and password, I can bypass those controls easily.

Some applications have a database ID and no password, or an easy-to-guess password. And very frequently, that ID has access to everything, including reads, writes and deletes.

If I have that ID and a network jack, I can log into your database using ODBC, Microsoft’s Open DataBase Connection client software that is installed by default on Windows operating systems. I can use Excel, Access, or other database software to pull all your data out.

Or change your data.

And P.S., connecting with ODBC uses clear text usernames and passwords, which is how I once captured a DBA’s ID and password with a sniffer.

Fortunately for all of us, there are usually other financial controls that can capture errors or changes in the database. Usually.

NEXT: How to Audit Databases

The story is that the thief had access to the Countrywide (a mortgage broker) set of databases, which, of course, held all sorts of private financial information. A treasure trove, in fact, for anyone seeking a quick buck on the Internet. Countrywide is owned by Bank of America, and I have to wonder if they had done a third-party vendor review anytime recently, or had relied on Countrywide telling them everything was secure (which lots of Banks do, despite the Office of the Comptroller of the Currency telling them NOT to do that).

According to Countrywide, “The thief took advantage of a lapse in policy.” What interesting language. What policy, exactly, and what defines a “lapse?” Sounds like nobody was really paying attention to database access. Did Countrywide or Bank of America discover the thief? No, the FBI did.

What would it have taken to catch the thief in the act? Given that the thief was a “senior analyst,” it means that controls would have to be really specific. Let’ s brainstorm a little bit, because we need to start thinking this way. Too often, insider access is left wide open, and excuses are made that “it’s too time intensive” or “it takes resources away from the server.” Those excuses will no longer hold in court of law.

Think about it: if your organization offers up those excuses, the judge will jump all over you. If you have done background checks and are monitoring access, a lot of time and money will not go to lawyers. Demonstrating “due diligence” with regard to your employees and your data is very effective.

So, how could we monitor that kind of data? Two thoughts occur to me: first, only allow the employee to access records he works directly with, and require approval for access to any other records. This won’t rule out collusion, but it will make it harder for a single thief.

Second, log use of flash drives. This could be “silent” logging, but you could put two and two together, if the databases were also logging access.

How would YOU catch him?

Have you ever been in the position of having a software vendor say: “We’re not going to test that patch yet, you’ll have to wait for the next software release from us. If you patch it, we won’t support it.”

Or finding a security flaw in the application, reporting it to the vendor, and having them say they will charge your company to fix it as a “feature request.”

Or examining roles and rights in the database, and finding out everyone is sysadmin. Or better yet, the developer hardcoded his ID into the application.

I bet you have, and you know I have. Once the software is installed and in production, they have you over a barrel and they know it. Time to build a better barrel.

Time after time, I’ve found software applications that don’t secure the application user inside the database (giving that user rights to EVERYTHING). Why? Because it’s easier to code. You don’t have to spend time finding out what broke and fixing it when you lock user rights down. Some applications hardcode usernames and passwords right into the software so that it can never be changed (unless, of course, you pay for an upgrade). Even worse, I’ve seen it when the ID is hardcoded with a blank password. Why? It’s fast, cheap and easy.

How do YOU change it? Two ways:

First, raise management awareness so that you are at the table with the software salespeople to ask some hard questions. Is security part of their SDLC (Software Development LifeCycle)? Management can often be “wowed” by a product without ever looking under the hood. Ask how their product is secured, especially since it will probably be holding important data. Don’t be wowed by application level controls – get some hard answers on how the data is accessed.

Second, be there at contract time. This is the most important. Make sure it is written into the contract that they will fix all security flaws found in their product within 30 days. Make sure that they are responsible for testing OS patches quickly and reporting to you if it is OK to patch. Pick a timeline you can live with. After all, you’re paying them for a service.

If not, you’ll have to live with buggy code and I’ll have to audit it. We’re in this together.

Usually we’re looking at a nice fat application wrapped around data. It looks great, manipulates the data into all sorts of interesting reports, and adds a lot of value to the data. But ultimately, without the hamburger, the bun is useless.

Many IT Auditors and business managers tend to approach security by testing the application controls – synonymous with testing the hamburger by sampling the bun. The bun looks great, controls are all set, no one can see what they shouldn’t, right?

BIG Wrong. This is why the PCAOB is now requiring SOX auditors to examine the configuration of databases. I’m delighted to see that this is finally a requirement, because that’s where the beef is, and always will be. Inside the databases.

First question to ask: How does the application talk to the database?
Every connection to the database requires a username and password (even if the password is blank). EVERY CONNECTION. So, what is the application using? I’ve found IDs and passwords hard-coded inside applications (well, you won’t be changing the password on that one!), inside ASP code on the web server that serves up the application (hack the web server, get the database, too!) and sniffed it online using port 1433 and/or ODBC connections (IDs and passwords run in clear text). Another fun one is to examine a user’s workstation and find the ID in the ODBC configuration (pushed out via script) so that the user can use their Excel application or nice Access database code. (Of course, I’m talking Microsoft SQL server here, but these items are applicable across the variety of databases.)

Second question to ask: What rights does the application ID have?
It’s ironic that so many software companies cut expenses by enabling an application ID that has db-owner rights to the database. That way everything works, right? I’ve seen it dozens of times, and it’s always painful. Often there is no DBA at the company, or it was installed and the DBA is stuck with it. If it’s in production, removing that access will probably break the application, and no one wants that. So everyone crosses their fingers and toes that no one discovers this easy in. And, PS, hopefully there is a password to that ID.

Third question to ask: Is logging enabled to critical database tables?
Don’t believe anyone who says, “Logging can’t be turned on due to performance issues!” Sure, if you turn EVERYTHING on to be logged, the server will tank. But setting up triggers that send reports and logging access to half-a-dozen tables is not going to impact the server (unless, of course, the application is already a hog). It means more work for the DBA, but if she is skilled, it shouldn’t be a problem. A good DBA can do that in his sleep.

This way you can watch who is getting to the beef. Why does the application ID matter so much? Because if I have that ID and password, I don’t need that application to get into the database. The application is just a pretty face – I can connect via Excel, Access, or any other database-connecting application as long as I have a username and password. Just to see what I can get. And if that application ID has dbowner access, or even better, sysadmin access, I can get everything in the database.

The application may have good controls, but outside the application, or using MY application, those controls won’t exist. Beef, lots of it, sans bun. In short, hacker hamburger.

So, OK, how does this make a good audit, Eigen, you ask? Sounds like a pain in the neck, right? Two magic words for you: compensating controls.