Database security

The Purpose of Audit

Bruce Schneier’s last cryptogram contained a discussion about the purpose of audit. He was commenting on the fact that Barack Obama’s phone records, passport file and aunt’s immigration status was inappropriately accessed by employees of the State Department, Immigration and Verizon employees.

Because of good audit controls, the State Department electronic monitoring alerted supervisors when information was inappropriately accessed. Verizon fared less well, and Immigration has no idea who accessed the information.

“Audit helps ensure that people don’t abuse positions of trust.” Too bad Countrywide didn’t have such alarms in place to catch the guy siphoning off information to sell. Or the guy who walked out the building with hundreds of thousands of dollars of hardware over the course of 10 years.

With hard statistics this year that insiders, either by ignorance or malfeasance, have been a large source of data breaches, having good audit trails and controls in place makes more and more sense.

With so many large databases out there holding such private information, how can we continue to pretend that it only happens to other businesses? And complaints about the cost of security just aren’t cutting it anymore. The incredible COST of a data breach just keeps rising.

Pointing fingers and saying the other guy should be responsible for security doesn’t work either. Ultimately, responsibility rests with those who have the data to safeguard the data - no matter what form it takes: inside a database, on a backup tape, on a laptop, on a web server.

If we’re going to use personal information to make money for our business, we’d better be prepared to protect that information - from ourselves and other employees.

Getting What You Pay For…..2008

In my travels as an auditor this year, I’ve visited 15 states and seen approximately 20 different networks, both LAN and WAN. I’ve audited hospitals, lotteries, racetracks, banks, small businesses, large online retailers, metal fabricators, telco service bureaus and health care service bureaus.

I continue to see networks that are not patched. “It might break our custom code,” is the most common excuse, followed by, “Gee, we just didn’t get around to it.”

Software coding continues to be a security disaster in the making. Developers continue to open up databases by giving too many rights to users and application IDs. I still find individual developer IDs inside production databases.

Management continues to be unwilling to invest the money in a secure architecture. In the last three years, I can count on the fingers of one hand the organizations I’ve seen that follow secure best practices. And not use all the fingers.

I still hear people try to tell me that they don’t need a firewall because they have really good routers. And then they don’t update the IOS on the routers and/or leave the default SNMP strings in place.

If you are paying for these services, and you are getting the above, there is a problem waiting to happen on your network. If you don’t know what’s going on in your databases, time to find out before another Countrywide happens in your back yard.

Have a safe holiday. And remember: who is responsible for good security? You are. I am. Let’s keep trying to do it right.

Thank you, Federal Trade Commission…

For saying the blindingly obvious:

“Companies and schools should find new ways to authenticate the identities of customers, employees and students that do not involve social security numbers, a U.S. consumer protection agency said on Wednesday as part of recommendations to fight identity theft.”

Now here is the real challenge: could the FTC, a government agency, please communicate this point with Medicare? You, know, the government agency that puts the social security number on the medical benefits card it requires members to carry? The report addresses the use in the “private sector,” but medical use of social security numbers is a huge factor in medical identity theft, synthetic identity theft, and plain ol’ identity theft.

The FTC released the report on December 17, 2008, and you can read it here. All 21 pages of it in double space.

The “Social Security Number” was created in 1936 for the purpose of tracking workers’ earnings for benefits purposes. Not as a universal identifier. Any good DBA will tell you that only using one “identifier” predicates a high risk of false positives. Newer techniques, such as full name, address, date of birth, place of birth, etc, as a group predicate a much more accurate positive response (”Yes, this is the right person”).

But this additional data is “out there” as well, along with social security numbers. The genie IS out of the bottle.

The report worries about social security numbers data already being out of control. Given how many databases are out there (public and private) with ALL of the above information in storage, I think it is already way out of control, and the other identifying data along with it. Daily reports from the “Breach Blog” saturate my email box. Reading Pogo Was Right only confirms my opinion.

The FTC report seems to be an exercise in “too little, too late.”

Auditing MS SQL - Roles, and Why They Matter

SQL “Server” runs on top of MS Windows, and it has groups inside of it that are not seen on the Windows server or even the Windows Domain. That’s why we have to check and make sure that inappropriate users don’t have complete access to everything inside the database. Not everyone should be looking at those payroll files!

So, to confuse us a little more, instead of calling them groups (like Users, Administrators, Power Users, etc) SQL calls them “roles.” There are roles based on SQL as a whole (the SQL “Server”) and roles based on individual databases. Kind of like Domain Admins vs Local Admins.

SQL Server roles are equal to Domain Admins inside SQL Server. They give rights to many different core functions of SQL, and the sysadmins role has rights to everything inside SQL. By default, sa is here, but you should look very carefully at any other user in this group. DBAs like to give themselves a “backdoor” into the server this way, but that should be removed. Best practices recommend always having the DBA use their Domain ID for insertion into this role. This way you can log and monitor their access, and when they leave the company and their ID is disabled, they won’t be able to access the SQL databases. It also let’s them know that you are watching, and people tend to behave better that way.

I have found that software developers LOVE to put their names in this group, and not their Domain IDs, either. If it’s a production database, they have no business having anything other than SELECT rights anywhere. Kick ‘em out!

Here’s the SQL Server roles default permissions:

Fixed Server Roles
Sysadmin – can perform any activity (and Builtin\Administrators are part of this group by default)
Serveradmin – can set SQL-server config options and shut down SQL Server.
Securityadmin – manage logins and CREATE DATABASE permissions, read error logs and change passwords (within SQL, not Windows)
Setupadmin - manages linked servers and startup procedures
Processadmin – can manage processes running in SQL server
Dbcreator- can create alter and drop databases
Diskadmin - can manage disk files
Bulkadmin - can execute BULK INSERT statements

How do you find out who is in these roles? Use the results of a stored procedure: sp_helpsrvrolemember with the results taken from the Master database.

For individual databases, there are database roles. People in these roles can be all powerful, but only within that individual database, not all of SQL. How do you know how many databases are inside one SQL Server installation? Again, a stored procedure: sp_helpdb

You’d be surprised how DBAs like to create their own little mini databases (on production boxes!) just to “do things.” They may be great “things,” but those databases need to go somewhere else.

In order to see the members of all the database roles, run another stored procedure: sp_helprolemember run from each production database in SQL Server. It’s a little tedious, but you will get the information you need.

I commonly monitor the membership in the db_owner role (equivalent to a local server Administrator). By default, the dbo, or db_owner is the only one in this role. If the DBAs are already sysadmins, they don’t need to be in this role, unless you want to be very granular in your controls (NOT a bad idea). Developers shouldn’t be in this role, or just as bad, the application ID of an application accessing data.

This is where a lot of software development falls off the security bandwagon. If the application ID is db_owner, it means that the ID has access to everything in that database. If I can acquire the application ID’s password, (and some of them don’t have one) I can get into all the data. I wouldn’t use the application, just connect directly via ODBC or even Excel.

It’s easier to write applications using the application ID as db_owner, but unless you are using middleware to vet everyone’s login via the Windows Domain, you take the risk of losing all that confidential data out the back door.

If you can acquire this information on a quarterly basis, you will go a long way towards having an easy audit and a better night’s sleep!

FREE Tools for Auditing MS SQL Server

There’s a lot of really nice application tools to audit SQL databases out there. They have lots of bells and whistles and write out a really nice report with professional formatting. If you’ve got one of those, LUCKY YOU. But most of us Admins and Auditors have to scrounge for what we can find with the budget we’ve got (read $0).

So I always like to start out with SQLPing A nice tool from SQLSecurity.com that scans the network both actively and passively, testing SQL’s default listening port 1433 and it reports on what it finds (nothing fancy, just text). This includes those desktop versions of SQL Server: MSDE, that are often configured with SA-no password. This tool will tell you what version SQL is running, and will even do a test for SA-no password. Since SA cannot be locked out, you won’t damage the server with one attempt.

NOTE: Check with your network administrator before running this test. It’s also a great test of your intrusion detection system, because, if the IDS is configured properly, it should catch it and alert for it. Make sure management knows you are using this, if that’s what you’re going to use it for. No IDS? No worries.

I also recommend SQLSecurity. com for a lot of great information and scripts for DBAs. They know all about SQL Injection (Unlike a CIO I recently interviewed three years ago) and they have lots of MS SQL information well worth your visit.

The other free tools I use are found inside SQL Server. Yes, inside the SQL database, and they are called “Stored Procedures.” This is a fancy name for pre-written sequenced query language batch files. The folks at Microsoft have done us all a great favor by writing hundreds of them. Inside the Master database are the stored procedures you want to run, or have a DBA run for you and output to .csv format files. (Each database also has stored procedures, but the Master database SPs are the ones you want.) There is a table full of them, and here are the ones I use:

sp_helpdb Names and file locations on the Windows server of all SQL databases. (And you should find out who or what has access to those files)

sp_helpuser Review usernames, groups the user belongs to, and their default login database.

sp_helplogins a. Identify and review any external users and groups; b. Look for mappings of login name to UserOrAlias or as DB Owner. c. Check AUser and ARemote columns to identify who has remote access to what database.

sp_helpsrvrolemember Users and groups assigned to each server role (More on this later)

sp_helprotect Permissions in the database. Examine this list carefully for what rights are granted and denied to whom.

Get the results of these queries, results from the last post, and SQLPing. You’ll have some very interesting items to review. and remember, Google is your friend.

Inside the Database Server - MS SQL

The first question to answer is: “Is the SQL system patched?” You or a DBA can confirm this inside Enterprise Manager (the software client that runs on SQL or from a remote installation of it) by right-clicking the primary database icon and selecting Properties. You can also run a query inside Query Analyzer “SELECT @@ VERSION” and make sure the latest service packs from Microsoft are installed. No excuses! Make sure it’s updated. MS SQL doesn’t have a lot of patches and updates (unlike the operating system) but they are important.

MS SQL Server is one of the easiest systems for an auditor to review. I have a script I use (when they let me ON the server) or hand off to the DBA/Network Administrators and watch them run it. The most difficult part is sorting all the data into something readable. I always ask for the output in .csv format, which I can then port to Excel.

SQL Server always has a core of databases installed by default. A “tempdb” database used by SQL during its operations, a “sample” database, Northwind (which you can delete, by the way), a “model” database, and the critical one (to auditors and IT Security) the Master database. I do a thorough audit of this database before I move to any other databases installed by applications.

The Master database holds the core functions and controls of MS SQL. SQL has its own users and groups, that are created outside of the Windows server and any Windows Domain the server might belong to. You won’t see them in the local server administration; only the “user” SQL installs to run the SQL services. You can add Windows users and groups to SQL users and groups, and this causes a lot of confusion amongst auditors and non-DBAs.

For auditing purposes, we can keep it simple. But I do want to add one important note first: None of the queries and stored procedures I’m going to discuss with you do ANYTHING in SQL other than READ. All of the procedures use the SELECT command, which is equivalent to READ in SQL.

This is really important in auditing, as the first rule is always: Do No Harm to the System. Always! And it means I don’t accept excuses from DBAs who are afraid of “adding more stress” to the server. Reading a table takes a nanosecond.

Tables in the Master database hold information about the management of SQL as a whole. There is a table in the Master database called syslogins. In SQL 2005 it’s called sysxlogins. This contains information about all users that have a login ID to SQL, regardless of what database they access. (And you want to make sure that very few can log into the Master database).

This is where you will find out about the “sa” ID, which is the equivalent to “administrator” inside MS SQL. You won’t see this ID anywhere in the Windows operating system, but it is critical to examine inside SQL. It can be disabled in SQL 2005, but NOT SQL 2000.

So, here’s what we use: SELECT sid,name,password,dbname FROM syslogins If you can, grab it off your SQL system (even a development box) to get a sense of the structure.

I look for several things in the results of the query, but the two most important are: Who has access and do all the users have a password?

Yes, a password. I am STILL running across production systems that use the sa ID with no password. You might as well write HACK ME across your forehead, poke yourself with a pointed stick and resign. Developers still do this in their coding because it’s easy and fast. You get what you pay for.

Under the Password column in the results, you should see gobbledygook or the word NULL. If there are users and groups from a Windows Domain, they will be identified by the domain name or server name and then the ID, thus: Server\Name (for a local account) or Domain\Name. These IDs always have NULL in the password column, because SQL expects them to be authenticated by Windows.

I look for a user ID with NULL in the password column. And then I test to confirm the ID is blank by using ODBC to connect to the SQL server. I am regularly horrified, I am sorry to say.

Examine the list of users for IDs that have been created inside SQL. DBAs and developers are notorious for creating “backdoor” IDs that are not part of the Windows domain and only exist inside SQL. The problem is, when they leave, those IDs don’t get deleted or disabled when you disable or delete their Domain ID. They just sit there, waiting to be used and abused. Again, I’m regularly horrified, I’m sorry to say.

Finally, the query results will show you each user ID’s default database. Make sure IDs only have access to the database they need.

Next: MS SQL internal groups

European Hotel Chain Has Their Customer Data For the Past Year Accessed

Visited Europe in the last year and used a Best Western Hotel? Your credit card, expiration date, the company that employs you, your name, address and future bookings may be in the possession of a Russian Mafia website. An enterprising Scottish newspaper, the Sunday Herald, noticed on Thursday night that an Indian hacker offered to sell access to Best Western and notified Best Western about the breach. Although Best Western closed the hole on Friday, the horse is out of the barn.

Eight million people stayed at 1,312 locations from 2007. Is this “Identity Theft?” It’s a darned nice start. Only the Social Security number is missing. Certainly the names, addresses, business information, details of employment, credit card numbers and expiration dates could be used for synthetic identity theft.

According to the Herald:

“The Sunday Herald understands that a hacker from India - new to the world of cyber-crime - succeeded in bypassing the system’s security software and placing a Trojan virus on one of the Best Western Hotel machines used for reservations. The next time a member of staff logged in, her username and password were collected and stored.”

One of the first things I learned doing penetration testing was that you don’t have to have some fancy piece of coding to break in. It can be the simplest thing - finding a set of keys in someone’s desk - that gets you into the server room. In fact, it usually IS the simplest thing. Their web site may have great security, but that was easily bypassed by a user login.

Best Western evidently had not noticed all the activity that account was generating - sucking all the data out of their databases. Which takes us back to auditing databases, doesn’t it?

Best Western’s response? Tim Wade, head of marketing for Best Western GB, said it was “unlikely” that whoever was responsible got hold of the details of “every booking at every hotel” in Europe because of the way their system worked. Has anyone mentioned to Best Western that letting a marketing guy handle communications for a data breach is not always the best choice? “Unlikely” is not a word that I find comforting. What are the facts? Why don’t they know exactly how much was taken? Because they probably don’t have any security logging in the right place. It’s why they didn’t notice the breach in the first place.

Let’s hope they didn’t get all the way into the American side of the company. Or maybe they have. How would we know?

How to Audit Databases: Part I

Databases are enormous, powerful repositories of data. They can hold payroll, HR personnel data (think social security numbers) stock prices, Accounts Receivable, Client Relationship Management, and customer information. Banks can’t live without them. Most medium and many small sized businesses use them, too.

They are the motherlode of the organization and the last line of defense in a hack. It’s critical that DBAs have the tools at their disposal to monitor and provide reporting. If your database isn’t secure, the hacker won’t care how well indexed it is.

And there are a lot of ways in. If I have administrative access to the server, I can copy all the database files, take them away and reload them on my own database server. If I have unencrypted backups of those files, I can do the same thing.

So the first step in auditing the database is to examine the server the database is running on. This gets confusing to non-DBAs and auditors because many of the terms used inside the database are similiar to server terms. It’s important to keep them separate, and to make sure that access to the database files on the server is monitored. Server administrators do not need to have access to those files, but they may have to, in order to manage/backup the server. So, set up logging.
Make sure everyone who has a need to access that server administratively has a unique ID. Remove access to root(*NIX) or Administrator (Windows). They can have administrative rights, just make sure you can identify them by ID and IP connection.

Finally, what about the backup tapes? If they are not encrypted, you can join the “breach list” of companies that have lost their data when tapes were misplaced, stolen, or “disappeared.”

NEXT: Inside the Database “Server”

I Can Make Your Database Lie to You

So many financial auditors, CEOs, CFOs and others rely on electronic data to understand the complexities of General Ledger, Accounts Payable, etc. In this era of SAP, ADP, electronic time clocks, etc., the one common denominator is the database underlying each application.

Applications aren’t something you just run on one PC anymore (I know I’m preaching to the choir, here). Financial applications, especially, are all networked, and the storage is usually a relational database like Oracle, MS SQL, Sybase, DB2 or MySQL. Relational databases are wonderful for business because you can correlate so many different facts.

So why are they so scary to me? Because they are rarely audited.

I need a network ID to log in, so the database is safe, right? No.

The application has security controls, so my database is safe, right? No.

DBAs (Database Administrators) know exactly what I am talking about here. All those items are just the outer edge of security. If I have a network jack and a database ID and password, I can bypass those controls easily.

Some applications have a database ID and no password, or an easy-to-guess password. And very frequently, that ID has access to everything, including reads, writes and deletes.

If I have that ID and a network jack, I can log into your database using ODBC, Microsoft’s Open DataBase Connection client software that is installed by default on Windows operating systems. I can use Excel, Access, or other database software to pull all your data out.

Or change your data.

And P.S., connecting with ODBC uses clear text usernames and passwords, which is how I once captured a DBA’s ID and password with a sniffer.

Fortunately for all of us, there are usually other financial controls that can capture errors or changes in the database. Usually.

NEXT: How to Audit Databases

Monitoring Insider Access to Databases

The recent report on the Countrywide data theft got me thinking again about how to monitor insider access to databases.

The story is that the thief had access to the Countrywide (a mortgage broker) set of databases, which, of course, held all sorts of private financial information. A treasure trove, in fact, for anyone seeking a quick buck on the Internet. Countrywide is owned by Bank of America, and I have to wonder if they had done a third-party vendor review anytime recently, or had relied on Countrywide telling them everything was secure (which lots of Banks do, despite the Office of the Comptroller of the Currency telling them NOT to do that).

According to Countrywide, “The thief took advantage of a lapse in policy.” What interesting language. What policy, exactly, and what defines a “lapse?” Sounds like nobody was really paying attention to database access. Did Countrywide or Bank of America discover the thief? No, the FBI did.

What would it have taken to catch the thief in the act? Given that the thief was a “senior analyst,” it means that controls would have to be really specific. Let’ s brainstorm a little bit, because we need to start thinking this way. Too often, insider access is left wide open, and excuses are made that “it’s too time intensive” or “it takes resources away from the server.” Those excuses will no longer hold in court of law.

Think about it: if your organization offers up those excuses, the judge will jump all over you. If you have done background checks and are monitoring access, a lot of time and money will not go to lawyers. Demonstrating “due diligence” with regard to your employees and your data is very effective.

So, how could we monitor that kind of data? Two thoughts occur to me: first, only allow the employee to access records he works directly with, and require approval for access to any other records. This won’t rule out collusion, but it will make it harder for a single thief.

Second, log use of flash drives. This could be “silent” logging, but you could put two and two together, if the databases were also logging access.

How would YOU catch him?