Deleting the data from the database is the easy part, a simple delete loop will handle that nicely.  The problem is when you delete rows from a table which contains LOB data the LOB pages aren’t cleared when they are deallocated.  We can see this by running the following code.

CREATE DATABASE Lobtest
GO
use Lobtest
GO
CREATE TABLE t1 (c1 int IDENTITY(1,1) PRIMARY KEY, c2 ntext)
GO
INSERT INTO T1 (c2) VALUES (replicate('a', 20000))
GO
DBCC IND ('LobTest', 't1', 1)
GO
DBCC TRACEON(5201, -1)
GO
DELETE FROM t1
GO
DBCC IND ('LobTest', 't1', 1)
GO
DECLARE @dbid as int = db_id('Lobtest')
DBCC PAGE (@dbid, 1, 231, 3)
GO

You can see that page 231 is a LOB page which is allocated to the table t1. When you look at the actual page using DBCC PAGE after the row has been deleted we can see that there is data in the page, and that the page header shows that the page is still allocated to the table t1. This can be seen by looking in the header of the page for the header value labeled “Metadata: ObjectId = 245575913″.

When you go to shrink the database the SQL Server engine will get to the LOB pages and it will need to figure out if the LOB row is a part of a row which still exists or not. In order to do this SQL Server will need to scan through the pages which make up the table looking for any rows which reference the page it is trying to delete.

When doing shrinks after deleing large amounts of LOB data SQL Server will generate large amounts of IO while figuring this out and the shrink operation will take an extremely long time. (Paul Randle talks more about it here.)

So the question that this person at SQL Saturday had was, how can I reclaim the space from my database within a reasonable time.

The solution that we came up with was actually pretty simple. Do the database deletion as normal. Then backup and restore the database. Then do the shrink, followed by rebuilding the clustered indexes in order to fix the fragmentation issue which the shrink will introduce.

This works for a pretty simple reason, because the PFS page shows that the LOB page isn’t allocated even though the page is full of data (you can verify this by looking at page 1 in file 1 in the sample database created by the script above). When the database engine backups up the database the database engine looks at the PFS pages to figure out which pages to back up. Because the PFS pages show that the pages are empty the database engine doesn’t bother to backup the pages, so when the pages are restored they are restored as blank pages. This means that after the restore the shrink operation can run without an issue.

In the case of this application there was a maintenance window which could be taken advantage of which would allow the backup and the restore to happen.

Another option which we came up with which would require less downtime involved using database mirroring. By configuring database mirroring (which is initialized via a backup and restore process giving us the same basic approach) and then failing over to the mirror we would end up in the same position. We could then shrink the database without issue (probably pausing database mirroring so that we didn’t have to wait for the second server to process the shrink in real time) and then fail back the database to the original server.

As geeky as it was, Mladen, Andre and I had a great time figuring this out, and the attendee had a great time watching us go through all the possible options as we excluded them one by one. And most importantly she got her problem solved.

So if you end up in this situation here’s a solution that will help you shrink the database so that you can reclaim the space that the LOB data pages are taking up without having to wait forever.

Denny

Some background

For those that don’t know what session state is, let me take a step back and fill you in.  Session state allows the web developer to store values in variables up on the web server, but in a shared location so that it doesn’t matter what web server the end user connects to the values are all there and the web servers understand the session information from each other.  This is very important for shopping websites, or pretty much any site with a login prompt and a load ballancer that is configured to send the users connection to the next available web server.

What is the problem?

The problem becomes that most people when they need to scale out session state opt to put the session state information into SQL Server because that is a nice easy repository that is probably already up and running, not to mention that the session state code in ASP.NET easily supports it.  The issue with putting the session state information into SQL Server is that you don’t give a crap about persisting the session state information to disk.  The information doesn’t get persisted more than an hour or so, and if the information within the database is lost, the only impact is that any values in session variables is lost.

So what are the options?

There are a few options besides using SQL Server for session state.

1. Keep using the in-process option.
2. Use the ASP.NET Session State Service

The easiest option is to keep using the in-process option in IIS for session information.  This means that you will need to configure what are called sticky sessions on the load balancer so that the user always goes to the same web server every time.  In the event that a web server fails, all the session information for those users would be lost, and they would need to start over by logging back into the site, or they would have an empty shopping cart, or whatever the site does.

The ASP.NET Session State Service is a Windows service that provides a memory only session state repository so that session information isn’t ever written to disk, and it isn’t kept on the web server.  You can either stand up a dedicated machine for this, or setup a couple and cluster the service manually so that you have an HA solution for your session state service.  If you have an existing SQL Server cluster you can even use this cluster for it, if you don’t have anywhere better to put it.  Just configure the cluster to have the SQL Server run on one node as the preferred node, and have the session state run on another node as its preferred node.  This way the services won’t ever run on the same node unless the other node is offline.  The session state service doesn’t take a lot of CPU power, and the amount of RAM it needs to be completely dependent on the amount of information that you are stuffing into session variables on the web servers.

In either case, either solution is better than putting the session state information into a SQL Server database.  The information doesn’t need to be written to disk, ever.  The information that is written into the session state database is in a blob binary form, not any sort of relational form so you can’t really do anything with it.

How do I know if sessions state is in SQL?

That is an easy one.  You’ll have some funky database, probably in simple recovery mode usually name aspnetdb (maybe with a prefix and/or a suffix).

What other problems can session state cause?

Well first it’ll take away buffer pool resources from your other databases.  Because the session state database is hit very hard the data from the database will always be in memory.  How ever much data is in the session state database, you are missing that much buffer pool space for your other databases.  Because of the way that session state works, every single time a page is clicked on the website, at least one call is made to the session state database pretty much forcing the database server to keep the data from the database in the buffer pool.

Another problem that you can see if ghost records.  Paul Randal describes ghost records perfectly on his blog post Ghost cleanup in depth.

When a record is deleted, apart from it being marked as a ghost record, the page that the record is on is also marked as having ghost records in one of the allocation maps – the PFS page (post coming soon!) – and in its page header. Marking a page as having ghost records in a PFS page also changes the database state to indicate that there are some ghost records to cleanup – somewhere. Nothing tells the ghost cleanup task to clean the specific page that the delete happened on - yet. That only happens when the next scan operation reads the page and notices that the page has ghost records.

He also talks about how to fix the problem in the same blog post, so I’ll leave you to click through for all that information.

Another problem goes back to the fact that the SQL Server will be persisting everything to disk, and with the information in this database changing all the time, as the website grows you’ll need faster and faster disks under the session state database just to keep up.  As the site grows more popular you will end up spending more and more money on faster and faster disks, just to keep the session state working, much less anything else on the SQL Server.  You may even get to the point where you actually need a dedicated SQL Server just to run the session state database.

Another problem is good old locking and blocking.  SQL Server likes to take page level locks when it does insert, update and delete operations.  Well, unless each page on disk only has a single row in it you are going to have processes being blocked for short periods of time as other users session state information is updated.  You can work around this to some extent by hacking the session state database’s stored procedures and forcing row locks, but now you are taking more locks (and more memory for locks), etc.

How do I change to the session state server?

First you need to install the service on the server that will be your session state server.  On Windows 2008 just install all the .NET components and that should do the trick.  You’ll probably want to start the service as well.  When you bring up the list of services it is the one called “ASP.NET State Service”.

Then on each web server you’ll want to change the session state information. You can either use IIS Manager to do this, or change the web.config. However you configured session state the first time, that’s how you’ll want to change it this time.

To use the IIS Manager select the website (or application) to configure and double click on the “Session State” icon on the right (shown below).

From there select “State Server” from the list and change “localhost” to the server you’ll be using for session state. If you have changed the TCP port number from 42424 to something else you can adjust that here as well.

To set the session state setting via the web.config file find the existing session state information and edit it, or add in the session state information. Set the “mode” to “StateServer” and set the “stateConnectionString” to the same value that goes in the IIS config setting.

<configuration>
  <system.web>
    <sessionState mode="StateServer"
      stateConnectionString="tcpip=SampleStateServer:42424"
      cookieless="false"
      timeout="20"/>
  </system.web>
</configuration>

Hopefully I’ve convinces you to move your session state information out of SQL Server and into a repository that it actually belongs in.

Denny

After the blogger in question (Peter) removed all his SQL content (no idea if it was all plagiarized or just some was and taking it all down was easier) everything settled down a bit.

Later that day Todd McDermid posted a blog post giving a bit of an overview of today’s events and his opinions about the plagiarism and our reaction to it.

Before I continue I want to say that I fully respect Todd and his opinions on the subject, even though I don’t agree with them. Anyone willing to sit around and write for other people’s education for little or no money deserves to be respected.

Now as for the DMCA style take down nasty grams (and I like that way better then take down notices for some reason) there are other issues at play than just our giving information away to everyone.

For example, if you dig through Microsoft’s TechNet and MSDN sites there’s plenty in there which says that you can’t republish the information posted without written consent.

“Unless otherwise specified, the Services are for your personal and non-commercial use. You may not modify, copy, distribute, transmit, display, perform, reproduce, publish, license, create derivative works from, transfer, or sell any information, software, products or services obtained from the Services.” *Stolen directly from the TechNet ToS which I found by clicking the Terms of Use link at the bottom of every page on the TechNet site.

As for my articles which were plagiarized, the company which paid for those articles to be written (I’m assuming that it comes as no shock to anyone that people are paid to write articles for websites and magazines) owns the copyright on those articles for as long as they choose to enforce them. Some companies hold a lifetime exclusive on the articles, while others hold a shorter one.

Either the blogger in question or not was aware of the law on plagiarism or not, but he states on his website that he’s a college graduate. Pretty much every college will toss you out for plagiarizing other peoples works, and the students are well aware of it. He should have assumed that in the professional world that plagiarizing was still unacceptable.

Unfortunately for Peter there is a lot of information out there which can be freely read and referenced, but not copied. This blog would be included in that. For those of you reading this on SQLServerPedia.com it took a decent amount of work to get permission to post my blog posts up on the SQLServerPedia.com blog feed. Why did it take so long? Well because I don’t own the copyright to these blog posts. I write on behalf of Tech-Target and they own the copyright to anything and everything which I post on my blog, unless I have posted it somewhere else first (and then they still own the rights to the version on their site).

Would I like to have everyone be able to read what I write? The big reason that I spend so much time writting (especially this last month) is so that people can read it. I have information to share, and hopefully people find the information that I have useful. But a the copyright ownership of information has to be respected for web based articles just like it does for SQL Server Magazine, TechNet Magazine, etc.

These are my thoughts and opinions, on today’s events. Take them for what they are worth, god knows I’m no copyright lawyer. Please feel free to post your own here, on twitter, on Todd’s post (no registration required on his blog, unlike here) or your own blog if you have one.

Denny