I opened them both up in SSMS and the code for the procedures was identical (as you would expect), but there was something different.  The procedure that had the really high CPU % times was compiled with the SET QUOTED_IDENTIFIER ON setting, while the procedure that had the really low CPU % times was compiled with the SET QUOTED_IDENTIFIER OFF setting.

I have no idea why there was a difference, but I changed the one which was ON to OFF and pushed the procedure into the database.  As soon as I did the CPU % for that procedure dropped down into the single digit range where it should have been.

Let this be a lesson, those setting definitely matter.  And don’t trust that they are correct, even in Microsoft provided code like Session State.

Denny

Some background

For those that don’t know what session state is, let me take a step back and fill you in.  Session state allows the web developer to store values in variables up on the web server, but in a shared location so that it doesn’t matter what web server the end user connects to the values are all there and the web servers understand the session information from each other.  This is very important for shopping websites, or pretty much any site with a login prompt and a load ballancer that is configured to send the users connection to the next available web server.

What is the problem?

The problem becomes that most people when they need to scale out session state opt to put the session state information into SQL Server because that is a nice easy repository that is probably already up and running, not to mention that the session state code in ASP.NET easily supports it.  The issue with putting the session state information into SQL Server is that you don’t give a crap about persisting the session state information to disk.  The information doesn’t get persisted more than an hour or so, and if the information within the database is lost, the only impact is that any values in session variables is lost.

So what are the options?

There are a few options besides using SQL Server for session state.

1. Keep using the in-process option.
2. Use the ASP.NET Session State Service

The easiest option is to keep using the in-process option in IIS for session information.  This means that you will need to configure what are called sticky sessions on the load balancer so that the user always goes to the same web server every time.  In the event that a web server fails, all the session information for those users would be lost, and they would need to start over by logging back into the site, or they would have an empty shopping cart, or whatever the site does.

The ASP.NET Session State Service is a Windows service that provides a memory only session state repository so that session information isn’t ever written to disk, and it isn’t kept on the web server.  You can either stand up a dedicated machine for this, or setup a couple and cluster the service manually so that you have an HA solution for your session state service.  If you have an existing SQL Server cluster you can even use this cluster for it, if you don’t have anywhere better to put it.  Just configure the cluster to have the SQL Server run on one node as the preferred node, and have the session state run on another node as its preferred node.  This way the services won’t ever run on the same node unless the other node is offline.  The session state service doesn’t take a lot of CPU power, and the amount of RAM it needs to be completely dependent on the amount of information that you are stuffing into session variables on the web servers.

In either case, either solution is better than putting the session state information into a SQL Server database.  The information doesn’t need to be written to disk, ever.  The information that is written into the session state database is in a blob binary form, not any sort of relational form so you can’t really do anything with it.

How do I know if sessions state is in SQL?

That is an easy one.  You’ll have some funky database, probably in simple recovery mode usually name aspnetdb (maybe with a prefix and/or a suffix).

What other problems can session state cause?

Well first it’ll take away buffer pool resources from your other databases.  Because the session state database is hit very hard the data from the database will always be in memory.  How ever much data is in the session state database, you are missing that much buffer pool space for your other databases.  Because of the way that session state works, every single time a page is clicked on the website, at least one call is made to the session state database pretty much forcing the database server to keep the data from the database in the buffer pool.

Another problem that you can see if ghost records.  Paul Randal describes ghost records perfectly on his blog post Ghost cleanup in depth.

When a record is deleted, apart from it being marked as a ghost record, the page that the record is on is also marked as having ghost records in one of the allocation maps – the PFS page (post coming soon!) – and in its page header. Marking a page as having ghost records in a PFS page also changes the database state to indicate that there are some ghost records to cleanup – somewhere. Nothing tells the ghost cleanup task to clean the specific page that the delete happened on - yet. That only happens when the next scan operation reads the page and notices that the page has ghost records.

He also talks about how to fix the problem in the same blog post, so I’ll leave you to click through for all that information.

Another problem goes back to the fact that the SQL Server will be persisting everything to disk, and with the information in this database changing all the time, as the website grows you’ll need faster and faster disks under the session state database just to keep up.  As the site grows more popular you will end up spending more and more money on faster and faster disks, just to keep the session state working, much less anything else on the SQL Server.  You may even get to the point where you actually need a dedicated SQL Server just to run the session state database.

Another problem is good old locking and blocking.  SQL Server likes to take page level locks when it does insert, update and delete operations.  Well, unless each page on disk only has a single row in it you are going to have processes being blocked for short periods of time as other users session state information is updated.  You can work around this to some extent by hacking the session state database’s stored procedures and forcing row locks, but now you are taking more locks (and more memory for locks), etc.

How do I change to the session state server?

First you need to install the service on the server that will be your session state server.  On Windows 2008 just install all the .NET components and that should do the trick.  You’ll probably want to start the service as well.  When you bring up the list of services it is the one called “ASP.NET State Service”.

Then on each web server you’ll want to change the session state information. You can either use IIS Manager to do this, or change the web.config. However you configured session state the first time, that’s how you’ll want to change it this time.

To use the IIS Manager select the website (or application) to configure and double click on the “Session State” icon on the right (shown below).

From there select “State Server” from the list and change “localhost” to the server you’ll be using for session state. If you have changed the TCP port number from 42424 to something else you can adjust that here as well.

To set the session state setting via the web.config file find the existing session state information and edit it, or add in the session state information. Set the “mode” to “StateServer” and set the “stateConnectionString” to the same value that goes in the IIS config setting.

<configuration>
  <system.web>
    <sessionState mode="StateServer"
      stateConnectionString="tcpip=SampleStateServer:42424"
      cookieless="false"
      timeout="20"/>
  </system.web>
</configuration>

Hopefully I’ve convinces you to move your session state information out of SQL Server and into a repository that it actually belongs in.

Denny

In preparation for the onslaught of security patches which were released on October 14, 2009 I went ahead and patched all our severs the day before as it had been a couple of weeks since I had triggered patching and I wanted to get everything else installed so that there wouldn’t be any dependency issues with installing the new patches.  All of our servers, about 60 in all, patched correctly except for two of the three servers which host our web application which our customers use.  Now I say that two of the three servers didn’t patch correctly, but the third server wasn’t allowed to reboot so I don’t know if that one patched corrected or not at the time.  But with two machines of the three offline, I wasn’t willing to chance it.

These machines are not exactly complex machines.  They are Windows 2008 Web Edition virtual machines (under VMware vSphere 4.0), with IIS 7 installed.  Each machine has 5 private IPs as each of the sites has an SSL certificate on it.  The only other software which is installed is the VMware Tools and the Trend Micro Anti Virus.  Our ASP.NET site is sitting on the machine, but it requires no extra DLLs to be registered so doesn’t really count as being “installed”.

Unfortunately we don’t have Software Assurance on our servers as it was outside of our budget so we have to settle for the paid support with Microsoft.  As I didn’t really want to front the few hundred dollars to call Microsoft at midnight, and knowing that we had a decent support contract with Dell from when we purchased our servers from them, I figured that Dell would be a good place to call, then they could get Microsoft involved.

The symptoms for the server were that when trying to boot normally the machine would hang at the Applying Computer Settings. If the network ports were disconnected then the machine would boot but nothing would function correctly as no services would start.

When I booted into safe mode there was an error in the event log which says “The COM+ Event System detected a bad return code during its internal processing. HRESULT was 8007043c from line 45 of d:\vistasp1_gdr\com\complus\src\events\tier1\eventsystemobj.cpp. Please contact Microsoft Product Support Services to report this error.”  To me this is a pretty clear that there’s some major problem with the machine.

Dell support wanted to look at some stuff, so I was OK with that.  I figured how long can this take?  Well 5 hours after that we had gotten no where.  Some of the highlights of the stuff that we did.

• Uninstall all the patches that were installed (there were 5 including IE8 on this server)
• Attempt to reinstall VMware Tools, except that the installer wouldn’t start.
• Check the and IP and DNS settings
• Disable IPv6
  
• Check to make sure we can ping the domain controller
  
• Run SFC against the system.

That’s pretty much all that was done except to reboot the machine a bunch of times and watch it sit and spin at the applying computer settings section.

At one point I asked her if she knew what COM+ was.  A valid question considering the error message that I saw in the error log.  After several minutes of silence (I assume I was on mute and she was searching the net for the answer) she told me something about it having to do with the .NET architecture.  While .NET can use COM to do stuff, um no.  Points for trying, but no.  She was very upset and seamed quite offended that I called her bluff on this.

After 5 hours of this BS i finely had enough and unloaded on the support person and the Technical Account Manager at Dell and they got Microsoft involved.  But because this wasn’t an OEM license, but a Microsoft Volume License they had to open a ticket under the normal hotfix broke my machine queue.  This lovely queue has an 8 hour callback window to call you in.  At this point with one machine running I went to get some sleep while I waited for the call back.

Working with Microsoft

It took a couple of tries to get in touch with the actual Microsoft engineer.  For some reason when he called me at 10:30 in the morning it didn’t wake me up (I got to bed around 7 or so).  I got up at 2 when my alarm went off so that I could get to my doctors appointment.  I emailed the engineer and explained that the phone didn’t wake me, and asked for a call back at around 5.  The engineer told me not a problem, he’d schedule a call for 5, but with another engineer as he was off work before then.  No problem, I wouldn’t want to work late either if I didn’t have to.

Come 5pm no call.  Come 6pm no call.  Come 7pm no call.  So I call into the technical router and I’m told at there is an 8 hour call back Window.  I tell him it’s been 12 hours.  He then tells me that its after hours and I’ll need to wait until the next day.  At this point I pretty much go ballistic on him and get to a supervisor.  The supervisor looked at the case listened to be rant, understood that I was pissed off, and an MVP to boot, and found someone to help me.

Once I got on the phone with the Engineers it took Microsoft 2 hours and 3 guys to find the problem, but the end result is that it’s a very simple problem, with an even simpler solution.

The Problem

To understand the problem you have to dig into how the OS boots.  When the OS starts the drivers and services are grouped together, and those groups are started as a group.  In this case we are concerned with the HTTP.SYS driver and the Cryptographic Services service.  They are grouped together in a single group.  The problem comes into play when you have websites which are secured by SSL certificates and the SSL certificate is hosted in IIS.

In some cases the HTTP.SYS driver will start and attempt to contact the Cryptographic Services service, which hasn’t yet started.  Because of this the HTTP.SYS driver waits for the Cryptographic Services service to start.  Now the problems comes into play with the services database which Windows maintains.  When a service begins to start is takes a lock on this database.  When the service finishes starting it releases the lock.  So HTTP.SYS begins to start it takes a lock, then looks for the Cryptographic Services service.  When it can’t find the Cryptographic Services service it waits for it to be found.  But because the HTTP.SYS driver hasn’t finished starting it holds a lock on the service database.  We are now in a deadlock situation.  HTTP.SYS is waiting for Cryptographic Services to start, but it can’t without the HTTP.SYS service to finish starting.

The Resolution

The resolution for this is sadly simple, it is much simpler than the 7 hours of time on the phone it took to get to this point.  Set a startup dependency between the HTTP.SYS driver and the Cryptographic Services service.  Doing this requires a manual registry change, so all the normal warnings about editing the registry apply.

To fix the problem, open RegEdit.  Navigate to HKLM > System > Current Control Set > Services > HTTP.  Create a new Multi-String Value named “DependOnService” (don’t forget that Caps counts).  Edit the new value and type in CryptSvc (again Caps counts).  Reboot the server.

Now when the server boots up the HTTP service won’t attempt to start until after the Cryptographic Services service has finished starting.

So the end solution to fix the problem required adding 23 characters to the registry in the right location.

Why did it happen

I was told that sometimes this just happens.  For some reason the Cryptographic Services service just takes longer to load.  In this case, I had installed a bunch of hot fixes, several of which were security hot fixes.  Which would have made changes to the Cryptographic Services service which would have slowed down it’s starting up during boot while it tried to finish dealing with any install pieces.  This allowed the HTTP.SYS driver to start first and started this whole mess.

A simple reboot didn’t fix the problem because the Cryptographic Services service wasn’t ever allowed to do what it needed to so it wouldn’t ever start because every time it tried to load the HTTP.SYS didn’t let it finish.

Finding more information

Where can you find more information about this?  That’s an excellent question.  At the moment you can’t.  Microsoft has seen this several times now on Windows 2008, but they haven’t yet released the MSKB which says how to resolve the issue.

As best as I can tell this is the only info on the web about this so far.

The reason that I didn’t see the same problem on any other servers?  Well that’s because this cluster of servers is the only one that is hosting the SSL certificates them selves.  All the other clusters of servers the SSL certificates are being hosted by out Cisco Load Ballancer so they don’t have this problem, because if there’s no certificates being used by IIS then HTTP.SYS driver doesn’t try and talk to the Cryptographic Services service.

If you are having this same strange kind of issue some things you can look at to see if you might be having the problem.

First check to see if there is a lock being held on the services database.  This can be done with the sc command line tool.  From a command line run:

sc querylock

If the following services aren’t running, then your HTTP.SYS driver may not have started completly.

• Print Spooler
• Routing and Remote Access
• SSDP Discovery
• UPnP Device Host
• Windows Event Collector
• World Wide Web Publishing Service (IIS)

If you have SSL certificates on Windows 2008 it is recommended that you put this registry setting in place in order to prevent this problem from coming up until Microsoft is able to address it within Windows.

Hopefully this saves someone some of the headache that I had to deal with.

While it turned out that the problems wasn’t with COM+ like I suspect, I’m guessing that while the COM+ Event Service I’m guessing that it does actually depend on it for encrypting any communication which it does which would be what caused the error to be thrown (I’m guessing here).

Now if anyone from Dell would like to contact me to suck up to me and make me happy with Dell again, you’ve got my contact information.

Denny

If you’re like me you probably upgrade when you have the chance.  If you’re like my company you don’t run just one website per server.  In our case this cluster of web servers runs 6 different sites, and the session state for one of them keeps resetting just about every page view.

Apparently one of the values which is used to create the session id is the ID number of the website in IIS.  Up through IIS 6 this number was a hash of the name of the site, so as long as the sites all had the same name you were set.  In IIS 7 this ID number is now just a running number based on the order you create the websites in.  So if you create the websites in a different order on one or more machines then your ASPState information will keep expiring since the hash values don’t match correctly.

The good news is that you can change the ID number by clicking the Advanced Settings link on the right hand side.

Denny