So a couple of nights ago Paul Randal (Blog | Twitter), Wendy Pastrick (Blog | Twitter), Jonathan Kehayias (Blog | Twitter) and I were talking about how some of our demo’s have completely and totally failed, often in front of a live audience. As we all were chatting about our various demo’s crapping out we came to the realization (ok, we had all already come to this realization separately) that when your demo fails, how you handle it is how you prove that you are a truly good speaker.
Speaking by it self isn’t all that easy for a lot of people (oddly not me, but I’m kind of an attention whore) but when that demo blows up, and it will at some point, how you handle that is where you really show that you know the product. Since it is a little hard to tell a good story in 140 characters, I decided that a blog post would be a good way to give my story.
Windows just won’t cluster
So my first mega fail was actually not in front of a live audience (unless you count the recording engineer). The first time I did a recording for the SSWUG conference (I think it was the first, I’m going to record by third or fourth set of sessions soon) one of the sessions that I did was on SQL Clustering. During the session I walked through the Windows 2008 clustering process and got the SQL Installation started. I would jump back to it through out the session so the demo was throughout the session because sitting there watching the wizard running which I do nothing isn’t very exciting.
So I tested everything before I left home and everything was working perfectly. I decided to test everything from the hotel the night before (I always fly in the night before when I do the recordings) and when I ran through the clustering validation wizard it took 3 or 4 times for me to get a successful validation. At this point it was late, so I called it a night. When we did the recording the next day, wouldn’t you know it the damn validation wizard just wouldn’t work. Which means that SQL Server wouldn’t install on the cluster as SQL won’t install on a cluster that hasn’t passed validation. So I just had to wing it, and explain that this is what you should be seeing on this screen, etc.
Needless to say the session went great, and was a pretty good length (I think I went really close to the one hour limit on it) and the reviews were great. I even got some positive reviews for leaving in the problem demo and not refilming it to make it look correct.
OK, who’s set off the fire alarm?
My second story is more recent, and was in person. At the 2010 SQL PASS Summit I was doing a quick 45 minute storage session during Buck Woody’s post-con on Sharepoint. During my talk the fire alarm for the building started going off. It wasn’t so loud that you couldn’t here me, but it was a bit annoying to listen to (I suppose that is the point). Someone come in to tell us that we didn’t need to leave, which was good since I was still talking, with occasional pauses to yell at the building, but we worked right through it.
What’s happened to you when you present?
So what sort of fun would this telling of the nightmares be if I didn’t open it up for you to tell us about what sort of problems you’ve had with your demo’s or sessions (hopefully Buck Woody (Blog | Twitter) will post something, he’s got some great stories). If you aren’t syndicated on SQLServerPedia.com or SSC please link back to me, so I can compile a master list later.
So it is time for another vote for the second category that I have a SQL Rally session submitted for. This time the vote is for the “Enterprise Database Administration & Deployment” track. I’ve got a session up titled “Using SQL Server Denali’s Always On” again in the Summit Spotlight section.
The abstract for my session is:
In this session we will look at the features which are provided with Microsoft SQL Server “Denali” as part of the “Always On” features including site to site configurations to allow of a large scale high availability solution without the need for any high end SAN storage solution. Additionally we will be looking at the ability to have redundant servers which can be used for reporting or for taking your backups reducing the load from the production database.
So, I’m guessing that my first solo book must be getting close to shipping. You can now flip through the first 11 pages of “Securing SQL Server” (preview page) on Amazon’s website. Now the images in the online preview are in color, but the book won’t be (or so I was told) so they must have used my original images to put together what Amazon is showing.
This is what I’m guessing is one of the last steps before the book is actually shipped.
Now for those of you that are going to wait for the Kindle edition, you’ve got a little longer to wait. While it is available to view on Amazon, you can’t purchase it yet. I’ve been told by the publisher that it’ll be about 3 months behind the print version.
So I got an email from someone (and yes he told me that I can publish this post about it) because he was having an argument with his storage / systems team about LUN config for the SQL Servers. He wanted smaller LUNs so that they could be more easily moved around if performance problems showed up on the array. His storage / system folks big argument against this was that it was complex to setup, so they didn’t want to do it. When I got to this “argument” I knew that this blog post was coming.
As IT professionals our job is to design and build systems which meet the performance requirements of the applications which will sit upon them. And to keep those systems running for as long as the business determines that those systems are useful to making the company money. We as professionals (those that know me can stop snickering when I call myself a professional) shouldn’t be designing systems a specific way just because it is easier to do so. If that way doesn’t support the needs of the application then it is the wrong way.
In the case of storage, there are lots of systems out there that needs lots of LUNs for one reason or another (usually to spread the load out). After carving up the LUNs and presenting them to the SQL Server, you mount them as mount points and setup the dependencies in the cluster admin tool (this server is clustered I’m told) and you are done. There’s not much do “manage” from the disk level at this point on a day to day basis. If everything is sized correctly then the LUNs won’t need to be grown for quite some time and everything can just be left as is.
In short, quit being so lazy when designing systems. Don’t take the easy way out just because you can. Design the system correctly so that you don’t have to redesign it later. I can’t tell you the number of months that I’ve spent redesigning systems because they weren’t done correctly the first time. If you need help designing a system ask someone, there are lots of people who will be happy to help. But don’t make poor planning decisions just because you don’t want to do a little bit of extra work.
The SQL PASS organization has taken a different approach to session selection for the SQL Rally than they normally do for the annual summit. Instead of a full program committee which selects sessions then announces them, they are leaving the session selection up to the community via a vote. This weeks voting is for the Database/Application Developer track. Within that track sessions are broken down into 11 categories. As a former SQL PASS summit speaker my session does into the Summit Spotlight track, which is probably the toughest track to be in as everyone else in the Summit Spotlight track is also a former summit speaker.
The session which I’ve submitted as a Database/Application Developer session is called “Where should I be encrypting my data?”. In this session we’ll be looking at all the various places within the application stack that data can be encrypted. This includes the application layer, the database layer, encrypting over the wire, transparent data encryption, encrypting using your MPIO driver and offloading encryption to your HBAs. We will look at why and how you encrypt data at all of these levels, and learn when to use each one. As data encryption is a must for any business doing business in the United States, this session is a must for any database or application developer.
Voting for this round of the SQL PASS sessions is being done the week of January 17th-23rd, 2011 only, so you’ll need to vote soon for your vote to count. If you would like to be emailed when the next rounds of voting happen, you can register for that as well.
The way the session selection will work (as I under stand it) is that the number of sessions have already been selected. The top sessions from each group will be given spots at the SQL Rally event, so getting the word out is very important. As is the fact that people need to be sure to vote for sessions that they want to see.
I gave two talks at the recent SQL Saturday 62 in Tampa, FL and I had a blast doing it. As promised I’ve posted them on Monday (it is still Monday my time, so that counts right)?
Thanks everyone for coming to my session. Hopefully I’ll see you at the Tampa SQL Saturday next year.
The short answer is as much as possible.
The more exact answer is you typically can’t allocate cache to a specific LUN. You can usually carve up the total amount of cache into read and write cache. You want as much write cache as possible so that the database writes directly to the disk as little as possible.
There is no number that I can give saying that you need X amount of write cache for a 500 Gig database, because it doesn’t depend on the database size. It depends on the amount of data change per minute, how fast the data can be destaged from the write cache to the disks, and how much load is being placed on the disk for reading data.
You’ve really got two questions here. The first is if I like iSCSI for SQL, and the second is if I like NAS for SQL. NAS doesn’t support iSCSI as iSCSI is a SAN technology.
iSCSI is just fine for SQL Server as long as the server and the storage are on the same subnet, and the network has enough bandwidth available to support the data transfer.
NAS isn’t supported for SQL Server, unless the NAS has a SAN head and the SAN head is used for access to the storage.
This great phrase (which is way more clever than anything I could come up with) was tweeted by Daniel Taylor (Twitter) last week while talking about the pre-cons which are coming up this week in Tampa. And today is the last day to sign up for $99 (plus a few bucks for fees). After today the price goes up to $109 per person.
There is a better deal available if you have a few people that want to go. If you buy two pre-con tickets for the full price of $109 you get a third ticket for free. If you and a friend were already planning on coming, you may as well bring a third person for basically nothing.
Pre-registration for the pre-con sessions is required so that the restaurant knows how many people to expect for lunch. The same is requested for the main SQL Saturday event as well.
One of the all to common configuration mistakes that are made is to put the session state within the SQL Server when a website begins to scale to multiple web servers.
For those that don’t know what session state is, let me take a step back and fill you in. Session state allows the web developer to store values in variables up on the web server, but in a shared location so that it doesn’t matter what web server the end user connects to the values are all there and the web servers understand the session information from each other. This is very important for shopping websites, or pretty much any site with a login prompt and a load ballancer that is configured to send the users connection to the next available web server.
What is the problem?
The problem becomes that most people when they need to scale out session state opt to put the session state information into SQL Server because that is a nice easy repository that is probably already up and running, not to mention that the session state code in ASP.NET easily supports it. The issue with putting the session state information into SQL Server is that you don’t give a crap about persisting the session state information to disk. The information doesn’t get persisted more than an hour or so, and if the information within the database is lost, the only impact is that any values in session variables is lost.
So what are the options?
There are a few options besides using SQL Server for session state.
- Keep using the in-process option.
- Use the ASP.NET Session State Service
The easiest option is to keep using the in-process option in IIS for session information. This means that you will need to configure what are called sticky sessions on the load balancer so that the user always goes to the same web server every time. In the event that a web server fails, all the session information for those users would be lost, and they would need to start over by logging back into the site, or they would have an empty shopping cart, or whatever the site does.
The ASP.NET Session State Service is a Windows service that provides a memory only session state repository so that session information isn’t ever written to disk, and it isn’t kept on the web server. You can either stand up a dedicated machine for this, or setup a couple and cluster the service manually so that you have an HA solution for your session state service. If you have an existing SQL Server cluster you can even use this cluster for it, if you don’t have anywhere better to put it. Just configure the cluster to have the SQL Server run on one node as the preferred node, and have the session state run on another node as its preferred node. This way the services won’t ever run on the same node unless the other node is offline. The session state service doesn’t take a lot of CPU power, and the amount of RAM it needs to be completely dependent on the amount of information that you are stuffing into session variables on the web servers.
In either case, either solution is better than putting the session state information into a SQL Server database. The information doesn’t need to be written to disk, ever. The information that is written into the session state database is in a blob binary form, not any sort of relational form so you can’t really do anything with it.
How do I know if sessions state is in SQL?
That is an easy one. You’ll have some funky database, probably in simple recovery mode usually name aspnetdb (maybe with a prefix and/or a suffix).
What other problems can session state cause?
Well first it’ll take away buffer pool resources from your other databases. Because the session state database is hit very hard the data from the database will always be in memory. How ever much data is in the session state database, you are missing that much buffer pool space for your other databases. Because of the way that session state works, every single time a page is clicked on the website, at least one call is made to the session state database pretty much forcing the database server to keep the data from the database in the buffer pool.
Another problem that you can see if ghost records. Paul Randal describes ghost records perfectly on his blog post Ghost cleanup in depth.
When a record is deleted, apart from it being marked as a ghost record, the page that the record is on is also marked as having ghost records in one of the allocation maps – the PFS page (post coming soon!) – and in its page header. Marking a page as having ghost records in a PFS page also changes the database state to indicate that there are some ghost records to cleanup – somewhere. Nothing tells the ghost cleanup task to clean the specific page that the delete happened on – yet. That only happens when the next scan operation reads the page and notices that the page has ghost records.
He also talks about how to fix the problem in the same blog post, so I’ll leave you to click through for all that information.
Another problem goes back to the fact that the SQL Server will be persisting everything to disk, and with the information in this database changing all the time, as the website grows you’ll need faster and faster disks under the session state database just to keep up. As the site grows more popular you will end up spending more and more money on faster and faster disks, just to keep the session state working, much less anything else on the SQL Server. You may even get to the point where you actually need a dedicated SQL Server just to run the session state database.
Another problem is good old locking and blocking. SQL Server likes to take page level locks when it does insert, update and delete operations. Well, unless each page on disk only has a single row in it you are going to have processes being blocked for short periods of time as other users session state information is updated. You can work around this to some extent by hacking the session state database’s stored procedures and forcing row locks, but now you are taking more locks (and more memory for locks), etc.
How do I change to the session state server?
First you need to install the service on the server that will be your session state server. On Windows 2008 just install all the .NET components and that should do the trick. You’ll probably want to start the service as well. When you bring up the list of services it is the one called “ASP.NET State Service”.
Then on each web server you’ll want to change the session state information. You can either use IIS Manager to do this, or change the web.config. However you configured session state the first time, that’s how you’ll want to change it this time.
From there select “State Server” from the list and change “localhost” to the server you’ll be using for session state. If you have changed the TCP port number from 42424 to something else you can adjust that here as well.
To set the session state setting via the web.config file find the existing session state information and edit it, or add in the session state information. Set the “mode” to “StateServer” and set the “stateConnectionString” to the same value that goes in the IIS config setting.
<configuration> <system.web> <sessionState mode="StateServer" stateConnectionString="tcpip=SampleStateServer:42424" cookieless="false" timeout="20"/> </system.web> </configuration>
Hopefully I’ve convinces you to move your session state information out of SQL Server and into a repository that it actually belongs in.