To solve this problem, I added a DDL trigger to the server which will cause the new database to be backed up as soon as the database is created.  The trigger itself is rather simple.  Most of the trigger is setting variables.  Then I make sure that the database isn’t a database snapshot, as database snapshots can’t be backed up.  If it isn’t a snapshot we continue with everything else.

Then I create a subfolder for the backups to be put into (the backups for each database go into their own folder, so as this is a new database the folder needs to be created).  Then I commit the transaction, as database backups can’t be taken within a transaction.  Then we do the actual database backup.  I then throw a message to the end user using the RAISERROR statement telling them that they can ignore the other error that they are going to get about the transaction being closed before the trigger was complete.  This is just an annoyance of my needing to commit the transaction before taking the backup.  Sure I could have setup a job which takes the backup and emails if there was a failure, but that just seems to complex for something so simple.  The code for my trigger is below.

CREATE TRIGGER BackupNewDatabase
ON ALL SERVER
FOR CREATE_DATABASE
AS
declare @database sysname, @event_data XML = EVENTDATA(), @folder nvarchar(4000), @file nvarchar(4000)

SET @database = @event_data.value(‘(/EVENT_INSTANCE/DatabaseName)[1]‘, ‘sysname’)

set @folder = ‘X:\Backups\’ + @database

set @file = @folder + ‘\’ + @database + ‘.bak’

if exists (select * from sys.databases where name = @database and source_database_id is null)
BEGIN
EXEC master.dbo.xp_create_subdir @folder

COMMIT

BACKUP DATABASE @database to disk=@file

raiserror( ‘You can ignore the error message which says that the transaction ended within the trigger.’, 16,1)
END
GO

Hopefully you find this solution helpful if you get into a situation like this,
Denny

I’ve moved lots of companies into data centers before, and each time I’ve done so it has included a site visit during to RFP process to check on things like cooling, generator, site configuration, etc.  During these site visits I stick my head through every door that the sales guy and the engineer who are doing the tour will open.  If they won’t open a door for me they better have a damn good reason to not show me what’s in there.

If I went to do a site visit at a facility that was located just at sea level just a few blocks from the ocean, they’d better have some pretty impressive plans for how they are going to handle flooding.  If the answer was “we’ve never had a problem with flooding” or something similar they’d be off my list as they haven’t done their due diligence to insure that the site will be online during the worst emergencies possible.

Now before you start telling me that I’ve got no idea what I’m talking about, and that data centers in the North East have different problems from data centers in the South West.  I actually do as I’ve moved companies into data centers on both coasts.  Most recently I moved the data center for Phreesia from a managed service facility in Texas to a colo in New Jersey.  As a part of this move we looked at a number of facilities in the New York / New Jersey area.  Many of the New York City data centers were eliminated due to cost, or being just to close to the water as we didn’t want to deal with situations like this exact one.

The data center we settled on is in New Jersey about a 30-40 minute drive from the Phreesia office (once you get out of New York City).  While the data center is near a river, the river is a little over a mile away.  The data center itself is on a slope with a  cinder block wall on the outer edge which will divert water away in the event of a river overflow (it also protects the facility from someone driving a car or truck into the facility).  The generators and fuel pumps are all mounted on raised feet (not very tall, but tall enough) so that they keep running in the event of a flood.  The cables from the generators to the building have all been buried under the ground so that tree branches which are torn loose during the hurricane can’t cut those cables.

Our diligence in selecting a data center paid off.  While the folks mentioned in that article were dragging buckets of fuel up 17 floors worth of stairs the team at Phreesia just sat back and rode out the storm with their families.  The sites didn’t go down and the team didn’t have to rush into a very hazardous situation. The team was able to focus on their families and neighbors without having to worry about the systems.  Those of us that aren’t in the New York area monitored the systems remotely and founds that everything was running perfectly normally just on generator power instead of city power.

This entire event just shows that when doing data center evaluations the worst possible case situation needs to be planned for and expected.  If they aren’t they are going to come back and bite you.  Especially in todays world of storms which are ever increasing in destructive power.

If you are in a facility which has risks such as fuel pumps which are below sea level (they are in the basement and the road is at sea level) then a second data center becomes very important very quickly.  This became very clear during this hurricane when some very large important websites went offline because they didn’t have that DR facility that they could fail over to, unlike sites like StackOverflow (and ServerFault and the Stack Exchange network).

If you are at risk now is an excellent time to sit down with management and go over all the “what ifs” of your current hosting solution and think about the cost of a DR site along with the cost of not having one.

Denny

The reason for this is pretty basic.  How is the SQL Server supposed to know what settings to use, or how often to do those things?  Every database needs to have the indexes rebuild at a different interval.  There are a variety of recovery options which would change the backup policy for a database.  Not to mention that to properly build backup settings the SQL Server would need to know things like your RPO (Recovery Point Objective aka. how much data are you willing to loose in the event of a major failure).  Without a way to tell the SQL Server these sorts of things the SQL Server wouldn’t be able to properly maintain your environment.

So there’s the why.

Denny

The big thing that you loose when doing this is that you can no longer restore the SQL Server database to a specific point in time if you have a system failure.  If you don’t need the ability to restore so a specific point in time, and restoring to your latest full or differential backup that you have (you do know that you need to backup the vCenter databases right?) then changing the recovery mode from FULL to SIMPLE is just fine.  However if you do need that ability (and I would recommend that you have that ability so that if the vCenter server fails or someone deletes data you can get back as much trending data as possible) then you will want to leave the recovery mode as FULL, but instead setup transaction log backups (hopefully you have a DBA or a database consultant that can help you with this) so that they run every few minutes (I typically recommend every 10-15 minutes).  Just keep in mind that when selecting how often they should run in the event of a server failure you may loose up to that amount of data from the database.  So if you aren’t aloud to loose more than 10 minutes worth of data, don’t backup the log every 15 minutes, back it up every 10, or maybe every 8 just to be safe.  Make sure that you get in writing from your management that this is the config that they want so if you get yelled at you are covered.

If you haven’t ever backed up the transaction log your disk is probably pretty full (or totally full).  In this case you will want to shrink the database just once.  But before you do you need to know what size you should shrink it to.  This can be much easier if a DBA does it, but I’ve got faith in you to not screw it up.

First connect to the SQL Server using Management Studio.  Take a backup of the transaction log to disk by right clicking on the database, selecting Tasks from the context menu then selecting Backup from the pop out menu.  Select transaction log backup, set a file to backup the log to (you’ll need a lot of disk space for this) and make sure that there’s only one file listed in the file list then start the backup.  This will probably take a while.

If you don’t care about loosing the log for now change the database from full to simple recovery and back.  Do this by running this script.

ALTER DATABASE vcdb
SET RECOVERY SIMPLE
GO
ALTER DATABASE vcdb
SET RECOVERY FULL
GO

Now do a full backup as shown above (just do a full database backup instead of a transaction log backup).

Now with everything running as normal wait for how ever long it’ll be between transaction log backups.  Then connect to the SQL Server in SQL Server Management Studio.  Right click on the vCenter database (vcdb) and select Reports from the context menu, then “Disk Usage” from the list of reports.  At the top it’ll give you the size of the transaction log in MB, and the graph on the right will show you a percentage breakdown of how full the transaction log file is.  Take the Used percentage and double it.  Now take that percentage and multiple it by the size of the transaction log.  That’s the minimum size you should set your transaction log for.  Personally I would round up to the 8 gig size above that size.  If you need 150 Megs of log space, set the log for 8 Gigs in size.  If you need 10 Gigs of log space, let the log for 16 Gigs of space.  A little extra space used is fine, it is better for performance of the database if we don’t need to grow this transaction log.  You can shrink the transaction log by using the DBCC SHRINKFILE statement as shown.

DBCC SHRINKFILE ('vbdc_log', 8388608)

Keep in mind that the number, in this case 8 Gigs, is passed in in megs.  Do NOT for any reason create any sort of scheduled task to regularly shrink any database file.  This will do nothing good and will cause performance problems of its own.  If anyone has told you to setup a job which shrinks the database file regularly they are WRONG.

Denny

While you can normally attach a database without the log file, it doesn’t always work.  Sometimes SQL Server won’t be able to recover from this and create a new log file and you’l have lost the database.

If your database is growing and has gotten very large, then you’ll want to change the recorvery mode from FULL to SIMPLE.  This will help keep the transaction log file small, and is perfectly safe to do.  The only thing you will loose is the ability to do point in time restores of the transaction log.  However point in time restores are only available if you backup the transaction log (which would have prevented the log from growing out of control to begin with).

The reason that many people do this, is that the name of the file is misleading.  Normally logs are very safe to delete, however the transaction log isn’t your normal log file.  A transaction log keeps a record of every change to the database, along with the before and after values.  All changes are written to the transaction log before they are commited to the database itself so that if the server fails the database is left in a consistant state.

If your log file has simply gotten huge, and you need to make it smaller, you can use the DBCC SHRINKFILE command to shrink the log file to a smaller size.  This shouldn’t be done on a regular basis.  If you are doing this on a regular basis leave the file large it’s at the size it needs to be or change the database recovery to SIMPLE.

Denny

 The backups are simple, use the SQL Maintenance plan to backup the logs, and then copy them to the remote machine.  But how do you restore the logs to the reporting server nightly.

 Well I’ve got a two step SQL job which should help you out.

Step 1 kills all current sessions in the database, and step 2 does the actual restores.

The code for step 1 is: 

declare @spid varchar(20)
declare cur CURSOR FOR
select spid
from sys.sysprocesses
where dbid = db_id('Your Database Name Here') /*<---Put your database name here*/
and spid > 50
open cur
fetch next from cur into @spid
while @@FETCH_STATUS = 0
BEGIN
exec ('kill ' + @spid)
fetch next from cur into @spid
END
close cur
deallocate cur

The code for step 2 is:

create table #Files
(FileName nvarchar(4000),
Depth int,
IsFile bit)insert into #Files
exec xp_dirtree 'd:\', 1, 1

I hope this makes your process easier. Now this code only works on SQL Server 2005 and up as the system stored procedures which I use were not included until SQL Server 2005.

Denny

The basic explanation is “Used when waiting for a backup thread to complete.  Wait time may be very long (minutes, hours).”  Basically what this means is that there is a backup running and something is waiting for it to complete.

When i saw this show up I was running a restore.  That restore had three entries in the sysprocesses table.  The first was the main kpid, with two child kpids.  The parent kpid was the wait type of BACKUPTHREAD while it was waiting for the child kpid to finish processing.  In my case the wait time was short, and it seamed to switch from this wait type to an IO wait type.

Denny