DR

Oracle Dataguard Replication Scare

I had to bring down the standby server for the power outage in our DR location a few nights agon. No big deal, shut off replication from the primary, go to the standby and do a shutdown imediate, since you can not do a shutdown on the standby server. When things cam up I go to the secondary and do a startup nomount; and then alter database mount standby database; Well I thougt that was all I needed. About 10 minutes later I get some pages about the primary not being able to connect to the standby… DUH I forgot to start the listener on the standby. No Problem. I run lsnrctl start on the standby host and I am in business now. No more pages all is good. Well I thought so until after the cold backup that we run overnight from the standby database. Now I get a page saying that recover managed standby database cancel can not execute since this is not a managed standby database. I think that is odd and I check to see if archive logs are being applied and they are not and have not been for over 13 huors now. I find that I forgot to run alter database recover managed standby database; I run that command and look at that. Now my Archive logs are being applied. It took about 4 hours to get caught up but it did. So when bringing up the standby just do the following:

STARTUP NOMOUNT;

ALTER DATABASE MOUNT STANDBY DATABASE;

ALTER DATABASE RECOVER MANGED STANDBY DATABASE;

If you have enabled the primary to send archive logs then you are all set.

Sybase Instance Scare

Monday morning my company was putting in a new UPS at our DR facility. This is great but it means that all database instances that are running at that location are coming down as power to the entire site was down for 2 hours. No big deal right, I got a list of the instances and shut them all down properly before the host machines were shutdown. A couple hours later I get a call saying that all the power is back on and that all my hosts are ready as well. I login and I start to run my startup scripts for all my Sybase instances. I run into two servers that are not starting up and this is what I find in the log files.
The configuration area in master device appears to be corrupt.
NOT COOL… I know that I have backups so that is OK but Master really in two instances… I dig around a bit more and I find that the two instances that are having issues run on the same physical host but are in Solaris Containers that use Veritas Disk mounts. I google the error to see if it may be disk related and not corruption in Master. I found it.

I called my storage guy and he looked into it. Called me about 30 minutes later and asked me to try again. WOW everything came up and I was a happy camper. Sure made for a long morning though and also made me realize the benefit to practicing DR situations. I think I will work with my team to have outages on out Test instances just so we can rebuild Master. It will be good to know and be practiced up on if it is ever not the disk.