Monday morning my company was putting in a new UPS at our DR facility. This is great but it means that all database instances that are running at that location are coming down as power to the entire site was down for 2 hours. No big deal right, I got a list of the instances and shut them all down properly before the host machines were shutdown. A couple hours later I get a call saying that all the power is back on and that all my hosts are ready as well. I login and I start to run my startup scripts for all my Sybase instances. I run into two servers that are not starting up and this is what I find in the log files.
The configuration area in master device appears to be corrupt.
NOT COOL… I know that I have backups so that is OK but Master really in two instances… I dig around a bit more and I find that the two instances that are having issues run on the same physical host but are in Solaris Containers that use Veritas Disk mounts. I google the error to see if it may be disk related and not corruption in Master. I found it.
I called my storage guy and he looked into it. Called me about 30 minutes later and asked me to try again. WOW everything came up and I was a happy camper. Sure made for a long morning though and also made me realize the benefit to practicing DR situations. I think I will work with my team to have outages on out Test instances just so we can rebuild Master. It will be good to know and be practiced up on if it is ever not the disk.