Posted by: Colin Smith
Communication, Planning, Sybase, Upgrade
So last week was our big go live of Sybase 15 in a production environment. Wow it did not go as smooth as I had hoped. 24 hours of College Basketball, 24 Hours of PASS, and perhaps 24 hours of some other things are pretty cool and fun, but 24 hours of Sybase was not. Now do not get me wrong, we had some fun moments but when an upgrade turns into a 24 hour marathon it is not usually about having a good time.
We did get the Production system up and runnin without too many hitches and I think we were about on schedule for that. We also needed to get the Reporting Server up and online and loaded from production and that did not go so well. I thought that we would be done and good by 10:00 AM on Tuesday after starting at about 8:00 PM in Monday. Well that did not work out. We were in the office working on it for a full 24 hours before going home. After I got home I was called a few times through the night and then back in the office at 8:00 AM Wednesday and putting out fires.
This all sounds bad, like we did not have a good plan or like we did not do our due dilligence and test. Well here is the thing, we had, for the most part, a good plan. We, the DBA team, had done our testing and validated that all of our stuff would work and worked through any issues that the application teams brough to our attention. The problem really lies in the application groups doing testing on all batch jobs and thuroughly testing the application. This was a huge upgrade and it inclded new client side drivers for the Sybase ODBC connections that are used all over the place. Well the application teams just did not do all the testing that they needed to.
So I guess, what I have learned from this is to babysit the application teams a bit more and make sure that the testing that needs to get done does get done. Make sure that they know what the runtimes of jobs were and what they are going to be. Do any code modification that you need to, since 15 is essentially a new optimizer, before the go live.
This is just me ranting about such a bad week last week that I feel could have been better if others had been a bit more proactive. That is OK though as I think that it made me a better DBA for having to work all these issues and it helped me understand them more as well. I think, if we ever do this again, that we will be better able to communicate and have a better upgrade.