Attention DBAs: listen carefully to this tale of woe…
At the Alaska Dept. of Revenue recently, a “technician” committed not one, but two, serious errors: first, he mistakenly deleted over 600,000 customer records (which were scanned paper applications from state residents); second, he reformatted the backup drive!
I’ll wait for the gasps and nausea to subside among you DBAs reading this.
But wait, it gets better. The department had a third line of defense: the data was also on tape. Lo and behold — you guessed it — the backup tapes were unreadable. Did I mention that the data was worth $38 billion (not million, billion)? The phrase “criminally incompetent” comes to mind.
Believe it or not, there was a happy ending to this sad story. The “old school” backup was still available — 300 boxes containing the original paper applications. These were re-scanned and the database was reconstructed. It took 70 people six weeks to do this, to the tune of $200,000 in overtime.
The morals of the story are obvious: 1) have a backup and recovery plan that includes worst-case scenarios, 2) test your backups regularly and 3) check the references of “technicians” that you hire!