This is why to test your backups!

Attention DBAs: listen carefully to this tale of woe…

At the Alaska Dept. of Revenue recently, a “technician” committed not one, but two, serious errors: first, he mistakenly deleted over 600,000 customer records (which were scanned paper applications from state residents); second, he reformatted the backup drive!

 I’ll wait for the gasps and nausea to subside among you DBAs reading this.

But wait, it gets better. The department had a third line of defense: the data was also on tape. Lo and behold — you guessed it — the backup tapes were unreadable. Did I mention that the data was worth $38 billion (not million, billion)? The phrase “criminally incompetent” comes to mind. 

Believe it or not, there was a happy ending to this sad story. The “old school” backup was still available — 300 boxes containing the original paper applications. These were re-scanned and the database was reconstructed. It took 70 people six weeks to do this, to the tune of $200,000 in overtime.

The morals of the story are obvious:  1) have a backup and recovery plan that includes worst-case scenarios, 2) test your backups regularly and 3) check the references of “technicians” that you hire!

Cheers, Tim