Eye on Oracle

Mar 20 2007   10:09AM GMT

This is why to test your backups!

Ken Cline Profile: Clinek

Attention DBAs: listen carefully to this tale of woe

At the Alaska Dept. of Revenue recently, a “technician” committed not one, but two, serious errors: first, he mistakenly deleted over 600,000 customer records (which were scanned paper applications from state residents); second, he reformatted the backup drive!

 I’ll wait for the gasps and nausea to subside among you DBAs reading this.

But wait, it gets better. The department had a third line of defense: the data was also on tape. Lo and behold — you guessed it — the backup tapes were unreadable. Did I mention that the data was worth $38 billion (not million, billion)? The phrase “criminally incompetent” comes to mind. 

Believe it or not, there was a happy ending to this sad story. The “old school” backup was still available — 300 boxes containing the original paper applications. These were re-scanned and the database was reconstructed. It took 70 people six weeks to do this, to the tune of $200,000 in overtime.

The morals of the story are obvious:  1) have a backup and recovery plan that includes worst-case scenarios, 2) test your backups regularly and 3) check the references of “technicians” that you hire!

Cheers, Tim

3  Comments on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.
  • Zachi
    this no way is worth 38 billion $, if it cost $200,00 to recreate the data from the applications. The most it would have cost is mailing each and every one of the Alaska residents the forms and ask them to refill them. Therefore, several million $ is the true cost.
    0 pointsBadges:
  • Sukaina
    As the value of data was in billions, the concerned company is also reponsible, as the management should have taken precaution for having more backups at different location and media. Instead of spending $200,000 in recovery, lesser investment in better technologies for storage and recovery would have saved them from this problem.
    10 pointsBadges:
  • Tim DiChiara
    Zachi: You're correct. It seems that the value of the data itself is unclear in the original article. However, what I believe that the article meant was that the content or meaning of the data -- i.e., the money being disbursed to Alaska residents -- was worth $38 billion.
    0 pointsBadges:

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: