Posted by: Dave Raffo
business continuity, disaster recovery, i365
New Orleans is no stranger to natural disasters that emphasize the need for a good disaster recovery plan. But people in the city are struggling to deal with a business continuity situation stemming from crashed servers and a backup service gone bad.
Two servers that hold the Parish of Orleans Civil District Court’s conveyance and mortgage records going back to the 1980s crashed simultaneously on Oct. 25, and the court has been without critical digital real estate records since then. The documents stored on the servers’ database exist on paper, but the computerized index linking them to their physical location was on one of the failed servers.
Members of the court’s IT staff are blaming the problem on Seagate’s i365, according to a series of stories on the incident in The Times-Picayune newspaper. The court is under contract with i365 to use its EVault Remote Disaster Recovery Service, EVault SaaS and SaaS plus, and Evault Express Recovery Appliance. i365 had been backing up records and purging them every 30 days from August 2009. Last July, i365 sent a software update to the court to install. The court’s IT staff said it installed the update and received a message saying it was installed correctly.
But no data was backed up since July, and other records were purged after their 30-day expiration date. The court did recover digital conveyance records from the 1980s up to March 27, 2009, and mortgage data through Aug. 6, 2009. But that left more than 150,000 documents without digital records or indexes to them.
While i365 has refused to comment publicly, an email from one of its executives to the chairwoman of the court’s technology committee obtained by the Times-Picayune put at least some of the blame with the court:
“The continued exposure of this situation hurts all involved — i365, Orleans Parish and the Civil District Court,” Dave Hallmen, head of i365′s Worldwide Sales and Marketing division, wrote to [judge Piper] Griffin on Nov. 5. “We have instructed our staff to refrain from publicizing our service call records which support our position that Civil District Court IT personnel failed to properly maintain the on-site software and backup jobs.”
According to the Times-Picayune, the court’s IT staff also inadvertently lost or corrupted database information when trying to troubleshoot the failed Dell servers.
Earlier this month, the court contracted with a data management firm, The Windward Group, to restore 35,000 missing conveyance records and 119,000 lost mortgage records by Jan. 2 at a cost of hundreds of thousands of dollars, according to the Times-Picayune. About 30,000 records have been restored.
The snafu is taking its toll on real estate sales in the area. According to the Times-Picayune, title insurance companies are reluctant to underwrite home sales and some refinancing deals without up-to-date records.
Regardless of who’s at fault, i365 competitors will be sure to use the incident to push alterative DR solutions. Larry Lang, CEO of QuorumLabs, said the New Orleans incident highlights problems with backing up to the cloud. QuorumLabs sells onQ backup appliances that can provide DR when used in pairs.
“Sometimes your data goes up into the cloud and when you go to pull it back it’s not there anymore,” Lang said. “It’s like freeze drying stuff, you don’t know what will happen when you add water. When [the court’s IT staff] went back to add water, there was nothing there.”
Lang said the incident also shows the importance of DR testing. onQ appliances allow customers to automatically test their restore capabilities. “You should consistently run tests to make sure your snapshots are good,” he said.