April 30, 2011 3:31 PM
Posted by: Sharon Fisher
, disaster recovery
This week featured millions of people glued to computer screens, waiting for all to be revealed, sharing their predictions, and crying when they finally saw the reality.
Oh, yeah, and there was a Royal Wedding.
But ten minutes before that (not that they were trying to hide anything, of course), Amazon also released the post-mortem of its extended Elastic Compute Cloud (EC2) outage of the previous week.
In case you were under a rock, a number of major computer sites — including foursquare, Reddit, and Quota — were down for a day, sometimes more, on April 21, due to a problem with Amazon’s web hosting business. It wasn’t until Monday or Tuesday of this week that all the sites really recovered.
If you’re familiar with the concept of “thrashing,” where a too-full hard disk or computer memory is so busy trying to find places to work that it doesn’t get anything done, that’s basically what happened to Amazon, on a mammoth scale. Due to a configuration problem, the cloud went down, and the first thing all the servers did when they came up was try to re-mirror themselves — which they couldn’t do because all the other servers that were up were trying to do the same thing. The actual summary goes into a lot more detail, if you really want to know, but that’s basically it.
So now the Internet is seeing a storm of a different kind: A pundit storm where people talk about 1) What It All Means and 2) Where We Go From Here and 3) Could It Happen Again?
1) S*** happens. 2) Don’t have a single point of failure, duh. 3) Of course.
Oh, you wanted more detail?
What it all means is that people are human and machines are stupid. This does not change, and will not change. Count on it. Problems happen. Then we institute new systems that help us protect against the most recent problem, and wait for a new problem to happen.
You know, like the TSA.
Where We Go From Here is that Amazon is instituting a number of changes in processes and procedures, both human and machine, that are intended to keep this from happening again.
Organizations that use the cloud — anybody’s cloud, not just Amazon’s — should take this as a wake-up call. Even if you weren’t affected by this outage, you could be on the next one. Don’t just have a backup. Have a backup for the backup. Yes, it costs money. How much money does it cost for your business to be out for a day? (Even if Amazon did give all its affected customers a freebie.) Forrester analyst Rachel Dines wrote a blog post listing a number of questions organizations should ask their cloud provider about backups and failover strategies.
Finally, accept that it’s going to happen — whether it’s from a natural disaster like the earthquake in Japan or the tornadoes in the American South, government action to shut down the Internet like in Egypt, widespread electrical failures, or simply a flu pandemic. As Dines says, “Assume nothing” — check every step in the disaster recovery plan, and figure out what the alternative is for every component that could fail.
April 24, 2011 8:08 PM
Posted by: Sharon Fisher
, law enforcement
When you absolutely, positively have to keep people from being able to look at your data, what do you do? Last week a number of people were surprised to find out that the popular cloud storage site Dropbox, which had advertised itself as encrypting its data so thoroughly that even its employees couldn’t look at it, actually could decrypt data after all — if required to do so by U.S. law enforcement.
Dropbox made a point of telling Steve Kovach at Business Insider, who broke the story, that this was a rephrasing of its terms of service, not a change in policy. “The TOS update was merely a clarification for users, not a policy update,” the company said.
Dropbox also pointed out that it wasn’t alone in this. “It is also worth noting that all companies that store user data (Google, Amazon, etc.) are not above the law and must comply with court orders and have similar statements in their respective terms of service.”
A number of articles about the incident concurred with this, including Business Insider’s. “This is nothing groundbreaking, but Dropbox has updated its security Terms of Service to say that if the government asks, they will have to decrypt user’s files and turn them over. That’s standard practice for any online storage service from Gmail to Amazon”.
But Business Insider went on to say, “and shouldn’t affect the average user unless they’re doing something wrong.”
That’s where it gets sticky.
Several other articles on the subject made similar comments. “In the meantime, don’t go doing anything that’ll get you in so much trouble that the G-Men need to decrypt your email or cloud storage,” said David Gerwitz of ZDNet, whose article headline, “If you have something to hide from the government, don’t use Dropbox” also implied that only those who had something to hide should be concerned. “Ok, so no worries–so long as you’re not doing anything wrong, you should be fine,” agreed Sarah Jacobsson Purewal of PC World. Comments in the PC World story went so far as to say that the only people who would be concerned about this would be pedophiles.
Recall that in 2005, the New York Times revealed that the National Security Agency was monitoring telephone calls, without warrants, of domestic callers. A few months later, USA Today revealed that this was going on with the cooperation of a number of telephone companies, including AT&T, Verizon, and Bell South.
“[T]o say that only the “guilty” have any reason to care about privacy shows a dangerous lack of awareness of how easy it is to violate some law or regulation and thereby become “guilty” yourself,” says William Morriss, a Senior Associate patent attorney of Frost Brown Todd, writing in the Ephemeral Law blog. “Even worse, when the government goes about collecting enormous amounts of data without having to justify itself and without any oversight, there will inevitably be false positives which have the potential to literally ruin someone’s life.”
The one solution Dropbox has to offer is that users can encrypt their own files before upload them to a data storage service like Dropbox — so that if the data storage service decrypts stored files, they continue to be encrypted, which only the user can decrypt. “Dropbox does not discriminate between the types of files stored in your Dropbox nor the applications used to open those files. This means you can use your own software encryption methods, such as third-party encryption software, to keep your files secure on your terms,” the company’s Terms of Service said.
However, it doesn’t say exactly how one goes about finding or using third-party encryption software. Moreover, there are those who fear that any encryption software — unless it’s open source, where people can examine it — could have a “back door” that would allow government agencies to decrypt it without user assistance. Attempts have been made, and continue to be made, to require such a back door. Some people, consequently, are sticking with “better safe than sorry” and using only open source encryption software. Unfortunately, this goes beyond the area of “easy to use” for the average — law-abiding — user.
April 18, 2011 8:28 PM
Posted by: Sharon Fisher
, western digital
A major Asian manufacturer is looking to get out of the storage business so it can invest in new areas.
Didn’t we just hear about this?
In this case, however, it’s not Hitachi GST that’s doing the selling, but Samsung Electronics, which — like Hitachi — was primarily involved in the spinning disk market and had less of a presence in the solid-state disk (SSD) market and would face expensive retooling to support it, according to the article in the Wall Street Journal on Sunday that sparked all this.
The potential purchaser? Seagate Technologies, which was leapfrogged by the Western Digital-Hitachi GST merger, which took up almost 50% of the market, according to iSuppli. Seagate accounted for 29% of hard disk drive shipments in the fourth quarter, while Samsung accounted for 10%, iSuppli said. In addition, sales of hard disk drives are down 4% in Q1 compared with Q4, iSuppli said.
Perhaps Seagate — which considered and rejected a Hitachi purchase itself — didn’t want to miss out a second time. And unlike a Hitachi purchase, which might have courted an antitrust claim, a Samsung purchase would be in the consumer marketplace, rather than the enterprise market Seagate and Hitachi share, according to Jason Mick at DailyTech.
The source for all this? “A person familiar with the matter,” who said the Korean Samsung was hoping for $1.5 billion (compared to the $4.3 billion Hitachi fetched), but might settle for $1 billion.
Seagate itself wouldn’t comment, but Chris Mellor of The Register noted earlier this month, in a piece about Seagate’s earnings, that its chairman and CEO, Stephen Luczo, was spending three months in the Far East, and that the Seagate’s earnings report had noted, “The preliminary results for the fiscal third quarter do not include the impact of any potential new restructuring activities, future mergers, acquisitions, financing, dispositions or other business combinations the company may undertake.”
Samsung, meanwhile, estimated lower earnings earlier this month, and is getting involved in areas far removed from the hard disk business, such as biopharmaceuticals.
The Journal quoted Richard Kugele, an analyst at Needham & Co., as saying “there is really no legitimate alternative” to a sale of the unit to Seagate other than for Samsung to shut it down.
The last disk drive up when the music stops playing? Toshiba (which started all this in 2009 with its acquisition of Fujitsu
), with 11% of the market — which could either be considering a Samsung purchase itself, or planning its own exit strategy with a sale to Seagate, Mellor suggested.
April 12, 2011 9:26 PM
Posted by: Sharon Fisher
Republican presidential candidate Mike Huckabee, former governor of Arkansas, is being slammed in the press due to accusations that, when he left the governor’s office in 2007, he ordered a number — by some reports, up to 91 — hard disks of information from computers in his office destroyed, which he later said was to protect sensitive information such as citizens’ Social Security numbers on them.
An Arkansas citizen filed a lawsuit, but Attorney General Dustin McDaniel in July, 2007, said he could find no reason to pursue any action against Huckabee over the destroyed hard drives, according to a Politico story at the time. A total of three suits were filed, none of which went anywhere.
Why is this coming up now? After Huckabee announced his 2012 candidacy, left-leaning Mother Jones researched the issue and published an article on it.
“In February, Mother Jones wrote to the office of Arkansas Gov. Mike Beebe seeking access to a variety of records concerning his predecessor’s tenure, including Huckabee’s travel records, calendars, call logs, and emails. Beebe’s chief legal counsel, Tim Gauger, replied in a letter that “former Governor Huckabee did not leave behind any hard-copies of the types of documents you seek. Moreover, at that time, all of the computers used by former Governor Huckabee and his staff had already been removed from the office and, as we understand it, the hard-drives in those computers had already been ‘cleaned’ and physically destroyed.”"
Several blogs and other publications, ranging from Reason
to Gizmodo, have picked up the story. Matt Browner Hamlin of Americablog Elections
, for example, suggested that Huckabee’s actions could have been related to a pardon he gave to a man who then committed further crimes.
“It doesn’t seem like a far stretch to connect the destruction of these hard drives to wanting to prevent information about his decision to pardon Maurice Clemmons, who went on to murder four police officers in Washington state after his release,” the blog noted.
Similarly, the AllGov blog also speculated on Huckabee’s motives, including another pardon gone wrong.
“During his tenure he faced numerous ethics complaints relating to use of state funds for personal expenses and failure to report gifts and outside income. Huckabee was also embarrassed when he agreed to an early release of convicted serial rapist Wayne Dumond, who, once back out in the world, raped and murdered at least one more woman.”
Huckabee himself also responded, denying the claims and suggesting in U.S News and World Report that Mother Jones was acting out of partisan motives.
“The absurd insinuation that my office ‘destroyed’ state records or that records are ‘missing’ is the same old political canard that was attempted years ago and failed then for the same reason it will fail now—it’s factually challenged,” U.S. News quoted Huckabee as saying.
This led Mother Jones to substantiate its claims, posting a copy of the 2007 memo to Huckabee from his IT department that all the disks had been rewritten seven times and then crushed. (Organizations that are lax about protecting data can certainly take a lesson from this, and even Huckabee’s detractors must admire his thoroughness.) A mirror copy of the network drives was also made and given to a Huckabee aide, who refused to speak with Mother Jones.
“Can Huckabee—a potential presidential contender who extols the cleansing virtue of transparency—explain why these records were destroyed, and what happened to the backups handed to his aide?” Mother Jones wrote.
It’s interesting to look at more contemperaneous accounts of the incident. For example, Computerworld said at the time that
“Hard drives in 83 PCs and four servers were destroyed, according to Claire Bailey, director of the Arkansas Department of Information Systems (DIS). She said that her office backed up information from the servers but not the PCs, and gave the backup tapes to Huckabee’s former chief of staff. The DIS apparently did not retain a copy of the data on the backup tapes.”
Huckabee, who did not respond to Mother Jones‘ request for a comment, had commented to Computerworld at the time.
“”This is not about destroying state property, this is about honoring our obligation to protect the privacy of the thousands of people who had personal data on those hard drives,” Huckabee said in a statement e-mailed to Computerworld. “We carried out recommendations from the Department of Information Systems to destroy the hard drives.”
Perhaps, but some people felt differently.
“We were taken a little by surprise that he went to the extreme lengths that he did to crush the hard drives without informing anyone ahead of time and without proper authority,” said Arkansas state Sen. Jimmy Jeffress, (D-Crossett).
In addition, the Arkansas Times at the time blasted Huckabee for his “graceless” leaving of his office, citing the destruction of the hard drives and the required replacement of them, which ended up draining a fund intended to help Arkansans in the case of emergencies such as tornadoes. “Maybe Huckabee had learned from Richard Nixon, who neglected to destroy the evidence and paid a price,” the editorial said.
There was also some question about the cost to replace the drives. According to Huckabee, both in 2007 and in his response to Mother Jones, replacing the disk drives cost $13,000. However, the 2007 Computerworld article quoted a spokesman for the incoming governor that the staff had to use $335,000 from the governor’s operating budget to purchase new computers as well as new hard drives: A total of 22 refurbished computers with new hard drives, 27 new desktop computers and 22 new laptops. Huckabee said in his response to Mother Jones, which used a $350,000 figure, that that was because the new governor wanted new computers.
The Times went on to follow the lawsuit story, noting that Huckabee perhaps had good reason to want to destroy the drives.
“Critics, however, recalled that early in Huckabee’s term as governor, documents, e-mails and memos stored on hard drives just like the ones that were destroyed formed the basis of embarrassing stories about Huckabee, including a 1998 story in the Arkansas Times detailing how Huckabee and his family were using the $60,000-a-year Governor’s Mansion fund as their personal piggy bank. As revealed in documents provided to the Times by a former governor’s office employee, the Huckabee family had used the mansion fund — which was supposed to be used only for purchases related to official state business — to buy everything from pantyhose and dog houses to meals out and loaves of Velveeta cheese.”
Ironically, Huckabee ran for President in 2008 on a platform of transparency, proposing that every federal government expenditure should be published online within 24 hours, Mother Jones reported, quoting Huckabee as saying, ”We should demand transparency and accountability from our government.”
However, one thing seems clear: Regardless of whether the governor’s office spent $13,000 or $350,000 to replace the hard drives, and regardless of whether the data on the hard drives was incriminating or merely sensitive, the appearance
is an issue. As the saying goes, in politics perception is more important than reality, and even if Huckabee had nothing but the best intentions in mind, his destruction of the disks gives the perception that he was hiding information that should legitimately be available to the people of Arkansas and the U.S. With legal experts now saying that governments are subject to the same electronic discovery requirements
as any corporation, Huckabee’s actions could be considered suspect or legally liable.
As one commenter to Mother Jones noted, “Yes. I’m sure his motive was responsible data management. Nice try.”
April 6, 2011 6:35 PM
Posted by: Sharon Fisher
CDMI stands for Cloud Data Management Interface and is an industry standard defined and controlled by the Storage Networking Industry Association (SNIA).
“The SNIA CDMI architecture standard defines the functional interface that applications will use to create, retrieve, update and delete data elements from the cloud,” according to Mezeo Software (quoting the SNIA), which announced this week that it planned to support the standard in its cloud storage products. “Based on a REST HTTP protocol, the CDMI standard requires adopters to implement strong access controls and to provide for encryption of the data on the underlying storage media for secure multi-tenant cloud environments.”
The SNIA goes on to say that CDMI lets clients discover the capabilities of the cloud storage offering, use this interface to manage containers and the data that is placed in them, and lets administrative and management applications manage containers, accounts, security access and monitoring/billing information. In addition, metadata can be set on containers and their contained data elements through this interface, SNIA says.
In other words, CDMI means that users have a standard interface for performing such functions as backups, and defines a set of standard terminology regarding users and types of data, regardless of the underlying storage technology in the cloud.
Vendors such as Bycast, Cisco, Hitachi Data Systems, Iron Mountain, NetApp, Olocity, Oracle, and QLogic have taken part in developing the specification, which came out in February, 2010 after the group was formed in 2009. There is also a mailing list devoted to the specification.
Like other industry standards before it, such as TCP/IP, vendors will be holding “plugfests” to ensure that their different implementations of the CDIA specification can work together. One will be held later this month in Colorado.
CDMI is increasingly becoming of interest to users; according to a recent survey of users from Storage Strategies NOW, 53% said that SNIA’s CDMI will be part of their cloud storage RFPs/proposals; and 30% of respondents said SNIA’s CDMI was very important for a public/hybrid cloud standard.
March 31, 2011 11:32 PM
Posted by: Sharon Fisher
You know, it’s not even that March was all that unusual. But here, on World Backup Day, it’s worth looking at some of the incidents that happened this month:
- The personal information — including the names, Social Security numbers, addresses, phone numbers, and dates of birth – of 13,000 individuals who had filed compensation claims with BP after last year’s disastrous oil spill may have been potentially compromised after a laptop containing the data was lost by a BP employee.
- The world’s largest stem cell bank, Cord Blood Registry, mailed data-breach warning letters to some 300,000 people after storage tapes and a laptop were stolen from an employee’s car
- Insurer Health Net waited until March 14 to disclose a data breach discovered on Jan. 21 involving the loss of nine server drive and the data of 2 million customers, employees, and health care providers.
- A USB memory stick containing the details of around 4,000 people has been lost by Leicester City Council.
- Taxpayers’ Social Security numbers, confidential child abuse reports and personnel reviews of New Jersey workers nearly went to the highest bidder after the state sent surplus computers out for auction.
What the heck is going on?
Sadly, it’s not even all that unusual. And to make matters worse, such breaches are getting more expensive. According to the Ponemon Institute
, which did a survey for Symantec Corp., data breaches continue to cost organizations more every year. The average organizational cost of a data breach this year increased to $7.2 million, up 7 percent from $6.8 million in 2009. Total breach costs have grown every year since 2006. Data breaches in 2010 cost their companies an average of $214 per compromised record, up $10 (5 percent) from last year, the Institute said.
Such incidents are so prevalent that the Online Trust Alliance recommends
that organizations have a plan in place for dealing with them, indicating it’s an issue of not if, but when. The only winners in these situations appear to be the credit-monitoring bureaus.
Part of the problem is that the lost data wasn’t always encrypted (though in the Leicester case, it appears the data was encrypted and the stick was stolen deliberately). On the other hand, how often does it happen that people lose the password or the key, or through some other action lose legitimate access to their data?
One thing does seem clear: People aren’t learning. The Leicester, New Jersey, and Health Net incidents were followups to similar incidents in 2009.
March 26, 2011 8:13 AM
Posted by: Sharon Fisher
When new rules for electronic discovery of documents in civil cases went into effect in December, 2006, there was some discussion at the time about whether governments and other public entities would also be subject to the same rules.
It’s taken more than four years, but it’s starting to look like they do.
According to a recent article in Law Technology News, “Recent decisions indicate that, despite the narrower scope of pretrial criminal discovery, the government may well be held to the same high standards of preservation and production of electronically stored information (ESI).” The article goes onto cite several such decisions.
In one case, the Federal Bureau of Investigation was criticized for not retaining copies of BlackBerry messages sent to a defendant. Consequently, the jury was given what’s called “adverse inference instructions,” which the article said “permits (but does not require) a jury to presume that the lost evidence is both relevant and favorable to the innocent party.” The jury subsequently found the defendant not guilty of all charges.
The FBI was lucky. Companies, in similar cases, have been fined up to $1.5 billion for failing to maintain records that the court considered discoverable.
In another case the article cited, the judge explicitly said that ”[l]ike any ordinary litigant, the Government must abide by the Federal Rules of Civil Procedure. It is not entitled to special consideration concerning the scope of discovery, especially when it voluntarily initiates an action.”
In fact, in some courts, there is a movement afoot to use the e-discovery rules — originally defined for civil procedures — for criminal procedures as well, because there is not a corresponding set of rules for such procedures, the article went on to say.
Is the government ready? Since 2007, IE Discovery Inc. said it has surveyed legal, records management, and information technology (IT) personnel within the federal government about trends in e-discovery, and it recently released its 2010 survey, 2010 Benchmarking Study of Electronic Discovery Practices for Government Agencies Survey. The survey included 46 government attorneys, paralegals, and IT personnel from 24 government agencies.
Results included the following:
- More than two-thirds of participants report that e-discovery processing is handled in-house.
- 61% of those surveyed claimed to be “more confident” in their ability to manage e-discovery.
- Government agencies have no standard approach to impose and manage litigation holds.
- Many agencies do not engage in early data assessment to reduce the amount of data that must be processed and reviewed.
- More than 40 percent of the agencies say that their e-discovery burden grew in the past year.
- The number of agencies reporting budgeting as a top concern jumped by almost 30 percent from 2009 to 2010.
- Almost one-half of agencies are now collecting “structured data” in repositories, databases, and similar systems.
- The form of production varies greatly. Almost 40 percent of respondents report producing discovery requests in image and text formats, 37 percent in native file formats and only 41 percent on paper.
Now, not all of these results are good news. 41% still respond to requests on paper? Are you kidding? More than half aren’t collecting structured data in systems? How in the world are they doing it, in Longaberger baskets? They have no standard approach and do no early data assessment? Oy! Still, I’ll take IE’s word for it that this is an improvement. The courts seem to be indicating, though, that they’d better improve faster.
March 18, 2011 3:35 PM
Posted by: Sharon Fisher
, ipad 2
Several analyst firms have come out with reports in the past week saying that flash storage production could be affected by the Japanese earthquake. In particular, this could delay manufacturing of Apple’s popular iPad 2.
iSuppli, in particular, has issued three separate press releases in the past week regarding the issue, one about the iPad 2 specifically, one about delays in components in general, and one about the industry’s dependence on Japanese-made components. For example, “Japanese companies, mainly Toshiba Corp., account for 35 percent of global NAND flash production in terms of revenue,” the company said.
Reuters quoted DRAMeXchange as saying that spot prices of NAND flash chips increased on Tuesday by nearly 3 percent after a 20 percent jump on Monday.
Micron, based in my home state of Idaho, stands to gain, according to several analysts quoted in an article by Matt Phillips in the Wall Street Journal. While Micron, too, has manufacturing facilities in Japan (despite what Raymond James chip analyst Hans Mosesmann was quoted as saying in Barron’s), they were located in south central Japan and were undamaged, according to an article by Anne Wallace Allen in the Idaho Business Review.
However, even undamaged facilities might take time to start up again, iSuppli warned. “While some of these suppliers reported that their facilities were undamaged, delivery of components from all of these companies is likely to be impacted at least to some degree by logistical issues now plaguing most Japanese industries in the quake zone. Suppliers are expected to encounter difficulties in getting raw materials supplied and distributed as well as in shipping out products. They also are facing difficulties with employee absences because of problems with the transportation system. The various challenges are being compounded by interruptions in the electricity supply, which can have a major impact on delicate processes, such as semiconductor lithography.” Aftershocks are also a factor, the organization warned.
iSuppli also noted that actual shortages aren’t likely to hit until later in the month or April, because there is typically a two-week inventory in the supply chain. However, prices are already going up due to the “psychological effect” of the earthquake, the company said.
While Japan is no stranger to earthquakes, the power of this one dwarfed previous quakes, said Jim Handy of Objective Analysis in a report on March 11. “The Taiwan earthquake in 1999 that caused significant damage in Taipei and stopped fabs in Hsin Chu was a magnitude 7.6, less than one tenth the power of Japan’s earthquake. The 1989 Loma Prieta earthquake that stopped production in Silicon Valley measured 6.9, or one hundredth the strength of today’s earthquake. Prior Japan earthquakes that have caused concerns to the semiconductor industry have been far smaller than today’s, including a 5.9 magnitude earthquake in September 2008, two measuring 6.0 and 6.8 in July 2007, and one measuring 6.9 in March of 2007.”
Handy also updated the company’s mondo chip map to reflect information it had learned from the various manufacturers since the earthquake.
Earthquakes can have multiple effects on fabrications plants, Handy said in an earlier report on a 2007 Japanese quake. “Typically an earthquake will disrupt the processing of any wafers that are on a photolithographic tool at the time that the earthquake struck,” he said. “Although a very large earthquake in close proximity to a fab can cause physical damage to the structure that is greater than the damage the building is designed to sustain, most fabs are designed to accommodate the kind of earthquake that is typical to the area. Fabs are built on a special floating floors that isolate the internal equipment from external vibration ranging from tiny earth tremors or vibrations from a passing truck to minor earthquakes. Greater earthquakes may not cause damage but their vibrations can result in incidental damage to the products being processed.
“If there is a power loss, no matter how brief, wafers in a high-temperature process may have to be scrapped,” Handy continued. “If the power loss lasts 20-30 minutes or longer there may also be a period of unexpected downtime as furnaces are brought back to a stable temperature. Another possible difficulty would be possible breaches in the clean environment. Earthquake damage may even require recalibration and further losses of work in progress (WIP) than are spelled out here. Losses could run into multiple days, stopping product flow for a week or more.”
Ironically, typically flash memory chips are cheaper this time of year, according to PCB Design 007. However, due to the earthquake, as well as to increased demand for iPads, that may be different this year, the website said.
March 14, 2011 11:46 PM
Posted by: Sharon Fisher
It’s difficult to write this, when the full extent of the earthquake damage to northern Japan isn’t yet clear and the nuclear crisis is still escalating. But now — when the images are fresh not only to you but also to the managers who approve your disaster recovery projects — is the best time to think about how your company would handle being in a similar situation, as overwhelming and impossible to believe it might be right now.
Think about it. How many places are really safe from natural disasters? We’ve already seen how the Icelandic volcano shut down flights all over Europe. The Bay Area, Seattle, and Portland are all geologically active; many parts of the Southeast are vulnerable to hurricanes; the central U.S. is prone to tornadoes. In addition to earthquakes and volcanoes in the Asia Pacific area, the region is also subject to typhoons.
It’s easy to think that having a backup or replication system in place is enough, but after watching the widespread devastation in Japan, it’s clear that we need to be thinking about how to scale up our ideas of what kind of disaster we’re planning for.
1. Where are your backups, replicated servers, etc. located? Same building? Same city? Same state? If you didn’t realize it before, it’s clear now that a disaster can cover a massive distance and that backups need to be geographically dispersed, perhaps through the cloud. Also, even if you’re using the cloud, where is the data center actually located? If it’s someplace subject to natural disasters, such as earthquake- and wildfire-prone areas in California, it may not help you much. I know some companies that choose to have their backup sites located near Spokane, Wash., because it’s geologically boring.
2. And while you’re at it, how well is your company set up for remote employees? If employees are evacuated, is there a way they can work from where they are? Can employees in other parts of the world pick up the slack?
3. How well is your site and your backup site set up for emergency power? A big part of the problem with the Japanese nuclear reactors was that they didn’t plan for an extended power outage. While there were batteries to operate the cooling system, they lasted only a few hours. Some colocation facilities keep diesel fuel on hand to run generators; does yours? How long will it last?
4. The good news — and there is some — is that the Internet reportedly held up remarkably well. Renesys, which has performed some interesting analyses of Internet shutdowns in Libya, has observed that much of the country’s Internet traffic was unchanged. “It’s clear that Internet connectivity has survived this event better than anyone would have expected,” the company wrote in its blog. “The engineers who built Japan’s Internet created a dense web of domestic and international connectivity that is among the richest and most diverse on earth, as befits a critical gateway for global connectivity in and out of East Asia. At this point, it looks like their work may have allowed the Internet to do what it does best: route around catastrophic damage and keep the packets flowing, despite terrible chaos and uncertainty.”
Consequently, communication with people outside the disaster zone has been better than after some natural disasters, with many people able to check in with loved ones fairly quickly, using social media such as Facebook and Twitter.
So think about your plan. Meanwhile, there are many ways to follow the developing situation in Japan, and to help victims in the ravaged country. Google, in particular, has collected a list of resources to keep informed about what’s happening. It could just as easily have been any of us, so think about how you can help.