Oh, yeah, and there was a Royal Wedding.
But ten minutes before that (not that they were trying to hide anything, of course), Amazon also released the post-mortem of its extended Elastic Compute Cloud (EC2) outage of the previous week.
In case you were under a rock, a number of major computer sites — including foursquare, Reddit, and Quota — were down for a day, sometimes more, on April 21, due to a problem with Amazon’s web hosting business. It wasn’t until Monday or Tuesday of this week that all the sites really recovered.
If you’re familiar with the concept of “thrashing,” where a too-full hard disk or computer memory is so busy trying to find places to work that it doesn’t get anything done, that’s basically what happened to Amazon, on a mammoth scale. Due to a configuration problem, the cloud went down, and the first thing all the servers did when they came up was try to re-mirror themselves — which they couldn’t do because all the other servers that were up were trying to do the same thing. The actual summary goes into a lot more detail, if you really want to know, but that’s basically it.
So now the Internet is seeing a storm of a different kind: A pundit storm where people talk about 1) What It All Means and 2) Where We Go From Here and 3) Could It Happen Again?
1) S*** happens. 2) Don’t have a single point of failure, duh. 3) Of course.
Oh, you wanted more detail?
What it all means is that people are human and machines are stupid. This does not change, and will not change. Count on it. Problems happen. Then we institute new systems that help us protect against the most recent problem, and wait for a new problem to happen.
You know, like the TSA.
Where We Go From Here is that Amazon is instituting a number of changes in processes and procedures, both human and machine, that are intended to keep this from happening again.
Organizations that use the cloud — anybody’s cloud, not just Amazon’s — should take this as a wake-up call. Even if you weren’t affected by this outage, you could be on the next one. Don’t just have a backup. Have a backup for the backup. Yes, it costs money. How much money does it cost for your business to be out for a day? (Even if Amazon did give all its affected customers a freebie.) Forrester analyst Rachel Dines wrote a blog post listing a number of questions organizations should ask their cloud provider about backups and failover strategies.
Finally, accept that it’s going to happen — whether it’s from a natural disaster like the earthquake in Japan or the tornadoes in the American South, government action to shut down the Internet like in Egypt, widespread electrical failures, or simply a flu pandemic. As Dines says, “Assume nothing” — check every step in the disaster recovery plan, and figure out what the alternative is for every component that could fail.]]>
Dropbox made a point of telling Steve Kovach at Business Insider, who broke the story, that this was a rephrasing of its terms of service, not a change in policy. “The TOS update was merely a clarification for users, not a policy update,” the company said.
Dropbox also pointed out that it wasn’t alone in this. “It is also worth noting that all companies that store user data (Google, Amazon, etc.) are not above the law and must comply with court orders and have similar statements in their respective terms of service.”
A number of articles about the incident concurred with this, including Business Insider’s. “This is nothing groundbreaking, but Dropbox has updated its security Terms of Service to say that if the government asks, they will have to decrypt user’s files and turn them over. That’s standard practice for any online storage service from Gmail to Amazon”.
But Business Insider went on to say, “and shouldn’t affect the average user unless they’re doing something wrong.”
That’s where it gets sticky.
Several other articles on the subject made similar comments. “In the meantime, don’t go doing anything that’ll get you in so much trouble that the G-Men need to decrypt your email or cloud storage,” said David Gerwitz of ZDNet, whose article headline, “If you have something to hide from the government, don’t use Dropbox” also implied that only those who had something to hide should be concerned. “Ok, so no worries–so long as you’re not doing anything wrong, you should be fine,” agreed Sarah Jacobsson Purewal of PC World. Comments in the PC World story went so far as to say that the only people who would be concerned about this would be pedophiles.
Recall that in 2005, the New York Times revealed that the National Security Agency was monitoring telephone calls, without warrants, of domestic callers. A few months later, USA Today revealed that this was going on with the cooperation of a number of telephone companies, including AT&T, Verizon, and Bell South.
“[T]o say that only the “guilty” have any reason to care about privacy shows a dangerous lack of awareness of how easy it is to violate some law or regulation and thereby become “guilty” yourself,” says William Morriss, a Senior Associate patent attorney of Frost Brown Todd, writing in the Ephemeral Law blog. “Even worse, when the government goes about collecting enormous amounts of data without having to justify itself and without any oversight, there will inevitably be false positives which have the potential to literally ruin someone’s life.”
The one solution Dropbox has to offer is that users can encrypt their own files before upload them to a data storage service like Dropbox — so that if the data storage service decrypts stored files, they continue to be encrypted, which only the user can decrypt. “Dropbox does not discriminate between the types of files stored in your Dropbox nor the applications used to open those files. This means you can use your own software encryption methods, such as third-party encryption software, to keep your files secure on your terms,” the company’s Terms of Service said.
However, it doesn’t say exactly how one goes about finding or using third-party encryption software. Moreover, there are those who fear that any encryption software — unless it’s open source, where people can examine it — could have a “back door” that would allow government agencies to decrypt it without user assistance. Attempts have been made, and continue to be made, to require such a back door. Some people, consequently, are sticking with “better safe than sorry” and using only open source encryption software. Unfortunately, this goes beyond the area of “easy to use” for the average — law-abiding — user.]]>
Didn’t we just hear about this?
In this case, however, it’s not Hitachi GST that’s doing the selling, but Samsung Electronics, which — like Hitachi — was primarily involved in the spinning disk market and had less of a presence in the solid-state disk (SSD) market and would face expensive retooling to support it, according to the article in the Wall Street Journal on Sunday that sparked all this.
The potential purchaser? Seagate Technologies, which was leapfrogged by the Western Digital-Hitachi GST merger, which took up almost 50% of the market, according to iSuppli. Seagate accounted for 29% of hard disk drive shipments in the fourth quarter, while Samsung accounted for 10%, iSuppli said. In addition, sales of hard disk drives are down 4% in Q1 compared with Q4, iSuppli said.
Perhaps Seagate — which considered and rejected a Hitachi purchase itself — didn’t want to miss out a second time. And unlike a Hitachi purchase, which might have courted an antitrust claim, a Samsung purchase would be in the consumer marketplace, rather than the enterprise market Seagate and Hitachi share, according to Jason Mick at DailyTech.
The source for all this? “A person familiar with the matter,” who said the Korean Samsung was hoping for $1.5 billion (compared to the $4.3 billion Hitachi fetched), but might settle for $1 billion.
Seagate itself wouldn’t comment, but Chris Mellor of The Register noted earlier this month, in a piece about Seagate’s earnings, that its chairman and CEO, Stephen Luczo, was spending three months in the Far East, and that the Seagate’s earnings report had noted, “The preliminary results for the fiscal third quarter do not include the impact of any potential new restructuring activities, future mergers, acquisitions, financing, dispositions or other business combinations the company may undertake.”
Samsung, meanwhile, estimated lower earnings earlier this month, and is getting involved in areas far removed from the hard disk business, such as biopharmaceuticals.
The Journal quoted Richard Kugele, an analyst at Needham & Co., as saying “there is really no legitimate alternative” to a sale of the unit to Seagate other than for Samsung to shut it down.
An Arkansas citizen filed a lawsuit, but Attorney General Dustin McDaniel in July, 2007, said he could find no reason to pursue any action against Huckabee over the destroyed hard drives, according to a Politico story at the time. A total of three suits were filed, none of which went anywhere.
Why is this coming up now? After Huckabee announced his 2012 candidacy, left-leaning Mother Jones researched the issue and published an article on it.
“In February, Mother Jones wrote to the office of Arkansas Gov. Mike Beebe seeking access to a variety of records concerning his predecessor’s tenure, including Huckabee’s travel records, calendars, call logs, and emails. Beebe’s chief legal counsel, Tim Gauger, replied in a letter that “former Governor Huckabee did not leave behind any hard-copies of the types of documents you seek. Moreover, at that time, all of the computers used by former Governor Huckabee and his staff had already been removed from the office and, as we understand it, the hard-drives in those computers had already been ‘cleaned’ and physically destroyed.”"
“It doesn’t seem like a far stretch to connect the destruction of these hard drives to wanting to prevent information about his decision to pardon Maurice Clemmons, who went on to murder four police officers in Washington state after his release,” the blog noted.
Similarly, the AllGov blog also speculated on Huckabee’s motives, including another pardon gone wrong.
“During his tenure he faced numerous ethics complaints relating to use of state funds for personal expenses and failure to report gifts and outside income. Huckabee was also embarrassed when he agreed to an early release of convicted serial rapist Wayne Dumond, who, once back out in the world, raped and murdered at least one more woman.”
Huckabee himself also responded, denying the claims and suggesting in U.S News and World Report that Mother Jones was acting out of partisan motives.
“The absurd insinuation that my office ‘destroyed’ state records or that records are ‘missing’ is the same old political canard that was attempted years ago and failed then for the same reason it will fail now—it’s factually challenged,” U.S. News quoted Huckabee as saying.
This led Mother Jones to substantiate its claims, posting a copy of the 2007 memo to Huckabee from his IT department that all the disks had been rewritten seven times and then crushed. (Organizations that are lax about protecting data can certainly take a lesson from this, and even Huckabee’s detractors must admire his thoroughness.) A mirror copy of the network drives was also made and given to a Huckabee aide, who refused to speak with Mother Jones.
“Can Huckabee—a potential presidential contender who extols the cleansing virtue of transparency—explain why these records were destroyed, and what happened to the backups handed to his aide?” Mother Jones wrote.
It’s interesting to look at more contemperaneous accounts of the incident. For example, Computerworld said at the time that
“Hard drives in 83 PCs and four servers were destroyed, according to Claire Bailey, director of the Arkansas Department of Information Systems (DIS). She said that her office backed up information from the servers but not the PCs, and gave the backup tapes to Huckabee’s former chief of staff. The DIS apparently did not retain a copy of the data on the backup tapes.”
“”This is not about destroying state property, this is about honoring our obligation to protect the privacy of the thousands of people who had personal data on those hard drives,” Huckabee said in a statement e-mailed to Computerworld. “We carried out recommendations from the Department of Information Systems to destroy the hard drives.”
“We were taken a little by surprise that he went to the extreme lengths that he did to crush the hard drives without informing anyone ahead of time and without proper authority,” said Arkansas state Sen. Jimmy Jeffress, (D-Crossett).
In addition, the Arkansas Times at the time blasted Huckabee for his “graceless” leaving of his office, citing the destruction of the hard drives and the required replacement of them, which ended up draining a fund intended to help Arkansans in the case of emergencies such as tornadoes. “Maybe Huckabee had learned from Richard Nixon, who neglected to destroy the evidence and paid a price,” the editorial said.
There was also some question about the cost to replace the drives. According to Huckabee, both in 2007 and in his response to Mother Jones, replacing the disk drives cost $13,000. However, the 2007 Computerworld article quoted a spokesman for the incoming governor that the staff had to use $335,000 from the governor’s operating budget to purchase new computers as well as new hard drives: A total of 22 refurbished computers with new hard drives, 27 new desktop computers and 22 new laptops. Huckabee said in his response to Mother Jones, which used a $350,000 figure, that that was because the new governor wanted new computers.
The Times went on to follow the lawsuit story, noting that Huckabee perhaps had good reason to want to destroy the drives.
“Critics, however, recalled that early in Huckabee’s term as governor, documents, e-mails and memos stored on hard drives just like the ones that were destroyed formed the basis of embarrassing stories about Huckabee, including a 1998 story in the Arkansas Times detailing how Huckabee and his family were using the $60,000-a-year Governor’s Mansion fund as their personal piggy bank. As revealed in documents provided to the Times by a former governor’s office employee, the Huckabee family had used the mansion fund — which was supposed to be used only for purchases related to official state business — to buy everything from pantyhose and dog houses to meals out and loaves of Velveeta cheese.”
Ironically, Huckabee ran for President in 2008 on a platform of transparency, proposing that every federal government expenditure should be published online within 24 hours, Mother Jones reported, quoting Huckabee as saying, ”We should demand transparency and accountability from our government.”
As one commenter to Mother Jones noted, “Yes. I’m sure his motive was responsible data management. Nice try.”]]>
“The SNIA CDMI architecture standard defines the functional interface that applications will use to create, retrieve, update and delete data elements from the cloud,” according to Mezeo Software (quoting the SNIA), which announced this week that it planned to support the standard in its cloud storage products. “Based on a REST HTTP protocol, the CDMI standard requires adopters to implement strong access controls and to provide for encryption of the data on the underlying storage media for secure multi-tenant cloud environments.”
The SNIA goes on to say that CDMI lets clients discover the capabilities of the cloud storage offering, use this interface to manage containers and the data that is placed in them, and lets administrative and management applications manage containers, accounts, security access and monitoring/billing information. In addition, metadata can be set on containers and their contained data elements through this interface, SNIA says.
In other words, CDMI means that users have a standard interface for performing such functions as backups, and defines a set of standard terminology regarding users and types of data, regardless of the underlying storage technology in the cloud.
Vendors such as Bycast, Cisco, Hitachi Data Systems, Iron Mountain, NetApp, Olocity, Oracle, and QLogic have taken part in developing the specification, which came out in February, 2010 after the group was formed in 2009. There is also a mailing list devoted to the specification.
Like other industry standards before it, such as TCP/IP, vendors will be holding “plugfests” to ensure that their different implementations of the CDIA specification can work together. One will be held later this month in Colorado.
CDMI is increasingly becoming of interest to users; according to a recent survey of users from Storage Strategies NOW, 53% said that SNIA’s CDMI will be part of their cloud storage RFPs/proposals; and 30% of respondents said SNIA’s CDMI was very important for a public/hybrid cloud standard.]]>