Yottabytes: Storage and Disaster Recovery

August 16, 2012  12:57 PM

Storage No Longer a Barrier to Ubiquitous Government Surveillance

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

Big Brother image via Shutterstock

Big Brother image via Shutterstock

“Within the next few years an important threshold will be crossed: For the first time ever, it will become technologically and financially feasible for authoritarian governments to record nearly everything that is said or done within their borders – every phone conversation, electronic message, social media interaction, the movements of nearly every person and vehicle, and video from every street corner.”

This is due to a 1-million-times improvement in the ability to store information since 1985, according to John Villasenor, an electrical engineer at the University of California, Los Angeles, as well as a senior fellow at the Brookings Institution.

As an example, it would cost just $50 for an entire year of location data to 15-foot accuracy for 1 million people, updated every five minutes, 24 hours a day, Villasenor said in a seminar earlier this year. Similarly, storing the audio from telephone calls made by an average person in the course of a year would require about 3.3 gigabytes and cost just 17 cents to store, a price that is expected to fall to 2 cents by 2015, he said.

Scott Shane in the New York Times blog called attention to Villasenor’s work, which was published in a Brookings Institution paper date last December called Recording Everything: Digital Storage as an Enabler of Authoritarian Governments.

“In the 1960s, the National Security Agency used rail cars to store magnetic tapes containing audio recordings and other material that the agency had collected but had never managed to examine, said James Bamford, an author of three books on the agency,” reported Shane. “In those days, the agency used the I.B.M. 350 disk storage unit, bigger than a full-size refrigerator but with a capacity of 4.4 megabytes of data. Today, some flash drives that are small enough to put on a keychain hold a terabyte of data, about 227,000 times as much.”

Civil liberties organizations have increasingly been concerned about the amount of government surveillance that has been permitted, ranging from data that can be obtained from cellphones with no warrant required (though Rep. Markey (D-Mass.) has put forth a bill to limit that) to location data that the Sixth Circuit Court has ruled doesn’t require a warrant.

Moreover, Villasenor notes, individual people are providing a great deal of such data themselves, through the use of social media, mobile location apps, and so on.

But it is the rapidly declining cost of storage that makes such surveillance possible, Villasenor says. “Over the past three decades, storage costs have declined by a factor of 10 approximately every 4 years, reducing the per-gigabyte cost from approximately $85,000 (in 2011 dollars) in mid-1984 to about five cents today,” he writes. “In other words, storage costs have dropped by a factor of well over one million since 1984 [My note: an ironic benchmark to use]. Not surprisingly, that fundamentally changes the scale of what can be stored.”

These technological improvements put it within the reach of a country to store all the data it can obtain, Villasenor says. “For a country like Syria, which has a population of 15 million people over the age of 14, the current cost to purchase storage sufficient to hold one year’s worth of phone calls for the entire country would be about $2.5 million – a high number but certainly not beyond governmental reach,” he writes. “If historical cost trends continue, the annual cost in 2011 dollars to purchase enough storage for Syria’s government to record all calls made in that country will fall to about $250,000 by 2016 and to about $25,000 by 2020.”

While video data takes up much more space, limited video data –such as recording license plate numbers — is becoming increasingly prevalent in various states throughout the U.S., most recently in Minnesota. “Over the course of a full year, a system of 1,000 roadside license plate reading cameras each producing 1 megabit per second would generate image data that could be held in storage costing about $200,000,” Villasenor writes. “The resulting database of license plate numbers (as opposed to the images used to obtain the numbers) could be stored for a small fraction of this cost.”

The so-called “Peaceful Chongqing” universal video surveillance project in China — ostensibly for public safety — could cost as little as 25 cents per person per year by 2020, Villasenor writes.

Villasenor’s paper was focused on what he called authoritarian governments. Extrapolating costs to the U.S. was presumably left as an exercise for the reader.


July 31, 2012  10:47 AM

Don’t Have an Encryption Key? Go to Jail

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

Christian Szell: Is it safe?… Is it safe?
Babe: You’re talking to me?
Christian Szell: Is it safe?
Babe: Is what safe?
Christian Szell: Is it safe?
Babe: I don’t know what you mean. I can’t tell you something’s safe or not, unless I know specifically what you’re talking about.

— Marathon Man

The topic of whether one is or is not compelled to produce a key to an encrypted disk drive in response to a law enforcement request has come up before (and, in fact, in the U.S. has come up at least five times, according to a DefCon presentation by Electronic Frontier Foundation attorney Marcia Hoffman).

Now that, in at least some cases, U.S. judges are ruling that individuals need to surrender the passwords to decrypt their disks, an interesting side issue has also come up, particularly for those countries that already have laws requiring people to provide encryption keys to law enforcement: What happens if the person doesn’t have the key? Or says they don’t?

Such key disclosure laws are in effect in countries including India, Australia, and a variety of European nations.

But most of the attention has fallen on the U.K., where in October, 2007, a law was put in place that people could be jailed for up to five years for refusing to release an encryption key.  Ostensibly, it was for national security and to help prevent terrorism.

And the law has been used. In 2007, it was applied — against a group of animal rights activists. Then in 2009, the first person – who had been diagnosed as mentally ill – was jailed for nine months for refusing to turn over a key. In 2010, a teenager was sentenced to 16 weeks in prison under a similar charge.

Earlier this month, a U.K. blogger, Rick Falkvinge, brought attention to the possibility that, should law enforcement decide that a random number generator, for example, was actually an encrypted file, one could be jailed for refusing to provide the nonexistent key.

This is not the first time the subject has been raised – Cisco blogged about a similar topic in 2009 – but with the increased focus on encryption and decryption – typically using the spectre of child pornography (according to Hoffman, four out of the five U.S. encryption key cases had to do with alleged child pornography), there is new concern being brought to the issue and how to deal with being forced to prove a negative: That you don’t have a key that law enforcement insists you do.

“So imagine your reaction when the police confiscate your entire collection of vacation photos, claim that your vacation photos contain hidden encrypted messages (which they don’t), and sends you off to jail for five years for being unable to supply the decryption key?” writes Falkvinge.

July 29, 2012  3:23 PM

Black Hat Vendors Resort to Booth Babes #blackhat

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

“Hey, did you see the girls dressed up in the BDSM outfits in the exhibit room?”

And people wonder why women don’t want to get involved in the computer industry.

To be fair, a number of men, as well as women, objected to the booth babes in the exhibit room at Black Hat this week in Las Vegas. In fact, the person who told me about them said, “What do they think this is, the 1980s?”

But really? We’re still having this discussion? In 2012?

To be sure, this is not criticizing the young women who choose to make their living this way. Nor is it ignoring the Las Vegas context of the conference, where women in similar costumes, or less, could be seen in just about any casino. The criticism is to the vendors in a professional conference who believe that this is the best way to attract attendees to their booths.

(Interestingly, the more counter-culture DefCon, also held that week and also held in Las Vegas, apparently didn’t feel the need to do this.)

Women security experts attending a meeting of the Executive Women’s Forum, a Scottsdale, Ariz.-based organization of more than 750 female security professionals in the computer industry, generally rolled their eyes about the two vendors that chose to promote their products this way. Several of them expressed surprise that a major vendor such as RSA would feel the need to resort to such tactics, noting that it’s typically expected more from smaller vendors.

Indeed, the other vendor featuring scantily clad women in its booth was a smaller vendor, SecureNinja – but on the other hand, to judge from the swag people were carrying during the conference, attendees seemed primarily interested in the toy ninja swords the company was also handing out. Which only goes to show that vendors don’t have to resort to scantily clad women to attract visitors, male or female.

This is not to pick on Black Hat in specific – this is apparently endemic among security shows. And what annoyed some EWF members was not just the attire of the booth babes at these shows, but the fact that, typically, the women aren’t actually capable of discussing the companies’ products – with one of them reporting that she took it upon herself at one show to teach them what public key infrastructure meant.

EWF members also pointed out what they saw as a more egregious offense: that of all the speakers in the conference, only three were women. Moreover, some male speakers made jokes about women in their presentations, such as the one who pretended to be confused between new Yahoo! CEO Marissa Mayer and Sports Illustrated swimsuit model Marisa Miller.

Ironically, one of the other speakers — Mark Weatherford, Deputy under Secretary for Cybersecurity for the National Protection and Programs Directorate (NPPD), who was representing the Department for Homeland Security — mentioned in his presentation, “We have a problem. There are not enough smart people in the public or private sector to help us defend our country.”

If this is the case, can we really afford to alienate half of them?

July 22, 2012  8:48 AM

EMC, VMware Move Executives Around

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

EMC isn’t known for having drama-filled management shakeups, which is why the most exciting part of the recent executive shuffle between it and VMware was when word leaked that VMware CEO Paul Maritz was on the way out before it became clear where he was going — to EMC as “top strategist.”

Much handwringing ensued at first, with VMware stock dropping 3% and some analysts criticizing the company for taking its eye off the virtualization ball, with some industry watchers speculating he was being “pushed out” and might head an EMC cloud spinoff.

Now that Maritz’ future at EMC is settled, and VMware is now headed by EMC chief operating officer Pat Gelsinger — himself often said to be a potential future EMC head — people are back to talking about how well EMC and VMware are doing, and stock for both companies went up.

In a world where CEOs resign 20 minutes after their appointment so they can collect a $44 million severance payment, you can’t blame people for getting excited about EMC and VMware, where the most exciting aspect is whether CEO Joe Tucci is going to retire in 2012 or 2013. (He’s now planning to retire in 2013. Unless he stays another year.)

Maritz is a former Microsoft executive with a lovely South African accent whose biggest claim to fame is coming up with the phrase “eating your own dog food” for companies that use their own products while Gelsinger is a former executive at Intel who was said to be being groomed for that CEOship. As always, EMC has a deep bench of qualified, non-drama-king executives.

VMware and EMC have an interesting relationship. VMware is 79% owned by EMC (and accounts for 60% of EMC’s own value), but operates fairly independently for all that. While some observers speculated this may mean further grooming for Maritz as a potential CEO, and that Gelsinger moving to VMware meant he was no longer in the running, the Wall Street Journal and others seemed to believe the EMC CEO slot would primarily be between Gelsinger and CFO David Goulden, who was named COO.

Whether either Maritz or Gelsinger is expecting a child didn’t come up — apparently that’s only a issue for female tech executives — though the Wall Street Journal did mention that they were each planning to stay in the Bay Area to be close to grandchildren.

July 17, 2012  7:59 AM

Maine Politician du Houx Accused of Behaving Badly With Data Storage

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

We’re used to hearing stories about politicians erasing things from hard disk drives that they shouldn’t have. Now we’re hearing about a politician allegedly putting things onto a hard disk drive that he shouldn’t have.

Maine legislator Rep. Alex Cornell du Houx (D-Brunswick) was the subject of a temporary protection from abuse order this spring filed by Rep. Erin Herbig (D-Belfast), after their relationship ended, according to the Bangor Daily News. The two people made a private agreement, which was not filed in court but was said to be legally binding, around the relationship and their future contact.

“In addition to having no contact with Herbig, the private agreement, which was obtained by the Bangor Daily News, requires that Cornell du Houx pay all of Herbig’s $9,000 in legal fees and turn over any computer hard drives and data storage devices with pictures of Herbig or ‘any other women sleeping or in a state of undress’ to Herbig’s lawyer.”


Apparently, after their relationship ended, du Houx had allegedly “stalked, harassed and threatened her,” including taking more than 100 photos and video of Herbig sleeping.

Without addressing the veracity of the charges, let’s look at the data storage issues involved. (The private agreement doesn’t appear to be available online; excerpts that the paper posted didn’t cover these aspects.)

  • The agreement would appear to mention only hard drives and data storage devices. Apparently, if du Houx had any pictures of sleeping or undressed women stored in the cloud, those are fair game. (Not to mention, sleeping or undressed men.)
  • As Bangor Daily News letter writer Sid Duncan legitimately pointed out (in an otherwise kind of gross and disgusting letter to the editor on July 16 that the paper has since retracted), what if some of the data storage devices are actually state-owned and contain state documents?
  • What if the data storage devices contain communication with du Houx’ attorney? Passing them to Herbig’s attorney could violate attorney-client privilege.
  • What is Herbig’s attorney intending to do with the data storage devices? Is he a computer forensics expert who could legitimately retrieve any such images, even if deleted, from the devices? Is he intending to do so? Is he going on a fishing expedition to see what other charges could be filed – such as, conceivably, child pornography or stalking? Has he obtained a warrant for this? And what is he planning to do with the images and data storage devices? Is he planning to destroy these images? Or, as Duncan implied, will he be, um, “perusing” them? If he does destroy those images, could he be accused of destroying evidence (as in 2007’s United States vs. Philip D. Russell)? But if there is child porn on any of the devices, could the attorney himself then be charged with possession of child pornography?
  • What about the privacy and rights of any other sleeping or undressed women who might also have images – obtained legitimately or otherwise – stored on du Houx’ devices? Especially since he appears to be a professional photographer? Or, in fact, any other personal pictures he might have of other people? Or, for that matter, of himself?
  • Will du Houx get his data storage devices and data back, including all the data that isn’t of sleeping or undressed women?



July 11, 2012  8:05 PM

Gartner Updates Backup Magic Quadrant, Sort Of

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

If you’ve been holding your breath waiting for the new Gartner Enterprise Backup/Recovery Magic Quadrant to come out so you could see how vendors had moved around since last year, you can let go. Other than FalconStor moving from Visionary to Niche, and several players in the Niche quadrant changing around primarily due to acquisitions, there wasn’t much change. Commvault, EMC, Symantec, and IBM are still in the Leaders quadrant; HP (now Autonomy, an HP company) and CA are still in the Challengers quadrant; and NetApp is now all alone in the Visionary quadrant. The Niche Players in this Magic Quadrant are Acronis; Asigra; EVault, a Seagate Company; FalconStor Software; Quest Software and Syncsort.

However, Gartner did change some of its strategic planning assumptions (that is, predictions) between last year and this. While it still believes that by 2014, 80% of the market will choose advanced, disk-based appliances and backup software-only solutions over distributed VTLs, it now believes that one-third, not 30%, of organizations will change backup vendors due to frustration over cost, complexity and/or capability, and it will be by 2016, not 2014.

It also now believes that by 2015, at least 25% of large enterprises will have given up on conventional backup/recovery software, and will use snapshot and replication techniques instead, and that by the end of 2016, at least 45% of large enterprises, up from 22% at year-end 2011, will have eliminated tape for operational recovery.

And it is no longer predicting that by 2014, deduplication will cease to be available as a stand-alone; rather, it will become a feature of broader data storage and data management solutions; while the company didn’t say so explicitly in this report, product descriptions seemed to indicate this has largely taken place already.

Other interesting tidbits from the report include:

  • Gartner end-user inquiry call volume regarding backup has been rising at about 20% each year for the past four years.
  • “The rising frustration with backup implies that the data protection approaches of the past may no longer suffice in meeting current, much less future, recovery requirements,” Gartner notes. “As such, companies are willing to adopt new technologies and products from new vendors, and have shown an
  • increased willingness to switch backup/recovery providers to better meet their increasing service levels.”
  • Companies are increasingly considering cloud-based recovery systems, predominantly for midsize enterprise servers and branch-office and desktop/laptop data.
  • Symantec currently owns 34.1% of the market, which has decreased over the past five years. IBM and EMC have 17.3% and 17.0% market share, respectively.
  • No other vendor has more than a 7% market share.
  • In 2011, CommVault and EMC increased their market shares.
  • Along with Symantec, CA Technologies, IBM and Quest Software slid slightly in market share in 2011.

Gartner also identified five trends that it believes will  emerge over the next several years:

  • Re-expanding the number of backup solutions and technologies
  • Backup application switching
  • Decreasing backup data retention
  • Backup modernization
  • Deployment of new technologies and vendors

June 30, 2012  3:24 PM

Cloud Storage Proves Not So Resilient After All

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

Oh, hey, just put all your stuff in the cloud. It has IT people to watch it and take care of it, and if there’s any sort of problem, it’s replicated in other places so you’ll still have access to it whenever you want.

Remember that?

It turns out, not so much.

Major Internet services such as Netflix, Pinterest, and Instagram were taken offline on Friday night due to thunderstorms that took out power to the Amazon Web Services site in Virginia.

Amazon’s Cloud services status page was full of power-related error messages,” wrote MSNBC’s Bob Sullivan. “Amazon’s ElastiCache, for example, indicated that starting at 8:43 p.m., the service was “affected by a power event.”  At 9:25 p.m., this message was posted: “We can confirm that a large number of cache clusters are impaired. We are actively working on recovering them.””

But, oh, Sullivan continued, this wasn’t really Amazon’s fault, because the storm was just so very bad. Why Amazon didn’t have backups or replicated copies elsewhere on the network, he didn’t say. (Apparently the problem may have been a routing issue.)

“Outages like this morning’s are a reminder of how fragile, still, our digital architectures actually are,” intoned The Atlantic‘s Megan Garber. “As much as we try to bolster them against the elements, they are made of sand, not stone. Buildings can be brought down by storms; but so, today reminds us, can their digital counterparts. Even the structures that lack structures can be torn by nature’s whims. That is, in its way, terrifying. And yet — here’s the other sliver — it is also, just a tiny bit, reassuring. No matter how advanced we get, today reminds us, nature will always be one step ahead.”

This is probably not a great consolation to those companies that have moved their operations to the cloud because they were assured it would still be there in a disaster. And this is a disaster? A thunderstorm (albeit one that has caused at least a dozen deaths)? Are those companies feeling “reassured” today by discovering that their disaster recovery systems are, in fact, vulnerable to an outage that might be in a completely different part of the country?

What happens if there’s an earthquake, hurricane, or some more severe natural disaster? If the Red Cross or FEMA loses its connectivity because it depended on the cloud, will its managers philosophically fold their hands and talk about how in the great scheme of things this shows just how little we all are? Will the people asking to be saved or helped see it this way?

Instead, hopefully Amazon and other cloud providers will take this as a wake-up call before hurricane season gets going, and ensure that the virtual cloud can stand up to the real thing.


June 28, 2012  7:04 PM

Google Docs Adds Offline Support

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

In a move that has been expected for the past month — and desired for much longer than that — Google has made it possible for Google Docs users to edit documents offline and then synchronize them when the user logs back in again. Google called offline editing “one of its most requested features” for Google Drive, which the company said has 10 million users.

At present, the function works only with Google Docs — that is to say, document files using Google Drive — but is expected to be available for spreadsheets and presentations at some point in the future, Google said in its instructions. It is also only available for the Chrome browser, and Google didn’t say whether it expected to make the feature available to other browsers.

Pundits are claiming this functionality will negatively affect other consumer cloud storage systems, such as Dropbox. “Google Kneecapped Dropbox,” proclaimed Business Insider.

The one other point that’s worth noting is that changes made to the online file while the user is offline take precedence over whatever changes the offline user makes.

“If an online collaborator deletes the text you edit while offline, their changes will override yours. If a collaborator deletes the document you’re editing offline, your changes will be lost when you come back online because the document will no longer exist. Try to use offline editing for documents that you own and that won’t be deleted without your knowledge.”

Well, yeah, but that sort of defeats the purpose of using Google Drive for collaboration in the first place, doesn’t it?

Another point — only enable Google Drive on your own computers.

“Enabling offline access on public or shared computers can put your data at risk, since others may be able to view your synced Google documents and spreadsheets.”

Which also sort of seems to defeat the purpose — wouldn’t a great use case for this feature be “I’m traveling without my computer or it broke, and so I’m using somebody else’s”? Might be nice to have a “Mr. Phelps” feature that automatically deleted itself after syncing a file, or on command.

Needless to say, features that require the Internet won’t be available online — sharing, publishing, reporting a problem, etc. Surprisingly, however, so is inserting an image or a picture.

Google made the announcement at its sold-out I/O conference in San Francisco.


June 27, 2012  11:45 PM

New Mitt Romney Email Messages Surface

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

As you may recall, a few months back I did a piece on Massachusetts Governor and candidate (now Republican presidential nominee) Mitt Romney, talking about how he not only deleted all his email messages when he left office, but had his staff members buy 17 hard disk drives. used in his office (reportedly spending $100,000 of government funds to do so).

Turns out, he wasn’t quite thorough enough. The Wall Street Journal, upon discovering that email of one cabinet member, then-Administration and Finance Secretary Thomas Trimarco, had been accidentally retained, made a public records request for copies of emails between Trimarco and top Romney officials, and reportedly got 73 pages’ worth.

Some of them are available here but it doesn’t really matter. They’re about the implementation of his health care plan (aka “Romneycare”), and honestly, I don’t care what they’re about. What’s interesting is the process of finding them, and how even someone who went to as much care as Romney to scrub his past was still tripped up by missing one guy’s email cache.

The Journal article, by Mark Maremont, also mentioned in passing that Romney occasionally used a private email account for discussing official business, which is non-optimal on a business  basis, let alone politically. Moreover, such a tactic doesn’t protect a person from an electronic discovery request; typically they cover all email accounts that a person might use, not just the business one — it just makes complying with the request more of a challenge for the IT department.

This wasn’t the only recent news on the Romney disk drive front. A few days later, while leaving a note to journalists following his campaign teasing them about the cushy bus they got to travel in, he added, “PS — erased your hard drives.

While he obviously didn’t do that, the note gave all the reporters the opportunity to rehash the erased disk drive story from his gubernatorial days, as well as gave President Obama’s campaign the opportunity to criticize him.

“Mitt Romney may joke about how his staff erased government hard drives to keep his records secret, but what we do know about his record as Governor is anything but funny — he left the state 47th in job creation and number one in per capita debt in the nation,” Obama spokesman  Danny Kanner said in a statement.

June 20, 2012  9:48 AM

Protecting Your Data Center During Wildfire Season

Sharon Fisher Sharon Fisher Profile: Sharon Fisher

With data centers increasingly being built in less-urban areas, and with the increasing number of wildfires in recent years, this sort of disaster needs to be added to the panoply of hurricane/tornado/earthquake for disaster recovery.

Last summer, the data center at the Los Alamos National Lab in New Mexico was surrounded on two sides by a 60,000-acre fire, while in 2007, the data center at Pepperdine University was threatened by a 1,200-acre fire in Malibu, Calif., that came within 100 feet of it.

More recently, we’ve had the fires in Colorado, which led one data center manager to post to Slashdot asking for advice. While there were the usual number of jokes, tangents, and speculation about his motives, there was also useful advice for data center managers as fire season approaches. (And most of this advice is useful for disasters in general.)

  • Have a disaster recovery plan and make sure it’s updated — for example, are all the contacts and their phone numbers correct? “DR plans are a living document that should be updated for every significant change to your infrastructure,” noted Slashdot user Macgrrl. “They should have an annual ‘trial run’ to see if they work. The worst time to find out your DR plan doesn’t work is in an actual disaster event.”
  • Priorities are people, data, equipment.
  • This is one advantage of using the cloud — data is by definition offsite.
  • Perform regular offsite backups.
  • Make sure the network is documented and up-to-date, with the documentation available electronically and offsite. Save configuration settings to a text file and store it both electronically and on paper.
  • Label everything — including AC adapters to keep from zapping things afterwards.
  • Take pictures of the cabling for documentation purposes.
  • If you have to save equipment, focus on disk drives and servers first. And keep in mind that insurance that reimburses for equipment lost in a fire might not reimburse for equipment damaged in a bugout.
  • To save time, use wire cutters to disconnect cables (*not* power cables!).
  • Cover things you’ve left behind with plastic or trash bags to help protect them from water and smoke.
  • Consider setting up your data center to be portable in the first place — set it up in a shipping container, put racks on wheels (and make sure doors are wide enough to move them through and you have a forklift if necessary), use quick-disconnect hard drive enclosures, buy a truck or van to store onsite, etc.

Any disaster plan should be able to cope with ‘and then a giant foot appeared above the building and squished it flat,'” noted Slashdot user GirlInTraining. “Yours should be no different. It might not be a wild fire that threatens your servers… it could be a UPS that shorts out, or a tornado, flood, a failed fire suppression unit, or simple human incompetence.”

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: