Yottabytes: Storage and Disaster Recovery

September 30, 2017  5:44 PM

Cobbling Together the Toshiba Memory Chip Sale

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Apple, Flash, samsung, toshiba, western digital

We’ve been waiting a while to find out about the Toshiba memory chip sale, it looks like it might finally be settled: A sale, sort of, with an eventual IPO.

Sort of?

“While it says that it sold its chip division, it didn’t really,” writes Tim Culpan for Bloomberg Gadfly. “It merely pawned it to Bain until it can afford to buy it back again.”

As you may recall, this all started this spring when Toshiba revealed it had lost a lot of money constructing, of all things, nuclear plants. At that time, the company said it intended to sell its memory chip division, hoping to raise at least $18 billion from it, and was also hoping to complete the sale by June. Which it obviously didn’t do. Without the sale, Toshiba faces delisting from the stock exchange due to its losses.

There’s a lot of moving parts to this deal:

  • Toshiba is investing 350.5 billion yen and in return gets 40.2 percent of the company.
  • SK Hynix (which used to be Hyundai Electronics) is investing 395 billion yen (around $3.5 billion) and in return will get less than 15 percent of the company.
  • Hoya is investing 27 billion yen — 1.4 percent of the money — and in return gets 9.9 percent of the company.
  • A partnership led by Bain Capital — appropriately called Pangea — is investing a total of 415.5 billion yen (around $3.7 billion) and in return gets 49.9 percent of the company. Those partners include:
  1. Apple, which is investing 165 billion yen (around $1.47 billion). Why does Apple care? Because Toshiba is the second-biggest manufacturer of the flash memory chips that its iPhone and iPad use. Why can’t it use the number-one manufacturer? Because that’s iPhone competitor Samsung.
  2. Dell, Seagate and Kingston, which are investing a total of 250 billion yen, with Seagate specifically saying it would invest up to $1.25 billion.
  3. Bain Capital, which itself is investing an additional 212 billion yen.

Why don’t the percentages of investment and equity match? Because, for example, SK Hynix is taking a smaller percentage of equity to avoid antitrust issues, Culpan writes. Another factor is that between Toshiba and Hoya, Japanese companies still retain a majority interest in the company – “a keen wish of the Japanese government,” Reuters writes. SK Hynix and the American companies will not have voting rights; their primary interest is access to the unit’s chips.

There were, in fact, so many moving parts that a press conference on the deal was cancelled because the participants hadn’t agreed on some of the details, according to Reuters.

In addition, Western Digital is continuing to throw a cog into the works. “A Western Digital subsidiary, SanDisk, shares ownership with Toshiba of a flash memory production operation in Japan,” explains Jonathan Soble in the New York Times. “Because of that, the American company contends that its approval is necessary for Toshiba to sell the chip unit. Western Digital – which in September had been rumored to have bought the Toshiba unit itself — said this week that it would seek an injunction against the deal.” In the meantime, Toshiba and SanDisk are undergoing arbitration to settle the multiple lawsuits they’re filing against each other, according to Toshiba.

That has also led to state-sponsored Innovation Network Corp. of Japan and Development Bank of Japan backing out of the consortium, write  Pavel Alpeyev and Yuki Furukawa for Bloomberg. In the meantime, what could happen is that the three joint ventures owned by Western Digital could be withdrawn from the sale, they write.

Assuming the whole thing works out – it is expected to close by March 31 — Culpan expects an IPO around 2020.

September 26, 2017  1:57 PM

Time to Get Baked: Baked Tapes, That is

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Storage, Tape

Okay, now it’s time to get baked. Baked tapes, that is. What did you think we were talking about?

The concept of baking tapes has been known for some time – there’s even a 1993 patent on it – but the subject came up again recently in the context of a series of British text-based adventure games from the 1980s, Magnetic Scrolls. Adherents were excited to find backup tapes from Magnetic Scrolls, which were thought to have been lost, and wanted to convert them to a form of magnetic media from this century so they could be revitalized.

“For over 30 years the only copy of the data lay on now obsolete TK50 tape cartridges,” writes Hugh Steers, a founding member and core developer of Magnetic Scrolls who has now founded Strandgames to bring them back, in a blog post describing this process. “You might think, a bit like I did (foolishly), that high quality backups such as these, on proper DEC backup tape media, made with an 1980s DEC MicroVax, would essentially be able to remain on the tape almost indefinitely, providing the tapes themselves are kept in good condition – like in a cupboard, drawer or even perhaps an attic. You would think then, the biggest problem is somehow locating someone with a compatible system and tape drive set up and working. If you can find that system, just pop in the tapes and read away – job done! No problem. Back in time for tea and biscuits!”

(I did say they were British.)

“Immediately we hit the problem,” Steers laments.”The tapes would not read. Turns out that old tape suffers the, so-called, Sticky Shed syndrome.”

“Sticky shed” doesn’t refer to an adhesive small building in the back yard, but to a phenomenon where tape components attract water and as a consequence become both “sticky,” which makes it hard for them to go through the tape heads, and “shed” the oxide in which the information is stored, meaning they lose data.

“The problem goes back to the 1970’s when most tape manufacturers made an ill-advised decision to change the formulation of the ‘binder’ used to glue the magnetic tape particles to the plastic base material,” explain Mike Rivers and Graham Newton in one of the canonical guides to tape baking. “Unknowingly, the new formulation attracted moisture, and eventually enough accumulated to make the tape go ‘sticky.’” Certain kinds of tape are more susceptible to the problem than others.

“Do you remember old music cassettes?” Steers writes. “Remember those times when, all of a sudden the music goes a bit weird and muffled, followed by a disconcerting tape chewing sound. You open the tape compartment to find a massive ball of tangled tape knotted up and totally ruined! Well, imagine the same thing, but with your only data backup, Uh-oh!”

The solution to this problem is to remove the water from the tape medium. Hence, baked tapes.

“The purpose of ‘baking’ is to drive out all the moisture that the tape binder has accumulated, which is what caused it to go sticky in the first place,” Rivers and Newton write. “This will give a few weeks to a few months of ‘normal’ tape functioning… enough time to transfer the affected recordings to a stable medium before the problem reappears when more moisture is absorbed.”

The exact methodology for baking tapes is pretty mystical. The temperature needs to be 130 degrees Farenheit (plus or minus 5-10 degrees), which means a typical oven won’t work because they can’t go too low. Other devices people have used to bake tapes include:

It should go without saying, don’t use a microwave, especially if the tapes are on a metal spool. If the tape is on a plastic spool, it should probably be rewound on a spool of a different material before baking. Experts also say not to use a gas oven. “Gas produces water vapor when it burns, and that is what you are trying to drive out of the tape,” Rivers and Newton write. And while we’re doing disclaimers, this apparently doesn’t work on acetate tapes, but sticky shed typically doesn’t happen with acetate tapes.

How long to bake the tapes also varies, ranging from 2 to 8 hours. Needless to say, all of this is dependent on the specific tapes in question: How much water they have, how much they’ve already deteriorated, how much moisture is in the air, and so on. Hence the mysticism.

(There is also a huge amount of arcane detail on how best to treat the tapes before, during, and after baking.)

The final question remains: How long is a baked tape good for? And that also varies. Some say it’s only good for one read after that, so be sure to read it immediately into a new device. Others say the tape could be good for some time. Certainly, immediately transferring it to another medium seems like the wisest course of action.

Finally, I haven’t done this myself (though I know people who have), and I can’t make any promises or commitments about how well this could work or what damage you could do to your tapes by doing so. No warranty expressed or implied, etc.

The happy ending in this particular case is that, between baking the tapes and finding a way to clean the heads simultaneously while reading them, the sticky shed Magnetic Scrolls tapes have now been read into a new medium and can now be rebuilt, Steers writes.

September 19, 2017  11:33 AM

Forget Your Password, Go to Jail

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Encryption, privacy, Security

Remember the guy who got put in jail for contempt for forgetting his hard disk drive encryption passwords? He’s still in there, and doesn’t have any prospects for getting out anytime soon.

Francis Rawls, a former sergeant in the 16th district of the Philadelphia Police Department, was accused of having child pornography on two encrypted Macintosh hard drives, which were seized in March, 2015. He was ordered by a judge in August, 2015, to provide the passcode to decrypt the drives, but he claims to not remember it. He was put in jail for contempt of court. Prosecutors claim Rawls is “forgetting” his password on purpose to keep from being charged with possessing child pornography, which could put him in prison for 20 years.

Earlier this year, Rawls appealed the ruling, and it was denied. More recently, he attempted to be let out of prison by claiming that there was an 18-month limit on how long someone could be jailed for contempt of court. In addition, in the meantime, he hopes that the case will eventually go to the Supreme Court.

Prosecutors said, though, that his claims weren’t valid because the precedent he was citing had to do with someone who was a witness, and Rawls wasn’t a witness. In fact, prosecutors have defined the case very narrowly in a way that doesn’t give him a lot of protection. “After the government had seized the contemnor’s computers and was unable to decrypt several of the hard drives, it filed a motion with Judge Rueter under the All Writs Act, 28 U.S.C. § 1651, for an order directing Rawls to produce a decrypted copy of the hard drives,” notes a recent briefing. “The procedural posture is significant. The government did not proceed before the grand jury. It did not subpoena him as a witness.”

As you may recall, the whole legal issue of whether people can be compelled to give up their passwords is still being fought in the courts. Courts have been deciding back and forth on the issue for several years now, with some ruling that a phone password is more like the combination to a safe than a physical object such as a key. It matters because something that is the expression of one’s mind, like the combination to a safe, is protected under your Fifth Amendment rights not to incriminate yourself. A physical key, something you possess, is something you can be forced to produce.

In this particular case, prosecutors are claiming that the Fifth Amendment doesn’t apply because it is a “foregone conclusion” that the hard disk drives contain child pornography. That’s because, even though they can’t read the files, they know what the encryption scheme hashes them to. And the hashes of those files apparently are equivalent to the hashes of other common child pornography files.

The appeals court earlier this year also found that the Fifth Amendment doesn’t apply because Rawls didn’t use that defense in the first place, when he showed up to decrypt the hard disk drives and then said he couldn’t remember the passwords. “[B]y failing to appeal the original All Writs Act order or to raise the Fifth Amendment as a defense to the contempt proceeding, he had procedurally defaulted on his Fifth Amendment challenge,” prosecutors write.

Not to mention, prosecutors – who made a point of defining the term “chutzpah” in their briefing and applying it to Rawls – set a new definition for “disingenuous” themselves, by saying it wouldn’t really be “testifying” against himself, because they weren’t really asking for Rawls’ passwords. They were just asking him to type them in. “The government deliberately chose not to call Rawls as a ‘witness’ to minimize Fifth Amendment issues,” prosecutors write. Those Fifth Amendment “issues” being his ability to use it to protect himself. “Thus, the government did not seek to compel him to produce his password. Rather, it sought to compel him to perform a physical act.” That “physical act” being typing in his passwords.

Right. I’m not robbing you, because I’m not asking for your money. I’m just asking you to take it out of your wallet and put it on the table, which happens to be within my reach.

(Though I do have to thank prosecutors for teaching me a new word: “contumacious” – stubbornly or willfully disobedient to authority.)

Prosecutors also implied that 18 months for contempt was nothing, citing other cases where people had been jailed for five years and seven years for contempt. And because Rawl has already lost in appeals court, they are not sanguine that the case will ever make it to the Supreme Court, they add.

Earlier this month, Judge Cynthia Rufe agreed with prosecutors, citing the finding of the previous appeal as the reason. “The ruling of the Court of Appeals compels the conclusion that Mr. Rawls is not a witness to a proceeding as contemplated by § 1826, and that the 18-month limitation therefore does not apply to this matter,” she writes. “This matter exists before the Court solely because Mr. Rawls has prevented the search warrant from being fully executed.”

Consequently, Rawls stays in jail, though prosecutors said they should check in on him now and then to see if, after two years of largely solitary confinement, he suddenly remembers his passwords. “Theoretically, he could be held in jail for contempt forever … until he’s dead,” Dan Terzian, a lawyer from Duane Morris, tells Olivia Solon in The Guardian.

The moral of the story? Don’t forget your password. You could go to jail.

September 13, 2017  3:44 PM

Data Centers Weather Harvey, Irma

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Data Center, Disaster Recovery

Hurricanes in the Southeast in September aren’t a surprise, or shouldn’t be. That said, having two hit the region within a matter of a few days, as well as having another potential one waiting in the wings, tested the mettle of operators. But all told, damage to data centers appeared minimal thus far.

Some data center regions have been hit pretty seriously by hurricanes, most notably Hurricane Sandy in 2012. Hitting New York City, the home of many high-tech firms, the storm took out many data centers that were below ground level. Power failures resulted in bucket brigades of diesel fuel being taken up stairs to the data centers on higher ground. And even thunderstorms have taken out cloud data centers such as Amazon Web Services in Virginia in 2012.

But Harvey and Irma don’t appear to have done massive damage to data centers thus far. Four major Internet providers in Houston stayed up, though the data centers themselves were inaccessible due to flooding, reports Yevgeniy Sverdlik in Data Center Knowledge. The biggest problem was due to fears that they would run out of diesel fuel, he reports, though at least during the thick of the storm they hadn’t even lost utility power. Other sources also indicated that Houston data centers were by and large unaffected.

Some staff stayed in Houston data centers for days. “The facilities had showers and were stocked with food, cots, video games, and books,” Sverdlik writes in a different Data Center Knowledge piece. “Stocking up on sleeping cots and supplies is a customary part of data center operators’ emergency preparedness plans.” In previous disasters, data centers have warned that the most critical resource is people and making sure that they’re safe, and a number of data centers had to put up some of their people when their homes were uninhabitable, he writes.

For Irma, Miami was particularly critical because it serves as a hub linking the U.S. with Latin America, Sverdlik writes in another Data Center Knowledge piece. However, most of the networks using that facility had alternate paths, he added. The building, like many Florida data centers, was rated for Category V winds and was 32 feet over sea level. While reports are still coming in, Florida data centers appeared to also pretty much stay up, though some were on backup power and generators for a time.

Either way, Verizon declared a “Force Majeure event” – essentially, an Act of God —  for Hurricanes Harvey and Irma that let it off the hook any delay or inability by Verizon or its vendors to provide services.

Even for companies that aren’t located in regions affected by hurricanes, these events were a useful wake-up call to update disaster recovery plans. In addition, the fortuitously timed DCD>Colo+Cloud conference, in Dallas on September 26, is planning to expand its coverage of disaster recovery and resiliency topics.

Incidentally, in a bravo-for-little-ironies department, Nirvanix — the company that was notorious for sending out press releases during a natural disaster encouraging everyone to use its products — went out of business in 2013.

August 31, 2017  10:55 PM

How to Destroy a Hard Drive? Ask Terry Pratchett

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Hard disk, hard drive, Storage

If you really want to make sure that nobody’s going to be able to read your data, the late author Terry Pratchett just showed you how it’s done: Per his instructions, his executor just ran over his hard disk drives with a steamroller.

“Pratchett’s hard drive was crushed by a vintage John Fowler & Co steamroller named Lord Jericho at the Great Dorset Steam Fair, ahead of the opening of a new exhibition about the author’s life and work,” reports The Guardian.

Pratchett, who died in March, 2015, at 66 from Alzheimer’s disease, reportedly told author Neil Gaiman of his wish, who revealed it in an August 2015 interview with the Times of London. “The fantasy author Terry Pratchett wanted his unfinished work to be run over with a steamroller, according to his close friend, the writer Neil Gaiman,” the paper reported at the time. “Gaiman, the award-winning author of The Sandman and Coraline, reveals that Pratchett, his confidant of 30 years, told him that he wanted ‘whatever he was working on at the time of his death to be taken out along with his computers, to be put in the middle of a road and for a steamroller to steamroll over them all.’”

Rob Wilkins, who carried out the instructions in the will, manages the Pratchett estate, and tweeted from an official Twitter account that he was “about to fulfill my obligation to Terry” along with a picture of an intact computer hard drive – following up with a tweet that showed the hard drive in pieces, the Guardian reports. The pieces will also become part of the exhibit.

Richard Henry, an official at The Salisbury Museum, where the exhibition will be held, told NPR that the task actually wasn’t easy. “It’s surprisingly difficult to find somebody to run over a hard drive with a steamroller. I think a few people thought we were kidding when I first started putting out feelers to see if it was possible or not.”

Even the steamroller didn’t destroy the hard disk drive, Henry continued. “The steamroller totally annihilated the stone blocks underneath but the hard drive survived better than expected so we put it in a stone crusher afterwards which I think probably finally did it in,” he told the BBC.

Why not just erase the hard disk drive, which reportedly had ten unfinished works on it? Because as any number of criminals have found out to their sorrow, “deleting” a file doesn’t really delete it — just the pointer to the file gets deleted. Much of the data in the file is still on the hard disk drive and can be scraped off by a diligent forensic analyst. Even deleting the file multiple times, rewriting the disk, and so on might not fully eliminate the data.

This is not to say that there weren’t plenty of people who were sad that Wilkins had been so thorough. (Including Gaiman, who said in the August 2015 interview that he was “ridiculously glad” the destruction had not yet happened.) In his lifetime, Pratchett wrote more than 70 books, selling more than 85 million copies worldwide, and no doubt many of his eager fans would have loved to see even an incomplete work.

But the author did not want his unpublished works to be completed by someone else and released, Henry told the BBC. In fact, Wilkins told the BBC in 2015 that what Pratchett really wanted was to have a device connected to his heartbeat so when his heart stopped it would wipe the contents of his hard drive.

Assuming, of course, that the hard disk drive that was crushed was actually the one that Pratchett had used. After all, he had Alzheimer’s; maybe he didn’t know what he was asking for? Maybe someone made a copy of it in the two years after Pratchett died. (Why it took two years before it was destroyed, no one has said.) We can always still hope. “It’s not impossible that some further fragment might surface in years to come, and this will all turn out to have been an elaborate joke on Pratchett’s part,” writes Stephanie Merritt in the Guardian. “I wouldn’t put it past him.”

“Mr. Pratchett is hardly the first author to request that his unpublished work be destroyed or hidden from public view,” reports Sophie Haigney in the New York Times. “Franz Kafka wanted his diaries, manuscripts and letters burned. Eugene O’Neill wanted the publication and performance of ‘Long Day’s Journey Into Night’ to be delayed until 25 years after his death. Vladimir Nabokov left instructions that fragments of a manuscript be destroyed. In all of these cases, though, the requests were ignored, and the unpublished work came to light.” Edward Albee has left a similar request but it isn’t clear whether it will be honored, she adds.

If you don’t happen to have a steamroller handy, other methods for ensuring the destruction of a hard disk drive include a sledgehammer, a .45, or taking it apart and destroying the disks inside.

August 29, 2017  12:41 PM

Nerd Out on Backblaze Hard Drive Statistics

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Backblaze, hard drive, Storage

Periodically, I like to pass on what Backblaze is reporting about the hard drives that make up its cloud backup service. This is for two reasons. First, Backblaze uses a truly massive number of hard drives, and so they come up with a lot of statistically significant results. Second, the company is absolutely nerdy about big data and hard drive statistics, and does quarterly and annual reports on its experiences that are a great example of how a company could do this sort of report about any hardware it happened to have.

Backblaze just came out with its second quarter report, but because I haven’t written about its individual hard drive statistics for more than a year, I’ll also catch you up on the last couple of quarters as well.

The company is now up to 83,151 hard drives altogether. In the first quarter, it added more than 10,000 hard drives in total, and in the second quarter, it added 635 new hard drives in total – some due to failure but many of them due to migrating to larger, higher density (as well as newer) hard drives. For example, Backblaze has been migrating its 3 TB hard drives to 8 TB  — which, the company said, more than doubles its storage capacity in the same footprint while only increasing its electrical use a little bit.

In addition to upgrading the hard drives themselves, Backblaze is also creating much larger collections of hard drives. Instead of using its “pods” of 45 hard drives, the company has been using “vaults” made up of up to 20 even bigger “pods,” each of which hold up to 60 hard drives. With the increased size of hard drives it’s now using, each “vault” can now store up to 14.4 petabytes of data.

Another interesting thing that Backblaze has been doing lately is testing enterprise-grade hard drives. As you may recall, the company became well known for building its storage system with commodity consumer hard drives rather than the monolithic gigantic storage devices made by companies such as EMC. That was cheaper, especially when it became time to upgrade, and was more granular. But the company has been criticized over the years for using consumer hard drives rather than enterprise hard drives, which some people (including the vendors whose hard drives weren’t very reliable in the Backblaze setup) said would be better suited for the way Backblaze used its drives.

So Backblaze has been testing enterprise hard drives, and surprisingly found that they were actually more prone to failure than consumer ones, as well as generally being more expensive. On the other hand, the company apparently found a batch of Seagate 8 TB enterprise hard drives on sale, and at that price they were worth getting, so the company is using some of them. While they are still showing a slightly higher failure rate, the company cautions us not to jump to conclusions, indicating that it might simply be burn-in failures because of how new they are (which the company calls the “bathtub curve”).

“The enterprise drives have 363,282 drives hours and an annualized failure rate of 1.61%,” writes Andy Klein, director of product marketing for Backblaze. “If we look back at our data, we find that as of Q3 2016, the 8 TB consumer drives had 422,263 drive hours with an annualized failure rate of 1.60%. That means that when both drive models had a similar number of drive hours, they had nearly the same annualized failure rate.”

In other developments, it may surprise you, but Backblaze doesn’t always leap to a new, more dense hard drive model as soon as it comes out; since it’s using a commodity model, it waits until the cost per megabyte for the more dense models is equivalent to that of the less dense models it’s already using, and then tests them. Consequently, the company is just now starting to test 12 TB hard drives. “In the next week or so, we’ll be installing 12 TB hard drives in a Backblaze Vault,” Klein writes. “Each 60-drive Storage Pod in the Vault would have 720 TB of storage available and a 20-pod Backblaze Vault would have 14.4 petabytes of raw storage.”

As it is, Backblaze spends 23 percent of its revenue on hardware, 90 percent of which is devoted to pods and vaults. The rest of the 47 percent of revenue devoted to costs includes space for the hard drives, electricity to run them and keep them cool, personnel to keep them happy and functioning, bandwidth to transfer data, and so on. The company’s remaining 53 percent of revenue is devoted to the operational expenses of keeping it running, such as developing new features, marketing, sales, office rent, and other administrative costs.

As always, the company releases an Excel spreadsheet with its data, as well as the entire datasets themselves, so you can geek out on hard drive data to your heart’s content.

Disclaimer: I am a Backblaze customer.

August 25, 2017  10:54 PM

Yet Another Installment of the Spokeo Case

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Data, law, legal

In the criminal justice system, cases sometimes rise and fall based on incredible arcane and trivial bits of the law.


Such is the case with Spokeo. As you may recall, it all has to do with Spokeo, a data aggregator, getting some data wrong, and the guy in question, Thomas Robins, suing them for it. The importance of the case was less a matter of the effect on the particular guy, and more a matter of what legal precedent would be set, based on court rulings. Hypothetically, it could have meant that people would be able to sue based on simple errors of procedure that happened to violate a law. That would have made the lawyers happy, but not many other people.

The Spokeo case went all the way to the Supreme Court, which ruled in May that the Ninth Circuit Court hadn’t done the job right in the first place and sent it back to them for a do-over. And apparently the Ninth Circuit wanted to be very thorough this time, because it took them more than a year to essentially cut-and-paste from the Supreme Court’s ruling.

Specifically, the Court directed the Ninth to determine two things about the case:

  • Whether it was particular. “There was no dispute that Robins had satisfied the particularity requirement of injury in fact,” writes James McKenna of Jackson Lewis, in National Law Review. “The injury he alleged was based on the violation of his own statutory rights and was due to the inaccurate information about him.”
  • Whether it was concrete. “A concrete injury must be ‘real’ and not ‘abstract,’” McKenna writes. “There are three aspects of concreteness. First, it is not synonymous with being tangible. A concrete injury need not be tangible. Intangible injuries, such as the abridgment of free speech rights, can be concrete.” Second, while Congress implementing a law is important, just because a law provides the right to sue doesn’t mean they can, he writes. “Third, the risk of real harm’ can constitute a concrete injury, even if the harm may be difficult to prove or measure,” he adds.

Where the Ninth had erred, the Supreme Court ruled, was by not considering those two points separately in the first place, especially the concreteness one.

To a certain extent, this seems like failing the calculus test because, even though you got the right answer, you didn’t show your work. But for attorneys, it’s all about showing your work and crossing all the Ts and dotting all the Is.

So, the Ninth Circuit came back earlier this month to say, yes, it was, too, concrete. “The Ninth Circuit looked to whether the FCRA [Fair Credit Reporting Act] was established to protect consumers’ concrete interests (as opposed to their purely procedural rights),” write Hanley Chew and Eric Ball in Mondaq. The Ninth Circuit went on to point out that both the Supreme Court and Congress have indicated in the past that having incorrect data in a database is a Bad Thing, going on to quote the Ninth Circuit ruling. “’The relevant point is that Congress has chosen to protect against a harm that is at least closely similar in kind to others that have traditionally served as the basis for lawsuit.’ Thus, informed by both Congress and historical practice, the Ninth Circuit held that Congress enacted the FCRA to protect consumers’ concrete interest in accurate credit reporting,” they continue.

Did that settle the case and Robins gets what he wants? No! It simply rules that he has standing, meaning it kicks the can back down the road to the District Court, and he can proceed with his case. “The District Court for the Central District of California originally dismissed the case, holding Robins failed to allege any injury-in-fact and, therefore, did not have Article III standing,” explains David Anthony in InsideARM. “The Ninth Circuit reversed, holding the alleged violation of Robins’ statutory rights alone was sufficient to satisfy Article III’s requirements, regardless of whether the plaintiff can show a separate actual injury.”

Keep in mind that Robins is suing Spokeo not for making him sound worse, but for making him sound better. “The report erroneously said Robins was married with children and that he was older, better educated, wealthier and more accomplished than he actually was,” write three attorneys from Sidney Austin LLP in Mondaq.

Those scoundrels.

The Ninth Circuit also took pains in its ruling to say that, just because the data was wrong in Robins’ case, didn’t mean that everybody who finds an error in their personal information in a database gets to sue. “The Ninth Circuit emphasized the case-specific nature of its hybrid approach, ‘caution[ing] that [its] conclusion on Robins’s allegations does not mean that every inaccuracy in these categories of information (age, marital status, economic standing, etc.) will necessarily establish concrete injury under FCRA,’” write four attorneys from K&L Gates in the National Law Review. “This is because “[t]here may be times that a violation leads to a seemingly trivial inaccuracy in such information (for example, misreporting a person’s age by a day or a person’s wealth by a dollar).”

And that’s the good news for Spokeo, and every other company that collects data on people: Having to determine just how much and what sort of bad data is acceptable – the data equivalent of the allowed number of insect parts in a jar of peanut butter – will make it harder for people to file class-action suits against data aggregators, the company told Perry Cooper of Class Action Litigation Report.

Even without the Ninth Circuit’s do-over, numerous – on the order of three a dayrulings are now referencing the Spokeo case. The problem now is that even after the Ninth Circuit decision, some questions still remain — which means this case could go back to the Supreme Court again. “The Ninth Circuit did not provide broad guidance about whether and under what circumstances a single inaccuracy in a credit report, a certain type of inaccuracy or another combination of inaccuracies would be sufficient to constitute a concrete injury,” write the Sidney Austin attorneys. “The court made clear that de minimis violations may not confer standing, but other than holding that Robins’s alleged facts were enough to confer standing, it did not provide clear guidance on where the line falls between sufficient and insufficient injuries.”


August 16, 2017  11:36 PM

Dreamhost Fights DoJ Inauguration Warrants — All 1.3 Million

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
government, privacy, Security

As you may recall, in February we covered the question of how many search warrants the government could legally expect to serve on Facebook at once, given that the company felt that the 381 it had received was too many.

“For example, could the government get a warrant for everyone who posted on Facebook that they had attended the Women’s March so it could arrest them or put them in some sort of database?” we asked presciently.

Little did we know.

As it turns out, the Department of Justice is asking the web hosting company Dreamhost to provide information about every visitor to a particular website, www.disruptj20.org, which was intended to help organize protests at the inauguration of President Donald Trump. Some 230 people, including six journalists, were arrested. (“J20” referred to January 20, the date of the inauguration.) But the DoJ is asking for information on all 1.3 million visitors to the website.

And it is asking for a lot of information: “names, addresses, telephone numbers and other identifiers, e-mail addresses, business information, the length of service (including start date), means and source of payment for services (including any credit card or bank account number), and information about any domain name registration,” as well as the content each person viewed.

In other words, even if you simply visited the website once, didn’t post any information, and didn’t attend any protests, the government would now have your information. Incidentally, disruptj20 did not keep logs of this data itself, but Dreamhost did, according to NPR.

So, just by virtue of researching this story, I’m now on this list. Twice.

“No plausible explanation exists for a search warrant of this breadth, other than to cast a digital dragnet as broadly as possible,” writes Mark Rumold of the Electronic Frontier Foundation, which is helping Dreamhost with its defense. “But the Fourth Amendment was designed to prohibit fishing expeditions like this. Those concerns are especially relevant here, where DOJ is investigating a website that served as a hub for the planning and exercise of First Amendment-protected activities.” The organization is also helping Facebook fight a similar request for information, but doesn’t even know whether it’s also about the inauguration, because of a gag order.

Dreamhost, which spilled the beans on all this on August 14 , is fighting the warrant on First and Fourth Amendment grounds, saying it is “overbroad.”

You think?

A hearing is scheduled for Friday.

Interestingly, the DoJ sent out its warrant on July 12. For an event on January 20? It doesn’t necessarily mean that the DoJ attorneys are slow, though they have been fighting with Dreamhost about this data since a week after the inauguration. The Electronic Communications Privacy Act Stored Communications Act changes the rules at 180 days. “Under the ECPA, emails on a server for more than 180 days is considered ‘abandoned’ by users and can be accessed through a subpoena instead of a search warrant,” explains Ryan Reilly in the Huffington Post. To what degree that is actually a factor here is hard to tell, because many of the outlets reporting on this aren’t technical enough to say. But it’s interesting timing.

The DoJ made its initial request, a subpoena and an order to preserve records, on January 27. However, Dreamhost, perhaps disingenuously, didn’t understand what the government was actually asking for. “Within three weeks of service of the subpoena, DreamHost produced its records responsive to these categories,” the company writes. “In its correspondence accompanying the production, DreamHost’s General Counsel made clear that he understood the subpoena was directed to records regarding the registrant, and not records regarding third party visitors to the website.”

Dreamhost also points out in its response that the request is more like a subpoena than a search warrant, because “it requires DreamHost itself to execute the warrant and provide the responsive records to the government.” The company also notes that the information the government is asking for is really more like evidence of a violation than a violation itself, despite how the warrant is worded.

It also isn’t clear exactly what the DoJ is trying to find out, or if it’s simply going on a fishing expedition, because that part of the warrant is sealed. But assuming it gets away with this request, it is making its requests for Microsoft data overseas look like child’s play. If companies can be forced to provide this much data about every single visitor to its customers’ websites, no matter how innocent, this could have a seriously chilling effect on, well, everything.

July 31, 2017  9:03 PM

E-Discovery Data Breach is a Lesson for All of Us

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
E-discovery, Security

Be careful with e-discovery: You might discover something you didn’t intend.

That’s what one attorney recently learned when collecting data for a legal case. “The 1.4 gigabytes of files that Wells Fargo’s lawyer sent included copious spreadsheets with customers’ names and Social Security numbers, paired with financial details like the size of their investment portfolios and the fees the bank charged them,” write Serge Kovaleski and Stacy Cowley in the New York Times – data from some 50,000 customers altogether.

Typically, such personally identifiable information is redacted, or removed, from e-discovery data sent to the opposing counsel, Kovaleski and Cowley explain.

Initially, the attorney blamed the software vendor (of course), which wasn’t named but appeared to include both software and service. But as it turns out, the attorney hadn’t realized how much data the e-discovery request had obtained, writes Christine Simmons in Law.com. Using the software, the attorney reviewed “what I thought was the complete search results” and marked some documents as privileged and confidential, and then coordinated with the vendor to withhold from production anything she tagged as privileged and confidential, Simmons writes.

“What I did not realize was that there were documents that I had not reviewed,” the attorney tells Simmons, adding that her view showed only a set limit of documents at one time. There also appeared to be some confusion about who actually performed the redacting of the documents, and whether any of the data was redacted, according to court documents (which are a thing of beauty, and you really should read them to get the full effect).

Moreover, the files were handed over to opposing counsel with no protective orders and no written confidentiality agreement in place. Consequently, it would be perfectly legal for counsel “to release most of the material or include it in their legal filings, which would then become part of the public record,” Kovaleski and Cowley write.

And it didn’t end there. Because Wells Fargo had released the personally identifiable information, it then became a data breach and was subject to all the laws governing data breaches. Sending the data without redactions or confidentiality agreements violates “various privacy protection laws, Financial Industry Regulatory Authority Inc. guidance and U.S. Securities and Exchange Commission regulations, according to opposing counsel in court documents,” she writes. The attorney who had sent the files to the other attorney asked that the data be returned, but at that point it became evidence in the data breach case.

Wells Fargo and its attorney have been using various legal maneuvers to get the opposing counsel to return the data, as well as destroy any copies it had made of it, Simmons writes. The attorney also noted, however, that the CD was encrypted, and that she’d written “Confidential” on the envelope. Thank goodness.

Regardless, Wells now needs to follow standard data breach protocols, such as notifying the customers that their data has been improperly released, Kovaleski and Cowley write. “And some of the accounts are listed as having a foreign owner, which would potentially trigger a separate set of overseas regulations, such as Europe’s stricter privacy statutes,” they add.

Such data breaches could happen more often as e-discovery becomes more common and more voluminous, Simmons warns.

July 31, 2017  6:38 PM

IBM Mainframe Encryption Apparently Okay

Sharon Fisher Sharon Fisher Profile: Sharon Fisher
Apple, Encryption, Google, government, IBM, privacy, Security

When Apple and Google released cellphones with encryption being the default, law enforcement had kittens, with dire warnings about terrorism and child pornography if there wasn’t a back door into it. And governments all over the world, including the U.S., have insisted that data shouldn’t be encrypted unless a back door was available, in case evil people were hiding evidence of their nefarious deeds.

But so far, law enforcement hasn’t complained about IBM’s new Mainframe Z, announced earlier this month. “IBM has launched a new mainframe system capable of running more than 12 billion encrypted transactions per day, in a bid to wade further into the financial cybersecurity market,” writes Ryan Browne for CNBC. “IBM claimed that its new mainframe can encrypt data at a rate 18 times faster than other platforms. The mainframe will be used initially as an encryption engine for IBM’s cloud computing technology and blockchain (distributed ledger technology) services.”

IBM didn’t say when the system would be available, though it said the technology was already in use at six of its own blockchain service centers, and at least one article indicated that the system was would be available in mid-September. The company already supports 87 percent of all credit card transactions, totaling nearly $8 trillion worth of payments each year, Browne writes.  The system is intended to “enable companies to comply with new data protection laws, such as the European Union’s General Data Protection Regulation (GDPR) and the U.S.’s Federal Financial Institutions Examination Council (FFIEC) guidance on the use of encryption in the financial services industry,” he adds. “The GDPR holds that businesses should encrypt personal data to prevent a compromise of confidentiality, while the FFIEC’s guidance states that management should ‘implement the type and level of encryption commensurate with the sensitivity of the information.’”

But by announcing the system, IBM is also drawing a line in the sand and siding with Apple, writes Brian Fung in the Washington Post. “IBM fully supports the need for governments to protect their citizens from evolving threats,” he reports the company said in a statement on the issue. “Weakening encryption technology, however, is not the answer. Encryption is simply too prevalent and necessary in modern society.”

Maybe law enforcement thinks that hackers and terrorists can’t afford mainframes like this one, which according to Fung is supposed to cost $500,000 a pop? But companies like Microsoft can, and the U.S. government has been fighting with Microsoft for several years to gain access to data that it stores overseas. What if Microsoft said fine, here’s the data – but it’s encrypted, so good luck?

Indeed, with some governments wanting to outlaw encryption altogether, is IBM going to be allowed to sell the equipment in those countries? Will people in those countries be allowed to use it? Is IBM releasing the system in hopes that it will be grandfathered in should countries implement anti-encryption laws?

Experts also point out that IBM statements about the encrypted data being more safe from hackers isn’t necessarily true. Commenters to the Washington Post article noted that only the data at rest would be encrypted, while data within an application would still be decrypted and vulnerable. In addition, hackers don’t have to be able to read data to wreak havoc, noted another. “I do not need to know what is in your data for a ‘WannaCry’ attack to work,” writes JoeFromBoston. “Even if YOU have encrypted your data, if I encrypt your encrypted data a second time, you are still in big trouble.”

So far, no comment from the FBI or other law enforcement organizations.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: