This morning Plan B failed. I have high speed internet access into my office, but I pay a small monthly fee to keep a pay-per-minute dial-up account just in case my high speed internet provider ever goes offline. So, this morning, when I lost my internet access, I mentally started preparing myself for 56K upload and download speeds. What I had not mentally prepared myself for, was my phone lines also being down. Thankfully, I still had my cell phone and was able to reach the outside world and let some individuals know about my situation.
But, it occurred to me that this was fairly typical of how disasters go. Not that losing internet access or phone service is necessarily a disaster, but disasters are rarely neat and tidy, they never happen when it is convenient and you can generally count on them not to follow the plan you laid out.
In no way am I implying that companies should abandon either their data protection or disaster recovery planning efforts. What I am suggesting is that after you have backed up all your data, laid out your recovery plans and then tested them, introduce some reality back into the situation.
For instance, a concern that one records management provider recently expressed to me is that companies should evaluate their disaster preparedness after they have just finished a disaster recovery exercise. Tapes are out of order, the recovery environment is not properly configured and people are exhausted. How quickly and how well could your company recover in this situation if a disaster happened then?
Another important aspect to include in your plan is to identify someone who knows the plan but is not afraid to think outside the box. I was once in a disaster recovery situation where an entire production database had failed and there was not enough unallocated disk in the free disk pool at that site to recover the database. The plan called for us to recover to another site, but one individual asked “Do we have a SAN?” and “Can we move some allocated but unused disk on another server over to this one?” In both cases the answer was yes, and we were able to recover the application in 2 or 3 hours instead of 8 to 12 hours.
Disaster recovery plans are just that, plans — no more, no less. But like all plans, they were created at a past point in time and may not reflect the current reality. That is where having someone around who can assess the entire situation and not just follow the script becomes imperative if one is to turn the disaster into a recovery.
Former Brocade CEO, Greg Reyes, went to court Monday to face the music for options backdating. His trial is the first of over 100 cases against companies accused of options backdating and the results could signal a collapsing house of cards for the technology industry. The criminal indictment against Reyes charges him with conspiracy to commit securities fraud, mail fraud, making false statements in filings with the SEC and falsifying books and records. He faces decades in prison and millions of dollars in fines if convicted.
Options backdating refers to the practice of reaching back to a date when the company’s stock price was at a low, and selecting that date for the option grant’s exercise price, or the price an employee will pay for the stock. The goal is to boost the potential windfall for the recipient. It’s said to have been common practice during the hay days of the dotcom era to lure talented employees. The criminal part of this action is when a company hides this practice from its shareholders, therefore not having to pay the correct compensation on options at the time they were awarded.
But the trial is not about whether Greg Reyes did this or not. His signature is all over hundreds of documents signing off on the practice. The question is whether he knew it was wrong, but did it anyway. Reyes’ defense says that he didn’t know he was doing anything wrong.
So, for the sake of argument, let’s say he was totally clueless and scribbling over financial statements and falsifying board meeting documents seemed perfectly normal to him.
Now think about this. The IRS calls to audit your taxes and they discover you claimed more than you should have. Can you say sorry officer, the rules are so complex, I must have filled out the form wrong? From my limited knowledge of tax laws, even if you get the wrong information from an IRS agent, you are still liable!
The most complex part for the government in this case is proving intent. And Reyes claims he didn’t profit personally from any options backdating. That may be so, but what about his staff and close associates? Should they be called to account as well?
I was talking with a friend in the business world about this topic and he says that CEOs hire legal advisors and executive management to advise them on such matters, as they themselves cannot be expected to know, in detail, every aspect and legal loophole of the law. So where are these guys then? Perhaps they should be in the dock? My friend also felt that with hindsight it’s clear that backdating was a nefarious practice, but at the time, it really didn’t appear to be.
In high-profile trials such as Reyes’, the defense and prosecution legal eagles will spend weeks jousting over semantics. In the end the jury will get worn down by the attorneys, who will create enough confusion and doubt in their minds as to Reyes’ actions, and he will be let off, perhaps with a slapped wrist and a fine. That’s my bet.
In the meantime, is it possible for the legal system to monitor the business world a little more closely and for the business world to try to act ethically, to prevent years of wasted legal wrangling and millions of dollars in fees? Or am I just hopelessly optimistic that things can change for the better? Anyone have any thoughts?
Welcome back to Las Vegas, where water (a precious commodity in the desert) will cost you $4 a bottle. This week, storage users moved from the Venetian to the Mandalay Bay hotel, drinking in its tropical theme and all the news they could handle on HP. A few miscellaneous items from the show:
HP CEO: “no confusion” about need to improve business
During his keynote speech Monday night to officially open the show, HP’s CEO Mark Hurd said he’s proud of the progress the company has made “over the last two and a half years” (i.e. since the “Carly Era” ended and the “Mark Era” began), but said “there isn’t any confusion about how much work we have in front of us.”
Hurd told the gathered audience of thousands in Mandalay Bay’s Event Center (where Pay-Per-View boxing matches are also held) that “we’ve invested millions in R&D, only to underdistribute our products in the market.” He estimated the market for HP’s products at $1.1 trillion, compared to HP’s revenues of $100 billion. The company has added 1000 salespeople and is working with its 140,000 channel partners “to get them deeper into the markets they cover,” Hurd said, particularly in midmarket accounts.
“The biggest complaint I get is how hard it is to do business with HP,” Hurd said. “We have to take the complexity that comes with being a big company and turn it into a capability that works for you.”
HP backup tapes climb Everest
Among the presentations at this year’s conference for media was the video-illustrated story of the brothers Clowes, who climbed Everest in 2006 carrying an HP DAT 160 tape drives and LTO-2 tape cartridges to backup photos and video they took on their ascent to the summit. HP “helped out” with the climbing trip and loaned the brothers a laptop as well as the drives, according to Thomas, the excursion’s photographer and the owner of a small residential building company in the UK. He said his two-man climbing team (consisting also of brother Benedict, a UK financial analyst) bailed out a professional video team from the BBC filming a documentary when their laptop choked on the dust flying around the air at base camp. HP also had the brothers do temperature testing on the tapes, which it is reported survived the wintry ordeal intact.
HP consolidates data centers
What would a conference be without more keynotes? On Tuesday morning, Randy Mott, EVP and CIO of HP, took attendees inside HP’s data center consolidation, in which it was estimated the company invested approximately 2% of its gross revenue (gross revenue, according to Mott’s presentation: $97.18 billion) in a three-year project to consolidate 85 data centers into three “zones”, each with two data center locations, in Houston, Austin and Atlanta. Construction is expected to finish on the data centers by this November, and the majority will have been completed in the next 60 days. The consolidated data centers now boast 21.6 football fields’ worth of raised floor space, 40,000 servers, 16,000 racks and 4 PB of storage. Mott said HP has halved the cost of its storage for double the capacity, though the consolidation won’t stave off storage growth for long–the company expects to be at 10 PB by the end of its three-year project.
That’s all the news that’s fit to print from where I sit–if you’re attending the HP Executive Forum, StorageWorks Americas or Technology Forum conferences this week, feel free to add your thoughts below.
One of the common questions I am getting from IT people that I meet with is how can they protect remote offices, typically those with no local, at least officially anyway, IT staff. It is important to accept upfront that you may need a couple of solutions to address this challenge. This is especially true if you have remote offices that vary in size. There may be local databases and, more often than not, local file servers.
The first option is to eliminate the problem by eliminating the local need. Products like Citrix or Windows Terminal Server can eliminate the need for local applications and a wide area file system (WAFS) can eliminate the need for local file servers. The Citrix/Windows Terminal server solutions are best explained by those two companies so we’ll spend our time on WAFS. WAFS essentially places a cache at the remote site. At a high level, this cache is a server appliance with a small local disk that can replicate changes as they happen to a central server at a primary data center. The most frequently accessed data is stored on the remote file server, which serves up data to those users in a local fashion and at local performance. Typically, proprietary but enhanced network protocols are utilized to get higher performance on data transfers for data that is not in the remote cache. Some of the WAFS companies are also providing the ability to utilize data deduplication on the network, in a similar fashion to how some disk-to-disk backup products use data deduplication to optimize storage. NAS that can do data deduplication would be an ideal central NAS for this environment since typically there is a high level of file redundancy between remote offices and the primary data center.
From a data protection standpoint, the centralized repository for all the remote cache’s data is now also a server at your primary data center and can fall under the umbrella of your normal protection scheme. Other advantages to WAFS is that it can eliminate the need for buying additional servers and storage for remote offices, delay the need for bandwidth upgrades and can even enable better collaboration between offices.
Another option is to use replication. This is ideal for sites that in addition to a remote file server also have some remote database applications or email, especially if there are just a few of this type of sever present. While most data replication products are considered for disaster recovery solutions, they also make for an ideal remote or branch office backup solution. With these products all data is replicated as it changes to a centralized disk at the primary site. This disk can then be mounted to a backup server and backed up locally at the primary data center. Cost can be a concern if the local server count is more than just a few servers and you are not leveraging the other value points (disaster recovery and failover) of replication. Also, unless your data replication solution can produce snapshots to freeze moments in time or you can leverage snapshot technology by replicating to storage that supports it at the primary data center, be aware that if you do experience corruption at the remote site, you will then very quickly replicate that corruption to the primary site.
If you don’t need the near instant protection of WAFS or replication you can also leverage some D2D solutions to perform a backup locally and utilize the D2D solution’s replication capability to send that data to a similar device at the primary data center. This typically makes sense when you have a fairly large data set or number of servers and maybe a small remote IT staff. It will also almost certainly require a data deduplication device, as straight disk-to-disk backup will not work well for remote backups. You need to be able to deploy a D2D solution that can leverage data deduplication. Standard backup to disk essentially consists of several large backup files. Those files are typically created too fast and are too large to be copied across the common WAN segment in time to meet the nightly backup window. Data deduplication appliances on the other hand only have to send the changed block between the backup jobs, greatly reducing the WAN requirement. Also, you now have the data locally at the branch office instead of only at the primary data center, allowing for faster recovery at the branch if needed.
In the past four or five years, we have expanded from hardly any options for remote office data protection to many. Which of these solutions you deploy is a function of budget and business requirement, and in some cases it may make sense to blend the solutions. Assessing the needs of the remote offices while focusing on the business realities at the primary data center, will help you make those choices.
Coming to you from sunny Las Vegas, in the “Blogger’s Lounge” inside Symantec’s Vision City (their name for the show floor). Seemed apropos.
In addition to the Blogger’s Lounge, which consists of picnic tables set up on a large field of Astroturf in the middle of the floor that has been dubbed “Central Park”, Vision City has a Financial District, consisting of business partners’ exhibits, and Tech Row (no relation to Skid Row, presumably), where product demos are being held. Central Park also has Wii video games, robotics competitions, and obstacle courses for remote-control cars.
In each of this year’s seven keynotes, Symantec has shown similarly slick production values, beginning with an address given by data center group president Kris Hagerman on Tuesday.
Quite often when they roll video at a conference general session, attendees can be seen checking their watches, email, or taking advantage of dimmed lights to catch some shuteye. But the video during Hagerman’s address, called “The Complexity Master”, was a hit with the crowd, who responded with genuine belly laughs to the Office-Space stylings of the short film, which documented a day in the life of a backup administrator trying to sell his company on standardizing on the NetBackup platform (the vid was, of course, not completely without marketing).
The biggest laughs came when the Complexity Master, a Southern California comedian hired for the role, and his supporting cast (presumably of other comedic actors), were given the dialogue most backup admins either imagine or wish would happen during meetings with business units–witness the one higher-up who chews his donut thoughtfully before saying, every time he’s called upon, “that sounds…complicated.”
We can only hope it gets leaked to YouTube.
Video hijinks weren’t the end of the three-ring circus at Hagerman’s keynote, either. Senior director of product marketing Matt Fairbanks bounded onto the stage sporting one of the “Storage United” soccer jerseys (after Manchester United, DC United, et al), and was later joined by a girls’ under-10 soccer team from Silicon Valley, who handed out soccer balls and jerseys to the crowd. The talk of the conference following the session was the tiny pigtailed soccer player who flawlessly rattled off a spiel about replacing her TSM and Legato environments and achieving huge ROI when asked what she thought of the new NetBackup. Somebody get that girl an agent.
More nuggets from the show for your reading pleasure are below the fold.
Word of tapes “falling off the back of trucks” is almost a once-a-month event these days, but the way companies handle the disclosure of these albeit embarrassing incidents is shameful.
A coworker at TechTarget told me this morning that he had just recevied a letter from IBM informing him that the company had lost tapes containing sensitive current and former employee data, including and potentially his social security number. This is old news [May 15], but a few things stuck me as interesting about it.
1) He has not worked for IBM in over 20 years, yet the company is still storing information on him. Ever heard of ILM over there guys? I think Tivoli has something…
2) IBM announced this publicly on May 15 but my friend did not receive the letter until June 7.
3) IBM lost the tapes on Feb. 23, 2007.
“Time was needed to investigate the incident, determine the nature of the information on the lost tapes, and conclude that recovery of the tapes was unlikely,” IBM said in an FAQ sheet sent to its employees. “In order not to impede any continuing investigative efforts, we are not disclosing the numbers of individuals affected,” it added.
Come on! We weren’t born yesterday. IBM’s excuse for the delay in informing its employees, as well as the number that were affected seems disingenuous, probably to avoid further embarrassment. It’s a poor response not to mention bitterly ironic given IBM’s focus on security.
My friend was given a year’s worth of free credit reporting to help him track whether anyone is using his stolen information. If IBM thinks this is enough to rescue its relationship with its employees it might want to take a look at this survey of people who were notified that their personal information had been lost. It found that 20% of the people had already stopped doing business with that company and another 40% were considering it.
We all know that deleting a file doesn’t actually “delete” anything. Deletion only marks the file’s clusters as free for re-use — data actually remains tucked away within the sectors of each cluster until they are overwritten by new data. To really destroy data, it must be overwritten multiple times. This ensures that the magnetic traces of previous recordings cannot be read with advanced laboratory equipment (even when new data is on the media).
But how many times do you really have to overwrite that deleted data before it’s actually considered secure? Once? Twice? Ten times? Experts say that multiple overwrites are worthwhile — even required — noting that anywhere from 7 to 11 writing passes may be needed to fully overwrite the old data.
And there’s no shortage of tools that promise to kill your old data. Professional products like FDRERASE/OPEN from Innovation Data Processing can securely erase the magnetic disk using three to eight passes. Even end-user products like File Shredder from HandyBits.com promise to overwrite file data with random information up to 15 times, claiming that “it is practically impossible to recover the original data”.
Now there are circumstances when it pays to be extra thorough, but personally I think it’s overkill — a practice based on old MFM/RLL drive technologies. US DoD specification 5220.22 calls for three overwrites, while NIST standard SP 800-88 was revised in 2006 to call for only one overwriting pass on modern (post 2001) hard disks.
But I want to hear what you think. What tools are you using? How do you ensure that your old files are securely deleted? Does it even matter to you?
In the mean time, listen to this FAQ on Storage Security where Kevin Beaver offers practical answers to the most common storage security questions he hears from storage pros today.
Arun Taneja and I have focused on the topic of data protection management (DPM) in recent posts. He sees what I also see – DPM software is undergoing a significant transition in purpose. While DPM software is not new and companies like APTARE, AGITE Software, Bocada, Tek-Tools, ServerGraph and others have offered it for years, only now are vendors and customers figuring out how to use in a larger context within organizations.
Companies tend to only think of and use DPM software in a singular context – day-to-day operations. For the most part, this software does a good job of monitoring and reporting on the successes and failures of backup jobs, the identification of failed tape drives and the utilization of media in tape libraries. However, this did not raise DPM software’s value proposition much beyond the purview of the day-to-day operations staff.
Now, some DPM vendors are reworking their messaging to make their products more appealing to a larger corporate audience – with capacity planners and storage architects their primary focus. Part of the motivation for this change is that more companies want to bring disk into their backup scheme. But, these organizations lack information on their current backup environment to confidently make these types of changes to their infrastructure.
For many companies, it’s a roll of the dice as to how well a new disk library will work in their environment. I have spoken to more than one user who purchased a disk library with high amounts of disk capacity only to find out the controllers on the disk library could not keep pace with the amount of data backup software fed to it. This required them to purchase additional disk libraries which created new seta of management problems.
DPM software can help to address these types of sizing issues by quantifying how well current backup resources are being used and trend their use over time. This allows companies to select and implement appropriately sized disk libraries based on facts, not assumptions. It may also give them the facts they need to justify waiting on a disk purchase, since they may identify better ways to utilize their existing tape assets.
Performing trending and capacity planning is very different than sending out an alarm that a tape drive or a backup job has failed. Companies need to be sure that vendors are actually delivering a product makeover regarding reporting and analysis capabilities and not simply covering up their product’s deficiencies with its latest rendition of marketing literature.
Data Robotics, a new storage startup just coming out of stealth, has announced both itself and its first product today. Data Robotics was formerly working under the name Trusted Data, and is a venture led by BlueArc founder Geoff Barrall. The product is a consumer storage unit called Drobo that Data Robotics says it intends to scale up into the enterprise market.
Why should you care? Because if Data Robotics has its way, Drobo could change the concept of RAID storage.
Here’s how it works: the Drobo box, a black cube that will fit on a desk, contains four disk bays. Any SATA disk drives from any manufacturer of any size can be added and the box will automatically stripe data across them; the box uses no management software, and instead has a system of lights which show red, yellow or green. If it’s red, replace the disk. If it’s yellow, the disk is filling up. Green and all is well.
When disks fill up, they can be swapped out for a larger size and data is restriped automatically, using RAID levels that change according to the disk capacity left over. For example, in a system with a 120 GB, 250 GB, 500 GB and 750 GB drives installed, 120 GB of each disk would use RAID 5 striping in a 3 + 1 configuration. As the 120 GB drive fills up, the system would put the remaining capacity into a 2+1 configuration, and then finally into a mirrored pair (in case you’re doing the math at home, the final 250 GB of the 750 GB drive would remain empty in that scenario).
Data Robotics president Dan Stevenson said the system was designed for the non-technical customer–it could perhaps even be termed the extremely non-technical customer. Hence the lights. “If you can figure out a traffic light, you can figure out how to manage this storage box,” Stevenson said.
Of course, you may have to understand a bit more than that to know that you shouldn’t take the 750 GB drive out to replace it without enough capacity left in the remaining disk slots to absorb the data while it’s missing. And Stevenson said so far Drobo isn’t pursuing any distribution deals with Best Buy, instead presenting this as an alternative to homegrown RAID arrays for Bit-Torrent addicted power users or “prosumers”, professional digital photographers, and small businesses where, to quote Stevenson, “the IT guy is Frank’s son.”
One such company is Michael Witlin Associates (MWA), a five-man company that produces corporate events in Silicon Valley. Its owner, namesake and executive producer received a free eval copy of Drobo and says it’s been humming along nicely in the few months he’s tested it, in contrast to a Maxtor-drive-based RAID 5 array he struggled to manage. Every so often partitions on that array would “drop off” in Witlin’s user interface, “And I could never figure out what the problem was,” he said. Drobo plugs in via USB 2.0–something Witlin said he prefers–and “there’s nothing I have to do after that.”
Meanwhile, Data Robotics is aiming to take its approach to RAID to the big time. Stevenson envisions a Drobo-ruled utopia in which lower-paid “Tier 3” IT admins also manage Tier 3 nearline storage, automatically striped using Drobo’s RAID method, who merely need the expertise to observe and act on a red, yellow or green light.
“Their algorithms could have a huge impact on enterprise storage, as well,” wrote Brad O’Neill, senior analyst with the Taneja Group in an email. “If you can begin to create heterogeneous arrays right down to the drive level, with no interruption of availability or performance, you’ve done something extremely disruptive to the market–capacity upgrades would become the equivalent of simply plugging in drives, waiting for a green light, then adding more. I could imagine large service providers using drive robots running swaps and upgrades not unlike tape robots do today. ”
In the interest of full disclosure, O’Neill’s not just an enthusiastic supporter of Drobo’s vision, but also of their bottom line: “I have a Drobo plugged into my laptop right now via USB providing 700 gigabytes of…storage…with four different drives of varying capacities and vendors,” O’Neill confessed. “I [also] bought four of them and gave them to my friends.”
(His friends are probably used to those special Christmases with Brad…)
You have seen my writings on (and may even have heard me speak about) Cross Correlation (CC) analytics engine as a necessary part of a Data Protection Management (DPM) product. DPM products make your backup and restore environment work more efficiently. Recently, I have seen the application of CC techniques to solve problems on the primary storage side. And much to my pleasure, I have also seen the technique applied to manage application performance.
Several players are delivering products in the DPM market including Aptare, Bocada, Illuminator, Servergraph, Tek-Tools and WysDM, and most recently, Symantec, with their NetBackup Reporter product. These products, as a category, are delivering real value, based on my conversations with many of you. EMC, who resells WysDM as Backup Advisor, is apparently shipping in large quantities. All big data protection vendors have gotten religion on this recently, and they are all scrambling to add DPM functionality via in-house R&D or through a partnership.
To be sure, not all products are created equal in terms of the strength of the CC engine (or even the existence of one), which to me is the essence of the product. Without a sound CC engine, the best a product can do is rudimentary analysis and basically report on changes.
I have seen two new and interesting uses of CC recently. First, WysDM announced WysDM for File Servers. Essentially, that means the same CC engine is being used to look at NetApp filers (primary storage) to determine if the filer is behaving as it should. Much as before, the product gathers data from the application and through all hardware and software layers that reside between it and the filer, and applies analytics to determine if the system is behaving within acceptable boundaries. Are response times to file requests deteriorating? Is capacity being utilized efficiently? Is a file system ready to run out of storage? What needs to be done to solve the problem? Will an additional GE connection make a difference? You get the point.
I know you are probably saying to yourself, “I get some of that information from filer’s integral management tool?” Of course, you do. But, just like on the data protection side, the amount and type of information about the environment that was being delivered before this tool was available was rudimentary and static. Unless one escapes outside of the filer and looks at the entire picture from end-to-end it is hard to determine the root cause of a problem that exists or is in the making. That can only be done with a sophisticated CC tool. And only a sophisticated tool will give you predictive information with a high degree of confidence.
Another company that has applied CC to the primary storage is Illuminator Software, whose DPM product now includes functionality about snapshots and replication. But, the product is still true to its data protection roots. In this case, the product provides information on the readiness of volumes from a data recoverability point of view. Whether the volume is protected using snapshots or replication or secondary disk or tape, its recoverability is established and reported on. The product also offers advice on the actions necessary to improve recoverability.
The third company, Akorri Networks, has applied a CC engine for an entirely different purpose: to provide insight into application performance. Of course, application recoverability is improved when application availability is improved so there is an underlying connection here. But, the overt focus is to provide insight into how storage resources are being used to deliver a certain level of performance at the application level. In other words, given a particular SLA for an application, does one have adequate or inadequate storage resources applied? Would extra resources (higher throughput storage, more storage, another pipe to storage, etc.) help to bring application performance back into SLA boundaries? Or would it be a waste? What would help the most? With this kind of information the right type and quantity of resources can be applied thus saving time and resources.
The progress in these areas has been truly phenomenal in the last three years, and yet, we are still in infancy stages of utilizing these tools. Most of these technologies have become available from smaller companies, whose reach is limited. Given that your environment is only getting more complex it behooves you to check these out! Send me an email if you need any help.