I’ve personally reviewed and/or audited somewhere close to fifty business continuity/disaster recovery (BCP/DR) plans over the past decade.  I’ve also written or edited several of those as well in the past five years since moving into professional services for financial institutions.   Furthermore I’ve participated in roughly a half-dozen tests while still working within the infrastructure during the first part of my career.  Suffice to say I have at least an informed opinion regarding the viability of any such BCP/DR strategies.

Fundamentally there are a few varieties of  BCP/DR plans:  Those that are current and viable, those that convince your examiner that it’s current and viable and those that may have been viable years ago but bear no resemblance to your current business profile.  And beyond those there’s the worst of BCP/DR realities, the non-existent one.  But really in the end what your current state of preparedness comes down to is this – either you’re ready for an event or you’re not.   And in the past forty-eight hours that’s been made abundantly clear in the form of how many of my clients affected by Hurricane Sandy have navigated through what’s now clearly one of the worst weather events in my lifetime.

Around noontime yesterday (October 29, 2012) as weather conditions worsened and major metropolitan areas were literally shutting down for business I started checking up on a few clients.  The first thing I did was visit the website of every client that my practice has assisted with their BCP/DR strategy – each of them had updated their website to announce that branches in the affected areas were closed.  Some had a pop-up window with the update, others had a message displayed in either bright red letters, bold font or both.  As a standard design consideration each of them also had phone numbers clearly displayed and when I called a sampling real people answered and were available to assist me.  I inquired of a few of them where they were physically located and they were all located remotely and not on site in affected areas (much to their credit they were reluctant to share too much information).   The second thing I did was visit the website for a few clients whose BCP/DR plans were tagged during an audit/assessment as either being deficient or missing.  The websites were not updated and in all but one case I only learned that they were closed for the day after calling into a branch (one had an 800 number that was redirected to a real person).

Now I know this wasn’t a very deep or meaningful test of anyone’s ability to continue operations in the event of a disaster.   But what it did prove is that those institutions who had plans that were current and whose management team knew to rely upon had already thought through the little things that make a difference.   Someone knew to update the website, management knew to reroute calls away from unmanned branch locations.  I can only assume that the appropriate parties desginated to do so also contacted their regulators to inform them of their closing and that a phone chain was initiated informing staff thus keeping them off the roads and safe.  And because an important part of the plan creation/update process is both training and testing stakeholders are able to navigate through the decision tree and take appropriate related steps without having to think through it – one of the biggest challenges confronting management during a crisis.  The very best part of having a viable and current plan is that all the thinking has been done in advance and has been reviewed and validated which greatly reduces the chances that something (or someone) will be missed.

Here’s a sanity test:  If you didn’t know exactly where to begin the decision-making process or who to engage you’re in need of a new plan.  And if you did know but can’t be absolutely certain that others would be able to do the same in your absence, you’re in need of a new plan.  One of the rebuttals I’ve heard all too often when identifying a deficient or missing BCP is that management knows what to do should some manner of disaster strike.  That may be true but what happens if key people are unavailable or can’t be reached?

Seriously, when something like Hurricane Sandy occurs it’s the best time to consider how you’re institution would fare when navigating such an event.  Block off an hour within the next week with your key people, pull out your BCP/DR documentation and try and step through how you’d handle things under similar circumstances.  In a very short time you’ll gain a sense of whether or not you’re prepared and if necessary afford you the opportunity to improve.

Trust me on this – you don’t want to be in the middle of a disaster scenario and find out that your plan doesn’t work.

I’ve been working on BCP’s since the late 90′s, cutting my teeth on a plan for the technology business unit I worked in at Citigroup and have continued working with clients on their plans in a variety of business verticals in the years since.  Whether the client is a multi-billion dollar enterprise or a single branch bank, there remain commonalities that defy the entities complexity. On one hand it’s difficult to compare the plan I worked on at Citigroup to one I recently reviewed at a banking client with a single physical location (everything was quite literally under one roof) but on the other hand, the key elements were exactly the same.

Ask questions about who is responsible for activating the plan, who has copies and where are they located and you’d get similar replies (mostly shoulder shrugs, lots of “um’s” and finger tapping).  Select a sampling of employees and ask them what they’d do in the event of a business disruption and you’ll get a wide range of answers that are typically intelligent and sensible but have nothing to do with what’s documented in the plan. Review the plan and conduct a logical walk through to determine if someone without intimate knowledge of the various sections could rely on it in order to help navigate through a disruption and you’re likely going to have a list of questions longer than your own arm. Of course one of my favorite measures of a plans effectiveness is to gauge its overall size and complexity relative to the entity it’s supporting. The single branch banking client had a binder filled with a plan that was nearly twice the size of the one I worked on at Citigroup.  Despite the fact that the Citigroup entity clearly dwarfed the small banking client, you couldn’t tell from the plan.  I’m not suggesting there’s a size rule to apply but typically the thicker the plan the less effective it becomes after a certain point.

However, the reason we’re talking business continuity to kick-off the new year isn’t so I can rant but rather to illuminate an important aspect of a BCP (and perhaps any of your regulatory activities as well).  Your business environment is dynamic, it’s ever-changing with new considerations, concerns and risks emerging almost daily.  Employees come and go, business needs change to keep pace with the economy and your physical and logical infrastructure changes to accommodate both.  It’s just about impossible that any plan you developed last year remains relevant this year.  Thus the reason why the FFIEC guidance hammers home the point about conducting frequent risk assessments and conducting periodic reviews of your key compliance activities.  You simply cannot rely upon any documented procedure that hasn’t been reviewed recently and assessed for accuracy and relevance.

In terms of a BCP, you need to conduct an annual business impact analysis to determine if each critical area of the institution is properly factored into the plan, if the area’s needs have changed since the last update and if the current set of procedures adequately support its needs. You need to update your contact lists, inventories and your escalation plans.  You need to reissue the updated plan and make sure that all stakeholders are aware of it’s changes and have access to the new version readily available. Perhaps the  most important recurring activity is to conduct a basic test of the BCP to ensure that it will work and that your staff knows how and when to rely on it.

As for the client for whom the report was issued, they’re in good shape.  The test revealed some common issues (e.g. critical stakeholders answers were often extemporaneous and did not come from the plan itself, many in the room did not think to bring a copy of the BCP) but by and large they did well. They did well because the plan had been updated earlier in 2010 and reflected on what they knew had to be done in the event of a disruption. Although they didn’t rely upon the actual document, they didn’t need to because they were the ones who contributed to its content and were able to react rather than read. Unfortunately they’re part of the minority because typically the plans I review are detached from reality to the point where they’re almost fictional and almost completely useless as written.

Like all compliance requirements there’s real value to be derived from addressing them properly and you shouldn’t need an examiner, an auditor or a blogger to point that out. It’s the first week of the first month of a new year; is there a better time to plan reviews of your key procedures and activities?