I’ve personally reviewed and/or audited somewhere close to fifty business continuity/disaster recovery (BCP/DR) plans over the past decade.  I’ve also written or edited several of those as well in the past five years since moving into professional services for financial institutions.   Furthermore I’ve participated in roughly a half-dozen tests while still working within the infrastructure during the first part of my career.  Suffice to say I have at least an informed opinion regarding the viability of any such BCP/DR strategies.

Fundamentally there are a few varieties of  BCP/DR plans:  Those that are current and viable, those that convince your examiner that it’s current and viable and those that may have been viable years ago but bear no resemblance to your current business profile.  And beyond those there’s the worst of BCP/DR realities, the non-existent one.  But really in the end what your current state of preparedness comes down to is this – either you’re ready for an event or you’re not.   And in the past forty-eight hours that’s been made abundantly clear in the form of how many of my clients affected by Hurricane Sandy have navigated through what’s now clearly one of the worst weather events in my lifetime.

Around noontime yesterday (October 29, 2012) as weather conditions worsened and major metropolitan areas were literally shutting down for business I started checking up on a few clients.  The first thing I did was visit the website of every client that my practice has assisted with their BCP/DR strategy – each of them had updated their website to announce that branches in the affected areas were closed.  Some had a pop-up window with the update, others had a message displayed in either bright red letters, bold font or both.  As a standard design consideration each of them also had phone numbers clearly displayed and when I called a sampling real people answered and were available to assist me.  I inquired of a few of them where they were physically located and they were all located remotely and not on site in affected areas (much to their credit they were reluctant to share too much information).   The second thing I did was visit the website for a few clients whose BCP/DR plans were tagged during an audit/assessment as either being deficient or missing.  The websites were not updated and in all but one case I only learned that they were closed for the day after calling into a branch (one had an 800 number that was redirected to a real person).

Now I know this wasn’t a very deep or meaningful test of anyone’s ability to continue operations in the event of a disaster.   But what it did prove is that those institutions who had plans that were current and whose management team knew to rely upon had already thought through the little things that make a difference.   Someone knew to update the website, management knew to reroute calls away from unmanned branch locations.  I can only assume that the appropriate parties desginated to do so also contacted their regulators to inform them of their closing and that a phone chain was initiated informing staff thus keeping them off the roads and safe.  And because an important part of the plan creation/update process is both training and testing stakeholders are able to navigate through the decision tree and take appropriate related steps without having to think through it – one of the biggest challenges confronting management during a crisis.  The very best part of having a viable and current plan is that all the thinking has been done in advance and has been reviewed and validated which greatly reduces the chances that something (or someone) will be missed.

Here’s a sanity test:  If you didn’t know exactly where to begin the decision-making process or who to engage you’re in need of a new plan.  And if you did know but can’t be absolutely certain that others would be able to do the same in your absence, you’re in need of a new plan.  One of the rebuttals I’ve heard all too often when identifying a deficient or missing BCP is that management knows what to do should some manner of disaster strike.  That may be true but what happens if key people are unavailable or can’t be reached?

Seriously, when something like Hurricane Sandy occurs it’s the best time to consider how you’re institution would fare when navigating such an event.  Block off an hour within the next week with your key people, pull out your BCP/DR documentation and try and step through how you’d handle things under similar circumstances.  In a very short time you’ll gain a sense of whether or not you’re prepared and if necessary afford you the opportunity to improve.

Trust me on this – you don’t want to be in the middle of a disaster scenario and find out that your plan doesn’t work.

So with the ratifying of the most recent bit of the legislation by the FDIC last month I’m all the more curious to see how the industry reacts.  For those of you who don’t know what it is I’m referring to it’s Section 165(d) – the new law that requires banks with non-banking assets in excess of $50B to draft a plan that would in effect put its sponsoring institution out of business in a neat and orderly fashion.  Or rather, as I’ve come to think of it, it’s a Business Continuity Plan of a whole ‘nother color (or perhaps it’s simply a Business Discontinuity Plan – BDP).

Think about what this law requires.  Banks that are in-scope for this now have to draft a plan that would allow regulators to step in and break off the various spokes of the wheel and either sell things off or shut them down in a way that is as minimally disruptive to the financial system as is possible.  What I don’t understand about this is how would that even be possible?

You’d have to make an awful lot of assumptions to even draft such a plan.  In 2007 when the banks started spiraling out of control you would have thought the very first thing to do was to divest themselves of the root cause of the problem, their consumer loan portfolio and primarily their mortgage business.  But who was buying that pile of rotting paper for anything other than pretend money (how’d Countrywide make out playing that market)?  So documenting in a plan that you’d sell off your various units assumes that there’s a market looking to buy them and you can’t really count on that, can you?  The truth of the matter is all you can really lay down on paper is a very high-level approach that specifies how each segment of the business needs to be evaluated to determine what if any value it possesses and than shop it to the market and see if there’s a buyer.   But how is that any different then how a business is dissolved in bankruptcy?  And we already have all sorts of laws on the books to guide that process.

How do you even test such a plan?  In order for this legislation to deliver on its promise the regulators would need to know that the plan would work somewhere close to as it’s designed to if ever it was needed.  How can you possibly step through it and know for sure?  Wouldn’t the various markets need to participate as well and how reliable would that be?  Wouldn’t everyone need to pretend that it was real?  Say Citi, would you be willing to buy BoA’s commercial loan portfolio and if so, how much would you be willing to pay for it?  If I’m Citi I’m thinking make the pretend offer high because it’s not binding and if a once in a lifetime disaster occurs again we can totally low-ball on the real offer if it ever comes to that.  So how reliable is that test?

But here’s the thing that keeps tap-tap-tapping away at the back of my mind – those who have to comply are the same institutions who can’t successfully design, implement and support a viable business continuity plan and that’s something they’ve had years to perfect and still haven’t even come close to doing.  And they’ve actually had disruptions where they needed to rely on these plans and still haven’t quite gotten them done right.  If they can’t successfully design a viable BCP when that’s something they’ve often enough desperately needed how are they supposed to design a viable plan to dissolve their business relying entirely on speculation and imagination?  Seriously, was the lack of such a plan one of the the primary reasons that the banking behemoths weren’t allowed to fail or was it simply our leadership being fearful that if a Citi or Chase went belly-up the economy might never recover?

Lewis Black loves to poke fun at the whole banking mess and about how banks were instructed by Capital Hill shortly after things got ugly to make sure that before making a loan they needed to be certain the person has the financial wherewithal to repay it.  He suggested that the next piece of direction should have been to remind the banking leaders to breath occasionally because it was about as simplistic and obvious.  Well perhaps the lawmakers should have kept things simple here as well.  Rather than require a BDP, let the FDIC oversee things as they have for many, many years and shepherd a failing institution through the various stages of liquidation finding suitable buyers for the pieces that are worth selling off or that need to be absorbed.  Sure the bigger institutions would present some issues and complexities that would require a certain degree of creative thinking but isn’t that better than trying to rely on a plan that was conceived of pure speculation and whimsy?

The real problem wasn’t that any of the monster banks couldn’t fail, it was that they weren’t allowed to.  Even if any of them had something drafted that specified how they should be dismantled the government wouldn’t have let it happen.  Much like a financial institution tends to look at their BCP well after the disruption occurred (happens all the time) I suspect a BDP would serve in much the same capacity.  And if I was an examiner and was going to hold the feet of one of my institutions to the fire for something I’d rather they focus on having an actual, honest-to-goodness BCP that would help them navigate the next hurricane, earthquake, blizzard, blackout, etcetera rather than preparing for something that may never happen again.

Most of what I do falls under GLBA-defined requirements and what that really means is that any institution I work with has identical goals.  The designs of the related programs and procedures certainly can look different because everything that falls under the guidance of FFIEC is supposed to be adjusted based on the size and complexity of your institution.  But they all need to conduct risk assessments, they all need to have current, up-to-date and recently tested business continuity plans, they all need to have viable vendor management programs and so on and so on… And I have many years of experience building out and/or supporting these very activities and know quite clearly what works, what doesn’t, what presents well to the examiners and what falls well short of expectations.

Sometimes though I’m caught off guard when a client rejects my advice because they’re confidant that what they’re doing or intending to do is consistent with their examiners expectations.  I’m a fan of confidence, I sort of dabble a bit in the discipline myself and appreciate how it can be very effective when trying to sell something to the audience.  But with regards to compliance there’s really not a whole lot of wiggle room.  In fact sometimes it can be interpreted as binary – either you’re compliant or you’re not.  So when I encounter a client who hasn’t updated or tested their BCP in years (if ever) and tell them that’s going to be a problem with their regulator I recoil when their reply is the dreaded “well the examiners haven’t had anything to say about it”.   “Yet”, I typically reply, “they haven’t had anything to say about it yet.”  All because the examiners haven’t dinged you for something doesn’t mean that you’re in good shape, it often means that they simply had bigger issues to focus on and haven’t quite gotten to it.  I have a list longer than my arm regarding vendor management and the common mistakes most institutions make and how those mistakes are going to lead to trouble with the examiners.  But when I bring this to the attention of the appropriate stakeholders I’m often treated as if though I’m simply trying to sell them my services and not giving them solid advice.  It can be very frustrating particularly because our practice was built on giving out solid and oft times free advice.  We’re willing to make the trade-off between generating revenue and doing right by our clients.  However you can lead a horse to water, but, well y’know.

I have an idea, maybe a great idea that might help solve the problem.  What if the examiners created a list of findings and issues culled directly from their reports and compiled them in a repository?  They could make the verbiage appropriately anonymous to avoid any privacy issues but share with the public what it is they’re finding out in the field.  The findings can be sortable based on the related requirement and or size/complexity of the institution so that any institution that shares the regulator can figure out where they may have issues.  Remember, the purpose of compliance and the regulators charged with ensuring that it’s being addressed satisfactorily is to protect us, the customer.  So it’s a very good thing to use all available resources to make sure that everything that can be done to make that happen is being done.  If your bank or credit union is able to access such a repository and use that information to identify where they’re weak or deficient doesn’t that help protect all of our sensitive information?  And it also removes the thin veil of ignorance associated with the logic that all because your examiner hasn’t documented any issues with a particular activity that must mean that you’re doing things right.  And when a client tells me that they don’t need to conduct a periodic review of all high risk vendors I can show them where that’s recently been an issue in a report.  Or when they tell me that testing their DR plan satisfies the need to test the BCP it’s part of I can show them how that logic failed to hold up under recent scrutiny.

Really in the end this isn’t so different than what happens now.  All of us practitioners gather information from the field regarding what the examiners are focusing on and use that information to update our own guidance and advice.  For example, when we recently heard that examiners are looking for greater scrutiny to be placed on SLA tracking as part of the vendor management program we made sure to include that advice in any of our audit and assessment reports.  But why should the industry need to rely on an informal approach?  Why not make it formal, take ownership and put the right information in the right hands to affect the desired results?

Is this idea a bit self-serving?  Sure, at least a little.  But really in the end if it helps get the right things done and in place who really cares?  If I can prove to a client that a Red Flags program that’s recorded only a handful of incidents during the previous twelve months is likely ineffective and be able to get them to do something about it everybody wins.  And can something like the proposed repository actually happen?  Maybe.  I’m sure the lawyers would weigh in with all kinds of issues.   But it’s difficult to argue against the merits of such an offering and in this age of greater accountability this would potentially be well received.

Anyone have any better ideas?

I just heard that Mayor Bloomberg is evacuating low-lying areas in New York City and that mass transit will more or less be cut off tomorrow (Saturday) sometime around mid-day.  New York’s Governor Cuomo also discussed the possibility of closing the bridges as well if weather conditions become so severe that using them might be dangerous.  Upon hearing this my first thought was “how the heck are key stakeholders going to get to their disaster sites if they’re called in?”  The obvious answer is that many companies will likely require that the important people go to their DR sites tonight so that they’re already there “just in case”.  How wonderful for these people to have to leave their families in the midst of a potentially epic natural disaster.  I can’t help but wonder how many are willing to comply and how many are going to insist that they can’t make it.  Did any BCP/DR test ever take into consideration the possibility that key stakeholders would simply refuse to show up?

And with the enormous range of Hurricane Irene is it at all possible that certain recovery sites might not be able to provide the proper services, resources and support to meet such a potentially large demand?  I know that they all claim that they’ve factored that in to their models and are able to provide sufficient capacity.  But until they know for sure how do they really know for sure?  Who among us has yet to witness any BCP/DR plan that didn’t start experiencing hiccups and delays during testing?

One element of a BCP that I’m also now wondering about is the day-after scenario.  I’ve reviewed dozens of plans during my career and upon reflection cannot recall any that placed significant attention on what happens after the official disruption is at an end.  I’m looking at pictures of severe flooding from Irene from those places already affected and have to wonder how many business are going to be able to open on Monday despite the fact that the roads are clear and the skies sunny and blue.  In thinking about some of the more common disruptions over the years (e.g. heavy snow, ice, etc.) it was somewhat obvious that once the roads were passable it was safe to head back to the office.  But that may not be the case this time around.  How many plans are designed to accommodate that?  Is someone from facilities charged with the responsibility of conducting a site inspection on Sunday night to see if their buildings are ready to open the next day?

Admittedly I’m picking on the entire concept of a business continuity plan but you can’t blame me, Hurricane Irene is only one reason.  Middle of last week I was in the Northeast and experienced my very first earthquake event.  Now I realize that anyone from California or Japan would chuckle at that statement because what I personally experienced was little more than an overloaded truck driving past me on a pothole-ridden street to those who deal with the phenomenon regularly.  But still, for me it was a big deal.  In the aftermath I asked around to see what happened in other places where the tremors were felt to see if anyone was formally evacuated from their building – no one was.  I expected in the days following to read about how companies had dedicated time and resources to inspect their structures to ensure that everything was as it should be and that there were no signs of damage from the unexpected movements – again, almost nothing to be found.  Well for all those BCP’s that I’ve reviewed where the likely threats were documented and addressed as part of their plan, how many think that maybe they should update their documentation to cover earthquakes?  They can no longer justify leaving it out because it’s not a likely threat, it just happened.  And now that they know it happened once they need to accept that it not only could happen again but likely will. But I’m willing to bet that a year from now I won’t find a single plan that has been modified to include what should happen in the event of an earthquake.

I’m just thinking that regulators and auditors need to stop rewarding those they’re responsible for monitoring for simply having a plan in place.  At some point they’ll need to shift their focus from simply checking off that a plan exists and start digging into it a bit more.  The same degree of scrutiny that emerged in 2009 because of the “Great Swine Flu” threat and making sure that BCP’s had a thorough pandemic response component now needs to become standard fare for the overall plan.  Companies need to conduct more than tacit testing exercises and really start thinking things through.  Between companies having antiquated and irrelevant plans, to those who have partially baked plans and worse yet, those who don’t even have one in place it’s time to do something about it.

The worst time to discover that you need a viable plan and don’t have one is, well, when you actually need it.  If enduring both an epic hurricane and your first earthquake don’t inspire you to action nothing will.

Japan’s tragedy will serve as a fertile source of both proving and disproving the myriad business continuity and disaster recovery techniques being used around the world today.  The most prepared and best trained companies will have very likely fared about as well as could be expected while those who weren’t, those who either had partially baked plans or no plans at all will be lucky to survive in any measurable way.  And it’s hard to imagine that most companies didn’t have plans to deal with earthquakes and tsunami’s because they’re credible and consistent threats in the region.  But after a quarter century in corporate life and little more than half those years focusing on audit and compliance I’m no longer surprised by anything I encounter.

However there was one story to emerge from Japan this week that I found to be quite shocking.  It was about how a banks vault came open during the series of events and someone stole forty million yen (about $500k USD).  It happened in the prefecture of Myagi in a town known as Kesennuma and police said that between the wave’s power and the ensuing power outages, the vault came open.  What with all the flooding and chaos it took more than a week for someone to get back into the building and discover what had happened.

For many the story seemed plausible if not mildly amusing because who wouldn’t love to wander into a bank and be able to scoop up all the cash floating around.  And because in this particular situation no one died or was hurt as a result it’s benign enough to be more entertaining than tragic.  It sort of reminded me of a scene in the movie “Ground Hog Day” where Bill Murray’s character figured out the perfect timing to be able to steal a bag of cash out of the back of an armored truck.

But I sort of have a problem with this story because I don’t think it happened the way it’s being portrayed.  My very first thought upon reading the details was that either someone left the vault door open as they were fleeing the bank or someone who knows a thing or two about how to open a vault went back in after the fact and exploited the situation to their advantage.  The odds that a vault door simply flew open due to what was really a massive flood at that point just doesn’t hold up under scrutiny.

Have you ever actually seen what a door on a bank vault looks like?  I have and I’ve probably seen about three dozen or more since I started working in the banking sector and I couldn’t think of how any one of them, if closed properly would ever just come open due to rushing water for a relatively short period of time.  First of all they’re all seated within a metal frame and so for the rods or pistons that create the seal to come undone the metal itself would need to have been bent or twisted.  Second, they weigh a ton (not as much of an exaggeration as you might think).  Even the weakest vaults I’ve encountered have doors that have some serious density to them and would not likely bend under most natural forces.  I would sooner believe that the walls that the door and its frame were attached to failed then believe that the door simply “flew open”.

If I had to put on my most skeptical mindset to use I would venture a guess that the person responsible for making sure the vault was properly closed before safely exiting the building rushed through the procedure, didn’t properly lock the vault and in their heightened state of panic just didn’t think about it.  While that’s the most likely scenario the second most likely version is that someone who knows how to open the vault door and who knew after a day or so that no one would ever be concerned with theft while there were still lives to save made their way into the crippled building with its security systems down and manually opened the door and had at it.  But under either scenario it’s almost entirely likely that the person(s) who stole the money had an idea about what to do and took advantage of the situation.  I mean, they obviously entered the bank after the disasters struck and they weren’t likely looking for survivors if they were of a mindset to grab what had to be a sizable physical haul.

And the thing is that there’s no viable lesson to be learned from a story such as this.  I’m certain the bank had a procedure in place that specified how all cash drawers were to be placed in the vault and that the vault itself should be locked upon exiting during a disaster.  While in certain physical disaster scenarios it’s possible to install an individual to monitor the facility during and after the event this wasn’t one of those times as everyone needed to flee the area.  And having someone come back the next day to keep an eye on things was probably the last thing anyone associated with the bank was concerned with (and rightfully so) as they had lives to save and keep safe.

So no usable lesson to learn and probably no way to ever find out what really happened.  For my money I hope they find the people behind this because it makes me angry to think that while so many people struggled to search for survivors or to recover bodies there were people looking to profit from the situation.

And if there’s anything for the BCP community to glean from this story it’s that no plan can truly account for every possible scenario.  It’s a hard lesson to learn but perhaps one that serves a purpose if for no other reason than to underscore the need for adequate insurance coverage.

This is something of an annual post that I’ve been issuing over the years where I bang the proverbial spoon on the proverbial pot trying to warn everyone that there’s work to be done.  I’m not talking about running through the paces to prepare for an exam but rather having work done that ensures the protection of your customer/member information.  I used to work for a company whose primary sales approach was to  tell current and prospective clients that they had to conduct all manner of tests and assessments because of the regulations.  The firm’s angle was that in order to be compliant you “must do this work,” which not coincidentally dovetailed with services we offered.

I always thought that the “because I said so” logic was flawed.  My thinking then and now was that we should educate clients  on why they need to have regular audits and assessments: How scheduling the work at proper intervals and coordinated activities so that they flow naturally into one another greatly reduces their risk of exposure and improves their reputation as a bank or credit union that can be trusted.   But  what if an institution’s basic strategy is to wait until an exam is a week away and then pull long hours and work all weekend to update what’s needing updating?

The regulatory compliance trinity is fairly simple and straightforward at its highest level:  You document your controls and related activities (the infamous policies and procedures collection), periodically assess your risk factors to determine if you need to add or modify those controls and related activities, and then  test the controls to determine if they’re in place and effective.  GLBA at its core is actually that simple and really quite effective.  It’s GRC 101 and there’s no doubt that by complying with its basic tenets you’re doing the right thing to protect your account holders.

And yet you’d be surprised by how many financial institutions routinely reach this point in the calendar year having deferred scheduling much (if not all) of their compliance work.  You can’t go an entire year without having conducted both an audit and a risk assessment.  No business infrastructure goes through a 12-month period without something significant changing, without risk factors emerging that haven’t been present before that need to be managed.  By extending your compliance work to align with your exam cycle, you’re opening up a huge gap through which a truckload of problems is likely going to drive.  Based on the size and complexity of your institution, you can arrange your compliance program so that not everything needs to occur annually.  I’ve worked with clients where their program called for a risk assessment and audit to occur in alternate years and where only the ongoing programs (e.g. vendor management, penetration testing, business continuity planning, etc.) needed to be addressed and validated annually.  And while it’s true that you don’t need to shoehorn everything into a 12-month period, you do need to have a clearly defined plan on how your institution complies with the various regulations.  You simply can’t get two-thirds of the way through the year without having conducted or scheduled any manner of testing or assessments.

We’re about to turn another page on the calendar and enter September.  While you may count that as four months to year-end and think there’s plenty of time to get things done you need to consider that it’s more like three months.  Between the major holidays, the minor holidays and people taking time off as the year winds down you’re going to find it hard to secure resources to conduct the work and even harder to have them complete tasks while people are constantly out of the office.  So with three effective months of working time left in the year, you need to move quickly to come up with a plan.  What are you committed to accomplishing by year end and how are you going to succeed?  Remember, there’s no more obvious red flag to an examiner than finding a pile of documentation where the ink is still wet or the update/completion dates are suspiciously recent.

And don’t come back at me with the logic that it doesn’t clearly state anywhere in GLBA/NCUA regulations that you need to conduct an audit, a risk assessment or any manner of security-based testing.  As I’ve stated here in my blog several times, FFIEC guidance clearly indicates a need to have a recently conducted risk assessment available.  FFIEC guidance also clearly specifies the need to conduct an audit at a frequency appropriate for the size and complexity of an institution.  All you need to do is look at the Master Table of Contents in the FFIEC examination handbooks to see which parts of your infrastructure need to be tested periodically (why do you think the agency authored the handbooks?).  Considering that both the FDIC and NCUA rely on FFIEC guidance to support their examination process, there’s little doubt (actually no doubt) that’s where you need to look to figure out what work to schedule.

Three months to go, what’s your plan?