Which is why I don’t much care for any manner of compliance-based assessments that are self-administered.

Companies have had this crazy notion for more than a decade now that the best way to identify and address risks inherent within the infrastructure is to ask key stakeholders a somewhat generic set of questions and use their responses to figure out what’s what.  Most of the time the people driving these initiatives are either information security professionals or corporate compliance people who either believe they already know where the problems are or are looking for the simplest and easiest way to satisfy some requirement.  But what they often fail to grasp is that it’s almost impossible to draft a common set of questions that either apply to the vast majority or worse, will be interpreted consistently across the stakeholder population.  Plus the perceived benefit of using a self-assessment approach to reduce effort and required support resources is almost always an illusion.  Most of the time saved in not having someone ask the questions and record the answers is instead consumed by needing to explain the format, explain the questions or trying to clarify and clean up the responses.  While supporting one such program recently each assessment required a kick-off meeting, a follow-up meeting to review the status of the assessment, a third meeting to review the initial draft of the questionnaire, a fourth meeting to review the resulting report(s) and a largely untracked number of hours to help generate all of the related support documentation.  Regardless of the size of the entity being assessed each one consumed somewhere close to eight hours.  While that might seem like a scary large number, the really scary part was that based on which risk analyst was responsible for the assessment and the personality/mindset of the stakeholder completing it the results looked very different from one another.  It was almost impossible to generate meaningful metrics across the assessment population because a “Yes” answer for one question might mean the same as an “N/A” in another; there was no way to know that.

Another issue I’ve always had with the self-assessment approach is that while some stakeholders take it seriously and do a remarkably thorough job, others race through it with little hesitation just to fill in the blanks and get it off their desk.  Sometimes you can detect which is which, sometimes you can’t.  Plus the approach fails to capture much of the rich and relevant information related to each question and the underlying risk behind it.  I recall conducting a team-driven risk assessment years ago where one stakeholder after the next covering a very broad sampling of the infrastructure kept lamenting on the lack of a proper disaster recovery plan.  They had something to show auditors/examiners but to a person no one believed it was a truly viable plan.  All but the CIO brought it up as a concern and when pressed a bit about why that was they all shared a common concern: If their main office was closed unexpectedly for twenty-four hours, regardless of the reason, they were likely out of business.  A related self-assessment question would ask “Do you have a current and recently tested DR plan?” – most respondents on that engagement would simply have selected “Yes” and moved on to the next question without ever being challenged to share their concerns.  Where’s the value in having a repository of questions and answers when it fails to capture the true essence or dimension of risk? 

And the biggest issue I’ve always had with self-assessment questionnaires and their related templates is that they’re so often poorly designed.  I can guarantee you that each of them has at least one question which makes zero sense to anyone who reads it.  They either answer it based on what they think it’s asking, answer with an “N/A” or require follow-up with the people managing the process to have it explained.  And you’d be amazed how many times even the author is challenged to provide a meaningful answer (including this guy).  One thing’s for certain, a self-anything needs to be designed and written so that everyone understands what they need to do without having their hand held.  Plus it’s rare that questionnaires are customized so that each stakeholder is only asked those questions that truly make sense.  An application owner should never be asked if their anti-virus solution is current and up-to-date.  A business process owner should never be asked about software change management.  Yet seldom have I encountered a self-assessment process which does anything like this and so the audience is burdened with time consuming yet unnecessary questions.

Really though in the end my overriding problem with the self-assessment approach is that it fails to capture the expertise and guiding hand of true risk and assurance people.  The process is often supported by analysts who don’t really have a feel for conducting assessments and are satisfied that all of the blanks are filled in.  I have a nose for when there’s something beyond a simple answer and know when to scratch at the surface to bring it to light.  By not allowing expert hands to guide the process potentially huge amounts of valuable and possibly critical details are being missed thus undermining any perceived value of the process.  When you consider that all tolled and tallied the self-assessment approach versus the guided assessment approach doesn’t really save you much time (if any) and that it results in a weaker finished product, why would you elect to use it?   One answer is that regulators push for it because perhaps it’s better than nothing (I can’t get any of those I know to comment).  Another is that the people sponsoring these initiatives lack the fundamental comprehension to understand their options and chose what they perceive as the less complicated approach (again, I don’t know for sure it’s just a theory).  What I do know is that when done right a risk assessment is managements best friend, a fundamental belief behind the recent spike in ERM activity.

While recently having my car serviced the mechanic discovered a nest of some sort in the engine block, he thinks it was probably squirrels.  Because of this discovery he went searching for all the wired connections to make sure they weren’t chewed up and destroyed, quite a few were as it turns out (the car had been idle for several months).  The bill only added the cost of the replacement wires but nothing significant for the time it took to first find which were affected and then replace them.  Had I attempted the repair myself I might have noticed the nest and likely would’ve cleared it but know for certain I never would’ve thought to check the wires, where to look for them or what to look for.  I was smart enough to rely on a professional with a nose for that sort of thing and it saved me time, money and best of all the aggravation of having the car break down somewhere unexpectedly.  Good thing I didn’t go the self-repair route.

I just heard that Mayor Bloomberg is evacuating low-lying areas in New York City and that mass transit will more or less be cut off tomorrow (Saturday) sometime around mid-day.  New York’s Governor Cuomo also discussed the possibility of closing the bridges as well if weather conditions become so severe that using them might be dangerous.  Upon hearing this my first thought was “how the heck are key stakeholders going to get to their disaster sites if they’re called in?”  The obvious answer is that many companies will likely require that the important people go to their DR sites tonight so that they’re already there “just in case”.  How wonderful for these people to have to leave their families in the midst of a potentially epic natural disaster.  I can’t help but wonder how many are willing to comply and how many are going to insist that they can’t make it.  Did any BCP/DR test ever take into consideration the possibility that key stakeholders would simply refuse to show up?

And with the enormous range of Hurricane Irene is it at all possible that certain recovery sites might not be able to provide the proper services, resources and support to meet such a potentially large demand?  I know that they all claim that they’ve factored that in to their models and are able to provide sufficient capacity.  But until they know for sure how do they really know for sure?  Who among us has yet to witness any BCP/DR plan that didn’t start experiencing hiccups and delays during testing?

One element of a BCP that I’m also now wondering about is the day-after scenario.  I’ve reviewed dozens of plans during my career and upon reflection cannot recall any that placed significant attention on what happens after the official disruption is at an end.  I’m looking at pictures of severe flooding from Irene from those places already affected and have to wonder how many business are going to be able to open on Monday despite the fact that the roads are clear and the skies sunny and blue.  In thinking about some of the more common disruptions over the years (e.g. heavy snow, ice, etc.) it was somewhat obvious that once the roads were passable it was safe to head back to the office.  But that may not be the case this time around.  How many plans are designed to accommodate that?  Is someone from facilities charged with the responsibility of conducting a site inspection on Sunday night to see if their buildings are ready to open the next day?

Admittedly I’m picking on the entire concept of a business continuity plan but you can’t blame me, Hurricane Irene is only one reason.  Middle of last week I was in the Northeast and experienced my very first earthquake event.  Now I realize that anyone from California or Japan would chuckle at that statement because what I personally experienced was little more than an overloaded truck driving past me on a pothole-ridden street to those who deal with the phenomenon regularly.  But still, for me it was a big deal.  In the aftermath I asked around to see what happened in other places where the tremors were felt to see if anyone was formally evacuated from their building – no one was.  I expected in the days following to read about how companies had dedicated time and resources to inspect their structures to ensure that everything was as it should be and that there were no signs of damage from the unexpected movements – again, almost nothing to be found.  Well for all those BCP’s that I’ve reviewed where the likely threats were documented and addressed as part of their plan, how many think that maybe they should update their documentation to cover earthquakes?  They can no longer justify leaving it out because it’s not a likely threat, it just happened.  And now that they know it happened once they need to accept that it not only could happen again but likely will. But I’m willing to bet that a year from now I won’t find a single plan that has been modified to include what should happen in the event of an earthquake.

I’m just thinking that regulators and auditors need to stop rewarding those they’re responsible for monitoring for simply having a plan in place.  At some point they’ll need to shift their focus from simply checking off that a plan exists and start digging into it a bit more.  The same degree of scrutiny that emerged in 2009 because of the “Great Swine Flu” threat and making sure that BCP’s had a thorough pandemic response component now needs to become standard fare for the overall plan.  Companies need to conduct more than tacit testing exercises and really start thinking things through.  Between companies having antiquated and irrelevant plans, to those who have partially baked plans and worse yet, those who don’t even have one in place it’s time to do something about it.

The worst time to discover that you need a viable plan and don’t have one is, well, when you actually need it.  If enduring both an epic hurricane and your first earthquake don’t inspire you to action nothing will.

Japan’s tragedy will serve as a fertile source of both proving and disproving the myriad business continuity and disaster recovery techniques being used around the world today.  The most prepared and best trained companies will have very likely fared about as well as could be expected while those who weren’t, those who either had partially baked plans or no plans at all will be lucky to survive in any measurable way.  And it’s hard to imagine that most companies didn’t have plans to deal with earthquakes and tsunami’s because they’re credible and consistent threats in the region.  But after a quarter century in corporate life and little more than half those years focusing on audit and compliance I’m no longer surprised by anything I encounter.

However there was one story to emerge from Japan this week that I found to be quite shocking.  It was about how a banks vault came open during the series of events and someone stole forty million yen (about $500k USD).  It happened in the prefecture of Myagi in a town known as Kesennuma and police said that between the wave’s power and the ensuing power outages, the vault came open.  What with all the flooding and chaos it took more than a week for someone to get back into the building and discover what had happened.

For many the story seemed plausible if not mildly amusing because who wouldn’t love to wander into a bank and be able to scoop up all the cash floating around.  And because in this particular situation no one died or was hurt as a result it’s benign enough to be more entertaining than tragic.  It sort of reminded me of a scene in the movie “Ground Hog Day” where Bill Murray’s character figured out the perfect timing to be able to steal a bag of cash out of the back of an armored truck.

But I sort of have a problem with this story because I don’t think it happened the way it’s being portrayed.  My very first thought upon reading the details was that either someone left the vault door open as they were fleeing the bank or someone who knows a thing or two about how to open a vault went back in after the fact and exploited the situation to their advantage.  The odds that a vault door simply flew open due to what was really a massive flood at that point just doesn’t hold up under scrutiny.

Have you ever actually seen what a door on a bank vault looks like?  I have and I’ve probably seen about three dozen or more since I started working in the banking sector and I couldn’t think of how any one of them, if closed properly would ever just come open due to rushing water for a relatively short period of time.  First of all they’re all seated within a metal frame and so for the rods or pistons that create the seal to come undone the metal itself would need to have been bent or twisted.  Second, they weigh a ton (not as much of an exaggeration as you might think).  Even the weakest vaults I’ve encountered have doors that have some serious density to them and would not likely bend under most natural forces.  I would sooner believe that the walls that the door and its frame were attached to failed then believe that the door simply “flew open”.

If I had to put on my most skeptical mindset to use I would venture a guess that the person responsible for making sure the vault was properly closed before safely exiting the building rushed through the procedure, didn’t properly lock the vault and in their heightened state of panic just didn’t think about it.  While that’s the most likely scenario the second most likely version is that someone who knows how to open the vault door and who knew after a day or so that no one would ever be concerned with theft while there were still lives to save made their way into the crippled building with its security systems down and manually opened the door and had at it.  But under either scenario it’s almost entirely likely that the person(s) who stole the money had an idea about what to do and took advantage of the situation.  I mean, they obviously entered the bank after the disasters struck and they weren’t likely looking for survivors if they were of a mindset to grab what had to be a sizable physical haul.

And the thing is that there’s no viable lesson to be learned from a story such as this.  I’m certain the bank had a procedure in place that specified how all cash drawers were to be placed in the vault and that the vault itself should be locked upon exiting during a disaster.  While in certain physical disaster scenarios it’s possible to install an individual to monitor the facility during and after the event this wasn’t one of those times as everyone needed to flee the area.  And having someone come back the next day to keep an eye on things was probably the last thing anyone associated with the bank was concerned with (and rightfully so) as they had lives to save and keep safe.

So no usable lesson to learn and probably no way to ever find out what really happened.  For my money I hope they find the people behind this because it makes me angry to think that while so many people struggled to search for survivors or to recover bodies there were people looking to profit from the situation.

And if there’s anything for the BCP community to glean from this story it’s that no plan can truly account for every possible scenario.  It’s a hard lesson to learn but perhaps one that serves a purpose if for no other reason than to underscore the need for adequate insurance coverage.