Which is why I don’t much care for any manner of compliance-based assessments that are self-administered.

Companies have had this crazy notion for more than a decade now that the best way to identify and address risks inherent within the infrastructure is to ask key stakeholders a somewhat generic set of questions and use their responses to figure out what’s what.  Most of the time the people driving these initiatives are either information security professionals or corporate compliance people who either believe they already know where the problems are or are looking for the simplest and easiest way to satisfy some requirement.  But what they often fail to grasp is that it’s almost impossible to draft a common set of questions that either apply to the vast majority or worse, will be interpreted consistently across the stakeholder population.  Plus the perceived benefit of using a self-assessment approach to reduce effort and required support resources is almost always an illusion.  Most of the time saved in not having someone ask the questions and record the answers is instead consumed by needing to explain the format, explain the questions or trying to clarify and clean up the responses.  While supporting one such program recently each assessment required a kick-off meeting, a follow-up meeting to review the status of the assessment, a third meeting to review the initial draft of the questionnaire, a fourth meeting to review the resulting report(s) and a largely untracked number of hours to help generate all of the related support documentation.  Regardless of the size of the entity being assessed each one consumed somewhere close to eight hours.  While that might seem like a scary large number, the really scary part was that based on which risk analyst was responsible for the assessment and the personality/mindset of the stakeholder completing it the results looked very different from one another.  It was almost impossible to generate meaningful metrics across the assessment population because a “Yes” answer for one question might mean the same as an “N/A” in another; there was no way to know that.

Another issue I’ve always had with the self-assessment approach is that while some stakeholders take it seriously and do a remarkably thorough job, others race through it with little hesitation just to fill in the blanks and get it off their desk.  Sometimes you can detect which is which, sometimes you can’t.  Plus the approach fails to capture much of the rich and relevant information related to each question and the underlying risk behind it.  I recall conducting a team-driven risk assessment years ago where one stakeholder after the next covering a very broad sampling of the infrastructure kept lamenting on the lack of a proper disaster recovery plan.  They had something to show auditors/examiners but to a person no one believed it was a truly viable plan.  All but the CIO brought it up as a concern and when pressed a bit about why that was they all shared a common concern: If their main office was closed unexpectedly for twenty-four hours, regardless of the reason, they were likely out of business.  A related self-assessment question would ask “Do you have a current and recently tested DR plan?” – most respondents on that engagement would simply have selected “Yes” and moved on to the next question without ever being challenged to share their concerns.  Where’s the value in having a repository of questions and answers when it fails to capture the true essence or dimension of risk? 

And the biggest issue I’ve always had with self-assessment questionnaires and their related templates is that they’re so often poorly designed.  I can guarantee you that each of them has at least one question which makes zero sense to anyone who reads it.  They either answer it based on what they think it’s asking, answer with an “N/A” or require follow-up with the people managing the process to have it explained.  And you’d be amazed how many times even the author is challenged to provide a meaningful answer (including this guy).  One thing’s for certain, a self-anything needs to be designed and written so that everyone understands what they need to do without having their hand held.  Plus it’s rare that questionnaires are customized so that each stakeholder is only asked those questions that truly make sense.  An application owner should never be asked if their anti-virus solution is current and up-to-date.  A business process owner should never be asked about software change management.  Yet seldom have I encountered a self-assessment process which does anything like this and so the audience is burdened with time consuming yet unnecessary questions.

Really though in the end my overriding problem with the self-assessment approach is that it fails to capture the expertise and guiding hand of true risk and assurance people.  The process is often supported by analysts who don’t really have a feel for conducting assessments and are satisfied that all of the blanks are filled in.  I have a nose for when there’s something beyond a simple answer and know when to scratch at the surface to bring it to light.  By not allowing expert hands to guide the process potentially huge amounts of valuable and possibly critical details are being missed thus undermining any perceived value of the process.  When you consider that all tolled and tallied the self-assessment approach versus the guided assessment approach doesn’t really save you much time (if any) and that it results in a weaker finished product, why would you elect to use it?   One answer is that regulators push for it because perhaps it’s better than nothing (I can’t get any of those I know to comment).  Another is that the people sponsoring these initiatives lack the fundamental comprehension to understand their options and chose what they perceive as the less complicated approach (again, I don’t know for sure it’s just a theory).  What I do know is that when done right a risk assessment is managements best friend, a fundamental belief behind the recent spike in ERM activity.

While recently having my car serviced the mechanic discovered a nest of some sort in the engine block, he thinks it was probably squirrels.  Because of this discovery he went searching for all the wired connections to make sure they weren’t chewed up and destroyed, quite a few were as it turns out (the car had been idle for several months).  The bill only added the cost of the replacement wires but nothing significant for the time it took to first find which were affected and then replace them.  Had I attempted the repair myself I might have noticed the nest and likely would’ve cleared it but know for certain I never would’ve thought to check the wires, where to look for them or what to look for.  I was smart enough to rely on a professional with a nose for that sort of thing and it saved me time, money and best of all the aggravation of having the car break down somewhere unexpectedly.  Good thing I didn’t go the self-repair route.

On occasion I receive phone calls from recruiters looking for resources to take on contract work. An important part of our practice is comprised of services work and so I’ll look into the opportunity; if it’s consistent with what we do and it’s a good fit for someone in our practice we’ll try and make it work. In this particular instance, the hiring client had some very specific requirements that presented as unusual. It wasn’t so much in what they were looking for from a definition perspective but rather their method of vetting the candidate. The recruiter told me right up front that any candidate needed to present proof of their certifications before being considered for the position. In more than a dozen years working in audit and compliance I can’t recall ever being asked right up front for such information and it caught me off guard.

The certification in question was the Certified Information Security Auditor (CISA) designation issued by ISACA. Generally speaking it’s the defacto standard when it comes to my professional space but only because it’s the only one available. While there are a number of IT auditors who also have the CIA designation it’s somewhat rare and unusual. But while it may be the standard cert for IT auditors, it’s certainly not a hard requirement and not something that all practitioners aspire too. I probably know more excellent IT auditors who don’t possess a CISA than I do those who do. I sat for the exam (and passed) back in 2005 because I was looking for a way to bookmark my audit experience; too many recruiters saw my resume and thought of me more as an IT practitioner than as an audit/compliance resource. I wanted to distinguish myself as an auditor and that seemed to be the best, most direct way to do so.

What I learned during the period of time while studying for the exam was that I already knew what was necessary to pass the test. There were a few disciplines covered during the exam that exceeded my knowledge (primarily around cryptography, encryption and key management) but I was okay with that because those were areas I would never pursue work in (we throw that stuff to our CISSP’s). Midway through the exam preparation experience, I questioned the validity of the certification. I genuinely believed that my previous eight years of experience spoke more to my expertise than any certification ever could. A year or so later I came to learn that when ISACA issued new certifications they also allowed for grandfathering – you could simply pay for the certification if you could prove that you already had the experience doing that sort of work. That cemented my opinion that experience was far more significant than the cert (and it also meant that a solid number of CISA’s I knew never had to pass the exam).

Within the first two years after I passed the exam I knew three people with almost no audit experience who studied for and passed the CISA exam because they believed audit and compliance work was their best way to stay employed. None of the three knew how to conduct a risk assessment, develop an audit plan, write an audit program or build work papers after taking the exam, yet all three were CISA’s. With some minor modifications to their resume they could present themselves as true audit professionals. That also cemented my opinion that the certification wasn’t as much of an indicator of ability as I once thought.

I recall a conversation with someone who was an IT audit instructor, but who at the time didn’t possess the CISA certification. His issue with the certification was that he didn’t believe multiple choice exams proved competency because you knew one of the provided answers was correct and so you just needed to be good at taking exams and making educated guesses. I don’t know if I completely agree but I have come to believe that the CISA certification would be that much more meaningful if the candidate had to display a basic ability in conducting the related work. Give them a set of criteria about an environment (e.g. software, networking, etc.),  have them create a risk assessment to determine what should be assessed, develop an audit plan based on the identified risks and write the audit programs to test the necessary controls. A panel of reviewers could grade the material and decide if the candidate possesses the necessary competencies. At least with such an approach you would know that if you hire a CISA certified practitioner, they have the skills to do the job. By the way, of the three aforementioned  practitioners who are CISA-certified, only one could actually pass such an exam today, three-plus years after having obtained the designation.

And so in an industry where you don’t need a certification to work (unlike the medical or legal professions), I’m not sure that a similar value should be placed on possessing one.