Right there on the front page of today’s News and Observer was a story about how a recent audit claimed corruption at a local college (North Carolina Central University).  I’m sort of trained in a Pavlovian sort of way to notice anything having to do with audit and so I gave it a cursory read.  Cursory turned into focused when I reached the part about how the school’s chancellor Charlie Nelms called the report draft “sloppy” and went on to say that some of its harshest accusations might not be true.

One of the oldest tricks in the business book when it comes to audits is to start questioning the quality and veracity of reports that are perceived as not being favorable.  Instead of focusing on the audits findings and trying to validate them (because a good audit is your best friend if you really want to do things right) the auditee goes into a series of tactical maneuvers to deflect attention away from the report’s contents and feigns disgust and outrage.

The school chancellor went on to say that, after firing the auditor who produced the report, he “ordered his staff to gather more information before he releases a final version to the public.”   He went on to say that the “draft audit was so poor that he doesn’t trust it, and he does not want to damage the reputations of people who might not have done anything wrong”.

A few years ago, I conducted a risk assessment for a client with an odd configuration of infrastructure pieces that clearly defied anything close to typical, so it was difficult to measure them against the norm.  Just the same, I tried.  I took a step back after conducting all of my interviews and gathering as much information as was available and filtered it through the lenses of an examiner.  I surfaced gaps and issues that were likely to be viewed in a negative light, explained why that was and offered clear and concise remedial steps.  Senior management went bonkers (for lack of a better word) when they received the report.

They were outraged because the report was delivered a week late (which was true), they were insulted that there were typos (not factual errors, just a few grammatical/spelling hiccups which are common in draft versions) and charged that some of the issues listed were completely false.  In summary, they called into question the accuracy and reliability of the entire report.  It was startling for me because in my more than two decades working in the business world with more than 10 years conducting audits and assessment, I’d never had a client react anywhere near this way before.

But it was really more about using diversionary tactics intended to gain a negotiating advantage.  Their end game was to soften the report’s contents so that it looked better when the examiners came back around; by pushing us back into a defensive position, they were almost successful.  Fortunately, I’m stubborn when it comes to standing behind my findings and need incontrovertible proof that I was wrong about something before changing or removing things.  I may not be the best auditor but I have well honed instincts around IT, the myriad processes necessary to support the infrastructure, and I know good from bad.  I never put anything into my reports that doesn’t resonate with me and my peers (and typically the report’s audience).

So you can imagine where my head was at while reading the story today.  Mr. Nelms also said, “I want to see the source documents, and I want to see the field notes from the audit, because I want it to be accurate.  I don’t want it to be hearsay, because some of the allegations are just mind-boggling.”

Well that’s good to hear because any audit worth its weight in paper needs to be supported by solid work papers.  But considering that he fired the auditor, I’m hoping someone in his office thought to secure that beforehand.  And I’d need to understand why he’s gathering more information when all he really needs to do is use the work papers and have another independent auditor re-perform the tests.

Oh and another thing, who hired the auditor to begin with?

Also, now that the report’s findings are semi-public (it’s available despite not having been formally released), where’s the value in conducting a follow-up audit?  Anyone involved with any alleged wrongdoings now has a clear roadmap in front of them on how to cover their tracks.

Here’s my thinking on all of this: The audit is likely somewhere close to 100% accurate but far from perfect (I know that’s a contradiction).  If the chancellor was really interested in handling this properly, he’d quietly set about having independent people digging into the findings, not as a CYA exercise but simply to get to the bottom of things and deal with whatever is found.  I’m not saying that where there’s smoke there’s always fire but unless Mr. Nelms can offer a credible explanation why he would think that the fired auditor would fabricate stories or offer poorly formed conclusions I’d have no choice but to question his position on all of this.  I guess what I’m asking for is a credible explanation as to where the smoke is coming from and an explanation why he thinks it’s benign.

What I’d like is to hear the auditor’s side of the story.  I’m betting that would be an enlightening conversation.  But if Mr. Nelms was successful in his very public tongue lashing of this auditor, he/she will do anything and everything to avoid having their name outed.  And so the diversionary tactics score another point.

And the best part of this?  I almost never read the paper.

But this leads to another interesting quirk about how the examiners often operate. Generally speaking, if the reports are available, they don’t dig much (if at all) beyond the reports contents. And so the information security and IT components of many exams become more about inventorying recent reports and not much else. We see evidence of this all of the time when we conduct a first-year audit or assessment and discover gaps or issues that have been in place for years and which the exams never picked up on.

I’ve written in the past about how surprisingly few institutions maintain a current business continuity plan and even fewer properly test that plan. But what surprises me more is that these conditions have existed for years spanning many exam cycles. How is that even possible?

I’ll tell you how: There’s a documented plan that is provided upon request and by and large the examiner conducting the fieldwork checks off that they received it and voila, you have a non-issue. And because the people in the field are typically given too few hours to cover too much landscape, they don’t have enough time to dig in deeper. Sometimes it happens where an examiner happens to actually open the document and vet it for key details – every now and again we come across a DoR or an MoU where the absence of a recent business impact analysis was tagged – but that happens almost never.

I’m fond of advising clients that you conduct much of the required compliance work for one of two reasons: You do it because it’s the right way to manage your institution and reduce your risk or because you have to. Because of the approach taken by examiners, way too many institutions lean towards the latter and simply want to have a report available to hand over when asked. But is this really the right way to run a financial institution?

And when you consider that the value of the report is largely defined by its contents and the competency of the practitioners conducting the fieldwork behind it, isn’t there an increased likelihood that there are important issues that go undetected? If all you do is pay for the report (often issued by the firm submitting the lowest bid) and all the examiners do is check off that the report was available and issued during the appropriate time frame, is there any real value in even bothering with this process?

I’m a bit biased regarding the value of reports. My firm is on a constant hunt for real risk and not just simply working our way down a checklist to kick out a document and collect our money. We tend to examine our clients infrastructure as if though we have our own money deposited with them and tie what we see straight back to GLBA and NCUA requirements. The value in this approach is that we produce a report that the board of directors can relate to, not just the IT folks.

But again, if no one really even cares about the content of the report and only that it exists, why bother doing a good job?

Maybe our industry needs to adopt an approach similar to the PCI folks. Maybe the FDIC and NCUA should issue certifications to practitioners validating them as properly trained and educated experts with regards to GLBA. There would still be a variance from firm to firm to a certain degree, but at least there would be a recognized standard and an increased likelihood that if an examiner is going to rely on the competency and completeness of a report there’s some justification behind that decision.

Something’s going to have to change though and hopefully sometime soon. Because using the “last minute” logic is flawed and only serves to reinforce my own bad habits.

]]> http://itknowledgeexchange.techtarget.com/regulatory-compliance/should-examiners-rely-on-audit-and-assessment-reports/feed/ 0 http://itknowledgeexchange.techtarget.com/regulatory-compliance/lets-audit-the-review/ http://itknowledgeexchange.techtarget.com/regulatory-compliance/lets-audit-the-review/#comments Tue, 01 Sep 2009 15:29:17 +0000 David Schneier http://itknowledgeexchange.techtarget.com/regulatory-compliance/?p=185 I had mentioned in my last post a recent conversation with my partner regarding a proposed IT general controls (ITGC) audit. My primary role in our practice is to head up regulatory compliance services which includes audits, assessments and program development; my partner’s primary role is head of sales and business strategy. However, there’s a significant amount of overlap between our two sides and I sometimes forget that I’m the compliance expert when we’re discussing the industry. His knowledge of the myriad regulations is impressive and there are times where I’ll vet ideas through him to validate my own thinking.

Anyway, he’d mentioned a conversation with the client around the proposed ITGC audit in which the project sponsor asked what the difference is between an audit and a review. I knew right away where the question stemmed from because of my experience in the industry. Many firms we compete with (and some I’ve even worked for in another time and place) don’t conduct audits, they conduct reviews. Sometimes they don’t even conduct a review or an audit but rather an assessment. I’ve struggled with this blurred use of terms because in my mind there are very clear delineations. The lifecycle of the governance, risk and compliance (GRC) domain is as such: identify and assess risk, design controls to mitigate those risks and test to validate that those controls are functioning as expected. And so, a risk assessment is conducted, governance elements are introduced in the form of policies and procedures, and regularly scheduled tests occur to make sure that the whole enchilada is actually getting the job done. And so when practitioners offer services that aren’t specific to one of the key areas of the GRC spectrum, it bothers me.

See, an audit is an audit is an audit; you determine the control objectives that are supposed to be supported by the entity that are in scope for the type of audit, identify the related control activities that either are or should be in place and then design a series of test steps to determine if those activities are occurring and tie back to the overall objective. The auditor has some leeway when it comes to offering an opinion as to what the results actually mean (one auditors pass is oft times another auditors finding) but fundamentally the audit results tend to be binary, you either pass or fail. It’s all fairly straight forward.

Now a risk assessment is not an audit; it’s a bit more arbitrary. Management generally is polled to determine what areas of their infrastructure (including finance, operations and technology) they are most concerned with, factor in regulatory and industry requirements and then come up with a plan for conducting the risk assessments. As these assessments occur, what they reveal would be factored into the overall audit plan to make certain that the areas presenting the greatest risk to the business are being examined closely and in a timely fashion.

So here’s my question: what exactly is a review? If you’re conducting tests and examining evidence you’re conducting an audit. If you’re interviewing stakeholders and determining what controls are in place but not testing their effectiveness then you’re conducting a risk assessment (assuming you’re asking the right sort of questions – a post for another day). Is there some odd dimension between the risk assessment and the audit I’m unaware of where this review occurs? And what exactly are you expecting from the results of this review? Because I’ll tell you this much, examiners only recognize risk assessments and audits. You can present them with a report that’s called a review in the title and tell them it’s actually an audit but you’d better be prepared to produce work papers because they’re going to ask you for them, trust me on this point. But my experience is that reviews don’t produce work papers because evidence is generally not formally collected in support of the report and requires significantly less effort to conduct. Which is why when we discussed the semantics of our industry, my partner is fond of saying that the difference between an audit and a review is about 50% of the proposed amount.

And as long as I’m ranting on the topic, how many of you out there include work papers as a deliverable when you contract with external entities to conduct audits? I’ve long been amazed at how many audit projects I’ve managed where work papers weren’t required or provided because it wasn’t included in the statement of work. I’m of the opinion that a summary audit report by itself is useless if you don’t have the supporting documentation because it’s the only true way to confirm that the testing occurred and support the findings. And of course it’s important to loop back to my earlier comment: Examiners will absolutely ask you for the work papers. I routinely receive phone calls from clients I’ve worked with at my previous firms asking if I still had access to their work papers because their examiner is asking for them. It’s awful for me because I have to tell them that while the evidence is still available (I keep everything for seven years for fieldwork I’ve conducted) there are no formal work papers to provide. Thus the reason that when we started our firm and developed the methodologies, I made certain that part of the scoping process included asking the client if work papers were to be included in the deliverables. It adds time to the project and so there’s a cost implication, but it’s the right way to run an audit and so really a need-to-have. You can take a short cut where applicable and only document findings but even so, how can you prove that a successful test was actually successful? For those practitioners looking for shortcuts it provides the wrong incentive.

If you want/need an audit, schedule an audit. If you want/need a risk assessment, schedule a risk assessment. If you’re considering a review or a generic assessment reconsider; decide which of the two proper categories your work fits into and than schedule accordingly. This isn’t rocket science; the FDIC and NCUA have been quite clear as to what you’re required to do so keep it simple. And if the resources you’re using aren’t speaking in the common audit and compliance vernacular force their hand and make them do so. Audit or risk assess, pick one.