In the days since that conversation I’ve put some thought into the frameworks because in the end the aforementioned CISO was committed to finding the NIST experience and eventually did.  But what did that really mean?  Having fairly recently had the occasion to have both NIST 800-53 and the ISO 27000 documents  in front of me it was striking how similar they both were with only a few obvious distinctions to be made between the two.  Essentially the differences reflected more on the cultures that created them than the risk factors they were focused on (NIST = U.S.A and ISO = European).  But information technology architectures fundamentally are identical the world over so despite formatting and spelling they both are addressing the same challenges whether or not they realise it. And for those of us who have familiarity with both, to know one is to know both, even if those who are committed to either one disagree.  If you’ve worked on audit/assessment projects leveraging ISO 2700o material you’re immediately qualified to work on projects using the corresponding NIST framework and vice versa.   And if you have experience working with PCI standards guess what?  You can pretty much step in and work with either NIST or ISO content (except of course you have to expand your sights to include the entire infrastructure, not just on whatever touches PAN data).

My preference is that we would consolidate globally into the ISO frameworks where applicable and maybe even fit that in to the SSAE 16 process.  I’ve read enough toothless SAS 70/SSAE 16 reports to know that it’s easy enough to rig the system to your advantage.  And unless you’re a government agency that has to comply with NIST there’s little meaningful value to using NIST whereas being ISO 27000 certified carries a great deal of weight within the audit/assurance community.  Plus there’s the added benefit of having InfoSec practitioners all getting trained and practiced at both building out ISO 27000 compliant solutions and also knowing how to test the related controls.  Think about that, a single global security standard regardless of where you enter into the profession.  Having run a few practices in my career and way more than my fair share of engagements I can tell you that has great appeal.  Plus it would help eliminate awkward dialogues where my sixteen years of real and relevant experience is at least partially marginalized because it hasn’t all been with one particular standard.

Ultimately in the end a frameworks only meaningful advantage is that it theoretically ensures consistency in how controls are identified and assessed.  If you have someone who knows a framework but doesn’t really understand the details within that sort of defeats the process anyway, no matter how robust or thorough it may be.  Perhaps that’s why I consider it a non-issue when it comes to which frameworks a practitioner has used.  I’d much rather work with someone who understands the technology and has a good feel for the details rather than someone who knows that SDLC is addressed in SA-3 for NIST or Section 12.5 for ISO 27002.  But than again, I’ve always been more concerned with real risk, not perceived risk so this shouldn’t be surprising to anyone who’s read my content in the past.

A security framework by any other name would be just as comprehensive, you know what I mean?

Now I’ve designed and supported more than my fair share of related content.  I understand that sometimes the best way to tell a story is to paint it in the form of a picture; I get that part.  But way too many times I’ve witnessed such initiatives spiral out of control to the point where it takes an army of people working ridiculous hours to pull together a deck of metrics that either fails to answer anyone’s questions or, even worse, generates requests for more metrics to provide clarity.  And once a metric becomes a standard part of any reporting package it often stays there until management changes, and sometimes even beyond.

But I think there’s a bigger issue with metrics that exceeds my simply not thinking they’re “all that and a bag of chips”.  Where are the controls around generating them?

Seriously, we have this vastly complex framework wrapped around financial reporting (SOX) to provide reasonable assurances that what management is reporting to its investors is accurate.  We have industry, federal and state legislation requiring all manner of controls around sensitive information.  There are auditors (internal and external) and regulators from all over the place that comb over everything with a fine tooth comb (or at least claim to) to make sure everything being done is done right – but in my nearly fifteen years in the audit/assurance industry I have never heard of a finding or issue regarding the veracity of metrics.  Which is only a problem if the people running an institution or company rely on them to make decisions.

The reason why it’s a problem is because so much of the metrics out in circulation is pulled together from disparate sources, cobbled together in spreadsheets or non-production databases and manually generated.  There’s no easy way to verify the source data, or know that it’s unaltered in any way or even know if it’s the right information.  And even if the data source used is from a secured production-like environment, still there’s no real auditing conducted to ensure the information is accurate or better yet, is even the right information.

I once took over a change management process and assumed responsibility for a series of reports that were generated for the Managing Director who in turn used that as part of his reporting package shared with the CIO.  One of the key metrics being reported on was scheduled releases and the IT departments on-time implementation percentage.  The numbers looked great showing that they were on-time more than ninety-five percent of the time over a two year period.  The only problem I could see with the metric was that it was misleading to the point where it was almost a lie.  The scheduled release date was being pulled from the system used to migrate changes into production and that date was only determined once the development team had completed all of their work.  So the scheduled implementation date was chosen once they knew they were ready to move into production.   Of course the on-time numbers looked great, they always knew they were ready before committing to it.  The Managing Director incorrectly assumed that there was a legitimate release schedule with forecasted dates and that the on-time numbers reflected on a well run process; wrong.   No one ever questioned the numbers or their source and had I not inserted myself into what was described as a well honed, efficient process the problem might have never been identified; and there a few more just like it.  My trust in metrics was permanently altered after that.

Metrics represents an excellent way for decision makers to quickly understand status and identify problems.  I’ve quoted here before about how someone I respect quite a bit was fond of asking her team “If you can’t measure it, how can you manage it” and she’s absolutely right.  Metrics is the ultimate management means of measuring key activities and issues within their world.  But how far do you go and how much effort do you expend pulling the related reports together?  And even if you’re able to automate the process and shorten the time necessary to generate the reports, how do you know that you’re either measuring the right things or that the underlying data is unaltered?  Ultimately I think that senior managers should be provided with something akin to a cost-benefit analysis for each metric they’re given.  Have them understand the degree of complexity and the amount of effort required to generate a number before deciding whether or not it’s worth it.  Perhaps I’m being naive but I’d like to think that most C-level executives would eliminate a significant amount of their reporting if they could see how much it was really costing them.

Here’s the part that should really concern you the most though: Metrics is a key component of Board reporting, they make all sorts of decisions based on what these reports tell them.  How can that be allowed unless the process used to generate them is locked down and audited?  Where are the regulators in all of this?

That first experience has arguably tainted my opinion of the role played by internal audit for nearly twenty years.  Subsequent to that first encounter I’ve been audited a few more times, assisted clients in preparing for internal audits many times and have had hundreds of interactions either directly or indirectly with a variety of companies internal audit function.  And despite all of this experience and having eventually become an auditor myself I’m not sure I could present a credible argument as to where there’s real value being generated by the process beyond maintaining appearances.

The first problem is that for most companies there’s an unhealthy fear of auditors.  There’s often real concern that if any major issues are uncovered someone’s head will roll.  At the aforementioned Fortune 100 company, it was widely believed that if your group was found to have a material finding (or anything remotely resembling one) the highest ranking person in the group was doomed.  To their credit the company also had a mechanism in place so that if you figured out that you had a problem before anyone else and self-reported it you were allowed appropriate time to remediate.  But that wasn’t always effective enough because most application and business managers weren’t auditors and couldn’t always recognize when a control was either missing or failing and so there was still an enormous amount of work and panic leading up to a scheduled audit.  I remember thinking that the company should remove the threat of termination and encourage both auditor and auditee to work openly and honestly together so that in the end issues were surfaced, defined and repaired.  In the two decades since I’ve worked with and for a few companies who believed they had this healthier sort of dynamic in place between their internal audit department and its business and technology functions but really in the end it’s almost always the same problem.  Internal audit is viewed as an unforgiving and punishing agent and no one ever want them snooping around.

The second problem is that there’s a degree of incompetence found within many internal audit functions.  While conducting my first technical audit back in 1997 (my company was managing an outsourced audit plan) I identified a significant issue with the methodology used to make production changes in a certain database environment.  It resulted in there being virtually no clear or simple way for the DBA to back out a change if it didn’t work.  If a change failed it would require bringing down production for several hours in order to restore things to the previous state.  The first person who challenged my finding was the internal auditor who had audited the same platform for years and didn’t either understand or agree with the finding.  It took me nearly an hour to first educate him as to why the technical issue existed, prove that it did and finally to agree with the associated risks.  He had worked there for years, had never had the chance to see how other companies managed similar infrastructures and was way more concerned with his authority and capabilities being challenged than with the fact that his company had a significant risk to be repaired.  In the time since I’ve met many more people just like that one, auditors who stay at one company for years, fall into bad habits and fail to keep their skills relevant.  They wind up relying too much on the Internet to try and update their knowledge base, don’t have the perspective of understanding how other companies are managing similar challenges and are happy enough to bring out the same whipping stick and a feeling of empowerment to scare the daylights out of internal control owners while conducting their audits.  It results in poorly formed and often irrelevant findings that waste everyone’s time.  I wish I had a ten dollar bill for every instance I knew of where something was being fixed because it was easier to appease the auditor than it was to convince them their finding was flawed or even wrong.

Now I’m not saying all internal auditors are incompetent, they’re not.  I’ve met some brilliant and extremely effective internal auditors along the way.  And in those environments audits weren’t feared because there was a high degree of confidence that if an issue was identified it was something worth knowing about.  But in almost all of those cases the auditors involved had only been with their company for a few years, not decades.

The third problem is that audit needs to be seen as adding value, not creating unnecessary delays or work.  Practically speaking internal audit is playing for the same team as the control owners whose processes they assess.  Their primary goal shouldn’t be to notch as many findings as possible on the board but rather to identify weaknesses and deficiencies so that they can be remediated and help further harden the infrastructure and reduce risks.  I understand the need for the function to maintain independence and separation but only so they can remain objective not so they can operate as if though they’re the ultimate authority on right and wrong and beyond reproach.  If they’re invited to participate early in a project and find issues they should issue interim findings so that small problems don’t become bigger problems further on down the project road.  If you wait for the post-implementation audit to document early stage issues you’re not really helping anyone.  If they abuse being granted access to meetings and documentation long before the audit function is typically engaged the only predictable outcome is that access will be denied until someone forces the issue.  And one more major issue I routinely find with internal audit is that no matter how strong or weak a finding may be, no matter how poorly or strongly worded, no matter how relevant or irrelevant they all too often defend it as if though it’s gospel that’s beyond reproach.  Why is that?  Why can’t the control owner question the finding, demand clarity or try to frame it’s relevancy?  All auditors should feel an obligation to issue a final report which resonates with everyone involved as being accurate and hopefully fair.

Until internal audit is seen as part of the solution, not part of the problem it’s going to remain, well, a problem.  Until control owners gain a sense that by developing a healthy dialogue with their auditors it will only help things and not hurt them it will continue to be a problem.  And until all involved parties working for the company feel as if though they’re working towards a common goal it will remain a problem.

Is it possible that this incredibly important and still developing concept known as GRC can be hijacked and used instead to almost marginalize the sum total of all it’s related parts?  Until this week I would never have even thought of something like this as possible but there it was, right in front of me and a bit of a shock.

There are likely two main drivers behind this disturbing trend: GRC software and the overwhelming volume of compliance-based activities.  So many of the GRC solutions currently on the market tend to be rather broad in their scope.  While most of them are oriented towards one particular point within the GRC spectrum they have all expanded to try and touch on as much as they can justify.  So whereas you have a product that may have been designed to manage policy content it now also offers risk assessment, audit and overarching governance features.  But still what it does best is manage policy content.  The license for the product isn’t cheap and senior management has been sold to some degree on the promise of automating much of the required work via this new and costly solution.  Thus we have the first driver behind the blurring of GRC lines: “We paid a lot for it so we better use the heck out of it”.  And so there’s a slow but steady march through the organization looking for things that can be brought into the fold.  However not everything belongs in every GRC solution because as noted previously, each offering no matter how effective tends to favor one specific location within the GRC spectrum.

But even when you have a solution that’s broad enough to accommodate most of what you need to accomplish there’s the other driver coming into play, massive compliance requirements.  I’ve had clients who don’t even care so much about if what they need to do to comply makes sense for them but will do anything to pass an exam.  And so there’s this mad, Lemming-like dash in a single direction to shoehorn everything and anything into this thing called GRC that might even be remotely related.  There’s little thought put into how best to get the work done with the primary concern being “we have to have something to show the examiner”.  The result is a hodgepodge of seemingly related activities being coordinated under a single function or initiative but with almost zero effort made to try and normalize the workload and gain the efficiency’s that GRC promises.  How thoroughly depressing for us practitioners.

And it’s fantasy to think that once things are setup to be done a certain way they’ll ever change.  Unless an examiner or auditor tells you something needs to change everything stays the same.  So a poorly designed GRC function remains poorly designed forever.  And an unnecessary GRC activity continues because no one typically cares if you’re doing too much, only too little.  It’s almost like people just want to stuff everything remotely related to the discipline into the GRC closet and then make sure guests never open that door.

I know we’re still early in the GRC life cycle (Michael Rasmussen recently noted in an article that it’s been ten years since he first conceived of the acronym and concept) but what if this trend isn’t derailed sometime soon?  What if because of the weak economy (I’m being polite, I should swap “weak” for “horrible”) companies continue to just sweep everything under the GRC rug and don’t exploit the benefits of the concept?

I’m reminded of the old joke about the immigrant who decides he’s going to use his lumberjack skills in the U.S.A. to make a living and invests his life savings in a chainsaw.  After repeatedly failing to achieve any appreciable gains in his productivity he finally returns to the store to find out what’s wrong with the machine.  Once they pull the ripcord and fire it up he jumps back in surprise asking “what’s that noise”.  I have this image in my head of some internal controls manager managing his/her company’s GRC program ten years from now stumbling across an OCEG document, reading it and jumping back in surprise and exclaiming “what a great idea, why aren’t we doing this sort of thing”.  Don’t laugh, I can all but guarantee it’s going to happen at this rate.

Honestly?  Bombs on airplanes?  How could this even be possible?  Anyone who has traveled in the past decade knows all too well exactly how insane airport security has become.  I’ve had nail files broken off of nail clippers, toiletries confiscated, water bottles thrown away and have had to empty the contents of my laptop bags so often I wouldn’t dare even attempt to count the number of times.  But a bomb makes it through?

Sadly it’s the perfect example of why controls and their related activities aren’t nearly as effective as any of us would like to believe.  They’re a starting point and not much else.  Just like in any controlled environment, we try to identify as many risk factors as is possible and then design controls to either manage or mitigate them.  But risk factors continue to change, evolve, mature and move on.  And those who would exploit them to their advantage understand this and seek to identify the opportunities that are created in that gap between when they emerge and the world catches on to them.

It’s why compliance by itself is never enough.  It’s why risk assessments are vital to the security and soundness of your institution.  You can’t manage what you can’t measure and when it comes to risk factors the only way to measure them is via an assessment process.  Ever wonder why just about every piece of banking guidance makes reference to your “most recently completed risk assessment”?  And trust me, ignorance isn’t bliss; it’s a bloody nightmare when it comes to the financial domain.

While financial institutions don’t typically have to worry about bombs, they do need to understand threats presented by the ever shifting technology and business landscape.  They need to monitor their employees activities and assess risks presented either by newly emerging business practices (e.g. mobile banking) or growing dependencies on existing ones (e.g. ACH).  Waiting for your regulator to tell you what to do will definitely result in there being a gap that someone is poised to exploit.  Are you OK with that?  As a banking customer, I know I’m not.

The next day I received another message from him with a different link, thus confirming my earlier suspicions that something was amiss. After letting him know about the wayward messages, I started thinking about what had just happened.  This is someone who lives security every minute of every day.  He knows about every threat old and new, the tools and techniques to combat them and is one of those people I go to for advice when I don’t know where else to turn.  And his Facebook session was sending out phantom messages without his prior knowledge.  A little scary when you get right down to it.

But wait, it gets just a bit scarier for me.

Fresh on the heels of the Facebook incident, I came across an interview on a security website I visit now and again in which the interviewee offered his opinion that security threats from social media sites are greatly exaggerated.  Really?  Based on what?  Here I am having just been presented with evidence that the threats are real, swift and plentiful and I’m being told just days later that it’s really not that bad.  And why I’m writing about it here is because although the person being interviewed is not offered as a security expert, the website itself conveys a certain degree of legitimacy.  The opinion was followed up by a recommendation that if you’re concerned about the threats imbued in the use of these sites that you should simply not use them. Hmmm.  My takeaway from the interview boils downs to “security threats from social networks sites are not so bad” and “if you’re concerned about threats, don’t use them.”  So your choices are either ignorance or avoidance; nice.

I remember way back when Palm Pilots first became popular.  Corporate IT reacted by banning them, claiming it would be a support nightmare.  Not long afterward, the use of personal email became pervasive and people wanted to be able to access it from their work place.  Corporate IT reacted by blocking access to most common external email sites.  A short while later, USB storage devices started showing up and almost a minute later corporate IT reacted by, you guessed it, banning them.  Fast forward to 2010 and smart phones (the modern day equivalent of the Palm Pilot) are common place within corporate infrastructures, USB devices are allowed, and the demand for access to external emails has subsided quite a bit (thanks to the aforementioned smart phones).

Now the greatest threat presented by the most recent wrinkle in the ongoing evolution of technology is access to social media sites.  I keep reading articles and coming across polls exploring whether or not companies should allow access to Facebook and LinkedIn.  I’m wondering why anyone seems to think it’s optional.  Exactly which technological advance has corporate America successfully derailed since technology first landed on our desks 40 years ago?

Here’s my take on all of this:

• First, the threats presented by social networking sites like Facebook, MySpace, and LinkedIn are real.  Hackers were among the first to see the potential of these social networks and have quickly moved to take advantage.  I’m hammered on Twitter with suspicious links and receive odd communications via Facebook all the time.  And I consider it remarkably irresponsible for anyone remotely having to do with information security to claim anything else.
• Second, you’d better figure out how to safely manage use of social networks.  While I can make an intelligent argument why all but the professional social networks should be blocked by your Web filters, I’ve personally witnessed over the not quite two years I’ve been using Facebook that it’s fast becoming the most common way for people to keep in touch.  Accordingly, your users will continue to seek out ways to access their network of choice and bypass your controls.  So you have a choice: Try to stop the next advance in the digital evolution or figure out a way to manage it better.  But remember, historically telling users to not use something and trying to prevent them from doing so has proved to be a flawed and largely ineffective  strategy.
• Third, and this is a biggie: Educate your users on the types of threats they’re likely to encounter, how to identify them, and how to handle them when they appear.  Rather then spending all of your time trying to prevent this already entrenched advance in technology from being used, split off some of that time to prepare your user community on best practices.  And have rules in place so that if someone fails to follow them you retain the option to take action.

But every now and again I find a situation that makes me think that maybe, just maybe, an exception can be made.

In working with a client on implementing a compliance program, it became apparent that by adhering to the exact letter of the law specified within the documentation, they’d immediately be out of compliance on day one in a very large, obvious way.  Typically when dealing with such a situation, I advise the client to develop a schedule indicating the dates by which they expect to get all their work done and be fully compliant.  For vendor management, I usually recommend twelve months, for Red Flags it’s usually six months and for security awareness it’s three months.  As long as the plan and related schedule is documented and you can prove that you’re adhering to it, examiners and auditors alike will usually give you a free pass until the next time around.

Even so, in this instance nearly half of all the in-scope work would be displayed as overdue right up front.  No one wants to see that on a screen or in a report, no one wants to risk having senior management see that information and absolutely no one ever wants to explain to an examiner/auditor why they have so much work still to do (even with a solid explanation and plan).

And so I blinked.  I considered in this instance a way to introduce a new rule that would allow the client to theoretically use my approach of scheduling all the work to be completed within a set time frame (twelve months in this case) but wouldn’t have to show anything as being overdue.  It didn’t seem so much like the right thing as much as the kind thing to do.  I even went so far as to scope out my idea in writing and share it with my fellow compliance experts in our practice.

As it turns out, I apparently have had an influence in how all of us view such matters because the first question I was asked was what would I do if I was managing the program.  I wouldn’t come up with any special rules to avoid being accurate and honest, that’s for certain; it is what it is.  I was then asked if I was willing to bend the rules in other projects, say like an audit for example.  Well considering I’ve excused myself from audits in the past because management (at previous companies) elected to remove findings or soften them in order to keep the clients happy I knew the answer was a resounding “no.”  So I was asked why I was looking to bend the rules now.  Good point.

What audit and compliance practitioners have to do is often unpopular and sometimes very difficult.  We’re often perceived as inflexible or unreasonable.  But the truth is that your compliance and/or controls framework is only as effective as its weakest link; if you start making exceptions in one area it quickly becomes expected in others.  Once one control is weakened in exchange for making things easier or more palatable, the integrity of the whole enchilada suffers.

Compliance requires hard decisions, thick skin and consistency.  If you’re more inclined to be affected by acceptance rather than respect, it may not be the right line of work for you.  Or as I’m fond of saying, it requires that you’d rather be right than popular.

One of the key considerations in sorting through the irony that’s my place in this world is that I’m nothing like the auditors I used to deal with in my application development days on Wall Street. What I audit, how I examine related controls and activities and review supporting evidence is heavily biased by my first-hand knowledge of the IT infrastructure. I understand technology and how it’s used, and so when I’m conducting fieldwork, I’m able to see things from a blended perspective. Most of the auditors I dealt with understood audit way better than they understood technology and so they’d ask question after question, not really knowing if the answers made sense, only if they matched expected results. For me, if the answer doesn’t make sense or is the wrong one, I immediately switch gears and seek out compensating controls because they’re often there if you know where to look.

Audit is heavy on my mind this week because I’m in the process of wrapping up a report for a client about the exit meeting. It’s interesting how the names and faces change from engagement to engagement but the script rarely varies. You’d think it would get old or boring but curiously it never does. The client never likes to see anything negative in print and it usually sets off a flurry of activity from report issuance to the first review meeting. There are almost always a series of requests to move things around, change the way things are worded and occasionally to reevaluate ratings. And I can’t recall a single audit where additional evidence wasn’t submitted for review after the initial draft was distributed to offset findings – artifacts that often have that “new car” sort of smell. But that’s actually a good thing and I’ll explain why.

An auditor’s job is to find control gaps and weaknesses. I’ve often compared what we do to fishing: You cast your line, see what you can catch, and keep at it until you either fill up your basket or have exhausted all available time and resources. Sometimes the bounty is rich and sometimes not so much. But there are always things to catch (I’ve never been shut out yet) even in the very best managed IT shops. The payout for the auditor is to identify legitimate issues that resonate with client. You want for those who own the controls to understand what the issues are and take swift action to remediate. I know some auditors take offense to after-the-fact evidence being provided because they perceive it as if though it’s implied that they missed something. Not me. When the client comes back quickly with viable solutions to make the findings go away, I consider that a bonus even if they didn’t exist a week earlier. That means that real risk is being further mitigated and managed and that’s the only reason to ever conduct an audit, in my opinion.

The client I’m working with, as it turns out, has fast become a favorite of mine. They’ve made great strides over the past year or so in enhancing their security posture and have gone a very long way towards putting in place effective controls to protect themselves, which ultimately results in their better protecting their customers. They take this sort of thing very seriously and as such, they have earned my respect. So when they come back to me with newly available information to offset findings in the draft report I’m happy to factor that into my findings. I did my job, they did theirs and in the end, the world is a little more secure.

So I guess I’m a minority on a couple of fronts: I’m more than satisfied with my job and I’m an IT auditor who genuinely understands the technology infrastructure. So much for there being strength in numbers.

]]> http://itknowledgeexchange.techtarget.com/regulatory-compliance/the-best-part-of-audit-yes-i-said-audit/feed/ 0