Now I’ve designed and supported more than my fair share of related content.  I understand that sometimes the best way to tell a story is to paint it in the form of a picture; I get that part.  But way too many times I’ve witnessed such initiatives spiral out of control to the point where it takes an army of people working ridiculous hours to pull together a deck of metrics that either fails to answer anyone’s questions or, even worse, generates requests for more metrics to provide clarity.  And once a metric becomes a standard part of any reporting package it often stays there until management changes, and sometimes even beyond.

But I think there’s a bigger issue with metrics that exceeds my simply not thinking they’re “all that and a bag of chips”.  Where are the controls around generating them?

Seriously, we have this vastly complex framework wrapped around financial reporting (SOX) to provide reasonable assurances that what management is reporting to its investors is accurate.  We have industry, federal and state legislation requiring all manner of controls around sensitive information.  There are auditors (internal and external) and regulators from all over the place that comb over everything with a fine tooth comb (or at least claim to) to make sure everything being done is done right – but in my nearly fifteen years in the audit/assurance industry I have never heard of a finding or issue regarding the veracity of metrics.  Which is only a problem if the people running an institution or company rely on them to make decisions.

The reason why it’s a problem is because so much of the metrics out in circulation is pulled together from disparate sources, cobbled together in spreadsheets or non-production databases and manually generated.  There’s no easy way to verify the source data, or know that it’s unaltered in any way or even know if it’s the right information.  And even if the data source used is from a secured production-like environment, still there’s no real auditing conducted to ensure the information is accurate or better yet, is even the right information.

I once took over a change management process and assumed responsibility for a series of reports that were generated for the Managing Director who in turn used that as part of his reporting package shared with the CIO.  One of the key metrics being reported on was scheduled releases and the IT departments on-time implementation percentage.  The numbers looked great showing that they were on-time more than ninety-five percent of the time over a two year period.  The only problem I could see with the metric was that it was misleading to the point where it was almost a lie.  The scheduled release date was being pulled from the system used to migrate changes into production and that date was only determined once the development team had completed all of their work.  So the scheduled implementation date was chosen once they knew they were ready to move into production.   Of course the on-time numbers looked great, they always knew they were ready before committing to it.  The Managing Director incorrectly assumed that there was a legitimate release schedule with forecasted dates and that the on-time numbers reflected on a well run process; wrong.   No one ever questioned the numbers or their source and had I not inserted myself into what was described as a well honed, efficient process the problem might have never been identified; and there a few more just like it.  My trust in metrics was permanently altered after that.

Metrics represents an excellent way for decision makers to quickly understand status and identify problems.  I’ve quoted here before about how someone I respect quite a bit was fond of asking her team “If you can’t measure it, how can you manage it” and she’s absolutely right.  Metrics is the ultimate management means of measuring key activities and issues within their world.  But how far do you go and how much effort do you expend pulling the related reports together?  And even if you’re able to automate the process and shorten the time necessary to generate the reports, how do you know that you’re either measuring the right things or that the underlying data is unaltered?  Ultimately I think that senior managers should be provided with something akin to a cost-benefit analysis for each metric they’re given.  Have them understand the degree of complexity and the amount of effort required to generate a number before deciding whether or not it’s worth it.  Perhaps I’m being naive but I’d like to think that most C-level executives would eliminate a significant amount of their reporting if they could see how much it was really costing them.

Here’s the part that should really concern you the most though: Metrics is a key component of Board reporting, they make all sorts of decisions based on what these reports tell them.  How can that be allowed unless the process used to generate them is locked down and audited?  Where are the regulators in all of this?

Of course I don’t know anyone by the name Rebecca Keen and knew instantly that it was a phishing scam. It’s not the email by itself that made this a blog-worthy item.  What made Rebecca’s email this week’s topic was the reaction of someone close to me and their attitude about how to handle it.

At the risk of embarrassing anyone, I won’t go into specifics as to who the person is, but when I told them about the email as a way of educating them on how to identify and manage phishing attempts, they asked me how I knew it wasn’t legitimate.  Beyond the obvious fact that I don’t know now and have never known anyone by that name I’m not sure what else I’d need as proof this was a scam.

Here was the ensuing exchange:

“That may be true but what if they sent you the email by accident?  What if they misspelled the email address?”

To which I replied, “Still not my problem and I won’t respond because that establishes a dialogue which will only encourage the person further.”

“But shouldn’t you at least let the person know they reached the wrong person,” I was asked with a tinge of real concern.

“If I reply, that will send the message that they reached the right person. They’ll think I care, which will only open me up to additional pressures from the scammer”.

“People are so mistrusting these days.  I’d at least want to make sure this wasn’t someone who needed my help”.

And therein lies the problem: Despite this being a very obvious phishing attempt, it was only obvious to me. Despite the endless stories about people being exploited and robbed by an endless array of online and email scams, there are still people who respond favorably to these sort of things because of their basic decency. The person to whom I was talking wasn’t lacking in intelligence and isn’t typically naive, but when presented with these situations uses a different set of rules.

To make matters worse, the email from Rebecca Keen was properly formatted without spelling errors and actually looked like something I might receive from a legitimate source.  As a matter of fact, it presented itself so well that I actually opened it, which is a step further along than these things usually get.  But of course I knew instantly that it was just the latest example of how people are using the Internet to try and steal money.  And while the scam was obvious to me, there is at least one person I know who might actually have taken action upon receiving something similar.

You know what occurred to me today?  The reason that scammers continue to send out phishing emails is because they still generate the desired results.  Despite the endless marketing campaigns by a wide range of financial institutions to educate online users, there are still a large enough number of people who are victims waiting to happen.   And as long as even one person responds favorably to a phishing campaign, it’s considered a success.

I’m thinking that as a former New Yawker I should create a program for the FDIC based on my experiences growing up in New York City.

• Do not engage in any dialogue with anyone you don’t know about money in an unusual or inappropriate setting e.g. street corners, subway platforms, etc.
• If someone is selling something, offering to buy something or trying to distract you somehow when in an unusual or inappropriate setting (e.g. stopping you on the street, walking up to your table at a restaurant, etc.) immediately disengage and continue on your way or return to what you were doing without allowing the conversation to develop and/or continue.
• And if at any time your instincts tell you that something is wrong, amiss, out of place or odd err on the side of caution and do everything and anything to remove yourself from that situation.

P.T. Barnum was often credited with having said that “There’s a sucker born every minute” and apparently online there are somewhere between two and too many scammers waiting to take ‘em.

P.S. As I was about to publish this post I received an email update from Rebecca Keen letting me know that someone temporarily stole her email account and that there’s no emergency whatsoever.   Glad to hear it but I still have no clue who she is.

Right there on the front page of today’s News and Observer was a story about how a recent audit claimed corruption at a local college (North Carolina Central University).  I’m sort of trained in a Pavlovian sort of way to notice anything having to do with audit and so I gave it a cursory read.  Cursory turned into focused when I reached the part about how the school’s chancellor Charlie Nelms called the report draft “sloppy” and went on to say that some of its harshest accusations might not be true.

One of the oldest tricks in the business book when it comes to audits is to start questioning the quality and veracity of reports that are perceived as not being favorable.  Instead of focusing on the audits findings and trying to validate them (because a good audit is your best friend if you really want to do things right) the auditee goes into a series of tactical maneuvers to deflect attention away from the report’s contents and feigns disgust and outrage.

The school chancellor went on to say that, after firing the auditor who produced the report, he “ordered his staff to gather more information before he releases a final version to the public.”   He went on to say that the “draft audit was so poor that he doesn’t trust it, and he does not want to damage the reputations of people who might not have done anything wrong”.

A few years ago, I conducted a risk assessment for a client with an odd configuration of infrastructure pieces that clearly defied anything close to typical, so it was difficult to measure them against the norm.  Just the same, I tried.  I took a step back after conducting all of my interviews and gathering as much information as was available and filtered it through the lenses of an examiner.  I surfaced gaps and issues that were likely to be viewed in a negative light, explained why that was and offered clear and concise remedial steps.  Senior management went bonkers (for lack of a better word) when they received the report.

They were outraged because the report was delivered a week late (which was true), they were insulted that there were typos (not factual errors, just a few grammatical/spelling hiccups which are common in draft versions) and charged that some of the issues listed were completely false.  In summary, they called into question the accuracy and reliability of the entire report.  It was startling for me because in my more than two decades working in the business world with more than 10 years conducting audits and assessment, I’d never had a client react anywhere near this way before.

But it was really more about using diversionary tactics intended to gain a negotiating advantage.  Their end game was to soften the report’s contents so that it looked better when the examiners came back around; by pushing us back into a defensive position, they were almost successful.  Fortunately, I’m stubborn when it comes to standing behind my findings and need incontrovertible proof that I was wrong about something before changing or removing things.  I may not be the best auditor but I have well honed instincts around IT, the myriad processes necessary to support the infrastructure, and I know good from bad.  I never put anything into my reports that doesn’t resonate with me and my peers (and typically the report’s audience).

So you can imagine where my head was at while reading the story today.  Mr. Nelms also said, “I want to see the source documents, and I want to see the field notes from the audit, because I want it to be accurate.  I don’t want it to be hearsay, because some of the allegations are just mind-boggling.”

Well that’s good to hear because any audit worth its weight in paper needs to be supported by solid work papers.  But considering that he fired the auditor, I’m hoping someone in his office thought to secure that beforehand.  And I’d need to understand why he’s gathering more information when all he really needs to do is use the work papers and have another independent auditor re-perform the tests.

Oh and another thing, who hired the auditor to begin with?

Also, now that the report’s findings are semi-public (it’s available despite not having been formally released), where’s the value in conducting a follow-up audit?  Anyone involved with any alleged wrongdoings now has a clear roadmap in front of them on how to cover their tracks.

Here’s my thinking on all of this: The audit is likely somewhere close to 100% accurate but far from perfect (I know that’s a contradiction).  If the chancellor was really interested in handling this properly, he’d quietly set about having independent people digging into the findings, not as a CYA exercise but simply to get to the bottom of things and deal with whatever is found.  I’m not saying that where there’s smoke there’s always fire but unless Mr. Nelms can offer a credible explanation why he would think that the fired auditor would fabricate stories or offer poorly formed conclusions I’d have no choice but to question his position on all of this.  I guess what I’m asking for is a credible explanation as to where the smoke is coming from and an explanation why he thinks it’s benign.

What I’d like is to hear the auditor’s side of the story.  I’m betting that would be an enlightening conversation.  But if Mr. Nelms was successful in his very public tongue lashing of this auditor, he/she will do anything and everything to avoid having their name outed.  And so the diversionary tactics score another point.

And the best part of this?  I almost never read the paper.

What is new is the ambivalence we’re experiencing from management.  It seems that a little known byproduct of our currently sad economic state is that keeping the doors open seems to be the only goal that really matters.  Management is not particularly concerned with much else, or so it would seem.  Not that this by itself is a new phenomenon either but there’s almost a reckless undertone emerging.

We’ve encountered some glaring issues recently that underscore a fundamental problem that I’ve struggled with for a long time: The FDIC and NCUA examiners just don’t pay enough attention to IT-based risks.  In some instances they touch on high-level issues and in rare instances can get a bit more granular, but we’ve collected empirical evidence that an in-depth review hasn’t been conducted for the vast majority of institutions that we’ve worked with.

Forget about industry best practices and forget about the fact that financial institutions are required by law to implement and maintain certain basic safeguards.  We live in an age where identify theft and credit card fraud are rampant.  Every day we are presented with more stories, more guidance and more information about how the criminal element is finding newer and more insidious ways to get at our money and credit.  My senior citizen mother and my grade-school aged children are all aware of the term phishing, have all been coached as to which email is safe to open versus which isn’t and know not to share personal information.  If I can convince them of the threats out there in the great digital void you have to think it’s fairly obvious, right?

So why is it that the examiners aren’t paying more attention to the IT infrastructure?  I had a chance to ask someone from the NCUA office a few months back that very question and while I didn’t like his answer, it made sense particularly considering the more pressing issues banks and credit unions are currently dealing with.  It comes down to resource availability.  Only so many hours are allocated to an exam  based on their size.  And so for the smaller institutions, the examiners prioritize the work based on risk.  Can anyone argue that scrutinizing the books is less important than auditing the IT infrastructure?

Even so, some of the institutions we’ve worked with and which I’ve personally reviewed have had issues for what has to be several years.  How is it possible that in the past five years not one examiner has ever noticed the absence of a business continuity plan?  Or any form of security around the firewall (and an unusually permissive firewall at that)?  Or the lack of strong (or even reasonable)  password controls?

Something has to give.  When you combine the lack of proper examiner supervision with a less than concerned management mindset the potential for serious issues becomes much greater (and likely).  Somehow the various entities that are responsible for providing oversight for those places we trust with our money need to figure out a way to provide reasonable assurances that at least the bare minimums are being met when it comes to IT controls.  With all the money being spent to keep the banking industry afloat can’t someone figure out a way to slice off a little bit in order to hire enough IT people to conduct the necessary examinations?  Congressman?  Senator?  Mr. President?  Anyone?