Posted by: David Bressler
Analysis, Big Data, BigMemory, Financial services, In-Genius, in-memory, In-memory data management, in-memory intelligent action, market surveillance, real-time, Real-time actionable intelligence, Real-time analytics, Real-time Big Data
Human behavior fascinates me. Especially into today’s workplace, where people are asked to do more with less, yet everyone is surprised when there are gaps in what gets delivered.
The intersection between human behavior and technology is even more fascinating. Is there another area of life where so much of how things work stated absolutely, when it’s really just ‘opinion masquerading as fact obfuscated by marketing’?
Forgive me for taking a customer’s point of view, but when my bank denies me a small credit increase to get me through a business trip, but in the same week gives me a whole new line of credit for $20,000… what do you think that experience does for my confidence that they’re properly surveilling their own trading operations for compliance?
Why’s it so hard?
Well, it’s not the technology. It’s the will to do something correctly. Past all the biases, past all the marketing, past all the default decisions… really rethink what you’re trying to accomplish, and how to really make it work for your business.
It’s a human problem (you might call it a political problem within the organization). It’s a skills problem, the skill being problem-solving not programming language. People like to reuse what they know to solve problems. What if we actually think differently about the problem?
What if, instead of the “put all the data we need in a database (because that’s what one does with data, isn’t it?) and run queries against it (of course we’ll run queries, is that how one gets value out of data in a database?)”, you decide on a different way of doing things?
Getting data into a database implies latency. Once data’s in a database, it’s hard to work manipulate. It’s hard to enrich the data with additional sources. It’s hard to work with time (harder than it needs to be). It’s hard to personalize data visualization at scale.
But that’s what one does with data. We put it in a database. Oh, and if there’s a lot of data, we call it a data warehouse (where things really go off a cliff, and not in a good way).
If we all just stopped for a second, we’d realize this sort of architecture doesn’t work. Well, we probably have. That’s why we’ve gone out to buy off-the-shelf software to solve market surveillance (and fraud, and money laundering) problems… in exchange for data integration and transparency/customizability difficulties (not to mention bringing another proprietary platform into the bank).
And, given the assumptions we’ve made about our databases, it makes perfect sense to bring in multiple off-the-shelf products, each for a different facet of the same problem – there are fraud management solutions, money laundering solutions, along with the different components of global market surveillance. I’d even argue that data privacy and one-customer-view solutions are subsets of the same class of problem.
That class of problem being… the ability to sense your business in real time, respond as appropriate, and gain insight that enables you to optimize business operations in order to achieve competitive advantage, while complying both with the laws governing your business and the intent behind those laws. (By the way, based on a call I just got off regarding healthcare, this same class of problem is prevalent in healthcare too, another regulated environment plagued by a lack of transparency and insight into business execution.)
How should you approach this class of problem?
Change your perspective about what’s possible by keeping the following in mind:
“Instead of throwing data in a database and running rules against the data, think instead of running the data against the rules as the data flows through your enterprise.”
With this perspective, you’ll benefit greatly:
- The data is already in motion, this perspective eliminates latency.
- It eliminates scalability challenges, because the database bottleneck is removed.
- Native time-based rules solve the difficulties of event-driven responsiveness to time-based patterns.
- Increasing the temporal window is simply a matter of putting more information in memory (as is adding new data sources and data enrichment).
- You’ll have better insight into “what’s running where”, preventing rogue algorithms from destroying the business.
- You have a single platform where you can execute all different forms of “market surveillance, fraud management, data privacy, and one-customer initiatives” enriching each together as the platform grows, rather than building in silos and failing to capture cross-area insights and optimizations.