A guest blog by Matt Jones, Analytics Strategist at Tessella
Big tech vendors have been piling into analytics and now AI. IBM’s Watson has been charming the media with disease diagnosis and Jeopardy prowess, and Palantir has been finding terrorists. Other major vendors such as Oracle and SAP have also been joining the scene, albeit with slightly less fanfare.
These companies have been leading tech innovation for years; one would expect them to offer a credible AI solution, and indeed their technology is good.
But the black box approach that many such companies currently offer presents problems. The first is that analytics is not a plug and play solution; it needs to be built with an understanding of data and context.
The second is that, in buying a black box solution, you lose control of your data. This means you are not sure what it’s telling you, you allow others to benefit from it for free, and you may not be able to access it in future.
And now you may even be in for a big bill for the privilege. A recent court case ruled that drinks company Diageo had to pay SAP additional licences for all customers that indirectly benefitted from SAP software used in their organisation, costs that could run to £55m.
If upheld, this could have a chilling effect on the analytics platform industry. Could every beneficiary of data insights across and beyond your organisation now need a licence? Companies should now think a lot harder before committing to a platform on which their whole business relies.
Your data is your company’s lifeblood – value it
Even before this case, it’s stunning how much control of data companies were willing to give up to vendors, and how little thought goes into the consequences of over reliance on one technology.
Vendor lock-in is nothing new of course, consumers have been allowing Apple and Google to track their every move, and global businesses putting everything in the Microsoft cloud, for years.
But data and AI represent this problem on steroids. Data projects done right are embedded throughout the entire organisation. Some solutions will suck in all your data from every system, lock it away, and even refuse you access to it. So these platforms have all your data, the context, and the insights it provides. And now they have even more power to charge you to benefit from it.
How to plan for an analytics solutions
So, how can you benefit from data analytics without storing up future problems?
Before you even think about technology, get the right mix of technical and business people to look at what your business needs to achieve and how data can help you. Then get people in with data science expertise who can explore how your data can be used to support those needs.
Only then should you look at what platforms you need to achieve this. The most powerful is not necessarily the most suitable, find one suited to your need. In doing so, consider licensing models and demands from your data – does it leave your site? Can you access it when you need to? Look at the company’s overall culture – are they transparent or opaque? This will guide you in how they are likely to handle your data.
Perhaps more important is to consider whether you need a black box at all. Google, Microsoft and Facebook, amongst others, all offer openly available Artificial Intelligence (AI) APIs on which anyone can build bespoke AI or machine learning platforms – which are as sophisticated as any black box on the market. Furthermore this allows you complete control and transparency of how the data is fed in, processed and presented, so you can identify causal links between data and outcomes, rather than having to trust someone else’s insights into your business are correct.
If you do need a black box solution – and there are times when they are the right option – you should ask whether the vendor is a partner or just a platform. Do they understand your business context? Do they integrate with your particular data setup? Do they leave you with control of your data? Do they make the data analysis process clear, so you can understand whether your business insight is based on a causal links or just an unsupported pattern spotted in the data.
The approach you take should be driven by the most appropriate approach to solving your challenge or finding the insight you require to make better decisions – not by the platform itself – and it should consider what level of data control and oversight you are willing to give up. Once you have properly defined that, then you can make the best decision about how to use your data to meet those goals.
This is a guest blogpost by Stuart Wells, Chief Technology Officer, FICO.
Recent research by Industry Week found that 73% of manufacturing companies recognise the potential benefits of successful supply chain optimisation (SCO) projects. The same report highlights that while manufacturers have been investing heavily in SCO projects for a long time, and continue to do so, not all of these projects achieve the return on investment they potentially could.
Even though SCO has been around for years, manufacturing firms still struggle to implement optimisation effectively. Each organisation faces it own challenges carrying out those projects, but there are two key issues that most manufacturers share: data swamps and siloed legacy systems.
Data and integration: the key challenges
The ‘Big Data’ hype pushed firms to invest in storage tools and solutions that quickly collected as much data as possible. However, very few businesses thought ahead, and having a lot of data without a plan for how to use it does not solve any issues, or add any business value.
Manufacturing firms are also struggling to find off-the-shelf applications that fit their existing environment. Ripping and replacing all current tools and platforms is not an option, as many contain critical business data. Many manufacturers struggle with data dispersed across disconnected systems and platforms. This limits their ability to optimise business and process decisions.
Over 85% of organisations tell Gartner that they are unable to exploit Big Data for competitive advantage, while at the same time 95% of IT decision makers expect Big Data volumes and the number of data sources they use to grow further. There is a critical industry need to evolve from classic data management to an approach that leverages information and knowledge across all platforms and systems.
Shell and Honeywell: how optimisation paved the way to success
One company that has taken this productive approach and implemented a successful optimisation project is Honeywell Process Solutions. In many oil-refineries, and other companies in the continuous process industries, the production schedule is created through a surprisingly low-tech approach; humans working with spreadsheets. Manual scheduling is restricted by the analytic limitations of the human brain, which is prone to error, so Honeywell Process Solutions developed an analytics-powered optimisation tool.
The company used mathematical algorithms to analyse hundreds of variables in a short time to determine the best solution out of many thousands of possible scheduling scenarios. This has driven significant economic impact. For example, the downstream effect of scheduling demand-driven production of 100,000 barrels equates to an annual profit increase of more than £2.3 million.
Optimisation is not only useful in production scheduling. Shell, a global group of energy and petrochemicals companies, deployed powerful optimisation tools to improve asset utilisation and maintenance requirements of the chemical plants while improving plant stability and profitability. Shell has nearly 600 advanced-process control applications around the globe, with each processing plant using its own set to run plant operations. It needed a solution that would provide plant-wide control, and help maximise the economic operating benefits for each plant.
For Shell, it was all about making the plants as safe as possible. The team at Shell deployed a platform that runs calculations in real-time by balancing numerous constraints and objectives. After the system calculates the best actions, the onsite team interacts with a visual tool that gives them the control and flexibility to explore trade-offs and make the best possible decisions for the plant. The tool provides recommended actions for every situation, as well as the optimal values for flows and temperatures, which keeps the plant and its crew safe.
Innovate to stay in the game
Manufacturers who use data and analytics in innovative ways will be ahead of their competitors, according to the Centre for Data Innovation. The application of analytics can help many business areas gain significant advantage, in particular when it is used for optimisation as analytic models help organisations to reduce inefficiencies.
Optimisation can help to enhance your workforce and line scheduling, better mix products to meet quality standards, improve production planning, enhance truck loading, optimise production across plants, and rebalance inventory. Like Honeywell and Shell, it is important that manufacturers embrace advances in technology and innovate if they are to thrive in the data-driven marketplace.
This is a guest blog post by Bhavin Turakhia, Founder and CEO of Flock
Today, collaboration is a key to a company’s success. It is therefore critical to understand that collaboration itself can be a roadblock to optimum productivity. This has always been the case. For years, we have spent our time and energy on not-so-productive collaborative activities such as emails and meetings. For example, meetings have been known to consume a large share of our working hours, in many cases 40-50% of the workday.
In an environment where organisations are increasingly going global and teams are dispersed across time zones, teammates are required to be far more agile and quick when responding to one another to complete tasks in a timely manner. Email isn’t the right tool for employees needing instant responses from their global teammates, and thus, team messaging apps have emerged as a collaboration tool that’s much more conducive to productivity. Team messengers deliver exactly what most organisations need to stay relevant and competitive in today’s business landscape — faster sharing of information, increased efficiency, and the ability to make quicker decisions.
In fact, team messengers are touted as the collaboration tool of choice for the workplace of the future. No wonder some of the biggest companies in the world have jumped into the enterprise and team messaging chat space.
However, one of the most common concerns around the use of team messengers is whether they are really helping us become more productive or doing just the opposite by way of regular interruptions to our workflow. So should we stop collaborating to keep ourselves more productive at work? Of course not! Without collaboration, a business will stop working like a well-oiled machine. The answer lies in collaborating more effectively and ensuring that team messengers are used in a way that optimises productivity.
Eliminating workplace communication challenges
One of the commonly cited problems with the use of team messengers is that when conversations flow in real time, it’s difficult to pretend that a message went unnoticed (and therefore, un-responded to) as opposed to emails, which most people do not feel obligated to respond to immediately. In the world of fast, real-time communications, not responding immediately is unacceptable, and worse, it can even be perceived as rudeness — just like ending a face-to-face conversation abruptly. However, team messengers help workers tackle this problem by allowing them to create a time blocking schedule through a simple ‘Do Not Disturb’ feature and notify teams they’re part of that they do not wish to be disturbed at that time. This way, if they receive a message during this ‘blocked out’ time, they can delay their response without offending anyone.
One of the main reasons why people remain glued to their messaging apps or emails— even when messages do not involve them directly — is the fear of missing something important. Doing so hurts their productivity to a great extent. This is another area in which certain team messengers emerge as a better way to coordinate and communicate with colleagues.
Enterprise messaging apps allow users to simultaneously be part of multiple teams. They can access all conversations that concern them from a single interface and set notifications for critical projects or discussions. At the same time, they can also mute less important conversations or groups that they can respond to at leisure, for example the office football group. Or they can simply activate the ‘do not disturb’ mode. These features stop unnecessary messages diverting users’ attention and allow them to focus on their most critical tasks.
Blocking distractions: using team messengers effectively
Unlike emails, that are both interruptive and slow in response, team messengers not only help users increase their productivity, but also help users work more efficiently by offering time-saving features such as sharable to-do lists, file sharing, team polls and so much more.
Team messaging apps also offer an effective platform for teams to share knowledge that often gets lost among big teams. This is a major plus considering that the average knowledge worker experiences a 20% productivity loss while looking for company information or colleagues who can help with specific tasks.
Unlike other traditional forms of collaboration such as email, team messengers allow various app integrations. For example, Google Drive can be integrated within a team messenger, allowing users to collaborate on documents while accessing their messages within one place. Interestingly, businesses can also build and integrate customised apps into their team messenger, to suit their unique needs. For example, companies can customise the Meeting scheduler app that helps users schedule meetings, invite participants, get participant feedback on meeting slots, and view team calendars at a glance – all from within their team messenger.
Thus, when used properly, team messengers are not a productivity killer. Their features enable users to disconnect from the world at large when they want to focus on work, and to be notified if a message concerns something urgent and collaborate in real time when they need to. Some tools also allow users to pin their most important conversations to the top of their chat window, so that they can filter chats by level of importance. This way, nothing important gets missed.
As most users will attest to, interruptions are a necessary evil when it comes to collaboration. Therefore, it is important for employees across teams and functions to learn to balance collaboration with task management. When used properly, team messengers enable employees to manage their time while collaborating with their teammates seamlessly, in real time. In this way, collaboration tools can be seen as a solution for the productivity vs. collaboration conundrum.
This is a guest blogpost by Gal Horvitz, CEO, PNMsoft
Artificial Intelligence (AI) refers to a wide variety of algorithms and methodologies that enable software to improve its performance over time as it obtains more data. This technology is currently the hottest thing in the business process management (BPM) industry and it is continuing to heat up. In fact, it’s on fire!
But, the funny thing is that although we see a lot of innovation with AI algorithms and methods, the concept of AI isn’t new. What is really new is how businesses are using AI. We like to call it the new generation of AI — one example is deep reasoning —which breaks AI’s traditional dependency on known datasets. Deep reasoning performs unsupervised learning from large unlabeled datasets to reason in a way that can be applied much more broadly. In other words, AI can “learn to learn” for itself.
Forrester Analyst Rob Koplowitz’s February 2017 report, “Artificial Intelligence Revitalizes BPM,” – where PNMsoft was one of the companies interviewed – explains that:
“The primary driver for BPM investments just two years ago was cost reduction through process optimization. Today it is customer experience, with enterprises expecting to put top priority on digital automation in two years.”
AI has the ability to take human cost and latency out of processes, as well as provide new interfaces that customers enjoy. With faster and more user-friendly operations, customers are happier, stay loyal to the business and are more favorable to buy more products and/or services from their current provider.
That’s why it’s no surprise that customer experience (CX) and business transformation are expected to skyrocket to the top two primary focuses of businesses looking to improve their processes.
But, in order to progress the practice of AI, companies must first feed the system initial data for the AI algorithms to analyze and suggest data-driven improvements from deep reasoning. Forrester reports 74% of firms say they want to be “data-driven,” but only 29% are actually successful at connecting analytics to action. If these companies want to drive positive business outcomes from their data, they must have actionable insights. Enterprises are realizing this is the missing link and have begun to invest in and grow large sandboxes of data sets that will ultimately help build the AI algorithms that can inspire significant digital transformation for their businesses.
We agree with Brent Dykes’, director of data strategy at Domo, theory that there are many key attributes of actionable insight. We will break them down for you so that you can put them into motion and begin to build the infrastructure needed to put AI and BPM to work.
- Alignment – Make sure the data you’re gathering directly feeds into the success metrics and key performance indicators (KPIs) you desire.
- Context – Determine why you need the data in the first place. Do you need it to make comparisons or benchmark your success? Context will enable your AI to make more accurate predictions.
- Relevance – Pulling the right data at the right time will help narrow down which actions need to occur and when.
- Specificity – The more specific the data is, the better sense of the next action to take will be.
- Novelty – When analyzing customer behavior, it is easier to spot a one-time occurrence over something that has repeatedly happened. Novelty occurrences are areas to pay close attention to and AI will be able to pinpoint them quickly.
- Clarity – How the data is communicated can say whether it can be acted on or not. If the data is not communicated well, it can be lost in translation.
AI or machine-learning technology can bring huge benefits to any industry. Here’s an example of a healthcare organization that has adopted data-driven AI and has connected analytics to action.
- By analyzing large amounts of medical data, the healthcare organization’s AI is helping clinicians give faster and more accurate treatment to their patients, and has the ability to learn to make better decisions going forward. For patients, the AI-driven healthcare system alleviates some of the burdens on a system struggling to keep up with ever-growing demand. By implementing these technologies, the organization can make better health decisions, diagnose disease and other health risks earlier, avoid expensive procedures, and help their patients live longer- which are all actionable insights driven from data and analytics.
Moving forward, we will continue to see companies adopt new technologies, like AI, as a means to improve their bottom lines and their efficiencies.
This is a guest blogpost by Kirk Krappe, author, CEO and chairman of Apttus
The thing about technology evolution that the movies don’t quite tell you is that it desperately relies on adoption. The most amazing invention on the planet could be created and if there’s not widespread consumer demand, or if it’s prohibitively expensive for businesses, that breakthrough isn’t going to actually break through. The old Henry Ford quote is more relevant than ever: “If I had asked people what they wanted, they would have said faster horses.” However, by demonstrating the value and efficiency of new tools, we can push modern business forward exponentially – unignorably.
To wit, the most exciting thing going on right now is the rapid acceleration of the machine learning space, which often gets associated with artificial intelligence and advances in bot technologies. Here we have a topic that has been researched, built out, and implemented for many years now, but which is finally breaking through into mainstream usage. Why is this? We can look principally to two related advancements in the field.
- Machine learning is more practical, accessible and applicable than ever before.
- As a direct result, it is more affordable and useful than it’s ever been, increasing its adoption and making it part of mainstream business.
If you’re looking to demonstrate value, the most direct example is that of sales. Particularly in the enterprise world, where many transactions rank in the six, seven, eight figure range, getting an extra bit of speed and efficiency into the equation makes a tremendous difference. Now consider this: CRM systems and other financial tools employed by enterprise companies are still relying on manual input from their sales teams to pick out the best products, the most appropriate discounts, and perfect pricing. Machine learning has changed the game here. Accessing historical and predictive data, the tools to create a perfect sale, optimized for both the salesperson and their customer, are at their fingertips. Product recommendations aren’t necessarily relying on either party knowing everything about their business – that’s how opportunities are missed on both sides. Geo-specific, industry-specific, and even future needs can be acknowledged and addressed in moments, creating a better experience for everyone involved.
Think about how that might affect a business – if every deal had a safety net ensuring that it addressed every need of the customer, and maximized the deal for a company, all while speeding up the entire cycle, that makes the entire sales process significantly faster and cheaper. It creates business advantages faster. In the B2B world, it takes a notoriously slow-moving operation and turns it into an asset – then passes along that advantage to an entire new ecosystem of partners and customers. One adoption benefits everyone who touches the business. Slowly but surely, the technology revolution continues on until it’s unquestionably part of the mainstream.
This is how we experience breakthroughs on a global scale; they blatantly improve the way we, and our businesses, perform in our daily lives. It’s not a flying car or a lightsaber (yet), but it’s incredibly exciting all the same.
This is a guest blog by Ben Calnan, head of the smart cities practice at people movement consultancy Movement Strategies
Autonomous vehicles will soon be a reality – in fact, industry commentators believe that the first fully autonomous cars will be available to the UK public in five years’ time and perhaps sooner in other countries. However, for car manufacturers, the transition from prototype development to mainstream deployment is full of challenges. While navigating the issues surrounding data ownership will prove difficult, for those who successfully establish a role in the AV data eco-system, commercial opportunity awaits.
For driverless cars to work effectively in the real world, they’ll have to integrate with existing infrastructure and transport modes. Modelling the potential demand for AV ownership vs. rental, the effect on public transport usage, the requirement and location of parking and charging facilities, and capacity for cars to ‘circulate’ are all essential. However, while the datasets required to unlock this insight are available, in practice, accessing the relevant information to inform this modelling will not be easy.
Integration and sharing of data is crucial. One of the challenges for smart cities in recent years has been the interface between information and services provided by the public and private sectors. Guaranteeing the quality of data from different parties, as well as navigating the issues surrounding data ownership is a challenge. For example, mapping AV demand would require accessing data from in-vehicle, public transport usage and cellular data tracking. No one organisation can access this information without purchasing from or collaborating with others.
There are now a series of projects and organisations seeking to address these issues such as the Fiware consortium for interoperability standards, and major technology companies are building online data brokerages and promoting API integration. These projects are the real enablers of effective collaborations, as competing automotive industry stakeholders bring their products to market, and cities attempt to facilitate the introduction of this new transport mode.
As well as supporting the predictive analyses needed to accelerate the mainstreaming of AVs in our cities, data collected by AVs themselves will also prove a valuable commodity. Driverless cars need and therefore collect, analyse and combine vast quantities of data as they navigate the road network and the hazards that entails, constantly sending back information to servers, helping to improve their algorithms. In this sense, they are the ultimate environmental data collectors, frequently updating a virtual picture of our world. The quantity of data being collected will increase significantly, with gigabytes of lidar, radar and camera footage acquired every second.
The information generated by a fleet of AVs will present numerous opportunities – real-time journey information, not just speed data, might enable improved network management, which will benefit all road users, including emergency services. Alternatively, live in-dash cameras could be remotely accessed to detect and monitor crime, or inform emergency services as to the required level and scope of assistance. However, a key risk is that the size and nature of the data collected by AVs will be too difficult to interrogate securely and expensive to manage. Organisations looking to extract value from this information must invest in analytics tools and skill sets and also in information security processes and awareness to efficiently capture and process this data for applied use.
We’re on the cusp of a hugely disruptive technology, the impacts of which will permeate our society – movement data and analytics will be at the heart of this innovation.
This is a guest blogpost by Larry Augustin, CEO, SugarCRM.
It’s a perennial discussion: what’s going to be making headlines in the world of business technology in the coming twelve months? Usually, this brand of prediction piece is little more than a few half-baked thoughts thrown together to fill a few column inches, not to mention the fact that change in any industry is invariably slow and difficult to frame in twelve month chunks.
However, I do believe that in 2017 we’re going to see the beginnings of some serious shifts in the CRM market: general advancements in key technology areas like Internet of Things (IoT) and Artificial Intelligence (AI), along with a continued focus on modernisation among legacy CRM users in areas of social, mobile and user experience. All this will lead to some interesting changes in the market for both vendors and practitioners.
With this in mind, here are six predictions that I think may well make CRM headlines over the year.
Predictive analytics becomes the norm
The promise of predictive analytics has been talked up for quite some time. But as more companies eschew old-model SaaS deployments for cloud-based CRM and data warehousing, the throughput and storage issues that hindered truly predictive analytics initiatives will start to go away. This happens because the analytics are embedded in the application, at the point of usage. In reality, what we’re really seeing is what I call “embedded analytics”; analytics that may not technically be predictive, but is an integral part of the application.
Companies looking to better know their customers, and provide truly proactive service and delivery models, will be the first to take up predictive analytics. This will be put to best use in retention and servicing – the “give the customer what they need before they realize they need it” scenario, rather than the “offer someone something they might want to buy.” The benefits to retention of the former are huge, versus the intrusive and sometimes risky path of “best offer” models of predictive analytics.
Businesses get thirsty for more
The trend towards more modern, flexible CRM technology is going to continue. As legacy systems begin to really show their age, businesses looking for a truly integrated, seamless cloud-based CRM product, with engaging mobile user experiences, are going to have a number of options. This will be the year of the CRM maverick, with rewards for those looking to break the status quo and build exciting, different and innovative custom deployments that meet the demands of tomorrow’s customer.
AI gets ever closure
There were many announcements and concept-type demos around AI-powered CRM from various providers in 2016 but nothing of any material weight has been released for general use. I believe we’ll see the same in 2017: everyone will continue to talk about AI, but we are still a couple of years away from getting the technology in the hands of users.
While both SugarCRM and Salesforce will be releasing “1.0” versions of products they are aligning with AI, truly AI-powered CRM will not be available until 2018 at the earliest. Why? Because this is hard stuff, and even deep-pocketed providers have development issues to resolve before really bringing broadly available AI-powered tools to market.
Data, Data, Data
The battle for data-enriched CRM will continue to heat up in 2017. Data is a great way to extend the value of CRM to businesses of all sizes, especially those in the small-to mid-size range. By providing pre-populated data sets, the amount of “busy work” done by sales and other CRM users is reduced, and the better the data, the more effective individuals can be every moment of the day.
A lot of mergers and acquisitions as well as in-house development and partnerships, will fuel more data-powered CRM announcements in 2017. The key, of course, is seeing which providers provide the most seamless and most sensible use cases out of the box for their customers.
Customer experience – the key differentiator
Some may find it a bit ironic, but it is actually more disappointing that most CRM user organisations do not have a great relationship with their CRM vendor. If we are truly selling the promise of exceptional customer relationships as an industry, we need to walk the walk as well.
I hear all too often from prospects how the “market leaders” come to the table with arrogance, terrible terms and an overall unfriendly demeanor. That has to change. In short, just being “number one” or a multi-billion-pound company means nothing. CIOs and line-of-business decision makers know that there are alternatives on the market.
Mobile CRM takes a new direction
Mobile CRM is nothing new, far from it. However, at a time when both internal users and customers are demanding truly flexible interactions with companies then mobile development is more important than ever before.
Mobile CRM is no longer about “shrinking” the mobile app to fit the smart phone or tablet screen real estate. It is no longer about offline access; to really nail mobile CRM in 2017, organisations will need more than just extension apps but rather entire platforms, inextricably linked to their core CRM. It should be fast, easy and cost-effective for companies to build wholly new and customer-focused mobile experiences (whether the user be an employee or a customer) but if this isn’t done in the coming year then it’s probably going to be too late.
So, those are my handful of predictions for the CRM world in 2017. If there’s a unifying theme, then it’s that customer demands continue to change quickly and companies need to work harder than ever to keep up. It’s a challenge but also an exciting opportunity to make headway in a world that’s increasingly customer-centric.
This is a guest blog post by Ravi Shankar, chief marketing officer, Denodo.
For over two decades, the traditional data warehouse has been the tried-and-true, single source of truth in support of BI. However, BI is rapidly evolving, so the traditional data warehouse will have to evolve as well, to keep pace. Traditionally, data warehouses have required data to be replicated from source systems and stored within it, in a format that enables the data to be readily consumed by BI applications. These might seem like reasonable requirements, but they’re stunting the growth of this venerable technology.
BI analysts are now seeing the potential for delving into new kinds of data, such as machine-generated readings (from vehicles, packages, temperature sensors, manufacturing equipment, etc.) and output from myriad social-media platforms. Traditional data warehouses cannot support these new forms of data, as they appear “structureless” to the ETL processes dedicated to extracting, transforming, and loading the data into the warehouse. These processes would have to be re-written every time a new data source is introduced, which is neither practical nor sustainable, and quickly becomes costly. More importantly, batch-oriented ETL processes are just not set up to accommodate dynamic, real-time data streams.
Also, data sources are getting exponentially larger, which puts a strain on replication and storage, not to mention security, and further contributes to steadily rising expenses.
If data warehouses could accommodate streaming data, they could re-establish themselves as the single source of truth, but at what cost?
Adapting to the limitations
Companies are using open-source frameworks like Kafka and Spark to accommodate the new machine-generated, streaming, and social media data sources, and they are using distributed storage systems like Hadoop to offload data from the data warehouse. These solutions work well, and Hadoop is an extremely cost-effective, scalable alternative to physically expanding the storage capacity of a data warehouse. However, such companies are now saddled with a new problem: Data cannot be queried across the data warehouse, the Hadoop cluster, and the Spark system, severely limiting BI potential.
In this all-to-familiar scenario, the data warehouse is no longer able to be the single source of truth, simply because of its physical limitations: its need to physically replicate data to a central repository, its natural physical storage capacity, and its need for a programmer to physically update ETL scripts to accommodate every new source.
For this scenario, the solution is clear: not a physical data warehouse but a logical data warehouse.
Logical data warehouse
A logical data warehouse doesn’t physically move any data. Instead, it provides real-time views of the data, across all of its myriad sources, and these can be cloud sources, such as Kafka, Spark, and Hadoop, as well as traditional databases of any stripe.
This means, of course, that logical data warehouses can easily accommodate traditional data warehouses or data marts as sources, to support all of an organization’s standard reporting needs. In this way, logical data warehouses are perfectly capable of fulfilling Gartner’s ideal of a bimodal IT infrastructure, one mode characterized by predictability, and the other by exploration. The first mode can be met by the production of standard, highly audited reports, facilitated by the traditional data warehouse, while the second can be met by the ad-hoc, experimental capabilities of self-serve analytics, facilitated by the logical data warehouse.
As far as BI analysts are concerned, all of the company’s data, along with select external sources, sit in a single, logical repository. They do not need to know where different sets of the data may be stored, which data sets needed to be joined to create the view, or what structures define the various source data sets. They only see the data that they need, when they need to see it.
For the IT team, the BI infrastructure built around a logical data warehouse is much easier to manage than one that is built around a physical data warehouse. Since the logical data warehouse doesn’t actually “house” any data, merely the necessary metadata for accessing the various sources, there is no replication or storage to manage, and no ETL processes to maintain. If one source needs to be replaced by another, data consumers will not know the difference; they will experience no downtime, and the IT team can proceed with the migration at their own pace.
A logical data warehouse is the only logical choice for a data warehousing solution that serves as an organization’s single source of truth. It provides seamless, real-time access to virtually any source, including traditional data warehouses and data marts, and it’s easy to introduce new ones into the mix, without affecting users and with minimal impact on IT. Logical data warehouses can scale to accommodate any volume of data, across any number of sources, to meet current and future needs.
In a guest blog, Neo Technology’s CEO Emil Eifrem says graph databases are being looked into by government to manage big data.
While classical business database software, RDBMS (relational database management systems), still has an important role to play in the government context, these systems struggle to tackle new types of data-based problem that the public sector in particular wants to get a handle on.
Why? Relational databases are adept at managing transactional and analytical requirements and are easy to set up, access and extend. But they are challenged by the large amounts of data that agencies now need to manage, specifically in the context of the connections between data that so much of the real world is based on.
Public services – especially the new class of digitally-enhanced or delivered ones governments want to see – depend on being able to spot these connections for legal or improved service delivery reasons. In response, a way of working with data, graph database technology, is emerging as the tool that could help the public sector for this class of applications.
The NoSQL contribution
Graph databases are a big part of a new generation of database technologies for managing large datasets out of the NoSQL family. NoSQL (‘Not SQL’) includes the key-value store, the column family database, the document database and the graph database, while a fifth technology, Big Data data stores like Hadoop, oriented at large-scale batch analytics, has also emerged.
Each of the growing band of post-RDBMS databases has different strengths, but all are aimed at harnessing large volumes of data better than their SQL forebears. That matters, as we are all generating more and more data every day – a data mountain that’s a major challenge for government looking to gain actionable insight into issues.
However, while NoSQL can power all sort of big data work, for tasks that require examining the connections between people, places and events in the real world the tool that can help is graph technology. That’s because graph databases are fantastic at handling both data connectedness even with huge amounts of data. With graph databases civil servants can start to see patterns emerge by connecting multiple legal, welfare and demographic datasets thanks to the connections graph technology highlights/become apparent with – and which are starting to translate into potentially innovative new ways of helping us, the digital citizen.
Let’s consider a real example of how graph database technology is enabling functionality that RDBMS and Big Data/Hadoop technology would struggle to produce. A G8 country needed help in better visualising the relationships and connections powering modern social life. In this case, this was a case management application, helping its civil servants identify individual cases of potential criminal interest, including national security, immigration abuse (visa fraud) or attempts to obtain benefits illegally. Immediate access to such information has been deemed to be the difference between identifying and stopping a criminal and leaving it too late – time no-one can afford to lose as a relational database is still spinning away, crunching data too complex for it to process in real-time.
Meantime this particular government customer is deriving insights, which aren’t just helping it deal with this first problem, but which are becoming the data-driven basis for truly informed, data-based policy creation. As a result, graph technology is starting to enable a new, highly responsive informal learning system for this government – a move with major implications for not just it, but all e-government work.
Judging by these experiences and the kinds of conversations we have with public sector ICT and policy people, we believe graph databases could make a contribution in the drive to deliver cheaper, more integrated, and richer digital public services.
The author is co-founder and CEO of Neo Technology, the company behind Neo4j (http://neo4j.com/).
This is a guest blog by Daniel Cohen, a solutions engineer at DataStax
MiFID II – or to give it its full title, the updated Markets in Financial Instruments Directive – is an updated set of regulations that covers investment banks, hedge funds and alternative investment firms. It provides a list of areas where new compliance steps have to be taken by the companies involved in trading, covering everything from voice recording through to collection of trade data. It’s this area of data collection that will be interesting for these companies to consider.
At a roundtable I attended this year, the issues around MiFID II were not so much led by technologies as by methodologies for organisations to deal with the wider problems of compliance projects. As part of the discussion, areas like the availability of skills and recruitment in the wake of Brexit received the most attention, while making the most of investment in compliance was also discussed.
Can we do more with compliance?
As with much compliance regulation, the rules have been put together to catch up with the potential that new technologies can deliver. New financial instruments are being developed all the time that can stretch the rules that exist around trading, while the volume and variety of trades continues to go up all the time. Under MiFID II, both the industry regulators across Europe and the trading firms will be required to keep a complete and accurate list of all trades taking place.
Invoking the trilogy of variety, velocity and volume – Gartner’s traditional description of big data – should make MiFID II a fairly obvious use case for big data systems. The IT teams at banks are already investing in their compliance efforts to meet the deadline of 03 January 2018, while hedge funds and smaller IT organisations are looking at how they can work with providers to solve these problems as well.
What was most interesting during the roundtable discussion was how much variation there was in mindset in the IT teams involved. Some see the January 2018 deadline as the only end goal, some are already looking at ways to get around the legislation and take things out of scope, while others are looking ahead at what new things can be delivered using the data that has to be gathered. These differences demonstrate that IT is still seen in very different ways within businesses, from the traditional ‘keep the lights on’ maintenance role through to more strategic and forward thinking.
Ade Dickson, solution director of Sopra Steria made a great point here: “It’s important not to see the deadline as the end goal, but just the starting line. It’s pointless to build solutions that will only cope with two years’ worth of data, when this regulation will be in place for seven to ten years.”
Bringing together voice and data compliance plans
Alongside the trade data that IT teams will have to capture, mobile calls and data will also have to be tracked for compliance. Whereas previously, traders could leave their phones away from the trading desk and this would be compliant with MiFID I, MiFID II will force companies to capture all voice calls and data created on phones. According to Alex Phillips, Head of Mobile at Adam Phones, this change in the rules may be a difficult one to start off with.
The reason behind this is that, for many people, phones run apps as well as voice calls. While mobile call recording now works at the network level, apps like Facebook, WhatsApp or Linkedin on those phones can also be used to communicate. Each of those transactions would have to be recorded and kept for a minimum of five years. However, many of these services are fully encrypted, so the data saved would not be clear and it would be very difficult to force employers to release keys to apps that are not actually theirs.
The likelihood is that many firms will have to look at their mobile device management strategies over the next 18 months, preventing people from installing and using these kinds of apps on work phones.
Alongside this, companies will have to start planning for their compliance management. One example given in the roundtable was how compliance officers should listen to a set percentage of calls every month to check that recording of conversations is taking place. Alongside this, compliance teams should be making preparations on the processes they will use to link up trading data concerning a specific customer account or trader with all the relevant voice calls made by that trader would be required in the event of an investigation.
What is “relevant” is still up for definition – a lot of this will be determined by the first investigations into compliance status by the Financial Conduct Authority after the initial deadline passes. Being prepared for any questions is going to be a key skill for the future.
What preparations should companies be making?
From the roundtable, there were four steps that companies should be considering around MiFID II. These lessons should be relevant beyond the investment banking sector too:
- Prepare for the years ahead, not just the immediate deadline. With so much data coming in, IT teams should look at how much data they will have to support over five or seven years, rather than just in the next two to three years. Changing a storage infrastructure so soon after compliance regulations are brought in is something that you can plan ahead to prevent.
- Make a selling point of compliance. When MiFID II asks for data to be stored over five years … why not store seven, or even ten? This can be an opportunity to sell a service as going above and beyond.
- Brexit will have an impact … but how much is still to be determined. Alongside the general challenges for the financial services sector that Brexit represents, there will be more difficulty in finding people with the right talent around compliance. Looking at retention and training is therefore a key area to develop for IT teams.
- Developing new services around data will be a key differentiator. Across the banking sector, there are a lot of skilled and innovative individuals involved in these projects. The market potential from using data gathered for compliance purposes is huge, but these services still need to be developed and supported over time.