This is a guest blogpost by Luiz Aguiar, data scientist at GoCompare.
We produce a massive amount of data every day.
Not only that, our attitudes towards the data we produce are also changing. We’re becoming more comfortable sharing the data we produce with apps, businesses, and other entities, if it means getting better services.
Most of us are happy for companies like Google, Amazon or Netflix to know our preferences to better tailor the content we are served, or recommend the things we want to buy. We’re even inviting these companies into our homes by embracing AI systems, like Alexa, Google Home or Siri to make our lives easier, by using the data we provide them.
So if we produce data at exponential speed and are happy to share it for get tailored services, why aren’t more companies taking advantage of this? Why do so many still rely solely on traditional market research and guesswork?
The key problem is that the sheer amount of data available means it’s hard for companies to analyse it effectively. It would take forever for a person to be to be able to analyse all the data we provide and get some insight from it, let alone being to design better services as a result.
The problem of unstructured data
Not only is the sheer volume of information a problem for analysts, another issue is that the majority of this data is unstructured making it incredibly hard to classify and compare.
That’s because the information we produce is not in the right format, shape or requires some enrichment.
As an example, imagine you are in a restaurant deciding what to order. The likelihood is you’ll look through the menu and choose one of the options based on the information available – this is structured data.
In comparison, unstructured data would be like sitting down to a list of every single raw ingredient and cooking utensil available in the kitchen, then having to piece it all together to figure out what you want. All the information is there – but just not in an easily accessible way.
Obviously, the first option is the easier one to process, the second would be too daunting and complex for a person to analyse and make a quick decision – and this is where machine learning can help.
Machine learning runs information through a series of algorithms that classify and group data and then uses this to find patterns and subsequently predict future behaviours, all on an enormous scales. In short, machine learning techniques are able to extract insights deeply hidden inside your data, that otherwise would be impossible to detect.
Thinking back to our restaurant example, while a person might struggle to sift through the unstructured data for just one establishment, a well-trained AI could do this for any restaurant in the country, or even the world.
Then, using other information about you it could make an informed decision of what you should eat, when you should eat and where you should eat – giving you the best possible experience, without you having to even think about it.
And that’s just one example. Algorithms as Artificial Neural Networks, that try to mimic the functions of a biological neural network are very powerful in pattern recognition and image classification. They have the potential to do a better job than humans at recognising stock market trends, house prices, insurance costs, medical diagnoses, you name it. The possibilities are almost endless.
This is why you should care about machine learning, and why over the next few years machine learning and AI won’t just be the buzzword that everyone is talking about, but will be the fundamental difference between successful tech companies and those that get left behind.
GoCompare has opened access to its APIs to other fintech organisations through a new community development, Machine Learning for Fintech. For more information, or to apply for a developer token, go to https://www.communityapis.com/
Originally from Rio de Janerio, Luiz holds completed his an MSc in Computer Science Optimisation and Machine Learning from the Pontifical Catholic University of Rio de Janerio.
Luiz moved to England in July 2015 and worked for Formisimo as lead data scientist on the Nudgr project and Perform Group as a Data Scientist, before joining the Data Science team at GoCompare.