Write side up - by Freeform Dynamics

Page 1 of 212

June 18, 2018  9:00 PM

Enterprise AI: fiction and reality

Bryan Betts Bryan Betts Profile: Bryan Betts

If you are starting to suspect that the AI hype is overblown, and that it’s just the latest term that marketing folk love to add to make their product sound more interesting, you are not alone. You’d be wrong though – the AI effect is very real, and is set to make serious waves across many industries.

That’s one of the conclusions of a report I co-authored recently, but it comes with caveats. A big one is to understand that not all AI is equal, and that when the marketeers say ‘AI’ in relation to a serious offering, what they usually mean is ‘deep learning’, which is just part – albeit an important part – of the overall AI picture.

Even with this principle in mind, terminology can still trip you up. Where the practitioners say ‘deep learning’, many researchers prefer ‘deep neural networks’. it’s also important to realise that deep learning (DL) is not ‘intelligence’ as most people understand it. Rather than the clever androids of Humans and other popular science fiction, DL is the super-savant of AI: often able to out-perform a human expert, but only in a narrow domain.

Deep learning is AI at work

Still, what if that narrow domain is the quality control system on your production line, or the chatbot answering standard enquiries on your website? In that case, DL could be just what you’re looking for, both to improve response times and to offload the dull and repetitive stuff from your skilled people.

So while it is true that AI has been over-hyped, and that it was about time for a negative correction, there’s no need to fall into a trough of disillusionment. Approach it realistically, with an understanding that AI is not magic and has its limits, and it could pay off handsomely.

To learn more about Deep Learning, what it can offer and how to plan for implementation, you can download our free report here.

June 18, 2018  11:14 AM

Your PC is rooted and jailbroken: deal with it!

Richard Edwards Richard Edwards Profile: Richard Edwards

One of the defining characteristics of the modern digital workplace is a computer on every desk and a smartphone in every pocket. The ability to centrally discover, provision, deploy, update, and troubleshoot these devices is, without doubt, essential.

Discussing this topic at a recent vendor briefing uncovered the thought that the Windows PC is, by its very nature, a rooted/jailbroken device. Thus, in its current form, the PC constitutes a significant potential security risk to those organizations that use them, even with added protections.

The only route to Microsoft-verified Windows security

You’ve probably never considered rooting your Android device or jailbreaking your iPhone, but they’re common enough activities among curious tech enthusiasts. While different in approach, both activities usually result in the user of the device being able to do things that the device maker never intended.

Rooting an Android device gives you access to the entire operating system, while jailbreaking an iOS device enables you to run custom apps rather than just those in the App Store. If you use a Windows computer these capabilities sound unremarkable, right? However, managing these “capabilities” across a large desktop estate requires constant effort and vigilance, even with endpoint protection tools and configuration management solutions.

Mobile device management (MDM) products and enterprise mobility management (EMM) solutions can detect and restrict compromised smartphones and tablets from accessing corporate data and applications, but the Windows computer continues to present additional challenges. There are plenty of tools to help IT departments secure and protect PCs, but they inevitably add some level of cost, complexity and user dissatisfaction. So, what’s to be done?

We can all see that Windows 10 is slowly but surely turning into a modern mobile operating system, but legacy application compatibility issues are likely to prevent total transformation. So, if verified security is what you’re after, and with Windows 7 reaching end-of-life in January 2020, now’s the time to start seriously considering the pros and cons of Windows 10 in S-mode.

Evaluate the pros and cons of ‘S-mode’

S-mode is a locked-down, enhanced security mode of Windows 10. It constrains users to apps from the Microsoft Store, the Microsoft Edge web browser, and Azure Active Directory Domain Services. You can switch to “regular” Windows 10 (free of charge) if you buy a computer supplied with Windows 10 in S-mode and find it too restrictive, but this is currently a one-way street.

Enabling the switch back to Windows 10 is understandable during this introductory phase, but I’d like to see a feature that actively encourages organizations and individuals to explore the world of S-mode and the Microsoft-verified security capabilities it offers. A sensible location for this option would be under the ‘Reset this PC’ settings.

S-mode isn’t going to be compatible with everyone’s IT requirements, but there are many business and user scenarios it could suit. If you’re heading down the Microsoft/Office 365 route, and use Microsoft Office, web applications and virtualised apps, then it’s probably worth looking at what Windows 10 in S-mode has to offer. Be sure to let me know what you think.


June 4, 2018  10:29 AM

Xero highlights the role of the modern software ecosystem

Dale Vile Profile: Dale Vile

I recently caught up with Gary Turner, co-founder and managing of director of Xero. I was interested in meeting him as Xero has a reputation as the company that broke all the rules of small-business accounting. I’m obviously not talking about anything dodgy here – simply that it streamlined financial management and made it a whole lot more user-friendly by rethinking processes for a connected world. As Turner pointed out, it’s about providing seamless connectivity to everyone that matters to you in the context of small business financials. This includes your accountant, your bank, other financial service providers and, not least, solution providers who allow you to enrich or extend your accounting system.

The reason this strikes a chord with a lot of small business owners and managers is because financial accounting has traditionally fallen into category of ‘thankless chore’. Too often it was about scorekeeping, using software with an almost 18th-century feel that ran on an isolated PC in the corner of the office. It involved excessive amounts of paper, transcription, manual reconciliation of numbers from different sources and so on. It was easy to make mistakes, and hard to find and fix them.

And if all this was delegated to a bookkeeper, accounting became just a black box that periodically shot out reports that were never quite what you needed, and were probably too late to act on. Meanwhile, preparing the end-of-year books for your accountant to work on was always a nightmare.

Enabling the ecosystem

When you consider how Xero has helped its clients put most of this behind them, it’s easy to see it as predominantly a technology play – a more modern replacement for those clunky old PC packages. What’s really going on, though, is ecosystem enablement. For example, you can also look at Xero through a financial advisor lens. If you are an accounting firm with a trusted login to all your clients’ finance systems, which is essentially what Xero enables, your relationship with each customer becomes more intimate and continuous.

This has obvious benefits for both parties; you can track trends, anticipate needs, and sell additional services to meet them, increasing your revenue while helping the customer jump on opportunities and risks early. From a complimentary solution provider perspective, a cloud-based ecosystem like Xero’s allows you to tap into a well-defined market, and service it efficiently.

The big takeaway from all this is that when making technology decisions, it’s important to consider the ecosystem you are buying into. I know this is not a new principle – SAP’s attraction back in the 1990s was largely driven by the consulting and skills community that grew up around it, and even today, Microsoft Windows is still important to many businesses because of the ISV and OEM partners that support it. And we obviously can’t forget players like Salesforce.com that has one of the largest and most vibrant cloud-enabled ecosystems on the planet.

The element that is new – and Xero is a good example of this, in the way it has brought financial advisors and banks into the equation – is that ecosystems in today’s connected world are not just about technology partnerships.


May 31, 2018  5:56 PM

Cloud storage is hot – again!

Bryan Betts Bryan Betts Profile: Bryan Betts

But what do people mean when they talk about cloud storage – and do they really know what it can do these days?

Not so long ago, once you got beyond Dropbox, for most professional users cloud storage meant online backup, and the biggest concern with that was getting your data back. That hasn’t gone away – some of the biggest cloud service providers make it effectively free to upload data but then charge you to get it back. So backing up your 5TB system is fine, but the cost of restoring it might sting a little…

Cloud storage is now a lot more than just that, though. Indeed, I’m constantly impressed by the new use models that people come up with for it. For instance, there’s now multiple ways to deploy cloud storage as a replacement for NAS. There’s even some interesting ways to deploy it on-prem in appliance form as primary storage, as for example Cloudian and Qumulo can do, targeting capacity-intensive and performance-intensive uses respectively.

Then there’s the ways that cloud storage can help with GDPR. Regulatory compliance means you need to do data governance and management properly, and that’s a lot easier if you have a single data store or central system of record. A consolidated backup and archive service is just that, and for some companies it’s easier to acquire and run in the cloud than on-prem.

Specialists such as Commvault, Mimecast and Veritas recognised this a while ago, Box and Dropbox know it too, and so of course do the major cloud platform providers. It’s clear to smaller players as well – for instance, when I met with Redstor recently, one of the opportunities we talked about was leveraging cloud-based data management for GDPR subject requests.

Storage evolution

I see three things going on here. One is that hybrid cloud is now the norm, with the on-site element dealing with the latency and bandwidth issues that still afflict public cloud storage. Another is that although by its nature cloud storage is object-based, there’s more and more ways to deploy it as blocks or files to support existing applications – sure, born-in-the-cloud apps are object-based, but most current apps and end user requirements are file-oriented.

And then there’s our changing relationship with data, which is becoming virtual, like everything else. In this world, data lives “somewhere out there”, and a system backup is just a set of metadata that provides a view onto your data cloud that looks like a backup. A different set of metadata makes that data cloud look like a searchable archive, another view provides user self-service file retrieval, yet another supports GDPR data subject requests, and so on.

No longer do we need to think in terms of discrete systems, and of regular full system backups to tape volumes, real or virtual. Your governance systems still know where the data is really, of course, and apply the necessary security, but in data-centric ways, not system- or application-centric.

An over-simplification perhaps, but you get the picture. Today’s cloud storage isn’t what many people think it is – it might not even be in ‘the cloud’! In many ways, we’d do better to lose the term altogether and find a new one that better encapsulates those hybrid notions such as virtual-yet-consolidated, and local-yet-remote. If anyone reading this can think of one, please suggest it in the comments!


April 30, 2018  10:46 AM

Windows, modularity, and the Law of Unintended Consequences

Bryan Betts Bryan Betts Profile: Bryan Betts

For some time, my office PC had suffered from multimedia niggles. Some were relatively minor, such as Facebook and Twitter videos failing to play, but others were more problematic. For example, some work-related videos on YouTube would play, but others would not.

The straw that broke the camel’s back was when I discovered that I couldn’t record calls made within Skype for Business (“the app formerly known as Lync”). With a couple of typically-cryptic Windows error messages as my starting points, I started digging, and it wasn’t long before clues emerged pointing to what we used to call the Windows Media Player, or WMP.

Those who’ve installed Windows more often than I have will already have guessed it: this PC is running not Windows 10, but Windows 10N. The N versions of Windows lack media playback and were produced in response to the 2004 European Commission antitrust ruling which found Microsoft guilty of anti-competitive behaviour in bundling WMP as standard.

The problem is that in taking WMP out, it seems that Microsoft took out a lot more than just a media player – it took out a chunk of stuff that is pretty much a necessity these days. It’s a bit like saying, “You don’t want Internet Explorer? Ah, that must mean you don’t want the Internet – right, we’re removing all the networking…”

Fortunately, the fix was relatively easy: you can download the missing bits from Microsoft’s website for free. A few minutes and a reboot later, and all that media stuff was working at last.

Building a system or an app to be truly modular

Of course the world has moved on a lot since then, as has Microsoft. And perhaps the original N versions of Windows XP were just a quick fix in response to that EC ruling.

However, it seems extremely odd that Windows 10N should still be in this state over a decade later. Sure, you don’t want WMP, Skype, the browser and so on to all have their own individual multimedia subsystems, when they could all use the same one. But you don’t do that by making everything else work through WMP – you modularise it properly.

Either you make the absolute minimum bits that everything needs a core part of the operating system, and WMP is then one of several programs that can call those, or you flip it around and use APIs that allow other applications to call WMP – or any other media player – when they need video playback.

This is a fairly coarse-grained example, but as modularity grows more popular in many areas of software design, whether it’s via containers, hyperscale computing, microservices or some other paradigm, it’s going to be interesting to see whether it pops up again – or perhaps, how often it pops up again.

Edit… And now the update to Windows 10 v.1803 has broken the WMP add-on, necessitating a fresh reinstall. That’s clever, Microsoft – way to go!!


April 20, 2018  9:28 AM

Growth mindset lets Microsoft love Linux

Richard Edwards Richard Edwards Profile: Richard Edwards

Timed to coincide with the RSA security conference, Microsoft has announced Azure Sphere, a new solution for creating highly-secured, Internet-connected microcontroller devices. But there’s a bit more to it than that. Enabled by the company’s newly found ‘growth mindset’, Microsoft engineers can happily use the best technology for the job which, on this occasion, just happens to be Linux.

More than 9 billion microcontroller-powered devices are produced each year according to Microsoft’s blog post, and these are increasingly being connected to the Internet-of-Things (IoT). Similar to, but less sophisticated than, system-on-chip (SoC) designs, microcontrollers are used in many of the products and devices that surround us. These range from automobile engine control systems and medical devices, to building controls, appliances and children’s toys. The size of this market dwarfs that of the PC and mobile markets combined, and Microsoft clearly sees a lucrative opportunity if it can establish Azure Sphere as the protective ‘mother ship’ of the world’s smart things.

Azure Sphere, a secure home for smart things

In a world where so many ‘things’ are connected, almost anything, it seems, can be disrupted. It therefore follows that every connected thing needs to be protected. This is the ethos that underpins Microsoft Azure Sphere. Azure Sphere includes three components that work together to protect devices at the edge of the network: Azure Sphere certified microcontrollers, Azure Sphere Security Service, and Azure Sphere OS. Unpacking the announcement reveals a couple of surprises. First, Microsoft said it will license its technology to chip makers on a royalty-free basis. Next, the new microcontroller operating system is based on a custom Linux kernel, not Windows.

The Linux kernel has formed the basis of embedded operating systems and microcontroller-based products for many years, so Microsoft developing its own custom Linux kernel isn’t technically remarkable. However, it does jar somewhat with the Windows 10 IoT everywhere proposition of 2015, although the company was just starting to look at microcontrollers back then. Azure Sphere is still in private preview, but the first Azure Sphere chip, the MediaTek MT3620, is expected to ship in volume this year. The first wave of Azure Sphere devices are expected by end of year and, who knows, Microsoft might even ship a new device of its own. The much talked about Andromeda perhaps?

The seven properties of highly secure devices

Microsoft researchers began exploring the microcontroller-powered devices market in 2015. Then, in March last year, they published a research paper, The Seven Properties of Highly Secure Devices, that could have, theoretically at least, prevented the Mirai botnet attacks of 2016, when an estimated 100,000 compromised IoT devices took down several high-profile websites using DDoS attacks. In short, the paper concludes that a redesign is necessary if we want our IoT devices to be safe and secure. The paper also details a proof-of-concept project with MediaTek, a Taiwanese company that develops SoC for mobile devices, home entertainment systems, network and connectivity equipment, and other IoT products.

Here’s the list of properties, and their tests, which Microsoft deems critical if we are to have highly secure, network-connected devices:

  1. Hardware-based root of trust: Does the device have a unique, unforgeable identity that is inseparable from the hardware?
  2. Small trusted computing base: Is most of the device’s software outside the device’s trusted computing base?
  3. Defence in depth: Is the device still protected if the security of one layer of device software is breached?
  4. Compartmentalization: Does a failure in one component of the device require a reboot of the entire device to return to operation?
  5. Certificate-based authentication: Does the device use certificates instead of passwords for authentication?
  6. Renewable security: Is the device’s software updated automatically?
  7. Failure reporting: Does the device report failures to its manufacturer?

If we think about the damage caused by malicious botnets over the past decade, each comprised of tens-of-thousands of vulnerable Windows PCs, then it’s easy to see why these principles have become a bit of a thing at Microsoft. The company clearly wants to avoid the mistakes of the past, and while nothing has been explicitly stated, it would be a comforting thought if Microsoft were to apply the seven tests listed above to all its device efforts going forward. Having said that, I’m struggling to see how today’s Windows operating system model would ever make the grade. Maybe we’ll hear how in the coming months.

From security chumps to security champions

Continuing his mission to establish a ‘Digital Geneva Convention’, Microsoft President and Chief Legal Officer, Brad Smith, used his RSA security conference keynote to remind delegates of the perils of not updating and patching the Windows operating system. He pointed to last year’s massive cyberattack when, on May 12, more than 300,000 computers running Microsoft Windows were affected by WannaCry ransomware. Mr. Smith also highlighted the NotPetya cyberattacks that targeted Ukraine businesses and institutions, affecting the Windows-based systems of banks, government departments, newspapers and energy companies. These events were not cataclysmic, this time, but they do show us how the disruption of non-patched or out-of-date computer-controlled social infrastructure can affect businesses, governments, and millions of people.

Championing cybersecurity, Brad Smith also used the RSA conference to announce the Cybersecurity Tech Accord, a public commitment among 34 IT companies to “protect and empower civilians online and to improve the security, stability and resilience of cyberspace”. In essence, those companies signing-up to the accord pledge not to assist governments in cyberattacks. This list includes key enterprise IT suppliers, such as Arm, BT, CA Technologies, Cisco, Cloudflare, Dell, HP, HPE, Microsoft, Oracle, RSA, SAP, Telefonica and VMware. However, notable names not yet on the list include Apple, AWS, Google, IBM and Lenovo. We’ll have to wait and see if the accord has any real impact on the escalating political tensions associated with state-sponsored cyberwarfare, but it’s at least a start.

Nadella sets Microsoft free with ‘growth mindset’

The Windows operating is still a very important asset to Microsoft, and will be for many years to come. However, Satya Nadella has made it very clear where the company is heading: Intelligent Cloud and Intelligent Edge. The departure of Terry Myerson, Executive VP of the Windows Devices Group, signalled the start of this new epoch, and this week’s announcement, choosing Linux over Windows, provides yet another glimpse of the cultural change and ‘growth mindset’ that Nadella is driving at Microsoft, one in which nothing should be taken for granted and where employees must always be willing to check their assumptions as new data is revealed.

Microsoft is clearly changing its approach to security as with many things these days, but it still tends to blame the ‘bad guys’ and laggard IT departments for the woes caused by rogue Windows-based computers. The company has made billions of dollars from sales of its software, so we shouldn’t feel sorry for it when it has to spend millions of dollars cleaning up the mess caused by its legacy products, even those that are no longer supported. It’s clearly the right, and moral, thing to do.


March 21, 2018  11:00 PM

Will Alexa kill the radio star?

Jason Stamper Jason Stamper Profile: Jason Stamper
AWS, radio

I am fond of my Amazon Echo smart speaker (other brands are available). Some would say perhaps too fond. I am wont to ask, “Alexa, do I look good today?” and ‘she’ sometimes says I look great, which is in equal measure reassuring, but also rather unlikely.

By the way, before you ask, I do know Alexa is not really a ‘she’, but a combination of basic machine learning, speakers, a microphone and that mysterious Internet ‘thingy thing’. But I digress.

You see, a report this week by the BPI and Entertainment Retailers Association has asserted that the Amazon Echo and similar devices from companies that will never, ever be evil, has suggested that such smart speakers will buoy music streaming but have a negative effect on radio.

The logic is that said devices, with their ability to stream vast arrays of music at relatively low cost, will make it less likely that people who own such devices also, or ever, listen to the radio.

So Alexa, not video, could kill the radio star.

Smart speakers replacing other audio devices

An estimated 27 million smart speakers were sold last year worldwide and most people who have bought one listen to music on them. More to the point, the report found that 39% of smart speaker owners said that time they would have spent listening to the radio is now spent listening to music streamed to their smart speaker.

If that all sounds a little bleak for radio stations, may I offer a few words of comfort? Yes, it’s quite possible that smart speakers will temporarily distract listeners from their FM or DAB radio stations, as they become accustomed to being able to stream a huge amount of music to their speakers, on demand – indeed without even getting off the sofa.

But in my view, that may be a temporary aberration. In this era of fake news or ‘alt-news’ I know more and more people that are looking to some sort of curation of their news sources – an editor, a pundit, an organization with checks and balances – that they can trust more than Twitter, Facebook, or the myriad of social websites that we tend to visit every day.

There’s also the fact that my Echo will play most of the DAB radio stations for those times I want to listen to the news, some banter or unexpected songs.

Voice vs commodity music streaming

But the commoditisation of streaming music (Amazon says there are “tens of millions” of songs on Amazon Music and it’s about £1 a week to subscribe) may, in fact, make radio stations focusing on the spoken word the winners in all of this. BBC Radio 4’s figures have been weathering the storm, and chat radio LBC has seen some exciting growth on the back of challenging caller and celebrity or politician interviews.

Are smart speakers going to kill off radio? No, they aren’t. Might they force incumbent radio stations to consider their mix of music and spoken word? Probably, yes. In the mean time, have a listen to Frank Skinner on Absolute Radio on a Saturday morning. It’s irreverent and witty, but also have a listen to how the music – though frequent – takes a back-seat to the humour. In the face of Amazon Echo speakers and the like, the best DJs are fighting back with aplomb, and they are anything but traditional disc jockeys. Echo won’t kill the radio star, at least not in my lifetime.

But what do you think? Am I talking out of my Tweeter? Drop me a comment, I’d love to hear your views.


March 21, 2018  5:23 PM

Amazon Neptune: a shot in the arm for the graph database?

Jason Stamper Jason Stamper Profile: Jason Stamper
AWS, Graph database, HANA, IBM, NoSQL, Oracle, Redis, SAP, Teradata

Amazon Web Services (AWS) announced its entry to the graph database market at its AWS reINVENT conference in Seattle in November last year. It was a notable announcement for a couple of reasons: it was the first graph database from the company (it offers a range of relational and NoSQL databases as a service). But it also shone a rather bright light on a database category that has often been considered niche, complex and expensive.

Neptune is currently in preview before it reaches general availability, but we expect that to happen soon. So should you be bothered?

A graph database is one that uses graph structures to enable the data to be queried, using the concepts of nodes, edges and properties to represent and store data. The key concept is the fact that the graph directly records the relationships between different data items in the database. Because the graph links related objects directly, it means those that have a relationship with one another can often be retrieved in one operation.

In relational databases, there are no such direct connections between related objects as data is stored in rows and columns. To create a relationship between different elements developers must write a ‘join’. But joins can become unwieldy and affect database performance.

The characteristics of graph databases enable the simple and fast retrieval of complex hierarchical structures that would be harder or even prohibitively time-consuming to model in relational databases.

The slight drawback with graph databases is that they cannot easily be queried with the de facto querying language for relational databases, Structured Query Language (SQL). Not only that, but in the graph database world there is not yet an equivalent de facto query language — there are a number of industry standard languages but there is likely to be a shakeout of some of these as graph databases become more popular and a clear winner possibly emerges.

Amazon says it built Neptune specifically for the cloud, which has its pluses and minuses. The drawback is there isn’t an on-premises version. The advantage though is that due to its economies of scale AWS tends to be able to offer good value subscriptions. As with other AWS managed services Amazon Neptune is highly available, with read replicas, point-in-time recovery, continuous backup to Amazon S3, and replication across AWS Availability Zones.

It can store billions of relationships and the graph can be queried with milliseconds latency. Neptune supports encryption at rest and in transit. As for that thorny issue of which query languages to support, AWS has hedged its bets with the option of Apache Tinkerpop Gremlin or SPARQL (Microsoft’s cloud graph offering, Azure CosmosDB, supports Gremlin or Gremlin-compatible languages such as Apache Spark GraphX).

I would have liked to see the addition to both of Cypher, a language developed by graph database pioneer Neo4j, as we believe it has very widespread adoption. Neo4j donated it to the openCypher Project in 2015 and as well as Neo4j it’s supported in SAP HANA Graph, Redis and AgensGraph databases.

Use cases and early adopters

Early adopters of Neptune are likely to be existing AWS users who have some or all of their data in the cloud already: AWS already offers a range of databases including relational and NoSQL options.

Amazon envisages that Neptune will power graph use cases such as recommendation engines, fraud detection, knowledge graphs, drug discovery, and network security. Security is probably the most common area where graph databases have been pressed into action, but they are also used in logistics, supply chain management, master data management, life sciences, e-commerce and even the hospitality industry.

Companies having a play with Neptune in preview include AstraZeneca, Thomson Reuters, Siemens, and the Financial Industry Regulatory Authority (FINRA). Amazon has been looking into how it can use it to improve its own Amazon Alexa system.

I believe AWS’ move into the graph database space is significant for the sector. It will make it simpler than ever for people to have a play with a graph database inexpensively. With Neptune, you don’t need to worry about hardware provisioning, software patching, setup, configuration, or backups.

It’s not that there are not other graph-as-a-service offerings, but few have quite the reach of AWS. With so many companies already having at least some of their data on AWS, this is an opportunity to see what a graph database can do for you.

There are too many graph databases to mention them all here, but here is a selection of firms large and small (in alphabetical order) to add to those mentioned above. Most offer some kind of pre-production free trial, so you can kick the tyres before you jump right in.

AllegroGraph
ArangoDB
Graph Base
Graph Story
HypergraphDB
IBM
Oracle
Ontotext
OrientDB
Teradata
Titan

Do you have any experience of using graph databases? I’d be interested to hear your thoughts in the comments section.


March 1, 2018  4:46 PM

What’s all the fuss about in-memory databases for IoT?

Jason Stamper Jason Stamper Profile: Jason Stamper
Data Grid, IN MEMORY, In-Memory Database, iot, Spark

IoT Back to Basics, chapter 4: IoT projects risk failure without careful consideration of data management processes and analytics. Their ultimate goal after all is to glean valuable information from data coming from the ‘things’ on the network – sensors and smart devices – in order to act on it.

So I thought I’d look at some of the novel trends in in-memory data processing: in-memory databases as well as data fabrics and data streaming engines.

The use of memory in computing is not new. But while memory is faster than disk by an order of magnitude, it is also an order of magnitude more expensive. That has for the most part left memory relegated to acting as a caching layer, while nearly all of the data is stored on disk. However in recent years, the cost of memory has been falling, making it possible to put far larger datasets in memory for data processing tasks, rather than use it simply as a cache.

It’s not just that it is now possible to store larger datasets in memory for rapid analytics, it is also that it is highly desirable. In the era of IoT, data often streams into the data centre or the cloud – the likes of sensor data from anything from a production line to an oilrig. The faster the organization is able to spot anomalies in that data, the better the quality of predictive maintenance. In-memory technologies are helping firms see those anomalies close to, or in, real-time. Certainly much faster than storing data in a disk-based database and having to move packets of data to a cache for analytics.

I expect take-up of in-memory data processing to accelerate dramatically, as companies come to grips with their data challenges and move beyond more traditional data analytics in the era of IoT. In-memory databases are 10 to 100 times faster than traditional databases, depending on the exact use case. When one considers that some IoT use cases involve the collection, processing and analysis of millions of events per second, you can see why in-memory becomes so much more appealing.

There’s another big advantage with in-memory databases. Traditionally, databases have been geared toward one of two main uses: handling transactions, or enabling rapid analysis of those transactions – analytics. The I/O limitations of disk-based databases meant that those handling transactions would slow down considerably when also being asked to return the results of data queries. That’s why data was often exported from the transactional database into another platform – a data warehouse – where it could more rapidly be analyzed without impacting the performance of the system.

Hybrid operational and analytical databases

With in-memory databases, it’s becoming increasingly common for both operational and analytic workloads to be able to run in memory rather than on disk. With an in-memory database, all (or nearly all) of the data is held in memory, making reads and writes an order of magnitude faster – so much so that both transactional duties and analytic queries can be handled by the same database.

There are a number of in-memory database players vying for what has become an even more lucrative market in the era of IoT. The largest incumbent database vendors such as Oracle, IBM and Microsoft have added in-memory capabilities to their time-tested databases. SAP has spent many millions of dollars educating the market about the benefits of its in-memory HANA database, saying it will drop support for all other third party databases under its enterprise software by 2025. There are also smaller vendors vying for market share such as Actian, Altibase, MemSQL and VoltDB.

Data grids & fabrics

Then there is the in-memory data grid (sometimes known as a data fabric) segment. This is an in-memory technology that you ‘slide’ between the applications and the database, thereby speeding up the applications by keeping frequently-used data in memory. It acts as a large in-memory cache, but using clustering techniques (hence being called an in-memory grid) it’s possible to store vast amounts of data on the grid.

In recent years their role has evolved beyond mere caching. They still speed up applications and reduce the load on the database, and have the advantage of requiring little or no rewriting of applications, or interference with the original database. But now as well as caching, they are being pressed into action as data platforms in their own right: they can be queried (very fast, in comparison with a database), they add another layer of high availability and fault tolerance – possibly across data centers – and they are increasingly being used as a destination for machine learning.

There are data grid offerings from a handful of vendors, amongst them Oracle, IBM, Software AG, Amazon Web Services, Pivotal, Red Hat, Tibco, GigaSpaces, Hazelcast, GridGain Systems and ScaleOut Software.

Data streaming engines

The third category, streaming, is also notable in the context of the Internet of Things. Data streaming involves the rapid ingestion and movement of data from one source to another data store. It employs in-memory techniques to give it the requisite speed. Streaming engines ingest data, potentially filter some of it, and also perform analytics on it. They can raise alerts, help to detect patterns, and start to form a level of understanding of what is actually going on with the data (and hence with the sensors, actuators or systems that are being monitored).

While streaming was largely confined to the lowest-latency environments, such as algorithmic trading in the financial sector, more and more use cases in the IoT space are latency sensitive: e-commerce, advertising, online gaming and gambling, sentiment analysis and more.

There are relatively few vendors with data streaming technology. But they include IBM with Streams, Amazon Web Services’ Kinesis in the cloud, Informatica with its Ultra Messaging Streaming Edition, SAS’ Event Stream Processing (ESP), Impetus Technologies with its StreamAnalytix and also TIBCO, Software AG and SAP (which bought StreamBase Systems, Apama and Aleri, respectively).

Smaller competitors include DataTorrent, which has a stream processing application that sits on a Hadoop cluster and can be used to analyze the data as it streams in, and SQL-based event-processing specialist SQLstream. Another young company is Striim.

In the open source space, Apache Spark Streaming and Apache Storm both offer streaming – most vendors have added support for Spark rather than Storm. But that, as they say, is a story for another day.

You can read the previous chapter in this series here.


February 22, 2018  1:45 PM

Where are security and data governance in IoT?

Jason Stamper Jason Stamper Profile: Jason Stamper
attacks, Data governance, Hackers, iot, Security

IoT Back to Basics, chapter 3: It’s no surprise that security and governance are important considerations when it comes to the IoT, but quite how incredibly important they are may not be immediately obvious.

Ensuring that users of IoT systems and smart devices remain safe and secure – which requires that their data stays protected and carefully governed – is vital if businesses and public sector institutions are to initiate successful IoT projects. There isn’t just the risk to a user’s privacy, and the possibility of big fines from regulatory bodies when things go awry, but also the issue of reputational risk and the commercial consequences of confidence in your brand being undermined.

Of course, security should be high on the agenda in all areas of IT. A targeted and sustained ransomware attack on the NHS, in May last year, was just one example of how sophisticated some of the hackers – and their malware – have become. At a machine data analytics conference last year, the chief security officer at Travis Perkins, a British builders’ merchant and home improvement retailer, told us that his organization had faced 3,851 ransomware attacks in just one month last summer.

Attack surface

The extra problem with IoT is that it vastly increases the potential ‘attack surface’ – there are more connected devices and gateways, and hence more areas of potential vulnerability, which gives those with nefarious intent greater opportunity to wreak havoc. And while many existing technologies and data governance methodologies can also be used in the era of IoT, they cannot make up for the broader attack surface.

Some of the ‘things’, such as sensors, are relatively dumb and therefore unlikely to bring much gratification to hackers. There’s not a huge amount of twisted satisfaction to be gained from interrupting temperature or wind-speed readings from a sensor in a wind turbine, for example.

But when you consider that IoT also includes the likes of connected vehicles, wear-at-home medical devices, industrial and hospital equipment, you can see why security is such a vital consideration.

For instance, in 2015 a group of researchers from the University of California, San Diego, discovered a serious weakness in vehicle security that allows hackers to take remote control of a car or lorry, thanks to small black dongles that are connected to the vehicles’ diagnostic ports.

These are common in both cars and lorries, fitted by insurance companies and fleet operators, as a way of tracking vehicles and collecting data such as fuel efficiency and the number of miles driven.

But the researchers found that the dongles could be hacked by sending them SMS text messages, which relayed commands to the car’s internal systems. The hack was demonstrated on a Corvette, where the researchers showed they were able to apply the brakes or even disable them (albeit as long as the car was at low speed).

You can imagine the repercussions of such a hack as we move ever-closer to driverless cars.

Home invasion?

There have been other worrying security lapses around IoT that give pause for thought. In 2013, for instance, the US Federal Trade Commission (FTC) filed a complaint against TRENDNet, a Californian maker of home-security cameras that can be monitored over the Internet, for failing to implement sufficient security measures.

TRENDNet’s cameras were hacked via the Internet, leading to the display of private areas of users’ homes on the Web, and allowing unauthorized surveillance of adults as well as children going about their usual daily lives. As well as an invasion of privacy, there was the potential that such covert surveillance could be used to monitor the comings and goings of the occupants of a premises, and hence give rise to further criminal activity once the hacker knows when there is no one at home.

Clearly, some IoT initiatives have different risk profiles to others. For instance, ‘white hat’ hackers last year demonstrated that they had been able to hack into a smart domestic appliance network and turn off ovens made by the British company AGA. Being able to turn them on and adjust the temperature would be more dangerous, but the ramifications are still worrying.

Another penetration testing company discovered that hackers could remotely compromise a connected kettle with relative ease and thus potentially gain unfettered access to a person’s wireless network, from which they could change DNS settings and monitor all web traffic for access to bank accounts and other sensitive data.

It’s obvious that the companies involved in implementing IoT need to be just as sophisticated about their security processes and protocols as the most sophisticated hackers – but time and again we have seen companies outsmarted by either ‘white hat’ or, worse, ‘black hat’ hackers.

The potential security risks around IoT are very real

Organizations contemplating the benefits IoT projects (or in the case of local or federal government, their citizens) would be wise to consider security and data governance very carefully indeed. Authentication and authorization technologies are likely to be necessary. Data masking (removing attributes that would enable a hacker to identify specific people and their habits, for instance) may also be called for, and in some cases even mandated by law.

Ensuring privacy is also an issue. While some consumers or citizens are quite happy to share various data with organizations, others are not. Organizations must therefore ensure that they ask users to ‘opt in’ to IoT-related projects or systems, rather than opting them in without explicit consent (even if they subsequently offer an opt-out).

Companies that don’t do this run the risk of annoying customers and falling foul of auditors and legislators. If potential fines are not sufficient to deter some companies from taking security and data governance seriously, the potential reputational damage certainly should be!

You can read the previous chapter in this series here.


Page 1 of 212

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: