CW Developer Network

Page 1 of 9212345...102030...Last »

April 25, 2017  12:36 PM

Veritas survey provokes Talend opinion on GDPR data developer responsibility

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

A new study from information management firm Veritas Technologies has suggested that 86 percent of organisations worldwide are concerned that a failure to adhere to the upcoming General Data Protection Regulation (GDPR) could have a major negative impact on their business.

Nearly 20 percent said they fear that non-compliance could put them out of business.

NOTE: This is in the face of potential fines for non-compliance as high as $21 million or four per cent of annual turnover – whichever is greater.

Getting anal over data retention

There is also widespread concern about data retention.

More than 40 percent (42%) of organizations admitted that there is no mechanism in place to determine which data should be saved or deleted based on its value.

Under GDPR, companies can retain personal data if it is still being used for the purpose that was notified to the individual concerned when the data was collected, but must delete personal data when it is no longer needed for that purpose.

“There is just over a year to go before GDPR comes into force, yet the ‘out of sight, out of mind’ mentality still exists in organizations around the world. It doesn’t matter if you’re based in the EU or not, if your organisation does business in the region, the regulation applies to you,” said Mike Palmer, executive vice president and chief product officer at Veritas. “A sensible next step would be to seek an advisory service that can check the level of readiness and build a strategy that ensures compliance. A failure to react now puts jobs, brand reputation and the livelihood of businesses in jeopardy.”

So how can developers help?

As the new breed of developers starts to encompass data developers, data scientists and data engineers, what can data-centric programmers do to help?

Jean-Michel Franco is the product marketing for data governance man at open source integration data integration software products company Talend.

Franco says that GDPR is all about accountability and that in the past, only a few stakeholders in organisations felt concerned with the need to protect data related to privacy. This may have resulted in a somewhat uncontrolled proliferation of those data across systems and files folders.

“GDPR is game changing because it elevates the challenge at the enterprise level, with high stakes due to huge potential fine. Developers now need to establish privacy by design. This is game changing for a developer, they must consider privacy from each end every activity they manage,” said Talend’s Franco.

“This is not only true for new systems, but developers also must think about the legacy system that needs to be referenced and audited, this might need to be updated for compliance. For example, Business intelligence or analytics systems might expose sensitive data to a very wide range of users in an inappropriate way,” he added.

New data management challenges?

Franco says that the first challenge is privacy by design. As mentioned above, developers must consider privacy at the very beginning of the design of an application.

He urges us to think about a clickstream analytics application to track and trace customer journeys in the website. An application designer must consider whether this application contains privacy related data. If it does, they need to make sure they have collected the consent of the web visitors.

“In some cases, they might need to do an audit of the risk associated with this application in case of a data breach. Then they need to make sure this information can be referenced centrally in the organisation and reconciled with the other PII data, define policies such as those related to retention, and finally comply with the rights for the subject for accessibility and rectification,” said Talend’s Franco.

Talend’s Franco writes…

The concluding portion of this story is wholly attributed to Jean-Michel Franco.

The second challenge is data governance. Data governance is the new challenge in the digital era. There are so many data sources, users that want to be self-sufficient with data, and use cases that a central organisation cannot manage without engaging many users within the organisation for accountability.

As it is a team effort across the organisations, data governance policies are known to be challenging to establish. A regulation such as GDPR elevate data governance as a mandate, so companies now need the establish, or re-enforce, a dedicated organisation and set of best practices to deliver on the promise of data governance.

The third challenge relates to the transparency of algorithms. With the growing popularity of machine learning and artificial intelligence, decisions tend to be more and more automated. GDPR establishes the right to explanation, whereby a user can ask for an explanation of an algorithmic decision that was made about them. Again, this will have a huge impact, because it pushes accountability on data scientists and developers to enable explanation of the insight that are given as outcomes in their models.

It also a call for code of conduct to make sure that the use of advanced analytics avoid discrimination, knowing that a big step towards countering discriminatory algorithms is to ensure we can understand them.

Concluding thoughts

It is true, the rise of the data developer and the need to understand the difference between data engineers and data scientists has now come to the fore.

April 22, 2017  4:11 PM

People are pesky & picky, developers need in-app support power

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

Developer needs platforms, developers need (programming) languages, developers need DevOps, developers need automation with workflow controls… and of course developers need beer, soda and pizza.

But also, developers need the ability to placate those pesky people we call users with something that resembles support…

… and, ideally if possible, the option to be able to give those users that support right there inside the application they are using.

‘In-app’ support

We tend to call that ‘in-app’ support.

This scenario goes some way to attempt to validate the rationale behind customer service specialist Zendesk and its intention to put personalised customer support and self-service into any mobile app with its Zendesk kit on Fabric.

The Zendesk kit is intended to allow developers to install Zendesk Support for in-app support without users ever leaving their mobile app. Zendesk’s kit is the first customer service kit available on Fabric.

CRM explosion

According to the (not as magical as Gartner) analysts at IDC, the CRM applications market is forecast to reach $44.2 billion by 2020.

The suggestion (and, if you will, the call to action for developers) is that service and engagement need to be embedded into every application being developed today.

“Customers demand an intuitive and quick support experience on every channel and brands that don’t provide it will be at a disadvantage,” said Sam Boonin, vice president of product strategy at Zendesk. “Zendesk Support functionality includes contact forms and self-service functionality and is offered alongside other Fabric kits like Crashlytics and Stripe etc.”

As a use case example, when an agent receives a request, they will know information such as what customer service channel the customer is on, former inquiries, type of account and the version of the app are they using.

April 19, 2017  9:40 AM

Venafi: automate cryptographic functions in DevOps, or die, basically

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

DevOps is, obviously, a coming together of Developers and Operations teams. All well and good, but what about security?

Ah that’s okay, we have (well, the industry has) thought of that as well i.e. we now have DevSecOps to bring together Developers and Operations and the Security function that needs to serve them both.

Is that enough, already?

Cryptographic firm Venafi thinks it’s not and is launching ‘Venafi Cloud for DevOps Service’ in response to the fact that it says the action to enforce corporate key and certificate policies consistently in DevOps teams simply doesn’t happen enough.

Even among mature DevOps teams, insecure practices are rife with 80% allowing self-signed certificates and 68% allowing key re-use, says Venafi.

As a practical examlpe, Venafi also surmises that two-fifths (38%) of mature DevOps teams fail to replace development and test certificates when code rolls into production, leaving teams unable to distinguish between the identities of trustworthy and untrustworthy machines.

Automate cryptography, or else

So to Venafi Cloud for DevOps Service — a development designed to deliver cryptographic keys and digital certificates for platforms such as Docker Enterprise, HashiCorp, Terraform and SaltSack Enterprise.

Thanks to the fully automated and scaleable nature of key and certificate orchestration through Venafi Cloud for DevOps, enterprises are able to maintained accelerated application development while also remaining secure.

“It’s clear that most organisations are still struggling with securing the cryptographic keys and digital certificates used to uniquely identify machines,” said Kevin Bocek, chief security strategist for Venafi. “Although DevOps teams indicate that they understand the risks associated with TLS/ SSL keys and certificates, they clearly aren’t translating that awareness into meaningful protection. This inaction can leave organisations, their customers and partners extremely vulnerable to cryptographic threats that are difficult to detect and remediate.”

So it could well be an inconvenient truth then… cryptographic security risks are amplified in DevOps settings, where compromises in development or test environments can spread to production systems and applications.

April 19, 2017  7:59 AM

Don’t develop for hybrid cloud without hybrid deployment

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

Yes yes okay we get it. Hybrid cloud is better than public cloud (with its openness and multi-tenant-ness) & better than private cloud alone (with its expandability that doesn’t ever match the breadth of public cloud) and that’s the way it shall always be, for now.

But hybrid cloud on its own has implications — not least for software deployment concerns.

While the industry talks about hybrid deployment models, today’s transactional, operational and analytic data is typically managed in rigid silos, limiting performance and (some would argue) the option to get to actionable insights — this is not good for the new hybrid world.

Addressing ‘diverse data’

Tech vendors are aiming to address this issue and this is why we see firms like Actian position themselves specifically as ‘hybrid data management, analytics and integration’ specialists.

The firm’s Actian X product is claimed to be the first native hybrid database that combines the an OLTP database with a analytics query engine.

Rohit De Souza, CEO of Actian says that his firm embraces the entire hybrid data ecosystem, combining best-fit tools to bridge on-premise and cloud environments while also powering modern data-driven applications and services.

“Actian X brings the record-breaking performance of Actian Vector analytics into the heart of the OLTP database to effortlessly process transactional, analytical and hybrid workloads from a single database running on a single compute node,” said De Souza.

“Actian X’s common SQL language interface and management framework seamlessly deliver operational analytics and enable a new class of applications that can interleave OLTP and analytics queries on the fly,” he added.

What’s different here

What is (arguably) genuinely different here is that a huge number of data science, data engineering, data developer and data analytics software systems are tied to specific applications or are limited by available system memory.

As De Souza points out, hybrid data management integrates high performance analytics within an enterprise’s mission critical transactional data systems resulting (potentially, if we do it right) in a system that can analyse must faster. They say (as of 2017) about 10x faster than non-hybrid systems.

April 12, 2017  12:18 PM

Teradata analytics guru: one algorithm to rival them all?

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network written by Yasmeen Ahmad in her role as practice partner for analytic business consulting at database and data services firm Teradata

Ahmad writes as follows… 

Just as decision making in humans is improved by collective groups of people who individually contribute with diversity in knowledge and experience, machines also benefit from the notion of a collective mindset.

The idea and study of collective intelligence dates back many years – pooling together different information and perspectives can provide a better picture of the overall challenge or problem to be solved.

The collective computer brain

Similarly, individual algorithms operating on overlapping or the same datasets will make predictions of varying quality.

The strengths of one algorithm may help it outperform other algorithms given a specific set of data related to a situation. Leveraging this diversity in strengths of different algorithms means we can use multiple algorithms all working in concert with one another.

This technique is known as ‘ensemble modelling‘.

In fact more advanced Artificial Intelligence algorithms, such as neural networks, make use of this idea of collective intelligence.

Collective intelligence should not be limited to within the human and machine populations, but can be leveraged between human and machine. It is known that medical imaging and analytics is better at detecting cancer in a patient than a human pathologist. However, if we are able to take the expert opinion of a pathologist of how advanced a cancer is, this can augment the image analytics.

The power of automation

The ability of machines to be on par with human decision making in terms of accuracy is great. However, the game changer is the ability to automate in machines, which allows businesses to make millions of decisions that would otherwise be impossible. The speed of execution is a huge benefit of machine learning techniques that becomes a key differentiator.

KPMG predicts that relying on machine learning to partly automate the insurance claims process could cut processing time down from a number of months to just a matter of minutes. Similarly, in the oil industry, what could take eight weeks using human inspectors takes only five days using SkyFuture’s oil rig inspecting drones working with a single drone operator and engineer.

Parallel (decision) power

The power of automation allows tens or hundreds of tests to be run in parallel to compare decisions. This challenger methodology means we can assess whether decisions are good or bad.

Did we make the best move? A decision, when taken, inherently changes everything – the environment, market, customer opinion etc. To really know if it is was the best decision over all others, multiple decisions need to be executed and evaluated in parallel – a challenger methodology allows for this.

An assessment of multiple decisions also helps the machine learning algorithms. They can learn from the positives and negatives of multiple decisions and learn how to mitigate or enhance certain outcomes.

For example in sports science, sporting analytics companies analyse individual player performance and make recommendations that can be used by the coach. If a player was to follow only one strategy, they would become predictable and beatable, hence having multiple options to choose from allows a player flexibility to try different techniques and approaches which are continuously optimising his or her performance.

Intelligent Machines 2.0

This is just the beginning for machine learning in business.

Today, there is still a lot of effort for the human in creating and feeding algorithms to operate with precision and accuracy.

Tomorrow, Artificial Intelligence will enable a two way interaction. Machines will help to challenge our biases by asking questions that require additional or more precise data. Machines today are restricted to learn from data a human decides is relevant, this next wave will supercharge machine learning by providing machine’s the ability to navigate their learning. This human-machine partnership will also create benefits for leaders who will be free from biases to enable creative and insightful decisions.

Machine learning is becoming available to the masses. Through technological advances, such as the rise of cloud, computing power, scale and sophistication is available to all. The real challenge for executives to maximise on the opportunity of data driven decisioning is the internal culture.

The potential for many of our decisions, predictions and diagnoses to be informed by algorithms is here… 

… the human and machine collaboration is the key to unlocking that intelligence. 

About the author

Teradata‘s Ahmad speaks regularly at international conferences and events. After an undergraduate in computing, she followed a tangent into the world of life sciences, completing a PhD in data management, mining and visualisation at the Wellcome Trust Centre for Gene Regulation and Expression. Following this, she has also worked as a data scientist, building analytical pipelines for complex, multi-dimensional data types.

April 11, 2017  10:46 AM

LzLabs: The art of mainframe application ‘re-hosting’

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network written by Didier Durand in his capacity as VP of product management at LzLabs — the firm’s so-called ‘software defined mainframe’ product enables both Linux and cloud infrastructure to process thousands of mainframe transactions per second. It includes a faithful re-creation of the primary online, batch and database environments.

Durand writes as follows…

Modernisation is a continuum – machines need to be maintained and businesses are in essence built upon machinery. Enabling continuous modernisation is the key to future-proofing a business and those relying on legacy systems for core business processes run the risk of being unable to make use of new platforms and technologies in future.

History of the mainframe 

The mainframe began life as a powerful computing platform from which big business could scale resources and enhance functions to meet customer demand. Over the past half-century, it has been cemented as a pillar of industry. Today, over 70 percent of the world’s financial transactions are processed by a mainframe application.

There has long been a heated debate on whether or not mainframes are still viable platforms to run business from in a modern, competitive landscape, however this article is aimed at those who think an as-is ‘re-hosting’ approach to mainframe migration and modernisation sounds too good to be true – a long sought-after enigma. After all, the mainframe has been the subject of much pain (full application rewrite, discrepant results after migration, etc.) and anguish for those wishing to migrate and modernise their mainframe applications onto modern platforms and into modern languages.

While there are many reasons why one might wish to modernise their applications, there are just as many methods that have been used to enable that modernisation to take place. The ‘re-hosting’ approach is a classic in terms of system migration and there are a number of ways of doing so.

The latest and perhaps the fastest and most reliable method of doing this is centered around the (once-theoretical) idea that a migration can take place with no changes to the application code, so that the applications can be migrated onto the new platform as fast as possible, with no risk involved. Changes to code are often associated with system failures, as they have not been tried and tested across as many scenarios compared with the previous system.

So how does it work?

This type of “re-hosting” in essence, boils down to three stages:

  1. Packing up all of the application artifacts in their executable form on the source system
  2. Moving the package to the new platform
  3. Unpack and run

In order to achieve this, the system services running on the mainframe subsystem must be re-written natively for Linux. This includes systems for transactional activity, batch scheduling, indexed datasets and relational and hierarchical data.

To paraphrase Carl Sagan; if you wish to faithfully recreate a mainframe environment to run on modern, open platforms, you must first invent the universe.

A translator may also be needed in order to transpose the mainframe binaries, based on the assembly language of the mainframe architecture, into Intel x86 Xeon assembly language, so that it can be used on Linux. This process is called “Dynamic Instruction Set Architecture Translation.” These transformed binaries are then able to run in a managed container.

Tangible benefits 

There are numerous tangible benefits to this approach, including:

  • No issues with internal end-users, or with customers due to issues (discrepancies in results, etc.) as a result of re-writing the application code. The consequence of these issues, associated with traditional migrations, is often a cause of distrust emerging against the new system.
  • No need for retraining and no loss of productivity by end-users. They can continue with their daily tasks just as before – nothing has changed in terms of daily interactions with the application.
  • It’s an efficient way to validate the migration: iso-functionality makes the testing simple to define and validate. Results are either strictly identical or they are not. The approach doesn’t leave room for subjective interpretation, it’s entirely objective.
  • It enables easy automation of testing at large scale.
  • It solves the first half of the modernisation problem in a short time-frame. The applications have been migrated onto a modern long-term sustainable platform, based on very standard components, and can be operated by Linux sysadmins which are abundantly common in the job market, thus far-easier to recruit than mainframe sysadmins.
  • The cost savings presented by modern IT platforms are achieved instantly, before any longer-term application modernisation begins – a quick and highly positive ROI.

After the “re-hosting” takes place, applications can begin their modernisation process, using state-of-the-art technology, to rejuvenate the legacy application into a modern one, with a rich web-based interface, accessible through web services, restructured in containerised microservices, etc. It then becomes ideal for inter-operability.

A mainframe migration can be a bit like re-inventing the wheel. Traditionally, the mainframe applications would have to be re-invented before any migration can take place. In an as-is ‘re-hosting’ migration, the applications can be migrated straight away, with no changes to the application code. In essence, the mainframe has been reinvented, but it is operating on modern and open infrastructure – the benefits of which can be achieved immediately.


LzLabs’ software allows the executable form of legacy customer mainframe programs to operate without changes in what the firm calls ‘contemporary’ computing environments. The software enables mainframe data to be written and read in its native formats. This new environment works without forcing recompilations of COBOL and PL/I application programs or making complex changes to the enterprise business environment.

April 10, 2017  9:39 AM

Is RSI rescue in apps, or hardware?

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

Given the fact that there’s an ‘app for everything’, one would have thought that the software application developer community would, by now, have spent more time working on anti-RSI (Repetitive Strain Injury) apps to keep us protected from too much keyboard time.

Yes of course there is touchscreen and voice, but many of us still use keyboards all the time, especially those who spend their days down on the command line.

Pain diaries

The unfortunate reality is, if you happen to go sniffing around any of the major app stores looking for RSI-related apps, most of the software is flaky stuff related to hand exercises, yoga and stretching or so-called ‘pain diaries’.

What we might have (arguably, reasonably) expected was some form of collaboration platform that could plug into the apps we use all day long and alert us as to when we should stand up, move around and so on.

But not so much, so that’s why the following was of interest.

BakkerElkhuizen S-board 840

The BakkerElkhuizen S-board 840 Design USB keyboard that is described as compact i.e. meaning that it doesn’t include a numeric pad, which means the mouse can be placed much closer to the keyboard.

Research has shown that 90% of keyboard users rarely use the numeric pad. This means the S-Board 840 Design USB compact keyboard has greater comfort as it reduces the reaching distance to the mouse which significantly reduces strain on the shoulder and forearm.

Its design means its keys have dark letters on a light backgrounds which make them easier to read and are proven to contribute to increased productivity…. hmm, we don’ quite buy that claim, but let’s go on.


According to the supporting materials, “[The product’s] user-friendly keys include an extra wide space-bar and user-friendly positioned arrow keys. With many shortcuts, it’s the ultimate office, home and portable keyboard that makes typing feel pleasurable and fluid.  At just 2 cm thick it’s lightweight and fits easily in laptop bags . Lay-outs include US, UK, DE, BE, FR, Sw, Ge and others.”

User opinion

What is actually really nice about this product, if perhaps you are over 40-years of age, is that it kind of feels like the first keyboard you ever used on a bigger clunkier computer back in the 1980s. It has movement, but that movement is solid yet light. If you can remember the Research Machines 380z, well maybe a bit like that but with a more modern touch.

The CWDN blog is also trialling two vertical desk units from Varidesk and Ergotron, both of which we will aim to report on as hardware, but from a software perspective where possible.


The BakkerElkhuizen S-board 840 comes with multimedia keys and 2 USB ports (USB hub).

April 7, 2017  8:41 AM

Cirba VP: Brexit clears the skies for hybrid cloud

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network blog written by Ayman Gabarin is his capacity as senior vice president for EMEA at cloud infrastructure control software company Cirba.

Gabarin writes as follows…

To say that Brexit is causing confusion is an understatement. One day we’re told that businesses will suffer enormously when the UK leaves the European Union, the next that Brexit can only be good for both the country and its European neighbours.

Despite all the argument and speculation, however, nobody can say with any certainty what the consequences might be and, if there’s one thing business leaders hate, it’s any kind of uncertainty. Especially when it comes to IT and the burgeoning use of public cloud services which despite or, possibly, because they span geopolitical boundaries, will inevitably be affected by the UK’s exit from the EU.

Wrong time, wrong place

In many respects Brexit couldn’t have come at a worse time, as growing numbers of companies look to migrate workloads out of the data centre and onto public cloud platforms, many for the first time. Already grappling with concerns over data security, compliance and availability, many of those companies will have felt they were getting to grips with the issues, only to now be faced with even muddier waters that, because Brexit has yet to happen, are much harder to chart.

Where UK companies, for example, might have felt obliged to use services hosted within the EU in order to meet regulatory requirements, post-Brexit they could be emboldened to look further afield or retreat entirely and host applications only in the UK.

Likewise, European businesses may not want to continue using providers or work with partners based in the UK after the break, with widely differing opinions as to exactly what compliance standards will apply. Added to which, we’re already seeing currency fluctuations with yet more upheaval on its way as other European countries go the polls. So much so that some businesses could find it all too stressful and react by bringing already migrated workloads back in-house, where they at least have full control over their destiny.

The most likely scenario, however, will be for companies to take a more pragmatic approach to their use of the cloud. Instant scalability and pay-as-you-go services offered by Amazon, Microsoft and others are big incentives. Therefore, most will adopt a hybrid strategy, mixing public platforms with on-premise infrastructure, to not just provide a safe harbour from which to navigate choppy Brexit waters, but better equip the business to cope with future storms.

A growing trend already, Brexit and the uncertainty that comes with it, can only serve to focus minds and encourage greater commitment to the hybrid cloud model. More importantly, it will also motivate businesses to look more closely at where they host and how they manage application workloads in a hybrid world.

Simple cloud application choices, once upon a time ago

Before the word Brexit even entered the language, organisations were mostly challenged with binary decisions, such as what application workloads to migrate to the cloud closely followed by how best to monitor and manage workloads once they left the data centre. Post-Brexit, those issues are becoming a lot more pressing, further complicated by the desire to adopt a multi-cloud hosting strategy and migrate workloads in, out and between services within the cloud.

The cloud marketplace is also becoming a lot more diverse, increasingly dominated by a handful of major providers but, between them, offering a bewildering array of services. As a result of all this, the IT department is having to shift away from provisioning and operating infrastructure, to acting as a service broker, with a holistic understanding dynamic application requirements and the ever changing characteristics of all corporate infrastructure, both internally hosted and public cloud such as cost models, technical capabilities, and security options.

Application requirements vs. infrastructure options

Fortunately much of the information needed to make these decisions is available in the applications and platforms involved, waiting to be unlocked using predictive analytic models designed to evaluate detailed application requirements against the capabilities, availability and cost profiles of the different infrastructure options. Indeed it is only through deep analysis of all these variables that the best hosting environment can be chosen. Spreadsheet models and public cloud costing tools are a poor substitute and companies that fail to invest in the ability to perform this due diligence are taking a huge risk, and not just when it comes to dealing with the fallout of Brexit.

Change is one of the few constants faced by business IT teams and the best defence is a complete understanding of the hybrid environment in which modern IT systems need to operate backed up by investment in data-driven analytic tools, to not just react to change, but mitigate against the uncertainty that comes with it.

Editorial comment

Gabarin’s comments are wide ranging and (arguably) grounded in a sense of reality having worked with firms facing real cloud application workload data challenges. He makes these points, obviously, because Cirba’s Densify product is looking to address some of the operational issues here. It uses predictive analytics and an understanding of cloud platforms to do what the company claims is more than just monitoring and checking against Service Level Agreements (SLAs). The proposition from Cirba is that this technology can enable IT teams to become service brokers in their own right, whether affected by the imminent fallout of Brexit or simply looking to survive and grow in the wider global economy.

April 4, 2017  7:45 AM

Teradata CTO: beyond the data science hype cycle

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

Teradata CTO Stephen Brobst spoke at the firm’s day #2 keynote to add more colour to the data and services firm’s stance on how we should be working with analytics and all aspects of the data warehouse today.

A process for data

Brobst explains the process through which we must now work with data — measure, understand, optimise, execute, automate… with one step logically following the next.

Talking about the fact that we used to call big data analytics this thing called ‘data mining’, Brobst says that, “A lot of organisations are still overwhelmed by the hairball of data.”

Incomplete & dirty data

You’d think that big data analytics would be helping us at this point… but the problem is that the hairball is actually getting bigger. The answer, it appears, is being able to use machine learning to help automate our way out of the hairball and actually start to work with data that is incomplete and dirty.

Machine learning is an automation of the model building process (for data crunching)… but 95% of machine learning implementation is still using linear regression i.e the same technologies we were using in the 1990s to perform data mining.

Part of this is okay though, because we don’t need a human to program the learning elements of the data model that we need to bring to bear upon our current use of big data.

Riding the hype cycle

Brobst bemoans the fact that Artificial Intelligence today is off the chart in terms of it being too high up on the hype cycle.

“But don’t get me wrong, the hype of AI is going to continue,” said Brobst.

This is of course because all those CEOs and CTOs out there want to now go on the record to claim that they are using some form of AI in the approach to data… in reality though, if we take Brobst’s words as gospel, they are failing to use the more refined and strategic elements of machine learning automation intelligence that can build contemporary data crunching models.

AI in the neural network

Brobst explains that now, AI is enjoying something of a resurgence because it is being propelled by deep learning.

Can machines think?


But let’s also consider Alan Turing’s variation on this question.

Can machines do what we (as thinking entities) can do?

Yes! (at least in some cases, anyway).

How is deep learning differnet?

Multiple layers in the neural network (with intermediate data representations) can facilitate dimensional reduction in the data workload…. we can then interpret both linear and non-linear relationships with our data… and then, ultimately, we are able to derive patterns from data with very high dimensionality.

Why is this important asks Brobst? Because we now want to build highly scalable data systems for big data crunching with advanced algorithms, advanced real time data streaming (often using GPUs with highly parallel computation power) and create what we eventually call operational business intelligence.

The data scientists emerges

“It also helps us work with sensor data in the Internet of Things. We now need to be able to handle more data and work at deeper layers of the data network in the data warehouse. All these things are converging along with the emergence of the role of the data scientist as a job function inside organisations,” said Brobst.

With a million process variables (as seen in many complex modern businesses) … it is very hard to use brute force data processing techniques. This, essentially, a key rationale and validation for why machine learning is forming part of our new approach to data analytics argues Teradata’s Brobst.

Compelling stuff? Yes… as a CTO who clearly takes data science very seriously and wants to clarify and explain how we will be using data engineering in the future, this is the kind of keynote we need more of (Brobst talked about data, programming and data models for an hour without mentioning “customers” once)… deep learning for sure.

April 3, 2017  7:19 AM

Teradata Universe: It’s time to ‘instrument’ the business

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

Teradata has used its 2017 EMEA user, partner, customer, developer (and other-related practitioner) conference this week to explain how it now positions itself as a firm with a suite of data warehouse appliances and analytics products.

What does it take?

The firm has stated that it set out to use this year’s event to examine the following:

  • What it takes to move a production big data analytics workload to the cloud.
  • What it takes to develop an analytics company culture.
  • What it takes to move to a point where we can say — what get’s measured, gets managed.

The move to digital data business

So digging into the reality of what it means to become a digital data-driven business today… the audience tuned to Mikael Bisgaard-Bohr, executive vice president & chief business development officer.

Danish born Bisgaard-Bohr presented a series of thoughts on how firms are now becoming oh-so dependent on data. Commentators have discussed the state of the German automotive market and said that the golden age of engineering is now gone as we move to a future with software-driven vehicles that depend more on IT engineering than physical mechanical engineering.

Disney ‘instruments’ the business model

Bisgaard-Bohr also used the example of the Disney ‘MagicBand’ wristband system that allows the parks to track guests and know where they (within agreed privacy boundaries) and be able to provide them with access to rides, open their room doors, pay for restaurant meals and so on.

“If Disney can ‘instrument’ its operations with the MagicBand, then don’t tell me there is any industry that can’t be similarly data managed,” said Bisgaard-Bohr.

Not all data is created equally

So data is data in the data lake and in the data warehouse, right?

Well not so much… some data we need to clean, de-duplicate and start to use immediately. Other data we want to keep and use, but not just now. Other data we think there will could be some use for in the future, but we have no idea when, however we do want to still hold on to it… so we’ll just stick it in the data lake for now.

According to Teradata then, we need to build an infrastructure that we can manage data and allocate it dependent on its form, function and type.

Data will differ wildly dependent upon its:

  • Complexity.
  • Cleanliness and readiness for use.

Product news

The firm points to its latest product news which includes an all-memory update to its flagship Teradata IntelliFlex platform. This technology is focused on performance and storage density in a single cabinet.

The upgrades are driven by a move to all solid state drives (SSD), which makes it possible to reduce required datacentre space while delivering an increase in processing power for mission-critical analytics at the speed-of-thought.

Teradata also announced the new ready-to-run Teradata IntelliBase platform with its single cabinet support for multiple software technologies, re-deployable hardware and low, commodity hardware price-point.

Both products are offered on-premises, with Teradata IntelliCloud availability coming soon…. and both run the same Teradata database software to enable a high-performance hybrid cloud solution.

Page 1 of 9212345...102030...Last »

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: