We’re so done with DevOps. Today, it’s a case of DevOps + DataOps, but what does this latter term mean?
What is DataOps?
DataOps describes the creation & curation of a central data hub, repository and management zone designed to collect, collate and then onwardly distribute data such that data analytics can be more widely democratised across an entire organisation and, subsequently, more sophisticated layers of analytics can be brought to bear such as built-for-purpose analytics engines.
As more users start to demand data in more combinations in more locations in more use cases at more definable parts of the application and data analytics lifecycles, DataOps is there with its approach to distributed data architecture to serve this need.
As tamr database guru Andy Palmer puts it, “DataOps acknowledges the interconnected nature of data engineering, data integration, data quality and data security/privacy — and aims to help an organization rapidly deliver data that accelerates analytics and enables previously impossible analytics.”
For more here, let’s focus on TechTargets existing definition here and remind ourselves that as with DevOps, there are no DataOps software tools as such; there are only frameworks and related tool sets that support a DataOps approach to collaboration and increased agility.
“Such tools include include ETL/ELT tools, data curation and cataloging tools, and log analyzers and systems monitors,” notes the site.
What’s new in DataOps?
Why the back to school DataOps 101 you ask? Easy… firms are starting to use this ‘discipline’ (or approach) to drive new platform enhancements.
DataOps company Centiq has announced CenSQL, an open source contribution aimed at SAP HANA developers.
As a Command Line Interface developed in-house for its own needs, CenSQL is now offered free as part of Centiq’s effort to establish its innovations at the centre of the SAP HANA development community.
As HANA proves its worth in UK organisations, with some 74% of projects coming in on time and to budget, its developer community is growing strongly. These developers are increasingly spanning two different skillsets, that of development and those of operations, in what has become known as DevOps.
According to Centiq, “As the use of DevOps techniques increases amongst SAP HANA users, developers are struggling to meet continuously more exacting productivity targets. This constant battle to stay ahead of business demands can make the lives of IT professionals increasingly time-constrained. Basic processes, such as checking memory allocations, balancing workloads and avoiding any impact on production systems, cannot be neglected despite the constant demand for more efficient code production.”
CenSQL claims to makes life easier for DevOps teams as it:
- Provides a user interface with easily accessible system insights, allowing them to visually monitor key metrics during the coding process.
- Gives its users the ability to investigate and troubleshoot possible issues with data or system health and be more productive.
- Has many built-in commands useful for exploring/researching SAP HANA features.
Glyn Heath, CEO of Centiq has commented to say that his firm often has to break new ground, whether this means cutting new custom code, configuring new systems or creating from scratch new capabilities, like CenSQL.
“It became apparent that, CenSQL, just like Optiq, our state-of-the art monitoring and visualisation platform, was too good to keep to ourselves. The right thing to do was to offer it to the world as an open source download and share its ability to speed up HANA development,” confirmed Heath.
Not 100% open source, but open source-ish due to Java-like proximity, the latest news from Compuware sees the mainframe application development company announce the availability of Topaz for Total Test, a new package that brings ‘Java-like’ unit testing to the world of COBOL applications.
The firm is seeking to give “even novice developers” the chance to validate and troubleshoot changes they make to existing COBOL applications, if indeed those developers find themselves working in mainframe environments where COBOL exists.
Mainframes need love too
But let us not be quick to decry this fact i.e. many financial institutions, retailers and airlines run mainframes with a presence of COBOL.
Compuware claims that Topaz for Total Test “effectively eliminates” the notion that COBOL applications are legacy code that can’t be updated as frequently with the same confidence as other types of applications.
There is of course an ongoing loss of specialised COBOL programming knowledge… which leads to a risk and associated fear of introducing even the slightest defect into core mainframe systems of record.
“Compuware Topaz for Total Test transforms mainframe application development by automatically creating tests for logical units of code. Developers at all skill levels can thus quickly and easily perform unit testing on COBOL code just as they do in Java, PHP and other popular programming languages. In fact, Topaz is more advanced than typical Java tools, because it requires no coding and automatically generates default unit test result assertions for developers,” states the firm.
Visual analysis of even the most complex and poorly documented mainframe programs and data structures (Topaz for Program Analysis and Topaz for Enterprise Data).
Real-time quality control and error detection of mainframe coding syntax (Topaz integration with SonarSource)
Automated creation and execution of unit tests (Topaz for Total Test)
Agile cross-platform source code management and release automation (ISPW and integration with XebiaLabs)
“Unit Testing on the mainframe is overdue,” said Randy Dray, Director of IT at Pekin Insurance. “I’m glad to see Topaz for Total Test focusing on unit test automation.”
Compuware today also announced the acquisition of MVS Solutions, its fourth acquisition in the past twelve months.
In the wake of a handful of Linux projects pushing ever closer to consumer desktop adoption (think Ubuntu, Mint Cinnamon, Solus, Arch and Chrome OS)… members of the openSUSE Project have announced the next minor version of Leap — a professional Linux distribution for developers, system administrators… oh and yes, users too.
Specific ‘types’ of Linux
Now we are seeing Linux distributions in some part align themselves for specific types or use case or environment i.e./ e.g. Tails has been widely lauded as the best Linux distro with privacy and anonymity baked into it.
So then, in that regard, openSUSE Leap 42.2 Leap is made to give stability-minded users and conservative technology adopters peace of mind.
In terms of construction, openSUSE Leap 42.2 is powered by the Linux 4.4 Long-Term-Support (LTS) kernel and is a secure, stable and reliable server operating system for deploying IT services in physical, virtual or cloud environments.
“A selective process of including well-established packages in openSUSE Leap 42.2 gives new meaning to the term Linux Optimization; openSUSE Leap is simply the safe choice that offers Linux professionals a user-friendly desktop and a feature-rich server environment,” states the development team.
Leap is said to provide an ideal dev-to-production model with SLE (SUSE Linux Enterprise) for developers and system administrators who want to align their development and production environments.
The project team also reminds us that Leap also offers users a path to move to other operating systems like openSUSE’s faster more updated distribution Tumbleweed, with the newest upstream packages and software versions provided on a rolling release basis, or to an enterprise-level support system with SLE.
Bloomberg has published a blog post detailing its top five open source contributions of 2016 from a real commit history perspective.
Is that Bloomberg as in Bloomberg?
Well yes, that’s Bloomberg L.P. and its technology drives many of the world’s financial markets. Over 4,000 technologists define, architect, build and deploy complete systems to fulfil the needs of leading financial market participants globally, says the company.
The firm works with many (dozens) of open source projects and says that in the coming months it will be focused on involvement with Chef for configuration management; OpenStack and building mobile apps using Facebook’s React Native mobile application development framework.
So here’s a summary version of…
Bloomberg’s 2016 Open Source Contributions: Top 5 Projects
You may know the Jupyter Notebook as a web-based environment where programmers can do computational research with native support for code, maths and data visualization. This year saw the introduction of the JupyterLab preview, showing how future versions of Jupyter will provide a more desktop-like experience on the web. Bloomberg was one of three major contributors to this effort.
Bloomberg’s customers are dependent on the real-time delivery of financial data and information. The team is investing time and energy in V8’s high-performance, asynchronous processing functions, built on top of generators—also contributed through the Bloomberg collaboration with Igalia—to improve application performance.
The engineering team has been involved with the Lucene project and its Solr search engine for some time, ramping up contributions in mid-2012. Today,Bloomberg uses Solr as the basis of its Search-as-a-Service platform, home to more than 100 applications specially built to query unstructured text, tagged news, communications archives, legal databases and documentation.
You can read more here.
There’s support (in terms of platform and standards compliance) these days… and then there’s ‘first-class support’, is this the way the industry is going?
The eeGeo platform delivers a 3D representation of the world in which information, content, services and data visualisation can be presented in a visually attractive way that encourages discovery, interaction.
Specifically, it has released eegeo.js (based on the traditional open source 2D Leaflet.js, making it possible to use and deploy cool looking 3D maps just like any other leaflet.js base-layered map with an extra 2 lines of code) and a TfL (Transport for London) API to incorporate any public transport route/line/times into a map.
The firm explains that its maps are driven by libeegeo, a high-performance cross-platform C++11 library using OpenGLES and OpenGL.
According to the team, “This cross-platform library powers all of our platforms from a single codebase. This means reduced inventory, allowing us to stay quick and lean – we only have a very small core engineering team of 10.”
The company explains its evolution and says that it went from gaming first, to mobile second, then browser third.
With what amounts to literally hundreds of Linux distributions out there, Refracta version 8.0 and its suitability for the so-called “average user” might just be what makes it an above average contender.
Could Refracta be your next operating system of choice? As we know, a great OS needs to evidence that magical combination of utility, functionality, stability and robustness… and it also needs to have a decent user base.
Refracta is an offspring of Devuan Jesse GNU+Linux, which is a systemd-free fork of Debian.
Home computer users
Its makers say that it is an operating system designed for home computer users. It provides a simple and familiar layout that most users will find very comfortable.
“Refracta focuses on providing common applications and services that most users will need instead of trying to provide for more specialised uses. It provides a basic desktop with simple applications that are easy to use,” says the team.
The default install includes Internet focused software as well as graphics & multimedia plus an ‘Office’ type version of software along with various system tools and utilities. The default browser is Firefox ESR (Extended Support Release).
Hundreds more applications are available from a trusted source and can be easily installed.
This is a guest post for the Computer Weekly Developer Network blog ‘penned’ by Dr. Andy Phillips in his role as director of technical operations at LMAX Exchange — the company is a specialist in technology that facilitates the FX trading strategies of both buy-side & sell-side trading institutions.
It’s the question that keeps C-level execs awake at night: how do they attract and retain the best talent? To become best in class, you need the best people and that has become an ever-tougher task in the highly competitive world of tech talent.
No longer are the stock options, pool tables, free meals and beanbags enough to set you apart from the crowd. For star developers, there are plenty of good financial offers on the table too, so pay packets alone won’t necessarily seal the deal, especially among those who are most motivated by the technical quality of their working environment and peers.
City-chic is waining
Finance sector companies in particular, who may have relied on brand recognition and big paychecks to attract the best talent are now competing with companies that did not exist a decade ago. It’s no longer cool to work in the City compared to being part of a fintech startup in Clerkenwell trying to disrupt traditional finance. For more career risk averse developers, Internet giants like Facebook or Google have as much or more attraction than the giants of Wall Street.
In fact, the solution to what appears an HR problem lies in technology.
To attract the best technologists, you need to show a commitment to being the best place for technology. The best way to do that is to take a transparent approach: show them your code and run corporate open source projects that allow potential hires to assess you as a potential employer.
Deeds (or code) are louder than words.
Open source is an imperative
A commitment to open source might sound like a niche concern for the CTO, but it needs to be seen as a front-line business imperative for tech companies who want to build their brands among sceptical developers.
Developers don’t want to take what companies tell them at face value: they want to look under the bonnet, and assess the quality and design of the code for themselves. If you want to win credibility among the developer community and encourage the right people to your brand, you need to share your work and demonstrate best practice, not just talk about it.
The benefits to an open source approach don’t end with the positive impression it can help foster among developers.
By open sourcing, you also get the benefit of having your code peer-reviewed and tested, giving potential technical customers an independent check on what they are being promised by the sales team, helping build confidence in your technology. What’s more, popular open source software gains features and fixes bugs more quickly than in house software, lending itself to scale more readily than proprietary software.
No fly-tipping software onto GitHub, please
Of course, making a success of open source is not without its pitfalls and it brings with it a need to invest time and resources. You’ll need to have technologists answering questions, writing documentation, merging pull requests and generally tending any project. You need to put in to get back; it can’t just be a case of fly-tipping your software onto GitHub and assuming a grateful developer community will spring up around your abandoned code.
Another failure is to have an ‘open source strategy’ which amounts to no more than using it for free and not contributing back. This is strip mining open source software for your own benefit. You need to contribute and become part of a reciprocal community.
Open source should be core to how you gain corporate advantage and preference in the race for top talent.
It’s about showing leadership, proving your credentials and demonstrating that you are a company that understands the way that top talent you’re competing for wants to work. Open sourcing can both support your growth and act as a primary recruiting sergeant for the talent that’s hard to find and attract.
Cuts both ways
Moreover, the open-source approach in recruitment cuts both ways. It’s not just about creating the right impression within the developer community, but helping you identify the right talent in the first place. You can supplement the traditional CV by looking at a potential candidate’s open source footprint: activity on their github account, Stack Overflow reputation, participation in open source projects. Above all, you can learn about the quality and style of their output: what their work is like and how they might fit into your team.
With an open source approach, both employer and potential employee can find out more of what they need to know about each other; those small but important things that so often won’t surface in a generic interview, but which can be the making and breaking of a working relationship.
A prospective graphic design hire will bring their portfolio to an interview. The developers you are interviewing have their own online portfolio of open source software. As an employer if you don’t have a corporate open source portfolio you’re missing an opportunity. That’s ok, there are companies out there that still don’t have a website.
Source code speaks louder than words.
Because of this truth… if you’re worried about where your next generation of developers is coming from, go above and beyond the job spec and interviews.
Go open source to close the deal.
About the author and more…
LMAX Exchange (London Multi Asset Exchange) is a leading MTF for FX, regulated by the Financial Conduct Authority based in London, UK. Andy Phillips’ extended internal (but external facing) company blog is linked here.
Google is making news this week in developer circles. The search giant has come forward with a software fuzzing tool designed to fuzz open source code.
What is fuzz (testing)?
Fuzz testing or fuzzing is a software testing technique used to discover coding errors and security loopholes in software, operating systems or networks by inputting massive amounts of random data (called fuzz) to the system in an attempt to make it crash.
If a vulnerability is found, a tool called a fuzz tester (or fuzzer), indicates potential causes.
So then, the beta status Google OSS-Fuzz goal is to make common software infrastructure more secure and stable by combining modern fuzzing techniques with scalable distributed execution.
According to Google, Recent security stories confirm that errors like buffer overflow and use-after-free can have serious, widespread consequences when they occur in critical open source software. These errors are not only serious, but notoriously difficult to find via routine code audits, even for experienced developers.”
That’s where fuzz testing comes in. By generating random inputs to a given program, fuzzing triggers and helps uncover errors quickly and thoroughly.
Black Duck on Fuzz
The open source security team at experts Black Duck had plenty to say on Google’s news and contacted Computer Weekly’s Open Source Insider blog to make the following comments…
“OSS-Fuzz is a great new resource for the open source community to improve the quality of components and identify vulnerabilities very early. One outcome of this effort will be to increase user confidence in both open source software development as well as with specific components.”
“OSS-Fuzz potentially could become an essential tool for all open source projects during their development cycles, but will also increase the need for robust management systems. Many (Google) eyes will undoubtedly detect new vulnerabilities in older applications, which will flood the OSS community with new known risks to overcome.”
“Vulnerability reporting is a crucial component of any open source risk management to determine if any component used in the development of a product has disclosed vulnerabilities; even long after the product is released. Open source “consumers” will still need to be vigilant and take ownership of open source vulnerability management for their applications since there are millions of open source components and only a small portion of them will be tested with OSS-Fuzz.”
The team behind the Open Source Virtual Reality (OSVR) consortium has detailed news of a forthcoming Steam update that will accommodate for open platform Virtual Reality (VR) content.
The result of the new openness should mean that all games on Steam that are presented with the OSVR compliant logo will be ones that work with any VR headset.
Steam itself is a digital distribution platform primary designed for games. It offers multiplayer gaming and social networking services — its games are supplied over an Internet connection and are therefore automatically updated on the user’s machine. Other features include in-game voice and chat functionality.
The ‘problem’ that exists right now in the VR headset wars is that we have a handful of major manufacturers (including Oculus Rift and PlayStation VR) who each operate with a dedicated storefront but, crucially, their user options deny access to players who visit the store while using competing headsets.
The result is that software application development pros working to produce VR games have to end up porting releases to several platforms — a practice that, very arguably, does not push for the greater good in terms of games quality and development.
As reported on Gamespot by Jason Imms, “The backbone of OSVR is its software development kit (SDK), which allows developers to build support for all of the VR headsets on the market into their games. The intention being to reduce the hardware fragmentation that is forming around the big three.”
Could Windows every become open source? To be honest, although that discussion is fun to have, it’s really not on the cards right now or up for discussion. This was the avowed verbalisation made by more than one of the senior Microsoft developer division employees at the firm’s recent Connect () 2016 conference in New York.
Microsoft is in fact far more focused on it’s ‘any developer, any app, any platform’ message (rather than it’s previous ‘Windows Everywhere’ concept) and the firm has a lot to get right now in terms of helping to brings its tools for other platforms online (such as Visual Studio for Mac driven by Xamarin and .NET combined strengths) effectively.
So could any of the Windows upstarts ever supplant the Microsoft take on Windows? Ubuntu-based Zorin OS does indeed come with a taskbar on its GNOME-based desktop. A main system menu is also located on the left.
This is a stable distro of Ubuntu with a raft of customisations and desktop optimisations. Other press are lauding it as the best of both Windows and Mac OS X in one.
According to the Zorin blog, with a tap of the logo/super-key on your keyboard, you can instantly see everything that’s happening on your computer from the Activities Overview.
Every window and every workspace comes into view so you can browse, rearrange and organise your apps as they’re running. New workspaces are added dynamically, so you’ll never run out of room on your desktop says the team.
Others will like the universal search function.
“As soon as you start typing in the Activities Overview, you can search for anything on your computer and beyond. Not only can you find your installed apps, Universal Search also suggests new apps to download from the Software store as you type. In addition, you can now search for files and even inside of apps. Thanks to search providers, apps can extend the functionality of the Universal Search, so you can search for a city to find the local time or enter in a maths sum and see the answer live without even opening the Calculator app,” write the team bloggers.
You can read more and download at http://zorinos.com/