CW Developer Network

November 5, 2019  8:51 PM

Scylla’s ‘scale-up & scale-out’ mantra for monster database power

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

There are databases. Then there are big data databases. Then, then there are super-performance high-speed big data databases.

And finally, there are real-time big data databases that run at hyper-speed built around an efficiency mantra that champions scale-up (and then…) scale out engineering.

This is the spin (the server hard disk kind, not the marketing kind) that Scylla wants to put forward to describe its performance of millions of operations per second (MOPS) on a single node.

The company claims that independent tests show a cluster of Scylla servers reading 1-billion rows per second (RPS) – plus (and here’s the point of all of this) the organisation says that this is performance that ranks ‘far beyond’ the capabilities of a database using persistent storage.

“Everyone thought you’d need an in-memory database to hit [MOPS] numbers like that,“ said Dor Laor, CEO of ScyllaDB. “It shows the power of Scylla’s close-to-the-metal architecture. For 99.9% of applications, Scylla delivers all the power [a customer] will ever need, on workloads other NoSQL databases can’t touch and at a fraction of the cost of an in-memory solution. Scylla delivers real-time performance on infrastructure organisations are already using.”

NOTE: As TechTarget reminds us, NoSQL (not only SQL) is an approach to database design that can accommodate for a wide variety of data models, including key-value, document, columnar and graph formats – over and above ‘traditional’ relational databases (where data is sits in tables and data schema is designed before the database is built) NoSQL databases are useful for working with large sets of distributed data.


But let’s stop a moment. Scylla CEO Laor said ‘close-to-the-metal’… so what did he mean by that? We’ve detailed a complete definition commentary piece here, but essentially it’s all about database software that works in close proximity to and with knowledge of the actual instruction set and RAM addresses of the hardware system that it is built to run on.

In the company’s benchmark example, we see Scylla tell us that the test involved scanning three months of data (some served from cache and some from storage), which resulted in Scylla reaching speeds of 1.2 billion data points per second.

NOTE: Data points per second (DPPS) refers to the number of individual records inside any given architecture (or type) of database schema that the database query engine and management plane can accurately read, in one second.

Scanning a full year’s data, all from persistent storage (with no caching), Scylla scanned the entire database at an average rate of 969 million data points per second.

With one day of data, scanning everything from memory, Scylla achieved 1.5 billion data points per second.

Scylla uses power of modern hardware such as bare metal servers — with its shared-nothing, lock-free and shard-per-core architecture, which allows it to scale up with additional cores, RAM and I/O devices.

NOTE: A shared-nothing architecture is an approach to distributed computing architecture in which each update request is satisfied by a single node (where a node itself can represent a processor, block of memory or storage unit) so that there is (hopefully) no contention among nodes. In shared-everything, any data task (processing, memory or storage) can be served by an ‘arbitrary combinations of nodes’, so there could be traffic and the potential for collisions.

Scylla says that it stands in ‘contrast’ to Cassandra, because “Cassandra’s ability to scale-up is limited, by its reliance on the Java Virtual Machine, which keeps Cassandra from interacting with server hardware. Where Cassandra’s threads and locks slow down as the core count grows, Scylla can take full advantage of all of a server’s resources.”

The company claims that the performance Scylla demonstrated in these (above noted) benchmarks has implications for real-world applications. For example, analytics jobs that previously took all day to complete can now be run continuously to provide ‘intraday’ (i.e. inside one day) reports.

As with any Scylla story, it’s a bit like drinking from a firehose and the company presents itself with a degree of unashamed swagger and confidence. There’s also a lot to cross-reference and learn (hence the three clarification notes above and the separate close-to-the-metal explanatory story) in order to take all this in.

Scylla (the company) takes its name from the Greek sea monster of the same name in Homer’s Odyssey… let’s hope this stuff is no fairy tale.

November 2, 2019  9:01 AM

CI/CD series – Plutora: best practices for continuous code

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network in our Continuous Integration (CI) & Continuous Delivery (CD) series.

This contribution is written by Jeff Keyes, director of product marketing at Plutora — the company is known for its technology which works to capture, visualise and analyse critical indicators of speed and quality of software delivery.

Keyes writes…

CI/CD done right will build high quality software faster, as it’s one of the foundational parts of DevOps. To ‘do’ DevOps, teams need to focus on decomposing the applications as well as integrating automated checks as much as possible. Which in essence focuses on building quality in, instead of inspecting the quality after the code is built.

By putting in checks to verify the quality throughout the process, you’re doing it closer to when the code was actually written, meaning there is a more frequent feedback loop between the developer and what is taking place. Therefore, best practice of continuous integration is for the developer to be the one that not only writes the code, but also writes the tests for the code to ensure it works effectively.

For this to be successful, these tests need to be run quite frequently. By frequently running the automated tests, fewer bugs slip through to follow on phases of testing. By using CI/CD practices instead of traditional methods, the IT team will overall end up with higher quality code once it’s done.

Taking this a step further, when there are individual teams coming together and collaborating on this, the code that they integrate together is also of higher quality. This is because the quality of the individual sub-systems and component are of higher quality and the team can focus on ensuring the quality of the integration points.

Buggy bugs

It can be difficult when an IT team is faced with trying to find a bug and they are not sure whether it’s in their own code, someone else’s code, or in the touchpoint between the code. These automated tests help focus from where the core problems originate.

This enables the team to take the next step of integrating the code lines more frequently. This is the foundation of Continuous Integration. Continuously integrating code lines closes the feedback loop between the teams. So when the IT team puts it all together, there is a reduced risk of any errors due to multiple bugs being present.

In traditional software development pipelines, another common point of failure is getting the built software onto preproduction and production environments. Automation of the deployment is an integral part of CI/CD pipelines.

Automation ensures consistency and speed and it means that the IT team can regularly deliver code together with minimal effort. Defects in the process are addressed in a permanent fashion. The Continuous Deployment portion of CI/CD again raises the overall quality of the applications being developed.

So when CI/CD is implemented correctly, it can lead to a higher quality code being produced. But while it can facilitate this, in reality it depends on what it is used for, and how individual teams work together to ensure that when the code is brought together, they are sharing the best version of it.

November 2, 2019  7:45 AM

Nutanix VP: IT evaluation challenges in the subscription transition zone

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

Evaluating IT is complex.

As much as the press has something of a love-hate relationship with technology analysts (did somebody say Magic Quadrant?), we do know that enterprise tech market analysis is a long, complex and convoluted process.

One of the biggest issues is the question of metrics.

Just as we’re getting really clinically competent on machine file log data metrics, the higher tiers of the industry itself are (arguably) becoming ever-harder to track… primarily as a result of the industry (and its multifarious platform-level technologies) moving so fast.

The old technology tool metrics just don’t apply anymore.

Vice president of investor relations and corporate communications at Nutanix is Tonya Chin.

Pointing to the way companies like Salesforce came to market with a cloud subscription model, Chin argues that for Salesforce, early growth was a little less striking than some of the traditional enterprise software firms saw when they were startups.

“They [Salesforce] started by seeing rapid adoption of their subscription model and the company then worked to develop a ‘land and expand’ dynamism. This meant that Salesforce could grow steadily, without worrying about the troughs (that inevitably come along with the spikes) of the traditional software sales and deployment model,” said Chin.

New KPIs, please

Nutanix’s Chin asserts that today, in the Hyperconverged Infrastructure (HCI) sector, we need a new set of Key Performance Indicators (KPIs) and ways to unravel the confusion caused by a complex IT world. She notes that some companies still prefer to deploy software and hardware on-premises — and pay upfront for it. Other customers wish to adopt the cloud as their default choice and pay via a subscription model. What is clear is nearly all customers would like the flexibility of doing both, allowing them to put workloads where they are best suited and pay for them how they choose.

“To serve customers, companies like Nutanix are moving from selling hardware to software. This is perhaps not such a revolutionary change; for many years, makers of appliances only wrapped their software in ‘tin’ for the convenience of buyers or to get a higher revenue number. But the hardware-to-software metamorphosis has certainly affected how influential industry analysts understand what’s going on,” said Chin.

As with moving from traditional software to cloud applications, moving from selling boxes to selling software can cloud (pun intended) what’s really going on.

Splintering the spikes

In short, just like Salesforce, Nutanix knows that it (increasingly) won’t get as many of the big revenue spikes large deals will cause, because the new model is based on providing customers with the flexibility of shorter contract durations, which means enterprise software organisations and their partners receive less revenue up front, but have the opportunity to expand over the life of the customer relationship so long as customers are happy.

“Making the transition from hardware takes time for analysts to register the change — and it becomes harder for watchers that aren’t immersed in what has happened in the enterprise software space to see the wood for the trees. Just as annual recurring revenue (ARR) came to be a big indicator of cloud service success, analysts need to see annual contract value (ACV) as the critical KPI of companies like Nutanix during times of transition, rather than focusing on billings and revenue growth in the near term. As the company completes its transition, traditional software metrics, including ARR, will also be important indicators of commercial success,” said Chin.

Nutanix’s Chin notes that Adobe has shown how to pull off a software-to-cloud transition. She also notes that Microsoft, under Satya Nadella, is doing something similar as part of a broader reinvention of its brand and, being such a closely watched company, the tech sector analyst community has picked up on that very quickly.

“But these things can take time. Remember, it’s not so long ago that some analysts considered Amazon to be a disaster in the making because it trades off profitability for growth. Today, it’s recognised as one of the world’s most valuable companies,” said Chin.

The Computer Weekly Developer Network spoke to Nutanix’s Tonya Chin to get her VP-level-view on how we need to look at the longer-term signs of value and move away from short-termism in our quest to gain a 20:20 perspective on the way the market is going to move next.

Nutanix VP Tonya Chin: No short-termism allowed if you want a 20:20 perspective.




November 1, 2019  11:48 AM

CI/CD series – Contrast Security: pipeline tools must be robust, sharp & safe

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network in our Continuous Integration (CI) & Continuous Delivery (CD) series.

This contribution is written by David Archer, sales engineer at Contrast Security — the company’s technology works to automatically detect and fix applications vulnerabilities as it identifies attacks.

Archer writes…

As we know, the purpose of a CI/CD pipeline is to automate the checks and processes involved with delivering software into production so that it can be performed in a consistent way and is not affected by human error.

What is sometimes overlooked is that the ‘CD’ part of CI/CD can refer to either Continuous Delivery or Continuous Deployment and there is a subtle difference.

The process of Continuous Delivery means delivering code into production-like environments in order that you can confidently push a release into production as and when required. With Continuous Deployment the process is completely automated, and code is deployed into production.

The process starts with a developer committing code to a repository which triggers the CI/CD pipeline to start. During this, a number of tests are run. These include integration tests that verify that the changes the developer made do not affect any other components of software.

There is no limit to the number (or size) of the changes. Having said that, developers are encouraged to keep changes small and commit code regularly so that should any tests fail, the root cause can be quickly isolated.

CI/CD pipeline

The use of a CI/CD pipeline arguably becomes more important as organisations move from deployment of a single monolith application to a loosely coupled architecture using microservices. When microservices are used there are exponentially more components to release and teams must be able to deploy changes into production independently of other teams.

By eliminating human error in the delivery process and running a large number of automated tests in the CI/CD pipeline, it is possible to significantly reduce the number of configuration problems or bugs seen in production environments. However, the benefits do not stop there.

The CI/CD pipeline also results in a shorter feedback loop for developers. This allows bugs to be remediated not only earlier, but faster, as the developer still has good context around the code that they wrote. When you couple these benefits with a fast, automated and reliable delivery process, then it is obvious why companies have embraced this approach to delivery.

Having said that, the learning curve for CI/CD can be steep. There are a number of CI/CD tools available but figuring out the right tool for your organisation will depend on a number of factors including your code languages, deployment environments and complexity of your build process. When using microservices, each team will normally be able to choose the technology stack which best serves their requirements and therefore a single build process will not be enough. The CI/CD tool needs to be able to accommodate each team’s preferred language, build process and test frameworks, so better to look for a flexible solution.

The bottom line is that a poorly configured or maintained CI/CD tool can result in intermittent failures which will quickly frustrate developers, so reliability of your pipeline is key. In order to get the most from your CI/CD tool it requires up-front effort to create a robust pipeline. However, this will yield numerous long-term benefits. If you have a few issues at the start it is important to deal with any failures early so that you maintain trust in the delivery process.

Once a CI/CD pipeline is in place there can be a temptation to jam additional tools into the process, but you should approach this with caution. There are a number of tools which can be disruptive to the pipeline including security scanners which may extend your pipeline duration past the magical 10-minute feedback loop for developers.

David Archer — knows his way around the (code) pipeline.

November 1, 2019  9:14 AM

CI/CD series – Sumo Logic: the pipelines provide continuous data – are you using it?

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network in our Continuous Integration (CI) & Continuous Delivery (CD) series.

This contribution is written by Mark Pidgeon, vice president of technical services at Sumo Logic — the company is known for its cloud-native machine data analytics platform

Pidgeon reminds us that the model for CI/CD aims to make the process for getting software into production easier, faster and more productive. The pipeline model defines the stages that software has to go through and then makes those stages more efficient through automation.

Pidgeon writes…

The reality of the pipeline works well for getting software out.

However, it’s not the whole story when it comes to the data that this creates over time. Each stage of the development process will create data… and those applications will add data once they are in production.

While it is a by-product of running applications over time, this information acts like a ‘digital exhaust’ that developers can tap into and use over time. This data can – if approached correctly – support a more accurate feedback loop for development, for deployment and for business teams.

By looking at the impact of any changes on application usage, it’s possible to understand how well the application is performing from a technical viewpoint as well as from usage and business demand perspectives too.

Business-wide CI/CD

But truly effective CI/CD should inform the whole business.

This involves thinking about observability from the start. Application logs, metrics and traces can each provide insight into the application and any changes taking place, but the sheer amount of data coming through from these continuous changes has to be understood in context. Putting this data together from different sources – particularly from software containers – can be difficult as it involves a lot of normalisation, after which it can be linked up to data from any CI/CD pipeline.

Linking data sets from application components ahead of release into production provides a strong ‘before and after’ set of data. This can be useful for looking at the impact of changes as part of any move to production.

This data is created continuously during any development, and it provides data that can be used over time for making better decisions across the software process.

However, it does not end there – this data can show where more work is needed in security and compliance, where business decisions affect IT and where IT choices affect business performance too. This continuous stream of data can be made useful for more people over time.

As more companies expand their CI/CD processes to speed up their businesses, the end result should be more efficiency across the whole operation. This approach should help companies improve decisions using metrics and data. However, this is a mindset change. Moving to a continuous intelligence approach can help.

Sumo Logic’s Pidgeon: effective CI/CD should inform the whole business.

November 1, 2019  8:52 AM

CI/CD series – Qualys: Continuous deployment & security – how to avoid blockages

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network in our Continuous Integration (CI) & Continuous Delivery (CD) series.

This contribution is written by Marco Rottigni, chief technical security officer EMEA at Qualys – the company is known for its cloud security, compliance and related IT asset discovery services.

Rottigni reminds us that developers have been adopting CI/CD to speed up getting software through their work pipelines. This has led to faster code pushes taking place, which in turn has led to more updates getting made. Rather than months or weeks to develop new updates, teams can push out new updates in days or hours.

For the largest and most automated businesses, multiple updates can be taking place every day. In 2014, AWS estimated a new release took place every second.

Rottigni writes…

Now, not every business works at the scale and speed of Amazon. However, most companies are in the process of trying to speed up their release process. DevOps teams are in place at most companies – according to Statista, only nine percent of companies were not adopting DevOps in some way. However, in the rush to improve CI/CD pipelines, it’s easy to miss security.

The challenge with implementing ever faster CI/CD processes is that they move whatever is inside through to production faster. When everything is working well, this means great code and new features out for the business. When it is not – and when people come under more pressure to deliver new features faster without thinking them through – it is all too easy for faults to slip through. For IT security teams, stopping these faults making their way through to production is an essential goal to reduce risk and meet compliance goals. However, it’s no longer a case of IT security being able to block everything. Instead, more collaboration is needed between DevOps teams and security.

Continuous security

With so many changes coming through, security can’t just act as a gatekeeper at the end of the process. Instead, security has to get embedded into the whole approach to development. What does this mean in practice?

Embedding security into development involves making the tools used for tracking security vulnerabilities easy to consume within the software development pipeline. Rather than providing these tools separately, they should instead provide any necessary updates or recommendations directly into the developer environment. This makes it easier for the developers to consume this information in the same way as other bugs or feature requests, then go about fixing them.

This feed of data on software vulnerabilities and potential issues will need to be prioritised based on the severity of the issues, how widespread those issues are, and the risk levels associated with them. For example, a problem in an application may only be present in one specific circumstance and for a small subset of customers – in this instance, the risk profile may be low. Equally, a problem may be widespread, easy to exploit and need fixing rapidly.

These serious issues can then be pulled higher up the priority list for developers, rather than relying on the security team to flag the risk.

Automation situation

Similarly, CI/CD pipelines rely on automation in order to achieve results quickly and deliver releases out quickly. Security teams can support this process through providing data into that pipeline to guide developer teams when it comes to improvements that are needed and where potential issues have been discovered. Automating the discovery process around new vulnerabilities is one option, but it can quickly lead to security alert overload.

Providing prioritisation data based on what IT assets exist in the development and production environments can help the development team further. This makes it easier for security to provide the right data and to help streamline the work around fixing issues. This is particularly important for cloud and container environments that change regularly. Containers can be spun up from base images quickly, work for a time and then ben torn down or deleted. In a CI/CD pipeline, those base images should be kept up to date and tracked for potential issues. However, they can easily be updated with additional code added to images over time that is not sanctioned or can be out of date compared to potential issues. Tracking this process and checking that all images are managed properly should be a priority, but building this into the wider pipeline process can help.

With so many changes taking place over time, driven by the CI/CD pipeline, the aim for security is to provide developers with the ability to deliver all the necessary value to the business while controlling all the risk involved too.

Using a mix of security vulnerability data, IT asset information and integration, security can support CI/CD pipelines to be more effective and more secure at the same time.

Rottigni: it pays to build a ‘wider pipeline process’ 

October 31, 2019  5:55 PM

CI/CD series – Veracode: up the code pipeline without going down the rabbit hole

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network in our Continuous Integration (CI) & Continuous Delivery (CD) series.

This contribution is written by Paul Farrington in his capacity as EMEA CTO at Veracode — the company is known for its Veracode Application Security Platform that provides an automated cloud-based service for securing web, mobile and third-party enterprise applications.

Not all CI/CD is created equally… and Farrington appears to know a thing or two about the shape of the beast we are trying to tame here.

Farrington writes…

Platforms such as GitLab, AWS Developer Tools or Azure DevOps are increasingly favoured by teams looking to optimise software development, unlocking faster velocity by leveraging the power of the cloud to perform software build and tests in parallel.

In the past, development teams needed to wait-in-line, for on-premise build servers to complete tasks before having work completed. Today, the economics of these online CI/CD platforms allow steps to take place across a distributed compute model, so that release pipelines can be compressed.

Code isolation situation

I want to make note of the fact that, out in the real world of live production code, different CI/CD release patterns can be extremely powerful for DevOps teams.

Isolated code can be controlled by feature flags and routing rules that limit how many users experience new code. The ‘canary release pattern‘ is a popular example technique, as it allows deployments to take place at any time of day, but the user experience is carefully controlled so that initially only a trickle of users are exposed to new functionality. The gradual increase of flow of users can be controlled by scripts which look for errors appearing in logs. Based on certain tests, the roll-out could be automatically paused or rolled-back, without engineers needing to be paged in the middle of the night. These tests might also include automated security scans, for example with Dynamic Analysis, which could conceivably ‘vote’ for a rollback if say, a SQL injection flaw is found on the main login page. Once the isolated code has passed a series of tests during the release pipeline and once in production, it can then be allowed to survive if negative test results are below a designated threshold.

People also often ask about CI/CD in terms of how development sits next to [continuous] deployment.

Today, very few organisations have reached the level of maturity to support (automated) continuous deployment. CI/CD are terms which provide teams with the ability to release ‘deployable’ code at will, subject to when the business wishes for that to happen. In the recent Puppet State of DevOps report (2019), 56% of those surveyed agreed that business reasons mean that they choose not to automatically deploy. Consider that only 7% of developers, according to the same study, are able to remediate critical security vulnerabilities in less than one hour.

Getting funky on functional

Developers will typically write unit tests to help validate the correctness in a function. These can range in complexity, but in essence, they should be deterministic. For example, it could be validating that a function correctly sums inputs properly, or perhaps that it sorts a range of data points in the expected way. Because of their nature, unit tests will usually run quickly and are a normal part of a build process, fitting well within a CI/CD pipeline.

Increasingly, developers are using tools to perform security unit tests on code changes. In comparison, functional tests will often seek to ‘exercise’ a running application by executing a script and grouping together a number of user actions so that the behaviour of the application can be validated. Functional tests can also be undertaken within a CI/CD pipeline, but will generally require more time to complete. Depending on the nature of the project, the development team may elect for some functional tests to be performed following deployment, whilst retaining the ability to roll-back the release if need be.

Xebialabs has produced a lovely periodic table of DevOps tools that covers the full spectrum of tooling, such as source code management, deployment, security testing and release orchestration. Some of the online cloud platforms are becoming extremely popular, because they offer either free or cheap entry points for teams to experiment with development patterns, including how they configure CI/CD pipelines.

Veracode is one of the security elements (in blue) represented in the table, and its job is to ensure that developers are able to scan and receive results alongside other testing tools. Providing free integration tools is therefore a key principle to promoting plug and play interoperability, giving the development team the maximum amount of choice.

Jump, but don’t boil

Our advice is to be pragmatic. Be ambitious but realistic about what can be achieved in each ‘jump’. Take people with you by explaining the changes that need to be made and how these might be achieved. Don’t boil the ocean and try to automate everything in one step.

Anticipate (and perhaps, even embrace) failure by being willing to address feedback so that the next iteration will be perfect. A really useful read on that point is Epic Failures in DevSecOps, it’s free and provides insight into failure and potential success. Lastly, collaborate and try to future proof the decisions you make. For example, automating everything with the exception of security will only mean that you end up shipping vulnerable software more quickly.

Veracode’s Farrington — knows his code pipeline… and his onions, presumably.



October 31, 2019  8:02 AM

CI/CD series – GitLab: everyone deserves to deploy at high velocity

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network in our Continuous Integration (CI) & Continuous Delivery (CD) series.

This contribution is written by Jason Lenny, director of product management, CI/CD at GitLab — Lenny tweets at @j4lenn.

Lenny reminds us that CI/CD has roots in Agile… and both have roots in Lean manufacturing — they are all about applying iteration in different domains. Agile focuses primarily on software development, CI/CD on software delivery. Practicing DevOps means optimising the entire system end-to-end and applying iteration in every part.  

Lenny writes…

Iteration in a direction leads to an optimal solution. Iteration without direction leads to spaghetti code and Frankenstein UX. Simply because you are making changes faster doesn’t mean they are the right changes. The feedback loop is critical.

This is a big reason why GitLab focuses so much on delivering an entire DevOps platform in a single application. When Agile planning, Source Code Management, CI/CD and Monitoring are in a single application, you can have a single conversation thread throughout the entire Software Development LifeCycle (SDLC) and all teams from PMO and Dev to Security and Ops can have visibility into the feedback loop.

CI/CD enables smaller batch sizes.

Smaller changes mean code is easier to review, easier to test and easier to roll back when there’s a problem. This can result is fewer bugs and reduced defects in production. But, if you aren’t reviewing, aren’t testing adequately, and aren’t monitoring to know when to roll back then you won’t realize these benefits.

How frequent should frequency be?

Folks like DORA have proven the correlation between deployment frequency and business success.

More seems to be better, but at a certain scale, your system starts to break down. A key limiting factor is how long it takes your pipeline to complete. If you are deploying more frequently than your total pipeline time, you run the risk of breaking the master. The potential for conflict arises when additional changes get merged while a pipeline is still running on a previous change. High velocity means everything can pass in staging but still fail in master. Very sophisticated engineering organisations like Google, Facebook and Uber mitigate this by building complex custom functionality to queue and sequence changes ensuring that once a pipeline starts, other code can’t get merged ahead of that change. At GitLab, we thought everyone deserved to be able to deploy at high velocity and keep master green. It’s why we have queueing and sequencing built-in as a feature called, ‘Merge Trains’.

CI/CD is a functional change, DevOps is a cultural one.

You can technically implement an automated system to build, test, and deploy, but that doesn’t mean your development and operations teams are collaborating on the end-to-end lifecycle or that you are continually learning and improving. I’d say CI/CD is a subcomponent of DevOps. DevOps is about people, process, and technology. A key technology enabling DevOps is CI/CD. The disruption is intertwined.

Once upon a time, GitLab was a code repository, but today GitLab is an entire DevOps platform. GitLab includes built-in Agile project planning, monitoring, application security, package management, Kubernetes management, a WebIDE, and more, all in addition to source code management and CI/CD.

In regards to effective CI/CD, the ability to run on multiple machines is a result of our CI/CD Runner architecture. Runners are the agents they do the work of each CI/CD job in a pipeline. You can have many Runners each on their own machine so you can run many jobs in parallel. Runners can even auto-scale to spin up new instances when additional capacity is needed and spin down when it’s not. This means when everyone commits their changes at 4pm on a Thursday in preparation of Friday sprint demos, your CI/CD platform can handle all of that work quickly, without anything waiting in a queue, and without having to pre-provision and pay for your peak capacity during the week.


At GitLab we use CI/CD to mean continuous integration, delivery and deployment.

Choosing whether you want to deploy to production manually or automatically is simply a toggle in the configuration. Everything else about how you build your pipelines and use the product is the same regardless of which you choose. In fact, every job in a pipeline, not just your ‘deploy to prod’ job, can be either automatic or manual.

Your ruleset can even allow certain types of changes to go to production automatically while others require manual approval. Our design philosophy has been to build powerful small primitives that can be used as building blocks in diverse and flexible ways.

On the question of how functional tests differ from unit tests in CI/CD environments… the difference between unit and functional tests is the same whether those tests are automated or manual. Unit tests verify individual functions and methods in the code, while functional tests verify that actions return the expected results.

We also need to ask, if development teams use CI/CD tools to automate parts of the application build and construct a document trail, then what factors impact the wealth and worth of that document trail?

This is an interesting question, the most impacting factors are the compliance needs of the business and the scale at which they operate. In highly regulated environments, being able to produce a document trail can make the difference between passing or failing an audit. Similarly, a robust and accessible document trail significantly aids the troubleshooting process. When the system is down for 30 minutes, the value of the document trail differs depending on if you lost $100 during that 1/2 hour or $100,000.

From a technical perspective, the value of the document trail is based on how good of a trail it is. Do you have a single place to go to get all of your information, or are you having to cobble together a trail from data siloed in multiple different DevOps tools? Simplifying the toolchain is a key tactic for improving the usability and value of your records.

GitLab’s Lenny: positive about code from the start.

October 30, 2019  8:03 AM

CI/CD series – Perforce: polishing the production pipeline

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network in our Continuous Integration (CI) & Continuous Delivery (CD) series.

This contribution is written by Ido Benmoshe, VP of product and solution consulting at Perforce Softwarethe company is known for its software application developer-focused revision (and version) control system.  

One of the questions we have asked centres around the fact that we know that CI/CD is responsible for pushing out a set of ‘isolated changes’ to an existing application, but how big can those changes be… and at what point do we know that the isolated code is going to integrate properly with the deployed live production application?

Benmoshe writes…

CI/CD is aligned with Agile iterative development however CI/CD is a build, test and release automation process that complements an Agile development process. Smaller increments in development make CI/CD more efficient and allows faster feedback.

In general, the best practice is to launch a CI pipeline is onto the smallest code increments possible either on each 'check-in' or at a high frequency that suits the specific application, such as hourly or daily. However, keeping tight control over the number of changes made to each iteration requires developers to be disciplined -- and even minor changes may still have impact on critical application functionality. Reducing the chance of integration issues requires rigorous testing at multiple tiers – and it requires a high percentage coverage of unit, static code and functional testing, as well as load and endurance testing. Effective CD also requires full automation in building the application environment including the lower testing environments, so that they resemble production as much as possible. It is important to implement detailed monitoring and diagnostics capabilities in the application across all environments to ensure early problem detection and fast resolution.

CD players

In a fast paced and highly competitive digital environments, CD is implemented in order to accelerate time to market of new capabilities, defect fixes and security vulnerability remediation while improving the application quality. Companies therefore implement CD because it provides business ROI and drives innovation and competitive advantages. Failures are usually related to incomplete adoption or poor implementation of deployment automation and more often insufficient test coverage and test automation. Effective continuous feedback does require adoption of monitoring and diagnostics tools.

You asked me does CI/CD automatically lead to fewer bugs?

In general yes the rigorous testing pre and post check in 'shifts left' or identifies defects earlier, therefore resulting in faster problem resolution and reducing costly defects from making their way to production. Frequency depends on the business needs, scale of the environment and available resources.

DevOps is a wide space but if we refer to DevOps in the context of infrastructure automation then more effective CD environments implement infrastructure automation for provisioning all the environments however CI can be implemented on manually provisioned systems as well.

Defining deployment vs. delivery

CI/CD contrasts with Continuous Deployment (despite the proximity of the term), which is a similar approach in which software is also produced in short cycles but through automated deployments rather than manual ones… so allow me to clarify this point more clearly.

Continuous Delivery provides the organization with the capability to release high-quality software at will.

Continuous Deployment is essentially an extension of the process in which the changes are also pushed to production automatically without manual approval or intervention. This final production promotion step does not make the process less efficient but rather more related to the business needs.

Image: Perforce

October 29, 2019  1:02 PM

Pegasystems lays down the do’s & don’ts of RPA

Adrian Bridgwater Adrian Bridgwater Profile: Adrian Bridgwater

This is a guest post for the Computer Weekly Developer Network written by Francis Carden is his capacity as vice president for digital automation and robotics at Pegasystems — the company is known for its work in customer engagement software.

Robotic Process Automation (RPA) is already helping businesses reduce inefficiencies and deliver cost savings, but it’s not perfect.

IT teams must consider in equal measure what RPA should and shouldn’t be used for.

DO use RPA

DO use RPA…to automate simple tasks.

RPA typically automates simpler and mundane work. Years of business process optimisation has made it such that a lot of work that remains is not simple, repetitive work but rather complex and fragmented. This impacts RPA getting to scale. Companies switching from one unattended RPA vendor to another eventually realise it’s the complexity of the process/applications that prevent scale, not RPA itself.

Businesses should use attended RPA to automate simpler tasks within complex processes and optimise every worker rather than automating every worker. 50% automation on 1000 people is better than 100% of 20.

Also use RPA as a means to digital transformation, not the end game.

The goal of digital transformation is to transform business. RPA has a place – optimising parts of enterprise, but ultimately to digitally transform, RPA would require replacement with deeper intelligent automation technologies for many applications. Automating bad processes on old, costly-to-manage legacy apps are still bad processes on those applications.

No set-and-forget

In addition, RPA is not scalable for APIs and too breakable for IT to use as a set-and-forget. According to a recent Pega survey of 509 global decision makers on RPA, 87% of respondents experience some level of bot failures. Further, forty-one percent of respondents said that ongoing bot management is taking more time and resources than expected. RPA automations can be exposed to APIs to redesign new digital interfaces, but these RPA connections will be temporary.

There are far better technologies to automate intelligently for serious transformation than RPA and these include:

  • Low-code designs which supports centralised governed rules rather than siloed macros to eliminate cumbersome spreadsheets.
  • Centralised email management with natural language processing to read and respond to emails rather than RPA on outlook.
  • Orchestration and low-code together to replace old, tired processes and accelerate ending the life of many legacy apps that encourage poor processes.

DON’T use RPA…

Don’t use RPA to automate complex processes.

The core of RPA is to automate screens through the user interface (UI) – to read something off the screens and do something with the data before possibly sending the data back into them. RPA doesn’t need much intelligence for this because all RPA is limited by its operating system and the ability to support the multitude of complexly compiled applications that run on it. Some intelligence is being used via optical character recognition on-screen, but in these cases, app automation is even slower and more brittle than object-based automation systems in some RPA products.

Complex processes embedded with complex applications and GUIs are a no-go – they are fragile, expensive to support and break frequently.

Also, don’t use RPA for long-term enterprise application integration (EAI) projects.

Using RPA – UI automation – for long-term EAI projects is not architecturally sound. A simple UI change down the RPA chain could seriously impact EAI – disastrous as the objective of enterprise architects is to use robust, scalable, APIs connected using proven, secure technology. However, these aren’t always available in time to fulfil short-term optimisation or digital transformation. In this case, it is fine to at least consider RPA as a stop-gap/bridge so the value can be delivered.

It will be interesting to see how RPA develops. RPA’s lack of scale is at least drawing attention to bad processes and eventually, maybe, people will start to plan to transform them instead.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: