Two years ago, the drinking game of the tech industry was “Docker, Docker, Docker.” Venture Capitalists like Adrian Cockcroft were calling the growth of the Docker ecosystem “accelerating to plaid”. At the time, Docker was hosting two major events a year, in the US and Europe.
One year ago, Docker changed direction on several fronts:
- Narrowing the number of DockerCon shows to just one.
- More involvement with Microsoft, even after rumors of a buyout attempt.
- Moving from a community-centric strategy to more of a “delivered solutions” strategy with several acquisitions and more embedded technologies (e.g. SwarmKit).
- Clashing with the community about the separation between Docker, Inc and the docker open source project.
So as DockerCon 2017 approaches, what could developers and operators expect from Docker?
Hint…here’s what world famous industry prognosticator @cloud_opinion expects during the week:
[note – Rancher did announce “Project Longhorn” earlier today to focus on block storage in containers, which is often stated as a challenge for customers. But building storage systems from scratch is very, very difficult, as CoreOS learned last year with “Project Torus“, and DellEMC’s Chad Sakac has been saying for years.]
Docker vs. OCI vs. Containerd
A couple years ago, Docker worked with many other vendors to create the Open Container Initiative (OCI) and begin to standardize on the a container format and runtime. From this work, projects like “runc” were created. Last year, Docker separated the runtime into a new project called “containerd” and this was recently donated to the CNCF. All of these efforts were targeted at creating distinction between Docker, Inc (commercial company) and docker project (open source), as well as to create a container standard that could evolve to being stable (and boring) for the long-term.
runc is now at v1.0RC3 and close to shipping, and containerd is at v0.2 and will take time to complete and mature. Since Docker is now planning to maintain their runtime on a different schedule than the containerd project, and bundle it in the Docker EE commercial offering, it will be interesting to see how Docker messages the idea of an independent container standard vs. Docker offerings.
Kubernetes vs. Docker Swarm
After the SwarmKit announcement in 2016, some analyst thought that Docker Swarm would become the leading container scheduling/orchestration framework because it was built into the Docker Machine engine. Much debate and discussion (here, here) ensued about how the broader docker open source community would respond to this change, both in technology and how Docker communicated roadmaps with the community.
Based on recent survey data (here, here, here, here), that prediction has not played out in the marketplace, with Kubernetes having the lead (by varying amounts). Over the past year, the Kubernetes community has grown significantly in terms of developers (1100+), projects and offerings in the marketplace.
Will Docker attack the technical merits of Kubernetes, or will they potentially announce an integration with Kubernetes platforms?
Given the continued push by Microsoft to work closely with Docker, I expect that we’ll hear more about Docker for Windows Containers. Microsoft has been more open to open source technologies of late, as well as working with popular container technologies (Mesos, Kubernetes, Deis Workflow, etc.). I predicted that the two of them would get together a couple years ago, but we’ll see if that becomes strategic or something more.
More Details on Customers or Revenues?
After taking $180M in VC funding, over 7 rounds, the market is very interested in hearing some specifics about customers and more importantly revenues. Docker changed the pricing and bundling model earlier in the year with Docker EE, as well as new details about how long they will support customers on a given release. While Docker is not a public company and is not obligated to disclose any financials, those details are desired by both customers and partners to understand the viability of this market and to determine what their commitment to Docker should be long-term.
This past week, monitoring company Sysdig released the findings of their latest container survey data. The report used actual monitored data as the source, rather than use user-feedback via survey. The reports focused on many different container usage aspects, with orchestration being a key element.
These results are not surprising, as we’ve see the number of community developers (1100+) working on Kubernetes grow to 5-8x as large as any other container orchestration framework (including Mesos, Docker SwarmKit and Cloud Foundry Diego).
The other aspect that was not surprising is that the number of “DIY” scheduling frameworks – essentially people building their own container platforms with scripts and other customer tooling – continues to decrease over time. Given how quickly projects are moving (quarterly releases) in the container ecosystem, and the associated applications being built on top of them, teams are quickly coming to realize that they aren’t adding business value by constantly having to update and maintain those platforms.
The Growing Kubernetes Community
Obviously one survey does not define an industry or an industry trend. But lets look at what has been happening around Kubernetes over just the last 6 months.
- The number of contributors to Kubernetes continues to grow and diversify.
- Microsoft has announced support for Kubernetes on Azure (as well as acquiring Deis, and hiring Kubernetes co-creator Brendon Burns from Google).
- IBM added Kubernetes support to their Container service.
- Red Hat continued to have QoQ and YoY growth, bolstered by OpenShift (Kubernetes) adoption.
- Over 60+ Kubernetes projects are now available in the marketplace.
- KubeCon Berlin drew 50% more people than the KubeCon Seattle event just 4 months prior, with nearly 2500 expected in Austin in December.
Kubernetes is Getting Easier to Use
A year ago, the container community went through some growing pains as Docker decided to embed Docker SwarmKit into the Docker Engine, rather than keeping the engine independent from the orchestration/scheduling layer. Some of the finest analysts in the industry thought this might destroy Kubernetes, claiming that Docker would just become the default orchestrator/scheduler because it was simple to setup. Instead, this seem to bring the Kubernetes community together to focus on this challenge (which was well known at the time) and a broad range of projects has come of this re-focus on setup and simplifying Kubernetes environments. Projects like minikube, minishift and several “Quick Start” efforts on public clouds (e.g. on GKE) have made it much easier to setup a local or remote cluster. And free training from Katacoda and the CNCF is expanding the knowledge base of certified operators and developers. Throw in 250+ meetups around the world and the community of Kubernetes users and operators continues to not only expand, but make it simple to get engaged and learn.
If you’re an IT Operator, you most likely fit into one of three categories:
- Silo’d Technologist (Networking, Storage, Security)
- Trying to Make Sense of Software-Defined Data Center technologies (e.g. SDS, SDN, HCI)
- Trying to Make Sense of DevOps
If you’re in one of the latter two groups, then your learning curves (or curiosity) tends to follow these trends:
- Where do the lines blur/overlap between technologies that were once silo’d (e.g. Servers, Storage, Networking, Virtualization)?
- How to apply automation to the new instantiations of these technologies?
- How to learn enough programming (or advanced scripting) to drive that automation?
- How to drive the monitoring/logging/analytics tools that can be applied to these faster moving, software-centric environments?
- How to think about operations as a set of lifecycle pipelines, to better align to the application lifecycle pipelines that development teams use?
Let’s face it, that’s a long list of stuff to understand. In order to be effective in a world that will begin to value speed and agility more than stability, the requirement to understand quite a bit about the entire operations “stack” will be critical. Just as we’ve seen developers begin to become more “full stack developers”, I believe that we’ll soon begin to see IT operations teams targeting “full stack operations” skills.
A great example of where this is already happening today is with containers. Depending on who you ask, containers can either be a developer-centric technology or an operations-centric technology. And if you’re focused on using containers to run applications in production, you’re most likely going to leverage some sort of CaaS or PaaS platform. Within that platform, here are all the things that not only touch containers, but are deeply integrated into the platform architecture:
- Platform OS Management
- Image Management (Container Registry)
- Image Security and Scanning (Container Registry)
- Software-Define Networking
- Platform Availability Management
- Platform Inter-Process Communication Security
- Software-Defined Storage
- Application and Platform Monitoring
- Application and Platform Logging
- Platform Troubleshooting
It’s a full-stack of requirements without the clear lines of separation that used to exist for physical environments or virtualized environments. All of these things are now in the domain of the full-stack operator to automate and evolve to keep up with the increasing speed of developers. And the goal of all that work is to make the developers even less aware of the Infrastructure and Operations.
Now let’s take it a step farther. Suppose your developers want to start using one of the “serverless” frameworks instead of using containers? In theory, the lines between developer and operator would then begin to blur even more. But who will monitor and troubleshoot those serverless applications? It will require an even deeper knowledge of how the applications work in this new environment.
Will we see “full-stack operator” take off as a real thing, or just a resume thing (like “full-stack developer”)? Time will tell. But we’re definitely in the early days of it becoming a necessity as the platforms and applications that run many business quickly evolve.
Maybe I’ve been reading too many articles about Artificial Intelligence and Machine Learning. Maybe I’ve been listening to too many TV segments about how automation is the reason we don’t have as many factory jobs in the United States. Or maybe I’ve just been curious how how coal miners can be become coders. One way or another, the “jobs in the future?” question has been knocking around my brain quite a bit recently.
And then I was sitting in a session at CloudNativeCon and KubeCon in Berlin, listening to the technical nuances of how the default Kubernetes scheduler works, and it dawned on me that my kids will most likely not work on jobs that exist or are mainstream when they become adults.
Huh? How did you jump from future jobs to Kubernetes and back again? Well, it goes something like this…
In a couple weeks, we’re going to record our 300th episode on The Cloudcast. Since we always like to do something different on the anniversary shows, we thought it might be fun to have our kids and spouses record a quick bit about “What do you think Aaron and Brian talk about on the podcast?. I suspect it’ll be something like, “my dad works on computers…”. And if I really think about it, that’s probably all they will ever know. And in the future, there probably won’t be nearly as many people that have to care about the low-level nuances of computers because people will just use resources in the cloud. It’s already happening. One of the podcasts I listen to (Software Engineering Daily) recently had their 20-something host mention that he’s never touched any computers (or networks or storage) in a data center before as he’s only ever used the public cloud.
But all of this is OK, because as we move from one generation to the next generation, the jobs of the past don’t always carry forward. For example, my grandfathers were farmers and auto-factory workers, respectively. While farms and factories still exist, the type of work they (individually) did two generations ago no longer exists. My father was an accountant for much of his career, but those are now being replaced (or augmented) by Artificial Intelligence. My wife’s father was a long-haul truck driver, and we can see where those jobs are going. At one point, my mother worked for a company that made voice PBX systems. Those are being replaced by cloud services like Twilio and Amazon Lex.
The real question going forward is what jobs will still exist in 15 or 20 years for the next generation of workers?
If you’re in the 35-45 year old demographic, in the tech industry, there is a frequent conversation that you have with colleagues of a similar age. Is a mix between “What’s next?” and “How are you dealing with things?. At that age, you typically have a few common characteristics:
- A reasonably well paying, mid-level to senior role, often times moving up a management chain.
- An established place of residency, which may or may not be located close to the headquarters of your company.
- In many cases, a spouse and a family, with children that are beginning to have close ties to their school or clubs/groups associated with their activities.
If this was your situation and you lived in the Silicon Valley over the last 5-7 years, there were lots of stories about rising housing prices, and the shift in focus from the hardware companies in the South Bay towards the “social media and apps companies in San Francisco” that targets worker in their 20s who would be OK in the crowded apartments around the city. Many of the people in those situations looked to move to places like Portland or the Seattle suburbs to stay in the tech industry, but have a more family-friendly lifestyle.
But if you live outside Silicon Valley (or Seattle, Portland and maybe Boston….although Boston is less IT tech-centric these days), it’s a different story. For the last 10-15 years, the technology which allowed remote work has improved considerably – home broadband, WiFi, mobile devices, video conferencing, chat tools like Slack, social media, etc. – and many companies were able to grow their workforce and add expertise without requiring them to be in expensive headquarters buildings.
But it’s beginning to look like the trend around remote workers might be changing. Last year, there was a meme on Twitter about the requirement that workers “must relocate to San Francisco“. Unless you were moving from Seattle, NYC or Boston, you could expect at least a 3x cost-of-living increase with that move…so hopefully your options were going to be worth something BIG!
This year, we might be seeing the beginning of a more serious trend for the 35-45 crowd – IBM’s new policy which requires many workers to come into a HQ or Regional office, or their roles will no longer continue to exist. While IBM is pitching as a move to improve productivity, teamwork, and morale, it’s not hard to alternatively look at this as a move (in theory) to reduce the number of older employees without having to create an HR policy about “getting rid of older, more costly workers”. Many of those remote workers will not want to disrupt their established lives and hence may not be willing to pack up and move.
Where this may start to become interesting is if many of the other “incumbent” vendors, especially those that are struggling with declining margins or legacy business models, decide to follow IBM’s lead. Many of those companies have either been acquired by Private Equity firms (e.g. Dell/EMC) or have those PE firms influencing strategies at the board level (here, here, here). In those scenarios, cost-cutting tends to have as high a priority as innovation, sometimes more.
IBM is only one data point, but it’s a policy that will be watched by other management teams and should be watched by the 35-45 crowd too. The days of the remote office worker might be significantly changing…if you plan to stay with one of the incumbent vendors.
I get asked all the time about the differences and benefits of the various container technologies, especially on the orchestration side – often called the “container platform”. At some point, it’s better to write down the answer than to keep repeating it, so here goes.
Even though most people only think about containers in the context of Docker (circa 2014), containers and container platforms have been around for quite a while. But lets start with the modern-day platforms that most people know – Heroku, Google AppEngine, Cloud Foundry, OpenShift, AWS Beanstalk, dotCloud.
The Early PaaS Days
Around 2009-2011, most of these platforms came into the market with the goal of making is easy for application developers to “just push code into production”. By most definitions, these platforms were known as Platform-as-a-Service (PaaS). To some extent, these platforms did a great job of making things simple for developers, by hiding all the complexity of IaaS and associated services (LBaaS, DBaaS, Authentication, etc). And under the covers, most of these platforms used some variant of Linux containers to isolate and run the developer’s applications. These were the original homegrown/DIY container schedulers/orchestrators.
But these early platforms also had some limitations:
- Some of them only ran in a specific cloud environment
- Some of them only supported 1 or 2 languages
- Some of them were open-source, while others were various levels of proprietary software.
- Most of the open-source platforms didn’t have large community support.
So in the 2011-2013 timeframe, there were plenty of articles written about the death of PaaS and how it hadn’t gained the massive adoption.
The Open Standards Emerge
And around 2014, two very important things happened. dotCloud went out of business, but it spun out the docker project, simplifying how containers could be used by developers to package their applications. Usage of docker for container packaging took off, and the market now had a pseudo-standard for packaging applications. In addition, Google decided to open source the Kubernetes project and the Mesos project spun out of work at Twitter and UC Berkeley. This provided the market with choices about open source container schedulers/orchestrators that had web-scale DNA built-in. These three activities helped developers know where to focus their energies in building scalable container platforms.
Open Container Platforms Everywhere
As a move into 2016-2017, the market has evolved quite a bit since the early 2009-2011 platforms. Almost every major platform (or public cloud) is now supporting either Kubernetes or Mesos as the scheduler/orchestrator, and the docker project as the format for container image and runtime. The Open Container Initiative (OCI) is further standardizing the container image and runtime formats into standards that are less aligned with a single company. Kubernetes has gained almost 4x the developer support of alternative container schedulers/orchestrators, mostly because of the openness of the community and because it supports many application types with the default scheduler.
The future moving forward will continue to accelerate at a rapid pace. With all of the community focus on open frameworks and standards, businesses can feel much more confident in the platform decisions they are making for the next 5-10 years.
For the last 7-8 years, we’ve seen a large increase in the number of infrastructure software companies that were funded by Venture Capital companies. Companies such as Cloud.com (CloudStack), Piston Cloud (OpenStack), Active State (Cloud Foundry), Docker (containers), Mesosphere (Mesos) and many others were funded to contribute to large open source projects, as well as commercialize that software.
These open source projects have become the new “standards”. The projects and the code have replaced the function of previous standards-bodies like IEEE or IETF in defining how infrastructure would work. Working code has become the new “standard”.
While some of these infrastructure companies have been acquired by traditional infrastructure companies (such as HPE, Cisco, EMC, and Oracle), almost none of those transactions would be considered “big exits” in terms of VC returns.
The VC community is at a cross-roads with open source. As we recently discussed with Scott Raney from RedPoint Ventures, open source is an opportunity and a challenge for VCs. It can enable their portfolio companies to invest less in software and more in engineering. But it can be difficult to monetize their projects unless their gain a significant community following and user-base. This challenge has also been addressed several times on the a16z podcast with former Nicira CEO, and current Andreessen Horowitz VC Martin Casado (here, here). The challenge for VCs is that many developers only want to work with open source software.
This brings up an interesting question…
If the returns for VC aren’t going to be at the levels their funds require, except in very special cases, would it be unexpected to see VCs begin to stop investing in this space?
So how would the markets react if that VC funding were to slow down or stop entirely? There’s a few interesting areas to consider:
- For the last decade, open source communities have been rapidly innovating new areas – cloud, containers, big data, databases, IoT, software-defined infrastructure, etc. If the funding isn’t there for driving innovation or on-going engineering, then where will the new innovation come from? Is it possible that we’d see a resurgence of proprietary software offerings?
- Many public cloud companies are leveraging open source infrastructure to deliver their cloud services, both behind the scenes and as on-demand services. Will they be able to continue to add new functionality as the same rate that they have over the past few years?
This will be an interesting trend to track in 2017 and 2018. With more businesses looking to leverage multiple cloud (or hybrid cloud) services, and build consistent operational models across clouds, it could be challenging without those broad open source standards.
It’s been a little more than six months since I last wrote about “serverless” and a good bit of change has occurred in the market during that time. In a nutshell, serverless has evolved from a niche topic to one that is gaining more technology options and more actual deployments are starting to be recognized at various meetups and events.
For a high-level view of the marketplace, this is a really nice list of services and technologies that are available today. This is another great source of information. As you can see, serverless is covering much more than just AWS Lambda, and it extends from services (e.g. Email, CI/CD Pipelines, etc.) to frameworks to entire platforms focused on vertical elements of a business technology stack (e.g. payments and eCommerce).
A number of trends are evolving across the industry:
- Serverless or FaaS (Functions as a Service) is becoming more polyglot. The breadth of supported languages has evolved from Python or Node.js to include a much broader scope.
- Serverless is becoming either an add-on or native service to many of the CaaS (Container-as-a-Service) or PaaS (Platform-as-a-Service) platforms. For example, many new projects for serverless have been built to run on Kubernetes – Kubeless, Fission, Funktion, Funcatron. At some point I would expect that there will be some consolidation of ideas between these projects and Kubernetes will eventually have a native “functions” job, similar to “batch” or “stateful sets”. Cloud Foundry has announced that Spring Cloud Function is supposed to be available in 2017 for Spring Boot applications. I would also expect that Docker will announce some more formalized plan for serverless as the DockerCon2017 event in April, beyond the basic demos from 2016.
- Iron Functions, a previously proprietary technology is moving to open source and multi-cloud – Iron Functions.
- Serverless is gaining greater traction at events and meetups. ServerlessConf is now being held 3-4 times a year around the world, and events like FunctionConf are starting to pop up as well. These events are being augmented with a number of serverless-centric meetups, usually mixed into the local AWS, Docker, Kubernetes communities.
- Many companies are beginning to ask for serverless implementations that can either run within their own data center, or a framework that would consistently run on multiple cloud environments,
The big “next steps” for serverless will be a focus on enabling data services to easily tie into serverless computing frameworks – e.g. all notifications from a wide-range of data services to trigger serverless functions – and work somewhat consistently in multiple cloud environments. The is also a huge need to educate developers about which types of application patterns are emerging as the best fit for early serverless adoption.
While serverless movement is evolving quickly, and AWS is pushing for all-in usage by their customers, it is by no means a silver bullet for all applications. Serverless still has many limitations around operations. It still has limitations about what type of application can utilize the services. And most people still haven’t figured out microservices or containers or DevOps, so the prospect of breaking down applications into even more finite elements is not going to be an easy transition.
NOTE: We’ve decided to not spin off a separate Serverless podcast from The Cloudcast (@thecloudcastnet), but rather include serverless topics within the same podcast feed. But we are monitoring serverless activities and sending those out on the @serverlesscast Twitter feed.
This past week, I was having a conversation with an industry analyst about some of the fast moving trends in the industry – specifically about the growth of containers and Kubernetes. While the size of the container ecosystem is still being determined (here, here), it can often be difficult to measure because some leading companies are private, some public companies don’t break out individual product revenues, or users are leveraging the open source DIY (“trunk”) bits. But regardless of the revenues, many people are trying to determine how the docker project has grown to 6B container downloads and projects like Kubernetes have grown to over 1000+ developers in just over 1 year.
When looking at this space, there are several macro-level concepts that are driving this growth:
Businesses are becoming more digital
For many years, businesses have admired the speed at which web-centric companies have delivered innovation. Via startups and forward-looking companies, this pace of innovation is now becoming commonplace in many industries. At the core of this innovation is the people (developers), process (DevOps) and technology (containers, microservices) that have been pioneered at Google, Netflix, Etsy and others. The commercial delivery of containers and Kubernetes is beginning to bring these capabilities to companies outside of Silicon Valley.
Developers expect a self-service experience
For many years, public cloud services were called “Shadow IT”. They allowed anyone to get nearly frictionless access to IT-like resources through a self-service mechanism. Some IT vendors attempted to deliver similar capabilities to their on-premises products, but these were often via costly upgrades and additional tools which mostly looked like extensions of IT dashboards. The container-centric platforms are now embedding these capabilities, and often leading with a developer-centric UI (or APIs, CLIs) and integrated automation and scaling.
“Portability” is a design consideration
Whether or not people believe that Hybrid Cloud is a valid design approach, many companies are looking at how to leverage multiple cloud services in the future. And many of these companies are still interested in finding ways to make their technology choices portable between those clouds, in one way or another. Containers provide that portability in a way that VMs never really did. Containers can run anywhere a cloud provides a Linux host (and eventually a Windows host). Containers also provide a consistent “unit” to integrate with CI/CD systems, helping to evolve the speed and quality of application pipelines.
IT organizations continue to need cost-effective solutions
As much as “digital transformation” and “agility” dominate today’s IT headlines, many IT organizations are still being asked to consistently take costs out of their systems. For many companies, software licensing and repeatable manual tasks are the focus areas of those reductions because they are quantifiable. Platforms like Kubernetes are designed to be highly automated, as well as integrate with external automation platforms for deployment and testing. Combine this with the ability to run containers on bare-metal Linux hosts and the costs of the hypervisor can often be reduced (for appropriate workloads). This report from IDC shows how some of those cost savings can be measured in a container-based platform.
For the last couple years, the idea of “software eating the world” has gathered quite a bit of traction. While software-lead companies (Uber, AirBnB, Netflix, Facebook, etc.) have created considerable disruption in many vertical industries, the vast majority of companies are still struggling to managing the transition from existing monolithic applications to more agile microservices implementations.
Over the last couple weeks, I’ve had the opportunity to dig into what it means to balance monoliths and microservices with several industry thought-leaders (here, here). From those discussions, a few important considerations come through:
- Not every application needs to be built using microservices. Plenty of existing applications can be improved by rethinking the sprint process (length, frequency).
- Instead of focusing on monoliths vs. microservices, the focus should be on what is needed to build and ship software more rapidly. The focus should be on making the overall business more agile, and able to react to digital feedback loops about how users interact with those applications.
- The testing and deploying processes are just as important as the application building process. Many companies should initially be focused on how well they are organized and prepared (automated testing systems) to test and integrate software. This is often focused on CI/CD tools like Jenkins.
- The culture aspects of more to more modular, microservices-based platforms should not be under-estimated. It requires a different understanding about what it means to build an “independent service”, both from a technology perspective and an internal communications perspective.
- It’s critical to have a platform in place that provides developers with self-service access to agile infrastructure resources, and the platform should abstract many of the complexities (service discovery, high availability, networking, storage, auto-scaling and load-balancing) that developers face.
Managing a portfolio of applications can be complicated, especially as it goes through an evolution that involves more than just technology updates. Cultural shifts like DevOps, and organizational shifts like “2-pizza teams” can seems extremely complicated and uncertain in the early stages. Sometimes they require breaking out new organizations to prototype the new habits for a large organization. It’s often the willingness to adapt the culture and process to a more iterative model that lays the foundation for faster monoliths and more agile microservices applications.