Next week at Interop 2017 in Las Vegas, I’m giving a talk about managing containers. The focus of the talk is to look at the expanded interactions that are required as engineers move from having a single container, running on their laptop, to moving it into production. It looks at how much developers need to know about containers to get their applications working, and what operations teams need to plan for in terms of container scheduling, networking, storage and security.
Breaking down the talk, there are three critical messages to take away.
The Need for Container Platforms
Platforms that manage containers have been around for quite a while (the artist formerly known as “PaaS”), just like Linux containers have been around for much longer than docker. But as containers are becoming more popular with developers, as the native packaging mechanism for applications, it becomes increasingly important than operations teams have the right tools in place to be able to manage those containers. Hence the need for container platforms, and the emergence of technologies like Kubernetes.
The Developer Experience Matters
As platforms transition from PaaS to CaaS, or some combination of the two, it’s important to remember that the container is just a packaging mechanism for applications. It’s critical to make sure that developers are able to use the platform to rapidly build and deploy applications. This could means that they package the application on their laptop using a container, or push their code directly into a CI/CD pipeline. In either case, the container platform must be able to take that application and run it in a production environment. The platform shouldn’t restrict one development pattern or another.
Operational Visibility is Critical
While containers bring some interesting properties around packaging and portability, it’s important for operational teams to realize that they have different characteristics from virtual machines. Containers may run for a few seconds or for long periods of time. This means that the management, monitoring and logging tools have to be re-thought in order to be valuable in a container-platform environment.
The discussions and sessions at ServerlessConf Austin ’17 were a good mix of emerging technology and existing use-cases. The high level message is that serverless allows developers to focus entirely on their applications and all (OK, most) of the infrastructure and operations challenges get handled by the underlying services.
But as with any emerging technology, after a year+ of learnings, the focus of “what’s next?” begins to evolve. In discussions with several early-stage users, they said that there are two big areas they expect to see highlighted in 2017/2018:
- The evolution of frameworks from being focused on the functions, to being focused on the events and connected services. In particular, the Serverless Framework was mentioned as one that will need to evolve their focus from functions to events.
- The set of corner-cases and advanced use-cases that can’t be addressed with current services and tools.
From Functions to Events – The Natural Platform Evolution
If we go back a couple years, when AWS Lambda was first announced, it had somewhat limited functionality and had limited language support. Fast forward a couple years and the number of other AWS services that can now interact with Lambda has grown significantly, as has the number of supported languages. Services like API Gateway, Kinesis and CloudFormations are now capable of invoking Lambda functions.
Moving outside the AWS ecosystem, and we now see Azure beginning to expand not only their Azure Functions capabilities, but also the tools that connect Azure Functions to other services. At ServerlessConf, it was highlighted that Azure Logic Apps now supported over 120 connectors, to a mix of Azure and 3rd-party services.
As we move from being focused on the code within a function to the connections with other services, the mindset begins to change from “solving technical challenges” to “thinking about business logic”. With the broad range of connectors that will become more readily available in the marketplace, across many services, it’s not hard to imagine a line-of-business leader being able to bring together a new application to address to business model concept with very limited effort. It begins to democratize the development process.
This move from thinking about functions to thinking in terms of events and connectors also creates the possibility for more asynchronous business interactions. Instead of thinking about the end-to-end process of a transaction or process, the steps of interaction can be asynchronous and more loosely coupled. Or they can become more micro-interactions, eventually allowing a more customized experience to be tailored to individual customers.
There are still many use-cases and corner-cases to be worked out as serverless moves the focus from functions to events, but it is definitely an area that the community at ServerlessConf was spending quite a bit of time thinking about and discussing possibilities.
This past week was ServerlessConf in Austin. The event has now done a world tour over the last year, starting in New York in 2016 and coming back to the states in 2017.
We have been following the serverless space for over a year now (see @serverlesscast), but this was the first time we had a chance to attend an event in person. I went down there looking to better understand four key areas:
- The Community – Who is attending, what are they working on, and what level of progress have made?
- The Market – Is this as an area where people are using it for real business challenges, and is there a market to make money from this technology?
- Cloud options vs. Open options – Most of the discussion about serverless so far has been about services delivered via public clouds (e.g. AWS Lambda, Azure Functions, Google Firebase, Auth0, Netlify, etc)
- Market Investment – Is this a market where a lot of VC money has already been placed, or is funding coming from somewhere else?
The events are hosted by A Cloud Guru, which not only does training for cloud services such as serverless, but they have built their entire business on serverless technologies. They have done a great job building the community and attracting end-users that are building interesting things with serverless.
The show attracted about 250 people for the Day 1 hands-on training sessions, and almost 450 people for the Day 2/3 talks and networking. The audience still seems to be people that are doing early-stage projects with serverless, and it hasn’t (yet) been overtaken by the vendors. Nearly everyone in attendance was working on some serverless project for their business, from large (beta) rollouts to entire businesses being delivered via serverless architectures.
Serverless is still in the early days, as we don’t have any companies publicly announcing their revenues at this point. AWS is currently the leader, as their AWS Lambda services have been out in the market the longest and attracted the early-adopter following. But Microsoft Azure Functions had a very strong presence, showcasing now only Azure Functions but also a broad range of event-connectors through their Azure Logic Apps. Google has a split portfolio between Firebase, which has been a Backend-as-a-Service for a while and the beta Google Functions. Applications frameworks like Serverless Framework and Go Sparta are also starting to emerge.
There were several businesses talking about their real-world use cases, including consumer companies like iRobot and large Enterprise companies like Accenture.
Cloud options vs. Open options
Most of the event was focused on cloud services for serverless, but the market does have a few open options that are beginning to emerge. IBM has donated OpenWhisk to the Apache Foundation. Kubernetes has a few projects (Funktion, Fission, Kubeless) that are maturing for on-premises or cloud deployments. Also, a new project from stdlib was announced, called FaaSLang. Since one of the key value points of Serverless/FaaS is only paying for usage, it will be interesting to watch if any on-premises or open-source offerings catch hold in the market.
The amount of VC funding that was present, via startup funding, wasn’t very large at ServerlessConf this week. There were smaller companies such as IOpipe, stdlib, Fauna, A Cloud Guru and Serverless Framework, but none of them had raised a large amount of money yet. The market is still trying to figure out how quickly developers will be attracted to this new mode of developing applications, and if there will be any white space left after the public cloud providers begin to roll out more serverless tools and services.
Two years ago, the drinking game of the tech industry was “Docker, Docker, Docker.” Venture Capitalists like Adrian Cockcroft were calling the growth of the Docker ecosystem “accelerating to plaid”. At the time, Docker was hosting two major events a year, in the US and Europe.
One year ago, Docker changed direction on several fronts:
- Narrowing the number of DockerCon shows to just one.
- More involvement with Microsoft, even after rumors of a buyout attempt.
- Moving from a community-centric strategy to more of a “delivered solutions” strategy with several acquisitions and more embedded technologies (e.g. SwarmKit).
- Clashing with the community about the separation between Docker, Inc and the docker open source project.
So as DockerCon 2017 approaches, what could developers and operators expect from Docker?
Hint…here’s what world famous industry prognosticator @cloud_opinion expects during the week:
[note – Rancher did announce “Project Longhorn” earlier today to focus on block storage in containers, which is often stated as a challenge for customers. But building storage systems from scratch is very, very difficult, as CoreOS learned last year with “Project Torus“, and DellEMC’s Chad Sakac has been saying for years.]
Docker vs. OCI vs. Containerd
A couple years ago, Docker worked with many other vendors to create the Open Container Initiative (OCI) and begin to standardize on the a container format and runtime. From this work, projects like “runc” were created. Last year, Docker separated the runtime into a new project called “containerd” and this was recently donated to the CNCF. All of these efforts were targeted at creating distinction between Docker, Inc (commercial company) and docker project (open source), as well as to create a container standard that could evolve to being stable (and boring) for the long-term.
runc is now at v1.0RC3 and close to shipping, and containerd is at v0.2 and will take time to complete and mature. Since Docker is now planning to maintain their runtime on a different schedule than the containerd project, and bundle it in the Docker EE commercial offering, it will be interesting to see how Docker messages the idea of an independent container standard vs. Docker offerings.
Kubernetes vs. Docker Swarm
After the SwarmKit announcement in 2016, some analyst thought that Docker Swarm would become the leading container scheduling/orchestration framework because it was built into the Docker Machine engine. Much debate and discussion (here, here) ensued about how the broader docker open source community would respond to this change, both in technology and how Docker communicated roadmaps with the community.
Based on recent survey data (here, here, here, here), that prediction has not played out in the marketplace, with Kubernetes having the lead (by varying amounts). Over the past year, the Kubernetes community has grown significantly in terms of developers (1100+), projects and offerings in the marketplace.
Will Docker attack the technical merits of Kubernetes, or will they potentially announce an integration with Kubernetes platforms?
Given the continued push by Microsoft to work closely with Docker, I expect that we’ll hear more about Docker for Windows Containers. Microsoft has been more open to open source technologies of late, as well as working with popular container technologies (Mesos, Kubernetes, Deis Workflow, etc.). I predicted that the two of them would get together a couple years ago, but we’ll see if that becomes strategic or something more.
More Details on Customers or Revenues?
After taking $180M in VC funding, over 7 rounds, the market is very interested in hearing some specifics about customers and more importantly revenues. Docker changed the pricing and bundling model earlier in the year with Docker EE, as well as new details about how long they will support customers on a given release. While Docker is not a public company and is not obligated to disclose any financials, those details are desired by both customers and partners to understand the viability of this market and to determine what their commitment to Docker should be long-term.
This past week, monitoring company Sysdig released the findings of their latest container survey data. The report used actual monitored data as the source, rather than use user-feedback via survey. The reports focused on many different container usage aspects, with orchestration being a key element.
These results are not surprising, as we’ve see the number of community developers (1100+) working on Kubernetes grow to 5-8x as large as any other container orchestration framework (including Mesos, Docker SwarmKit and Cloud Foundry Diego).
The other aspect that was not surprising is that the number of “DIY” scheduling frameworks – essentially people building their own container platforms with scripts and other customer tooling – continues to decrease over time. Given how quickly projects are moving (quarterly releases) in the container ecosystem, and the associated applications being built on top of them, teams are quickly coming to realize that they aren’t adding business value by constantly having to update and maintain those platforms.
The Growing Kubernetes Community
Obviously one survey does not define an industry or an industry trend. But lets look at what has been happening around Kubernetes over just the last 6 months.
- The number of contributors to Kubernetes continues to grow and diversify.
- Microsoft has announced support for Kubernetes on Azure (as well as acquiring Deis, and hiring Kubernetes co-creator Brendon Burns from Google).
- IBM added Kubernetes support to their Container service.
- Red Hat continued to have QoQ and YoY growth, bolstered by OpenShift (Kubernetes) adoption.
- Over 60+ Kubernetes projects are now available in the marketplace.
- KubeCon Berlin drew 50% more people than the KubeCon Seattle event just 4 months prior, with nearly 2500 expected in Austin in December.
Kubernetes is Getting Easier to Use
A year ago, the container community went through some growing pains as Docker decided to embed Docker SwarmKit into the Docker Engine, rather than keeping the engine independent from the orchestration/scheduling layer. Some of the finest analysts in the industry thought this might destroy Kubernetes, claiming that Docker would just become the default orchestrator/scheduler because it was simple to setup. Instead, this seem to bring the Kubernetes community together to focus on this challenge (which was well known at the time) and a broad range of projects has come of this re-focus on setup and simplifying Kubernetes environments. Projects like minikube, minishift and several “Quick Start” efforts on public clouds (e.g. on GKE) have made it much easier to setup a local or remote cluster. And free training from Katacoda and the CNCF is expanding the knowledge base of certified operators and developers. Throw in 250+ meetups around the world and the community of Kubernetes users and operators continues to not only expand, but make it simple to get engaged and learn.
If you’re an IT Operator, you most likely fit into one of three categories:
- Silo’d Technologist (Networking, Storage, Security)
- Trying to Make Sense of Software-Defined Data Center technologies (e.g. SDS, SDN, HCI)
- Trying to Make Sense of DevOps
If you’re in one of the latter two groups, then your learning curves (or curiosity) tends to follow these trends:
- Where do the lines blur/overlap between technologies that were once silo’d (e.g. Servers, Storage, Networking, Virtualization)?
- How to apply automation to the new instantiations of these technologies?
- How to learn enough programming (or advanced scripting) to drive that automation?
- How to drive the monitoring/logging/analytics tools that can be applied to these faster moving, software-centric environments?
- How to think about operations as a set of lifecycle pipelines, to better align to the application lifecycle pipelines that development teams use?
Let’s face it, that’s a long list of stuff to understand. In order to be effective in a world that will begin to value speed and agility more than stability, the requirement to understand quite a bit about the entire operations “stack” will be critical. Just as we’ve seen developers begin to become more “full stack developers”, I believe that we’ll soon begin to see IT operations teams targeting “full stack operations” skills.
A great example of where this is already happening today is with containers. Depending on who you ask, containers can either be a developer-centric technology or an operations-centric technology. And if you’re focused on using containers to run applications in production, you’re most likely going to leverage some sort of CaaS or PaaS platform. Within that platform, here are all the things that not only touch containers, but are deeply integrated into the platform architecture:
- Platform OS Management
- Image Management (Container Registry)
- Image Security and Scanning (Container Registry)
- Software-Define Networking
- Platform Availability Management
- Platform Inter-Process Communication Security
- Software-Defined Storage
- Application and Platform Monitoring
- Application and Platform Logging
- Platform Troubleshooting
It’s a full-stack of requirements without the clear lines of separation that used to exist for physical environments or virtualized environments. All of these things are now in the domain of the full-stack operator to automate and evolve to keep up with the increasing speed of developers. And the goal of all that work is to make the developers even less aware of the Infrastructure and Operations.
Now let’s take it a step farther. Suppose your developers want to start using one of the “serverless” frameworks instead of using containers? In theory, the lines between developer and operator would then begin to blur even more. But who will monitor and troubleshoot those serverless applications? It will require an even deeper knowledge of how the applications work in this new environment.
Will we see “full-stack operator” take off as a real thing, or just a resume thing (like “full-stack developer”)? Time will tell. But we’re definitely in the early days of it becoming a necessity as the platforms and applications that run many business quickly evolve.
Maybe I’ve been reading too many articles about Artificial Intelligence and Machine Learning. Maybe I’ve been listening to too many TV segments about how automation is the reason we don’t have as many factory jobs in the United States. Or maybe I’ve just been curious how how coal miners can be become coders. One way or another, the “jobs in the future?” question has been knocking around my brain quite a bit recently.
And then I was sitting in a session at CloudNativeCon and KubeCon in Berlin, listening to the technical nuances of how the default Kubernetes scheduler works, and it dawned on me that my kids will most likely not work on jobs that exist or are mainstream when they become adults.
Huh? How did you jump from future jobs to Kubernetes and back again? Well, it goes something like this…
In a couple weeks, we’re going to record our 300th episode on The Cloudcast. Since we always like to do something different on the anniversary shows, we thought it might be fun to have our kids and spouses record a quick bit about “What do you think Aaron and Brian talk about on the podcast?. I suspect it’ll be something like, “my dad works on computers…”. And if I really think about it, that’s probably all they will ever know. And in the future, there probably won’t be nearly as many people that have to care about the low-level nuances of computers because people will just use resources in the cloud. It’s already happening. One of the podcasts I listen to (Software Engineering Daily) recently had their 20-something host mention that he’s never touched any computers (or networks or storage) in a data center before as he’s only ever used the public cloud.
But all of this is OK, because as we move from one generation to the next generation, the jobs of the past don’t always carry forward. For example, my grandfathers were farmers and auto-factory workers, respectively. While farms and factories still exist, the type of work they (individually) did two generations ago no longer exists. My father was an accountant for much of his career, but those are now being replaced (or augmented) by Artificial Intelligence. My wife’s father was a long-haul truck driver, and we can see where those jobs are going. At one point, my mother worked for a company that made voice PBX systems. Those are being replaced by cloud services like Twilio and Amazon Lex.
The real question going forward is what jobs will still exist in 15 or 20 years for the next generation of workers?
If you’re in the 35-45 year old demographic, in the tech industry, there is a frequent conversation that you have with colleagues of a similar age. Is a mix between “What’s next?” and “How are you dealing with things?. At that age, you typically have a few common characteristics:
- A reasonably well paying, mid-level to senior role, often times moving up a management chain.
- An established place of residency, which may or may not be located close to the headquarters of your company.
- In many cases, a spouse and a family, with children that are beginning to have close ties to their school or clubs/groups associated with their activities.
If this was your situation and you lived in the Silicon Valley over the last 5-7 years, there were lots of stories about rising housing prices, and the shift in focus from the hardware companies in the South Bay towards the “social media and apps companies in San Francisco” that targets worker in their 20s who would be OK in the crowded apartments around the city. Many of the people in those situations looked to move to places like Portland or the Seattle suburbs to stay in the tech industry, but have a more family-friendly lifestyle.
But if you live outside Silicon Valley (or Seattle, Portland and maybe Boston….although Boston is less IT tech-centric these days), it’s a different story. For the last 10-15 years, the technology which allowed remote work has improved considerably – home broadband, WiFi, mobile devices, video conferencing, chat tools like Slack, social media, etc. – and many companies were able to grow their workforce and add expertise without requiring them to be in expensive headquarters buildings.
But it’s beginning to look like the trend around remote workers might be changing. Last year, there was a meme on Twitter about the requirement that workers “must relocate to San Francisco“. Unless you were moving from Seattle, NYC or Boston, you could expect at least a 3x cost-of-living increase with that move…so hopefully your options were going to be worth something BIG!
This year, we might be seeing the beginning of a more serious trend for the 35-45 crowd – IBM’s new policy which requires many workers to come into a HQ or Regional office, or their roles will no longer continue to exist. While IBM is pitching as a move to improve productivity, teamwork, and morale, it’s not hard to alternatively look at this as a move (in theory) to reduce the number of older employees without having to create an HR policy about “getting rid of older, more costly workers”. Many of those remote workers will not want to disrupt their established lives and hence may not be willing to pack up and move.
Where this may start to become interesting is if many of the other “incumbent” vendors, especially those that are struggling with declining margins or legacy business models, decide to follow IBM’s lead. Many of those companies have either been acquired by Private Equity firms (e.g. Dell/EMC) or have those PE firms influencing strategies at the board level (here, here, here). In those scenarios, cost-cutting tends to have as high a priority as innovation, sometimes more.
IBM is only one data point, but it’s a policy that will be watched by other management teams and should be watched by the 35-45 crowd too. The days of the remote office worker might be significantly changing…if you plan to stay with one of the incumbent vendors.
I get asked all the time about the differences and benefits of the various container technologies, especially on the orchestration side – often called the “container platform”. At some point, it’s better to write down the answer than to keep repeating it, so here goes.
Even though most people only think about containers in the context of Docker (circa 2014), containers and container platforms have been around for quite a while. But lets start with the modern-day platforms that most people know – Heroku, Google AppEngine, Cloud Foundry, OpenShift, AWS Beanstalk, dotCloud.
The Early PaaS Days
Around 2009-2011, most of these platforms came into the market with the goal of making is easy for application developers to “just push code into production”. By most definitions, these platforms were known as Platform-as-a-Service (PaaS). To some extent, these platforms did a great job of making things simple for developers, by hiding all the complexity of IaaS and associated services (LBaaS, DBaaS, Authentication, etc). And under the covers, most of these platforms used some variant of Linux containers to isolate and run the developer’s applications. These were the original homegrown/DIY container schedulers/orchestrators.
But these early platforms also had some limitations:
- Some of them only ran in a specific cloud environment
- Some of them only supported 1 or 2 languages
- Some of them were open-source, while others were various levels of proprietary software.
- Most of the open-source platforms didn’t have large community support.
So in the 2011-2013 timeframe, there were plenty of articles written about the death of PaaS and how it hadn’t gained the massive adoption.
The Open Standards Emerge
And around 2014, two very important things happened. dotCloud went out of business, but it spun out the docker project, simplifying how containers could be used by developers to package their applications. Usage of docker for container packaging took off, and the market now had a pseudo-standard for packaging applications. In addition, Google decided to open source the Kubernetes project and the Mesos project spun out of work at Twitter and UC Berkeley. This provided the market with choices about open source container schedulers/orchestrators that had web-scale DNA built-in. These three activities helped developers know where to focus their energies in building scalable container platforms.
Open Container Platforms Everywhere
As a move into 2016-2017, the market has evolved quite a bit since the early 2009-2011 platforms. Almost every major platform (or public cloud) is now supporting either Kubernetes or Mesos as the scheduler/orchestrator, and the docker project as the format for container image and runtime. The Open Container Initiative (OCI) is further standardizing the container image and runtime formats into standards that are less aligned with a single company. Kubernetes has gained almost 4x the developer support of alternative container schedulers/orchestrators, mostly because of the openness of the community and because it supports many application types with the default scheduler.
The future moving forward will continue to accelerate at a rapid pace. With all of the community focus on open frameworks and standards, businesses can feel much more confident in the platform decisions they are making for the next 5-10 years.
For the last 7-8 years, we’ve seen a large increase in the number of infrastructure software companies that were funded by Venture Capital companies. Companies such as Cloud.com (CloudStack), Piston Cloud (OpenStack), Active State (Cloud Foundry), Docker (containers), Mesosphere (Mesos) and many others were funded to contribute to large open source projects, as well as commercialize that software.
These open source projects have become the new “standards”. The projects and the code have replaced the function of previous standards-bodies like IEEE or IETF in defining how infrastructure would work. Working code has become the new “standard”.
While some of these infrastructure companies have been acquired by traditional infrastructure companies (such as HPE, Cisco, EMC, and Oracle), almost none of those transactions would be considered “big exits” in terms of VC returns.
The VC community is at a cross-roads with open source. As we recently discussed with Scott Raney from RedPoint Ventures, open source is an opportunity and a challenge for VCs. It can enable their portfolio companies to invest less in software and more in engineering. But it can be difficult to monetize their projects unless their gain a significant community following and user-base. This challenge has also been addressed several times on the a16z podcast with former Nicira CEO, and current Andreessen Horowitz VC Martin Casado (here, here). The challenge for VCs is that many developers only want to work with open source software.
This brings up an interesting question…
If the returns for VC aren’t going to be at the levels their funds require, except in very special cases, would it be unexpected to see VCs begin to stop investing in this space?
So how would the markets react if that VC funding were to slow down or stop entirely? There’s a few interesting areas to consider:
- For the last decade, open source communities have been rapidly innovating new areas – cloud, containers, big data, databases, IoT, software-defined infrastructure, etc. If the funding isn’t there for driving innovation or on-going engineering, then where will the new innovation come from? Is it possible that we’d see a resurgence of proprietary software offerings?
- Many public cloud companies are leveraging open source infrastructure to deliver their cloud services, both behind the scenes and as on-demand services. Will they be able to continue to add new functionality as the same rate that they have over the past few years?
This will be an interesting trend to track in 2017 and 2018. With more businesses looking to leverage multiple cloud (or hybrid cloud) services, and build consistent operational models across clouds, it could be challenging without those broad open source standards.