A few weeks ago we had the privilege of talking with Gene Kim on The Cloudcast (.net) about his new DevOps book, The Phoenix Project. During the show I had a bit of a “lightbulb moment” and immediately went out and downloaded the book. I’m almost finished (page 206 as of today) but I wanted to share a few points from a Cloud Operations stand point based on my past experience, the podcast, the content of the book so far, as well as a recent post by Bart Copeland over at ActiveState.
First, a little bit of background. My initial job out of college was outsourced IT Operations for IBM Global Services supporting one account, Kodak. This was the late 90’s and for those of you that remember, cameras were not always digital. In the pre-digital days cameras used 35mm film (look it up kids) and each roll held 24 or 36 exposures. When you wanted to see your pictures, you dropped the roll of film off at your local drugstore or grocery store. Everyday around 4:00 the film for the day was collected by trucks and sent to about 70 large processing factories around the United States, Mexico, Puerto Rico and Canada to be developed. All of the film would be transported, unpacked, developed, billed, packaged, and sent back out to the store by 8:00 the next morning. At the height of 35mm photo finishing, Kodak held about 85%-90% of the market and overnight photofinishing processed somewhere in the neighborhood of 3 MILLION photos per night, all in a 14 hour window every night. My organization supported the IT (applications, servers, network, everything) for the factories.
While reading both Gene’s book and Brent’s blog I had a sense of deja vu because while they compared IT Operations to factory work, I actually lived a combination of both: supporting IT Operations IN a factory. I didn’t realize it until recently but this unique perspective has really shaped my approach to IT Operations over the years and many DevOps concepts just “made sense” to me. A few examples from IT Operations in a factory:
- The process is just as important as the work – While there are many courses today (and certifications to go along with them) for technical products, there usually isn’t a large focus in Operations on the HOW and WHY to get work completed. You can take a Microsoft, VMware, etc. class that will teach you to perform administration, but what about the questions you should be asking before you start the work? In any factory environment the concepts of flow and efficiency are paramount and this naturally bled over into our technical operations.
- Unplanned Work and WIP (Work in Process) can be the death of Operations – Gene discussed the concept of Unplanned Work (work that isn’t scheduled that ends up taking priority) and Work In Process (work that has been started but isn’t completed) as a killer to an organization. We supported a system spread across 70 locations that is processed over 100,000 orders per night. In our environment, things needed to run smoothly at all times because our time window was so short. Because of this, we mastered the concepts of flow very early on to make sure there were minimal constraints in our systems.
- Everything was Cookie Cutter – It took a massive team to support an infrastructure this large spread across 4 countries and 70 locations. Remember, this is pre-virtualization and pre-remote control/screen sharing days, all support was over the phone with a remote operator (often unskilled technically) typing in commands and reading back the screen output. When something broke, it was PAINFUL. Because of this, everything was tested and documented before it went out the door. We did this to remove the support system as a constraint to the Operations. Anybody in the factory could pick up the process and anybody on our end could remotely support the systems. We attempted to remove unplanned work (something is broken) as a constraint as much as possible.
As I read both Bart’s post and Gene’s book it dawned on me that most of the Operations world doesn’t see things this way. The concepts of flow, work in process, constraints, resource allocation and performance metrics were just how I was “raised” in Operations. I was taught the process is just as important as the support methods.
With Factories out of the way, let’s move on to Marathons. Both Gene Kim and Nick Weaver expressed on the podcast that DevOps as a model is often more of a journey than a destination for most organizations. The only way to get better is to practice over and over and to strive for constant improvement. This is very similar to training for a marathon. You don’t go from sitting on the couch to running a marathon over night. It takes breaking the activity down into smaller steps and mastering smaller goals over time to accomplish your larger goal.
Lastly we have Snowmen. How do you build a snowman? You start with a small snowball and roll it in the snow until it gets bigger and you have the size you need. Building a snowman is all about momentum and building on past progress. Building a DevOps practice is also all about momentum. Success breeds success. As I trained for a half marathon (no full marathon for me yet) I found that the more I ran, the more I wanted to run. It is painful at first but once you get past the initial resistance the snowball starts to build.
In summary, read Gene’s book, learn from factory operations, put a plan in place that starts small and builds over time. Change (especially in most IT Operations) doesn’t happen over night but it can happen. I was fortunate enough to work in a highly efficient Operations environment over fifteen years ago and it has stuck with me ever since.