1. Big Data is Difficult.
2. Data Huggers are the New Server Huggers - Company after company I spoke with highlighted that existing organizational structures are their #1 challenge to Big Data strategy success. Organizations love their data. Organizations don’t love sharing their data with other groups, even within the same business.
3. Forget the Economy, Big Data is the 1% Club - While Business Intelligence and Data Warehousing have been around for quite a while and are deployed at many companies, the companies that are able to leverage the newer technologies (Hadoop, NoSQL databases, R, etc.) to unlock business insight in real-time is still extremely small.
4. Big Data != Fast Data - It becoming clear that there is a big difference between Big Data and Fast Data, both in technologies and use-cases.
5. Hadoop is the Foundation, but beyond that… -  While the Hadoop market is competitive (Apache Hadoop, Cloudera, Hortonworks, IBM, MapR, Oracle, Pivotal, SAP) are all trying to sell a Hadoop-centric product, the real wars will be with the tools, frameworks and extensions that are layers on top of Hadoop.
6. “Telemetry” will make its way into your vocabulary – Whether it’s called “Internet of Everything” or “Sensor Data” or something else, you will begin to hear a massive push about how telemetry data will be attached to people and machines to drive real-time fast data and unlock new markets.
7. Connecting to the legacy is key – Many companies are focused on being able to not only integrated legacy datastores into Hadoop-based “Data Lakes” or “Data Reservoirs”, but also focusing on how to integrated existing SQL tools and skills into a Hadoop environment. The SQL aspect is attempting to overcome the shortage of Data Scientists and extend Big Data out to more generalist business users.
8. Data Scientists are in massive demand – This has been highlighted before, but it’s still a massive shortage in our industry. Not only is there demand for people to analyze the data, but also massive demand for people that can setup/run Hadoop environments and integrated legacy systems with Hadoop.
9. Huge Opportunities for Big Data On-Demand – While many Cloud Service Providers offer various types of on-demand IaaS resources or on-demand Database services, the ability to experiment on Big Data or Fast Data use-cases is massive. With setup being (still) complicated, there are huge opportunities for Cloud SPs to expand their offerings to be turn-key, as various sizes, to accelerate the time to analysis and action.
10. Bandwidth is Still a Problem – While Big Data might be a big deal, it still hasn’t overcome that pesky little physics issue – the speed of light. It will be interesting to watch how the location of data (on-premise vs. in public clouds) shapes the industry over the next 3-5 years.
11. Get familiar with Open-Source Frameworks - Whether you’re deploying with Puppet or Chef, coordinating resources with Zookeeper, or developing tools that leverage Pig or Hive, it’s time to start familiarizing yourself with open-source frameworks and community-based knowledge sharing. Big Data (or Fast Data) is attempting to solve challenges that are beyond a single organizations, so using the tools and frameworks of the community will help accelerate your chance at success.
12. Your Data is Your Next Product/Market – It was interesting to hear how many side conversations involved companies that currently possess massive amounts of industry-specific data that are now looking to unlock (and sell) this to external industries. For example, intelligent weather data could be extremely valuable to dozens of companies (finance, insurance, farming, transportation, grocery stores, airlines, etc.) that may be able to make better decisions from data that was never previously available to them.
13. Big Brother Knows About You – You’re welcome to keep fooling yourself into believing that you have a level of privacy or information security. Think again. Every device you interact with, every transaction you make and every location to visit is being tracked, correlated, analyzed and acted upon by someone.

Then reality sets in and they are faced with the challenges of Data Gravity, multiple variations of APIs, multiple variations of projects, and other nuances that highlight that we’re still quite a long way from being able to switch between any cloud computing service as simply as we plug into any Ethernet port.

So given that we’re still in the early days of this next wave, what is an appropriate level of concern to have for “lock-in” vs. focusing on bringing new value to the business via technology? And are there specific areas where lock-in might be more of a concern that other areas?

In the past, we often looked to standards bodies (eg. IEEE, IETF, W3, etc.) to define interoperability. But in today’s world, there are as many best-practices being defined via open-source projects as in any standards-body, so companies now need to decide if they value pace-of-change over standardization. IT organizations have traditionally leaned towards risk-migration as the higher priority, which is often in conflict with their business-side counterparts that want results now and aren’t thinking about the management and maintenance of legacy systems years into the future.

In some cases, we’re seeing levels of standardization between defined by loosely defined committees (eg. Open Data Center Alliance; podcast) or by groups of providers looking to establish market-baselines (eg. Cloud Foundry Core; podcast). Some companies view efforts like this as a good thing as it moves them farther along towards levels of interoperability, but there are still some in the industry who are never satisfied with the level of “openness” or often seek to find alternative motives from committees or vendors.

With bleeding edge technologies, we’re often faced with the challenge of no existing standards since the technologies are so new that a focus on standardization has not been anyone’s priority. Technologies like Software-Defined Networking (SDN) fit this category today.

Ultimately the answer to the trade-off between complete standardization and keeping up with change should be driven by business needs. Waiting for a piece of technology to become completely standardized or be “completely open” means trading off business opportunity now. Demanding the freedom of portability, whether it’s at the application or provider level, typically means giving up some of the advanced functionality that made the technology unique for the business needs at the time.

Inevitably, change creates costs. Either opportunity costs, capital costs or operational costs. Whether that change is high in the stack (applications, development frameworks, middleware) or down in the technology weeds (management software, operational process, worker retraining, etc.), there is always going to be a cost. It’s critical that both the business and technology “costs” are evaluated and considered when trying to determine whether to focus on standardization or openness. Neither come for free.

[NOTE: I'd recommending watching the two videos from Simon Wardley before continuing, as it's important to understand that what we're seeing in the evolution of computing is not unique to the computing industry. It happens for most products and markets over time.]

I believe there to be four critical elements to consider in looking at the parallels between 20^th and 21^st century factories:

1. Original Factory Designs – Flexible Containers and Assembly Lines
2. Supply Chains – Getting In and Out of the Factory
3. Evolving Factory Operations – Automation and Quality Improvement
4. Evolving People-Centric Process – Skills and Feedback Loops

Just as manufacturing companies leveraged Demming’s teachings to drive process efficiency for cost and quality of their physical products, so are the modern data center operators to reduce cost and improve response times for their services. PUE is replacing JIT as the input metric. DevOps is replacing Lean as the organizing structure. Fail-Fast is replacing Six Sigma as the quality measurement.

Factory Designs

Henry Ford revolutionized the automotive industry by introducing the assembly line and massively standardizing how cars were built. Over time, automated machinery was brought in as certain processes evolved to nothing more than repetitious human actions, further reducing costs and eliminating mistakes. Ford’s factories were designed such that multiple models could be build using common infrastructure and standardized processes.

Today, the largest cloud computing data centers emulate this same model, by running 100s of applications on consistent infrastructure (servers, virtualization, storage, network) and application platforms. Designed to be modular, these data centers are optimized for energy efficiency, as power can be 15-20% of the underlying cost to deliver the appropriate bits to the marketplace.

Supply Chains 

Factories that created durable goods not only had to be concerned with production within the factory, but also how raw materials would arrive and how the finished productions would be shipped to market. This is why many factories were built near waterways, or major railroad lines. It also explains the secondary network of suppliers that grew around the major factory towns.

Today’s bit factories are being located to best consume their two major inputs – bandwidth and power. Areas where colder weather (eg. Oregon) or low-cost power (eg. North Carolina) exist are beginning to become gathering areas for major data center construction. Other companies are choosing to co-locate near bandwidth aggregation (eg. Switch SuperNAP, Las Vegas) or bandwidth inter-connects (eg. Equinox).

Factories also evolved from end-to-end production to just-in-time, shifting the steps to bring a completed product to market. Data centers are seeing a similar evolution, as internet traffic has rapidly evolved from client-server to distributed applications, distributed data and caches, and API interactions between applications. Bringing together all these interactions for each web request is causing more and more companies to re-think where to place their data center assets to maximize efficiency and drive greater responsiveness to their users and applications.

Evolving Factory Operations 

Throughout the 20th century, factories became more and more automated. This automation was not just for production, but also inventory tracking, quality analysis, supply management and many other elements that ultimately effected the ability to bring products to market. In parallel to the automation, as lessons were learned, the operational models were also adjusted to give the business greater flexibility for variable market demands, new technologies and new partnering opportunities.

The common misconception by many Enterprises is that the Google, Facebook and NetFlix of the world only have a few applications so it’s simpler to operate vs. the 1000s of applications found in a typical Enterprise. That is not actually true, as highlighted with many examples here. While end-users may only see a single front-end to those services, they are literally made up of 10s and 100s of smaller applications behind the scenes.  They are made up of similar web-servers, databases, middleware, caching services, load-balancers and firewalls that run SAP or Oracle or Microsoft applications in an Enterprise. Beyond some of the more modern application environments for web-scale applications, the biggest difference is that Enterprise IT and Facebook don’t treat the cost of operations at similar levels of importance to their business, hence the lack of efficiency from most Enterprises.

Evolving People-Centric Process

While much of factory production has evolved to incorporate greater levels of automation or machinery, it still incorporates human intelligence to be able to find new ways for internal groups to work more closely together (eg. reducing steps to complete a task) or improve levels of quality. Movements like Lean manufacturing or Six Sigma were all about finding ways to improve output quality while simultaneously reducing the time to market or overall costs.

Within the data center, these new movements are centered around Converged Infrastucture and DevOps. As the underlying infrastructure technologies (storage, network, virtualization, server) begin to blur together, new management models are being created to create common terminology, as well as move intelligence and control further up the stack. From an applications perspective, development and operations groups are beginning to create development and deployment models that all new code to be introduced more frequently, allowing new features and bug fixes to drive a greater user-experience.

As we move further into the 21st century, IT will be under greater pressure to evolve just as other aspects of the business have. While it can be interesting to think of IT as the steward of technology for a business, its blueprint for evolution has already been defined in the 20th century. By looking at how the market has evolved to build physical products will give clear indications for how to enable the digital services of the 21st century. Companies that see the these parallels will have a much greater chance of getting successful ROI and competitive advantage.