Today, an application without data analytics is like a car without a steering wheel. It will go, but there’s no controlling its direction.
The cloud and big data are infiltrating company practices, and in some cases, they’re doing it together. The intersection of the two brings many challenges and opportunities for companies that choose to process and store big data in the cloud. Once companies decide to host big data in the cloud, they’ll want to assess their circumstances and decide what works best for them.
Nowadays, there are many emerging and evolving Web- and cloud-based technologies for controlling and using data inside applications, and they’re available as full-featured cloud suites, the popular Hadoop programming framework and other business intelligence tools that can be embedded into applications.
Recently, the SearchCloudApps team wrote a series of articles that looks at different platforms and tools developers can use to harness big data in the cloud. Here’s the lowdown on the advice, tips and information this series offers, both in our special report and supporting advice articles on SearchCloudApps.
First off, SCA News Writer Joel Shore shares advice on how software pros can use big data as a service (BDaaS) in the lead story of our special report, Pick the right tools for cloud and big data. BDaaS delivers a platform and suite of tools that can speed up builds of analytics applications. The article describes how BDaaS expands the capabilities of cloud-based analytics.
Is BDaaS the right data analytics development platform for your organization? CIMI Corp. CEO and cloud consultant Tom Nolle answers that question in the series’ second story, How to choose the best cloud big data platform. He describes and opines on BDaaS (big data as a service) and do-it-yourself options and covers the role databases play. He notes that “cloud planners need to decide on a database model, select between cloud database services and cloud database platforms, and review the features of each platform against their own special needs.”
Nolle questions a key big data tool assumption, that Hadoop fits all situations, in his tip. Cloud and big data don’t necessarily mean Hadoop. To Hadoop or not to Hadoop, he advises, depends on such variables as whether data access is centralized, the level of data distribution performance needs, database practices and more.
Joel Shore digs deeper into the pros and cons of BDaaS in his new report, Beware of the BDaaS double boomerang. What does that mean? Essentially, as data volume grows, there can be conflicts between IT, which manages data analytics processes, and marketing and other departments who demand more and more access and capabilities. “We often see the marketing and sales departments getting tired of waiting for projects, so they jump over IT and do it themselves,” Enterprise Strategy Group analyst Nike Rouda told Shore. Sounds like trouble in Data City, and that makes interesting reading.
SearchCloudApps coverage of the big data, BDaaS and cloud analytics scenes is ongoing, and our readers help us tailor our reports to your needs. Are you evaluating solutions for embedding data analytics into applications and can’t find the advice you need in these articles? Tell us about your projects, and our editors and resident experts will looks for answers to your problems.
Jan Stafford, Executive Editor
In a 2014 survey, readers of SearchCloudApplications and its sister sites identified application and data integration as the biggest challenges in software-as-a-service adoption. Industry researchers agree. Gartner analyst Eric Knipp calls app integration the No. 1 barrier to cloud deployment success, and 451 Research ranks it as the second-most-acute cloud pain point, beaten only by security.
Integration pain isn’t a cloud adoption deal breaker for most companies, though. Otherwise, the cloud applications market wouldn’t be so huge—$30 billion in cloud application subscription reviews in 2014—or growing so quickly, to the tune of $67 billion in projected subscriptions in 2018, according to IT researcher Apps Run the World.
Competitive pressure spurs that growth, and integration issues have to be handled. A starting point could be answering the nine questions on cloud application integration in this guide’s first article, by Valerie Silverthorne. Each query comes with an expert how-to or why-to discussion.
An overall cloud app integration strategy should include a unifying plan for adoption projects, writes consultant Tom Nolle in his tip on integrating applications across cloud boundaries. Without a step-by-step integration approach, each cloud app adoption project has to start from scratch. Nolle advises organizations to codify best practices for each application integration component or layer.
Making applications play nicely together is only the means to cloud application integration’s goal, enabling workflow. “The purpose of cloud application integration is to connect workflows through components,” writes Nolle in this guide’s third article. Beware of tool vendors that want to tie integration projects to their cloud services and not the project’s targeted workflow goals, he advised.
Planning provides the best relief from chronic integration pain. If your organization is feeling an ache we haven’t addressed in these pages, our experts can provide suggestions for remedies.
Find out more about application integration and SaaS adoption in our e-handbook.
Jan Stafford, Executive Editor
2015 is—wait for it—the year of platform as a service! Of course, pundits have already given PaaS the keys. But 2015 is when they work. This is largely because Amazon released AWS Lambda PaaS in January, cementing the notion that PaaS will drive growth for infrastructure-as-a-service vendors. Also this year, more new PaaS offerings will be tailored for the Internet of Things, mobile devices and other hot technology areas.
Essentially, a PaaS system provides an integrated cloud hardware and software platform and the tools used for software development and delivery. PaaS breaks the ice for those new to IaaS. Most software architects and developers enter projects via PaaS or a PaaS-like environment and aren’t as familiar to the infrastructure, said Kris Bliesner, CEO of consultancy 2ndWatch, an AWS partner. Overall, Amazon’s potential customers need both infrastructure and a platform, he said.
Building IoT applications has been difficult in the past because few development tools could facilitate machine-to-machine communication. In 2013, this type of open source tool emerged and got the ball rolling. New cloud platform offerings embrace IoT, and there’s a laundry list of features that these PaaS IoT options must have to be viable.
IoT presents a strong use case for investing in PaaS, but it’s only one in a crowd. Businesses can use PaaS for many types of application and cloud integration projects, and future use cases abound. We kick off this handbook by exploring PaaS uses, current and future.
So here’s your passkey to the Year of PaaS.
“Once burnt, twice shy” is a cliché that rings true when software development teams fail to build what their target user really wants. “Too many developers and companies know that pain of a user dropping off and never returning because the value was not there,” said Damion Heredia, IBM vice-president for Cloud Platform Services Product Management. Unfortunately, eliciting users’ true requirements often works against delivering applications as quickly as users demand. Speeding up the requirements phase of development with automated “personalization” tools is one approach to making both practices compatible.
Heredia talks about how developers can use the IBM Watson Personality Insights service in this interview. Personality Insights is available on Bluemix, IBM’s Cloud Foundry-based open cloud development platform. The IBM Personality Insights (PI) cloud service uses linguistic analytics to determine potential software users’ needs, usage patterns and other information.
What are some ways that developers try to speed through the user requirements phase of content, service and software creation?
Heredia: The kitchen sink approach comes to mind; just throwing in lots of features to give the impression of value. On the user end, there’s a frustration point at which they’re getting a lot of content served up to them that’s not relevant, that turns them off very quickly to the product.
How would software developers use Watson Personality Insights?
Heredia: We’re making available for developers an API that can help them access personality data about their users. There’s a lot of documentation on the API and also a sandbox environment that is completely free to use.
So, when building recommendation engines around a particular set of data, a developer can embed our service into their application via the API. They’ll take that data, and they’ll actually map it to the behavioral data that they are collecting through their app. For example, they could see a certain demographic in a geographical location and tie that data to personality data.
Could you give an example of how that user data is gathered?
Heredia: The information is gathered through the Watson visual personality assessment tool. For example, say, we show a user an image with a caption associated with it, and the user has the choice between saying ‘me’ or ‘not me’ to that image, based upon their reaction to it. Behind the scenes, Watson is scoring that user’s responses as related to personality traits. We measure on over 100 personality traits, and then return to the developers the score for that user on each of those 100 plus traits. They can use that for recommendations, personalization, targeted marketing. There are other requirements aspects to explore, of course, but PI gives a view of the user that previously was unavailable so quickly.
How much impact has the rising number of mobile applications user had on the demand for tailored features?
Heredia: It’s been a huge factor, because mobile devices have become the predominant way that people access the Internet, content and applications. Mobile has disrupted the traditional text-based delivery system because mobile is a predominately visual medium. Even this (IBM PI) service is an example, because we had to design it to be used on mobile devices. Try to do a traditional personality assessment, like the Meyers-Briggs survey, on your smart phone. Sure, you could, but you would be pulling your hair out, it would be so frustrating. So, we had to find a way for the developer to collect data in a way that reduces that friction, and makes it fun and enjoyable for the user so everyone wins in the end.
In 2014, news about developing cloud applications using IBM Watson’s cognitive capabilities and mobile-first strategies were top traffic-getters on SearchCloudApplications. An article about developing Internet of Things (IoT) apps and a piece about enterprise architects’ cloud projects were runners up.
Looking at why these four stories attracted over 12,000 readers, I talked with our expert contributor, George Lawton, who wrote all of them. Together, we offered some footnotes to each story.
The year’s top story was “IBM’s Watson supercomputer gives developers access to cognitive cloud.” In it, Lawton describes how software developers are using IBM Watson to build customer-focused cloud-based applications. Essentially, Watson provides the ability to analyze data about customer behavior and suggest — or, even, predict — future behaviors. For example, Watson technology has been used by Fluid Inc.’s retail business customers to increase customer engagement and sales. Also, Welltok Inc. is building a Watson app that provides regular wellness activity guides and rewards for users’ healthy behavior.
IBM’s Watson Group is putting $100 million into fostering work by third-party software developers, one of those being Welltok, according an article by reporter Stephanie Neil about cognitive computing. Talking about that investment, Lawton told me that setting up a new Watson Group headquarters spurred “the single largest movement of IBM research personnel in its history,” he said. He predicts spurt of Watson-based product releases in 2015, particularly in vertical markets such as financial services, retail, healthcare and travel.
Mobile-first report takes second
High interest in SearchCloudApplications’ feature on mobile-first development of enterprise apps shows that “mobile is now a premiere consideration in the rollout of new services,” Lawton told me. Mobile-first design of applications is now a must, because people don’t have to use a computer to access information and make transactions.
The mobile-first trend has far-reaching impact. “It’s helping to redefine the traditional notion of service-oriented architectures to embrace a more flexible infrastructure based on much simpler APIs,” Lawton said. At the same time, the gap between native and hybrid development environments is starting to shrink, which promises to lower the bar in keeping up with the proliferation of iPhone, Android and now Windows devices.
Developers, take note of IoT
The intersection of cloud computing and the Internet of Things (IoT) was a buzzy topic at the 2014 Silicon Valley EclipseCon conference. Developers there posited that the cloud could enable quick tests on new IoT projects, according to SearchCloudApplications’ third most popular news story, which covered the IoT cloud discussion at EclipseCon. Cloud testing will speed development and innovation in IoT projects by reducing the number of failed projects traditionally associated with early-adopter rollouts. “Leveraging the cloud to quickly test out new ideas will allow businesses to weed out bad ideas with less risk,” Lawton said.
Watch SearchCloudApps in 2015 for regular, in-depth coverage of cloud-based IoT development. Meanwhile, catch up on IoT with articles on design issues in mobile and IoT development and how IoT technologies are impacting product development.
An enterprise architect’s cloud apps success
Everyone likes to learn by example, so it’s no surprise that a first-person account of cloud development projects was fourth on our news traffic tally. Exploring enterprise architects’ first cloud projects, George Lawton tapped the experiences of veteran software pros. Check it out to learn about how cloud helped in landing NASA’s Curiosity Rover and onboarding two companies’ sales teams to mobile and cloud apps.
If first-person articles appeal to you, you’ll find a bunch of them in our Change Agents profiles of software developers and architects.
That’s the news round-up for 2014. I can’t wait to see what tickles readers’ fancy in 2015.
Happy New Year!
Based on a list of the most popular tweets by SearchCloudApplications.com this year, twitter users were interested in a variety of cloud application topics, from integration to mobile backend as a service (MBaaS) to automation. Here are five of the most clicked on tweets and the articles they referenced.
The above tweet linked to a feature on how mobile developers view building and buying backends. The tweet was the most popular of 2014, and the link grossed 179 clicks. The quote in this tweet reflects the opinion that investing in a backend service is more cost-effective than building an application’s back end. Companies in certain industries, however, may want to avoid MBaaS for compliance reasons. Check out the story for more specifics from developers who have carefully considered if MBaaS is right for their companies.
When integration platform as a service (iPaaS) began to pop up in the cloud app world, there was some confusion over what the service provided customers. Some of that confusion remains because iPaaS is still nascent. When SearchCloudApplications.com shared our definition of the term, different tweets with a link to the article where clicked on several times and retweeted a handful of times. The definition is a good starting point for those who are unfamiliar with iPaaS, as well as developers in need of a refresher.
Automation is a tricky but appealing part of IT. For companies that are able to automate some of their processes, automation can save time and money. SearchCloudApplications.com’s third most clicked on tweet of 2014 was about how cloud computing company Strategic SaaS adopted automated migration software to upgrade the company’s operation, a good case study for other companies considering automation.
Integration appears for the second time on this list with the above tweet, which was the fourth most popular tweet by SearchCloudApplications.com this year. The tip linked to in this tweet provided readers with information on some common and useful tools for cloud app integration. Covering iPaaS and software as a service vendors, this tip reviews some new products that readers might find useful to their companies.
MBaaS joins integration as the reoccurring topics in the top five tweets of the year. This feature highlighted in the tweet above examined an MBaaS success story. It’s another good read for mobile developers, particularly those who are struggling with handling the back end of their applications.
Follow us on Twitter @SearchCloudApps if you’re interested in receiving tweets like those above on a regular basis.
Integration is one of the key issues that arises for companies considering or using the cloud. Cloud app integration is central to cloud computing because it enables different programs to share data, but it is difficult to do well.
Executive site editor Jan Stafford recently wrote about the mistakes made and strategy needed for dealing with cloud integration. Stafford’s blog post focuses on the highlights from interviews with two Gartner analysts, emphasizing the need for enterprise architects to be well-versed on integration.
If that’s you or you’re looking for more information on application integration and cloud integration, SearchCloudApplications.com has published several articles about integration in the last couple of months. Recent tips and features provide information on integration basics, tools and more.
Part of the difficulty of integration is that the need for it is so pervasive in the cloud world. Integration takes place in the following use cases: integration platform as a service (iPaaS) providers, cloud to cloud, cloud to integrator to cloud, or within a hybrid cloud.
IPaaS, one of the solutions to integration issues, also makes things more complicated. New iPaaS and other integration providers emerge every quarter. With so many integration tools to choose from, IT decision-makers need to invest time to assess the market. Recent tool reviews on SearchCloudApplications.com might be a good starting point for your research.
Integration is hard, but it isn’t impossible. If you’ve been running into issue after issue, don’t despair. Remember that other companies have had success with integration. Sometimes the best advice isn’t from an expert pontificating on a subject but from IT professionals who have toiled with the technology that makes integration possible.
For those who are doing a deep dive on the subject, check out this essential guide on cloud app integration with even more information on basics, tools, strategy and iPaaS. Once you’re done, check out our quiz on cloud integration solutions, and test your knowledge on the subject.
Today’s hybrid clouds don’t live up to the name, in Gartner analyst Eric Knipp’s opinion. Without application portability, hybrid clouds are just a collection of remote and in-house applications that have been tailored for each environment and loosely integrated.
“When I think of hybrid cloud, I’m thinking of a place where I can write an application and then deploy it to any hosting location without changing the application at all,” said Knipp, Gartner managing VP. In a true hybrid cloud, workloads can be easily moved back and forth between private and public clouds to facilitate deployment and efficient scaling. Using either policy or some flick of a switch, a developer could make that same application scale to many other places, which may span private, public or even multiple public providers.
Deploying a single application across many environments with no real limitations is a capability that doesn’t exist today, said Knipp, who is co-author of Gartner’s 2015 Planning Guide for Cloud Computing. “To get there, we have to be architecting portable cloud applications in a way that dimensionally can live up to this hybrid cloud fantasy that we have today,” he explained. “I think it’s going to exist, because I see some building block technologies that are maturing.”
Using a platform as a service (PaaS) framework, like Red Hat OpenShift or Cloud Foundry, is a first step toward such “real” hybrid clouds. PaaS greatly increases application developers’ productivity by outsourcing the “plumbing” required for software development. PaaS provides infrastructure provisioning, development tools and cloud testing so that developers can focus on building the business logic and other important features.
Some say that Docker is the best basic building block for the hybrid cloud, but Knipp doesn’t. Docker is an open source container program in which an application and its dependencies can be bundled. A container is where a program building block, or component, can be run. “Docker is a useful enabling technology, but doesn’t solve most portability challenges,” Knipp said. All Docker enables is embedding software into the Docker image that can be shared. “It doesn’t help you with the external dependencies, and that’s where the developer’s problem really lies.” Even before Docker, Knipp said, it wasn’t that hard to build an application, even as a .jar, that could be deployed across many different instances of Tomcat.
“Backing services is what was hard to solve before Docker and still remains hard to solve after Docker,” Knipp explained. All the components involved — the database, the message queues, the in-memory data grids, the analytics service, and so on – add tremendous complexity. It’s not just what’s in that application that’s difficult to handle, it’s all the backend services, Knipp said. PaaS helps by providing and automating many of those services.
Portability has to be the key non-functional requirement consideration for any new application that’s being built in the cloud. “Just because I can take my application and drop it someplace else doesn’t actually make it portable,” Knipp concluded. “For me, hybrid cloud will only be achieved when we have a much higher degree of application portability.”
Twenty-one percent of respondents to a survey by Gigaom Research said they use the cloud to develop enterprise applications. The technology research company recently published the results of three different surveys in a report by analyst George Crump called, “Shadow IT: data protection and cloud security.”
The low percentage of developers creating applications in the cloud may be due to some of the inherent issues of using cloud. The surveys polled software-application developers and other IT workers and decision makers on a variety of cloud practices and issues, including what their top cloud concerns are. Security, performance, learning cloud-related skills, cost and support were the highest ranked, with security well in the lead.
Incidentally, employees may pose the greatest security risk to cloud adopters. Employees commit more than 70% of unauthorized data access, whether accidentally or intentionally. Employees also knowingly use unauthorized cloud applications. Eighty-one percent said they use SaaS applications without IT approval.
Another key finding of the report was that companies use many cloud instances. Of the companies surveyed, 32% said they have more than 50 cloud machine instances in production. An additional 30% use one to ten clouds, and 23% use 11 to 50.
A recent report by Forrester Consulting confirmed this multi-cloud reality. About 83% of respondents struggle with compiling their disparate cloud services, according to the report, which was commissioned by Quincy, Mass.-based software company Infosys.
The report, “Simplify and innovate the way you consume cloud,” also highlighted user experience as a hurdle to working in the cloud. Between 60% and 70% of survey respondents said they are concerned/very concern about the following: the complexity of managing and governing hybrid cloud, that application performance metrics such as availability and speed will be negatively affected in the cloud, and that integration with other applications will be difficult.
When enterprises adopt cloud computing, many of their legacy methods of software integration are instantly obsolete. Hanging on to old integration methods is like trying to fit square pegs into round holes, according to Eric Knipp, Gartner Inc. managing VP. Integration is the biggest barrier to cloud deployment success and the primary driver of change and new opportunities in developers and enterprise architects’ jobs, he said.
Application integration has been a pain point for organizations a long time, Knipp said. Initial efforts focused on point-to-point integration, which usually resulted in fragile integration points that, when multiplied, were not robust. The integrations would break during software upgrades, causing long release cycles. Over time, some, but not all, enterprises remediated integrations using integration middleware, including enterprise services buses (ESBs) and APIs.
“Yet, when organizations adopt SaaS, I see the same point-to-point integrations happening,” Knipp said. “Haven’t we been here before? Don’t we know how this movie ends? It’s not good.”
Today, cloud integration must be a core competency in the utility belt of the modern application developer in an enterprise. “When you’re talking about the mainstream enterprise that is not in the business of writing software unless they have no choice, integration has to absolutely be the first thing that you focus on,” Knipp said.
Gartner Research Director Kyle Hilgendorf agreed. “I like to refer to integration as kind of a four-headed monster,” he said. “You’ve got network integration, data integration, identity integration and then application or services integration.” Knipp and Hilgendorf are co-authors, along with four other analysts, of Gartner’s recent 2015 Planning Guide for Cloud Computing.
To handle all four types of integration well, enterprise architects must ensure that those integration architectures are “pre-plumbed or pre-connected” prior to adoption of public cloud, Hilgendorf said.
In advance of adopting cloud, build a cloud-friendly enterprise architecture and integration strategy, Hilgendorf advised. “Think about how to bridge your networks between your data center and one or more cloud providers,” he said. “Think about how you will do single sign-on and identity and access management federation or synchronization.” Consider how to synchronize the business’ identities for authentication and authorization requests. “All that work and much more has to be done up front by the enterprise architect and architecture team,” he said.
The integration challenges of cloud adoption alone give architects and developers a once in a lifetime opportunity to retool their skillsets for a long-term, successful career, according to both analysts. With the right skills, they’ll be valued leaders as businesses transition from traditional application architectures, deployment methodologies and sourcing arrangements.
“It is more critical than ever for integration to be a core competency in the utility belt of the modern application developer in an enterprise,” Knipp said.