There is something new afoot, he advised.  More applications have larger analytic components.

“For decades, there has been a split between transactional processing and analytical processing, and the split is widening in some ways now because of the volume of the data,” said Monash.

“For a number of reasons we are looking at a new generation of DBMSs,” said Monash. “Public and private cloud are just two of the reasons.”

There are new opportunities for new DB tools because there are DB tools now that apply more effectively to new use cases, Monash indicated.

That simplified things tremendously and has now been embedded in Carey’s personal philosophy – buy as much pre-made as possible and then perform tailoring where needed.  Looking back on his implementation experience Carey says that he was able to avoid complexity and many of the challenges others face by avoiding excessive customization. “We just used a little cement between the pieces.”

Likening the process to buying a suit, he says most people can’t afford and don’t need custom-made formal attire but instead can simply have a mass produced product tailored to fit. “In other words you need to figure out where on the cost-curve you want to be,” he says.

Joe Egan, a senior software developer most recently in a consultant role at BikeBandit.com, has faced similar integration challenges but didn’t find an easy pre-made solution. In several instances he has needed to integrate third party data services, for example a continually updated review and question and answer service that needed to sync with a local database of product offerings available to consumers. “In that space, I found [the challenge was] we were moving from less frequent, even weekly updates, to continuous updates based on web services,” he says.

Finally, notes Carey, “I believe that for any sizable project a company needs to have an architect/advocate on board who is knowledgeable about the issues and is looking out for the company’s best interest.  A third-party vendor may know the technology better, but their motivation is to sell and up-sell.  So the third-party’s guru is not really an advocate for the buying company.”

For more, see our featured story on application integration.

A long time ago, integration of various applications, application elements and databases became the main work of application development. Web services and SOA were first discussed in this context: Application architects and developers did not have to write every element from scratch.

Instead they would call in services, in some cases, as needed. These services were usually wrapped in XML and delivered over HTTP. They were called Web services, and were combined sometimes using techniques of service-oriented architecture. A classic example of an early Web service was a stock ticker feed.

With the stock feed, there was no need for you to write it yourself – you called it over HTTP. You could program against such a data service. For example, you might have five competitors’ stock prices feeding on to an executive’s PC screen. Maybe you would correlate this with another fee, say, money exchange rates. The Web services stock feed demo became so common that it became kind of boring.

Fast forward to today. Where B2B is in play, we now share services across corporate boundaries. Other than that, much of the services action still remains within the confines of the operation where components are shared across departments. These shared services bear some conceptual similarity with software object architecture of the ‘90s, although the implementation details are very different. Most discussion of SOA as it has evolved has been about in-house services.

Where are Web stock tickers now? They have become ubiquitous and something of an afterthought. But their siblings continue to grow. What is happening is that a genuine industry is growing up around this Web data services. The providers often have familiar names: Thomson Reuters and D&B, for example. But new entities have emerged such as StrikeIron, which offers a catalog of data services. Let us know what data services have worked for you. What has not worked? What questions do you have on data services?

This month, Kapow released Web Data Server (WDS) 7.2, an update to its flagship application used to build, test and deploy data feeds and services that integrate all kinds of Web data into the enterprise infrastructure. In this update, WDS includes a full design studio IDE with a data viewer, native XML support and FTP/file system interaction.

Andreasen said this release represents a three-year effort to merge the company’s RoboMaker and ModelMaker products into one offering. He said the focus was on usability and enterprise compliance.

Software like WDS is useful for enterprises looking to integrate data from partners whose APIs present a significant technical challenge, said Andreasen. Instead of having both IT departments go back and forth trying to make sense of things, you can pull up a website in WDS Design Studio as you would a Web browser and build a process flow by visually interacting with it. You can then deploy this as a service with little to no up-front coding.

“Most customers are focusing on applications that are business critical that they want to integrate with but there is no documented API or Web service,” said Andreasen.

Of course, since services built with WDS are controlled by local users, if the website a service integrates with changes, the service will break. Andreasen said this does happen, but building a new service is still a matter of minutes or hours – still a time-saver when compared to working with another company’s IT department.

Pricing for WDS starts at around $80,000 per year.

Now we at SearchSOA.com are asking you to do the same thing, sort of. It won’t involve wearing a kilt or smashing a coconut over anyone’s skull. We just want you to ask some good questions.

We’ve recently revamped our site experts roster and we’re looking to put them through their paces. The way it works is you ask a question and we send the question off to an expert to get you an answer. It’s a fairly illustrious list of folks:

• SOA standards and architecture – Anne Thomas Manes, vice president and research director at Burton Group
• SOA governance and BPM – Sri Nagabhirava, founder and chief architect nLeague Services
• SOA infrastructure – Dana Gardner, principal analyst Interarbor Solutions
• RIA and enterprise mashups – Jason Bloomberg, senior analyst ZapThink
• SOA testing and QA – Rami Jaamour, product manager of SOA solutions at Parasoft
• Data services – Larry Fulton, senior analyst at Forrester Research
• SOA development – Chris Haddad, vice president and service director at Burton Group

They’re already producing some top flight insight, like data integration best practices, where grid intersects SOA and the difference between WSDL 1.1 and 2.0. Yet good answers like that depend on good questions from the user community. We sift through heaping piles of “What’s the difference between an application server and a Web server?” (a perfectly legitimate question, but we answered it back in 2003) in order to get some of the top minds in the SOA space the best questions the user base can generate.

The process for submitting a question is simple. Just go to the topic where your question fits and click on “Pose a Question.” That will take you to a question submission form. After that, it’s as simple as typing in your query. Keep us busy. We like it that way.

(The other revelation was that the B-52s have a keyboard player who looks like Jose Canseco, but I digress.)

Here’s a smattering of comments made by SOA users at the show:

John Roach, director of architecture and governance at Wal-Mart, focused on using SOA to help manage store stock levels and customer demand. “If SOA doesn’t trace back to you finding the right thing when you walk into our store at the time you need it, then it isn’t material for us,” he said.

Kumar Murugan, application development manager at pharmaceutical manufacturer and marketer Novo Nordisk, talked about centralized policy management and stressed the need to view all SOA projects as part of a continuous process improvement cycle. He also highlighted the importance of having a rigorous QA process.

“You need to do a system discovery for any new service,” he said. “You need to understand how reuse affects your existing services.”

Manny Montejano, CTO at Cars.com, called governance “the key thing we need to resolve to be successful” as his company deals with explosive growth.

“It’s important to say no sometimes,” he said. “You have to let people know that some things are going to be more trouble than they’re worth.”

Anne McDiarmid, CIO for Australian fabric and crafts retailer Spotlight, made a case against trying to solve every problem with a software purchase.

“I’ve got middleware hanging out of my middleware,” she said. “I don’t need more middleware.”

A whirlwind of corporate acquisitions in foreign countries has created an integration challenge for SEB, a Swedish banking and insurance company. Enterprise architect Anders Jader targeted data as a key element in bringing together this international banking conglomerate.

“We are now in a phase where we need to transform everything into one data model and then be able to use that data as a service,” he said.

Tony John, domain lead architect at Allstate Insurance, echoed the importance of data in all things service-oriented, stating “we need more data analysts and data architects.” He noted that the bulk of a $30 million mainframe-to-SAP project “was spent on understanding the data.”

John also made the case that technologists have to understand the business they work for, not just how their niche of IT functions.

“No matter what machine or network it goes through, it’s still a group of people doing some business activity,” he said.

Here’s a few of the high points from the show:

• Anne Thomas Manes, VP and research director at Burton Group, noted that a lot of users want to standardize on a single enterprise service bus, neglecting the reality that most every company will need to support multiple ESBs. She also suggested not thinking of the ESB as a “bus” because it implies that there’s something in the middle of your services. Instead she suggested the term enterprise service network.
• Miko Matsumura, deputy CTO at Software AG, used the image of a crack pipe to illustrate a point during his presentation, namely that bad development habits can be hard to kick.
• Daud Santosa, CTO at the National Business Center inside the U.S. Department of the Interior, made a great point about choosing foundational pieces of technology — if the technology in question requires consistent and costly upkeep, then it shouldn’t be a foundational piece of technology. “This is hard enough,” he said, pointing to the detailed reference architecture he’s trying to implement at NBC. “Look for technology that makes your life easier.”
• Dan Foody, VP of Actional products at Progress Software, made a great observation in response to a question on how can you sell your business on the merits of SOA: take a sales course. His reasoning was you need to describe what service orientation means to your business and outside IT fiefdoms and that will require real professional sales skills.
• Many attendees bemoaned the communications difficulties that plague IT projects, but Matsumura offered that there is a common language everyone speaks: money. The line drew a hearty laugh from the Reston attendees, but later one person from the audience mentioned to me that the “money” line helped crystallize what he needs to do to get executive buy-in.
• John Woolbright, CTO at Synovus Financial Corp., noted that many real-time systems are undone due to a lack of data quality. He suggested defining systems of record for data. “If you want your SOA to be successful you need to know where that data is and how to access it.”
• Foody stressed creating visibility not only into the IT infrastructure, but to the business process itself. Failure to provide that visibility can lead you down the path of applications that don’t deliver as promised for the business, he noted.
• Manes continually stressed the importance of getting a handle on the producer/consumer relationship inside SOA as a key element for governance. Apparently too many users are running into problems caused by unchecked service consumption.

Most of all, a hearty thanks to our attendees. Rarely do you see audiences that are anywhere near that engaged during the presentations. It served as reminder that the practical implementation of SOA governance has become a pressing concern for app dev and IT shops.

[display_podcast]

The podcast will cover:

• The drivers for data integration
• Why data needs to be treated like a shared and reusable resource
• The importance of data quality
• Why you need more than an ETL tool to integrate your data
• The necessity for creating a standardized way of handling data