But it is not just data at rest that is in question. The need for big data in motion is growing, viewers claim. For its part, middleware stalwart Tibco sees big data, coupled with event processing and fast messaging, as a route to greater market penetration.

“We kind of own the big data problem as it relates to real-time events,” Tibco’s Vivek Ranadive told SearchSOA.com on a recent call. He maintains that even common tasks like pizza delivery – granted, for national chains – will be affected by big data. “When customers inadvertently get cold pizza, the company can pick that up,” and make things better with a free pizza, a coupon or what have you.

“When you think about big data, it is about running twenty-first-century risk. The planet needs an ‘eventing’ platform,” said Ranadive, author of “The Power of Now” (1999) and “The Two-Second Advantage” (2011).

The Tibco event architecture plays a role in a recent user story on SearchSOA.com. Our site recently profiled shipping giant OOCL’s Matt Rosen who shows how challenging markets can be, and how pivotal well-managed technology is in addressing those markets.

Shipping companies were in a tough bind when the 2008 downturn struck, and the going was not easier when recession hit big European markets. OOCL’s performance outpaced competitors, and in some significant part due to Rosen’s application development team, which better enabled efficient business processes for the global shipper.

Among a host of technologies Rosen’s OOCL crew employed was an event processing engine from Tibco Software. OOCL’s habitat – the shipping industry – is among those that advanced middleware maker Tibco is counting on to take it beyond its Wall Street techno roots. – Jack Vaughan

The time is winding down on 2012, and we were going through some reporter’s notebooks. Seems that earlier this year, when we caught up with David Besemer, Chief Technical Officer, CTO, Composite Software, he had some interesting comments on SOA’s role, now that it is a more mature practice.

“SOA got a lot of attention three or four years ago. Then it seemed to have waned a bit. But while the waning of the hype occurred, there were projects that showed people getting practical use out of services and APIs,” said Besemer. Among the practical uses he pointed to are new types of data integrations.

Besemer, whose special interest is data integration, said there is a change in focus going on; it is moving things away from a sole preoccupation with the data warehouse.  Cloud, big data, and analytical appliances, got the ball rolling, to the point where services-enabled technologies began eating at the edges of the data warehouse.

Non-technical business imperatives are driving the need for decoupled services in broader and broader swaths of computing. Business imperatives are calling for something faster than a data warehouse at times. Said Besemer: “All of the members of the enterprise architecture team are struggling to deliver on requests from the business in regard to data sets.” It is the data that the business needs to make decisions.

The name SOA may be heard less frequently these days. But the idea of abstracted, decoupled services is at the heart of the latest data integration advances. – Jack Vaughan

But there are also commercial tools available to tackle the problems. Among the tools that help development teams tackle modern integration jobs is Altova’s MissionKit. The latest version of the tool set adds interesting features that target the needs of the day.

Altova’s recently released MissionKit 2013 suite includes updates to its XML Spy tools that offer intelligence assistance for dealing with validation errors: updates to MapForce to support mapping for SQL stored procedures as well as an enhanced API for integrations into Java programs; and updates to its UModel tool that cover UML 2.4 and SysML 1.2. Importantly, UModel and other tools in the suite have improved support for XBRL and its most recent US-GAAP taxonomy, version 2012.

At release time, we talked to long-time industry analyst Peter O’Kelly about the trends driving these tools. Altova’s product line has evolved greatly since planting its original roots as an XML domain tool, said O’Kelly, who served as Altova’s product marketing manager and evangelist.

“It has expanded. It’s not just for people who work with XML,” said O’Kelly who now serves as principal analyst at O’Kelly Assoc.

The toolset tries to buffer developers from underlying complexity, he said, because teams always have to map between legacy and new technologies.  It is important to bring users a consistent framework, said O’Kelly.

With standard Java and standard Java servers, fairly strict and familiar three-tier architecture came about. The question to ask now is “Will it last forever?” Like so many things, the fundamental tiers of computing do come up for reconsideration once and a while.

These breezes have been blowing subtly since people cast about for lighter versions of Enterprise Java Beans. More recently, Node.js has arisen as a JavaScript alternative to Java on the server side. Increasingly, the client is the object of interest.

Node.js and other browser-influenced technologies seem to encourage software architects to cast skyward their monolithic three-tier components. As these flying components drift down to earth, they may not settle back up in the same alignment. The sudden near-hegemony of mobile clients is pushing things ahead quickly.  A variety of new architectures are brewing.

In some ways, there seems a growing reaction to the rule of Java and the server. That view emerges from a look at a reporters’ notebook. It’s not going away, but as described in an interview with James Strachan, now senior software consultant with JBoss: “The server side is becoming thinner and thinner.” When SearchSOA.com and TheServerSide.com spoke with Strachan earlier this year at the CamelOne event, the topic of Node.JS was on the docket, but Strachan was expansive.

He said, looking forward:

The server side might just be Amazon Simple DB or Mongo DB or something; there might not be much of a three-tier architecture anymore.

Meanwhile, with flair, he continued:

….the client side is becoming bigger and more and more complex; it’s real-time now, everyone’s doing Ajax, real-time updates, and people are doing lots of single-page applications – which is when one Web page starts up and the entire app is in there. There are lots of models, containers, relationships and persistence and “yada-yada.”

Strachan notes this is highly driven by mobile applications:

In many ways the browsers won. Almost every mobile platform has Web capabilities inside it – Android, iPhone, iOS all have Web browsers and so forth. So the Web has kind of won … most browsers use JavaScript and HTML 5. Silverlight’s dead, Flash is kind of dying … the browser is really where it’s at …  with HTML and JavaScript.

Are the new approaches overblown? Is real change far off? Do you see a shift in emphasis to the client? If so, do you think services or SOA have had a hand in breaking down the status quo? -Jack Vaughan

@searchsoa

The skills list – courtesy of the IT skills specialists at Foot Partners, LLC – includes Apache Hadoop, MapReduce, Hbase, Pig, Hive, Cassandra, MongoDB, CouchDB, XML, Membase, Java, .NET, Ruby, C++ and more.

Further, the ideal candidate needs to be familiar with sophisticated algorithms, analytics, ultra-high-speed computing and statistics – even artificial intelligence. The needs of big data, which arise in part from modern computing’s ability to produce more and more bits and bytes, mean that developers have to hone their skills significantly. Suddenly, SQL-savvy developers have to obtain NoSQL skills.

New technology like Hadoop is so raw that the developer is often forced to create his or her own software tools, which is a skill in itself.  Writes the Foote crew:

Hadoop is an extremely complex system to master and requires intensive developer skills. There is a lack of an effective ecosystem and standards around this open source offering and generally poor tools available for using Hadoop.

Foote warns that there is only more of the same to come, especially as unstructured data from sources such as sensors and social media pile up in the in-bin. Note to big data scientists of tomorrow: get ready for the deluge! – Jack Vaughan

The notion arose from our recent SearchSOA.com conversation on SimpleDB with Christopher M. Moyer, vice president of technology at Newstex LLC, who spoke with us on the topic of the Amazon cloud. The topic was no coincidence – Moyer is the author of Building Applications on the Cloud (Addison-Wesley, 2012), a book that neatly describes many  basic patterns of cloud computing based on concrete examples.

”SimpleDB is one of the hardest databases to comprehend,” Moyer said. ”Everyone is used to the idea that, if they write a record to the DB, it will be there. ‘’

But the standard approach of SimpleDB is that marvel: eventual consistency, which is great but unfamiliar to a large legion of developers.

It is a difficult topic for developers to understand and work with in their systems, he told us.

Amazon, like other cloud pioneers, has seen a way toward supporting the familiar, but this too needs special attention.

”They have worked to address [the gap] with a ‘consistent mode,’ but you should be aware that it can affect your performance and stability,” Moyer said.

”Simple is not always simple,” the SearchSOA.com editor said.

NoSQL may have a lot of momentum just now, but it requires some pretty extensive programming capabilities – enough to limit the extent of its eventual use for large-scale on-line transaction processing (OLTP), in the estimation of Stonebraker, CTO, VoltDB and adjunct professor, MIT.  Meanwhile, conventional relational databases will also fail to keep up as the number of online transactions “goes through the roof,” Stonebraker told an audience this week at the Large-Scale Installed System Administration (LISA) Usenix 2011 Conference in Boston.

OLTP data handling alternatives break down into the categories of OldSQL, NoSQL and NewSQL, says Stonebraker. OldSQL, or traditional SQL databases suffer from code bloat through years of wide use and adaptation, claims Stonebraker. These general purpose RDBMs spend too much time and effort on locking, latching and buffer management, and spend a small portion of their time doing ”useful work,” he claims. They do benefit from use of high-level SQL programming support, which is an area where NoSQL databases are faulty, in Stonebraker’s opinion.  The NewSQL approach he attributes to VoltDB and others claims better support for high-level SQL programming along with dramatically increased speed of transactions via more scalable architecture and built-in high availability.

“The NoSQL guys give up SQL and ACID to get scalability and performance,” he said. But, he asserted, SQL is not part of the problem with overhead that established RDBMSs exhibit.  Keeping SQL but cutting conventional RDBMS overhead is a worthwhile goal, he maintains.

“There is no reason you shouldn’t run DB systems with a high-level language,” he said. “With NoSQL, you get to do ACID in user-level code. That is a ‘tear your hair out kind of thing,’” he said. [Stonebraker does, however, admit that NoSQL players are working to improve their SQL support.]

Stonebraker has a long history in data bases in academia, as well as in industry, where he has been something of a serial entrepreneur . He was the major force behind the Ingres relational database, and subsequently started innovative data-related concerns such as Illustra, Cohera, StreamBase Systems and Vertica. Now he is with VoltDB.

The advent of the smartphone will boost transaction rates, which have already had to accelerate drastically with the move from customer service representatives doing computer input to end users over the Web doing input.  Stonebraker actually puts forward a fourth alternative, besides traditional RDBs, upstart NoSQL DBs and NewSQL, for meeting the new OLTP mandate. That alternative is ”roll your own.” It is used by some big sites, and, in Stonebraker’s estimation, is not an appropriate strategy for typical IT shops.

In his view, a top-tier site like Amazon is mostly a traditional shop using ”old SQL” for purchases and data warehousing.  Amazon does however make use of the SimpleDB non-relational data store.

“Most of the big Web properties have [created] ‘purpose-built’ data base systems. Most of them have simply rolled their own,” he said. “Simple DB was a purpose-built application. Cassandra for Facebook and Big Table for Google, too, are purpose-built ‘roll your own’ [applications].”

“If you are running at the unbelievable volumes, the ‘build versus buy’ decision is tilted toward build, especially if you have highly competent programmers. 

Roll you own is not the best approach for most shops, however. “The big Web properties are generally rolling their own DB systems because they have the volume and the personnel. There are only about six companies that are in that state,” Stonebraker told the LISA Usenix crowd. – Jack Vaughan

Enterprises can use the pre-integrated Sonic ESB and DataXtend SI enterprise solution alone or together with the Progress Actional SOA Management platform, according to the company.

Speed in development is at issue for SOA today, said Bloor Research founder and analyst, Robin Bloor. ”For IT to be successful in supporting business, it has to evolve its approach to SOA to be more responsive, particularly in how it handles change with regard to data semantics and policy,” he said in a prepared statement. At the Progress Revolution 2011 user event last week, he expanded on his thesis that ESBs must improve.

The limits identified for expanded use of SOA and ESBs are several. They include data limitations.

“Initially, SOA didn’t address data limitations, and early SOA implementations uncovered some additional challenges, of which one of them is data – data interoperability issues, data in context, and having a data model that doesn’t need to be changed every time you make a change to an application,” said Colleen Smith. A lot of the focus in the new product combo is meant to specifically address these data limitations, she said in an e-mail message.

Vaughan discusses this issue…

Though late to the party by some estimates, Red Hat claims its JBoss Enterprise Data Grid, which grows out of the ongoing Infinispan open source software project, brings new traits to this class of software – traits aimed at today’s caching data needs. The data grid offering will complement Platform-as-a-Service (PaaS) and contemporary shared-services architectures, the company said.

Red Hat execs told us they are proposing the Infinispan work for inclusion in a Java Community Process standard. There, they will encounter IBM and Oracle, who have their own Java-enabled data caching products. No problem, Red Hat says -  they would just like to “start the process” on some new caching standards for Java.