A long standing standards body is taking a stab at measuring SOA performance. The SPEC benchmark group is interested in hearing from people on this topic. Current SPEC member companies committed to developing a new SOA application measurement standard include IBM, Oracle and VMware.
The benchmark group is interested in hearing from enterprise architects and other users of SOA techniques to ensure that the working group understands customer needs and “can develop the best possible benchmarking solutions.”
The group plans an initial benchmark designed to cover Web services running on top of application servers, Enterprise Service Buses (ESBs) technologies that connect and mediate services, and composite applications choreographed through BPEL (Business Process Execution Language) technologies.
SPEC is a non-profit organization formed in 1988 to measure engineering workstation performance. The effort to achieve objective performance standards branched out in recent years to include benchmarks that cover Java (JVM and J2EE) systems. With its SOA effort, SPEC hopes to build on that earlier J2EE benchmark work.
“We want to do the same thing with SOA infrastructure that we did with [SPECjAppServer2004],” said Andrew Spyker, an SOA runtime architect and chair of the new group. “The SOA space is being used by just about [everyone] out there, but there is no standardized benchmark.”
Of course, benchmarks are challenging. We suggested to Spyker that services, which still seem to be something of a craft – if not an art – may be particularly hard to codify. Spyker does not disagree.
“We realize it is going to be a challenge to support all of the approaches that people have to a SOA. There are multiple ways to implement a service, an ESB, or a business process choreography. It is by definition, not a specification.”
“We have to try to audit what is an acceptable implementation and make sure it is something that a typical customer would do. A well described WSDL service is a good standpoint.”
According to a published statement, the SPEC benchmark group invites comments from enterprise architects and other users of SOA techniques as they work to understand needs and “develop the best possible benchmarking solutions.”
What do you think? Can SOA be benchmarked. Use the ”Comment” icon to share your opinion. Thanks!