In this blog, listing some more techniques that enable us to create advanced Mashups using EMML.

Adding HTTP Headers to the Mashup Result

HTTP headers can be sent with a Mashup result, when the mashup is invoked. Add <variable> declarations for HTTP header using reserved names in the form httpResponse.header-name. The header-name portion can be any standard HTTP header or a custom header (that must begin with X-). For example:

Setting the Mashup Response Character Encoding

If the response is generally going to be returned in an XML format, the XML declaration <?xml version=”1.0″?> in the response also has an optional encoding property. Currently, the XML declaration encoding property directly in EMML cannot be set directly. Instead, a custom XPath function as shown below needs to be used.

Opening and Using XML Files in Mashups

Using XMLConstructorFromFile macro, we can construct variables in a mashup from XML content in files. To use XML files as source data for mashup scripts, the file must be deployed in the EMML Engine classpath.

The optional encoding attribute in the XML declaration (<?xml ?>) at the beginning of an XML file identifies the character encoding that is used to read the file. If this attribute is not specified, the XML parser may deduce the character encoding (or) read the file assuming UTF-8 as the character encoding.

Formatting Results as CSV

Instead of the typical XML format, we can format the mashup results as CSV (comma-separated values), using the FormatAsCSV macro (installed with the EMML Reference Runtime Engine).

  <macro:FormatAsCSV xmldoc=”$sampleXML” outputvariable=”$csvString”/>

For this macro to work correctly, the structure of the input document must contain at least one node that repeats. The element names of all children nodes in the first ‘record’ (first node in the repeating set) are returned as field headers in the CSV output.

Ideally, repeating nodes should contain a single level of children with simple data that is converted to the CSV value. If a child of a repeating node also has descendants, then the CSV value for that child appends the values from all its descendants to the value of that child.

Wrapping POJO Classes with Mashups

In EMML, POJO classes can be used as services by wrapping them in mashup scripts. Include the POJO classes in the EMML Reference Runtime Engine classpath.

Create a separate wrapper mashup script for each POJO method that you want to expose as a service.

In the wrapper mashup script, add <input> declarations to accept input parameters for the Java method and an <output> declaration to receive the method return. Add a <script> statement to invoke the Java code.  An example is given below:

 Pre-Processing/Post-Processing Services with Mashups

Typically decoding encrypted data, encrypting data, decompressing or compressing data, filtering data or data scrubbing is done as part of pre- or post-processing. This can be handled by mashup that uses scripting to invoke pre- or post-processing code.

To implement pre- or post-processing, you must:

1. Write the logic to handle pre-processing / post-processing as a Java class or use Java libraries, as needed.

2. Add the custom Java classes to the classpath for the EMML Reference Runtime Engine.

3. Restart the EMML Engine.

Sample mashup script for pre-processing is given below:

Declare <input> variables to receive the data to be pre-processed and an <output> variable for the result.

Add a <script> statement to call the pre-processing logic. In this example, an encryption class that decrypts the accessed input parameter.

Invoke the service with the pre-processed data.

For detailed information refer to http://www.openmashup.org/omadocs/v1.0/emml/advMashupIntro.html.

Mashups are expected to be used by the business users directly (and not require programming skill) and the tag language EMML makes it easy to learn and use. Using the simple statements and commands of EMML, quite a few useful mashups can be created.

As usage of Mashup accelerates, the tendency to create more powerful and complex mashups would follow. In this blog, I have tried to give the crux of some of these advanced techniques.

Web clipping for direct use of HTML

Using the web clipping (or Screen scraping) technique, you can get the entire HTML from any URL as a service response. The EMML Reference Runtime Engine converts the HTML retrieved by the <directinvoke> statement is to XHTML in the http://www.w3.org/1999/xhtml namespace. You can filter, combine or transform with XHTML to create your own output.

Example of <directInvoke> resulting in web clipping:

The itemlink list in the XML result is as follows:

Normalizing Data for Effective Joins, Grouping or Filtering

Mashups – as the name suggests – is expected to get results from different services (not designed to work with each other). Though similar data is obtained from these services invoked, they would not be in identical forms. For effective usage of joins, grouping or filtering these results from different services, normalization of data to a single representation becomes essential. Creating a custom XPath function is the best method to normalize data.

The following example shows joining mortgage rates from two web sites – one refers to the APR and the second uses custom terms:

Create the custom XPath function – to normalize the custom terminology – as a Java class myFinanceFunction that extends org.oma.emml.client.EMMLUserFunction.

Compile this class, adding web-apps-home/emml/WEB-INF/lib/emml.jar to the classpath. Deploy the compiled class to web-apps-home/emml/WEB-INF/classes for the EMML Engine that host the mashups using this function.

Add a namespace for the class as an xmlns attribute to the <mashup> tag that uses this function.

Use the custom function in the XPath expressions in the <join> statement (or where data needs to be normalized).

Removing Duplicates With Filtering

To remove duplicates in a mashup, simply merge, join or group results. Sort the combined results (If needed) based on the key field that determines uniqueness to ensure that duplicates are contiguous. Use <filter> with a filtering expression that compares the key value of either the preceding or following ‘item’ to determine if this ‘item’ is unique. The filter expression can use the axis feature in XPath to compare preceding or following items.

Following is a simple <filter> statement:

<filter inputvariable=”$a” outputvariable=”$a” filterexpr=”/rss/channel/item[contains(title,'Java')" />]

In addition to the default XPath axis – the child axis, you can refer to previous nodes (preceding / preceding-sibling), following nodes (following / following-sibling), the parent node, ancestor nodes, descendant nodes and others. You can also use wildcards like following::* or following::node() to identify all following nodes of any name.

The following example checks the title of each item (after merging results from two RSS services) to remove duplicates:

I shall cover some more techniques in the second part of this blog. The Advanced Mashup Techniques are detailed in http://www.openmashup.org/omadocs/v1.0/emml/advMashupIntro.html.

Open Mashup Alliance (OMA) is a consortium aimed at successful use of Enterprise Mashup technologies and adoption of an open language – Enterprise Mashup Markup Language (EMML) – that promotes Enterprise Mashup interoperability and portability. EMML is an XML language to describe the processing flow for a mashup. The OMA provides the EMML schema and also a reference runtime implementation that processes mashup scripts written in EMML.

The EMML package can be downloaded from http://www.openmashup.org/download. Before installing the EMML package, install the JDK (1.5 or above) and Tomcat 5.5 or another compatible application server. EMML samples can be downloaded from http://www.openmashup.org/download.

You can create a mashup script using any XML editor. Adding the EMML schema, emml-install/EMMLSpec.xsd, to the XML editor – enables the editor to provide syntax assistance and validation. A Mashup script – an XML file that uses EMML – is created to define:

• the services and operations to be used by the mashup and
• the actions to apply to service responses to construct the results of the mashup.

Following is a simple mashup script that invokes a service (most viewed stories) and filters the output (business).

 Two categories of elements are added to mashups:

• Declarations for variables, parameters, data sources, namespaces, macros or metadata.
• Statements that perform actions – to invoke services, control flow, act on the results or use custom statements defined in macros – for the mashup.

Variables and Parameters

Variables and parameters must be declared (explicitly or implicitly) before they are used in any other EMML statement. Variables and parameters have a name, a datatype, data (a value), a scope (global or local) and optionally a default value. Variables can be created implicitly by assigning a name to the outputvariable attribute for any statement.

• <input> is an optional element to declare parameters that can be used as input to mashup, operation or macro.
• <output> is the parameter that holds the result returned from a mashup, operation or macro.
• <variables> is a list of variables to use for input, output or to hold any intermediate data or document content in the flow of mashup processing.

Accessing a database

To invoke SQL queries or statements, <datasource> with connection information for the database must be declared.  Each <datasource> declaration must have a URL that defines the JDBC connection and database login information (or) a JNDI name for connections to the database. Connection information can also be set dynamically as follows:

Note that, to use a <datasource>, you must add the JAR files containing JDBC drivers for your data source to the EMML Reference Runtime Engine classpath and add configuration for these drivers.

Use <sql>to issue individual SQL queries to a data source – for example:

Use <sqlUpdate> to execute any other SQL statement against a data source – for example:

<sql> and <sqlUpdate> can also be used to invoke stored procedures. SQL transactions can be managed using <sqlBeginTransaction>, <sqlCommit> and <sqlRollback>. EMML currently does not support distributed transactions.

Accessing a Web Service

Using <directinvoke>, you can invoke a publically accessible web service or website. In case of service invocation error, the EMML Reference Runtime Engine returns error information in two built-in variables: faultcode and faultmessage.

Transforming Intermediate Results

The mashup statements that can be used to select specific results, sort results, or otherwise transform the data in variables include:

• <assign> – to assign values or copy part of all of a variable
• <filter> – to filter a variable
• <group> – to group data in a variable and optionally filter or sort
• <sort> – to sort a variable
• <annotate> – to add nodes and data to a variable
• <xslt> – to transform data using XSLT stylesheets

Additionally, XPath functions can be used to transform data, change datatypes, perform calculations or determine boolean conditions.

Using the <script> statement, user-defined code can be executed within mashup processing. JavaScript and JRuby 1.0 are the scripting languages currently supported.

Combining Service Results

Results of two or more component services can be combined using:

1. <merge> – to merge service results that have homogenous, document-type structures (similar to database union operation)

2. <join> – to define how the data from disparate variables should be joined. Variable data should have repetitive structures that are related based on some criteria – say, the foreign keys that define the relation. It is comparable to inner joins for databases.

3. XQuery – to obtain outer joins in mashups.

Constructing the Mashup Result, Input or Intermediate Variables

The following statements can be used to define the structure of the result that is returned from a mashup, or to construct complex input parameters or any complex intermediate variable:

• <constructor> to construct a well-formed document.
• <appendresult> to add one or more well-formed items to a variable.
• <select> (within <mashup>, <operation> or <macro>)
• <select> within a <join> command to select specific items within a set of repeating items. See <join>.
• <group> constructs repeating structures from sorted and optionally filtered node sets.

Literal XML can also be used to construct the contents of variables or input parameters.

Controlling Mashup Processing Flow

The mashup commands to control the processing flow for a mashup operation includes:

• <if> for conditional processing
• <for> for looping based on simple counts
• <foreach> for looping through a set of nodes. This can be simple iterative loops or loops can be processed concurrently.
• <while> for looping as long as a condition is true
• <break> to forceably stop looping statements
• Concurrent Mashup Processing Flows using <parallel>.
• SQL Transactions using <sqlBeginTransaction>, <sqlCommit> and <sqlRollback>.

Using <emml-meta> and <user-meta>, metadata can be declared to allow setting of processing flags or pass other information to applications that will use a mashup, use a macro or handle mashup results.

EMML, like any other tag language, is quite simple to use and can be used by business users to create mashups quickly. EMML also supports various statements and commands that can be used to create complex Mashups depending on the needs of the user.