With the increase in the Web 2.0 applications, there are frequent situations where the application needs to insert or update data only for a subset of columns for a given table. And it becomes all the more difficult when it is not known which columns are being inserted to or updated until application execution time. For example, the customer address change request could mean changing the entire address or just the apartment number or any combination of columns (street name, city, zip code, apartment number and others).

Prior to DB2 version 10, the options for handling different combination of insert and updates included:

• Using a dynamic SQL statement constructed at execution time for each request. Obviously this has overhead and impacts performance.
• Coding SQL statements for all expected combinations of columns – and if we assume that every single combination is possible, the number of SQL statements to be coded grows exponentially. This indeed is a tedious process and has negative impact on application maintenance.
• Have one SQL statement that included all the expected columns. While the customer request comes with data corresponding to only a subset of columns, it becomes the application’s responsibility to figure out the correct values for the other columns – say the old values in case of update, default values in case of insert and NULL values for nullable columns etc. Again, this makes application program complex – and testing difficult.

DB2 10 for z/OS addresses this difficulty of supporting changes to only a subset of columns in a table by introducing the extended indicator variable support. Preparing separate INSERT or UPDATE statements for every combination of columns that are being inserted is no longer required.

Using these extended indicator variables we can indicate to DB2 that the value for the associated host variable is not supplied and also specify how DB2 should handle the missing value. By specifying the extended indicator variable value of -5 (implying default), we can indicate that the target column of the host variable is to be set to its defined DEFAULT value. By specifying the extended indicator variable value of -7 (implying unassigned) we can indicate that the target value of the host variable is UNASSIGNED and the target column is to be treated as if it is not included in the statement.

By enabling the extended indicator variables, you do not need to resubmit the current value for a column, or know the default value for a column. Extended indicator variables can be enabled at a package level, by using the EXTENDEDINDICATOR option on the BIND PACKAGE (or REBIND PACKAGE) command. Extended indicator variables can be enabled on a statement level for dynamic SQL by using the WITH EXTENDED INDICATORS attribute on the PREPARE statement.

When the extended indicator variable is used as an input variable:

When the extended indicator variable is used as an output variable, the values have the same meanings as for an indicator variable (negative value indicates NULL and 0 or positive indicates specified value). As this new variable is meant is intended for allowing applications to skip sending data for any of the host variables specified on the statement, in case of output, DB2 does not set indicator variables to any of the special values.

The following restrictions apply in specifying the special extended indicator variable values. It can be specified only for host variables that appear in:

• The set assignment list of an UPDATE operation in UPDATE or MERGE statements
• The values list of an INSERT operation in INSERT or MERGE statements
• The select list of an INSERT statement in the FROM clause of the SELECT statement
• The source-table parameter of a MERGE statement

For the INSERT statement, setting the extended indicator variable to -5 or -7 is the same, as the result is to insert a default value for any column that is missing an input value (that is, unassigned). With multi-row insert, the extended indicator variable values of default or unassigned can be used inside the indicator array.

For UPDATE or MERGE UPDATE, setting the extended indicator variable to -5 leads to the column being updated to the default value and setting the extended indicator variable to -7 leads to the update of the column not being applied.

It is important to understand the impact of these extended variables on UPDATE statement quite thoroughly, as it could result in behaviors other than intended by the developer. IBM provides the following example to get through the point.

While bind the DBRM to a package, it is required to the EXTENDEDINDICATOR(YES) option to enable the extended indicator variables.

After executing the above code, if we query the table the result is: (‘Josef’, ‘Earth’, ‘Unknown’, ‘00000′). As the value of -7 is specified in the extended indicator variable, the columns NAME, CITY and ZIP are not at all updated (i.e., the value Michael doesn’t have an impact at all). As the value of -5 is specified in the extended indicator variable, the COUNTRY column was set to the DEFAULT value (‘Earth’), rather than the given value (‘Australia’).

If the extended indicator variable for this package is disabled by rebinding with EXTENDEDINDICATOR(NO) option and tested, the results would be different. Re-running the UPDATE and querying will show that all the columns were set to the value of NULL. The reason being that the disabling of extended indicator variables results in these values being treated as the “indicator variable” and hence the negative value (-5 or -7) implies a NULL value.

As can be seen in the example, the simple enabling or disabling of Extended indicator variable and setting the values of -5 or -7, the application developer can control the actual columns that would be updated – irrespective of where the function is called from and what value it gets.

Thus these extended indicator variable enable applications to use a single SQL statement – for handling different combinations of INSERTS and UPDATES – without burdening the application with the need to fill the missing columns data explicitly with the current value or default value. Thus DB2 functions exploiting these extended indicator variables can be reused across different channels accessing them and form the core services in an SOA environment.

Service Oriented Architecture (SOA) that facilitates code reuse, interoperability, service management and system integrity is becoming a norm in today’s IT development. In latest version of DB2 (version 10), IBM has introduced quite a Application Integration features in DB2 version 10 aimed at further simplifying and improving the usage of distributed function of DB2 through SOA.

Single Load module across environments using DSNULI

Prior to version 10, the DB2 applications must be link-edited with the appropriate Language Interface modules – DSNRLI for RRSAF and Stored Procedures, DSNALI for Call Attachment Facility (CAF), DSNELI for TSO Attachment Facility, DSNCLI for CICS Attachment and DFSLI00 for IMS Attachment. Even if the same program logic is to run in different environments, multiple link-editing has to be done and this obviously resulted in usability issues.

DB2 version 10 provides the new Universal Language Interface module (DSNULI) to address this issue. Now you can produce a single module (link-edited with DSNULI) that is executable in any of the following runtime environments: CAF, CICS, TSO and RRS (Resource Recovery Services) and MVS Batch. DSNULI does not support dynamic loading or IMS applications.

DSNULI has nine entry points – associated with different attachment scenarios.  It dynamically loads and branches to the appropriate language interface module based on the entry point used by the application (say when a attachment facility entry point was used explicitly), or based on the current environment. Naturally, this kind of execution time detection hinders performance, so there is a trade-off between ease of application deployment and the application speed.

Enhanced performance and diagnostic monitoring support

DB2 10 for z/OS enhances performance monitoring support and monitoring support for problem determination- especially so for distributed workloads. Using Instrumentation Facility Interface (IFI), it captures and externalizes monitoring information for consumption by tooling.

It introduces a unique statement execution identifier (STMTID), defined at the DB2 for z/OS server, returned to the DRDA application requester and captured in IFCID records for both static and dynamic SQL. To support problem determination, the statement type (dynamic or static) and new statement execution identifier (STMT_ID) are externalized in several existing messages (including those related to deadlock, timeout, and lock escalation). In these messages, the STMTID associated with thread information that can be used to correlate the statement execution on the server with the client application on whose behalf the server is executing the statement.

DB2 version 10 also includes new trace records that provide access to performance monitoring statistics in real time, and allow retrieval monitoring data without requiring disk access. Some existing trace records that deal with statement-level information are modified to capture the new statement ID and new statement-level performance metrics.

Java query performance improvements

With version 10, IBM Data Server Driver for JDBC and SQLJ type 2 connectivity to local DB2 for z/OS data servers uses more efficient methods for processing forward only, read-only queries:

• Limited block fetch protocol
• True progressive streaming for retrieval of LOB and XML data
• Query buffers that reside in 64-bit addressable storage for applications that run in a 64-bit JVM

Elimination of DDF private protocol

With DB2 version 10 onwards, DDF private protocol is not supported. A REXX tool (DSNTP2DP) is provided to prepare a DB2 subsystem to convert from the private to DRDA protocol.

DRDA support of Unicode encoding for system code pages

DB2 version 10 includes DRDA support of Unicode encoding providing improved response time and less processor usage for remote CLI and JDBC applications as it removes the need for converting DRDA instance variables between EBCDIC and Unicode.

Support for 64-bit ODBC driver

In addition to the 31-bit ODBC driver, DB2 Version 10 provides a new 64-bit ODBC driver (XPLINK only) that allows 64-bit ODBC applications to take advantage of the expanded 16 million TB address space. The 64-bit ODBC driver runs in 64-bit addressing mode and reduces the virtual storage constraint, as it can access the user data above the 2 GB bar in the application address space.

One of the most important change is the introduction of Extended indicator variable that allows inserts and updates of different combinations of subset of columns – improving reusability. I shall elaborate on Extended Indicated variable in a later blog.

Clearly these enhancements are aimed at making DB2 - especially the DB2 in zOS – as the database of choice in an SOA environment. More such changes are expected to follow in the later versions – adapting DB2 to the needs of the today.

In DB2 version 10 for z/OS, IBM introduces a new access type called Hash Access and access method called Hash space. This new option of organizing tables using hash improves performance of queries that access individual rows using equal to predicate (say getting data using customer number or product number).

DB2 uses an internal hash algorithm with the Hash space to reference the location of the data rows. Thus using hash provides the advantage of selecting a hash access path which (in most cases) means only one I/O to retrieve a row from the table in turn reducing the CPU usage and improved response time makes it a very compelling proposition.

Using Hash access also means no need for maintaining the data sequence or clustering the index (for that matter, index clustering is not allowed if the table is hash organized). This results in efficient insert processing and avoids data sharing contention in maintaining a clustering sequence or clustering index.

While creating a table, the hash access can be enabled by adding the organization-clause in the CREATE TABLE statement.

• o ORGANIZE BY HASH UNIQUE (column-names) HASH SPACE (hash-space-value)

“Organize by Hash” specifies that a hash is to be used for organizing the data of the table.  The list of column names defines the hash key based on which the placement of a row is determined. Mentioning UNIQUE prevents the table from containing more than one row with the same value of the hash key (this applies even for the NULL value).

The amount of fixed hash space to be pre-allocated (default value is 64M) for the table can be specified. For tables partitioned by range, this space is for each partition. The value specified can be in KB, MB or even GB.

Hash organization is optimal when a table is of a stable or predictable size. When a table is organized by hash, DB2 automatically creates an overflow index. If the table exceeds the specified hash space, the extra rows are placed in the overflow index. The rows in the index are not hash access enabled and DB2 scans through the index to retrieve them.

An existing table can be altered to use hash access organization by specifying the “ADD ORGANIZE BY HASH” in the ALTER TABLE command. The table has to be reorganized and incompatible functions like index clustering would be disabled in the process. In case of existing tables by specifying AUTOESTSPACE(YES), we can let DB2 automatically estimate the best size for the hash space using the real-time statistics.

Using hash, obviously requires additional disk storage. Hash access can be monitored and the storage space used can be tuned for improved performance. The ACCESSTYPE column in the plan table has a value of ‘H’, ‘HN’, or ‘MH’, hash access is used to access the data. HASHACCESS (in SYSTABLESPACESTATS) indicates the number of times that hash access paths have been used to access the table.  Using HASHLASTUSED, we can find out if DB2 has used the hash access path recently.  

If HASHACCESS value is very low (even after several queries having accessed the table) or if DB2 has not used the hash access path recently – hash organization from the table can probably be removed thereby saving storage space.

The size of the hash space can be altered, if required, using the ALTER TABLE statement. Hash space must be increased:

• If the TOTALENTRIES (in SYSINDEXSPACESTATS) value is greater than 10% of TOTALROWS (in SYSTABLESPACESTATS).
• If the DATASIZE (in SYSTABLESPACESTATS) is greater than HASHSPACE (in SYSTABLESPACE). The HASHSPACE must at least be 20% larger than DATASIZE.

Hash organization is only available on Universal Table Spaces (UTS) that are both segmented and partitioned. Hash access cannot be used in the following cases:

o LOB and XML table spaces

o Tables defined with APPEND YES.

o Materialized Query tables

o Tables in basic row format (for hash, using reordered row format – RRF – is required)

Some of the restrictions in using hash organized tables include:

o Parallelism is not used for parallel groups when hash access is used

o Hash access is not used to access fact or dimension tables in a qualified star join

o For queries that use multi-row fetch, hash access is chosen only if the query contains an IN clause that returns multiple rows.

Hash table organization would be most useful where unique keys are accessed using equal predicates – customer id, product id, document id – and the order of the data is immaterial.

DB2 version 10 for z/OS, recently released, provides features for row versioning and time travel querying. With audit and compliance requirements demanding that the multiple versions of the data pertaining to a long period is stored and the changes made track able, we can almost say that it was high time DB2 came up with these features.

What we used to achieve using triggers, stored procedures and complex querying is now achievable using simple SQLs. I believe that it is not overstated when one of the insurance customers – who was involved in the Beta version – has said that over 80% of the applications requiring temporal features can exploit this and can save time and also make applications easier for business users to understand.

DB2 is providing support for both system time period as well as business time (or application time) periods. System time simply is based on the time when the transaction changes are made in the system, where application time is more logically tied to the application. When the application requires temporal support based on both, then IBM lets the usage of bitemporal tables.

The following points are worth noting as the system managed temporal features can be exploited without impacting the existing applications:

1. The historic data (or versions) is maintained in a separate history table. This also means less impact on performance of applications when they operate in normal mode (accessing only current data)

2. The temporal columns (begin time, end time, transaction time) etc. can be automatically generated and marked as hidden. So the existing applications can work as is – absolutely no changes required even in case of SELECT * or INSERT.

3. The temporal table and the history table can be associated and disassociated easily (just by using the ALTER TABLE command).

4. The historical data is accessed ONLY if the SELECT statement includes the period specifications.

The three key types of period specifications supported are:

• AS OF certain time
• FROM time1 TO time2
• BETWEEN time1 AND time2

DB2 handles these versioning and time travel querying, automatically and transparently – especially so in case of system-managed temporal tables. Business Time based is a bit more sophisticated, as the users have to handle the setting of timestamps and past, present and future effective dates and their associated business data are all maintained in a single table. But then it gives better control to the users and enables the applications to effectively manage and track transactions.

Applications that handle Insurance policies, credit cards, loans etc. are the ones that can directly start using these features. In addition to compliance, business analytics is an area which can make best use of these temporal features. More enhancements and performance improvements on the temporal features can be expected in the next releases.

IBM has been driving to improve DB2 utilization in the cloud and a series of announcements including the DB2 virtual appliance shows a trend where we can expect more. To quote IBM, its DB2 view of the Cloud Computing is “what technologies and business models can we bring to market to help our customers realize the promise of the Cloud Computing”. The promise of cloud computing is its potential to reduce costs and improve agility.

IBM’s target users for DB2 on cloud includes everyone – Developers, Startups, SMB, Enterprise, Service Providers, SaaS Vendors. Availability of no-charge DB2 options in cloud makes it worthwhile to start exploring DB2 in cloud.

IBM’s DB2 Strategy around cloud computing falls into four key points:

• Deliver key technologies to support its customers private cloud initiatives
• Partner with key public providers to fully integrate DB2 into the ecosystem
• Provide robust DBMS for SaaS vendors
• Offer terms and conditions and pricing to make DB2 the best DBMS for the Cloud.

Technically its threaded-engine Architecture that minimizes memory requirements and enables efficient use of multi-core processors used in virtualized systems, Autonomic features like Self Tuning Memory Manager that reduce administrative costs, Support for various development platforms like Java, .Net, PHP, Ruby on Rails etc. makes DB2 an excellent choice for cloud environments.

In cloud context being technically good is only one part of the solution. Cloud computing is expected to reduce costs, improve flexibility and enhance agility and unless these benefits are provided for success would remain elusive. IBM seems to be accepting this reality and showing its willingness to adapt and coming up with various licensing and pricing strategies.

DB2 support for Private cloud is centered on availability of optimized DB2 in Virtualized environments including DB2 virtual appliances, standardizing and enhancing DB2 server provisioning and automation features. The sub-capacity pricing (now available on all DB2 Editions) where the charges are based on the system usage and not the full-capacity of the hardware coupled with the option of licensing on a per-day basis provides the basis for cost-effective virtualization.

DB2 public cloud presence involves mainly partnership with leaders in the cloud space and includes:

• Amazon – public cloud infrastructure
• RightScale – Platform for cloud management
• Morph labs – Web application Hosting Infrastructure
• Corent – Rapid SaaS development
• Canonical – DB2 Virtual appliance running on Ubuntu cloud computing platform

Additionally, DB2 Enterprise Developer Edition is available in IBM Smart Business Development and Test Cloud.

DB2 in Cloud is available under a variety of pricing option. The free edition, DB2 Express-C is available at no cost but no IBM support. IBM points out that there is no limit on the database size or number of users making it an ideal choice for SME as well as developers. DB2 Express-C is easy-to-setup, easy-to-manage, and includes self-managing capability and also comes with the pureXML technology suited for Web 2.0 applications. For more scalable DB2 required to run large business database loads, customers would have to pay DB2 licenses.

Customers who already have a DB2 license can use it in the cloud with no additional cost and IBM would continue to support it. In between this free and perpetual license, various pricing options like pay-as-you-go, hourly usage fee, purchase optional DB2 support etc. are available.

IBM wants to position DB2 as the DBMS of choice for the SaaS vendors who aim at exploiting Cloud computing trend. The advanced autonomic features, technologies like Deep compression and pureXML coupled with attractive pricing is what IBM is betting on to achieve this.

DB2 in cloud – or for that matter any other Database – is not yet ready to handle mission-critical or I/O intensive applications. But there is a whole suite of areas that could right away start exploiting cloud – Prototyping, development and testing, periodically run applications (end-of-month processing or day end batch jobs), handling variable workloads without dedicating resources, database backups, databases for disaster recovery, Web 2.0 Applications.