As discussed earlier, VSAM RLS that enables simultaneous access of VSAM files across multiple CICS region and batch (to a limited extent) is quite relevant today. To get good performance of VSAM RLS, it is necessary to understand better how it works and also the parameters that can be used for tuning.

VSAM RLS performance and tuning involves focusing on the performance path in RLS – referred to as VSAM RLS I/O path.  VSAM RLS I/O path has four main components:

1. VSAM record management (VRM) – VRM provides the interfaces to the VSAM interfaces like GET, PUT, POINT, ERASE using with the application communicates with RLS. When we code one of these macro interfaces, the parameters to VRM is passed as an RPL control block.

2. Storage Management Locking Services (SMLS) – SMLS interfaces with VRM above and with XCF, the MVS component which provide locking services in the coupling facility. The lock structure in the couple facility is called IGWLOCK000.XCF is called to obtain, release or alter lock.

3. Ses Cache Manager (SCM) – SCM calls XCF caching services to obtain directory elements, or to read or write to the cache structure in the coupling facility. SCM also interfaces with the next component, BMF

4. Buffer Manager Facility (BMF) – BMF interfaces between VRM and SCM to locate the buffers and move them to local buffer pool if necessary. When BMF reads buffers into buffer pool, those buffers are kept there even after the data sets are closed. This is a powerful function, as it enables the use of the buffers read in previously when the data sets are re-opened and the impact is significant when the millions of records are involved. BMF manages the buffer space using the Least Recently Used (LRU) manager.

The four subcomponents that make up the I/O path typically behave as follows:

• When an application has issued a VSAM request (GET or PUT), VRM gets control.
• VRM then calls SMLS for performing record locking request.
• SMLS in turn calls XCF to obtain a lock for the particular record
• Then, BMF is called to find some buffers for the request.
• If the record cannot be found in the buffer pool, SCM is invoked to check if it is in the cache.
• If the record is not in the cache, the record is retrieved from DASD (the actual I/O happens).

Obviously, the shortest path leads to best performance.  Skipping the last step (which involves retrieval from DASD) would result in significant performance improvements.  The elapse time is improved by around one hundred times if you can get buffer hit. IBM test data shows the following results:

Details of the whole path is clearly elaborated in: http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/topic/com.ibm.iea.zos/zos/1.0/DFSMS/zOSV1R0_DFSMS_RLSPerformance_Tuning_Overview.pdf

Using the SMF type 64 and 42 records as well as RMF reports, you can collect the data on current performance of VSAM RLS. SMF 42 records have five subtypes – 15 to 19, each keeping different data about RLS statistics:

• Subtype 15 keeps statistics by storage class.
• Subtype 16 is statistics by each individual dataset.
• Subtype 17 is all statistics related to locking SMLS.
• Subtype 18 is caching statistics, which involves with the SCM component.
• Subtype 19 is BMF statistics, which involves the LRUs.

Tuning the RLS parameters described below based on the application need would result in significant performance improvements.

At the system level, RLS keeps all the parameters in SYS1.PARMLIB(IGDSMSxx) member of PARMLIB.

• RLS_MAX_POOL_SIZE specifies the maximum size of the SMSVSAM local buffer pool. Default size is 100 MB. It is a sysplex wide parameter. SMSVSAM attempts to not exceed the buffer pool size specified, although more storage might be temporarily used.
• RlsAboveTheBarMaxPoolSize specifies the total size of the buffer management facility (BMF) above the 2-gigabyte bar. This parameter can be mentioned for each system enabling different systems in a sysplex to have different values.
• RlsFixedPoolSize specifies the amount of real storage (both above and below the 2-gigabyte bar) to be dedicated to VSAM RLS buffering. This is used for page-fixing buffers – that can be used for critical data – that are not paged out by LRU. Using fixed buffers does provide significant performance improvements.
• RLS_MaxCFFeatureLevel specifies the method VSAM RLS caching uses to determine the size of the data that is placed in the CF cache structure. Value of Z indicates that RLS will cache CIs less than 4096 only. This options saves space in the CF cache structures and is useful if the data is read-only and remains valid in the local buffer pool. Value of A indicates that RLS will cache CIs up to 32768. This option requires more space in the RLS CF cache structures and is useful when shared data is updated across the sysplex.
• DEADLOCK_DETECTION specifies the interval for detecting deadlocks between systems.

In the dataset level, the following two parameters that can be specified in the data class are relevant:

1. RLSAboveTheBar that specifies if the SMSVSAM address space can take advantage of 64-bit addressable virtual storage during VSAM RLS buffering. The value of YES is recommended for high volume applications for best performance.

2. RLSCFCACHE specifies the amount of data that will be cached and can have the following values: ALL (default) specifies that RLS to cache both the data and index CI; NONE indicates that only the index data will be cached; Index CI that is always searched is more important than the data CI and hence has a higher priority than data CI. If you need to conserve space in coupling facility, this option of caching the index alone can be used. UPDATESONLY indicates that only WRITE requests will be placed in the cache structure;

While it is obvious that higher the buffer size set, better the performance, consideration should be given to the amount of real storage available. If the real storage available is limited and buffers specified are very high, frequent paging by LRU may occur.

Understanding how LRU works would be useful for setting these buffer sizes at optimal level. Refer http://publib.boulder.ibm.com/infocenter/ieduasst/stgv1r0/topic/com.ibm.iea.zos/zos/1.0/DFSMS/zOSV1R0_DFSMS_RLSPerformance_Tuning.pdf.

 As summarized by IBM:

• The VSAM “I/O” read path can see 100 times improvement when valid buffers are located in the local buffer pool.
• VSAM RLS 64 bit buffering allows for larger local buffer pools and increased buffer hits.
• Whether it is 31 bit or 64 bit pool, RLS Cache structures must be increased to accommodate the larger local buffer pool sizes.
• Adequate real storage must be available to accommodate the larger local buffer pool sizes.

In addition to this, the applications can do their bit for performance enhancements by having appropriate requests.

At the RPL level, using OPTCD you can specify if the request is asynchronous (SRB) or synchronous (TCB). Asynchronous request are better from performance point of view, as SRBs cannot be interrupted all the time and hence runs quicker. Note that in case of asynchronous requests, the CPU time of the request is going to be associated with SRB time and with the client space, not SMSVSAM (important factor in case of cost distribution).

Similarly the performance is impacted depending on whether the read is direct or sequential. In case of direct requests, RLS constantly searches the index to find the data CI and hence is more costly that a sequential read or horizontal read through the index.

When issuing GET or PUT or while opening the dataset, you can use the RLSREAD parameter to indicate the read integrity required. If read integrity is not required, specify NRI (No Read Integrity) which skips the step of calling SMLS for locking and hence provides improved performance.

The CICS Dynamic Scripting Feature Pack (optional product) seems to bring the best of both worlds – benefits of quickly developing scripted, Web 2.0 applications with simple and secured access to CICS application and data resources. The Dynamic Scripting Feature Pack basically embeds and integrates technology from WebSphere sMash into CICS TS run time and includes a PHP 5.2 runtime along with Groovy language support. Access to CICS resources is achieved using the JCICS APIs.

Dynamic Scripting Feature Packs can be used to:

• Rapidly develop, evolve, and deploy rich web applications, using dynamic scripting languages.
• Expose RESTful web services and widgets for use in mashups and consoles.
• Compose composite services, widgets, and applications that combine Web 2.0- style public and enterprise data feeds.

CICS dynamic scripting is part of project Zero (ref http://www-01.ibm.com/software/htp/cics/scripting/).  The idea of Project Zero is to provide a powerful development and runtime environment for dynamic web applications at the same time having the overall experience of being radically simple.

From a Project Zero developer’s perspective, the application is the server. Each dynamic scripting application is a standard (well-known) directory structure containing content within that structure. All you have to do is create an application, add application code, then start the application. The capabilities such as listening on a specified port and responding to HTTP, interacting with a database, using email, and so on are added to the application by adding dependencies.

In CICS Dynamic Scripting, all applications depend on zero.cics.core module. The zero.cics.core module provides much of the base functionality for a CICS-based Zero Application. The desired characteristics of the TCPIPSERVICE, URIMAP, PIPELINE, and JVMSERVER resources, can be specified in the zero.config and the zerocics.config files. If the application depends on additional features or capabilities that are related to HTTP, database interactions, Dojo support, email, and more, the dependencies are specified in the application’s ivy.config file in the application’s config directory.

With CICS Dynamic Scripting, developers can now use Web 2.0 technologies, such as Asynchronous JavaScript and XML (AJAX), Representational State Transfer (REST), Atom, JavaScript Object Notation (JSON), and Really Simple Syndication (RSS) to unleash and reuse enterprise content. The built-in Zero Resource Model (ZRM) provides a simplified way to create RESTful resource handlers with a data store that can be mapped to DB2 and Derby with improved resilience, security, and performance. Good samples are available at Refer to http://www.projectzero.org/wiki/Development/CicsSamples.

For persons who are new to CICS , to effectively write and implement a Project Zero application using the CICS Dynamic Scripting Feature Pack points to be noted are:

• Most CICS options and services that are traditionally accessed using EXEC CICS API are available to Java programs through JCICS. For example, the programs can be accessed using link(), xctl(), SetNextTransaction(), setNextCOMMAREA() and setNextChannel(). For complete list of JCICS Commands, refer to http://publib.boulder.ibm.com/infocenter/cicsts/v4r1/index.jsp?topic=/com.ibm.cics.ts.java.doc/topics/dfhpjla.html.
• The interface to the Project Zero technology, both in WebSphere sMash and in CICS Dynamic Scripting, is through a command-line interface (CLI). The CICS-provided CLI communicates to an associated CICS region using EXCI. Each ZERO command issued from the CLI, and each dynamic scripting application runs in its own CICS-based JVMServer resource.
• Data passed to or from CICS is in byte arrays whereas a Java program wants access to the data in an object-oriented fashion using getters and setters.
• When interacting with a CICS resource such as a VSAM file, Rational Application Developer can be very useful as it provides the ability to analyze a COBOL copybook and generate a corresponding Java object containing getters and setters for each field in the COBOL-like data structure.
• JZOS toolkit (i.e., additional Java support classes) included with the z/OS distribution of Java can be used to generate data objects to interface with CICS resources.

To reiterate, CICS Dynamic Scripting can be used to:

• Create reports, dashboards and widgets
• Expose CICS assets in mash-ups
• Quickly front end existing applications
• Create productivity applications
• Develop without a dedicated budget
• Quickly try out new business ideas
• Port existing unmanaged PHP into CICS
• Introduce new IT staff to CICS via PHP.

Speed, Simplicity and Agility are the keywords in dynamic scripting. It is important to note that while traditional CICS applications are expected to handle large number of concurrent users and high volume with rigorous availability requirements, the dynamic scripting applications (that is developed in days or weeks with low cost – by anybody with basic scripting skills) are expected to be tactical with fewer concurrent users, with low volumes and fit for purpose availability requirements.

COBOL is still the most widely used language in mainframe – both for online transaction processing as well as massive batch processing. Even minor improvement of performance of repeatedly executing COBOL programs directly provides savings of CPU and hence the costs. One of the simple things – and often neglected one – that would optimize COBOL performance is to use the right set of Compile options.

In quite a few performance tuning engagements, the culprit is that the compiler option (set in the JCL or the configuration tool) used by almost all programs having compiler options (mostly left to default) that bring down the performance. Even the veteran COBOL programmers, tend to ignore these and focus on programs alone while trying to improve performance. In this article, I would like to highlight the COBOL compiler options which impacts performance.

The OPTIMIZE compiler option can be used to improve the efficiency of the generated code. NOOPTIMIZE is the default. OPTIMIZE(STD) results in the following optimizations:

• Eliminate unnecessary transfers of control and inefficient branches (including those generated by the compiler).
• Simplify the PERFORM and CALL statement to a contained (nested) program, by placing the statements inline, eliminating the need for linkage code.
• Eliminate duplicate computations (such as subscript computations and repeated statements).
• Eliminate constant computations by performing them when the program is compiled.
• Eliminate constant conditional expressions.
• Aggregate moves of contiguous items (say with the use of MOVE CORRESPONDING) into a single move.
• Delete from the program, code that can never be performed (unreachable code elimination).

In case of OPTIMIZE(FULL) option, additionally it:

• Discard unreferenced data items from the DATA DIVISION, and suppress generation of code to initialize these data items to their VALUE clauses (If the program relies upon unreferenced level 01 or level 77 data items, do not use OPTIMIZE(FULL)).

Note that OPTIMIZE requires more CPU time for compiles than NOOPTIMIZE, but generally produces more efficient run-time code. It is suggested that NOOPTIMIZE is used while a program is being developed, as   frequent compiles would be happening and it also makes it relatively easier to debug a program since code is not moved.

With DYNAM option, all subprograms invoked through the CALL literal statement will be loaded dynamically at run time. NODYNAM is the default. DYNAM allows sharing of common subprograms, provides control of using the virtual storage (that can be freed using CANCEL statement), but with a performance penalty as the call must go through a library routine, whereas with the NODYNAM option, the call goes directly to the subprogram. Detailed information is available at http://itknowledgeexchange.techtarget.com/enterprise-IT-tech-trends/static-dynamic-linking-in-ibm-cobol/.

According to IBM, for a CALL intensive application, the average overhead associated with the CALL using DYNAM ranged from 40% to 100% compared to that of NODYNAM.

Using the FASTSRT compiler option improves the performance of most sort operations. NOFASTSRT is the default. With FASTSRT, the DFSORT product (instead of Enterprise COBOL) performs the I/O on the input and output files named in the SORT . . . USING and SORT . . . GIVING statements.

One program that processed 100,000 records was 45% faster when using FASTSRT compared to using NOFASTSRT and used 4,000 fewer EXCPs.

XMLPARSE(XMLSS) option (the default) selects the z/OS XML System Services parser as against  XMLPARSE(COMPAT) uses the built-in component of the COBOL run time. While XMLSS provides additional capabilities, at present COMPAT option is found to be faster by 20-108%. But it is important to note that as IBM would focus more on XML Parser, the performance difference is most likely to get lower.

THREAD option that enables multi-threading in COBOL and can be used in a non-threaded application (ref. http://itknowledgeexchange.techtarget.com/enterprise-IT-tech-trends/multithreading-in-cobol/) results in runtime performance degradation due to overhead of serialization logic that is automatically generated. NOTHREAD is the default.

ARITH option that allows controlling the maximum number of digits allowed for decimal numbers. ARITH(COMPAT), the default, allows the maximum digits of 18 (which should serve well for most requirements) while ARITH(EXTEND) allows up to 31.

ARITH(EXTEND) causes performance degradation for all decimal data types because of larger intermediate results. The performance impact on an average is 16%, while for programs with heavy use of decimals, it could be as high as 40%.

AWO option implicitly activates the APPLY WRITE-ONLY clause for all physical sequential, variable-length, blocked files (irrespective of whether it is specified in the program). NOAWO is the default option. Using the APPLY WRITE-ONLY clause makes optimum use of buffer and device space. With APPLY WRITE-ONLY specified, the file buffer is written to the output device only when the next record does not fit in the unused portion of the buffer. Without APPLY WRITE-ONLY specified, a file buffer is written to the output device when it does not have enough space for a maximum-size record.

According to IBM, a program using variable-length blocked files and AWO was 86% faster than NOAWO (as the result of using 98% fewer EXCPs to process the writes).

BLOCK0 option changes the default for QSAM files from unblocked to blocked thus gaining the benefit of system-determined blocking for output files. NOBLOCK0 is the default. BLOCK0 is applicable for each file that meets all of the following criteria:

• The FILE-CONTROL paragraph specifies ORGANIZATION clause as SEQUENTIAL or omits it.
• The FD entry does not specify RECORDING MODE U.
• The FD entry does not specify a BLOCK CONTAINS clause.

AWO might apply to more files than it otherwise would, if BLOCK0 is also specified (as AWO applies only for blocked variable-length records). One program using BLOCK0 was found to be 88% faster than using NOBLOCK0 (using 98% fewer EXCPs).

Note that specifying BLOCK0 for existing programs might change the behavior of the program – especially for files opened as INPUT without block size.

NUMPROC(PFD) improves the performance of processing numeric internal decimal and zoned decimal data. With NUMPROC(PFD), the compiler assumes that the data has the correct sign and bypasses the sign fix-up processing. But use this option only if your program data agrees exactly with the following IBM system standards.

Note that NUMPROC(NOPFD)is the default – and recommended if the numeric internal decimal and zoned decimal data might not use proper signs (especially if the program has to process external data files). Also note that NUMPROC(NOPFD) or NUMPROC(MIG) should be used if a COBOL program calls programs written in PL/I or FORTRAN.

NUMPROC(PFD) – that can provide performance benefit between 5-20% – is advisable for performance sensitive applications after ensuring that the necessary conditions are met.

TRUNC(OPT) is another performance tuning option for performance sensitive application and should be used only when the data in the application program conforms to the PICTURE and USAGE specifications.

While the above points are related to runtime efficiency, the following two options are worth noting in a development environment:

• Use BUFSIZE to allocate an amount of main storage to the buffer for each compiler work data set. Usually, a large buffer size improves the performance of the compiler.
• Use the COMPILE option only if you want to force full compilation even in the presence of serious errors. All diagnostics and object code will be generated. Do not try to run the object code if the compilation resulted in serious errors: the results could be unpredictable or an abnormal termination could occur.

With judicial use of the compiler options – suited for the given environment – performance benefits can be achieved without modifying the programs.

CICS is IBM’s online transaction processing facility – and WebSphere can be considered as an equivalent. The words transaction and task are used to describe units of work within CICS. Within a particular system, a series of transactions can be well defined and understood so that it becomes possible to talk about relative performance in terms of transactions per second (or minute, or hour).

 There are three modes of CICS operation:

1. Non-conversational mode – It is of the nature of one question, one answer; resources are allocated, used, and released immediately on completion of the task. In this mode the words transaction and task are more or less synonymous.

2. Conversational mode – Multiple questions (key strokes by user) and answers (response from the system) during which resources are not released. Resources are, therefore, tied up unnecessarily waiting for users to respond, and performance may suffer accordingly. Transaction and task are, once again, more or less synonymous.

3. Pseudo-conversational mode (most widely used) – Allowing for the slow response (think time as well as data entry time) for the user, transactions are broken into more than one task (without the user perceiving it – programming is done in such a way that for the user there are series of actions taken before the transaction gets completed). The key advantage is that the resources in demand are released at the end of each task giving the potential for improved performance (and large number of concurrent users). In this mode, the input/output surrounding a single task is known as a dialog.

In CICS context, the word “user” means the terminal operator and CICS performance (as perceived by the user) is the response time – which is the time between the last input action and the expected response. Several such responses could be required before the user completes a function.

A CICS region can support external requests (from clients, including internet or another CICS region) which comes to it over a network using a wide variety of transport protocols.  CICS supports distributed computing and the client/server model by means of:

• Internet Inter-Orb Protocol (IIOP)
• Distributed Computing Environment (DCE)
• Distributed Program Link (DPL)
• External CICS Interface (EXCI)
• External Call Interface (ECI)
• Function shipping
• Asynchronous Transaction Processing
• Distributed Transaction Processing and
• Transaction Routing.

Typically, CICS provides a transport-specific listener (a long-running task) that starts another task (a facilitator such as an alias or a mirror), to process the incoming request. The facilitator uses CICS services to access the application.

Mainframe operating system z/OS is a share-everything runtime environment that provides for resource sharing through its heritage of virtualization technology. z/OS gets work done by dividing it into pieces and giving portions of the job to various system components and subsystems that function interdependently.

The workload management (WLM) component of z/OS controls system resources, while the recovery termination manager (RTM) handles system recovery.

At any point in time, one component or another gets control of the processor – makes its contribution, and then passes control along to a user program or another component. The control typically gets passed when a job has to wait for information to be read in from, or written out to, a device such as a tape drive or printer.

As with memory for a personal computer, mainframe central storage is tightly coupled with the processor itself, whereas mainframe auxiliary storage is located on (comparatively) slower, and cheaper external disk and tape drives.

Typical z/OS middleware (between the operating system and an end user or end-user applications) includes:

• Database systems
• Web servers
• Message queueing and brokering functions
• Transaction managers
• Java virtual machines
• Portal services
• XML processing functions

System Address Spaces and Master Scheduler

Many z/OS system functions run in their own address spaces. The master scheduler subsystem runs in the address space called *MASTER* and is used to establish communication between z/OS and its own address spaces. Master initialization routines initialize system services, such as the system log and communication task, and start the master scheduler address space.

Batch processing is the most fundamental function of z/OS. Many batch jobs are run in parallel and Job control language (JCL) is used to control the operation of each job.

z/OS requires the use of various subsystems, such as a primary job entry subsystem or JES. An address space is created for every batch job that runs on z/OS. Batch job address spaces are started by JES.

Multiple initiators (each in an address space) permit the parallel execution of batch jobs. Correct use of JCL parameters (especially the DISP parameter in DD statements) allows parallel, asynchronous execution of jobs that may need access to the same data sets (a technique to use batch parallelism and improve availability)

There are address spaces for middleware products such as DB2, CICS, and IMS (referred to as “secondary sub-systems). Typically an automation package starts all tasks in a controlled sequence. Then other subsystems are started. Subsystems are defined in a special file of system settings called a parameter library or PARMLIB.

Workload Management (WLM)

WLM manages the processing of workloads in the system according to the business goals, such as response time. WLM also manages the use of system resources, such as processors and storage, to accomplish these goals.

WLM has three objectives:

1. To achieve the business goals that are defined by the installation, by automatically assigning sysplex resources to workloads based on their importance and goals (goal achievement).

2. To achieve optimal use of the system resources from the system point of view (throughput).

3. To achieve optimal use of system resources from the point of view of the individual address space (response and turnaround time).

Goal achievement is the first and most important task of WLM. Optimizing throughput and minimizing turnaround times – which come after that – are essentially contradictory objectives.

Optimizing throughput means keeping resources busy. Optimizing response and turnaround time, however, requires resources to be available when they are needed. Achieving the goal of an important address space might result in worsening the turnaround time of a less important address space. Thus, WLM must make decisions that represent trade-offs between conflicting objectives.

WLM is particularly well-suited to a sysplex environment. It keeps track of system utilization and workload goal achievement across all the systems in the Parallel Sysplex and data sharing environments.

A mainframe installation can influence almost all decisions made by WLM by establishing a set of policies that allow an installation to closely link system performance to its business needs. Workloads are assigned goals (for example, a target average response time) and an importance (that is, how important it is to the business that a workload meet its goals).

I/O and data management

The input/output architecture is a major strength of the mainframe. It uses a special processor to schedule and prioritize I/O: the System Assist Processor (SAP). This processor is dedicated to drive the mainframe’s channel subsystem, up to 100,000 I/O operations per second and beyond. The channel subsystem can provide over 1000 high-speed buses, one per single server.

Data management activities can be done either manually or through the use of automated processes. When data management is automated, the system uses a policy or set of rules known as Automatic Class Selection (ACS^TM) to determine object placement, manage object backup, movement, space, and security. ACS applies to all data set types including database and Unix file structures.

Storage management policies reduce the need for users to make many detailed decisions that are not related to their business objectives.

Today’s z/OS provides a disk device geometry called Extended Address Volume (EAV) that enables support for over 223 gigabytes (262,668 cylinders) per disk volume in its initial offering. This helps many larger customers having the 4-digit device number limitation to begin consolidation of disk farms.

Intelligent Resource Director (IRD)

Intelligent Resource Director can be viewed as Stage 2 of Parallel Sysplex. IRD gives the ability to move the resource to where the workload is. z/OS with WLM provides benefits from the ability to drive a processor at 100% while still providing acceptable response times for critical applications.

IRD is not a product or component, but consists of three separate but mutually supportive functions.

1. WLM LPAR CPU Management – provides a means to modify an LPAR weight to a higher value in order to move logical CPUs to that LPAR that is missing its service level goal.

2. Dynamic Channel-path Management (DCM) – designed to dynamically adjust the channel configuration in response to shifting workload patterns.

3. Channel Subsystem I/O Priority Queueing (CSS IOPQ) – z/OS uses this function to dynamically manage the channel subsystem priority of I/O operations for given workloads based on the performance goals

Predictive Failure Analysis and Health Checker for z/OS

Predictive Failure Analysis (PFA) is designed to predict whether a soft failure (abnormal yet allowable behaviors that can slowly lead to the degradation of the operating system) will occur sometime in the future and to identify the cause while keeping the base operating system components stateless. PFA is intended to detect abnormal behavior early enough to allow you to correct the problem before it affects your business.

PFA uses remote checks from IBM Health Checker for z/OS to collect data the installation. The objective of IBM Health Checker for z/OS is to identify potential problems before they impact z/OS’ availability or, in worst cases, cause outages.

Next, PFA uses machine learning to analyze this historical data to identify abnormal behavior. It issues an exception message when a system trend might cause a problem – thereby improving availability by going beyond failure detection to predict problems before they occur. To help customers correct the problem, it identifies a list of potential issues.

Mainframe has three major clustering techniques: Basic Shared DASD, CTC rings, and Parallel Sysplex. Parallel Sysplex – is analogous in concept to a UNIX cluster – allow the customer to operate multiple copies of the operating system as a single system. This allows systems to be added or removed as needed, while applications continue to run.

Basic Shared DASD

As the name suggests, this configuration allows multiple z/OS images to share the direct access storage devices (DASD). This is a preliminary form of sharing and has limited capabilities.

CTC Rings

The channel-to-channel (CTC) function simulates an input/output (I/O) device that can be used by one system control program (SCP) to communicate with another SCP. It provides the data path and synchronization for data transfer. When the CTC option is used to connect two channels that are associated with different systems, a loosely coupled multiprocessing system is established.

To a large extent, batch jobs and interactive users can run on any system in this configuration because all disk data sets can be accessed from any z/OS image. Jobs (and interactive users) can be assigned to whichever system is most lightly loaded at the time.

Parallel Sysplex

A Systems Complex, commonly called a sysplex, is one or more (up to 32) systems joined into a cooperative single unit using specialized hardware and software. A sysplex generally provides for resource sharing between communicating systems (tape, consoles, catalogues, and so forth).

Sysplexes are often isolated to a single system, but Parallel Sysplex technology allows multiple mainframes to act as one. It is a clustering technology that can provide near-continuous availability.

A Parallel Sysplex is a symmetric sysplex using multisystem data-sharing technology. This is the mainframe’s clustering technology. It allows direct, concurrent read/write access to shared data from all processing servers in the configuration without impacting performance or data integrity.

A Parallel Sysplex allows the linking up to 32 servers with near linear scalability to create a powerful commercial processing clustered system. Every server in a Parallel Sysplex cluster can be configured to share access to data resources, and a “cloned” instance of an application might run on every server.

Another unique advantage of using Parallel Sysplex technology is the ability to perform hardware and software maintenance and installation in a non-disruptive manner.

Recently one of our Unix experts – focusing on IT optimization – came to me to get an understanding of Mainframe and like most IT people who have not worked in Mainframe had quite a few obsolete notions. So I created a set of documents on Mainframe overview, on zOS (operating system) and CICS (the transaction processor) and sharing the same here.

Mainframe is a self-contained processing center, powerful enough to process the largest and most diverse workloads in one secure “footprint.” At the same time, Mainframe is also just as effective when implemented as the primary server in a corporation’s distributed server farm. There are different classes of mainframe to meet diverse needs of customers – Business Class and Enterprise Class.

Security, scalability, and reliability are the key criterions that differentiate the mainframe. Businesses today rely on the mainframe to:

• Perform large-scale transaction processing (thousands of transactions per second)
• Support thousands of users and application programs concurrently accessing numerous resources
• Manage terabytes of information in databases
• Handle large-bandwidth communication

Typically, the mainframe shares space with many other hardware devices: external storage devices, hardware network routers, channel controllers, and automated tape library “robots,” etc. Unlike in the past, now, the mainframe is physically no larger than many of these devices and generally does not stand out from the crowd of peripheral devices (earlier Mainframe used to have rooms for themselves but now they are just part of a data center).

Mainframe interfaces today look much the same as those for personal computers or UNIX systems. A business application is accessed through a Web browser, with the mainframe in the background.

It is now possible to run a mainframe operating system on a PC that emulates a mainframe. Such emulators are useful for developing and testing business applications before moving them to a mainframe production system (can be useful in cost savings).

Most mainframe workloads fall into one of two categories: batch processing or online transaction processing, which includes Web-based applications. Today’s mainframe can run standard batch processing such as COBOL as well as batch UNIX and batch Java programs.

A mainframe can be the central data repository, or hub, in a corporation’s data processing center. For example, centralizing the data in a single mainframe repository can save customers from having to manage updates to more than one copy of their business data, which increases the likelihood that the data is current.

Mainframe – Hardware

Mainframe hardware consists of processors and a multitude of peripheral devices such as disk drives (called direct access storage devices or DASD), magnetic tape drives, and various types of user consoles;

 o The term Box may refer to the entire machine or model; it is an expression used due to its shape. Mainframe systems today are much smaller than earlier systems-about the size of a large refrigerator. The mainframe’s power consumption today is 0.91 watts per MIPS and is expected to decrease with future models.

o The abbreviation CEC, pronounced keck, is for the Central Electronic Complex that houses the central processing units (CPUs).

o Central processor complex or CPC refers to the centralized processing hub that contains the processors, memory, control circuits, and interfaces for channels.

o All the processors (S/390 or z/Architecture) present in the CPU are referred to as processing units (PUs). The PUs are characterized as CPs (for normal work), Integrated Facility for Linux (IFL), Integrated Coupling Facility (ICF) for Parallel Sysplex configurations and so forth.

o A channel provides an independent data and control path between I/O devices and memory. Today, the largest mainframe can have over 1000 channels. A channel can be considered a high-speed data bus. Todays mainframe use ESCON (Enterprise Systems CONnection) and FICON (FIber CONnection) channels.

o Channels connect to control units. A control unit contains logic to work with a particular type of I/O device – printers, tape drives etc. Today’s channel paths are dynamically attached to control units as the workload demands – providing a form of virtualizing access to devices.

o Control units connect to devices, such as disk drives, tape drives, communication interfaces, and so forth.

o Sharing of I/O devices is common in all mainframe installations. A technique used to access a single disk drive by multiple systems is called multiple allegiance. Multiple paths to a device allows for effective disk sharing (across multiple servers) which in turn can provide improved performance and availability.

The IBM mainframe can be partitioned into separate logical computing systems. System resources (memory, processors, I/O channels) can be divided or shared among many such independent logical partitions (LPARs).

For many years there was a limit of 15 LPARs in a mainframe; today’s machines can be configured with up to 60 logical partitions. Practical limitations of memory size, I/O availability, and available processing power usually limit the number of LPARs to less than these maximums. Logical partitions are, in practice, equivalent to separate mainframes. Right sizing is the key when it comes to partitions. It can be used to control usage of resources, improved security, availability etc.

z/OS

z/OS – widely used mainframe operating system – system is a share-everything runtime environment that provides for resource sharing through its heritage of virtualization technology. z/OS gets work done by dividing it into pieces and giving portions of the job to various system components and subsystems that function interdependently.

A z/OS system usually contains additional, priced products that are needed to create a practical working system. IBM refers to its own add-on products as IBM licensed programs – and comes at a cost. Also independent software vendors (ISVs) offer a large number of products with varying but similar functionality, such as security managers and database managers. The typical products include:

• Security System (Resource Access Control Facility – RACF from IBM)
• Compilers (z/OS includes an assembler and a C compiler. Other compilers – COBOL, PL/1 etc are separate products).
• Relational Database (DB2 from IBM)
• Transaction processing facility (CICS, IMS, WAS from IBM)
• Sort (DFSORT from IBM)
• Utility Programs

Even the System Display and Search Facility (SDSF) program that people use extensively to view output from batch jobs is a licensed program (looking at these licensed programs to see if they are really necessary – and also that there is no other ISV product with similar functionality available in the installation provides another opportunity for cost saving).

Besides z/OS, four other operating systems dominate mainframe usage: z/VM, z/VSE, Linux for zSeries, and z/TPF. The use of z/OS, z/VM, and Linux on the same mainframe is common.

It is also important to note that if there are multiple versions of the same software in an installation, the license costs are actually multiplied.

Mainframe provides Specialty engines -zAAP as a specialized Java execution environment, IFL for Linux – which enables off-loading specific work to separate processors.  As attractive prices are offered for such processors, the overall total cost of ownership reduces by appropriate use of these processors. This also enables the general CPUs to continue processing standard workload increasing the overall ability to complete more batch jobs or transactions.

Consolidation of Mainframe

Data center consolidation initiatives have resulted in several smaller mainframes being replaced with fewer but larger systems.

Software license costs for mainframes have become a dominant factor in the growth and direction of the mainframe industry as mainframe software (from many vendors) can be expensive, often costing more than the mainframe hardware.  Though Software license costs are often linked to the power of the system, yet the pricing curves favor a small number of large machines – replacing multiple software licenses for smaller machines with one or two licenses for larger machines is cost effective (While consolidating mainframes, the licenses for software (with 3^rd party vendors) may have to be re-negotiated for cost savings ).

The relative processing power needed to run a traditional mainframe application (a batch job written in COBOL, for example) is far less than the power needed for a new application (with a GUI interface, written in C and Java). New powerful mainframes might need only 1% of their power to run an older application, but the application vendor often sets a price based on the total power of the machine, even for older applications.

As an aid to consolidation, the mainframe offers software virtualization, through z/VM. z/VM’s extreme virtualization capabilities and Linux on Mainframes make it possible to virtualize thousands of distributed servers on a single server. Consolidating distributed servers to mainframe can directly translate into significant monetary savings (IBM’s has conducted a very large consolidation project named Project Big Green to consolidate approximately 3,900 distributed servers into roughly 30 mainframes, using z/VM and Linux on System z. It achieved reductions of over 80% in the use of space and energy.).

In CICS there are three ways of transferring control to a subprogram:

1. Using EXEC CICS LINK Command

2. Using Static COBOL call

3. Using Dynamic COBOL call

The format of the LINK command in the calling program is:

EXEC CICS LINK PROGRAM(‘subpgname’)  or  EXEC CICS LINK PROGRAM(name)

Link command is always dynamic, irrespective of whether the subprogram name is explicitly given or as a variable with the subprogram name as the value. The subprogram that is invoked using LINK can either be local or remote and must be defined using RDO (Resource Definition Online).

Static and dynamic COBOL call is similar to that of normal COBOL programs. The dynamically called subprogram must be defined in the CSD (CICS system definition dataset).

If the called subprogram has CICS commands, then it has be translated by CICS. The CICS translator inserts DFHEIBLK and DFHCOMMAREA into the LINKAGE SECTION and also adds them as the first two parameters in the PROCEDURE DIVISION. In such cases the calling program should use the following syntax:

CALL ‘PROG’  USING DFHEIBLK  DFHCOMMAREA  PARM1  PARM2           (or)

CALL PROGID USING DFHEIBLK  DFHCOMMAREA  PARM1  PARM2

The CICS translator forces the compiler options of: RES, RENT, NODYNAM, and LIB which cannot be overridden.

In case of LINK, each time the subprogram is linked to, a new initialized copy of its working storage is provided and the run unit is reinitialized which can cause performance degradation in some cases. In case of dynamic call, the new initialized copy of the working storage is provided on the first entry to the called subprogram. For subsequent calls the same working storage is provided in its last-used state which can result in better performance than the corresponding LINK

The following are the factors to be considered with respect to the application programs:

• Presence of CICS statements in the subprogram would necessitate the use of CICS translator. CICS translator in turn will affect how the program is called (DFHEIBLK & DFHCOMMAREA).
• In case of COBOL call, the subprogram should return control to the calling program either by using GOBACK or EXIT-PROGRAM. In case of LINK, the subprogram should return control to the calling program either by using GOBACK or EXEC CICS RETURN.
• The error condition raised in case of failure in calling the subprogram is different. If the subprogram is unknown/unavailable then LINK fails with PGMIDERR condition. In case of dynamic call, if the subprogram cannot be loaded or not available, then it results in COBOL abend.
• It is understood that in ABACUS, the online programs use dynamic linking and then do a CANCEL after invoking the subprogram. Hence this approach may not have give any performance benefit over the LINK option. Also it is to be noted, to avoid CANCEL (and therefore the re-initialization) the programs should have been coded such that it would not have adverse effect.

More detailed information with sample code refer: http://cicswiki.org/cicswiki1/index.php?title=CALL_vs._LINK#Advantages_of_using_a_LINK.

Using linking, we can have one program call another program that is not contained in the source code of the calling program. Depending on the linkage options,  the object module of the calling program is linked with the object module of the called program “before or during” execution – referred to as static or dynamic linking.

Appropriately choosing the linkage as static / dynamic depending on the application requirement (and not a single technique for the entire application) can result in benefits related to performance, memory usage and maintenance.

 Static CALL

 In the static CALL statement, the COBOL program and all called programs are part of the same load module (self-contained). When control is transferred, the called program already resides in storage, and a branch to it takes place. Subsequent executions of the CALL statement make the called program available in its last-used state, unless the called program has the INITIAL attribute.

A static call occurs:

1. When the CALL literal statement is used in a program that is compiled using the NODYNAM and NODLL compiler options.. With these options, all calls of the CALL literal format are handled as static calls.

Because a statically called program is link-edited into the same load module as the calling program, a static call is faster than a dynamic call (though the load modules are larger).

The key disadvantage is that to change the behavior of the called program, relinking is necessary. As statically called programs cannot be deleted (using CANCEL), static calls might take more main storage. Also if more than one calling program accesses the called program, duplicate copies of the programs have to be loaded into memory.

Dynamic CALL

In the dynamic CALL statement, the called COBOL subprogram is not link-edited with the main program, but is instead link-edited into a separate load module, and, at run time, is loaded only when it is required (that is, when called).

When a dynamic CALL statement calls a subprogram that is not resident in storage, the subprogram is loaded from secondary storage into the region or partition containing the main program and a branch to the subprogram is performed. The first dynamic call to a subprogram within a run unit obtains a fresh copy of the subprogram. Subsequent calls to the same subprogram (by either the original caller or any other subprogram within the same run unit) result in a branch to the same copy of the subprogram in its last-used state, provided the subprogram does not possess the INITIAL attribute.

When a CANCEL statement is issued for a subprogram, the storage occupied by the subprogram is freed, and a subsequent call to the subprogram functions as though it were the first call. A subprogram can also be cancelled from a program other than the original caller.

A Dynamic Call occurs:

1. When the CALL literal statement is used in a program that is compiled using the DYNAM and the NODLL compiler options (or)

2. when the CALL identifier statement is used in a program. CALL identifier is always dynamic, even if the NODYNAM compiler option is used.

The program name in the PROGRAM-ID paragraph must be identical to the corresponding load module name or load module alias of the load module that contains it.

Typically, dynamic CALL statement is used in the following circumstances:

• For ease of maintenance. Applications do not have to be link-edited again when dynamically called subprograms are changed.
• The subprograms called are used infrequently or are very large. If the subprograms are called on only a few conditions, dynamic calls can bring in the subprogram only when needed. If the subprograms are very large or there are many of them, using static calls might require too much main storage. Less total storage might be required to call and cancel one, then call and cancel another, than to statically call both.

The key points to be noted are:

• CALL identifier is always dynamic
• With NODYNAM compiler option, all calls of the CALL literal format are handled as static calls.
• When a static CALL statement and a dynamic CALL statement to the same subprogram are issued within one program, a second copy of the subprogram is loaded into storage. 

IBM mainframe specialty processors – zAAP and zIIP – cost significantly less compared to the General purpose processors and these “do not count” in software pricing calculations and ISV licensing costs.  Third party tools are needed to make best use of this – as the system has to be tuned to use the right processors for the right load.

One such tool Neon zPrime has resulted in law-suits and has brought a whole stream of questions on the future of these specialty processors. More details on the lawsuits between Neon and IBM can be found at http://itknowledgeexchange.techtarget.com/mainframe-blog/neon-ibm-wrangle-over-trial-timeline-for-zprime-lawsuit/.

zPrime “facilitates” (and doesn’t guarantee) the movement from a general purpose processor (CP) to a specialty processor (zIIP or zAAP) potentially saving millions of dollars in software and hardware costs. The idea is to fully leverage the capacity of the specialty processors with significant loads moved away from the CPs.

What has made the issue interesting is that Zeon says that zPrime is a proprietary software solution governed by trade secrets and the details will not be publicly disclosed.  Neon also claims that zPrime uses publicly documented exits and doesn’t modify IBM licensed internal code, zOS or other IBM software. No hooks to z/OS dispatcher, SMF, RLM or WLM either.

Though how zPrime works is not disclosed by Neon (and looks like not yet figured out by IBM – as the issue is still left open), some of the following points hint at SRB being the key:

• In MVS, the dispatchable units of work can be represented as Task Control Block (TCB) or Service Request Blocks (SRBs). zIIPs can only run applications that have been coded in SRB mode.
• zPrime product author Tony Lubrana has pointed out that it requires a lot of expertise and special authorization to run in SRB mode.
• One of the issues highlighted by Neon is that in z/OS v1.9 and below, TPUT SRB generated is not eligible for zIIP or zAAP processing.

Irrespective of where zPrime uses SRB or not, SRB is an interesting area with a lot of potential which has been regularly exploited by IBM products. For example CICS features like VTAM high Performance option, VSAM sub-tasking was achieved by executing parts of the work in parallel under an MVS Service Request Block concurrent with the CICS TCB.

Service Request Blocks (SRBs) routines are meant for providing specialized services.  SRBs can be created only by units of work running in the supervisor state and key 0.

Scheduling a SRB is nothing but initiating the process. A program can schedule a SRM process using the SCHEDULE or IEAMSCHD macro.  A program can initiate a SRB as a process in the same or different address space.  An SRB in turn can schedule another SRB.  An SRB routine can exit from its processing and to pass control to a task with minimal system overhead.  The environment in which SRB routine runs and its ability to access address spaces and data spaces can be controlled by specifying appropriate parameters.

Being asynchronous in nature, these routines run independent of the scheduling program. Though SRBs advantage arises from its asynchronous nature, its processing can be synchronized with the scheduling program using WAIT / POST / SUSPEND / RESUME macros.

SRBs should not be a long-running program and usually represent very short pieces of work that complete much more quickly than TCB’s. Inline with this expectation, the system enforces the following restrictions on SRB:

• An SRB routine is generally not preempted by I/O interruptions once it is dispatched.
• To prevent them from going into a wait for any avoidable reason, system restricts SRBs from issuing SVCs (except abends). This implies that they cannot issue some of the system macros like ENQs and data management functions like opening datasets.
• An SRB cannot “own” storage areas. SRB routines can obtain, reference, use, and free storage areas, but the areas must be owned by a TCB. Any data that the requesting task and the service share must be placed in common storage.
• The Transfer Control (TCTL) macro requires that the SRB requesting the TCTL must not hold any locks and must be in primary ASC mode, where the home and primary address space is the same.

Detailed informatin on SRB available at http://www.cbttape.org/xephon/xephonm/mvs0401.pdf.

SRBs have been least exploited by organizations and understanding SRB better and trying to put it appropriate use can provide major benefits.