CEP Key to Value from Big Data

Enterprise data can lose its value in an instant. But when you can access, analyze and act on data in real-time, you can also extract big value from enterprise big data. Complex Event Processing (CEP) is a fundamental component of real-time data analysis and value extraction.

Today, our parent company, Software AG announced the welcome acquisition of Apama, the market-leading platform for building CEP applications, which will enhance our collective ability to address our customers’ big data challenges.

With Apama, organizations can monitor fast-moving event streams, detect complex patterns, and take action – all in real-time. Apama will allow them to correlate and analyze business events across multiple data streams to make fast and accurate business decisions.

As we continue to meet the challenges of big data with speed and scale, we look forward to adding this capability to our mix. Along with the Apama team, we will continue to bring big data solutions to leading industries including financial services, e-commerce and telecom, to name a few. Stay tuned to the Big Data Blog for more on Apama and other exciting technology news in the coming months.

In the meantime, I invite you to share your big data challenges, interest in CEP or thoughts on other related topics.

Mainframe Offloading: Slash Costs up to 80%

After the incredibly positive response to our recent post on mainframe offload with in-memory data management, I thought it would be interesting to dive deeper into a particular aspect of the challenge: offloading existing jobs from your mainframe using Terracotta.

Many Terracotta customers oversee large estates of mainframe hardware. This hardware is difficult and expensive to maintain, and it locks future development efforts into decades-old technologies such as JCL and COBOL. Engineers trained on these technologies are scarce and expensive.

Perhaps most costly to the organization is the resulting lack of agility. New technologies do not integrate easily with mainframe systems, making it difficult to move as quickly as competitors who have more modern estates.

But what if you could reduce mainframe load—and costs—by up to 80%? What if you could use modern big data technologies like Hadoop to augment expensive-to-run JCL?

Terracotta’s parent company, Software AG, has decades of experience in the mainframe integration field. Software AG’s Entire X provides real-time bi-directional integration with your IBM z Series and the webMethods Integration Server. These integrations allow your business data to move transparently into Terracotta BigMemory Max, our in-memory data management platform that is fast, flexible, and runs on inexpensive commodity hardware.

One architecture for mainframe offload using Terracotta BigMemory Max

Once your data is in BigMemory, it’s available to any cutting-edge technology. Key among these is Hadoop, which can be connected to BigMemory using the BigMemory-Hadoop connector. With the connector, you can easily get data from BigMemory into Hadoop, providing scalable, enterprise-strength batch processing  entirely on commodity hardware.

Once you’ve set  your data free from the constraints of the mainframe, what about your jobs?  Terracotta partners with consultants experienced in migrating JCL to Hadoop. With numerous successful mainframe offload projects under our collective belt, we’ve learned how to gracefully migrate of  jobs to Hadoop infrastructure.

All in all, users of Terracotta software have moved over 5 petabytes of data into ultra-fast machine memory. To find out if we can help with your mainframe challenges, contact me (ben.lackey@terracottatech.com) or Terracotta’s sales team (sales@terracottatech.com).

Improve Core Banking with Faster Analytics and Back-End Processing

Can you improve core banking with faster analytics and back-end processing? We think so!

A recent report estimated that between 2009 and 2014, the total number of U.S. online banking households will increase from 54 million to 66 million. This growth will result in increased load on the core banking systems, making it a bigger challenge to support the required performance. Performance is already a big factor in the success of a core banking system with the rapid increase in adoption of internet banking and mobile banking channels. A quick and easy way to overcome performance challenges is offered by In-Memory Data Management (IMDM). IMDM not only can scale to manage terabytes of heterogeneous data in-memory but can also do that with guaranteed latency in the low milliseconds. IMDM can improve core banking in the following three areas:

1. Faster online transactions: The performance of the core banking functions, offered through the internet and mobile channels, can be improved by integrating with IMDM. IMDM can maintain frequently accessed data like customer data, product information, offer information, etc., in-memory to improve the performance of online transactions. Faster transaction processing enables offering more personalized services resulting in increased rate of attracting and retaining customers.

2. Faster analytics: IMDM is being adopted by banks to transition from a batch reporting environment to a real-time reporting environment. IMDM can hold large data sets in memory whether those are banking transactions for the day, historical transactional data or a combination of the two. All this data in-memory reduces the time it takes to generate reports,leads to faster identification of insights and, as a result, faster action.Faster analytics also leads to faster identification of fraud that could prevent billions of dollars in losses.

3. Faster back-end processing: The overall performance of core banking systems and other interfacing systems like loan processing, general ledger,etc. can be improved by using IMDM to hold key data entities like credit information, customer transaction history, loan process details, associatedrules, etc. If the bank is already using straight through processing, IMDM makes it even faster by maintaining key data assets in-memory.

If you want to use IMDM to improve your core banking processes, please leave a comment below or contact me.

5 Reasons BigMemory Should Earn a Stevie

Big Data challenges are forcing innovative changes to enterprise architecture. Survival is only possible among the fastest.

BigMemory 4.0 is a Stevie award finalist for Best New Product or Service because it enables real-time access to Big Data at big scale.

We know what the Stevies think, now we’d love to have your opinion. Based on your big data needs, what’s the most useful or interesting product feature of BigMemory? Tell us what you think by taking the poll on our Facebook page. Please feel free to add any features not already listed.

5 Reasons BigMemory Should Earn a Stevie

Big Data challenges are forcing innovative changes to enterprise architecture. Survival is only possible among the fastest.

BigMemory 4.0 is a Stevie award finalist for Best New Product or Service because it enables real-time access to Big Data at big scale.

We know what the Stevies think, now we’d love to have your opinion. Based on your big data needs, what’s the most useful or interesting product feature of BigMemory? Tell us what you think by taking the poll on our Facebook page. Please feel free to add any features not already listed.

Check back next week to find out the results of the poll.

5 Reasons BigMemory Should Earn a Stevie

Big Data challenges are forcing innovative changes to enterprise architecture. Survival is only possible among the fastest.

BigMemory 4.0 is a Stevie award finalist for Best New Product or Service because it enables real-time access to Big Data at big scale.

We know what the Stevies think, now we’d love to have your opinion. Based on your big data needs, what’s the most useful or interesting product feature of BigMemory? Tell us what you think by taking the poll on our Facebook page. Please feel free to add any features not already listed.

Check back next week to find out the results of the poll.

Minimize Financial Risk in Real-time Using In-Memory Data Management

Financial Services industry is one of the first adopters of big data solutions. The regulatory and risk environment for a financial services company makes it mandatory to continuously innovate and implement solutions that will help overcome the challenges of big data. The financial crisis in the last few years has made the situation tougher by unleashing a new set of compliance mandates resulting in heightened emphasis on transparency and risk analysis. This is leading to a strong need for real-time analysis of risk that may require a significant upgrade to the existing application setup or a large investment to deploy a new one.

In-Memory Data Management (IMDM) can help overcome this challenge by enabling real-time risk management quickly, without significant changes to existing applications and in a cost-effective fashion. A large number of organizations in the financial services space are already benefiting from it. Here are some example use cases that demonstrate how IMDM can manage risk in real-time:

1. Risk and Compliance Reporting: Use in-memory data to speed analytics and reporting for faster, more informed decisions on how to best deploy capital and mitigate risk. Leading investment banks are benefiting from IMDM by real-time sharing of reconciliation data across multiple applications for reporting & risk management.

2. Collateral Management: Reduce risk exposure and support a higher volume of collateralized loans with real-time access to the data required to re-price and recall assets. One of the world’s largest financial services organization has deployed IMDM to hold all their collateral assets. This has enabled real-time risk mitigation, global visibility across all entities and geographies and support for cross-product collateralization and rehypothecation.

3. Fraud Management: Process transactions in-memory using fraud detection rules, also held in-memory, to detect and eliminate fraud in real-time. Large credit card companies are already using IMDM for fraud management.

If you would like to learn more, please visit Terracotta’s financial services page, leave a comment below or contact me at gagan@terracotta.org

Minimize Financial Risk in Real-time Using In-Memory Data Management

Financial Services industry is one of the first adopters of big data solutions. The regulatory and risk environment for a financial services company makes it mandatory to continuously innovate and implement solutions that will help overcome the challenges of big data. The financial crisis in the last few years has made the situation tougher by unleashing a new set of compliance mandates resulting in heightened emphasis on transparency and risk analysis. This is leading to a strong need for real-time analysis of risk that may require a significant upgrade to the existing application setup or a large investment to deploy a new one.

In-Memory Data Management (IMDM) can help overcome this challenge by enabling real-time risk management quickly, without significant changes to existing applications and in a cost-effective fashion. A large number of organizations in the financial services space are already benefiting from it. Here are some example use cases that demonstrate how IMDM can manage risk in real-time:

1. Risk and Compliance Reporting: Use in-memory data to speed analytics and reporting for faster, more informed decisions on how to best deploy capital and mitigate risk. Leading investment banks are benefiting from IMDM by real-time sharing of reconciliation data across multiple applications for reporting & risk management.

2. Collateral Management: Reduce risk exposure and support a higher volume of collateralized loans with real-time access to the data required to re-price and recall assets. One of the world’s largest financial services organization has deployed IMDM to hold all their collateral assets. This has enabled real-time risk mitigation, global visibility across all entities and geographies and support for cross-product collateralization and rehypothecation.

3. Fraud Management: Process transactions in-memory using fraud detection rules, also held in-memory, to detect and eliminate fraud in real-time. Large credit card companies are already using IMDM for fraud management.

If you would like to learn more, please visit Terracotta’s financial services page, leave a comment below or contact me at gagan@terracotta.org

Boosting Hibernate Performance with In-Memory Data Management

One of the great benefits of persistence frameworks like Hibernate is that they allow architects and developers to mange data in ultra-fast machine memory, or RAM. By default, a first-level cache — at the Hibernate session level — is always enabled. A second-level cache, at the session-factory tier, is optional, but can result in huge performance gains at scale. Additionally, Hibernate allows for query-level caching.

Terracotta BigMemory (Ehcache) is the default query- and second-level cache for Hibernate, and it can keep terabytes of data in memory with as few as two changes to a configuration file. Understanding how BigMemory works with Hibernate makes designing your enterprise applications much easier, so in this post I’ll share tips and best practices for using Terracotta BigMemory as a query cache and a second-level Hibernate cache.

1. How do I get started?

Documentation on using BigMemory with Hibernate is here: http://ehcache.org/documentation/user-guide/hibernate

Enabling the second-level cache or query cache requires only a single line of config in your hibernate.cfg file:

<property name=”hibernate.cache.use_second_level_cache”>true</property><property name=” hibernate.cache.use_query_cache “>true</property>

I typically use the Ehcache Singleton factory:

<property name=”hibernate.cache.region.factory_class”>

net.sf.ehcache.hibernate.SingletonEhCacheRegionFactory</property>

2. How do I know my query is hitting the second-level cache?

The simplest and safest way is to set “show_sql” to “true” in your Hibernate property file. When you query the database, if the SQL query prints to the console, it is probably not using your second-level cache. In addition, you can use the Terracotta Monitoring Console (provided as part of BigMemory Enterprise kit) or any Hibernate profiler (http://www.hibernatingrhinos.com/products/hprof) and look at the hits and misses against your cache.

3. How do I specify custom cache regions?

By default, Hibernate always points to the default ehcache.xml and the default cache region. This implies that Hibernate manages the cache regions for you.

Let’s take an example. Say you have two Hibernate objects, Account and Customer. By default, the settings of the default cache will be applied to these objects. Hibernate will create a cache with the fully qualified domain path (e.g. com.company.domain.Account with be the name of the cache)

For more control, you can also specify custom cache regions, you can do this in two different ways:

1. Specify this as a cache region in your Hibernate cfg or using Hibernate annotations

e.g. @Cache(region = “Account”)

1. Specify the cache region in your Hibernate domain configuration

Note that if you use query cache, Hibernate creates two caches internally for its purpose:

a)    org.hibernate.cache.StandardQueryCache

b)    org.hibernate.cache.UpdateTimestampsCache

The StandardQueryCache has the query that is executed as part of the key itself. The updateTimeStampsCache is used to track the timestamps for updates to particular tables.

Note that in case you want to cluster your query cache, you will need to specify the above as 2 separate cache regions to your ehcache.xml and cluster them using the terracotta tag.

4. My application is not using Hibernate, so why do I get the error java.lang.NoClassDefFoundError: org/hibernate/cache/CacheKey ?

You probably have different applications loaded by the same classloader that uses Hibernate. Separate your CacheManager config/ ehcache.xml as follows:

a) All Hibernate-related objects that require Ehcache as second-level cache should be defined in in ehcache.xml.
b) Plain old Ehcache objects will be defined in ehcache-nonHibernate.xml. Use CacheManager(“ehcache-nonHibernate.xml”).getInstance() to get a reference to this CacheManager.

5. How do I evict specific cache regions when I execute my hql?

Hibernate allows you to specify a synchronize tag within the class. This lets you specify the table(s) you are updating, and it will only clear the cache for the specified table(s). If you do not specify any table(s), it will clear the cache for all tables.

Here is a link and an example on the Hibernate forums on how this is accomplished:

https://forum.hibernate.org/viewtopic.php?t=959949&view=previous&sid=6032f5caaa1dea9d05c75587e400228d

<class name=”Summary”>
<subselect>
select item.name, max(bid.amount), count(*)
from item
join bid on bid.item_id = item.id
group by item.name
</subselect>
<synchronize table=”item”/>
<synchronize table=”bid”/>
<id name=”name”/>

…
</class>

Shows item and bid are accessed but since it is native SQL, Hibernate has no idea what is tables/entities are being touched. Synchronize informs Hibernate so it can deal with possible flush initiation, caching, etc.,  depending on how summary is being used.

6. How do I use Hibernate criteria, and what is the Native SQL “gotcha”?

To take advantage of the second-level cache, you will need to use Hibernate criteria. To take advantage of the query cache, you can use HQL.

If you run a stored procedure or issue an executeUpdate or execute Native SQL, there are two side effects of which you should be aware:

1. The second-level cache will not be used

2. The second-level cache will be completely purged in certain circumstances

Whenever a Query.executeUpdate() is run, for example, Hibernate invalidates affected cache regions (those corresponding to affected database tables) to ensure that no stale data is cached. This should also happen whenever stored procedures are executed.

Furthermore, if you run Native SQL through Hibernate you entire second level cache will be purged.

7. What about object lifecycles and read/write modes?

Hibernate likes to control the entire lifecycle of the object, from inception to destruction.

In read/write mode, Hibernate considers itself the owner of data, and tries to provide high-consistency guarantees. Without getting too technical here, let’s just say that Hibernate takes charge of all lock management and transaction management in this mode. As a result, if you need to enable rejoin/non-stop, you can only do so under non-strict read/write mode.

8. How do I cluster and enable BigMemory when using Hibernate?

This is simple! It takes just two lines of config to cluster using Terracotta:

1. Specify the terracotta config url (where is Terracotta Server deployed?)
2. Specify which cache regions need to be clustered (use the <terracotta/> tag)

You can continue to use ARC with Hibernate http://ehcache.org/documentation/2.5/arc/index

9. How do I cluster cache regions across multiple Hibernate session factories?

You don’t need to do anything. Just cluster the cache region as above and you will be set. Having separate Hibernate session factories should not matter. For a cache region to be clustered, it should belong to the same cacheManager-cacheRegion combination.

10. Can I use writebehind with Hibernate?

While you cannot configure a cacheWriter to work with Hibernate (due to the transaction semantics identified above), you can configure this using Ehcache putWithWriter and Ehcache writebehind. Use Hibernate as part of your CacheWriter interface implementation to define your persistence strategy. More documentation is here: http://ehcache.org/documentation/apis/write-through-caching

I hope these tips contribute to making your experience with Hibernate and BigMemory easier and more fruitful. If you have additional questions, please post them to the comments.

Too Big to Scale? Why Financial Services can Benefit from In-Memory Data Management

Financial services organizations make money by managing a variety of tradeable assets. These assets rely on data from multiple sources to make the business of managing money run efficiently. With the recent growth in data volumes and data velocity, the wheels of this financial engine are slowing down due to pressure on existing applications to a) continue to meet the performance requirements, and b) process large data volumes in real-time for the business to succeed.

These challenges can be overcome by using in-memory data management (IMDM) and integrating it with existing applications. IMDM provides several benefits that are unmatched by any other technology:

1. Ability to scale linearly with predictable performance: Most technologies support small data sets with millisecond or microsecond performance times but the performance degrades as the size of the data set grows. IMDM offers predictable performance regardless of data volume as it has been architected to scale linearly. As the data sets grow from hundreds of gigabytes to hundreds of terabytes, the IMDM setup can scale by adding more memory without worrying about scalability or performance.

2. Ability to integrate with almost any application: It does not matter if the application under pressure is related to investment banking, fraud management, settlement, compliance or something else, IMDM can integrate with any application regardless of the function or the use case. This flexibility extends to IMDM’s ability to work with several different technologies. For example – slow transactions can be offloaded from a mainframe to IMDM to make the end-to-end process run faster, IMDM can be setup as a private cloud instance to allow multiple applications to integrate with it using basic URLs or IMDM can integrate with your application, running in the same process, to give a huge boost to your application performance.

3. Short deployment timeframe: IMDM is a great technological innovation and it achieves greater heights by making it easy to deploy. It only takes a few weeks to implement IMDM in your environment.  All you need is to clearly define the use cases that are causing pain and IMDM can be setup to resolve those pain points. Before deploying the IMDM solution, it is important to keep your strategic needs in mind, mostly from a business growth perspective, to ensure the setup can support your business for the next several years.

If you are interested in learning more about the power of IMDM either leave a comment below or contact me at gagan@terracotta.org.