Posted by: Marc Breissinger
Agility, Architecture, Data Abstraction, Data Federation, Data Virtualization, Enterprise Architecture
Organizations today understand that better access to information assets can improve their bottom-line.
But they struggle with the variety of enterprise, cloud and big data sources, and all their associated access mechanisms, syntax, security, etc. Further, few data sources are structured properly for business user or application consumption, let alone reuse. And often the data is incomplete or duplicated.
Data Abstraction Overcomes These Challenges
Data abstraction overcomes data source to data consumer incompatibility by transforming data from its native structure and syntax into reusable views and data services that are easy for application developers and business analysts to understand and consume.
Data Abstraction Technology Options
Some data abstraction approaches used today work better than others.
For example, some organizations build data abstraction by hand in Java or use business process management (BPM) tools. Unfortunately, these are often constrained by brittleness and inefficiencies. Further, such approaches are not effective for large data sets since they lack the robust federation and query optimization functions required to meet data consumers’ rigorous performance demands.
Data warehouse schemas can also provide data abstraction. Data modeling strategies for dimensions, hierarchies, facts and more are well documented. Also well understood is the high cost and lack of agility in the data warehousing approach. Further, data warehouse based schemas don’t include the so many new classes of data (big data, cloud data, external data services and more) that reside outside the data warehouse.
Data Virtualization is a Superior Solution for Data Abstraction
Data virtualization is an optimal way to implement data abstraction at enterprise scale. From an enterprise architecture point of view, data virtualization provides a semantic abstraction or data services layer supporting multiple consuming applications. This middle layer of reusable services decouples the underlying source data and consuming solution layers. This provides the flexibility required to deal with each layer in the most effective manner, as well as the agility to work quickly across layers as applications, schemas or underlying data sources change.
Data abstraction using data virtualization provides the following benefits:
- Simplify information access – Bridge business and IT terminology and technology so both can succeed.
- Common business view of the data – Gain agility, efficiency and reuse across applications via an enterprise information model or canonical model.
- More accurate data – Consistently apply data quality and validation rules across all data sources.
- More secure data – Consistently apply data security rules across all data sources and consumers via a unified security framework.
- End-to-end control – Use a data virtualization platform to consistently manage data access and delivery across multiple sources and consumers.
- Business and IT change insulation – Insulate consuming applications and users from changes in the source and vice versa. Business users and applications developers work with a more stable view of the data. IT can make ongoing changes and relocation of physical data sources without impacting information users.
Data Abstraction Reference Architecture
With hundreds of installations in a variety of industries, Composite has developed a Data Abstraction Reference Architecture that architects and analysts can use this as a guide when abstracting data using data virtualization. In my next blog entry I will drill down into its components.