The Role of Data Integration in Providing Trusted Information

Data Integration Myth #2: Developing an information architecture is a waste of time, let’s just get on with building the data warehouse.
June 1, 2006
Data and Goliath
June 6, 2006
Show all

The Role of Data Integration in Providing Trusted Information

published in Information ManagementHow trustworthy is the data that you provide? Not very, if it has quality problems. Data quality is one of the biggest roadblocks to trusting corporate performance management (CPM) and business intelligence (BI) solutions. Picture this: a business group starts using a new CPM or BI solution only to discover that it cannot analyze the data to make business decisions or, worse, the data doesn’t match actual business results. Compliance with government regulations becomes impossible. Trust in the data is lost.

The problems grow because every CPM and BI project with a reputation for poor data quality begets another costly CPM or BI project to get it right. The proliferation of data silos costs the business time and money to determine what data to use and how to manipulate it. Meanwhile, the company gets more expensive IT projects and costly product licenses to solve the problem … this time for sure.

What happened? Data integration and data quality were too narrowly defined – and seen as products, not processes.

Data Integration is a Process

The narrow definition of data integration is that it is merely extract, transform and load (ETL) tools used to move data and transform it into a common format in a data warehouse. A more precise definition sees data integration as the processes that transform data into business information. What is often forgotten or shortchanged is that information must be usable by the business to measure performance and make business decisions.

Customer data integration (CDI) and master data management (MDM) are clear examples of efforts that confuse using tools with actually accomplishing data integration. Customer resource management and “360-degree view of the customer” initiatives were going to provide the single version of the truth. Likewise, enterprise resource planning (ERP) implementations and data warehouses were going to provide the single view of master data management (MDM). But people assumed that integrating those data subjects was simply implementing the right tools, rather than taking a new and more comprehensive approach to data integration.

Similarly, data quality efforts are hobbled when the focus is on the tools, not the process. The narrow focus on what data quality tools do, such as address matching, shortchanges the actual data quality.

It is more effective and cheaper to incorporate best practices in data integration right from the start, but for many of us that opportunity has passed. Do you need to redo everything and start from scratch? Many think so and give up, overwhelmed. But there are other ways to approach a data quality initiative.

More than Moving Data

Companies need a holistic approach to data integration that incorporates all the processes for creating usable business information from source system data. The result is comprehensive, consistent, relevant and timely information.

It’s not enough that the values in a CPM or BI system match those in the source systems. The purpose of source systems, such as transactional systems, is to record business events such as taking a customer order, requesting supplier materials and paying employees. Each of these systems is directed toward a certain constituency and organization.

The purpose of the CPM/BI system is to measure performance across an enterprise using its own metrics – metrics that may not apply to the source system transactions. When its data matches the source system, it may signify good data. However, good information is another story; it includes metrics and measures from the users examining business performance. Users need the data to be relevant, timely, complete and consistent across all source systems. A data quality tool is the first step to achieve this, but it’s not nearly comprehensive enough. The key is to include data quality processes through the entire process of transforming data into business information. Data profiling, data auditing and metadata integration also play roles in the data integration process.

Achieving data integration best practices may seem overwhelming, but companies can improve existing systems by adding portions of the expanded data integration process without ripping everything out and starting from scratch. Still, if there’s an opportunity to create a new system, doing it right the first time will improve the business value derived from your system. My next column will outline the steps you should take with existing systems to move you toward providing valuable business information that the business trusts and uses.

1 Comment

  1. On Shadows and Spreadsheets

    I read a couple of interesting articles this week that really helped to crystallize some thoughts I was having
    1. Slashdot had an article about the $10 Billion lost annually due to spreadsheet error and fraud.
    2. Rick Sherman wrote an intere…

Leave a Reply

Your email address will not be published. Required fields are marked *