Reining in the proliferation of expensive, redundant data silos requires a methodical approach that includes establishing a data integration center of expertise.

In their quest for a single version of the truth and a 360-degree view of the enterprise, businesses have invested in projects to improve efficiency, effectiveness and productivity. They have built IT systems for enterprise resource planning (ERP), customer relationship management (CRM), supply chain management (SCM) and systems for budgeting, planning and forecasting. They attempted to integrate data with data warehouses, data marts and business intelligence (BI) projects. They relied on enterprise application integration (EAI) software to integrate all their applications, but it failed to integrate the data. They tried enterprise information integration (EII) software to create virtual data warehouses, but it too failed to provide the silver bullet for avoiding the hard work of data integration. Despite all the investments, data continues to be inconsistent across the enterprise.

Vendors offer business performance management (BPM) packages — bundles of BI tools, ETL (extract, transform and load) tools, analytic engines and prebuilt data warehouses — for analyzing business data. Many of these packages provide great business value, but they also have the potential to create even more data silos with numbers inconsistent with ERP, data warehouse (DW) and other applications.

Stop the Madness!

Let’s get at the root of the problem: you need to integrate data, not just create another version of it. Ever-increasing data silos are costing businesses through the expense of creating and maintaining them, the waste of overlapping and redundant projects, the time to review and reconcile conflicting numbers from the various data silos and the loss of business confidence in the numbers and their transparency.

Even more data silos are created when companies undertake projects such as data migrations to support application consolidation. These can be due to corporate mergers, ERP instance consolidations and master data consolidations (product and customer reference data). Although these are data integration projects, they are treated as standalone, one-off projects — further fragmenting already constrained resources and budgets.

Blame Man, Not the Machine

It would be easy to pin the blame on a lack of technology. However, the reality is that we have all the technology we need. People – and people-related issues such as how groups are organized, project ownership, budgets and cross-functional politics – are the roadblock to data integration.

How do people make such a mess of things? First, IT systems are built on a project-by-project basis, self-contained and tactical. Project leaders start with a “green field” approach, selecting the “right” software and designing the “right” database. They assume they will do better than previous teams because they have picked the better technology. Because each team builds its own product and technology knowledge, too much time is spent learning what their predecessors already did, rather than examining and understanding data integration.

Second, there is a historical divide with DW/BI systems on one side and transactional systems and applications (ERP, CRM, SCM, financial systems, etc.) on the other. They differ in their perspective, approach, projects, organizations and even terminology. The transactional systems and enterprise applications manipulate data with their own built-in reporting, DW and BI solutions. At the same time, the DW/BI teams are sourcing that same data to build their own enterprise solutions.

Third, various technologies, such as ETL, EAI and EII are viewed as separate, competing integration technologies. Each technology is used for a separate project, and each project team feels that it alone has the answer to the company’s integration problems.

Finally, due to business and IT organizational issues, business functions and departments act independently – either building customized DW/BI applications to meet their needs or creating data shadow systems for reporting and analysis.

Admit It, You’ve Got a Problem

You must admit you have a problem before you can solve it. Many companies are blind to their data integration problems. All they see is their investments in ERP, CRM, SCM and DW systems and the terabytes of data stored in them. Smug with the knowledge that they have all the data that the business needs, they’re not even aware of the data silos surrounding them.

At some point, they start noticing inconsistent data, which is a symptom of a data integration problem. However, because they don’t understand the cause, they focus on relieving the symptoms with quick-hit solutions. For example, they may try to consolidate BI tools. While this is a worthy goal unto itself, using a single BI tool will do nothing about the fact that the underlying data is coming from many disparate systems (ERP, CRM, SCM, data warehouses, data marts, etc.). The business is not getting different numbers because it is using different tools, but rather because each tool is associated with a different database where the data had been transformed differently than the other databases. The BI tool used is the tip of the iceberg; the data integration issue is what is below the waterline.

Once you recognize a data integration problem, you can think out-of-the-box on how to define, approach and attack it.

Ending the Data Silo Cycle

Ending the data silo cycle requires a fundamental change – viewing data integration holistically. From an architecture level, we need to view all the pieces: information, data, technology and product. More importantly, we need to view data integration from political and organizational perspectives. Most IT projects fail due to people problems, not because of technology.

Starting at the top, create a business and IT case for viewing data integration as a problem in and of itself. Justify your case by highlighting the business value of the improvements versus the cost of business as usual. Be sure to get the sponsorship of your CIO and CFO.

Enterprise data is a key investment, so treat data integration projects like investment portfolios. Set priorities, determine budgets and expect a return on investment. The portfolio should include any project that involves data – from its creation in your transaction systems all the way through reporting and analysis.

Create a data integration center of expertise (DICE) to establish the scope of data integration, define the architecture and vision, and help implement that vision. You need an organization with people assigned to these tasks in order for this vision to become reality. Left on their own, projects and people will continually create more data silos.

Next Month’s Roll of the DICE

Next month, this column will continue our discussion of the DICE by exploring the pros and cons of the organizational options for a DICE and the drawbacks you need to understand as you initiate a DICE. Many centers of excellence, competency centers and program management offices have failed – not because of technical issues, but rather because of organizational issues they did not see or confront.