As Martha Stewart would say, getting information into the hands of business users is a “good thing.” Every enterprise needs to provide accurate, timely information for business groups to fuel the operational decisions they make each day.

So does it matter how they get this information? Business users may wonder what difference it makes as long as they get the information they need every day to keep their business selling products and services, working with suppliers and partners, and competing in the marketplace.

Well, it does matter how business users get information. It matters because, quite often, they get it from data shadow systems, which are groups of spreadsheets and local, customized databases – often Microsoft Access and statistical databases – created by business groups to gather data for their users. While these systems provide exactly the information that business users are asking for, they are rarely part of an enterprise’s official data warehouse corporate performance management strategy. Outside the purview of the IT group, they often spawn data silos with the usual problems of inconsistency and quality.

How to Spot a Data Shadow System

Picture this typical scenario: the finance group is cranking out reports to examine business performance. They’re looking at revenue and expenses versus budget and forecast. They’re figuring out what is selling, who is selling it, how much it costs to produce and deliver, and how much profit they’re making. And they are doing all of this in spreadsheets – dozens, maybe hundreds of spreadsheets.

And that’s not all. They’re using dozens and dozens of Microsoft Access databases to extract data from enterprise resource planning (ERP), enterprise applications and even the data warehouse to transform that data for use in their spreadsheets. Despite all the powerful business intelligence (BI) tools available, your company is using personal databases and spreadsheets for data integration!

Does this sound familiar? Not only is the finance group doing it, but marketing, sales and other organizations are also using Microsoft Access and Microsoft Excel to gather data from across your enterprise, transform it, report on it and analyze it. It is the same story at Fortune 100 companies and small to midsized firms. All the business users are doing it, including yours.

Meeting Business Needs

Business groups build data shadow systems to answer the business questions that the enterprise applications, data warehouse or reports fail to answer for them. They’re filling a gap in the services they receive from their IT departments. Users may not want to get the information this way, but they don’t see any alternative.

Data shadow systems satisfy the business groups’ need for relevant business information – exactly what they believed the elegantly designed, vendor-built enterprise applications and IT BI systems would do for them. As data shadow systems evolve over time, they encompass more and more information, and increasing numbers of business users come to depend on them.

The Dark Side of Data Shadow Systems

Data shadow systems give business groups what they want, and everyone is happy, right? Not quite. The fact is that business users do not want to spend so much time creating these systems. Nor should they. They should be spending their time gaining a better understanding of their business, not wrestling with technology.

Because dealing with technology is not what business users do best, they cobble data shadow systems together without an overarching design. Each addition gets more difficult to implement and more costly to maintain. And when data management principles and disciplines aren’t followed, data consistency and integrity suffer. Yes, data shadow systems often fulfill their business’s need, but they do so in a very costly manner that uses too many resources and sacrifices data quality.

Achieving a Balance

You’d be hard-pressed to find any company, large or small, that doesn’t already have at least one data shadow system. While they don’t offer an optimal solution from a technology standpoint, they have many business-oriented qualities that cannot be ignored. Keeping the needs of business users in mind, it is possible to replace or rework these shadow systems with solutions that dovetail with a company’s overall data warehousing architecture.

Replacement doesn’t need to be a huge effort, either. In fact, make sure it doesn’t turn into a multiyear IT project that implements the best-of-breed technology, standards and practices while you pay for the most expensive products in each software category. This overreliance on costly, silver bullet technology is the reason why we keep hearing about a 70 percent failure rate in data warehousing projects. It is not that data warehousing is a failed approach, but rather these mega IT projects generally don’t meet expectations, nor do they provide a reasonable return on their huge investment.

Rebuilding data shadow systems is the right thing to do to ensure consistent, quality information for running a business. How one approaches this task in a way that maximizes value and ROI is a story for another day.