CIOs are expressing a concern about shadow IT in the “2014 State of the CIO Survey,” CIO Magazine, January 2014. Approximately half feel it is a constant problem and 40% believe it is on the increase.

Data shadow systems have been prevalent for years at most companies, so why are the CIOs noticing now? Likely because in the past, most data shadow systems were built using spreadsheets, so CIOs would ignore them or think that when IT offered better BI tools, then the business people would “come to their senses.”

Well, they didn’t.

The Shadows Loom Larger

The reason why the problem of shadow IT is more visible now is that the CIOs are seeing business groups going off and getting cloud-based or SaaS technologies without consulting them. This certainly concerns them because they realize that the business people are not likely to come back once they start using applications like salesforce.com.

There are two ways that data shadow systems are part of this problem.  One is that business power users are populating the cloud apps with data from the shadow system. The other is that they are pulling data from the cloud app or the EDW and putting it into data shadow systems.  Either way, data silos are increasing and expanding in scope.

CIOs have a right to be concerned, but I don’t think that is the end of their worries. Not only are business groups getting cloud apps, but they are also acquiring data discovery tools like Tableau and QlikView on their own. These are terrific tools, but when a business group gets them without IT involvement, another tier of data shadow systems arises.

Again, the power users pull data into the data discovery tools, often from their spreadsheet-based data shadow systems, creating a “new and improved” data shadow system anchored by these cool tools. CIO are never going to pry data discovery tools out of the business people’s hands.

And, do not forget those spreadsheets. I co-authored a TDWI research report “Strategies for Managing Spreadmarts” in 2008 with survey results showing the typical enterprise had almost 30 spreadmarts or data shadow systems that were actively being used by business people in their decision-making. It has been six years, but does anyone think that the number of data shadow systems has decreased? In fact, with Microsoft’s expanded BI offerings tied in with Excel, such as PowerPivot, the data shadow systems are now on steroids and are even easier to build.

Blame the Game not the Player

It is easy to blame the business group or power users for building the data shadow systems, but maybe it is not entirely their fault. No one builds these things unless they feel there is no other way to get that functionality, data or analysis. Business people would prefer to do their jobs, not IT’s.  I am not trying to justify building data shadow systems, but you cannot fix a problem unless you understand the cause. Telling business people they are bad (and sending them to their rooms/cubicles) does not address the problem or fix it. More comprehensive and pervasive BI is the cure.

Speaking of the blame game, let’s not exempt IT. IT has been so focused on implementing cloud-based applications and big data projects that they have sometimes forgotten to consider what happens when they finish those projects. Cloud-based data and big data are useful, but not nearly as useful as when combined with an enterprise’s other data. Sentiment analysis is nice to know, but more valuable when you can act on it to increase sales and profits. It’s even more valuable when combined with data in your EDW to determine lifetime value of a customer, assess sales campaign effectiveness and determine what products are driving growth. Too often, enterprises do not get beyond the stand-alone cloud application or big data trove, so they end up creating IT-generated silos.

And then business people create more data shadow systems to bridge the new silos. It is like an episode of The Twilight Zone.

Let’s admit there is enough blame to be shared. Let’s put aside the hype and let’s get working on creating a data architecture that can blend in new data without creating silos, and eliminates the need for business people to glue their data together with data shadow systems.

The topic of data shadow systems will be covered in more detail in my upcoming book, The BI Guidebook – From Data Integration to Analytics, Morgan Kaufmann, October 2014)