(This is part of a series
of posts on business intelligence & data warehousing trends for 2008.)

How’s this for irony: a lot of integration projects create
more silos!

How can this happen? It’s because companies implement
integration technology and products on a project-focused
basis. The integration projects concentrate on specific applications and
processes, in other words they integrate
silos. Each tactical integration task is completed, and eventually the
company discovers that they need to integrate the silos they created with their
integration projects. 

The bad news is that many companies have already created
these integration silos. The business groups that invested in these integration
projects are now back to using spreadsheets, i.e. data
shadows systems, to reconcile the conflicting numbers coming out of these
silos.

The good news is that many IT and business groups are
educated consumers, i.e. they have seen the result of their myopic integration
efforts and understand they need to change their ways.

The Integration Sea Change

What should you do if you find yourself in this situation? Step
back from the tunnel vision of tactical data integration projects and design an
overall integration architecture. After you have this overall architectural blueprint
you can then design your individual projects, fitting them into this overall
blueprint. Just like you hand your house
blueprint to various contractors so that everything fits together, you need
your data integration blueprint to get things to work together too. This blueprint encompasses architectures for data,
technology, product and information (business data transformed.)

Enterprise
Data Management (EDM) is the blueprint (or holistic approach in consultant
lingo) that people should be striving for. Integration efforts, be it enabled
using EAI (Enterprise Application Integration), SOA (Service Oriented
Architecture), EII (Enterprise Information Integration) or ETL (Extract
Transform and Load) technologies, all involve integrating data.

Data integration processes involve mapping one or more data
sources to a data target and transforming the data along the way. Regardless of
whether you are using messaging, services or batch-driven technologies you are
performing the same processes (just the transport is different.)

But rather than taking a common approach most integration
projects start with an entirely different set of technologies or products. However,
these products and technologies overlap and, more importantly, the data they
integrate overlaps. The result is silos and business people manually
reconciling the data they just spent a lot of money and effort integrating.

Fortunately, if your company is in this situation you are
not alone (misery loves company!) and great options have emerged in the market.
First, there are best practices to design your data integration
framework (DIF) or blueprint. Second, there are proven program and project approaches
that incrementally build an EDM and migrate from your silos.

Finally, some of the top ETL (extract, transform & load)
vendors have recognized the need for comprehensive approach and have
transformed their ETL products to data integration suites. These suites have
expanded beyond batch-oriented ETL to include EAI, EII and SOA. In addition, it
is becoming increasingly common for these suites to offer data profiling and
data quality functions. No longer are you forced to buy separate best-of-breed
products supporting niche technologies.  You
can find products that will support your DIF.

The market leaders, according to Gartner Research (Magic
Quadrant for Data Integration Tools, 2007), are Informatica (INFA)
and IBM (IBM).
Both companies have expanded their data integration capabilities through
acquisitions; most notably IBM’s Information
Server obtained a significant portion of its functionality from its acquisition
of Ascential Software.

Forrester Research, , concurs with the leaders (The
Forrester Wave™: Enterprise ETL, Q2 2007)  but feels that Oracle (ORCL)
(via its Sunopsis acquisition) and SAP (via
its BusinessObjects acquisition) are catching up. I’d also watch that smaller
competitors such as Pervasive Software (PSVW) and Sybase (SY) are busy putting
together their own suites.

Conclusions

Change is in the air! Companies need to shift from their
tactical, project-focused approach to integration to an enterprise perspective. Many in the industry are realizing that
they need to make their integration efforts provide comprehensive and
consistent business information, not create yet another silo to then reconcile
with your other silos. The will and ability (products) are in place. This trend
will gain more momentum this year providing case studies and knowledge to
expand beyond technology’s early adopters.

fyi: This post was originally titled “No Quarter: Data Integration Suites
enable EDM (Enterprise Data Management)”. I have heard that some of my readers
are not Led Zeppelin fans and a few have not even heard of them, making my
obscure references a bit confusing.