Beyond ETL and Data Warehousing
BI/DW Index: Will tech lead us in the rebound?
February 17, 2009BI/DW Index: Hardware tarnishes the index
February 24, 2009
Data integration suffers from an image problem. It has become synonymous with extract, transform and load. Likewise, ETL has been regarded as a data warehousing technology. Both of these viewpoints fail to reflect current capabilities, and they greatly inhibit enterprises in their attempt to integrate data to provide the information their business needs. Because of this short-sightedness, companies have lost opportunities to harness information as a corporate asset. It increases the cost of integrating data, encourages the creation of data silos and forces businesspeople to spend an inordinate amount of time filling the information gaps themselves through data shadow systems or reconciling data.
This is the introduction to my article that was published in Information Management Direct (formerly DM Direct). To read the full article >>>
2 Comments
I agree with your assertion that cost has been a principle inhibitor to adopting the full breadth of data integration technologies. License costs have forced organizations to choose one approach as embodied by particular product offerings and then try to apply that to all use cases to justify the investment. There are changes afoot in this area however. Open source data integration offerings are enabling project teams to quickly derive benefits without huge investements and lenghty procurement processes. Lower overall TCO allows multiple technologies to be acquired and deployed leading to a richer solution portfolio at the organizations disposal. Companies like Talend, JasperSoft, Pentaho, EnterpriseDB, Red Hat/JBoss and others have compelling products to address data integration challenges. Organizations looking to build a broad DI toolset should educate themselves on the open source options available.
-Ken
Ken,
Thanks for the great feedback. I agree that license costs have been an inhibitor to greater use of data integration/ETL tools and resulted in hand-coding still being the “market leader” in ETL programming. I also agree that ETL open source software (OSS) presents an opportunity for companies to replace hand-coding ETL processes that in the long-run will offer a lower TCO, provide documentation and audit capabilities, and, quite frankly, create more consistent and higher quality ETL processes. BUT besides OSS there are other pockets of tools and vendors that provide an opportunity to achieve these goals without breaking an IT budget. “Bundled” ETL from Oracle and Microsoft is one pocket while another is companies such as expressor software, Pervasive Software and Sybase.
But the focus of the article was really about the “high end” of the data integration market were the DI leaders provide a suite of tools performing more than simply ETL, support large development teams and offer performance that most of the vendors above cannot match. In the past, companies with these high end needs bought ETL, EAI, EII (data virtualization) and SOA products/platforms separately. Nowadays there is an opportunity to leverage a single integration platform or product suite.
The data integration market is large enough to have a number of vendors across the spectrum of capabilities and costs.
Thanks, Rick