Published in Darwin Magazine (CXO Media)

Maybe applications integration isn’t really what you need.

Maybe you need to find a way to integrate your data.

If you read IT trade publications or attend conferences you can’t help but hear about how enterprise application integration (EAI) software is going to solve all your problems. Wherever you look, there’s another vendor hyping EAI to integrate all your systems.

Sounds good, right? Not so fast. EAI is not for everyone. Historically it has only made sense for very large companies with equally large budgets. A large financial services firm saddled with a massive application portfolio created through industry consolidation might be the right candidate for an EAI solution. But a smaller retailer or healthcare company might balk at a multi-million dollar price tag. The objective of EAI — to tie together disparate systems — is well-intentioned. It’s the way it goes about achieving that objective that doesn’t work.

Some of the wisest words on EAI are in Barron’s Plugged-In column by Mike Veverka, who writes about enterprise technology from a business perspective, providing valuable insights for IT managers concerned with ROI. Although he initially advocated EAI, Veverka wrote in his May 26, 2003, column that “this geeky ‘middleware’ software could have been a contender, but will never wear the belt.”

EAI is Dead

So how did EAI go from silver bullet to “geeky middleware?” In a way, its initial success contributed to its downfall. Well-funded early adopters like Deutsche Bank, Cable & Wireless, and Royal Dutch/Shell Group were able to use EAI successfully to tie together disparate systems. These successes led industry analysts and vendors to extrapolate huge markets for these tools. Their rosy forecasts were reinforced by industry surveys stating that systems integration is many companies’ top priority.

But the reality is that EAI is designed for large enterprises with thousands of different sources of data. When there are so many different sources of data involved, the business return of integrating them with an expensive EAI solution may be justified. That justification may dissolve, however, when the enterprise compares the huge cost of EAI with all the other projects competing for the same budget.

Contributing to the problem is the fact that some of the EAI vendors have not performed as well as analysts had expected, and their products were not only expensive, but also hard to use. The vendors’ performance is highlighted by their recent financial results in which they’ve posted net losses in their most recent quarters and revenues down from year-ago levels. It’s indicative of the trends, not just the overall economy.

But EAI Is Not Dead Yet

Just because EAI isn’t the silver bullet doesn’t mean that customers don’t need software that integrates their disparate systems. As Veverka said in his August 18 Barron’s column, “The relative failure of EAI to become the integration solution for the masses — to be sure, it has been successfully installed in a minority of massive conglomerates with bottomless pockets — doesn’t signal that demand has gone by the wayside for great software that knits together disparate systems across an enterprise. To the contrary, time and again, chief technology officers’ surveys tend to rank systems integration as businesses’ top priority and often its highest budget allocation.”

It’s time for EAI’s next incarnation: as an enabling technology within other applications. This shift has already started and will accelerate. You’ll probably notice EAI vendors starting to form partnerships or be getting acquired as the marketplace transforms.

Within the context of infrastructure technology, EAI does have a huge market potential. ERP, database and “business intelligence” vendors will be adopting EAI technology, either through partnerships, acquisitions, or building their own capability into their applications. This functionality will initially be a value-added feature and later it will be a competitive necessity. However, this time, the burden and cost of systems integration and interoperability will rest with software application vendors, not with their customers.

Forget the Systems — Look to the Data

Many of us in IT are geeks at heart. We tend to use a new technology because we want to rather than because we need it to satisfy a business need with a profitable ROI. Compounding the problem, the industry and its customers love to get caught up in the hype of silver bullets that offer quick fixes to tough, time-consuming problems.

Integrating data from many disparate source systems is a time-consuming, difficult task not because of the technical hurdles, but because of data management. How do you define, cleanse, consolidate and transform data from disparate systems so that it is consistent and correct? Not with EAI software. Welcome to another acronym: ETL (for “extract, transform, load”), a programming tool that helps you access and manipulate source data, then load it into a data warehouse. An ETL tool is what you want for integrating data.

No doubt about it, there is a legitimate need to integrate disparate systems. There is a much greater need by most enterprises, however, to integrate data to be able to measure and manage their business. Governance and financial transparency — a key concern of many corporate offices thanks to Sarbanes-Oxley — relies on data integration not systems integration. Companies, in order to drive revenue increases and improve profitability, need data integration for performance measurement and management.

Ignoring data integration often leads to expensive problems later on. A large telecommunications company found itself in this situation when it implemented EAI software to tie applications across its traditional phone, cell phone and Internet services businesses. It found that even with many applications interacting with each other via EAI software, it still couldn’t create a complete customer view. Customer names and addresses had been entered differently in various applications and people in the same household were not identified. In addition, the massive systems integration of applications failed to provide historical customer data, since these applications did not retain that data. This company learned that data integration with ETL tools would have been the most effective approach to providing a comprehensive view of each customer.

Unlike EIA, ETL software is a better and less expensive choice for data integration. EAI tools aren’t designed to transform large volumes of data efficiently. Rather, they’re used for passing information between applications in realtime. If you position EAI as a transportation utility, then it may be used rather than bulk extracts/loads, in the E and L of ETL. The ETL product interfaces with EAI software like another data source, providing the additional benefit of being able to operate in real-time. ETL vendors are already starting to offer EAI as part of their architecture, either through partnerships or acquisitions. Examples of this market trend are Informatica’s partnership with webMethods, and Ascential Software’s purchase of Mercator. Don’t be surprised if you see a more extensive use of this technology within ETL products.

Lessons Learned

If nothing else, remember these two things:

1. Don’t propose an EAI project, propose a systems integration project. There has to be a business need for that integration and an ROI better than what competing IT projects might offer.
2. If you need consistent, consolidated, correct data, you have to do the hard work of data integration. Sorry, but here’s no free lunch to achieving that goal — EAI is not the silver bullet many hope it is.