This decade, and especially this recession, has seen a tsunami in demand for the data needed to make sound business decisions. Yet businesses continue to fall behind when they don’t approach data integration as a business-wide effort that not only drives sales and profitability but also allows data to provide transparency, privacy and security.

As information needs have evolved and grown, so has the path of data integration. Some of the important data integration trends and business drivers are described here.

Businesses need more and more data

Businesses today are hungrier than ever for information. They depend on accurate, timely information to fuel efficient operations, growth and customer responsiveness. As the volume of data grows, so does the complexity of integrating it.

Some trends fueling the exponential growth of data:

Companies are generating more data internally. For example, the marketing group is collecting more detailed customer data from Web analytics and other customer touch points. Global companies have data from various countries to integrate, analyze and manage.

• There’s more external communication with partners and suppliers. As communication increases, so does the amount of data being passed back and forth. Inventory levels, ship dates, product descriptions – each company needs the latest information so it can share it internally and with its customers. Are we going to receive our shipment of widgets in time to meet our production goals? Is the partner’s product we’re selling on our website available in enough quantity to meet our holiday rush?
• There’s a movement from structured data to other, unstructured sources, such as spreadsheets and Web pages. Unstructured data can come from all over the enterprise. It’s easier to generate but harder to integrate. This data was often ignored in the past, but businesses now realize that it’s an invaluable source of company knowledge that needs to be integrated.
• Where batch data was once the norm, real-time data is now expected. With BlackBerrys and iPhones in hand, people expect immediate gratification. Getting more data faster contributes to the growing volume.

Businesses are understanding data integration benefits

In order to be useful, data has to be integrated. This may sound obvious, but businesses are really just starting to understand this. They’ve learned it the hard way: spending the last decade allowing spreadmarts to proliferate across departments. Not only did this not deliver the information they needed, it created silos that spawned more problems.

These spreadmarts provide inconsistent views of the enterprise and put businesses in the risky position of making decisions using faulty data. They’re expensive, because each one is usually created and babysat by professionals who should be spending time analyzing data, not gathering, massaging and attempting to integrate it.

Just knowing that they have a problem with spreadmarts doesn’t resolve the problem for businesses. It takes a methodical plan to renovate or replace spreadmarts in a way that preserves the value of their business information while yielding the highest information value. Many businesses across industries have embarked on projects to leverage the business knowledge in these spreadmarts while designing data integration processes that truly incorporate that data into business decision making.

Data integration is evolving

Data integration is moving beyond data warehousing and extract, transform and load (ETL). While the basic tasks of data integration – gathering data, transforming it and putting it into a target location – sound like ETL, new data integration trends and versions of data integration tools offer processes and technologies that extend beyond basic ETL tasks. These technologies help turn data into comprehensive, consistent, clean and current information. The tools support data migration, application consolidation, data profiling, data quality, master data management and operational processing.

These tools allow businesses to determine the state of the source systems, perform cleansing, ensure consistency and manage all the processing, including error handling and performance monitoring. In the past, IT groups had to manually build these processes into their data integration. Often, there was not enough time or the experience to build them properly. The latest tools come pre-built with these capabilities.

In the past, ETL was limited to batch-driven, overnight operations. Data integration suites now incorporate enterprise application integration, enterprise information integration and service-oriented architecture coupled with ETL to offer data integration in batch, interoperating with applications, or in real-time from BI applications . As the business demands more current information, IT can perform data integration to deliver it.

Hand-coding is a hard habit to break

Despite the fact that data integration tools have evolved substantially in recent years, there’s a battle in IT: hand-coding versus ETL tools. Enterprise data warehousing has standardized on ETL tools, but downstream applications like data marts and cubes are often hand-coded. The result is that IT cannot be as responsive as the business would like, so the business then creates spreadmarts in a do-it-yourself attempt to get what it needs.

Hand-coded applications are often undocumented, hard to update and costly to modify. There’s no need to reinvent the wheel and hand-code ETL when there’s a large range of excellent tools at different price points. Some are even free when bundled with other products. It is a better use of IT time and resources to use the pre-built processes to transform data, rather than building them from scratch.

What’s next in data integration trends?

Staying in touch with the evolving nature of data integration will help enterprises create deliberate processes for data integration, saving money and getting more people the information they need.

In Part 2 of this article, I will discuss data integration strategies and technology trends that one should use to meet the increasing demand for business information.