Analyst firm’s product ranking reports such as Gartner’s Magic Quadrants and The Forrester Wave are terrific, but are they being used appropriately?

Both provide snapshots of the state of a particular tech category. If you’re unfamiliar with a category, they provide value by giving a high-level assessment of the companies in that market segment. Analysts, consultants and IT use the “leaders” quadrant to build the “shortlist” for evaluating tech products.

That is certainly valuable, but I wonder if it is really helping decision-makers choose the best product for their company. This does not mean that the industry analysts’ research is not correct, but only that people may not be using the research in the most appropriate manner.

How does a product get in the leader’s position? It needs to have the most features, the most customers or the biggest company backing it (or all three.) So “best” is based on biggest in a lot of categories. This is especially true in more mature categories such as business intelligence (BI), data integration and corporate performance management (CPM).

It is a “no brainer” to recommend the industry’s leaders or “top” products, i.e. the products packed with the most features. You could probably pick those products by using industry analysts’ reports or simply by market share. The question is not whether those products are the market leaders and the industry “best,” but rather are they the best fit for you?

If you merely used the “leaders’ list” from the analyst BI reports, then you’d have picked products from one of the top three vendors – Business Objects, Cognos or Hyperion (acquired by SAP, IBM and Oracle respectively.) These companies’ products have led the market share wars for years. (SAS might argue that they have been one of the top three BI products, too, but that is a debate for another day.) And, by reading industry articles and publications, you would assume everyone is buying those three products.

But the collective market share for top three products (along with the market share of the software titans that bought them), according to IDC 2006 research, was 37.3%. That means that almost two-thirds of companies did NOT buy the “best” products.

Why? They were making their choices based on what was most appropriate for them. When companies are evaluating software, they have to determine what they need, how much they are willing to spend and what skills they have to implement the software.

This is true of many of the purchases we make in business or for personal use, and it is just as true for software products. The “leaders” are great, but they may be more expensive and complex, requiring a greater investment (time, money and skills) than other products. These other products may do everything you want at a lower TCO (total cost of ownership.)

The trap of the leaders’ quadrant is that if you use that research as your only input, you might not get the most appropriate software for your company. I have seen many analysts, consultants and IT professionals use the leaders’ list as a crutch. Analysts and consultants should know better, but sometimes they do not really know the other products in the market and sometimes there is a conflict of interest, i.e. they are partners with the leaders’ products.

I have seen too many companies create a shortlist from the leaders’ lists and not even look at other products. These companies then buy one of the leaders and get frustrated because they do not have the resources to deploy the product widely in their company. They got the best, but they might as well have thrown the money out the window. Instead of using the new BI, ETL (extract, transform & load), CPM or master data management (MDM), product they quickly go back to their old ways. What a waste.

Please, if you are developing a shortlist of vendors, start off with the industry analysts’ research but include additional research or advice, such as:

Industry print and online publications, especially articles written by practitioners or consultants who have experience with the technologies.

Conferences, events or local user groups, such as TDWI, DAMA or others — networking with other attendees at these events can be as valuable as the sessions. Please note that I have listed vendor-neutral events.

Online user forums, blogs and websites where people share their own first-hand experiences.

Experienced consultants, which I add not out of self-interest, but because we’ve seen many different kinds of systems in use at many different kinds of companies. But beware of consultancies that are tightly intertwined with one vendor, because their answer will always be that vendor.

Overpaying for something you do not use is a waste. Your company should get great business value from your data warehouse, BI, data integration and CPM efforts. If it doesn’t, maybe you bet on the wrong horse.