One of the techniques we use in our industry is the center
of some controversy in the news recently. Specifically, the use of data mining
(“The
Datamining Scare,” The Wall Street
Journal, May
13, 2006) to examine who people’s
phone calls or maybe it’s just their phone call records. Either way there is an
outrage from privacy advocates and people concerned with government becoming
too much like “Big Brother.” The Wall Street Journal article “Is the Phone
Company Violating Your Privacy?,” (May 13, 2006) reviews some people’s
concerns.

Without taking sides in this controversy, I find it
interesting that data mining is being used in many applications today including
trying to look for the “bad guys” (again, without commenting on the intentions
of this program which is NOT what this blog is about.)

For too long the classic example of data mining was from the
retail industry: on Saturday nights guys’ shopping baskets include both beer
and diapers. Even Wikipedia, the
online free encyclopedia, defines data mining with the beer
and diapers example in the second paragraph.

Although I am sure the beer/diaper connection was a useful
market basket analysis for that retailer, data mining has come a long way since
then and is being used in many industries. Companies continue to data mine all
sorts of demographic and consumer purchase data to determine who might buy what
and under what conditions like the beer and the diapers example. Data mining is
also used for fraud detection for your credit cards. More importantly for many
of us, data mining is also being used in medical research and clinical trials
to determine what might cure or lessen the impact of diseases.

Data mining has been advancing over the years. You may have
seen it called artificial intelligence, predictive analysis or other names.  Regardless of what people call it, the cost
and resource intensiveness has decreased, putting it within the reach of many
more applications over the years. It was once the province of only the most
deep-pocketed enterprises (like the federal government, oops let’s stay off
that topic!) but now is a cost-effective technique for many applications in
companies of all sizes.

But, just like the tip of the iceberg, there is something to
remember about any great data mining application. The data. Data warehousing
and enterprise applications that are built on relational platforms have enabled
these data mining applications.  Whether they’re
mining for information about diaper purchases or phone records, it’s ultimately
all just data.