Moss L.T., Atre S. Business intelligence roadmap: The complete project lifecycle for decision-support applications

Подождите немного. Документ загружается.

[ Team LiB ]

Defining Data Mining

Data mining capability is not something you can buy off the shelf. Data mining

requires building a BI decision-support application, specifically a data mining

application, using a data mining tool. The data mining application can then use a

sophisticated blend of classical and advanced components like artificial intelligence,

pattern recognition, databases, traditional statistics, and graphics to present hidden

relationships and patterns found in the organization's data pool.

Data mining is the analysis of data with the intent to discover gems of hidden

information in the vast quantity of data that has been captured in the normal course

of running the business. Data mining is different from conventional statistical

analysis, as indicated in Table 13.1. They both have strengths and weaknesses.

Table 13.1. Statistical Analysis versus Data Mining

Statistical Analysis

Data Mining

Statisticians usually start

with a hypothesis.

Data mining does not require a hypothesis.

Statisticians have to develop

their own equations to

match their hypothesis.

Data mining algorithms can automatically

develop the equations.

Statistical analysis uses only

numerical data.

Data mining can use different types of data

(e.g., text, voice), not just numerical data.

Statisticians can find and

filter dirty data during their

analysis.

Data mining depends on clean, well-

documented data.

Statisticians interpret their

own results and convey

these results to the business

managers and business

executives.

Data mining results are not easy to

interpret. A statistician must still be

involved in analyzing the data mining

results and conveying the findings to the

business managers and business

executives.

Tables 13.2 and 13.3 use specific examples (insurance fraud and market

segmentation, respectively) to illustrate the differences between traditional analysis

techniques and discovery-driven data mining.

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

381/631

Table 13.2. Example of Insurance Fraud Analysis

Traditional Analysis Technique

Discovery-Driven Data Mining

An analyst notices a pattern of

behavior that might indicate

insurance fraud. Based on this

hypothesis, the analyst creates a set

of queries to determine whether this

observed behavior actually

constitutes fraud. If the results are

not conclusive, the analyst starts over

with a modified or new hypothesis

and more queries. Not only is this

process time-consuming, but it also

depends on the analyst's subjective

interpretation of the results. More

importantly, this process will not find

any patterns of fraud that the analyst

does not already suspect.

An analyst sets up the data

mining application, then "trains" it

to find all unusual patterns,

trends, or deviations from the

norm that might constitute

insurance fraud. The data mining

results unearth various situations

that the analyst can investigate

further. For the follow-up

investigation, the analyst can then

use verification-driven queries.

Together, these efforts can help

the analyst build a model

predicting which customers or

potential customers might commit

fraud.

Table 13.3. Example of Market Segmentation Analysis

Traditional Analysis Technique

Discovery-Driven Data Mining

An analyst wants to study the buying

behaviors of known classes of

customers (e.g., retired school

teachers, young urban professionals)

in order to design targeted marketing

programs. First, the analyst uses

known characteristics about those

classes of customers and tries to sort

them into groups. Second, he or she

studies the buying behaviors common

to each group. The analyst repeats this

process until he or she is satisfied with

the final customer groupings.

The data mining tool "studies"

the database using the

clustering technique to identify

all groups of customers with

distinct buying patterns. After

the data is mined and the

groupings are presented, the

analyst can use various query,

reporting, and multidimensional

analysis tools to analyze the

results.

The Importance of Data Mining

Discovery-driven data mining finds answers to questions that decision-makers do not

know to ask. Because of this powerful capability, data mining is an important

component of business intelligence. One may even say that data mining, also called

knowledge discovery, is a breakthrough in providing business intelligence to strategic

decision-makers. At first glance, this claim may seem excessive. After all, many

current decision-support applications provide business intelligence and insights.

Executive information systems (EISs) enable senior managers to monitor,

examine, and change many aspects of their business operations.

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

382/631

Query and reporting tools give business analysts the ability to investigate

company performance and customer behavior.

Statistical tools enable statisticians to perform sophisticated studies of the

behavior of a business.

New multidimensional online analytical processing (OLAP) tools deliver the

ability to perform "what if" analysis and to look at a large number of

interdependent factors involved in a business problem.

Many of these tools work with BI applications and can sift through vast amounts of

data. Given this abundance of tools, what is so different about discovery-driven data

mining? The big difference is that traditional analysis techniques, even sophisticated

ones, rely on the analyst to know what to look for in the data. The analyst creates

and runs queries based on some hypotheses and guesses about possible

relationships, trends, and correlations thought to be present in the data. Similarly,

the executive relies on the business views built into the EIS tool, which can examine

only the factors the tool is programmed to review. As problems become more

complex and involve more variables to analyze, these traditional analysis techniques

can fall short. In contrast, discovery-driven data mining supports very subtle and

complex investigations.

Data Sources for Data Mining

BI target databases are popular sources for data mining applications. They contain a

wealth of internal data that was gathered and consolidated across business

boundaries, validated, and cleansed in the extract/transform/load (ETL) process. BI

target databases may also contain valuable external data, such as regulations,

demographics, or geographic information. Combining external data with internal

organizational data offers a splendid foundation for data mining.

The drawback of multidimensional BI target databases is that since the data has

been summarized, hidden data patterns, data relationships, and data associations

are often no longer discernable from that data pool. For example, the data mining

tool may not be able to perform the common data mining task of market basket

analysis (also called associations discovery, described in the next section) based on

summarized sales data because some detailed data pattern about each sale may

have gotten lost in the summarization. Therefore, operational files and databases are

also popular sources for data mining applications, especially because they contain

transaction-level detailed data with a myriad of hidden data patterns, data

relationships, and data associations.

Exercise caution with operational systems extracts

because the data could contain many duplicates,

inconsistencies, and errors and could skew the data

mining results.

Data mining tools could theoretically access the operational databases and BI target

databases directly without building data mining databases first, as long as the

database structures are supported by the tool (e.g., relational like Oracle,

hierarchical like IMS, or even a flat file like VSAM). However, this is not an advisable

practice for several reasons.

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

383/631

The data pool needs to be able to change for different data mining runs, such as

dropping a sales region or restricting a product type for a specific mining

purpose. Changing the data content of operational or BI target databases is not

possible.

The performance of operational as well as BI target databases would be

impacted by the data mining operations. That is unacceptable for operational

databases and not desirable for BI target databases.

A data mining operation may need detailed historical data. Operational

databases do not store historical data, and BI target databases often do not

have the desired level of detail. Archival tapes may have to be restored and

merged to extract the desired data.

Therefore, organizations often extract data for data mining as needed from their BI

target databases and from their operational files and databases into special-purpose

data mining databases (Figure 13.1).

Figure 13.1. Data Sources for Data Mining Applications

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

384/631

[ Team LiB ]

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

385/631

[ Team LiB ]

Data Mining Techniques

Data mining techniques are specific implementations of algorithms used in data

mining operations. The five most common data mining techniques are described

briefly below.

Associations Discovery

This data mining technique is used to identify the behavior of specific events or

processes. Associations discovery links occurrences within a single event. An

example might be the discovery that men who purchase premium brands of coffee

are three times more likely to buy imported cigars than men who buy standard

brands of coffee. Associations discovery is based on rules that follow this general

form: "If item A is part of an event, then X percent of the time (confidence factor),

item B is part of the same event." For example:

If a customer buys snacks, there is an 85 percent probability that the customer

will also buy soft drinks or beer.

If a person buys vacation airline tickets for an entire family, there is a 95

percent probability that he or she will rent a full-size car at the vacation

location.

With the help of scanners, retail stores use this data mining technique to find buying

patterns in grocery stores. Because of the context of a grocery store, associations

discovery is sometimes called market basket analysis.

Sequential Pattern Discovery

This data mining technique is similar to associations discovery except that a

sequential pattern discovery links events over time and determines how items relate

to each other over time. For example, sequential pattern discovery might predict that

a person who buys a washing machine may also buy a clothes dryer within six

months with a probability of 0.7. To increase the chances above the predicted 70

percent probability, the store may offer each buyer a 10 percent discount on a

clothes dryer within four months after purchasing a washing machine.

Classification

The classification technique is the most common use of data mining. Classification

looks at the behavior and attributes of predetermined groups. The groups might

include frequent flyers, high spenders, loyal customers, people who respond to direct

mail campaigns, or people with frequent back problems (e.g., people who drive long

distances every day). The data mining tool can assign classifications to new data by

examining existing data that has already been classified and by using those results

to infer a set of rules. The set of rules is then applied to any new data to be

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

386/631

classified. This technique often uses supervised induction, which employs a small

training set of already classified records to determine additional classifications. An

example of this use is to discover the characteristics of customers who are (or are

not) likely to buy a certain type of product. This knowledge would result in reducing

the costs of promotions and direct mailings.

Clustering

The clustering technique is used to discover different groupings within the data.

Clustering is similar to classification except that no groups have yet been defined at

the outset of running the data mining tool. The clustering technique often uses

neural networks or statistical methods. Clustering divides items into groups based on

the similarities the data mining tool finds. Within a cluster the members are very

similar, but the clusters themselves are very dissimilar. Clustering is used for

problems such as detecting manufacturing defects or finding affinity groups for credit

cards.

Forecasting

The forecasting data mining technique comes in two flavors: regression analysis and

time sequence discovery.

Regression analysis uses known values of data to predict future values or

future events based on historical trends and statistics. For example, the sales

volume of sports car accessories can be forecasted based on the number of

sports cars sold last month.

Time sequence discovery differs from regression analysis in that it forecasts

only time-dependent data values. For example, it determines the rates of

accidents during a holiday season based on the number of accidents that

occurred during the same holiday season in prior years. The property of time

can be:

- Work week versus calendar week

- Holidays

- Seasons

- Date ranges and date intervals

[ Team LiB ]

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

387/631

[ Team LiB ]

Data Mining Operations

Data mining tools enable statisticians to build analytical models, which the tools then

use during data mining operations. A predictive engine asks for a list of input criteria

and follows the steps and relationships from the analytical model to determine the

most likely predictions. The results of data mining operations are tables and files

loaded with analysis data that can be accessed with query and reporting tools. The

four main data mining operations are described below.

Predictive and Classification Modeling

This data mining operation is used to forecast a particular event. It assumes that the

analyst has a specific question he or she wants to ask. The model provides the

answer by assigning ranks that indicate the likelihood of certain classes. For

example, if a bank analyst wants to predict which customers are likely to leave, he or

she has to prepare for predictive modeling by feeding data about two types of

customers into the data mining tool.

Customer data that indicates which customers have already left. This data is

called "bad" data.

Customer data that indicates which customers stayed and are long-time

customers. This data is called "good" data.

The tool then sifts through the data to uncover the variables that identify classes of

profiles of typical customers who leave and classes of profiles of typical customers

who stay. The analysis results might be, "A female customer over 40 years of age

who has an income greater than $150,000 per year and owns her own home has a

35 percent chance of leaving the bank."

Typical probing questions for predictive data mining are those that look for

associations, patterns, trends, and facts in order to make decisions. For example:

Which offers will prompt customers to buy more? (Trend)

Which customers should be targeted for a new product? (Association)

What are the signs of fraudulent activity? (Pattern)

Which customers are better credit risks? (Fact)

Link Analysis

The link analysis data mining operation is a collection of mathematical algorithms

and visualization techniques that identify and visually present links between

individual records in a database. It is related to the associations discovery and

sequential pattern discovery data mining techniques. For example, link analysis can

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

388/631

determine which items usually sell together (e.g., cereal and milk).

Database Segmentation

This data mining operation is a set of algorithms that group similar records into

homogeneous segments. It is related to the clustering data mining technique. This

grouping is often the first step of data selection, before other data mining operations

take place. For example, database segmentation may group airline passengers as

either frequent flyer passengers or occasional passengers.

Deviation Detection

The deviation detection data mining operation is a set of algorithms that look for

records that fall outside some expectation or norm and then suggest reasons for the

anomalies. While deviation detection is mainly used for fraud detection, other uses

include tracing the potential reasons for declines in customer numbers or sales. For

example, "Customers who used to make frequent purchases but have not purchased

anything in a long time were either transferred by their companies or have moved

away from the area."

[ Team LiB ]

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

389/631

[ Team LiB ]

Applications of Data Mining

It is important to keep in mind that in spite of all the dazzling technologies, data

mining has to be driven by strong business needs in order to justify the expenditure

of time and money. One of the common business drivers for engaging in data mining

is to gain market share. This can be accomplished by either introducing new products

or by taking away market share from competitors. In either case, a data mining

application can help you decide how best to achieve your goals.

There are many types of data mining applications. The five most common ones are

briefly described below.

Market management

- Cross-selling: Identify cross-selling opportunities among existing

customers, who are likely to buy as a result of direct mail campaigns and

promotions (and thus minimize selling costs).

- Defecting customers: Determine which customers are likely to switch

brands by using vulnerability analysis that creates a predictive model of

behavior, so that the company can craft strategies to retain these

customers.

- Promotions and campaigns: Distinguish natural groupings in the

market, such as key sales periods for given items, by performing market

segmentation analysis as a way to fine-tune promotions and campaigns.

- Prospecting: Classify groups of prospects in order to find ways to

perform target marketing for each group.

- Market basket analysis: Evaluate which items people buy together

during a visit to a supermarket or store, using market basket analysis on

point-of-sale data. Then use the information to group products in store

displays, to adjust inventory, and to price and promote items.

Fraud detection

- Credit card fraud: Isolate credit card fraud by identifying meaningful

patterns of transactions as well as deviations from those patterns. Use

this model to predict an applicant's trustworthiness.

- Calling card fraud: Determine telephone calling card situations that look

suspicious and are likely to indicate fraud.

- Insurance fraud: Analyze large data pools of insurance claims to identify

possible fraud related to health insurance, car insurance, or property and

casualty insurance.

Risk management

北斗成功社区 BeiDouWeb.com 教育音视频/电子书/实用资料文档/励志音乐影视仅供免费试用/版权原著所有

390/631