????%20Business%20Intelligence - PowerPoint PPT Presentation

View by Category
About This Presentation
Title:

????%20Business%20Intelligence

Description:

Business Intelligence (Data Mining for Business Intelligence) 1002BI05 IM EMBA Fri 12,13,14 (19:20-22:10) D502 – PowerPoint PPT presentation

Number of Views:187
Avg rating:3.0/5.0
Slides: 112
Provided by: Myd9
Learn more at: http://mail.tku.edu.tw
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: ????%20Business%20Intelligence


1
???? Business Intelligence
????????? (Data Mining for Business Intelligence)
1002BI05 IM EMBA Fri 12,13,14 (1920-2210) D502
Min-Yuh Day ??? Assistant Professor ?????? Dept.
of Information Management, Tamkang
University ???? ?????? http//mail.
tku.edu.tw/myday/ 2012-03-16
2
???? (Syllabus)
  • ?? ?? ??(Subject/Topics) ??
  • 1 101/02/17 ?????? (Introduction to
    Business Intelligence )
  • 2 101/02/24 ?????????????
    (Management Decision Support System and
    Business Intelligence)
  • 3 101/03/02 ?????? (Business Performance
    Management)
  • 4 101/03/09 ???? (Data Warehousing)
  • 5 101/03/16 ????????? (Data Mining for
    Business Intelligence)
  • 6 101/03/24 ????????? (Data Mining for
    Business Intelligence)
  • 7 101/03/30 ????? (????) Banking
    Segmentation (Cluster
    Analysis KMeans)
  • 8 101/04/06 ??????? (--No Class--)
  • 9 101/04/13 ????? (????) Web Site Usage
    Associations (
    Association Analysis)

3
???? (Syllabus)
  • ?? ?? ??(Subject/Topics) ??
  • 10 101/04/20 ???? (Midterm Presentation)
  • 11 101/04/27 ????? (????????)
    Enrollment Management Case Study
    (Decision Tree, Model
    Evaluation)
  • 12 101/05/04 ????? (??????????)Credit Risk
    Case Study (Regression
    Analysis, Artificial Neural Network)
  • 13 101/05/11 ????????? (Text and Web
    Mining)
  • 14 101/05/18 ???? (Intelligent Systems)
  • 15 101/05/25 ?????? (Social Network
    Analysis)
  • 16 101/06/01 ???? (Opinion Mining)
  • 17 101/06/08 ????1 (Project Presentation 2)
  • 18 101/06/15 ????2 (Project Presentation 2)

4
Decision Support and Business Intelligence
Systems (9th Ed., Prentice Hall)
  • Chapter 5
  • Data Mining for Business Intelligence

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
5
Learning Objectives
  • Define data mining as an enabling technology for
    business intelligence
  • Standardized data mining processes
  • CRISP-DM
  • SEMMA
  • Association Analysis
  • Association Rule Mining (Apriori Algorithm)
  • Classification
  • Decision Tree
  • Cluster Analysis
  • K-Means Clustering

6
Data Mining at the Intersection of Many
Disciplines
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
7
A Taxonomy for Data Mining Tasks
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
8
Why Data Mining?
  • More intense competition at the global scale
  • Recognition of the value in data sources
  • Availability of quality data on customers,
    vendors, transactions, Web, etc.
  • Consolidation and integration of data
    repositories into data warehouses
  • The exponential increase in data processing and
    storage capabilities and decrease in cost
  • Movement toward conversion of information
    resources into nonphysical form

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
9
Definition of Data Mining
  • The nontrivial process of identifying valid,
    novel, potentially useful, and ultimately
    understandable patterns in data stored in
    structured databases. - Fayyad et al.,
    (1996)
  • Keywords in this definition Process, nontrivial,
    valid, novel, potentially useful, understandable.
  • Data mining a misnomer?
  • Other names
  • knowledge extraction, pattern analysis,
    knowledge discovery, information harvesting,
    pattern searching, data dredging,

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
10
Data Mining Characteristics/Objectives
  • Source of data for DM is often a consolidated
    data warehouse (not always!)
  • DM environment is usually a client-server or a
    Web-based information systems architecture
  • Data is the most critical ingredient for DM which
    may include soft/unstructured data
  • The miner is often an end user
  • Striking it rich requires creative thinking
  • Data mining tools capabilities and ease of use
    are essential (Web, Parallel processing, etc.)

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
11
Data in Data Mining
  • Data a collection of facts usually obtained as
    the result of experiences, observations, or
    experiments
  • Data may consist of numbers, words, images,
  • Data lowest level of abstraction (from which
    information and knowledge are derived)
  • DM with different data types?
  • - Other data types?

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
12
What Does DM Do?
  • DM extract patterns from data
  • Pattern? A mathematical (numeric and/or
    symbolic) relationship among data items
  • Types of patterns
  • Association
  • Prediction
  • Cluster (segmentation)
  • Sequential (or time series) relationships

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
13
Data Mining Applications
  • Customer Relationship Management
  • Maximize return on marketing campaigns
  • Improve customer retention (churn analysis)
  • Maximize customer value (cross-, up-selling)
  • Identify and treat most valued customers
  • Banking and Other Financial
  • Automate the loan application process
  • Detecting fraudulent transactions
  • Optimizing cash reserves with forecasting

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
14
Data Mining Applications (cont.)
  • Retailing and Logistics
  • Optimize inventory levels at different locations
  • Improve the store layout and sales promotions
  • Optimize logistics by predicting seasonal effects
  • Minimize losses due to limited shelf life
  • Manufacturing and Maintenance
  • Predict/prevent machinery failures
  • Identify anomalies in production systems to
    optimize the use manufacturing capacity
  • Discover novel patterns to improve product quality

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
15
Data Mining Applications (cont.)
  • Brokerage and Securities Trading
  • Predict changes on certain bond prices
  • Forecast the direction of stock fluctuations
  • Assess the effect of events on market movements
  • Identify and prevent fraudulent activities in
    trading
  • Insurance
  • Forecast claim costs for better business planning
  • Determine optimal rate plans
  • Optimize marketing to specific customers
  • Identify and prevent fraudulent claim activities

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
16
Data Mining Applications (cont.)
  • Computer hardware and software
  • Science and engineering
  • Government and defense
  • Homeland security and law enforcement
  • Travel industry
  • Healthcare
  • Medicine
  • Entertainment industry
  • Sports
  • Etc.

Highly popular application areas for data mining
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
17
Data Mining Process
  • A manifestation of best practices
  • A systematic way to conduct DM projects
  • Different groups has different versions
  • Most common standard processes
  • CRISP-DM (Cross-Industry Standard Process for
    Data Mining)
  • SEMMA (Sample, Explore, Modify, Model, and
    Assess)
  • KDD (Knowledge Discovery in Databases)

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
18
Data Mining Process CRISP-DM
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
19
Data Mining Process CRISP-DM
  • Step 1 Business Understanding
  • Step 2 Data Understanding
  • Step 3 Data Preparation (!)
  • Step 4 Model Building
  • Step 5 Testing and Evaluation
  • Step 6 Deployment
  • The process is highly repetitive and experimental
    (DM art versus science?)

Accounts for 85 of total project time
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
20
Data Preparation A Critical DM Task
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
21
Data Mining Process SEMMA
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
22
Data Mining Methods Classification
  • Most frequently used DM method
  • Part of the machine-learning family
  • Employ supervised learning
  • Learn from past data, classify new data
  • The output variable is categorical (nominal or
    ordinal) in nature
  • Classification versus regression?
  • Classification versus clustering?

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
23
Assessment Methods for Classification
  • Predictive accuracy
  • Hit rate
  • Speed
  • Model building predicting
  • Robustness
  • Scalability
  • Interpretability
  • Transparency, explainability

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
24
Accuracy of Classification Models
  • In classification problems, the primary source
    for accuracy estimation is the confusion matrix

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
25
Estimation Methodologies for Classification
  • Simple split (or holdout or test sample
    estimation)
  • Split the data into 2 mutually exclusive sets
    training (70) and testing (30)
  • For ANN, the data is split into three sub-sets
    (training 60, validation 20, testing
    20)

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
26
Estimation Methodologies for Classification
  • k-Fold Cross Validation (rotation estimation)
  • Split the data into k mutually exclusive subsets
  • Use each subset as testing while using the rest
    of the subsets as training
  • Repeat the experimentation for k times
  • Aggregate the test results for true estimation of
    prediction accuracy training
  • Other estimation methodologies
  • Leave-one-out, bootstrapping, jackknifing
  • Area under the ROC curve

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
27
Estimation Methodologies for Classification ROC
Curve
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
28
Market Basket Analysis
Source Han Kamber (2006)
29
Association Rule Mining
  • Apriori Algorithm

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
30
Association Rule Mining
  • A very popular DM method in business
  • Finds interesting relationships (affinities)
    between variables (items or events)
  • Part of machine learning family
  • Employs unsupervised learning
  • There is no output variable
  • Also known as market basket analysis
  • Often used as an example to describe DM to
    ordinary people, such as the famous relationship
    between diapers and beers!

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
31
Association Rule Mining
  • Input the simple point-of-sale transaction data
  • Output Most frequent affinities among items
  • Example according to the transaction data
  • Customer who bought a laptop computer and a
    virus protection software, also bought extended
    service plan 70 percent of the time."
  • How do you use such a pattern/knowledge?
  • Put the items next to each other for ease of
    finding
  • Promote the items as a package (do not put one on
    sale if the other(s) are on sale)
  • Place items far apart from each other so that the
    customer has to walk the aisles to search for it,
    and by doing so potentially seeing and buying
    other items

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
32
Association Rule Mining
  • A representative applications of association rule
    mining include
  • In business cross-marketing, cross-selling,
    store design, catalog design, e-commerce site
    design, optimization of online advertising,
    product pricing, and sales/promotion
    configuration
  • In medicine relationships between symptoms and
    illnesses diagnosis and patient characteristics
    and treatments (to be used in medical DSS) and
    genes and their functions (to be used in genomics
    projects)

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
33
Association Rule Mining
  • Are all association rules interesting and useful?
  • A Generic Rule X ? Y S, C
  • X, Y products and/or services
  • X Left-hand-side (LHS)
  • Y Right-hand-side (RHS)
  • S Support how often X and Y go together
  • C Confidence how often Y go together with the X
  • Example Laptop Computer, Antivirus Software ?
    Extended Service Plan 30, 70

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
34
Association Rule Mining
  • Algorithms are available for generating
    association rules
  • Apriori
  • Eclat
  • FP-Growth
  • Derivatives and hybrids of the three
  • The algorithms help identify the frequent item
    sets, which are, then converted to association
    rules

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
35
Association Rule Mining
  • Apriori Algorithm
  • Finds subsets that are common to at least a
    minimum number of the itemsets
  • uses a bottom-up approach
  • frequent subsets are extended one item at a time
    (the size of frequent subsets increases from
    one-item subsets to two-item subsets, then
    three-item subsets, and so on), and
  • groups of candidates at each level are tested
    against the data for minimum

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
36
Basic Concepts Frequent Patterns and Association
Rules
  • Itemset X x1, , xk
  • Find all the rules X ? Y with minimum support and
    confidence
  • support, s, probability that a transaction
    contains X ? Y
  • confidence, c, conditional probability that a
    transaction having X also contains Y

Transaction-id Items bought
10 A, B, D
20 A, C, D
30 A, D, E
40 B, E, F
50 B, C, D, E, F
Let supmin 50, confmin 50 Freq. Pat.
A3, B3, D4, E3, AD3 Association rules A ?
D (60, 100) D ? A (60, 75)
A ? D (support 3/5 60, confidence 3/3
100) D ? A (support 3/5 60, confidence
3/4 75)
Source Han Kamber (2006)
37
Market basket analysis
  • Example
  • Which groups or sets of items are customers
    likely to purchase on a given trip to the store?
  • Association Rule
  • Computer ? antivirus_software support 2
    confidence 60
  • A support of 2 means that 2 of all the
    transactions under analysis show that computer
    and antivirus software are purchased together.
  • A confidence of 60 means that 60 of the
    customers who purchased a computer also bought
    the software.

Source Han Kamber (2006)
38
Association rules
  • Association rules are considered interesting if
    they satisfy both
  • a minimum support threshold and
  • a minimum confidence threshold.

Source Han Kamber (2006)
39
Frequent Itemsets, Closed Itemsets, and
Association Rules
  • Support (A? B) P(A ? B)
  • Confidence (A? B) P(BA)

Source Han Kamber (2006)
40
Support (A? B) P(A ? B) Confidence (A? B)
P(BA)
  • The notation P(A ? B) indicates the probability
    that a transaction contains the union of set A
    and set B
  • (i.e., it contains every item in A and in B).
  • This should not be confused with P(A or B), which
    indicates the probability that a transaction
    contains either A or B.

Source Han Kamber (2006)
41
  • Rules that satisfy both a minimum support
    threshold (min_sup) and a minimum confidence
    threshold (min_conf) are called strong.
  • By convention, we write support and confidence
    values so as to occur between 0 and 100, rather
    than 0 to 1.0.

Source Han Kamber (2006)
42
  • itemset
  • A set of items is referred to as an itemset.
  • K-itemset
  • An itemset that contains k items is a k-itemset.
  • Example
  • The set computer, antivirus software is a
    2-itemset.

Source Han Kamber (2006)
43
Absolute Support and Relative Support
  • Absolute Support
  • The occurrence frequency of an itemset is the
    number of transactions that contain the itemset
  • frequency, support count, or count of the itemset
  • Ex 3
  • Relative support
  • Ex 60

Source Han Kamber (2006)
44
  • If the relative support of an itemset I satisfies
    a prespecified minimum support threshold, then I
    is a frequent itemset.
  • i.e., the absolute support of I satisfies the
    corresponding minimum support count threshold
  • The set of frequent k-itemsets is commonly
    denoted by LK

Source Han Kamber (2006)
45
  • the confidence of rule A? B can be easily derived
    from the support counts of A and A ? B.
  • once the support counts of A, B, and A ? B are
    found, it is straightforward to derive the
    corresponding association rules A?B and B?A and
    check whether they are strong.
  • Thus the problem of mining association rules can
    be reduced to that of mining frequent itemsets.

Source Han Kamber (2006)
46
Association rule mining Two-step process
  • 1. Find all frequent itemsets
  • By definition, each of these itemsets will occur
    at least as frequently as a predetermined minimum
    support count, min_sup.
  • 2. Generate strong association rules from the
    frequent itemsets
  • By definition, these rules must satisfy minimum
    support and minimum confidence.

Source Han Kamber (2006)
47
Efficient and Scalable Frequent Itemset Mining
Methods
  • The Apriori Algorithm
  • Finding Frequent Itemsets Using Candidate
    Generation

Source Han Kamber (2006)
48
Apriori Algorithm
  • Apriori is a seminal algorithm proposed by R.
    Agrawal and R. Srikant in 1994 for mining
    frequent itemsets for Boolean association rules.
  • The name of the algorithm is based on the fact
    that the algorithm uses prior knowledge of
    frequent itemset properties, as we shall see
    following.

Source Han Kamber (2006)
49
Apriori Algorithm
  • Apriori employs an iterative approach known as a
    level-wise search, where k-itemsets are used to
    explore (k1)-itemsets.
  • First, the set of frequent 1-itemsets is found by
    scanning the database to accumulate the count for
    each item, and collecting those items that
    satisfy minimum support. The resulting set is
    denoted L1.
  • Next, L1 is used to find L2, the set of frequent
    2-itemsets, which is used to find L3, and so on,
    until no more frequent k-itemsets can be found.
  • The finding of each Lk requires one full scan of
    the database.

Source Han Kamber (2006)
50
Apriori Algorithm
  • To improve the efficiency of the level-wise
    generation of frequent itemsets, an important
    property called the Apriori property.
  • Apriori property
  • All nonempty subsets of a frequent itemset must
    also be frequent.

Source Han Kamber (2006)
51
  • How is the Apriori property used in the
    algorithm?
  • How Lk-1 is used to find Lk for k gt 2.
  • A two-step process is followed, consisting of
    join and prune actions.

Source Han Kamber (2006)
52
Apriori property used in algorithm 1. The join
step
Source Han Kamber (2006)
53
Apriori property used in algorithm 2. The prune
step
Source Han Kamber (2006)
54
Transactional data for an AllElectronics branch
Source Han Kamber (2006)
55
Example Apriori
  • Lets look at a concrete example, based on the
    AllElectronics transaction database, D.
  • There are nine transactions in this database,
    that is, D 9.
  • Apriori algorithm for finding frequent itemsets
    in D

Source Han Kamber (2006)
56
Example Apriori Algorithm Generation of
candidate itemsets and frequent itemsets, where
the minimum support count is 2.
Source Han Kamber (2006)
57
Example Apriori Algorithm C1 ? L1
Source Han Kamber (2006)
58
Example Apriori Algorithm C2 ? L2
Source Han Kamber (2006)
59
Example Apriori Algorithm C3 ? L3
Source Han Kamber (2006)
60
The Apriori algorithm for discovering frequent
itemsets for mining Boolean association rules.
Source Han Kamber (2006)
61
The Apriori AlgorithmAn Example
Supmin 2
Itemset sup
A 2
B 3
C 3
D 1
E 3
Database TDB
Itemset sup
A 2
B 3
C 3
E 3
L1
C1
Tid Items
10 A, C, D
20 B, C, E
30 A, B, C, E
40 B, E
1st scan
C2
C2
Itemset sup
A, B 1
A, C 2
A, E 1
B, C 2
B, E 3
C, E 2
Itemset
A, B
A, C
A, E
B, C
B, E
C, E
L2
2nd scan
Itemset sup
A, C 2
B, C 2
B, E 3
C, E 2
C3
L3
Itemset
B, C, E
Itemset sup
B, C, E 2
3rd scan
Source Han Kamber (2006)
62
The Apriori Algorithm
  • Pseudo-code
  • Ck Candidate itemset of size k
  • Lk frequent itemset of size k
  • L1 frequent items
  • for (k 1 Lk !? k) do begin
  • Ck1 candidates generated from Lk
  • for each transaction t in database do
  • increment the count of all candidates in
    Ck1 that are
    contained in t
  • Lk1 candidates in Ck1 with min_support
  • end
  • return ?k Lk

Source Han Kamber (2006)
63
Generating Association Rules from Frequent
Itemsets
Source Han Kamber (2006)
64
Example Generating association rules
  • frequent itemset l I1, I2, I5
  • If the minimum confidence threshold is, say, 70,
    then only the second, third, and last rules above
    are output, because these are the only ones
    generated that are strong.

Source Han Kamber (2006)
65
Classification Techniques
  • Decision tree analysis
  • Statistical analysis
  • Neural networks
  • Support vector machines
  • Case-based reasoning
  • Bayesian classifiers
  • Genetic algorithms
  • Rough sets

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
66
Example of Classification
  • Loan Application Data
  • Which loan applicants are safe and which are
    risky for the bank?
  • Safe or risky for load application data
  • Marketing Data
  • Whether a customer with a given profile will buy
    a new computer?
  • yes or no for marketing data
  • Classification
  • Data analysis task
  • A model or Classifier is constructed to predict
    categorical labels
  • Labels safe or risky yes or no
    treatment A, treatment B, treatment C

Source Han Kamber (2006)
67
Prediction Methods
  • Linear Regression
  • Nonlinear Regression
  • Other Regression Methods

Source Han Kamber (2006)
68
Classification and Prediction
  • Classification and prediction are two forms of
    data analysis that can be used to extract models
    describing important data classes or to predict
    future data trends.
  • Classification
  • Effective and scalable methods have been
    developed for decision trees induction, Naive
    Bayesian classification, Bayesian belief network,
    rule-based classifier, Backpropagation, Support
    Vector Machine (SVM), associative classification,
    nearest neighbor classifiers, and case-based
    reasoning, and other classification methods such
    as genetic algorithms, rough set and fuzzy set
    approaches.
  • Prediction
  • Linear, nonlinear, and generalized linear models
    of regression can be used for prediction. Many
    nonlinear problems can be converted to linear
    problems by performing transformations on the
    predictor variables. Regression trees and model
    trees are also used for prediction.

Source Han Kamber (2006)
69
ClassificationA Two-Step Process
  • Model construction describing a set of
    predetermined classes
  • Each tuple/sample is assumed to belong to a
    predefined class, as determined by the class
    label attribute
  • The set of tuples used for model construction is
    training set
  • The model is represented as classification rules,
    decision trees, or mathematical formulae
  • Model usage for classifying future or unknown
    objects
  • Estimate accuracy of the model
  • The known label of test sample is compared with
    the classified result from the model
  • Accuracy rate is the percentage of test set
    samples that are correctly classified by the
    model
  • Test set is independent of training set,
    otherwise over-fitting will occur
  • If the accuracy is acceptable, use the model to
    classify data tuples whose class labels are not
    known

Source Han Kamber (2006)
70
Supervised vs. Unsupervised Learning
  • Supervised learning (classification)
  • Supervision The training data (observations,
    measurements, etc.) are accompanied by labels
    indicating the class of the observations
  • New data is classified based on the training set
  • Unsupervised learning (clustering)
  • The class labels of training data is unknown
  • Given a set of measurements, observations, etc.
    with the aim of establishing the existence of
    classes or clusters in the data

Source Han Kamber (2006)
71
Issues Regarding Classification and Prediction
Data Preparation
  • Data cleaning
  • Preprocess data in order to reduce noise and
    handle missing values
  • Relevance analysis (feature selection)
  • Remove the irrelevant or redundant attributes
  • Attribute subset selection
  • Feature Selection in machine learning
  • Data transformation
  • Generalize and/or normalize data
  • Example
  • Income low, medium, high

Source Han Kamber (2006)
72
Issues Evaluating Classification and Prediction
Methods
  • Accuracy
  • classifier accuracy predicting class label
  • predictor accuracy guessing value of predicted
    attributes
  • estimation techniques cross-validation and
    bootstrapping
  • Speed
  • time to construct the model (training time)
  • time to use the model (classification/prediction
    time)
  • Robustness
  • handling noise and missing values
  • Scalability
  • ability to construct the classifier or predictor
    efficiently given large amounts of data
  • Interpretability
  • understanding and insight provided by the model

Source Han Kamber (2006)
73
Data Classification Process 1 Learning
(Training) Step (a) Learning Training data are
analyzed by classification algorithm
y f(X)
Source Han Kamber (2006)
74
Data Classification Process 2 (b)
Classification Test data are used to estimate
the accuracy of the classification rules.
Source Han Kamber (2006)
75
Process (1) Model Construction
Classification Algorithms
IF rank professor OR years gt 6 THEN tenured
yes
Source Han Kamber (2006)
76
Process (2) Using the Model in Prediction
(Jeff, Professor, 4)
Tenured?
Source Han Kamber (2006)
77
Decision Trees
A general algorithm for decision tree building
  • Employs the divide and conquer method
  • Recursively divides a training set until each
    division consists of examples from one class
  • Create a root node and assign all of the training
    data to it
  • Select the best splitting attribute
  • Add a branch to the root node for each value of
    the split. Split the data into mutually exclusive
    subsets along the lines of the specific split
  • Repeat the steps 2 and 3 for each and every leaf
    node until the stopping criteria is reached

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
78
Decision Trees
  • DT algorithms mainly differ on
  • Splitting criteria
  • Which variable to split first?
  • What values to use to split?
  • How many splits to form for each node?
  • Stopping criteria
  • When to stop building the tree
  • Pruning (generalization method)
  • Pre-pruning versus post-pruning
  • Most popular DT algorithms include
  • ID3, C4.5, C5 CART CHAID M5

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
79
Decision Trees
  • Alternative splitting criteria
  • Gini index determines the purity of a specific
    class as a result of a decision to branch along a
    particular attribute/value
  • Used in CART
  • Information gain uses entropy to measure the
    extent of uncertainty or randomness of a
    particular attribute/value split
  • Used in ID3, C4.5, C5
  • Chi-square statistics (used in CHAID)

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
80
Classification by Decision Tree
Induction Training Dataset
This follows an example of Quinlans ID3 (Playing
Tennis)
Source Han Kamber (2006)
81
Output A Decision Tree for buys_computer
Classification by Decision Tree Induction
yes
yes
yes
no
no
buys_computeryes or buys_computerno
Source Han Kamber (2006)
82
Three possibilities for partitioning tuples
based on the splitting Criterion
Source Han Kamber (2006)
83
Algorithm for Decision Tree Induction
  • Basic algorithm (a greedy algorithm)
  • Tree is constructed in a top-down recursive
    divide-and-conquer manner
  • At start, all the training examples are at the
    root
  • Attributes are categorical (if continuous-valued,
    they are discretized in advance)
  • Examples are partitioned recursively based on
    selected attributes
  • Test attributes are selected on the basis of a
    heuristic or statistical measure (e.g.,
    information gain)
  • Conditions for stopping partitioning
  • All samples for a given node belong to the same
    class
  • There are no remaining attributes for further
    partitioning majority voting is employed for
    classifying the leaf
  • There are no samples left

Source Han Kamber (2006)
84
Attribute Selection Measure
  • Notation Let D, the data partition, be a
    training set of class-labeled tuples. Suppose
    the class label attribute has m distinct values
    defining m distinct classes, Ci (for i 1, ,
    m). Let Ci,D be the set of tuples of class Ci in
    D. Let D and Ci,D denote the number of
    tuples in D and Ci,D , respectively.
  • Example
  • Class buys_computer yes or no
  • Two distinct classes (m2)
  • Class Ci (i1,2) C1 yes, C2 no

Source Han Kamber (2006)
85
Attribute Selection Measure Information Gain
(ID3/C4.5)
  • Select the attribute with the highest information
    gain
  • Let pi be the probability that an arbitrary tuple
    in D belongs to class Ci, estimated by Ci,
    D/D
  • Expected information (entropy) needed to classify
    a tuple in D
  • Information needed (after using A to split D into
    v partitions) to classify D
  • Information gained by branching on attribute A

Source Han Kamber (2006)
86
Class-labeled training tuples from the
AllElectronics customer database
The attribute age has the highest information
gain and therefore becomes the splitting
attribute at the root node of the decision tree
Source Han Kamber (2006)
87
Attribute Selection Information Gain
  • Class P buys_computer yes
  • Class N buys_computer no
  • means age lt30 has 5 out of 14
    samples, with 2 yeses and 3 nos. Hence
  • Similarly,

Source Han Kamber (2006)
88
Gain Ratio for Attribute Selection (C4.5)
  • Information gain measure is biased towards
    attributes with a large number of values
  • C4.5 (a successor of ID3) uses gain ratio to
    overcome the problem (normalization to
    information gain)
  • GainRatio(A) Gain(A)/SplitInfo(A)
  • Ex.
  • gain_ratio(income) 0.029/0.926 0.031
  • The attribute with the maximum gain ratio is
    selected as the splitting attribute

Source Han Kamber (2006)
89
Cluster Analysis
  • Used for automatic identification of natural
    groupings of things
  • Part of the machine-learning family
  • Employ unsupervised learning
  • Learns the clusters of things from past data,
    then assigns new instances
  • There is not an output variable
  • Also known as segmentation

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
90
Cluster Analysis
Clustering of a set of objects based on the
k-means method. (The mean of each cluster is
marked by a .)
Source Han Kamber (2006)
91
Cluster Analysis
  • Clustering results may be used to
  • Identify natural groupings of customers
  • Identify rules for assigning new cases to classes
    for targeting/diagnostic purposes
  • Provide characterization, definition, labeling of
    populations
  • Decrease the size and complexity of problems for
    other data mining methods
  • Identify outliers in a specific domain (e.g.,
    rare-event detection)

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
92
Example of Cluster Analysis
Point P P(x,y)
p01 a (3, 4)
p02 b (3, 6)
p03 c (3, 8)
p04 d (4, 5)
p05 e (4, 7)
p06 f (5, 1)
p07 g (5, 5)
p08 h (7, 3)
p09 i (7, 5)
p10 j (8, 5)



93
Cluster Analysis for Data Mining
  • Analysis methods
  • Statistical methods (including both hierarchical
    and nonhierarchical), such as k-means, k-modes,
    and so on
  • Neural networks (adaptive resonance theory
    ART, self-organizing map SOM)
  • Fuzzy logic (e.g., fuzzy c-means algorithm)
  • Genetic algorithms
  • Divisive versus Agglomerative methods

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
94
Cluster Analysis for Data Mining
  • How many clusters?
  • There is not a truly optimal way to calculate
    it
  • Heuristics are often used
  • Look at the sparseness of clusters
  • Number of clusters (n/2)1/2 (n no of data
    points)
  • Use Akaike information criterion (AIC)
  • Use Bayesian information criterion (BIC)
  • Most cluster analysis methods involve the use of
    a distance measure to calculate the closeness
    between pairs of items
  • Euclidian versus Manhattan (rectilinear) distance

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
95
k-Means Clustering Algorithm
  • k pre-determined number of clusters
  • Algorithm (Step 0 determine value of k)
  • Step 1 Randomly generate k random points as
    initial cluster centers
  • Step 2 Assign each point to the nearest cluster
    center
  • Step 3 Re-compute the new cluster centers
  • Repetition step Repeat steps 2 and 3 until some
    convergence criterion is met (usually that the
    assignment of points to clusters becomes stable)

Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
96
Cluster Analysis for Data Mining - k-Means
Clustering Algorithm
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
97
Similarity and Dissimilarity Between Objects
  • Distances are normally used to measure the
    similarity or dissimilarity between two data
    objects
  • Some popular ones include Minkowski distance
  • where i (xi1, xi2, , xip) and j (xj1, xj2,
    , xjp) are two p-dimensional data objects, and q
    is a positive integer
  • If q 1, d is Manhattan distance

Source Han Kamber (2006)
98
Similarity and Dissimilarity Between Objects
(Cont.)
  • If q 2, d is Euclidean distance
  • Properties
  • d(i,j) ? 0
  • d(i,i) 0
  • d(i,j) d(j,i)
  • d(i,j) ? d(i,k) d(k,j)
  • Also, one can use weighted distance, parametric
    Pearson product moment correlation, or other
    disimilarity measures

Source Han Kamber (2006)
99
Euclidean distance vs Manhattan distance
  • Distance of two point x1 (1, 2) and x2 (3, 5)

Euclidean distance ((3-1)2 (5-2)2 )1/2 (22
32)1/2 (4 9)1/2 (13)1/2 3.61
x2 (3, 5)
5
4
3
3.61
3
2
2
x1 (1, 2)
Manhattan distance (3-1) (5-2) 2 3 5
1
1
2
3
100
The K-Means Clustering Method
  • Example

10
9
8
7
6
5
Update the cluster means
Assign each objects to most similar center
4
3
2
1
0
0
1
2
3
4
5
6
7
8
9
10
reassign
reassign
K2 Arbitrarily choose K object as initial
cluster center
Update the cluster means
Source Han Kamber (2006)
101
K-Means Clustering Step by Step
Point P P(x,y)
p01 a (3, 4)
p02 b (3, 6)
p03 c (3, 8)
p04 d (4, 5)
p05 e (4, 7)
p06 f (5, 1)
p07 g (5, 5)
p08 h (7, 3)
p09 i (7, 5)
p10 j (8, 5)



102
K-Means Clustering
Step 1 K2, Arbitrarily choose K object as
initial cluster center
Point P P(x,y)
p01 a (3, 4)
p02 b (3, 6)
p03 c (3, 8)
p04 d (4, 5)
p05 e (4, 7)
p06 f (5, 1)
p07 g (5, 5)
p08 h (7, 3)
p09 i (7, 5)
p10 j (8, 5)

Initial m1 (3, 4)
Initial m2 (8, 5)
M2 (8, 5)
m1 (3, 4)
103
Step 2 Compute seed points as the centroids of
the clusters of the current partition Step 3
Assign each objects to most similar center
Point P P(x,y) m1 distance m2 distance Cluster
p01 a (3, 4) 0.00 5.10 Cluster1
p02 b (3, 6) 2.00 5.10 Cluster1
p03 c (3, 8) 4.00 5.83 Cluster1
p04 d (4, 5) 1.41 4.00 Cluster1
p05 e (4, 7) 3.16 4.47 Cluster1
p06 f (5, 1) 3.61 5.00 Cluster1
p07 g (5, 5) 2.24 3.00 Cluster1
p08 h (7, 3) 4.12 2.24 Cluster2
p09 i (7, 5) 4.12 1.00 Cluster2
p10 j (8, 5) 5.10 0.00 Cluster2

Initial m1 (3, 4)
Initial m2 (8, 5)
M2 (8, 5)
m1 (3, 4)
K-Means Clustering
104
Step 2 Compute seed points as the centroids of
the clusters of the current partition Step 3
Assign each objects to most similar center
Point P P(x,y) m1 distance m2 distance Cluster
p01 a (3, 4) 0.00 5.10 Cluster1
p02 b (3, 6) 2.00 5.10 Cluster1
p03 c (3, 8) 4.00 5.83 Cluster1
p04 d (4, 5) 1.41 4.00 Cluster1
p05 e (4, 7) 3.16 4.47 Cluster1
p06 f (5, 1) 3.61 5.00 Cluster1
p07 g (5, 5) 2.24 3.00 Cluster1
p08 h (7, 3) 4.12 2.24 Cluster2
p09 i (7, 5) 4.12 1.00 Cluster2
p10 j (8, 5) 5.10 0.00 Cluster2

Initial m1 (3, 4)
Initial m2 (8, 5)
M2 (8, 5)
Euclidean distance b(3,6) ??m2(8,5) ((8-3)2
(5-6)2 )1/2 (52 (-1)2)1/2 (25 1)1/2
(26)1/2 5.10
m1 (3, 4)
Euclidean distance b(3,6) ??m1(3,4) ((3-3)2
(4-6)2 )1/2 (02 (-2)2)1/2 (0 4)1/2
(4)1/2 2.00
K-Means Clustering
105
Step 4 Update the cluster means,
Repeat Step 2, 3, stop when no more
new assignment
Point P P(x,y) m1 distance m2 distance Cluster
p01 a (3, 4) 1.43 4.34 Cluster1
p02 b (3, 6) 1.22 4.64 Cluster1
p03 c (3, 8) 2.99 5.68 Cluster1
p04 d (4, 5) 0.20 3.40 Cluster1
p05 e (4, 7) 1.87 4.27 Cluster1
p06 f (5, 1) 4.29 4.06 Cluster2
p07 g (5, 5) 1.15 2.42 Cluster1
p08 h (7, 3) 3.80 1.37 Cluster2
p09 i (7, 5) 3.14 0.75 Cluster2
p10 j (8, 5) 4.14 0.95 Cluster2

m1 (3.86, 5.14) (3.86, 5.14)
m2 (7.33, 4.33) (7.33, 4.33)
m1 (3.86, 5.14)
M2 (7.33, 4.33)
K-Means Clustering
106
Step 4 Update the cluster means,
Repeat Step 2, 3, stop when no more
new assignment
Point P P(x,y) m1 distance m2 distance Cluster
p01 a (3, 4) 1.95 3.78 Cluster1
p02 b (3, 6) 0.69 4.51 Cluster1
p03 c (3, 8) 2.27 5.86 Cluster1
p04 d (4, 5) 0.89 3.13 Cluster1
p05 e (4, 7) 1.22 4.45 Cluster1
p06 f (5, 1) 5.01 3.05 Cluster2
p07 g (5, 5) 1.57 2.30 Cluster1
p08 h (7, 3) 4.37 0.56 Cluster2
p09 i (7, 5) 3.43 1.52 Cluster2
p10 j (8, 5) 4.41 1.95 Cluster2

m1 (3.67, 5.83) (3.67, 5.83)
m2 (6.75, 3.50) (6.75, 3.50)
m1 (3.67, 5.83)
M2 (6.75., 3.50)
K-Means Clustering
107
stop when no more new assignment
Point P P(x,y) m1 distance m2 distance Cluster
p01 a (3, 4) 1.95 3.78 Cluster1
p02 b (3, 6) 0.69 4.51 Cluster1
p03 c (3, 8) 2.27 5.86 Cluster1
p04 d (4, 5) 0.89 3.13 Cluster1
p05 e (4, 7) 1.22 4.45 Cluster1
p06 f (5, 1) 5.01 3.05 Cluster2
p07 g (5, 5) 1.57 2.30 Cluster1
p08 h (7, 3) 4.37 0.56 Cluster2
p09 i (7, 5) 3.43 1.52 Cluster2
p10 j (8, 5) 4.41 1.95 Cluster2

m1 (3.67, 5.83) (3.67, 5.83)
m2 (6.75, 3.50) (6.75, 3.50)
K-Means Clustering
108
stop when no more new assignment
Point P P(x,y) m1 distance m2 distance Cluster
p01 a (3, 4) 1.95 3.78 Cluster1
p02 b (3, 6) 0.69 4.51 Cluster1
p03 c (3, 8) 2.27 5.86 Cluster1
p04 d (4, 5) 0.89 3.13 Cluster1
p05 e (4, 7) 1.22 4.45 Cluster1
p06 f (5, 1) 5.01 3.05 Cluster2
p07 g (5, 5) 1.57 2.30 Cluster1
p08 h (7, 3) 4.37 0.56 Cluster2
p09 i (7, 5) 3.43 1.52 Cluster2
p10 j (8, 5) 4.41 1.95 Cluster2

m1 (3.67, 5.83) (3.67, 5.83)
m2 (6.75, 3.50) (6.75, 3.50)
K-Means Clustering
109
Data Mining Software
  • Commercial
  • SPSS - PASW (formerly Clementine)
  • SAS - Enterprise Miner
  • IBM - Intelligent Miner
  • StatSoft Statistical Data Miner
  • many more
  • Free and/or Open Source
  • Weka
  • RapidMiner

Source KDNuggets.com, May 2009
Source Turban et al. (2011), Decision Support
and Business Intelligence Systems
110
Summary
  • Define data mining as an enabling technology for
    business intelligence
  • Standardized data mining processes
  • CRISP-DM
  • SEMMA
  • Association Analysis
  • Association Rule Mining (Apriori Algorithm)
  • Classification
  • Decision Tree
  • Cluster Analysis
  • K-Means Clustering

111
References
  • Efraim Turban, Ramesh Sharda, Dursun Delen,
    Decision Support and Business Intelligence
    Systems, Ninth Edition, 2011, Pearson.
  • Jiawei Han and Micheline Kamber, Data Mining
    Concepts and Techniques, Second Edition, 2006,
    Elsevier
About PowerShow.com