# A Tutorial on ROC Curve and AUC - PowerPoint PPT Presentation

PPT – A Tutorial on ROC Curve and AUC PowerPoint presentation | free to view - id: 175538-ZDc1Z

The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
Title:

## A Tutorial on ROC Curve and AUC

Description:

### Inductive Learning Algorithms. Inductive Learning Algorithms ... For a decision tree, it is the fraction of the examples of class in the leaf xi falls into. ... – PowerPoint PPT presentation

Number of Views:1080
Avg rating:5.0/5.0
Slides: 21
Provided by: valued937
Category:
Tags:
Transcript and Presenter's Notes

Title: A Tutorial on ROC Curve and AUC

1
A Tutorial on ROC Curve and AUC
2
Outline
• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

3
Inductive Learning Algorithms
• Inductive Learning Algorithms
• Given training examples x1, , xn, where xi(A1,
, An), and their classes ,, learn a
classifier f'(x)?, with probability
• or
• f(x) underlying true classifier
• P(x) given an example xi, the probability of
xi belonging to class

4
Evaluation Criteria
• Classification Accuracy
• The percentage of examples correctly classified
• Probability-based Ranking
• How close the result ranking to the underlying
ranking.
• Probability Estimation
• The difference between P(xi) and

5
Probability of Class Membership
• an estimate of P(xi)
• Most traditional classifiers also produce
probability of class membership as by-product.
• For a decision tree, it is the fraction of the
examples of class in the leaf xi falls into.

6
Why Ranking
• Classification assign a class label to an
example.
• Ranking rank a set of examples according to
their scores.
• Ranking is more desirable in some applications.
• Direct marketing.

7
Outline
• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

8
ROC Curve (1)
• True positive rate TP and false positive rate FP,
of a classifier are

9
ROC Curve (2)
• Given a ranking
• examples x1 x2 x3 x4 x5
x6 x7 x8 x9 x10
• 0.1 0.3 0.3 0.4 0.4 0.5
0.6 0.7 0.8 0.9
• Classes
• Try setting different cutoff

• TP5/5 FP4/5

• TP5/5 FP3/5
•

• TP1/5 FP0/5

10
ROC Curve (3)
11
• A perfect ranking
• A ROC curve A is said to dominate another ROC
curve B if A is always above and to the left of
B.
• Loosely, A is better than B.
• No clear dominating relation between two ROC
curves in many cases.

12
Outline
• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

13
AUC (1)
• The area under the ROC curve (AUC).
• summary'' for comparing the two ROC curves.
• One ROC curve dominates the other, its AUC must
be larger.
• Intuitively, the larger the AUC of a ROC, the
better.

14
AUC (2)
• Given a classifier G, let
• estimated probability of ith positive example in
class , i1, , n0
• estimated probability of ith negative example in
class , i1, , n1

15
AUC (3)
• Rank the set g1, , gn1, f1, , fn0. Let ri
be the rank of ith positive example.

16
AUC (4)
• AUC of G is given below

17
AUC(5)
• AUC reflects the quality of a ranking.
• For the perfect ranking,
• its AUC is 1.
• For the worst ranking,
• its AUC is 0.

18
AUC (6)
• Comparison between AUC and accuracy
• One example

Classifier1 AUC is 24/25, accuracy is
0.8 Classifier2 AUC is 16/25, accuracy is 0.8
19
AUC (7)
• One counter example

Classifier1 AUC is 0.889, accuracy is
0.667 Classifier2 AUC is 0.694, accuracy is
0.833
20
Outline
• The Evaluation of Learning Algorithms
• ROC Curve
• AUC and Ranking
• Related Research Topics

21
Related Research Topics
• What is the performance of traditional learning
algorithms, such as decision trees, Bayesian
networks in terms of AUC ?
• How to develop new algorithms that aim at maximum
AUC scores?