A Tutorial on ROC Curve and AUC - PowerPoint PPT Presentation

Loading...

PPT – A Tutorial on ROC Curve and AUC PowerPoint presentation | free to view - id: 175538-ZDc1Z



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

A Tutorial on ROC Curve and AUC

Description:

Inductive Learning Algorithms. Inductive Learning Algorithms ... For a decision tree, it is the fraction of the examples of class in the leaf xi falls into. ... – PowerPoint PPT presentation

Number of Views:1080
Avg rating:5.0/5.0
Slides: 21
Provided by: valued937
Category:
Tags: auc | roc | an | curve | in | inductive | tutorial

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: A Tutorial on ROC Curve and AUC


1
A Tutorial on ROC Curve and AUC
2
Outline
  • The Evaluation of Learning Algorithms
  • ROC Curve
  • AUC and Ranking
  • Related Research Topics

3
Inductive Learning Algorithms
  • Inductive Learning Algorithms
  • Given training examples x1, …, xn, where xi(A1,
    …, An), and their classes ,, learn a
    classifier f'(x)?, with probability
  • or
  • f(x) underlying true classifier
  • P(x) given an example xi, the probability of
    xi belonging to class

4
Evaluation Criteria
  • Classification Accuracy
  • The percentage of examples correctly classified
  • Probability-based Ranking
  • How close the result ranking to the underlying
    ranking.
  • Probability Estimation
  • The difference between P(xi) and

5
Probability of Class Membership
  • an estimate of P(xi)
  • Most traditional classifiers also produce
    probability of class membership as by-product.
  • For a decision tree, it is the fraction of the
    examples of class in the leaf xi falls into.

6
Why Ranking
  • Classification assign a class label to an
    example.
  • Ranking rank a set of examples according to
    their scores.
  • Ranking is more desirable in some applications.
  • Direct marketing.

7
Outline
  • The Evaluation of Learning Algorithms
  • ROC Curve
  • AUC and Ranking
  • Related Research Topics

8
ROC Curve (1)
  • True positive rate TP and false positive rate FP,
    of a classifier are

9
ROC Curve (2)
  • Given a ranking
  • examples x1 x2 x3 x4 x5
    x6 x7 x8 x9 x10
  • 0.1 0.3 0.3 0.4 0.4 0.5
    0.6 0.7 0.8 0.9
  • Classes
  • Try setting different cutoff

  • TP5/5 FP4/5

  • TP5/5 FP3/5
  • ……

  • TP1/5 FP0/5

10
ROC Curve (3)
11
Read ROC Curves
  • A perfect ranking
  • A ROC curve A is said to dominate another ROC
    curve B if A is always above and to the left of
    B.
  • Loosely, A is better than B.
  • No clear dominating relation between two ROC
    curves in many cases.

12
Outline
  • The Evaluation of Learning Algorithms
  • ROC Curve
  • AUC and Ranking
  • Related Research Topics

13
AUC (1)
  • The area under the ROC curve (AUC).
  • summary'' for comparing the two ROC curves.
  • One ROC curve dominates the other, its AUC must
    be larger.
  • Intuitively, the larger the AUC of a ROC, the
    better.

14
AUC (2)
  • Given a classifier G, let
  • estimated probability of ith positive example in
    class , i1, …, n0
  • estimated probability of ith negative example in
    class , i1, …, n1

15
AUC (3)
  • Rank the set g1, …, gn1, f1, …, fn0. Let ri
    be the rank of ith positive example.

16
AUC (4)
  • AUC of G is given below

17
AUC(5)
  • AUC reflects the quality of a ranking.
  • For the perfect ranking,
  • its AUC is 1.
  • For the worst ranking,
  • its AUC is 0.

18
AUC (6)
  • Comparison between AUC and accuracy
  • One example

Classifier1 AUC is 24/25, accuracy is
0.8 Classifier2 AUC is 16/25, accuracy is 0.8
19
AUC (7)
  • One counter example

Classifier1 AUC is 0.889, accuracy is
0.667 Classifier2 AUC is 0.694, accuracy is
0.833
20
Outline
  • The Evaluation of Learning Algorithms
  • ROC Curve
  • AUC and Ranking
  • Related Research Topics

21
Related Research Topics
  • What is the performance of traditional learning
    algorithms, such as decision trees, Bayesian
    networks in terms of AUC ?
  • How to develop new algorithms that aim at maximum
    AUC scores?
About PowerShow.com