Calibration Lecture

1

• The aim of this presentation is to explain how to
  use the confidence response scale so that your
  confidence responses show good calibration.

2
Calibration Graph

• In this slideshow, you will see graphs like the
  one to the right.
• This graph represents calibration, or the
  correspondence between each confidence response
  category you use and the percentage of time you
  were correct about which item was more expensive.
  

3

• The x-axis of the graph gives the confidence
  response categories that you chose among.
• The y-axis of the graph gives the percentage of
  time you were correct about which item was more
  expensive.

4

• For instance, we would count the number of times
  you said 50, 60, etc confident (doing this
  separately for each confidence response category).

• Then, we compute the percentage of times the item
  you chose was more expensive for that particular
  confidence response category.

5

• Lets say you said 100 chance 10 times
• and the item you chose as more expensive is more
  expensive 8 of those times (80)
• That would be demonstrated by the blue dot.

6

• Now lets look at a perfectly calibrated
  participant
• Out of all the times they said 50, the item they
  chose was more expensive 50 of the time
• Out of all the times they said 60, the item they
  chose was more expensive 60 of the time
• Etc.

• Note that to be perfectly calibrated, you dont
  have to be correct all the time. Instead, your
  confidence responses should match the proportion
  of time your chosen item is more expensive.

7

• Now lets look at a participant with poor
  calibration.
• Many of the points are far from the line.
• Out of all the times they said they were 90
  confident that they chose the more expensive
  item, their chosen item was more expensive only
  60 of the time.
• Out of all the times they said they were 100
  confident, they were correct only 70 of the
  time.

8

• And lets see an example with even worse
  calibration.
• Here, every one of their dots is far from the
  line of perfect calibration.

9

• To review, you want your points (the blue dots)
  to be as close to the line as possible.
• Note it is very difficult to have your points
  fall exactly on the line. Just give your best
  estimates, and try to get as close as you can.
  The graph on the left, for example, would
  indicate good calibration.