Signal Detection Homework due 99

1
Signal Detection Homework due 9/9
Needed IFR certified pilots and novices for
research. If interested, contact Mac Smith
(csmithg_at_gmu.edu)
[phone]
2
Signal Detection Theory
Detection tasks -- Examine aerial photos of Cuba
to detect missiles -- Examine x-ray for a
tumor -- Watch White House security monitors for
intruders -- Self examine for breast cancer --
Search police lineup for criminal suspect
3
Problem

• Problems with simple yes-no tasks
• Assumes that there is no bias in the responses
• For example, a conservative responder might be
  biased to respond absent when in doubt.

4
Signal Detection Theory
Possible states of the world
--signal --noise
Possible responses --signal --noise
How can we explain a given pattern of responses?
5
Signal Detection Theory
Possible states of the world
--signal --noise
Possible responses --signal --noise
State of the World
Medical Terms
Noise
Signal
False Positive
Signal
True Positive
Response
True Negative
False Negative
Noise
How can we explain a given pattern of responses?
6
Signal Detection Theory
Possible states of the world
--signal --noise
Possible responses --signal --noise
State of the World
Statistical Terms
Noise
Signal
Type-I error (?)
Power (1-?)
Signal
Response
Type-II error (?)
.
Noise
How can we explain a given pattern of responses?
7
Signal Detection Theory
Each trial, stimulus generates an internal
(neural) signal.
8
Signal Detection Theory
However, there is also internal noise to deal
with.
9
Signal Noise
Internally, this noise is added to the signal,
making the signal noisy This means that
detecting the presence of a signal at low
intensities is difficult and error prone.
10
Signal Detection Theory
However, as the signal becomes more intense, the
Signal Noise combination resembles the Noise
alone less and less
11
Signal Detection Theory
However, as the signal becomes more intense, the
Signal Noise combination resembles the Noise
alone less and less until finally there is very
little overlap.
12
Signal Detection Theory
Each trial, stimulus generates an internal
(neural) signal, X.
Signal and Noise stimuli tend to
generate internal signals of different strength
13
Performance is imperfect because signal noise
distributions overlap. There is no one point
where it is always signal or always noise
How do we decide whether an internal signal X was
produced by a signal or by noise?
14
How do we decide whether an internal signal X was
produced by a signal or by noise? --Have to
establish a criterion, Xc
If X lt XC, response Noise If X gt XC, response
Signal
15
XC
Noise
P (X N or SN)
X (Internal Signal)
If a Noise trial produces X gt XC, result will be
a False Alarm.
16
XC
Signal
P (X N or SN)
X (Internal Signal)
If a Signal trial produces X lt XC, result will
be a Miss.
17
Setting XC
FAR(False Alarm Rate) MR(miss rate)
18
Setting XC
XC
P (X N or SN)
FAR lt MR
XC
P (X N or SN)
FAR gt MR
19
Setting XC (criterion level)

• By adjusting XC, we introduce response bias
• Quantified by measure Beta
• ratio of signal-to-noise probability that you
  set as your criterion.
• Height of signal distribution over height of
  noise distribution at criterion point (Xc)
• XC moves left, Beta decreases, responses more
  liberal
• XC moves right, Beta increases, responses more
  conservative

20
Setting XC
Bias isnt (necessarily) bad. One state of the
world may be more likely than the other To
minimize errors, responses should favor likely
state If the probability that there is a
signal increases, then your bias will become
smaller and youll become more liberal. Example
airport x-ray detection -- probability of a
weapon is low, so you want to become more
conservative and reduce the likelihood that you
cry wolf
21
(No Transcript)
22
Setting XC
Responses may have different costs and
payoffs Goal will be to minimize costs
/maximize payoffs
V Value (or payoff/benefit) C cost
23
Describes an operators performance.
Indicate where observer should set criterion.
P(XS) height of signal curve at proportion
corresponding to MR (miss rate) P(XN) height
of noise curve at proportion corresponding to
CRR (correct rejection rate)
24
Assume. P(N) .75 P(S) .25
Overall P() of events V(CR) V(H) C(FA) 1
Costs and values of outcomes C(M) 10 Cost of
a miss is high, e.g. terrorist bomb MR .1
CRR .8 Miss and CR rates
Then P(XS) .176 P(XN) .28 Observed
values from MR CRR
Observed Bias
Observer should be more liberal
25
Sluggish Beta

• People dont optimize as they should especially
  at extremes, and especially with probabilities
• Called sluggish beta

26
(No Transcript)
27
Sluggish Beta

• Why?
• People dont like to respond with lots of Y or
  N in a row
• People dont understand probabilities--they tend
  to overestimate rare events
• People often show bias in proportion judgments
  (internal psychophysics may reflect biases in
  perceptual judgment)

28
Mean
1 SD
Normal Curve mean median peak spread
measured by standard deviation or variance a
Z-score indicates how many SDs above/below the
mean a given value is knowing Z-score, we know
what proportion of area under the curve is
below/above X knowing proportion, we can find
corresponding Z-score
29
Sensitivity -- ease of discriminating N from S --
determined by d, separation between N and S
distributions
30
Calculating d measured in SDs if S and N
distributions are normal with equal variance,
then d Z(Hit Rate) Z (False Alarm
Rate) e.g., if HR .8 FAR .15, then d
.84 (-1.04) 1.88 if distributions are
non-normal, or have unequal variances, an
alternative measure is necessary (more to come)
31
(No Transcript)
32
(No Transcript)
33
Sensitivity Bias
In SDT, criterion sensitivity are
independent sensitivity reflects
perception/encoding (affected by attention) bias
reflects decision making
34
By changing bias keeping d constant, we can
plot a Receiver Operating Characteristic.
Sensitivity is constant along each curve. As d
increases, ROC curve becomes more bowed.
35
Receiver Operating Characteristic (ROC)
Probabilities
36
ROC Curve (contd)

• The ROC curve is an isosensivity curvethat is,
  it connects points of equal sensitivity with
  differing bias.
• Tangent to curve at particular point indicates
  value of bias at that point
• Height of curve above negative diagonal along
  positive diagonal represents d

37
ROC curve can also be plotted in Z-scores.
d is the distance from the chance diagonal to
The ROC curve.
38
ROC curves can be collected... -- by varying
payoffs costs -- by varying probabilities
39
Why bother with an ROC curve? Tells us whether
assumptions for calculating d are met.
Z (HR)
Z (FAR)
If assumptions are not met, ROC curve in Z-scores
will not be parallel to chance diagonal -gt means
that d and beta are both varying When
assumptions can not be confirmed, an alternative
measure of sensitivity is
40
SDT measures of Sensitivity Bias
To measure sensitivity when assumptions are met
To measure sensitivity when assumptions can not
be confirmed
(area under ROC curve )
To measure bias
(May change when d changes)
(Less sensitive to d changes)
41
Implications of SDT
Designing experiments need all four cells of 2
x 2 SDT matrix in order to distinguish effects of
sensitivity from effects of bias Understanding
real-world phenomena change in performance
might result from change in sensitivity or change
in bias
42
Implications of SDT
Suspect Identification in Police Lineups Stage 1
Is the suspect here? (detection) Stage 2 Which
of these people is it? (recognition) If the
suspect is actually present, there is no way to
estimate the FAR or CRRand no way to distinguish
sensitivity from bias! Experiments show witness
bias toward suspect present response
witnesses may identify suspect (falsely or
correctly) based on characteristics besides
memory of crime
43
Implications of SDT
Suspect Identification in Police
Lineups Improving performance -- Tell witness
that suspect may not be present -- Use a blank
lineup control (a noise trial) to weed out
witnesses with liberal bias
44
(No Transcript)
45
Take home message SDT separates out bias and
sensitivity.
46
Information Theory
Engineering Psych treats human as information
processing system Perceive/encode Remember
Communicate How to measure information in the
environment? processed by observer? Information
theory provides a system for quantifying
information transmitted by a system.
47
Information Theory
Information is reduction of uncertainty following
the occurrence of some event. All of the
following events convey information conclusion
of a basketball game flashing of the Door
Open light on car dashboard draw of the
nights winning lottery numbers pronouncing of
the final word in a sentence sun rise What
determines the amount of information conveyed by
an event? Number of Possible Events
Probability of Potential Events Context
48
Information Theory
What determines the amount of information
conveyed by an event? Number of Possible
Events Example the conclusion of a race will
convey more information than the conclusion of a
ball game. why? There are more possible winners
in a race and only 2 possible in a ball game.
49
Information Theory
What determines the amount of information
conveyed by an event? Number of Possible
Events Information increases as the number of
possible events increases. If N potential
events are equally likely, then the info
conveyed by a single event is
units are bits Smallest number of possible
events which can produce uncertainty is 2. When
N 2, Hs 1 When N 1, there is no
uncertainty, and Hs 0
50
(No Transcript)
51
Information Theory
Probability of Potential Events Information of
a single event increases as the likelihood of
that event decreases Example The July it was
hot in Washington has less information than The
July it snowed in Washington
52
Information Theory
Probability of Potential Events Information of
a single event increases as the likelihood of
that event decreases If the Pi is the
probability of event i, then the info conveyed
by even i is
notice that when all events are equally
likely, then and
53
Probability of Events
Probability Information certain 100
log2(1/1) 0 likely 10 log2(1/0.1) 3.3 rare 0.
1 log2(1/.001) 9.9 bits
54
Information Theory
Probability of Potential Events Average info
conveyed by a series of events is
Note that the average is weighed by the
likelihood of the event. Therefore, high
probability (low information) events will be
weighed disproportionately. Note that average
information is maximal when all events are
equally likely
55
Example Assume that N 3 and P1
.33 P2 .33 P3 .33 Total 100 Then
HAve .33 1.6 .33 1.6 .33 1.6
1.58
Assume that N 3 and P1 .8 P2
.1 P3 .1 Total 100 Then HAve
.8 .32 .1 3.32 .1 3.32 .92
56
Information Theory
Context/Sequential Constraints Because some
events are more likely under some circumstances
than under others, context can modulate
information conveyed by an event Example What
is the second letter of the word? T----- Q-----
lt- for u, Q--- has less info
Example When I grow up, I want to be a
principal or a _____
Low info policeman, teacher
Ralph Wiggum (High info) caterpillar
57
(No Transcript)