Comparisons between the hybrid method J'L Hall, 1981 and QUEST Watson

1
Comparisons between the hybrid method (J.L
Hall, 1981) and QUEST (Watson Pelli, 1983)
Philip Jaekl
Hybrid Method
Hall (1981) implemented an algorithm which
combines elements of Taylor and Creelmans PEST
with components of maximum likelihood
estimation (mle). This approach runs the PEST
algorithm to collect responses at stimulus levels
near the 75 threshold. PEST uses a Wald
sequential likelihood ratio equation to determine
whether or not the stimulus level should be
changed. When the answer indicates the stimulus
level should be increased or decreased, a number
of simple rules determine the step size. The
staircase is terminated when the step-size is
reduced to a pre-determined fraction of its
initial magnitude. They state (as concisely as I
possibly can) Starting anew with each change in
testing level, the experimenter keeps running
total of the number of correct responses ( C) and
the total number of trials T . After each trial,
the test defines permissible upper and lower
bounds on N(C). If N(C) falls between these
bounds another trial is made at the same testing
level. If N(C) falls on or above the upper bound,
the decision is that the current level is too
high, and if N(C) is on or below the lower bound,
the current level is taken to be too low. If the
current testing level were exactly Lt, the
expected number o f correct trials E N(C) Pt
x T, after T trials. The sequential test bounds
are given by the expected number of events plus
and minus a constant
Nb(C)EN(C)W where Nb(C) is the bounding
number of events after T trials, and W is a
constant, called the deviation limit of the
sequential test.
2
Subsequent step sizes are guided by 4 rules 1.
On every reversal of step direction halve the
step size. 2. The second step in a given
direction if called f or, is the same size as the
first. 3. The fourth and subsequent steps in a
given direction are each double their predecessor
except that, as noted above, large steps may be
disturbing to a human observer and an upper limit
on permissible step size may be needed). 4.
Whether a third successive step i n a given
direction is the same as or double the second
depends on the se- quence of steps leading to the
most recent reversal. If the step immediately
preceding that reversal resulted from a doubling,
then the third step is not doubled, while if the
step leading to the most recent reversal as not
the result of a doubling, then this third step
is double the second. When the algorithm
terminates the experiment, the data is then
fitted using a maximum likelihood procedure.
This method runs on the principle of finding the
parameters of the psychometric function that
maximize the likelihood of the responses
ob- tained given the stimuli presented. On the
left in panel A, an adaptive track is
shown, which terminates after 50 trials (Hall,
1981). It was derived using a 2AFC auditory
task, for which the participant identified tones
in white noise. Panel B illustratesa maximum
likelihood logistic fit to these data.
3
QUEST Method
The theory behind QUEST (which is actually not an
acronym) uses Bayes rule to form a probability
density function (pdf) for the likelihood of a
threshold given the set of responses obtained.
Specifically, the implementation of Bayes rule
is FTD(TD) fT(T)fDT(DT)
fD(D) where FTD(TD) is the posterior pdf of
the threshold (T) given the data (D) fT(T) is the
prior density function of the threshold, based on
previous knowledge fDT(DT) is the pdf
likelihood of the data condition upon T and fD(D)
is a constant representing the prior pdf of the
data. The stimulus level in each sequence is
given by the output of this equation as the mode
of the posterior pdf. Watson and Pelli (1983)
suggest using a confidence interval to determine
the termination point of the adaptive run such
that the experiment ends when a confidence
interval for the location of threshold in the
likelihood function is smaller than a
pre-determined size. They also suggest using a
specified number of trials for the experiment. I
assume this suggestion is to be recommended under
the condition that pilot data (perhaps derived
from a simulation) shows stability in the
adaptive track after a certain number of
trials. As a final estimate they suggest using
the data, upon completion of the experiment, to
fit a maximum likelihood psychometric function to
derive the threshold.
4
Human data Hybrid Method
Human data was collected using participant pj. A
backward masking metacontrast paradigm was used
to determine the Weber contrast ((stimulus
luminance-background lumance)/background
luminance) at the 75 detection threshold in a
2AFC task. Under the guidline of Hall (1981) the
initial stimulus level was based on a
psychometric function obtained from pilot data.
Hall suggests using the 50 level (4the slope
of the function). In this experiment, the pilot
data was fitted using a Weibull function on a
pedestal of 0.5, which made the range between 0.5
and 1. Thus the initial contrast was quite high
as the mean was actually the 75 level. This may
be seen in the adaptive track of pj, illustrated
below.
The experiment terminated after 93 trials, which
is when the step size was reduced to 128th of
its original size of 5 units. The final
estimate of threshold in the adaptive track is
3.88. Observe the generally longer runs at a
given level as the stimulus approaches threshold.
5
Human data Hybrid Method
This figure displays the proportion correct at
each contrast level with a maximum likelihood
Weibull fit through the data. To centre on most
of the data, the abscissa in the figure ranges
between 0 and 7, although there are 2 other
levels (13.5 and 8.5) for which the proportion
correct were 1. The final estimate of threshold,
based on this fit is 75 3.68 beta 4.11
6
Human data QUEST Method
Human data was again collected using participant
pj in the same 2AFC contrast threshold task. The
experiment was set to terminate after 93 trials.
This limit was chosen such that the estimated
threshold could be directly compared with the
threshold obtained using the hybrid method, which
terminated after 93 trials. Since the observer
was experienced with the initial stimulus was
placed near the threshold obtained from pilot
data. Since Quest, as a maximum likelihood
method is more prone than PEST to getting stuck
within a given range (Hall, 1981 Leek, 2001)
this initial level also kept the testing level
within a probable threshold range.
The program which implements the QUEST method is
mquest.m. The data was again fitted using
fitmask.m.. The final estimate of threshold in
the adaptive track is 3.79, which is very close
to the estimate of 3.88 obtained at the end of
the PEST adaptive track. It is possible to see
how the change from one trial to the next reduces
as the likelihood function narrows. The track
appears to stabilize roughly after 45 trials,
which is about half of the required trials.
7
Human data QUEST Method
To the left the data obtained using QUEST are
fitted using the same method as the Hybrid data.
A Weibull function indicates The final estimate
of threshold, based on this fit is 75
3.30 beta 2.84 The threshold estimate is quite
comparable to the final estimate obtained from
the fit to the hybrid data (3.68) although, the
slope of 2.84 is considerably lower than the
hybrid estimate of slope of 4.12.
8
Human data QUEST Method
To the left the data obtained using QUEST are
fitted using the same method as the Hybrid data.
A Weibull function indicates The final estimate
of threshold, based on this fit is 75
3.30 beta 2.84 The threshold estimate is quite
comparable to the final estimate obtained from
the fit to the hybrid data (3.68) although, the
slope of 2.84 is considerably lower than the
hybrid estimate of slope of 4.12.
9
Simulated Observer data Hybrid Method
Estimated thresholds were derived from a
simulated observer with a normal cumulative
density function. The final estimate was derived
using the same Weibull fit as the hybrid data.
Some important parameters of the observer
function and simulation were 75 threshold
3.49 (the mean of the estimates obtained with pj
using the hybrid method and QUEST. beta1 (this
approximated the Weibull function fitted to the
hybrid data) Starting point3.5 (the same
starting point used for pj) Termination
rulestepsizelt5/1280.4 (also used for
pj) Number of simulations1000
A historgram of the frequency of the estimated
thresholds (bin size 0.05). (based on the
command to create the histogram in PESTsim.m, Im
not sure why the low thresholds are not
displayed Ive tried it a number of times and
this seems to be consistent)
This figure shows the estimated thresholds
across the run of simulations. 4 instances of
very low threshold estimates may be observed.
10
Simulated Observer data Hybrid Method
Bias According to Treutwein (1995), threshold
measurement bias can only be obtained when the
true threshold is known. Thus, the bias can be
obtained using these simulations by simply
calculating true threshold the mean of the
threshold estimates The mean of the estimated
thresholds from this simulation is 3.46. The
standard deviation was 0.33. The standard error
is 0.01. The bias is 3.49-3.460.03
impressively low.
11
Simulated Observer data Hybrid Method
Efficiency Taylor and Creelman (1967) have
defined the sweat factor K, as a measure of
efficiency for a psychophysical procedure. The
sweat factor is derived by
Which is a product of the fixed number of trials
in each simulation and the variance of the best
threshold estimate. To obtain the sweat factor,
the termination rule for the experiment was
changed to a fixed number of 93 trials the
number of trials it took pj to reach a step size
of 0.04. Thus, the efficiency is 930.1089
10.128. This number will be compared with the K
obtained from the QUEST simulation.
12
Simulated Observer data Hybrid Method
Bias and Efficiency Bias and efficiency may
also be described in other ways to compare
different parameters used for the same
psychophysical method. As stated the true
threshold was set to 3.49. The threshold
obtained when the termination rule for the
simulations was initial step size/128 was 3.46.
The illustration below shows a histogram of the
number of trials required to reach this
threshold. The bin size is 5. These data show
that using this stopping rule for PEST, the
distribution of trial number is positively skewed
and that it is possible for a simulated observer
with the parameters of pj to take up to
approximately 200 trials before the PEST
converges. The mean number of trials, however
was 93.7 with a standard deviation of 32.1.
Interestingly, the mean threshold obtained when
the trial number was fixed at 93 was 3.45, which
is still very close to the true threshold.
13
Simulated Observer data QUEST Method
Estimated thresholds were derived from a
simulated observer with a normal cumulative
density function. The final estimate was derived
using the same Weibull fit as the hybrid
simulation. Some important parameters of the
observer function and simulation were 75
threshold 3.49 (the mean of the estimates
obtained with pj using the hybrid method and
QUEST. beta1 (this approximated the Weibull
function fitted to the hybrid data) Starting
point of likelihood function (initial most
probable contrast threshold)3.8 (3.5 should have
been used to make a direct comparison with
observer pj) Termination rule93 trials (same as
QUEST used for pj) Number of simulations1000
A historgram of the frequency of the estimated
thresholds (bin size 0.05).
This figure shows the estimated thresholds
across the run of simulations.
14
Simulated Observer data QUEST Method
Bias As stated bias can be obtained using these
simulations by true threshold the mean of
the threshold estimates The mean of the
estimated thresholds from the QUEST simulations
is 3.57. The standard deviation was 0.28. The
standard error is 0.009. The bias is
3.49-3.57-0.08 units of Weber contrast also
impressively low, but not as low as the 0.03 unit
bias of Hybrid method.
15
Simulated Observer data Hybrid Method
Efficiency Again, the sweat factor K is derived
by
The effiency is 930.0784 7.291. This sweat
factor is smaller than that obtained for the
hybrid simulation set to 93 trials, which was
10.128. Thus, the QUEST method is more
efficient than the hybrid method. But what does
this mean, really? To make another comparison
the QUEST simulation was run with the trial
number set to 46, half the trials of the initial
simulation and also, approximately the amount
after which an eyeball estimate of stabilization
for the adaptive track of pj. In this case, the
mean threshold was 3.61 and the standard
deviation was 0.44, resulting in a bias of -0.12
and a sweat factor of 18. Relatively, a 0.12
difference in luminance is very small
however there was enough variation in the
estimates to result in more than a twofold loss
of efficiency.
16
Conclusion
The hybrid method appears to have an advantage of
less bias while QUEST appears to be more
efficient. If many conditions are to be tested
efficiency is important. Simulations may be run
based on pilot data to determine the bias and
efficiency. I would recommend trying different
trial numbers in the QUEST simulation to find an
appropriate number of trials for smallest
acceptable bias. This is currently the state of
one of my projects and what I plan to do.