Effect Sizes for Meta-analysis of Single-Subject Designs

1
Effect Sizes for Meta-analysis of Single-Subject
Designs

• S. Natasha Beretvas
• University of Texas at Austin

2
Beretvas grant

• Three studies
• 1.a) Summarize practices used for meta-analyzing
  SSD results
• 1.b) Summarize methods used to calculate effect
  sizes (ESs) for SSD results
• 2. Simulation study evaluating performance of
  selection of ESs
• 3. Conduct actual meta-analysis of school-based
  interventions for children with autism spectrum
  disorders.

3
Outline

• Large-n designs data
• Large-n Effect Sizes
• Single-n designs data
• Single-n Effect Sizes (sample)
• Problems
• 4-parameter model (AB designs)
• Explanation
• Continuing research

4
Large-n Studies Data

• Most simply consists of a randomly selected and
  assigned sample of participants in each of the
  Treatment and Control groups.
• Each participant is measured once on the outcome.
• Each participant provides an independently
  observed data point.
• The standard deviation provides an estimate of
  the variability of these independent data points.

5
Large-n Effect Sizes

• Provides a practical measure of the size and
  direction of a treatments effect.
• In large-n studies, the standardized mean
  difference is most typically used
• Represents how different the two groups means
  are on the outcome of interest.
• The standardized part originates in the
  difference being measured in standard deviations

6
Single-n Studies Data
7
Single-n Studies Data

• Most simply repeated measures on an individual
  over time in two phases (time series data)
• Baseline phase A control
• Treatment phase B treatment
• Score at time point t is related to score at time
  (t 1) not independent.

8
Single-n Studies DataVisual Analysis

• Plots are evaluated for the presence of a
  treatment effect by simultaneously considering
  the following
• Sustainable level and/or trend changes
• Baseline trends in expected direction
• Overlapping data between phases
• Variability changes within and across phases.

9
Single-n Effect Sizes

• Seems reasonable that a standardized difference
  between scores in phase A and B could be used as
  an effect size (ES)
• It seems feasible that this effect size would be
  on the same metric as for large-n designs?!
• No!!

10
Problems with d for single-n designs

• The standard deviation, s, for single-n designs
  describes different variability than for large-n
  designs.
• If these were not problems, then it would also
  only make sense to use d when there is no trend
  in the data.

11
Trend in A and B phases, tx effect
A single number cannot summarize changes in level
and slope
12
Trend in B phase, tx effect
13
Trend in A and B phases, no tx effect
What would d indicate about this pattern?
14
Alternative single-n ESs

• Percent Non-overlapping data (PND) is one of the
  most frequently used ES descriptors.
• If treatments effect is anticipated to increase
  outcome then
• Horizontal line drawn through highest point in
  phase A through points in phase B
• PND of phase B points above line
• The higher the PND, the stronger the support for
  a treatments effect.

15
PND
Baseline
Treatment
PND 6/6 100
16
PND
Baseline
Treatment
PND 11/13 84.6
17
PND

• PND is simple to calculate and interpret and
  takes into consideration
• Baseline variability
• Slope changes, but

18
PND
What would PND indicate about this pattern?
19
Alternative single-n ESs

• Assuming linear trends, it seems that two ESs
  should be used to describe change in level and
  trend.
• Huitema and McKean (2000) suggested using a
  four-parameter regression model (extension of
  piecewise regn suggested by Gorman and Allison,
  1996).
• Appropriate parameterization of this model
  provides two coefficients that can be used to
  describe change in intercept and in slope from
  phase to phase

20
4-parameter model

• The model
• where
• Yt outcome score at time t
• Tt time point
• D phase (A or B)
• n1 time points in phase A

21
4-parameter model interpretation

• Coefficients represent the following
• b0 baseline intercept (i.e. Y at time 0)
• b1 baseline linear trend (slope over time)
• b2 difference in intercept predicted from
  treatment phase data from that predicted for time
  n11 from baseline phase data
• b3 difference in slope
• Thus b2 and b3 provide estimates of a treatments
  effect on level and on slope, respectively.

22
4-parameter model - interpretation
23
4-parameter model - interpretation
24
4-parameter model - interpretation
25
4-parameter model - interpretation
b2
26
4-parameter model

• Model can be estimated using OLS or
  autoregression (to correct SEs if residuals are
  autocorrelated).
• The four-parameter model can be expanded for ABAB
  designs.
• Multiple baseline designs can be thought of as
  multiple dependent, within-study AB designs.
• b2 and b3 can be calculated for each individual
  and then summarized across individuals for a
  study.

27
4-parameter model

• How does estimation of these coefficients
  function for differing true coefficient values?
• How does an omnibus test work?
• F-ratio testing addition of both predictors (with
  coefficients b2 and b3)
• How to standardize regression coefficients for
  meta-analytic synthesis?
• No procedure yet established for regular
  regression.
• Comparison with long list of other SSD ESs.