Alternative methods for handling missing data: Impact on measured effectiveness of a riskstatus indi

1
Alternative methods for handling missing data
Impact on measured effectiveness of a
risk-status indicator

• E. Michael Foster Penn State
• Grace Yan Fang Penn State
• Damon Jones Vanderbilt
• Kenneth Dodge Duke
• June 1, 2001

2
(No Transcript)
3
Does prevention pay?

• At the very least, we must
• Deliver effective interventions
• Identify high-risk children and youth at optimal
  time
• Ex aggression

4
Can we identify at-risk children early and
reliably?

• Recent debate
• Added challenge attrition from longitudinal
  studies

5
How does attrition influence effectiveness of
high-risk screen?

• Examine data from the Fast Track intervention
• Review taxonomy of missing data problems
• Consider performance of risk measure under
  alternative methodologies
• ML
• MI
• MIenhanced
• Econometric methods

6
Outline

• 1. Describe data
• 2. Taxonomy of missing data
• 3. Possible solutions
• 4. Results
• 5. Future Research

7
1. Describe FT Project

• Multi-site trial of a multi-component
  intervention
• Durham, NC
• Seattle, WA
• Rural PA
• Nashville, TN
• Study participants attending schools in poor
  neighborhoods

8
1. Describe FT Project (cont)

• Analyses presented here include treatment and
  control groups identified in Kindergarten
• Focus on normative sample high-risk controls
• NO INTERVENTION CHILDREN

9
Data Collection

• Annual interviews with child and parents
• Includes reports of service use
• Teacher reports of behavior
• School record reviews
• Peer reports

10
FT Screening measure

• Step 1 Teachers rate children using modified
  TOCA-R
• Step 2 Parents of selected children were
  contacted to complete 24 items from CBCL
• Step 3 Measures combined to identify top 10

11
What do we know about attrition?

• Roughly 90 response rate by 6th grade.
• Some variation by respondent
• Non-response does vary by high-risk status.

12
Structure of Analyses

• Service use
• Reports of lifetime use of services at grade 6
• Review of school records
• Self-reports of delinquency at grades 5, 6 and 7

13
Service use outcomes

• IEP
• Repeat a grade
• Medications
• Specialized mental health
• General medical
• Overnight mental health
• School counselor
• Police contact

14
2. Taxonomy of Missing Data Problems

• Missing completely at random
• Missing at random
• Missing not at random

15
Possible Solutions

• Various inconvenient or downright misleading
  practices
• Listwise or pairwise deletion
• Substitution of means
• Regression predictions

16
Possible Solutions (cont)

• Two good alternatives
• Maximum likelihood estimation
• Multiple imputation
• Both assume MAR
• Differences in implementation

17
Implementing MI

• Creates a series of datasets.
• Missing values filled-in in a desirable way.
• Filled-in data reflects value of observed
  variables
• Independent across imputations

18
Implementing MI (cont)

• Steps
• Create series of imputed datasets
• Software NORM, PROCMI (SAS)
• Conduct analyses for each dataset
• Calculate parameter estimates as average across
  files
• Calculate variance as weighted sum of
• Average variance for each dataset
• Variation in parameter estimate across datasets

19
BIG DIFFERENCE imputation and analysis models
can differ.

• In the context of FT, we may lack child reports
  of TYD, but we have
• Peer nominations
• Parent daily reports
• Teacher reports of problem behaviors and school
  adjustment

20
4. Analyses of Fast Track Data
21
Risk status measure is potent.
22
(No Transcript)
23
Results

• Results
• Risk status is very potent.
• ML and MI results are very similar.
• Enriched strategy makes little difference.
• Practical problems
• Dealing with item v. unit non-response
• Distributional problems

24
5. What Next? Relaxing the MAR Assumption

• Allow unobserved determinants of response to
  correlate with unobserved determinants of outcome
• Difficult in a panel data model
• Lillard and Panis (1998)
• www.applied-ml.com

25
Further Reading

• Jones, Damon et al. 2001. Early Identification
  of Children At Risk for Costly Mental Health
  Service Use Working paper.
• Incremental value of parental reports (v. teacher
  only)
• Information loss created by categorizing
  individuals
• Variation by site and gender
• www.personal.psu.edu/emf10/

26
Further Reading

• Foster, E. Michael and Leonard Bickman. (1996)
  "An Evaluators Guide to Detecting Attrition
  Problems. Evaluation Review, 20(6) 695-723.

27
Further Reading

• Collins, L.M., Schafer, J.L., Kam, C. (under
  review). A comparison of inclusive and
  restrictive strategies in modern missing data
  procedures. Paper submitted to Psychological
  Methods
• Lillard, L., Panis, C. (1998). Panel Attrition
  from the Panel Study of Income Dynamics. Journal
  of Human Resources, 33(2), 437-457.