Missing Inaction: Why Do So Many People Ignore Missing Data in RCTs

1
Missing InactionWhy Do So Many People Ignore
Missing Data in RCTs?

• Temple-Merck Conference 17-Oct-08
• Janet Turk WittesStatistics Collaborative

2
Extent of missing primary outcome data

• Cardiovascular outcome trial 1-2
• Cancer progression-free survival 2-20
• Short-term blood pressure trial 5-10
• 12 week pain trial 20-40
• 12 week antipsychotic drug 30-50
• 12 week anti-infective 20-50
• Source informal experience

3
What others assume
4
What others assume
5
What others assume
6
What we fear (and assume)
7
What we fear (and assume)

• What we have left is different from what was
  there at first
• We cant characterize what is missing
• What is missing differs by group

8
Evidence of inaction hard to ferret out extent
and timing
Time
Missing values imputed with LOCF.
9
Rarely apparent in survival curves
10
Time to event
11
What I am not going to talk about

• MCAR, MAR, not MAR
• Ignorable/non-ignorable
• The effect of missing data on inference
• In sample surveys
• In experiments
• In randomized clinical trials
• Detailed methods of dealing with missing data

12
What I will discuss

• Once over lightly of the methods at hand
• Why others dont care about missing values
• Why our protocols encourage missing data
• What we can do to prevent missing data
• Even though prevention is boring

13
Underlying principle

• Our method of imputation shouldnt give us better
  results than what we would have seen from the
  complete cases

14
The cards in our deck binary outcomes

• Just ignore the missing observations
• Impute missing on basis of
• Proportion in own group
• Best case all pbo fail all rx succeed
• Worst case all pbo success all rx fail
• Proportion in placebo group (not unreasonable
  guess)
• Proportion in opposite group (reasonable worst
  case)
• Multiple imputation

15
Problems with the binary cards

• Too many degrees of freedom
• Some methods overstate effect
• Some methods understate effect
• Some methods are unreasonably pessimistic

16
Loss of 3 lines of vision

• Two groups treated and control
• 120 eyes per group (one per person)
• 40 in placebo 20 in treated
• Look at relative risk (
• Missing equal in both groups

17
Loss of 3 lines of vision impute own group
Imputation leads to increased sample size hence
false precision.
18
Binary example loss of 3 lines of vision
1.13
0.31
100 of missing behave the same way not
reasonable.
19
Binary example loss of 3 lines of vision
0.6
Not unreasonable to assume that untreated act
like placebo. Best reasonable case
20
Binary example loss of 3 lines of vision
0.67
Opposite arm is worst reasonable case.
21
What do binary cards do for us

• Bad
• Too many degrees of freedom
• Some methods overstate effect
• Some methods understate effect
• Good
• Sensible cases provide bounds
• Multiple imputation (if we have a good model)

22
The cards in our deck continuous outcomes

• Just ignore the missing observations
• Impute missing on basis of mean in
• Own group
• Combined group
• Placebo group
• Opposite group (worst reasonable case)
• Last Observation Carried Forward
• Baseline Observation Carried Forward
• Last rank carried forward
• Multiple imputation

23
The cards in our deck longitudinal

• Just ignore the missing observations
• Impute missing by carrying forward
• Last observation
• Baseline observation
• Own group trajectory
• Placebo trajectory
• Opposite group trajectory
• Last rank
• Longitudinal model
• Multiple imputation
• Proschan et al (2001)., J Stat Planning 96 155
• Obrien, Zhang, Bailey (2005). Stat Med 2434

24
Longitudinal outcome

• Pain at Day 4
• 325 patients per group
• 250 per group completed
• 7 point scale
• Placebo Treated
• Baseline 5.0 5.0

25
Effect size and p-values
26
Effect size and p-values
27
Continuous outcome

• Placebo Treated
• Baseline 5.0 5.0
• Day 1 3.7 3.2

28
Effect size and p-values
29
The cards in our deck survival

• Censor when missing
• Assume missing have event
• At same proportion as own, placebo, or opposite
  group
• Need to decide when the imputed event occurs
• At time of censoring
• At rate in assigned group

30
Message

• Analyses produce very different results
• Can affect
• Direction of effect
• Effect size

31
Why people dont care about missing data in
outcome trials

• In outcome trials
• we can censor doesnt matter what happens after
  people stop drug

32
Why people dont care about missing data

• Outcome trials are different from symptom trials
• Who cares about those who dont take drug?
• We know the drug wont work if you dont take
  it
• I am not interested in what happens after
  people stop.
• No evidence that the two groups differ in
  Prmissing
• We are interested in what we observe complete
  cases
• Too hard/expensive to bring back those who stop
  med

33
Informed consent documents unclear

• Participation in this study is entirely
  voluntary. Your treatment and your doctors
  attitude toward you will not be affected should
  you decide not to participate in this study
• You will be asked to return for follow-up visits
  and to provide follow-up information.
• If you agree to participate, you may withdraw
  from the study at any time without affecting any
  benefits to which you would otherwise be
  entitled.

34
Permissive protocols encourage missing data

• Drop-outs will not be replaced
• Suggests that it would be ok to replace them
• Suggests that analysis will ignore them
• Expect 10 drop out, therefore increase sample
  size by 10
• The primary analysis will use the
  intent-to-treat pop
• The ITT pop is defined as all those randomized
  who
• The ITT pop is defined as the evaluable group

35
Language about withdrawal an outcome trial

• The reason that a subject discontinues from the
  study will be recorded in the Case Report Form.
• A discontinuation occurs when an enrolled subject
  ceases participation in the study, regardless of
  the circumstances, prior to completion of the
  protocol.
• The final evaluation required by the protocol
  will be performed at the time of study
  discontinuation.

36
Outcome continuous measure at week 48

• Subjects must be withdrawn from the study (i.e.,
  from any further study medication or study
  procedure) for the following reasons
• At their own or their legally authorized
  representatives request
• If, in the investigators opinion, continuation
  in the study would be detrimental to the
  subject's well-being
• Occurrence of an intolerable treatment-emergent
  adverse event as determined by the investigator
  and/or the subject
• Failure of the subject to return to the study
  site for scheduled visits
• Persistent noncompliance
• Pregnancy

37
Prevention of missing values

• Education
• What is the effect of various analytic methods
• Why is missing important
• Revise informed consent forms
• Make protocols less permissive

38
Education of investigators

• Important to explain to investigators
• need for follow-up
• consequences to the study of failure to follow-up

39
Improved informed consent document

• Participation in this study is entirely
  voluntary. Your treatment and your doctors
  attitude toward you will not be affected should
  you decided not to participate in this study
• If you agree to participate, you may withdraw
  from the study at any time without affecting any
  benefits to which you would otherwise be
  entitled.
• You will be asked to return for follow-up visits
  and to provide follow-up information even if you
  are not taking study medication.

40
Protocols

• Be vigilant about permissive language
• Distinguish between
• Stopping meds
• Stopping active visits
• Withdrawing consent to be followed passively
• Explain to investigators the importance of
  follow-up
• (even for those who stop study medication)

41
Typical language about withdrawal in protocols

• The reason that a subject discontinues from the
  study medication will be recorded in the Case
  Report Form.
• A discontinuation from the study occurs when an
  enrolled subject ceases participation a
  participant in the study dies, is permanently
  lost to follow-up, or withdraws consent,
  regardless of the circumstances, prior to
  completion of the protocol.
• An final evaluation required by the protocol will
  be performed at the time of study discontinuation
  of study medication.

42
But, if there are missing data

• Choose analytic methods that
• Dont add false precision
• Are reasonably conservative
• Are interpretable
• Recognize that need for big increase in sample
  size
• Phil Lavoris rule 1 missing observation needs
  three additional
• So, if you expect 10 missing, inflate sample
  size by 1/3

43
Conclusion homework assignment

• Look at all your protocols
• Look at all your model informed consent forms
• Prevent permissive language in the future