Critically Appraising Research A Consumers Perspective Gary S. Gronseth, M.D. Department of Neurolog

1
Critically Appraising ResearchA Consumers
PerspectiveGary S. Gronseth, M.D.Department
of NeurologyUniversity of KansasOctober 12,
2006
2
Clinical
Outcomes
Basic
What causes disease and defining approaches to
diagnosis and treatment?
What works and what does it cost in real-world
settings?
How can disease be prevented or treated under
controlled settings?
3
(No Transcript)
4
Steroids
5
(No Transcript)
6

• The use of steroids for Bells palsy has become
  the standard of care in the community.

7

• The consequences of disfiguring facial weakness
  are so devastating that the use of steroids is
  mandatory.

8

• Fallacious
• Irrelevant
• Rhetoric
• Psychological appeal
• Emotion-Driven
• Persuasion

9
Common fallacious arguments in medicine

• Popularity
• Begging the question
• Irrelevant Outcomes

10
Irrelevant Outcomes
11

• Fallacious
• Irrelevant
• Rhetoric
• Psychological appeal
• Emotion-Driven
• Persuasion

12
Decison
13
Decision
14
Decision
15
Deductive ArgumentPrinciples

• Steroids
• Reduce inflammation
• Reduce compression within temporal bone

Good for hypothesis generation Often not
Convincing When convincing, no controversy
16
Decision
17
Decision
18
Decision
19
Decision
20
Anecdotal argument

• The last patient I saw with Bells palsy who
  didnt get steroids never got better.

21
ExperienceCollections of Anecdotes

• My Experience

22
(No Transcript)
23
Expert opinion

• From a practical view point, treatment with
  steroids is appropriate for the management of
  Bells Palsy.
• Bells Palsy remains without a proven
  efficacious treatment.

Stankiewicz 1987
May 1975
24
Limitations of Experience
More likely to remember anecdotes that
are Recent Extreme Support our preconceptions
25
EvidenceCollections of Recorded Anecdotes

• My Experience

26
Decision
27
(No Transcript)
28
The results of this study demonstrate no
statistically significant beneficial effect of
steroid therapy upon recovery from Bells
Palsy.P 0.32
May et. al. 1975

• Relative Rate
• Complete Recovery
• 0.92 (0.60 to 1.4)

Insufficiently powered to exclude a benefit
29
The control group included mainly those patients
who had refused to take prednisilone, because
they were afraid of developing complications, and
those with a relative contraindication to
steroids (e.g. moderate hypertension)
Shafshak et al 1994
30
The Literature is full of

• Fallacious arguments
• Contradictory expert opinions
• Misinterpretations of errors of chance
• Studies with a high risk of bias

31
Find and analyze all of the evidence in a
transparent and systematic fashion
or find someone who already has.
32
AAN clinical practice guidelines Above the fray
Franklin, Gary M. MD, MPH Zahn, Catherine A. MD
Neurology Volume 59(7) 8 October 2002  pp 975-976

• Avoid fallacious arguments
• Involve experts but dont allow their conclusions
  to go beyond the evidence
• Find all relevant studies
• Systematically assess and grade the quality of
  the studies to avoid spin

33
Clinical
Outcomes
Basic
What causes disease and defining approaches to
diagnosis and treatment?
What works and what does it cost in real-world
settings?
How can disease be prevented or treated under
controlled settings?
Patient-based
Pre-clinical
Organizational- based
Patient Population-based
Translation 2 Humans to Bedside/Curbside
Translation 1 Bench to Humans
34
Practice Guideline

• Systematic and Transparent

Many Specialty Societies
Cochrane collaboration
AAN
35
Consensus vs. Evidence
?
36
Consensus-Based Evidence-Based
?
37
Evidence-based Argument

• Question
• Evidence
• Conclusion
• Recommendation

38
Do steroids work for Bells palsy?
For patients with acute Bells palsy, do oral
steroids given within the first week improve
facial functional outcomes?
39
Do steroids improve facial functional recovery in
patients with for Bells palsy?
Patients Acute Bells Palsy Intervention Ora
l Steroids Given within 1 week Outcomes F
acial Function
40
Should we do this?

• For patients with (suspected) Bells palsy
• Therapeutic does early use of oral steroids
  improve facial outcomes?

41
Should we do this?

• For patients with (suspected) Bells palsy
• Therapeutic does early use of oral steroids
  improve facial outcomes?
• Prognostic does facial CMAP testing identify
  patients at increased risk for poor
  outcomes?

42
Should we do this?

• For patients with (suspected) Bells palsy
• Therapeutic does early use of oral steroids
  improve facial outcomes?
• Prognostic does facial CMAP testing identify
  patients at increased risk for poor
  outcomes?
• Screening does head MRI identify patients
  with treatable etiologies?

43
The Answer

• Qualitative
• Yes, (No, Maybe, Cant tell) for patients with
  condition A, intervention B is effective
  (useful, superior) to improve outcome C.
• QuantitativeHow much does the intervention
  improve outcomes
• CertaintyHow confident are we in the answer

44
Evidence-based Argument

• Question
• Evidence
• Conclusion
• Recommendation

45
(No Transcript)
46
Evidence

• Find all relevant studies
• Assess
• Risk of Bias
• Results
• Random Error
• Generalizability

47
Find the EvidenceThe Literature
48
Literature Search
Apriori inclusion criteria.
49
Literature SearchSteroids for Bells Palsy
230
Studies of outcomes in Bells Palsy patients some
getting steroids, some not getting steroids
8
50
Steroid Studies

• Yr Author
• 94 Shafshak
• 93 Austin
• 87 Abraham
• 82 Brown
• 78 Wolf
• 76 May
• 72 Adour
• 54 Taverner

51
How good is the Evidence?
52
Grading the Evidence
The quality of evidence is always judged as it
pertains to the clinical question.
53
AANs Evidence Classification Schemes

• Therapeutic
• Diagnostic/Prognostic Accuracy
• Screening

54
AANClassification of evidence

• Class
• I
• II
• III
• IV

Risk of Bias Low Moderately Low Moderately
High High
55
Study
56
AAN Grading the evidence

• Class
• I
• II
• III
• IV

Controlled

57
No Comparison Group
Poor Good
St
58
No Comparison Group

• Natural History
• Placebo Response
• Regression to the Mean

59
Confounding
Potential known and unknown confounding
differences between treated untreated group
St
-St
60
ControlledMatching
Potential unknown confounding differences between
treated untreated group
St
M
-St
61
ControlledRandomized Trial
No confounding differences between treated
untreated group
St
R
-St
...unless you are unlucky.
Or
62
ControlledConcealed Allocation
Is the study truly randomized
St
R
-St
63
Cross OversLost Randomization
St
R
-St
64
Cross OversIntent-to-treat
St
R
-St
65
Losses to Follow UpLost Randomization
St
R
-St
66
Losses to Follow UpIntent to Treat
St
R
-St
67
AAN Grading the evidence

• Class
• I
• II
• III
• IV

Controlled randomized matched comparative unc
ontrolled

68
AAN Grading the evidence

• Class
• I
• II
• III
• IV

Controlled randomized matched comparative unc
ontrolled
Masking
69
Masking
St
-St
70
Masking

• Patient
• Treating Providers
• Outcome assessors

71
AAN Grading the evidence

• Class
• I
• II
• III
• IV

Controlled randomized matched comparative unc
ontrolled
Masking masked outcome masked
outcome independent non-independent
72
Steroid Studies

• Author Random. Comp FU Masked
• May R 100 Y
• Taverner R 100 Y
• Brown R 100 Y
• Austin R 71 Y
• Shafshak NR 100 Y
• Wolf R 100 N
• Adour NR 85 N
• Abraham NR 100 N

• Class
• I
• I
• II
• II
• III
• IV
• IV
• IV

73
What does the evidence show?

• The studys results
• (Not the authors conclusions)

74
Effect of SteroidsRate Difference

Outcome


Treatment

Poor

Good

Steroids

15

85

No Steroids

33

67


75
Categorical Outcomes are the most clinically
relevant
76
Measures of Effect

• Therapeutic questions
• Rate Difference
• Relative Rates
• Odds Ratios
• Diagnostic questions
• Sensitivity/Specificity
• Likelihood Ratios
• Screening questions
• Yields

77
Rate Difference Good RecoverySteroids vs No
Steroids
60
I
II
III
IV
40
20
Rate Difference
0
-20
-40
May
Taverner
Brown
Austin
Shafshak
Wolf
Adour
78
Evidence-based Argument

• Question
• Evidence
• Conclusion
• Recommendation

79
Synthesis
60
I
II
III
IV
40
20
Rate Difference
0
-20
-40
May
Taverner
Brown
Austin
Shafshak
Wolf
Adour
80
Decrease Random Error (Include all studies)
Decrease Risk of Bias (Exclude studies with a
higher risk of Bias)
81
Cumulative Meta-analysis

• Do not Consider Class IV evidence
• Iteratively combine results of studies with the
  least risk of bias until you arrive at a
  sufficiently precise measure of effect or you run
  out of studies
• Grade the strength of the conclusion based on the
  most bias prone study used to determine the effect

82
Combine Class I evidence
60
I
40
20
Rate Difference
0
-20
-40
Pooled
May
Taverner
83
Hypothetical
60
I
40
20
Rate Difference
0
For patients with Bells palsy steroids are
established to increase the likelihood of good
facial functional recovery by about 12 percent.
-20
-40
Pooled
May
Taverner
84
Combine Class I and Class II evidence
60
I
II
40
20
Rate Difference
0
For patients with Bells palsy steroids probably
increase the likelihood of good facial functional
recovery by about 12 percent.
-20
-40
Pooled
May
Taverner
Brown
Austin
85
Hypothetical
60
I
II
III
40
20
Rate Difference
0
For patients with Bells palsy there is
insufficient evidence to conclude that steroids
increase or decrease the probability of good
facial functional recovery.
-20
-40
Pooled
May
Taverner
Brown
Austin
Shafshak
86
Conclusions

• For patients with Bells palsy...
• Based on consistent results from two class I and
  two class II studies, steroids probably increase
  the likelihood of good facial functional recovery
  by about 12 percent.

87
Evidence-based Argument

• Question
• Evidence
• Conclusion
• Recommendation

88
Relative Value
Benefits
Risks
89
Relative Value
Patient Preferences Consensus
Value of Steroids is Large
Good Facial Function Permanent
Steroid Side Effects Temporary
90
Strength of Recommendation
Strength of Evidence 2 Class I 2 Class II 2
Class III Value of Benefit/Risk Large Moderate Small Too
close to call
Level of recommend. A. Should be offered B.
Should be considered C. May be considered U. No
recommendation
91
Strength of Recommendation
Strength of Evidence 2 Class I 2 Class II 2
Class III Value of Benefit/Risk Large Moderate Small Too
close to call
Level of recommend. A. Should be offered B.
Should be considered C. May be considered U. No
recommendation
92
Recommendation

• For patients with Bells palsy...
• oral steroids should be considered to increase
  the probability of good facial functional
  recovery. (Level B)

93
Level A Recommendation
94
Level C Recommendation
95
(No Transcript)