Evidence-Based Medicine and Critical Appraisal - PowerPoint PPT Presentation

View by Category
About This Presentation
Title:

Evidence-Based Medicine and Critical Appraisal

Description:

Brief introduction to other study designs including types of pharmacoeconomic ... Commentators suggest NICE use bounded rationality approach. How do we use EBM? ... – PowerPoint PPT presentation

Number of Views:293
Avg rating:3.0/5.0
Slides: 50
Provided by: benan
Learn more at: http://www.ukmi.nhs.uk
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Evidence-Based Medicine and Critical Appraisal


1
Evidence-Based Medicine and Critical Appraisal
  • Ben Rehman
  • Director
  • London Medicines Information Service
  • Northwick Park Hospital

2
Introduction
  • Evidence-Based Medicine
  • What is it?
  • Why do we need it?
  • How do we do it?
  • Critical appraisalRCTs
  • Brief introduction to other study designs
    including types of pharmacoeconomic studies
  • How should we use it?
  • At patient level?
  • As part of broader decision making and resource
    optimisation?
  • Example critical appraisal workshop

3
Evidence-Based MedicineWhat is it?
  • the conscientious, systematic, explicit, and
    judicious use of current best evidence in patient
    care
  • best research evidence replaces or supports
    current practice
  • integration of evidence and patient or population
    values
  • optimising health gain
  • clinical evidence
  • pharmacoeconomics

4
Evidence-Based MedicineWhy do we need it?
  • Health professionals face problems ensuring best
    possible patient care
  • information overload
  • inadequate traditional sources
  • opinion based approach flawed
  • disparity between experience and up-to-date
    knowledge
  • Healthcare systems have finite resources
  • But its not a complete panacea!

5
Evidence-Based MedicineHow do we do it?
  1. Form an answerable question
  2. Obtain current best evidence
  3. Critically appraise the evidence
  4. Integrate appraised evidence with clinical
    expertise and patient values
  5. Evaluate our successes and failures

6
Evidence-Based MedicineHow do we do it?
7
How do we do EBM?1. Form an answerable question
  • Should contain
  • Intervention you are interested in
  • Patient or population you are interested in
    (clearly for drug interventions you require the
    indication here too)
  • Outcomes you are interested in
  • So for the example paper, our original question
    might have been
  • Do inhaled corticosteroids improve symptoms in
    adults with acute rhinosinusitis?

8
How do we do EBM?2. Obtain current best evidence
  • Group discussion
  • What are good sources of evidence?
  • What have you used in the past and found useful?
  • What makes them good?
  • What criteria do you apply?
  • How can we find good evidence?
  • What portals and sources do we have available?

9
How do we do EBM?2. Obtain current best evidence
  • Good sources of evidence

10
How do we do EBM?2. Obtain current best evidence
  • An appropriate search strategy is important
  • We should be aware of the hierarchy of evidence
  • Systematic reviews
  • Meta-analyses
  • Randomised-controlled trials
  • Cohort studies

11
How do we do EBM?2. Obtain current best evidence
The hierarchy of evidence
  • ?Its a guideits not necessarily this simple!
    Consider
  • A large RCT vs. a meta-analysis of small RCTs?
  • A meta-analysis of observational studies?
  • ?Hierarchy becomes complicated

12
How do we do EBM?3. Critical appraisal
  • Consider the following concepts
  • Internal validity
  • Does it prove what it set out to prove?
  • External validity
  • Are the results true in the wider world?
  • Bias
  • Is there systematic error?

13
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
14
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Are this studys results valid?
  • Did the trial address a clearly focused issue?
  • Consider the intervention and the outcomes
    considered
  • Is an RCT an appropriate method to answer this
    question?
  • Was the assignment of patients to treatment
    randomised?
  • Why is it important?
  • Acceptable vs dubious methods?
  • Is the process well described?
  • Was the process concealed?

15
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Are this studys results valid?
  • Were the groups similar at the start of the
    trial?
  • Could any differences have affected outcome?
  • Was any method used to balance randomisation
    (stratification)?
  • Was follow-up sufficient?
  • Length
  • Completeness
  • Was there sufficient power?
  • Were there enough participants to minimise the
    play of chance?
  • Was an intention-to-treat analysis performed?

16
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Are the results of this individual study
    important for us?
  • Define the population studied
  • Inclusion and exclusion criteria
  • Endpoints
  • Measurement of outcome
  • Clinical relevance
  • Surrogate and composite markers
  • Validity
  • Primary vs. secondary
  • Balance of beneficial outcomes against
    side-effects

17
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Are the results of this individual study
    important for us?
  • What is the magnitude of the treatment effect?
  • How is it described?
  • Proportions of people, a measurement (mean or
    median differences), a survival curve?
  • How is it expressed?
  • With proportions see terms like relative risk
    reduction, absolute risk reduction, number needed
    to treat
  • Is what is being expressed clear?
  • Is it clinically significant?

18
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Definitions for measures of association and
    effectiveness
  • Control event rate
  • the rate at which an outcome occurs in the
    control population
  • Experimental event rate
  • the rate at which an outcome occurs in the
    experimental population
  • Absolute risk (AR)
  • Probability an individual will experience a
    specified outcome during a specified period
  • Range of 0 to 1, or expressed as a percentage
  • Relative risk (RR)
  • How many times more likely (RR gt 1) or less
    likely (RR lt 1) an event is to happen in one
    group compared with another
  • Absolute risk reduction (ARR)
  • Absolute difference in risk between experimental
    and control groups
  • Relative risk reduction (RRR)
  • Proportional reduction in risk between
    experimental and control participants in a trial

19
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Definitions for measures of association and
    effectiveness
  • Number needed to treat (NNT)
  • A measure of treatment effectiveness
  • Measures the people who need to be treated with
    an intervention over a period of time to prevent
    an additional adverse outcome or achieve an
    additional beneficial outcome
  • Reciprocal of ARR
  • Odds ratio (OR)
  • Also a measure of treatment effectiveness
  • Likelihood of an event happening in the
    experimental group compared to the control group
  • Closer the OR is to one, smaller the difference
    in effect between the experimental intervention
    and the control intervention.
  • If OR greater than 1, effects of the treatment
    are more than those of the control OR less than
    1, effects of treatment less than those of the
    control.
  • Effects being measured may be adverse (e.g. death
    or disability) or desirable (e.g. survival).
  • If events rare OR analogous to relative risk
    (RR) as event rates increase the OR and RR
    diverge

20
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Absolute vs. relative risk reduction as
    measuresan example
  • Pros and cons of each measure
  • Clinical significance?
  • Proportional difference?
  • Ease of interpretation?

21
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • RCT example4S study
  • Stable angina or myocardial infarction more than
    6 months previously
  • Serum cholesterol gt 6.2mmol/l
  • Excluded patients with arrhythmia's and heart
    failure
  • All patients given 8 weeks of dietary therapy
  • If cholesterol still raised (gt5.5) randomised to
    receive simvastatin (20mg gt 40mg) or placebo
  • Outcome death or myocardial infarction (length of
    treatment 5.4 years ) were the outcomes

22
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • RCT example4S study
  • OUTCOME
  • Dead Alive
  • Control group (placebo) n 2223 256 1967
  • Experimental group (simvastatin) n 2221
    182 2039
  • Control event rate 256 0.115 (11.5) 2223
  • Control odds of event 256 0.130 1967
  • Experimental event rate 182 0.082
    (8.2) 2221
  • Experimental odds of event 182 0.089 2039

23
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • RCT example4S study

Relative risk reduction CER EER 0.115
0.082 0.29 (29) RRR CER 0.115 Absolute
risk reduction CER EER 0.115
0.082 0.033 (3.3) ARR Number needed to treat
_1_ _1_ 30.1NNT ARR 0.033 Relative
risk EER 0.082 0.71 RR CER 0.115
Odds ratio odds of event in experimental
group 0.089 0.69OR odds of event in control
group 0.130
24
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • RCT example4S study

25
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • NNT exampleswhen should we adopt therapy?

Intervention
Outcome
NNT
26
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Statistical approaches to uncertainty
  • Why do they exist?
  • Cannot include all individuals in a population in
    trial
  • Need to quantify uncertainty
  • p values and confidence intervals are the
    measures used, but what are they and what are the
    differences?

27
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Statistical approaches to uncertainty
  • p values
  • assess the significance of the difference between
    a sample estimate and a hypothesised value
  • tell us the probability that an observed effect
    occurred by chance if in truth there is no effect
  • But doesnt quantify the size of an effect, nor
    the direction
  • plt0.05 ? commonly reject null hypothesis
  • ideally pltltlt0.05 and trial reports the actual
    value

28
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Statistical approaches to uncertainty
  • Confidence intervals
  • Range of values around the sample estimate (i.e.
    the value found in the study)
  • The range has specified probability (usually 95)
    so CI acts as hypothesis test for the range
  • Should be seen with most measures of association

29
How do we do EBM?3. Critical appraisalplacebo
controlled RCTs
  • Statistical approaches to uncertaintyplease
    remember!
  • statistical significance ? clinical significance
    what is actually being measured?
  • sample size is still important
  • larger sample less uncertainty narrower CI
    and smaller observed effect considered
    significant
  • there can still be error
  • rejecting null hypothesis when its true (type 1)
  • Not rejecting the null hypothesis when its false
    (type 2)

30
How do we do EBM?3. Critical appraisalRCTs of
therapeutic equivalence
  • Increasingly used by pharma.
  • Designed to show a new treatment is not inferior
    to standard treatment by a predefined clinically
    acceptable endpoint
  • Rely on assumptions that can be hard to validate

31
How do we do EBM?3. Critical appraisalRCTs of
therapeutic equivalence
  • Trials based on the concept that
  • new treatment is non-inferior
  • but would exhibit therapeutic efficacy if a
    placebo controlled RCT could/was performed
  • new treatment offers ancillary benefits
  • Note non-inferior so efficacy determined by
    effect against placebo not comparator

32
How do we do EBM?3. Critical appraisalRCTs of
therapeutic equivalence
  • Estimation of non-inferiority margin
  • Use statistical and clinical reasoning to
    determine what is non-inferior
  • If there are a variety of placebo controlled RCTs
    this will be similar to a meta-analysis
  • Non-inferiority margin determined as a fraction
    (f) and specifies an acceptable magnitude for
    treatment effect that must be preserved

33
How do we do EBM?3. Critical appraisalRCTs of
therapeutic equivalence
  • What to look for
  • These trials should be designed in a manner
    consistent with the historical placebo-controlled
    trials
  • The active comparator should be well established
    with predictable quantifiable, and consistent
    effects
  • What constitutes non-inferiority should be
    determined prior to initiation of trial
  • Protocol deviations and poor adherence may have a
    larger impact on quality than in conventional
    trials
  • Analysis should be by intention-to-treat AND
    on-treatment, and should also report absolute AND
    relative effects

34
How do we do EBM?3. Critical appraisalRCTs of
therapeutic equivalence
  • So when appraising these trials ask yourself
  • Was the active control previously shown to be
    effective?
  • Were study patients and outcome variables similar
    to those in the original trials that established
    the efficacy of the active control?
  • Were both regimens applied in optimal fashion?
  • Was the appropriate null hypothesis tested?
  • Was the equivalence margin specified before the
    study?
  • Was the trial large enough?
  • Was the analysis intention-to-treat AND
    on-treatment, and should also report absolute AND
    relative effects?
  • PLUS usual assessment of size and precision of
    treatment effect!

35
How do we do EBM?3. Critical appraisalother
study designs
  • Many other types of study in the hierarchy e.g.
    systematic reviews, case-control studies, cohort
    studies
  • Details of appraisal not included today but worth
    thinking about most in terms of
  • Internal validity
  • Bias
  • External validity

36
How do we do EBM?3. Critical appraisalother
study designs
  • Cohort study

37
How do we do EBM?3. Critical appraisalother
study designs
  • Case control study

38
How do we do EBM?3. Critical appraisalwhat is
economic evidence?
  • Important in decision making
  • Considers costs and consequences of alternate
    courses of action
  • Scope depends on level of decision making they
    inform
  • Costs always measured the same way (although
    scope may vary), measurement of consequence
    varies according to study type

39
How do we do EBM?3. Critical appraisalwhat is
economic evidence?
  • Cost minimisation analyses
  • analyse difference in costs where there is no
    difference in outcome
  • narrow focus
  • Cost-effectiveness analyses
  • outcome expressed in natural units (e.g.
    validated single marker)
  • again fairly narrow scopegenerally 2
    interventions for single indication

40
How do we do EBM?3. Critical appraisalwhat is
economic evidence?
  • Cost-utility analyses
  • Costs and consequences but consequence measured
    as utility
  • Utility a value of health state rather than a
    natural marker (usually QALY)
  • Can compare incongruent interventions
  • Used by NICE
  • Consider outcome in terms valuable to patients
    i.e. life expectancy and quality
  • Uses a standard measure (utility)
  • Comprehensive consideration of costs

41
How do we do EBM?3. Critical appraisalwhat is
economic evidence?
  • Cost-benefit analyses
  • Broadest possible scope!
  • Allocation of resources between difference
    sectors of economy
  • Maximising social welfare
  • Rely on assigning cost to healthcare
    interventionincredible complicated in practice,
    particular in NHS where individuals dont pay at
    point of delivery

42
How do we do EBM?3. Critical appraisalfurther
reading
  • A lot is published! But texts Ive found useful
    include
  • Straus SE, Richardson WS, Paul Glasziou, Haynes
    RB. Evidence-based Medicine How to Practice and
    Teach EBM, Third Edition. Churchill Livingstone
    Edinburgh, 2005
  • Guyatt GH, et al. Users Guides to the Medical
    Literature II. How to use an article about
    therapy or prevention. A. Are the results of the
    study valid? JAMA 1993 270 2598-2601
  • Guyatt GH, et al. Users Guides to the Medical
    Literature II. How to use an article about
    therapy or prevention. B. What are the results
    and will they help me in caring for my patients?
    JAMA 1994 271 59-63
  • Evidence Based Medicine (EBM) journal notebook
    seriesavailable online
  • Clinical evidence online. Excellent terminology
    glossary and explanations

43
How do we use EBM?4. Applying results of
research to our individual patient
  • Is our patient so different from those in the
    study that its results cannot apply?
  • Is the treatment feasible for us?
  • What are the potential benefits and harms from
    therapy?
  • Are our patients values and preferences for both
    the outcome and the side-effects we may cause?

44
How do we use EBM?4. How should clinical
evidence inform wider decision making?
  • Briefly some food for thought!
  • Want maximum benefit for minimum cost! i.e. want
    to optimise utility
  • Decision analysis tries to understand and improve
    decision making in a rational way in the presence
    of uncertainty
  • Decision making for a population will include
  • Probabilistic data
  • Value judgements
  • Clinical considerations
  • Economic considerations

45
How do we use EBM?4. How should clinical
evidence inform wider decision making?
  • More food for thought!
  • Classic decision theory suggests
  • Known alternatives with known probability
    distributions
  • Ability to maximise outcome based on this
  • But, world is large and complex and knowledge is
    imperfect!

46
How do we use EBM?4. How should clinical
evidence inform wider decision making?
  • Some more food for thought
  • Bounded rationality approach relaxes assumption
    of perfect knowledge
  • Choose the option or course of action based on
    most important current needs
  • Commentators suggest NICE use bounded rationality
    approach

47
Example critical appraisal of RCT workshop
  • The CASP tool
  • available at http//www.phru.nhs.uk/Pages/PHD/reso
    urces.htm
  • Divide up into groups and use the tool to
    appraise the trial
  • Treating acute rhinosinusitis Comparing
    efficacy and safety of mometasone furoate nasal
    spray, amoxicillin, and placebo

48
Example critical appraisal of RCT workshop
  • Feedback from groups about the trial

49
RCT workshop
  • And finally what do others think about this
    study?
About PowerShow.com