Critical Appraisal of Health Care Intervention Studies

1
Critical Appraisal of Health Care Intervention
Studies

• Dr. Maureen Markle-Reid, RN, MScN, PhD
• Associate Professor and Acting Assistant Dean
  (Research), School of Nursing, Associate Member,
  Clinical Epidemiology and Biostatistics
• Nursing Graduate Seminars
• October 4, 2011

2
Objectives

• Define evidence-informed decision-making and
  describe its importance to nursing practice
• Identify the parts of a relevant, answerable
  question
• Describe the different types of questions
  relevant to nursing practice
• Define the term critical appraisal and discuss
  its relevance to nursing practice, policy and
  research
• Identify criteria appraisal criteria for health
  care intervention studies
• Develop skill in applying the criteria for
  critical appraisal of an intervention study to
  determine the quality and applicability of the
  research.

3
What is evidence-informed decision-making?

• The conscientious, explicit and judicious use of
  the current best evidence in making decisions
  about the care of individual patients
• (Sackett et al, 1996)

4
A Model for Evidence-Based Clinical Decisions
Health care resources
Patient preferences and actions
Clinical Expertise
Clinical Decision
Research evidence
Clinical state, setting, and circumstances
Haynes, R.B., Devereaux, P.J., Guyatt, G.H.
(2002). Clinical expertise in the era of
evidence-based medicine and patient choice. ACP J
Club, 136, A11-14
5
Why is evidence-based practice important?

• Demonstrate that nursing actions and decisions
  are clinically appropriate and result in positive
  outcomes for clients, their families and health
  care system as a whole
• Demonstrate professional accountability to
  clients
• Provide evidence for the cost-effectiveness of
  nursing care

6
(No Transcript)
7
Steps to Evidence-Informed Decision-Making

• Define Formulate a focused answerable question
  from the practice situation based on your
  information needs
• Search Efficiently search for research
• Appraise Critically and efficiently appraise
  the research sources
• Synthesize Interpret/form recommendations for
  practice based on the literature
• Apply Apply the results to the
  client/population
• Implement Decide whether (and plan how) to
  implement the adapted evidence into practice
• Evaluate Evaluate the effectiveness of
  implementation efforts

8
Formulate a Focused Answerable Question

• P opulation / Situation
• I ntervention / Exposure
• C ounter intervention
• O utcome
• T imeframe

9
Population/Situation

• Client group or clinical scenario of interest
• Single patient or group of patients with a
  particular condition or health care problem,
  e.g.,
• A person with a health condition
• People with hypertension (a group of people with
  a particular condition
• Primary health care for the elderly (an aspect of
  health care delivery)

10
Intervention/Exposure

• Interventions can be
• Therapeutic
• Preventive
• Diagnostic
• Organizational
• If exploring the meaning of a phenomena, the
  question may involve a situation rather than an
  intervention

11
Counter Intervention

• Clinical decisions involve choosing between
  alternative courses of action (or no action)

12
Outcome

• What is the outcome, or the effect we are hoping
  to achieve by the using the intervention?
• May be more than one outcome that is important to
  the question

13
Time Frame

• What is the period of time over which the
  question occurs?
• What is the optimal time to measure a change in
  the outcome(s)?

14
Step 1Formulate a focused answerable question

• Formulating Answerable Clinical Questions (Centre
  for Evidence-Based Medicine, Mount Sinai
  Hospital)
• The Well-Built Clinical Question (Duke University
  Medical Center Library and Health Sciences
  Library, UNC-Chapel Hill)
• Formulating Patient Centered Questions
  (University Library, University of Illinois at
  Chicago)
• Asking Focused Questions (Centre for
  Evidence-Based Medicine, Oxford)

15
Types of questions relevant to nursing practice

• Therapy/Health care interventions What is the
  effectiveness of different interventions?
• Causation and harm What might be causing
  disease/ill health/adverse effects?
• Diagnosis or assessment Does a diagnostic test
  differentiate between people with and without a
  condition?
• Prognosis What are potential future outcomes of
  a condition?
• Economic evaluation What is the
  cost-effectiveness of different interventions?
• Meaning Describing, exploring and explaining
  aspects of health and illness.

16
Different clinical questions require evidence
from different research designs
Type of Question Suggested Best Type of Study Design
Therapy/Health care intervention RCTgtcohortgtcase controlgtcase series
Diagnosis Prospective, blind comparison to a gold standard
Prognosis Cohort studygtcase controlgtcase series
Harm (Causation or etiology) RCTgtcohortgtcase controlgtcase series
Economic evaluation Economic analysis
Meaning of illness Qualitative
17
What type of research design will you use?
All studies
Q1
Analytic (PICO or PECO)
Descriptive (PO)
Theoretical (PS)
Q2
Experimental
Observational analytic
Qualitative
Survey (cross sectional)
Q3
(Randomised) Parallel group
Cohort study
(Randomised) crossover
Cross-sectional (analytic)
Case-control study
Glasziou, P., Heneghan, C. (2009). A spotters
guide to study designs. EBN Notebook, 12, 71-72
18
Step 2 Collect the best evidence Hierarchy of
Pre-Processed Evidence
EXAMPLES SYSTEMS Computerized decision support
SUMMARIES Evidence-Based Guidelines
Evidence-Based Texts   SYNOPSES OF
SYNTHESES DARE Evidence-Based Abstract
Journals   SYNTHESES (Systematic Reviews)
Cochrane Database of Systematic Reviews SYNOPSES
OF SINGLE STUDIES Evidence-Based Abstract
Journals   SINGLE STUDIES Clinical Queries
Adapted from Haynes, R.B. (2007). Of studies,
summaries, synopses, and systems The 5S
evolution of information services for
evidence-based healthcare decisions.
Evidence-Based Nursing, 10, 6-7
19
Step 3 Critically appraise the literature for
validity and applicability

• What is critical appraisal?
• Critical appraisal is the process of assessing
  and interpreting evidence by systematically
  considering its validity, results and relevance
  to an individual's work.

20
Relevance of Critical Appraisal to Nursing
Practice, Policy and Research

• Use the literature more effectively in answering
  clinical questions to guide clinical practice
• Distinguish stronger evidence from weaker
  evidence identify high quality research
• Identify the methodological strengths and
  limitations, results and relevance of the studies
  in answering a clinical question

21
Three Steps in Using an Article from the Health
Care Literature

• Are the results valid?
• What are the results?
• Are the results applicable (and useful) to my
  client care population?
• DiCenso, A., Guyatt, G., Ciliska, D. (2005).
  Evidence-based nursing A guide to clinical
  practice

22
Are the results valid?

• Are the study methods sufficiently rigorous to
  ensure that the study results represent an
  unbiased estimate of the true effect?
• OR
• Are the study methods sufficiently biased to lead
  to a false conclusion?

23
Are the results valid?

• Final assessment of validity is never a yes/no
  decision
• Validity as a continuum ranging from strong
  studies to weak studies
• Evaluation of the validity of a study involves
  some subjectivity

24
What are the results?

• Size and precision of the estimate of effect

25
Are the results applicable (and useful) to my
client care population?

• Can you apply the results to patients in your
  clinical setting?
• Were all important outcomes considered?
• Are the likely intervention benefits worth the
  potential harm and costs?

26
Applying appropriate criteria

• User Guides to Evidence-Based Practice
• 1993-2000 Evidence based medicine working group
  User guides to the Medical Literature in JAMA
• 2002 Guyatt et al. User guides to the medical
  literature A manual for evidence based clinical
  practice
• 2005 DiCenso, Guyatt, Ciliska.
  Evidence-based nursing A guide to clinical
  practice

27
Applying appropriate criteria

• Critical Appraisal Forms
• Critical Appraisal of Articles on CAUSATION
• Critical Appraisal of Articles on
  THERAPY/INTERVENTIONS
• Critical Appraisal of SYSTEMATIC REVIEWS
• Critical Appraisal of Articles on PREVALENCE AND
  INCIDENCE
• Critical Appraisal of QUALITATIVE RESEARCH
• Critical Appraisal of Articles on PROGNOSIS
  http//www.cche.net/usersguides/prognosis.asp
• Critical Appraisal of GUIDELINES AGREE
  Appraisal of Guidelines Research and Evaluation
  Instrument http//www.agreecollaboration.org/pdf/
  agreeinstrumentfinal.pdf

28
Example The Effectiveness of a Nurse-Led
Interprofessional Team Approach to Fall
Prevention in Older Home Care Clients at Risk of
Falling

• Markle-Reid, M., Browne, G., Gafni, A.,
  Roberts, J., Weir, R., Thabane, L., Miles, M.,
  Vaitonis, V., Hecimovich, C., Baxter, P.,
  Henderson, S. (2010). The effects and costs of a
  multifactorial and interdisciplinary team
  approach to fall prevention for older home care
  clients at risk for falling A randomized
  controlled trial. Canadian Journal on Aging,
  29(1), 139-161

29
Critical Appraisal Criteria for Health Care
Intervention Studies

• Screening Questions
• Did the study ask a clearly focused question?
• Consider if the question is focused in terms
  of
• The population studied
• The intervention given
• The outcomes considered

30
Research Question

• What is the effectiveness of a 6-month nurse-led
  interprofessional team approach to fall
  prevention compared with usual home care services
  in older home care clients at risk of falling
  with respect to the number of falls and fall risk
  factors (slip or trip, health-related quality of
  life and function, depressive symptoms,
  nutritional status, gait and balance, cognitive
  function, fear of falling)?
• P Frail older adults using home care services at
  risk for falling
• I Nurse-led interprofessional team approach to
  fall prevention
• C Usual home care services
• O Number of falls, slip or trip, health-related
  quality of life and function,
  depressive symptoms, nutritional status, gait and
  balance, cognitive function, fear of falling
• T 6 months

31
Critical Appraisal Criteria for Health Care
Intervention Studies

• Screening Questions
• Was this a randomized controlled trial (RCT) and
  was it appropriately so?
• Consider
• Why this study was carried out as an RCT
• If this was the right research approach for the
  question being asked

32
Critical Appraisal Criteria for Health Care
Intervention Studies

• Is it worth continuing?

33
Critical Appraisal Criteria for Health Care
Intervention Studies

• How were participants allocated to intervention
  and control groups?
• Consider
• How participants were allocated to intervention
  and control groups. Was the process truly random?
• Whether the method of allocation was described
• How the randomization schedule was generated
• Were the groups comparable at baseline on
  characteristics that might explain the outcome?

34
Follow-up Measures
Follow-up Measures
35
Strengths

• Random allocation of participants to groups
  ensures that groups are similar in all respects
  except exposure to the outcome
• Prospective design ensures that exposure to the
  intervention precedes the development of the
  outcome
• Greater likelihood that participants, health care
  providers, and outcome assessors can be blinded.

36
Limitations

• May not be ethical or feasible
• Volunteer bias limits generalizability
• Problems associated with a longitudinal design
• Time
• Expense
• Dropouts
• Final results may not be available for several
  years.

37
Critical Appraisal Criteria for Health Care
Intervention Studies

• Were participants, staff and study personnel
  blind to participants study group?
• Consider
• The fact that blinding is not always possible
• If every effort was made to achieve blinding
• If you think this matters to the study
• The fact that we are looking for observer bias

38
Critical Appraisal Criteria for Health Care
Intervention Studies

• Was follow-up complete?
• Consider
• If any intervention-group participants got a
  control group option or vice versa
• If all participants were followed up in each
  study group
• Was there loss to follow-up bias?
• If all of the participants outcomes were
  analysed by the groups to which they were
  originally allocated (intention to treat analysis)

39
Study Flow
40
Critical Appraisal Criteria for Health Care
Intervention Studies

• Were participants in both groups followed up and
  data collected in the same way?
• Consider
• Was data collected in the same way and at the
  same time interval for both groups?

41
Data Collection

• Multiple sources of data
• In-person interviews
• CCAC data
• RAI-HC data
• Data from service provider agencies
• Measurement of Clinical Outcomes Baseline and
  6-months

42
Outcomes

• Effects
• Frequency and Context of Falls (Falls
  Surveillance Report)
• Functional Health Status and Quality of Life
  (SF-36 Health Survey)
• Cognitive Status (SMMSE)
• Depressive Symptoms (CES-D)
• Gait and Balance (POMA)
• Nutritional Status (SCREEN II)
• Environmental Safety (HOME FAST)
• Perceived Self-Efficacy (MFES)
• Costs Health Services Utilization (HSSU and CCAC
  Utilization Data)

43
Critical Appraisal Criteria for Health Care
Intervention Studies

• How are the results presented and what is the
  main result?
• Consider
• What are the results?
• How large this size of result and how meaningful
  it is
• How you would sum up the bottom-line result of
  the trial in a few sentences

44
Results

• At 6 months, there was no difference in the mean
  number of falls between groups
• Subgroup analyses showed that the intervention
  was effective in reducing falls in men, 75-84
  years of age, with a fear of falling, or a
  negative history of falls
• Greater reduction in number of slips and trips in
  the intervention group
• Greater improvement in role functioning related
  to emotional health in the intervention group
• No additional cost from a societal perspective,
  e.g. both interventions cost the same.

45
Critical Appraisal Criteria for Health Care
Intervention Studies

• How precise are these results?
• Consider
• If the result is precise enough to make a
  decision
• If a confidence interval were reported. Would
  your decision about whether or not to use this
  intervention be the same at the upper confidence
  limit as at the lower confidence limit?

46
Critical Appraisal Criteria for Health Care
Intervention Studies

• Were all important outcomes considered so the
  results can be applied?
• Consider whether
• How the participants could be different from your
  population in ways that would produce different
  results
• Your local setting differs from that of the trial
• Is the intervention feasible in your setting?
• Consider outcomes from the point of view of the
• Individual
• Policy maker and professional
• Family caregivers
• Wider community

47
Critical Appraisal Criteria for Health Care
Intervention Studies

• Were all important outcomes considered so the
  results can be applied?
• Consider whether
• Any benefit reported outweighs any harm and/or
  cost
• Policy or practice should change as a result of
  the results of this trial

48
Methodological Strengths

• Study design randomized controlled trial
• Follow-up period was appropriate
• Follow-up was complete (lt20 loss to follow-up)
• Groups comparable at baseline

49
Methodological Limitations

• Small sample size
• Volunteer bias
• Potential for recall bias related to reporting
  falls
• Strategies for collecting data regarding the
  number of falls

50
Other Resources

• Evidence-Informed Decision-Making Resource
  Modules posted on Avenue to Learn
• Bandolier Evidence-Based Health Care
  http//www.medicine.ox.ac.uk/bandolier/
• Tutorials
• Introduction to Evidence-Based Medicine (Duke
  University Medical Center Library)
  http//www.hsl.unc.edu/services/tutorials/ebm/inde
  x.htm
• Directories
• Evidence Based Medicine (EBM) Tookit (New York
  University School of Medicine. The Frederick L.
  Ehrman Medical Library)
• www.urmc.rochester.edu/hslt/miner/digital_library/
  evidence_based_resources.cfm
• Resource Guide for Evidence-Based Practice
  (University of Alberta Libraries)
  http//www.library.ualberta.ca/subject/evidence/gu
  ide/index.cfm
• McKibbon, A. (1999). PDQ Evidence-based
  principles and practice. Hamilton, Ontario B.C.
  Decker Inc.
• Evidence Based Practice Tips http//www.ebmtips.ne
  t/risk001.asp