Critical Appraisal: An Introduction

1
Critical Appraisal An Introduction

• Melanie Browne HBHSc, MLIS
• Information Specialist

2
(No Transcript)
3
Agenda

• Introduction
• Define Evidence-based clinical practice
• How to read a research article
• Search for best evidence
• Apply critical appraisal in your library
• Our new role
• Case presentation
• Resources

4
Evidence-Based Medicine

• "...the conscientious, explicit, and judicious
  use of current best evidence in making decisions
  about the care of individual patients.
• from Sackett, DL, et al. "Evidence based
  medicine What it is and what it isn't." (BMJ
  1996 312 71-2)

5

• "Evidence based medicine (EBM) is an approach to
  health care that promotes the collection,
  interpretation, and integration of valid,
  important and applicable patient-reported,
  clinician-observed and research-derived evidence.
  The best available evidence, moderated by patient
  circumstances and preferences, is applied to
  improve the quality of clinical judgements"
  (McKibbon et al 1995).

6
Why EBM?

• A clinician needs to read 17 peer reviewed
  articles per day, every day of the year, to stay
  current (Haynes 1993).

Haynes, R. (1993) Where's the meat in clinical
journals? ACP Journal Club, 119 A23-4.
There are 20 million pieces of 'evidence' of
varying quality and sometime of contradictory
conclusions.
7
How Can You Access, Distil and Apply
Research/Evidence?

• Develop summaries
• Train clinicians and other health care decision
  makers to find and appraise relevant evidence
• Decision support systems

8
Summaries With Rigorous Methodologies

• Cochrane Collaboration
• National Centre for Clinical Excellence
• Centre for Clinical Effectiveness at Monash
  University

9
The Argument for Evidence-Based Medicine

• Stay up to date with the current literature
• Communicate effectively
• Make the best use of information from the
  history, physical examination, and diagnostic
  testing
• Avoid common pitfalls of clinical decision-making
  

10
Critical Appraisal

• The assessment of evidence by systematically
  reviewing its relevance, validity and results to
  specific situations. - Chambers, R. (1998).

11
What is best evidence?

• Using critical appraisal skills you can
  understand the methods and results of the
  research and then be able to assess the quality
  of the research.

12

• Critical appraisal Exact science
• Critical appraisal right answer

13
Information Mastery

• Slawson and Shaughnessy Formula
• Usefulness of Medical Information
• Relevance x Validity____________________
• Work to Access

14

• Smith R. What clinical information do doctors
  need? Br Med J 1996 313 1062-8.

15

• When doctors see patients they usually generate
  at least one question
• Most of the questions concern treatment
• Many of the questions are highly complex,
  simultaneously asking about individual patients
  and particular areas of medical knowledge

16

• Often doctors are asking not simply for
  information but for support, guidance,
  affirmation, and feedback
• Doctors are most likely to seek answers to these
  questions from other doctors
• The best information sources provide relevant,
  valid material that can be accessed quickly and
  with minimal effort

17
Basic elements of clinical decision making
Haynes RB. Loose connections between
peer-reviewed clinical journals and clinical
practice. Ann Intern Med 1990113724-8.
18
Why Should we Critically Appraise?

• Published research is not always reliable
• Published research is not always relevant
• To improve clinical effectiveness, we need a
  systematic framework to interpret research

19
Difficulties with Critical Appraisal

• Can be time consuming initially
• Doesnt provide an easy answer
• It could show a lack of good evidence in a
  particular topic

20
Key Steps to Effective Critical Appraisal

• 1. Are the results valid?
• 2. What are the results?
• 3. How will these results be relevant to
• the patient?

21
Validity and Reliability

• A test is valid when it measures what its
  supposed to.
• If a test is reliable, it yields consistent
  results.
• A test can be both reliable and valid, one or the
  other, or neither.
• Reliability is a prerequisite for measurement
  validity. 

22
Reliable, but Not Valid!
23
Not Reliable, Not Valid!
24
Reliable and Valid
25
Mark Newman - Middlesex University updated 04/2001
26
What a Physician can be Faced With on a Daily
Basis?

• 1 - 2 questions per patient (clinics)
• 15/patient/day (wards)
• 30 of questions are followed
• People
• Books
• Electronic resources
• 40 easy to answer
• 30 tough to answer
• 30 cannot answer
• Source Dawes M, Sampson U. Knowledge management
  in clinical practice a systematic review of
  information seeking behaviour in physicians. Int
  J Med Inform. 2003 Aug 71(1)9-15. Review. PMID
  12909153

27
Steps to EBM

• 1. Formulate a clear, focused clinical question
• 2. PICO model
• 3. Search the literature for the best external
• evidence
• 4. Critically appraise the evidence for its
  validity
• and usefulness
• 5. Implement the useful evidence in clinical
• practice
• 6. Evaluate the results

28
Question Formulation

• Not easy but EXTREMELY important
• Good questions will
• Focus/clarify your information need
• Give you some idea of where to look for
  information
• Give you searching concepts and terms

29
Background Questions

• Often broad in nature
• Often not patient-specific but fact based
• May not need to integrate knowledge
• More common early in training/new situations
•  

30
Foreground Questions

• Often for a specific patient or clinical
  situation
• Narrow in focus
• Need to integrate external information with
  clinical/situational data

31
PICO Structure

• Patients P
• Intervention I
• Comparison C
• Outcome O

32
Clinical Scenario

• What therapeutic agents can be used for rate
  control of atrial fibrillation (AF) in a patient
  with congestive heart failure (CHF)?

33
Starting Point

• Department Emergency
• Population Patients with atrial fibrillation and
  congestive heart failure
• Intervention Rate control
• Comparison N/A
• Outcome Mortality, effectiveness of rate control

34
PICO Schematic Model
Rate control
Patient w/ Atrial Fibrillation in congestive
heart failure
Effectiveness of rate control
Comparison
Rhythm control / No treatment
35
General Search Strategy
Clinical problem
Define the search question
Try another relevant resource
Choose a resource/database
Create a search strategy
Create a search strategy
Summarize the evidence
Poor yield
Summarize the evidence
Try another relevant resource
Apply the evidence
Adapted from Sackett, D. et al. 2000.
Evidence-Based Medicine How to Practice and
Teach EBM. 2nd Edition. Toronto Churchill
Livingstone.
36
Study Types for Question Types
Diagnosis Prospective cohort study with good quality validation against Gold Standard
Therapy Randomized controlled clinical trial (RCT)
Etiology/ Harm RCT, cohort or case-control study (probably retrospective)
Prognosis Prospective cohort study
37
Does this treatment work? systematic review, RCT
How good is a diagnostic test? (prospective) cohort study
Should we screen? RCT
What causes this disease? RCT, prospective cohort study, case control study (rare diseases)
What did people think or do? cohort study, cross-sectional survey, qualitative study
38
(No Transcript)
39
How can we do Critical Appraisal?

• Use common sense
• Use simple checklists
• Use different checklists depending on the
  different types of studies (i.e., RCTS,
  systematic reviews etc)
• Checklists help you focus on the important parts
  of the article

40
Research Methodology

• Who were the participants of the study?
• How were they recruited?
• Was there bias in the recruiting methods?
• How was the data collected?
• What statistical tests were used?
• Where the data collection methods accurate?

41
Critical appraisal questions

• What is the paper about?
• Why was the study done?
• What type of study was done?
• Was it primary research (experiment, RCT, cohort,
  case-control, cross-sectional, longitudinal, case
  report/series)?

42
Critical appraisal questions

• Was it secondary research (overview, systematic
  review, meta-analysis, decision analysis,
  guidelines development, economic analysis)?
• Was the design appropriate (for study on
  treatment, diagnosis, screening, prognosis, or
  causation)?

43
Critical appraisal questions

• Was the study ethical?
• Is the design right?
• (BMJ Editor's checklists)

44
How to Read a Research Article? First Glance.

• Purpose of reading the paper.
• Do not read the abstract.
• Read the title, find out who the authors are and
  where they work, look for sources of funding and
  conflicts of interest
• Look at the tables and figures
• Is there a diagram to show the flow of
  participants through each stage of the study
• Why was the study done and what hypothesis were
  the authors testing?
• What is broadly the topic of research?
• Therapy, Diagnosis, Screening, Prognosis,
  Causation?
• What type of study was done?
• Is this a primary (experimental, clinical trial,
  survey) or a secondary paper (review,
  meta-analysis, guideline, economic analysis)?
• Was the study design appropriate?
• Now proceed to a critical appraisal of the paper

45
Case Presentation

• Read the Ray et al(2008) paper and divide the
  group into 2 for a debate on the paper Breast
  size and risk of type 2 diabetes mellitus. CMAJ

46

• Ray and colleagues studied data from 92 106 women
  in conjunction with the Nurses' Health Study II
  and found that breast size at age 20, assessed by
  recall of bra cup size, correlated positively
  with the incidence of type 2 diabetes

47

• After adjustment for relevant factors, such as
  body mass index, waist circumference and family
  history of diabetes, the hazard ratio dropped to
  1.58 but remained significant.

48

• Nurses' Health Study II population data emanates
  from women who were mainly of white ancestry, and
  that their analysis is based on recall and
  self-report.

49
New Role of Health Sciences Librarians

• Teaching access to the literature and other
  information resources
• Teaching use of technology as a means to access
  and manage information
• Teaching skills in information organization and
  critical appraisal

50
Role of Librarians as Information Clinicians

• Information Clinician
• Medical Informatics Tutor
• EBM Educator

51
Informationist

• A discipline requiring a combination of the
  skills of a librarian, a clinical epidemiologist
  and a medical scientist
• Davidoff, F. Florance,V. (2000) The
  Informationist A New Health Profession? Annals
  of Internal Medicine, 132 996-998.

52
Effective INFORMAtician

• Knows categories of information resources
• therapy, diagnosis, prognosis
• synthesized resources vs original studies
• Knows strengths/weaknesses of information
  resources
• Knows when to use each category
• Knows that some resources are better than others
  in certain situations
• Cochrane diagnostic accuracy data?
• Knows what to do with the results

53

• Practical solutions - Librarian as a partner on
  the teaching team teachable moments
• Formal hospital library educational programs have
  focused on orientation and instruction in the use
  of bibliographic tools. i.e.- LATCH (Literature
  Attached to Charts) and clinical medical
  librarianship

54
Critical Appraisal Skills Training

• Research methodology
• Statistical techniques

55
Courses for Librarians Out There

• CILIP the Chartered Institute of Library and
  Information Professionals
• Critical appraisal skills for healthcare
  librarians building on the basics
• http//www.cilip.org.uk/training/training/2008/li
  binfo/criticalappraisalskillsforhealthcarelibraria
  nsbuildingonthebasics.htm

56
General Resources

• CASP (Critical Appraisal Skills Programme) - part
  of the Public Health Resource Unit based at
  Oxford, CASP runs training workshops on critical
  appraisal skills.  
• Evidence-Based Medicine Toolkit - hosted by the
  University of Alberta, this is an online "box" of
  handy tools to help you find, appraise, and apply
  in practice, evidence-based research
• How to read a paper - a set of ten guides from
  the BMJ (individual links given in the relevant
  section below).
• Levels of Evidence - a ranking system used to
  rank various study designs in order of
  evidence-based merit systematic
  reviews/meta-analyses and well conducted
  randomised controlled trials (RCTs) are usually
  seen as the best form of "evidence", with
  research based on the outcome of a case series
  placed somewhere near the bottom
• Netting the Evidence - search for the keyword
  "appraisal" to find a quality assessed list of
  appraisal resources
• User's guides to evidence based practice - based
  on a series of articles published in JAMA, these
  guides give comprehensive advice on how to find,
  appraise and apply research in practice

57
Keeping Current

• Daily InfoPOEMs http//www.infopoems.com/
• bmjupdates
• http//bmjupdates.mcmaster.ca/index.asp
• Tables of contents of journals
• (e.g. JAMA)
• From journal itself
• My NCBI from PUBMED

58
Predictive Value of tests
Confidence Intervals
Relative Risk Reduction
Intention to treat
P Value
Relative Risk
Absolute Risk Difference
Sensitivity
Absolute Risk Reduction
Absolute Benefit Increase
Specificity
Likelihood Ratio
Numbers Needed to Treat (NNT)
Odds Ratio
59
(No Transcript)