Is it True: Evaluating Medical Reviews

1
Is it True Evaluating Medical Reviews
2

• The review article itself should be the product
  of scientific investigation in which the
  participants are original investigations
  (research) rather than patients

3
Usefulness

• Work Low, good source for POEMs
• Relevance If title and abstract or article
  conclusion hold promise of POEM, continue
• Validity Uncertain

4
Reviews- Three Basic Types

• Textbooks
• Academic Reviews
• Translation Journals

5
Textbooks

• Collection of review articles
• Minimal, if any, supporting evidence
• Questionable validity, long lag time to publish
• Average 1-2 years
• Most useful for retracing, less hunting and
  foraging

6
Academic Reviews

• Summary Broadly paint landscape
• Validity uncertain- begin with conclusions and
  find supporting references
• References often inaccurate and out of date
• Expertise of author varies inversely with
  quality of review- Oxman/Guyatt
• More later
• Must confirm POEMs with original research,
  increasing work

7
Academic Reviews

• Synthesis Systematic reviews
• Meta-analysis or overviews
• Answer one or two specific questions
• Review primary literature with strict criteria
• Conclusions supported by available evidence
• Meta-analysis Achieve power not possible by
  single study

8
Academic Reviews

• Excellent source for hunting and foraging
• The Cochrane Library - Database of Systematic
  Reviews
• Clinical Evidence (BMJ-BMA)
• Clinical Inquiries (FPIN)- SORT
• AFP EB Reviews- SORT
• Dynamed - SORT
• Essential Evidence Plus- SORT

9
Translation Journals

• Quick reads for retracing and sporting
• Low work, but with low validity, may be zero
  usefulness
• Hunting/foraging Entering jungle on starless
  night

10
Translation Journals

• Common POEMs need original data for verification,
  greatly increasing work
• Watch for weasel words, based on DOEs and
  anecdotes
• it seems, may be effective, so one may
  assume, it appears, in my experience

11
Weasel Words
Patient Care
12
Translation Journals

• Buyer Beware Unsystematic reviews lead to
  unsystematic conclusions. Readers looking for a
  shortcut to understanding evidence about health
  problems and patient care should at least look
  for reviews by those who have not taken shortcuts

13
Determining the usefulness of reviews

• Onto the worksheets!

14
Reviews Determining Relevance

• A. Addressing specific clinical question?
• A. Patient-oriented evidence?
• B. Common problem?
• C. Change your practice?

15
Reviews Determining Validity

• Answer ALL worksheet questions
• Stop fatal flaw
• Notice how hard this is! Average time for a good
  systematic review- 2 years!
• Much different from your usual review/CME talk

16
Worksheet Qs Finding the studies

• Clearly stated?
• Terms appropriate? MESH-linked? None missing?
• Comprehensive?
• Medline another
• MEDLINE misses gt50 of articles
• Cochrane registry is especially good source
• Science Citation Index
• Bibliographic review
• Unpublished literature
• conference abstracts, personal correspondence
  with important investigators or pharmaceutical
  companies
• Done by more than one person and compared

17
Worksheet Qs Selecting the studies

• Inclusion Criteria
• Established a priori
• Minimum factors Population/problem
  intervention/comparison outcomes study design
• Prefer no language restriction
• Sometimes validity criteria incorporated (random,
  blinded, appropriate follow-up, gold standard,
  etc.)
• Best if done independently by 2 investigators
• Possibly blinded to author/journal/study results

18
Worksheet QsValidity of included studies

• Appropriate criteria?
• Assurance that criteria specific to type of
  article employed (therapy, diagnosis, prognosis,
  etc)
• If therapy randomization, blinding, concealed
  allocation, follow-up
• Process independent by gt 2 authors?
• Surprising differences!
• Why blinding may be important
• 2 sample articles, same study methods
• One finds benefit, other does not
• serious flaws in article without benefit

19
Worksheet Qs Validity

• Were the included studies valid?
• Garbage in garbage out
• If yes, no problem
• If no, how did authors handle this?
• Exclusion/inclusion criteria for quality of study
• Subanalysis with comparison of results
• Need to consider how these flaws affect
  results/conclusions

20
Worksheet Qs Analyzing the data

• Homogeneity vs Heterogeneity just finding the
  words and an explanation most important
• If NOT homogeneous?
• Need qualitative explanation. Is it due to chance
  vs study design, population, exposure, or
  outcome?

21
(No Transcript)
22
(No Transcript)
23
Worksheet Qs Analyzing the data

• Appropriateness of combining data
• Vote count not usually appropriate
• Important to include magnitude of the overall
  effect
• Cannot be done without some common ground-
  outcome
• Publication bias
• Small, negative trials less likely to be
  published
• Examined by funnel plot
• Number needed to change results

24
Funnel plot examples
From Cooper Hedges The handbook of research
synthesis. 1994
25
From Cooper Hedges The handbook of research
synthesis. 1994
26
Reviews Major Points

• Validity traps to avoid
• Assertions based on DOEs -- avoid perpetuating
  medical gossip
• Unassessed validity -- Personal experience
  unreliable as a basis for therapeutic
  interventions
• Missing pieces -- Quality of the review varies
  inversely with the expertness of the writer
• Failure to identify level of evidence Look for
  LOEs/ SORT

27
Reviews- Three Basic Types
28
(No Transcript)
29

• Effect on Patient-Oriented Outcomes
• Symptoms
• Functioning
• Quality of Life
• Lifespan

SORT A
SORT B

• Effect on Disease Markers
• Diabetes (microalbuminuria, GFR, photocoagulation
  rates)
• Arthritis (ESR, x-rays)
• Peptic Ulcer (endoscopic ulcers)

SORT C
Relevance of Outcome

• Effect on Risk Factors for Disease
• Improvement in markers (blood pressure, HbA1C,
  cholesterol)

• Highly Controlled Research
• Randomized Controlled Trials
• Systematic Reviews

• Physiologic Research
• Preliminary Clinical Research
• Case reports
• Observational studies

Uncontrolled Observations Conjecture
Validity of Evidence