Traditional ExpertBased Information Delivery Systems

1
Traditional Expert-Based Information Delivery
Systems

• Using an Expert, Being an Expert

2
Roles of Experts

• Consultation
• CME
• Review articles
• Practice guidelines
• Decision analysis

3
Using an Expert/Being an Expert

• Definition of an expert
• Subspecialist or primary care clinician with
  special interest
• Anyone/anything you go to for an answer to a
  question

4
Using an Expert/Being an Expert

• Never ask the barber whether you need a haircut
• So many specialists fall into the habit of
  looking where the light is -- that is, offering
  solutions only in territory familiar to them. . .
  Wonderful examples exist of otherwise excellent
  researchers who are unable and unwilling to
  recognize evidence contrary to their beliefs.

5
Usefulness Score

• Work Low
• Significant potential for usefulness
• Relevance Varies
• Validity Expert dependent
• If either relevance or validity is zero,
  usefulness is zero

6
Types of Experts

• Content Expert
• Clinical Scientist
• YODA

7
Content Expert

• Experienced, particularly diagnosis and
  procedures, not necessarily therapy
• Not trained in clinical epidemiology (validity)
• Traditional education favors DOEs (relevance)
• May not be current, may rely on anecdotes
• Risky extrapolation Information is only as
  current as the last consultation

8
Clinical Disagreement Between/Within Experts

• Same film disagree 29 of time
• Previous read disagree with self 20 of time
• Studied with venograms, fundi, MRI, angiography,
  mammograms, pathology (melanoma diagnosis)
• March 97 Bandolier on the Web Histology as Art
  Appreciation

9
Never ask a barber . . .

• Chalmers Recommendation highly correlated with
  training and source of income
• Management of acute GI bleed
• Surgeons surgery- 50 conservative- 15
• Internists surgery- 15 conservative- 50

10
Clinical Scientist

• Good at evaluating evidence up-to-date, dont
  have to be content experts
• Separation of therapeutics
• Medical Librarian, PharmD

11
(No Transcript)
12
YODA Your Own Data Analyzer

• Content expert and clinical scientist
• Consider POEMs first, even if this information
  conflicts with DOEs or clinical experience
• When POEMs not available, use best DOEs with an
  open mind
• Demonstrate appropriate validity assessments
• Not to be confused with YUCKs

13
YUCK

• YOUR
• UNSUBSTANTIATED
• CLINICAL
• KNOW IT ALL

14
Experts gone wrong YUCKs
15
YUCK

• Your Unsubstantiated Clinical Know-it-all
• Maladaptive
• Rigid, Dogmatic
• All personality types, but people who see things
  in Red and Green can fall into the YUCK trap

16
The Golden Question Thats interesting . . . Is
there any evidence that . . . ?
17
If its not a valid POEM, its just not
necessarily so
18
Making the Most of a CME Presentation
19
Dilberts Take on CME
20
(No Transcript)
21
Continuing Medical Education

• People remember 90 of what they do, 75 of what
  they say, but only 10 of what they hear
• How to make the 10 count

22
Do We Get Something From CME?
23
Is post-test performance improved? (DOE)

• YES
• Beware Chinese-Dinner Memory Dysfunction

24
Are patient outcomes improved? (POEM)

• No . . .Multiple RCTs have failed to find a
  benefit from traditional lecture format (passive)
• Maybe . . . with active (hands-on) workshops
  combined with close follow-up

25
Usefulness

• Validity Depends on the speaker
• Relevance Depends on POEMDOE ratio
• Work Higher than it seems
• NBA analogy (only last two minutes count)
• Tracking down validity of new POEMs

26
Role of the Speaker

• Present a good mix of POEMs highlighted by
  clinically relevant DOEs
• Augment POEMs with clinical experience
• Identify Level of Evidence (LOE) for listener

27
Role of the Listener

• Identify, before the talk begins
• What you want to learn
• What are the POEMs you need to know?
• Actively evaluate information (CME worksheet)
• When a change-inducing POEM is presented,
  validate
• By questioning the speaker
• By cross-checking with other sources

28
Identifying Common POEMS

• Will this information have a direct bearing on
  the health of my patients (is it something they
  care about)?
• Is the problem common to my practice?
• Is the intervention feasible?
• If true, will it require me to change my current
  practice?

29
Newer Models for CME

• Practice-based small group CME
• Educational prescriptions
• Point of care Sources
• Team-based learning
• Audience response systems
• CME worksheet

30
Experts IIIUsing Review Articles
31
Reviews- Three Basic Types

• Summary Reviews
• Synthesis Reviews
• Translation Journals

32
Summary Reviews

• Broadly paint landscape
• Validity uncertain Authors begin with
  conclusions and find supporting references
• References often inaccurate and out of date
• Expertise of author varies inversely with quality
  of review- Oxman/Guyatt
• Must confirm POEMs with original research,
  increasing work

33
Cecils Textbook of Medicine
34
(No Transcript)
35
Synthesis Reviews

• Systematic reviews
• Meta-analysis or overviews
• Answer one or two specific questions
• Review primary literature with strict criteria
  average time to produce a systematic review 2
  years.
• Conclusions supported by available evidence
• Only way to achieve LOE 1a
• Meta-analysis Achieve power not possible by
  single study

36
Systematic Reviews

• Excellent source for hunting and foraging
• The Cochrane Library - Database of Systematic
  Reviews

37
(No Transcript)
38
Translation Journals

• Quick reads for retracing and sporting
• Almost always written by experts, with their
  inherent biases
• Saturday morning conversion of lecture notes
• Low work, but with low validity, may be zero
  usefulness
• Hunting/foraging Entering jungle on starless
  night

39
Patient Care
40
Translation Journals

• Common POEMs need original data for verification,
  greatly increasing work
• Watch for weasel words, based on DOEs and
  anecdotes
• it seems, may be effective, so one may
  assume, it appears, in my experience

41
Weasel Words
Patient Care
42
Reading Review Articles

• Buyer Beware Unsystematic reviews lead to
  unsystematic conclusions. Readers looking for a
  shortcut to understanding evidence about health
  problems and patient care should at least look
  for reviews by those who have not taken
  shortcuts
• -- Brian Haynes, MD, PhD

43
Is it true? Evaluating Information from Reviews

• Comprehensive Search?
• Medline others
• MEDLINE misses gt50 of articles
• Cochrane registry is especially good source
• Unpublished literature, evaluate publication bias
• Done by more than one person and compared

44
Evaluating Information from Reviews

• Inclusion Criteria
• Prefer no language restriction
• Sometimes validity criteria incorporated (random,
  blinded, appropriate follow-up, gold standard,
  etc.)
• Best if done independently by 2 investigators

45
Evaluating Information from Reviews

• Appropriate criteria?
• Assurance that criteria specific to type of
  article employed (therapy, diagnosis, prognosis,
  etc)
• Process independent by gt 2 authors?

46
Evaluating Information from Reviews

• Were the included studies valid?
• Garbage in garbage out
• Homogeneity vs Heterogeneity just finding the
  words and an explanation most important
• 1a systematic review of high quality RCTs with
  homogeneity

47
Experts IVPractice Guidelines

• The Good
• The Questionable
• The Ugly

48
(No Transcript)
49
The UGLY

• Opinion based guidelines
• Consensus based guidelines

50
The UGLY - Opinion based guidelines

• Whose opinion?
• Conflict of interest?
• Perspective?

51
The UGLY - Consensus based guidelines

• Who?
• Dominant individual?
• Time pressure?

52
Consensus

• A process by which a group agrees to something
  which no individual in the group believes to be
  appropriate.

53
The Questionable - Evidence Based Guidelines

• How was the evidence used?
• POEMs vs DOEs?
• LOEs for individual studies?

54
American Cancer Society

• Review of all studies available
  (evidence-based)
• Oncology experts reviews recommends guidelines
• Volunteer board reviews modifies

55
National Cancer InstituteMammography Guideline

• Step 1 - Expert Panel Review
• The data currently available do not warrant a
  universal recommendation for mammography for all
  women in their forties. Each woman should decide
  for herself whether to undergo mammography . . ..
  Given both the importance and the complexity of
  the issue involved in assessing the evidence, a
  woman should have access to the best possible
  relevant information regarding both benefits and
  risks, presented in an understandable and usable
  form

56
National Cancer InstituteMammography Guideline

• Step 2 - Political Pressure
• Multiple letters to congressmen
• Senate vote (98 to 0) supporting mammograms for
  women in their 40s

57
National Cancer InstituteMammography Guideline

• Step 3 - Advisory Board Recommendation
• All women aged 40-50 should have a mammogram
  every 1 to 2 years

58
The Good - Evidence Linked Guidelines

• Exactly what is the evidence?
• How was the evidence obtained?
• How strong is the evidence for each
  recommendation?
• Where could you find the evidence to check it out
  for yourself?

59
The Good - Evidence Linked Guidelines

• Brief Summary Statement for each recommendation
• Detailed Discussion of the evidence
• Long Reference section pointing to original
  research
• Methods section showing how evidence was obtained
  and evaluated

60
(No Transcript)
61
Guidelines Tested?

• RESULTS Outpatients aged 60 years or younger
  with no comorbidity who were prescribed therapy
  consistent with ATS guidelines (ie, erythromycin
  with some exceptions) had 3-fold lower
  antimicrobial costs (5.43 vs 18.51 Plt.001) and
  no significant differences in medical outcomes.
• Outpatients older than 60 years or with 1
  comorbidity or more who were prescribed therapy
  consistent with ATS guidelines (ie,
  second-generation cephalosporin,
  sulfamethoxazole-trimethoprim, or beta-lactam and
  beta-lactamase inhibitor with or without a
  macrolide) had 10-fold higher antimicrobial costs
  (73.50 vs 7.50 Plt.001) despite trends toward
  higher mortality and subsequent hospitalization,
  no significant differences in medical outcomes
  were observed.
• Gleason PP, et al. Medical outcomes and
  antimicrobial costs with the use of the American
  Thoracic Society guidelines for outpatients with
  community-acquired pneumonia. JAMA. 1997 Jul
  2278(1)32-9.

62
Rapid Recognition ofEvidence Linked Guidelines

• Strength of Evidence Indicators
• Evidence Tables
• New SORT (Strength Of Recommendation Taxonomy)
  classification

63
Strength of Recommendation Taxonomy (SORT)

• Consistent and good-quality POE
• Standard LOEs for validity, POE vs DOE for
  relevance
• B. Inconsistent or limited-quality POE
• Consensus, usual care, opinion, DOE, case series

64

• Effect on Patient-Oriented Outcomes
• Symptoms (drivers license)
• Functioning (visual loss)
• Quality of Life (amputation)
• Lifespan

SORT A
SORT B

• Effect on Disease Markers
• Diabetes (Photocoagulation, GFR, NCV)
• Arthritis (x-ray, sed rate)
• Peptic Ulcer (endoscopic ulcer)

SORT C
Relevance of Outcome

• Effect on Risk Factors for Disease
• Improvement in markers (blood pressure,
  cholesterol, HBA1C, microalbuminuria)

• Highly Controlled Research
• Randomized Controlled Trials
• Systematic Reviews

• Physiologic Research
• Preliminary Clinical Research
• Case reports
• Observational studies

Uncontrolled Observations Conjecture
Validity of Evidence
65
The evolution of evidence subclinical
hypothyroidism

• Step 1
• Search of 10 databases
• Studies summarized
• 12 experts rated the evidence
• Recommendations
• Recommend against routine screening for
  subclinical hypothyroidism
• Recommend against routine treatment of 4.5
  10.0 mIU/L
• Surks MI, et al. Subclinical thyroid disease.
  Scientific Review and Guidelines for diagnosis
  and management. JAMA 2004291228-238.

66
The evolution of evidence subclinical
hypothyroidism

• Step 2
• Consensus meeting among members of the American
  Association of Clinical Endocrinologists, The
  American Thyroid Association, and The Endocrine
  Society.
• New recommendation statement
• Recommendations sent to leadership of the
  organizations

67
The evolution of evidence subclinical
hypothyroidism

• The result
• New recommendations from the three societies
• Most patients with TSH levels 4.5 10 mIU/L
  should be treated
• Should perform routine screening for subclinical
  hypothyroidism
• Why?
• Although good evidence is unavailable, there is
  a sizable amount of fair evidence and an
  abundance of opinion by experts . . . The
  (scientific panel recommendations) are contrary
  to the practice of many. . . experts
• Gharib H, et al. Consensus statement
  Subclinical thyroid dysfunction A joint
  statement on management from the American
  Association of Clinical Endocrinologists, The
  American Thyroid Association, and The Endocrine
  Society. J Clin Endocrinol Metab 200590581-5.

68
ACC/AHA Guidelines

• 16 current guidelines through Sept, 2008 with
  2711 clinical recommendations
• 11 Level A
• 48 Level C (expert opinion).
• The proportion of recommendations for which there
  is no conclusive evidence is also growing.
• Tricoci P, et al. JAMA 2009301(8)831-41

69
Guidelines - Validity Issues

• Explicit sensible process to identify, select
  combine evidence?
• Explicit sensible process to assess values of
  different outcomes?
• Account for recent developments?
• Subjected to peer review testing

70
Guidelines Tired Wired

• Authority
• Hidden Methods
• Theoretical
• Prescriptive Guidelines

• Evidence
• Open Methods
• Tested
• Information to guide patient choice

71
Experts VDecision Analysis

• Relevance - The Major Strength
• Turning DOEs into POEMs
• Incorporating multiple factors into a clinical
  decision
• Combinations of outcomes
• Patient preferences
• Costs

72
Validity - The difficult part

• Clinicians are unfamiliar with the basic
  conceptual framework
• Construction of a decision tree
• Assigning probabilities utilities
• Sensitivity analysis

73
(No Transcript)
74
(No Transcript)
75
Decision Analysis- Validity

• LOE 1
• Sensible costs/alternatives
• Check for inclusion of utilities based on
  outcomes of importance as defined by actual
  patient opinion.
• Systematic review of evidence (1a)
• Robust sensitivity analysis

76
Summary

• Experts are a source of quick and useful
  information
• Variations of the Golden Question
• Thats interesting . . . Have their been any
  studies?
• I didnt know that . . . Is there something I
  can read about that?
• Slawson DC, Shaughnessy AF. Obtaining useful
  information from expert based sources. BMJ
  199731494

77
Summary

• Traps to avoid
• Assertions based only on DOEs (Perpetuating
  PROSE)
• Unassessed validity
• Quality of the review (CME) varies inversely with
  the expertise of the writer (presenter)

78
Summary

• Look for levels of evidence (LOE), strengths of
  recommendations (SORT)
• Dont trust reviews, including print and
  electronic sources without these!

79
(No Transcript)