Introduction to the Methods Guide for Medical Test Reviews

1
Introduction to the Methods Guide for Medical
Test Reviews

• Prepared for
• The Agency for Healthcare Research and Quality
  (AHRQ)
• Training Modules for Medical Test Reviews Methods
  Guide
• www.ahrq.gov

2
Learning Objectives

• Describe the purpose of the Methods Guide for
  Medical Test Reviews.
• Understand the unique features of medical test
  reviews.
• Review the structure of the Methods Guide for
  Medical Test Reviews.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
3
Medical Test Reviews (1 of 4)

• Systematic reviews of medical tests help
  determine which tests can be relied on to improve
  health outcomes and the circumstances required
  for the tests to achieve this.
• Systematic reviews developed under the auspices
  of the Evidence-based Practice Center (EPC)
  Program
• Need to be technically comprehensive and
  practical
• Need to be completed with limited time and
  resources

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
4
Medical Test Reviews (2 of 4)

• The Methods Guide for Medical Test Reviews
• Provides practical guidance for those who prepare
  and use systematic reviews.
• Promotes consistency and rigor in how specific
  issues are addressed.
• Complements the Methods Guide for Effectiveness
  and Comparative Effectiveness Reviews developed
  by the Agency for Healthcare Research and
  Quality.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Agency for Health Care Research and Quality.
Methods Guide for Effectiveness and Comparative
Effectiveness Reviews. Available at
www.effectivehealthcare.ahrq.gov/methodsguide.cfm.
5
Medical Test Reviews (3 of 4)

• Medical tests are defined here as any test used
  in a health care context, for example, screening,
  diagnosis, or prognosis.
• Using medical tests depends on the context.
• Pre-existing conditions
• Results of other tests
• Skill and knowledge of providers
• Availability of therapeutic resources
• And others

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
6
Medical Test Reviews (4 of 4)

• Researchers have tended to focus on the ability
  of medical tests to
• Conform to technical specifications
• Classify patients into diagnostic or prognostic
  categories
• Influence thought or actions by clinicians and
  patients
• Rarely are medical tests evaluated in randomized
  controlled trials with representative patient
  populations and comprehensive measures of
  patient-relevant outcomes.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
7
Development of the Methods Guide for Medical Test
Reviews (1 of 3)

• Creation of the Methods Guide for Medical Test
  Reviews was guided by two tenets
• Evaluation of a medical test must be linked to
  the context of its use
• Medical tests are about improving patient outcomes

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Agency for Health Care Research and Quality.
Methods Guide for Effectiveness and Comparative
Effectiveness Reviews. Available at
www.effectivehealthcare.ahrq.gov/methodsguide.cfm.
8
Development of the Methods Guide for Medical Test
Reviews (2 of 3)

• The first tenet is to consider the context.
• Medical tests are not neutral reporters of
  reality. The context often affects sensitivity,
  specificity, and the clinical implications of a
  particular test result.
• Therefore, systematic reviews must specify the
  context in which the test is being used.
• The second tenet is to focus on patient outcomes.
• The ultimate purpose of medical tests is to
  improve patient outcomes.
• Improvement occurs by guiding the judgments of
  clinicians concerning disease state or treatment
  response.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
9
Development of the Methods Guide for Medical Test
Reviews (3 of 3)

• Most of the published literature
• Fails to address the clinical impact of tests.
• Focuses instead on test development and
  performance characteristics.
• The first step in a medical test review must be
  to establish a link between the use of a test and
  patient outcomes.

Causal Chain Diagram Test ? Result ?
Categorization ? Decision ? Patient Outcome For
example high risk, disease present, disease
progression
Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
10
Medical Tests Overview (1 of 2)

• Typical uses of medical tests are to
• Screen for the likelihood of a disorder.
• Diagnose the presence of disease.
• Assess response to treatment.
• Assess probability of desirable/undesirable
  consequences.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
11
Medical Tests Overview (2 of 2)

• Typical types of medical tests
• Laboratory tests
• Radiological imaging
• Patient history
• Physical examinations
• Scored questionnaires
• For screening
• For determining likely prognosis
• For assessing the potential response to therapy

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
12
Challenges Specific to Medical Tests (1 of 2)

• Assessing the impact of medical tests can be
  challenging.
• Medical tests have several steps between
  conducting the test and determining the outcome
  of clinical importance.
• Conversely, most treatments lead directly to
  intended results or adverse effects.
• Consequences result from the less direct
  relationship between medical tests and patient
  outcomes.
• Medical tests are often evaluated in isolation
  for their ability to detect a substance or a
  particular anatomic condition.
• Medical tests are not evaluated in terms of their
  impact on overall health outcomes.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Siebert U. Eur J Health Econ 20034(3)143-50 Ta
tsioni A, Zarin DA, Aronson N, et al. Ann Intern
Med 2005 Jun 21142(12 Pt 2)1048-55. PMID
15968029.
13
Challenges Specific to Medical Tests (2 of 2)

• Randomized controlled trials, generally
  considered the strongest form of evidence, are
  rarely performed for medical tests.
• Trials are difficult to carry out because of the
  numerous steps and the variables between the test
  and patient outcomes.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
14
Using Medical Tests (1 of 2)

• Medical tests can be likened to an electronic
  signal detector for the presence or absence of a
  clinical condition.
• Signal detection theory (e.g., sensitivity,
  specificity) can be used to calculate disease
  probabilities for positive or negative test
  results.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Green DM, Swets JA. Signal detection theory and
psychophysics. New York, NY Wiley 1966.
Reprinted 1988. Ledley RS, Lusted LB. Science
1959 Jul 3130(3366)9-21. PMID
13668531. Yerushalmy J. Public Health Rep 1947
Oct 362(40)1432-49. PMID 20340527.
15
Using Medical Tests

• Medical test evaluation studies fall along a
  continuum related to the study objectives.

Study Objective Terms Used Examples
Test conforming to technical specifications Technical efficacy Technical quality of a radiological image
Test conforming to technical specifications Analytic validity Accuracy of a chemical assay for the target analyte Concordance of a commercial genetic test with the true genotype
Test classifying a patient into a disease/phenotype or prognosis category Diagnostic accuracy efficacy Clinical validity Test accuracy Test performance Performance characteristics Operating characteristics Sensitivity and specificity Positive and negative likelihood ratios Positive and negative predictive value Test yield Receiver operating characteristic curve
Test directing clinical management and improving patient outcomes Diagnostic thinking efficacy Therapeutic efficacy Patient outcome efficacy Clinical utility Impact on mortality or morbidity Impact on clinician judgment about diagnosis/ prognosis Impact on choice of management
Test benefiting society as a whole Societal efficacy Incremental cost-effectiveness
Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. ? Fryback DG, Thornbury JR. Med Decis Making
1991 Apr-Jun11(2)88-94. PMID 1907710.
16
Analytic Frameworks (1 of 3)

• Reviewers need an explicit strategy for
  summarizing complex literature in a logical way
  to respond to key questions.
• Frameworks can be used to
• Maintain an orderly process.
• Clarify questions.
• Organize evidence.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Bravata EP, McDonald KM, Shojania KG, et al.
Ann Intern Med 2005 Jun 21142(12 Pt 2) 1056-65.
PMID 15968030. Mulrow C, Langhorne P, Grimshaw
J. Ann Intern Med 1997 Dec 1127(11)989-95.
PMID 9412305. Whitlock EP, Orleans CT, Pender N,
et al. Am J Prev Med 2002 May22(4)267-84. PMID
11988383.
17
Analytic Frameworks (2 of 3)

• Analytic frameworks
• Depict graphically the linkages that relate
  interventions and outcomes.
• Identify questions to help structure the
  literature review.
• Provide an evidence map after the review for
  identifying gaps and weakness in the evidence.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Agency for Healthcare Research and Quality.
U.S. Preventive Services Task Force Procedure
Manual. Available at www.uspreventiveservicestaskf
orce.org/uspstf08/methods/procmanual.htm.
18
Analytic Frameworks (3 of 3)

• An analytic framework has two key components
• A typology for describing the context in which
  the test is used
• Fits with the PICOTS (population, intervention,
  comparator, outcomes, timing, and setting)
  typology (the current standard approach).
• A visual representation of the relationship
  between the application of the test/treatment and
  the outcomes of importance to decisionmaking.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
19
PICOTS Typology (1 of 3)

• PICOTS stands for population, intervention,
  comparator, outcome, timing, and setting.
• This typology was developed to describe the
  context in which medical interventions might be
  used.
• It is important for defining the key questions of
  a review.
• It is important for assessing whether or not a
  given study is applicable.
• Variations in the typology exist, but the
  standard unchanging elements are PICO.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Chalmers I, Hedges LV, Cooper H. Eval Health
Prof 2002 Mar25(1)12-37. PMID 11868442.
20
PICOTS Typology (2 of 3)
PICOTS As Applied to Medical Tests Comments
Population Patient population includes results of other/previous tests Condition(s), disease severity and stage, comorbidities, and patient demographics
Intervention Index test includes clinical role of the index strategy in relation to comparator and the test-and-treat strategy in relation to clinical outcomes Description of index test includes administrator training, technology specifications, specific application issues Three main clinical roles in relation to the comparator replacement, add-on, and triage Description of index test performance and interpretation how the results of index test lead to management decisions/actions
Comparator Comparator test-and-treat strategy Description of comparator test performance and interpretation how the results of the comparator test lead to management decisions/actions
Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
21
PICOTS Typology (3 of 3)
PICOTS As Applied to Medical Tests Comments
Outcomes Relevant clinical outcomes includes any intermediate outcomes of interest Patient health outcomes includes morbidity (which also includes the adverse effects of the test and treatment), mortality, quality of life Intermediate outcomes includes technical specifications, accuracy, decisional, therapeutic impact
Timing Timing of outcome assessment Duration of followup single or multiple followup assessments
Setting Setting of test assessment Ambulatory settings (including primary, specialty care) and inpatient settings
Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
22
Organization of the Methods Guide for Medical
Test Reviews

• Module 1 Introduction
• Module 2 Topic Development and Structure
• Module 3 Choosing Outcomes
• Module 4 Search Strategies
• Module 5 Risk of Bias
• Module 6 Applicability
• Module 7 Grading Strength of Evidence
• Module 8 Meta-analysis With a Gold Standard
• Module 9 Meta-analysis Without a Gold
  Standard
• Module 10 Decision Modeling
• Module 11 Genetic Tests
• Module 12 Prognostic Tests

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
23
Overview of the Flow of Modules
Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
24
Summary

• Evaluation of medical tests presents challenges
• There are several steps between a medical test
  and outcomes of interest (indirect impact).
• Context has great importance.
• Few randomized controlled trials are aimed at
  comparing clinical outcomes of different tests
  and test strategies.
• Available guidance provides some suggestions
• Use the PICOTS (population, intervention,
  comparator, outcomes, timing, and setting)
  topology to clarify the context.
• Use an organizing framework for classifying the
  types of studies and their relationship to key
  questions.

Matchar DB. Introduction to the Methods Guide for
Medical Test Reviews. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
25
Practice Question 1 (1 of 2)

• How are analytic frameworks used in the context
  of this module?
• To graphically depict the specific questions that
  need to be answered by the literature review.
• To list the challenges associated with a review
  of a medical test.
• To demonstrate recurrent themes from the
  literature.
• To display all areas of potential bias.

26
Practice Question 1 (2 of 2)

• Explanation for Question 1
• The correct answer is a. The analytic framework
  should be used to clearly present the questions
  that need to be answered in the literature
  review. The questions are shown by linkages
  between interventions and outcomes.

27
Practice Question 2 (1 of 2)

• Test accuracy is included in which part of the
  PICOTS typology?
• P for patient population
• I for intervention
• C for comparator
• O for outcomes

28
Practice Question 2 (2 of 2)

• Explanation for Question 2
• The correct answer is d. The O in the PICOTS
  typology stands for outcomes, which include
  patient health outcomes and intermediate outcomes
  such as test accuracy.

29
Practice Question 3 (1 of 2)

• Most of the published literature on medical tests
  addresses the clinical impact of tests.
• True
• False

30
Practice Question 3 (2 of 2)

• Explanation for Question 3
• The statement is false. Most of the published
  literature on medical tests focuses on test
  development and test characteristics and not on
  the clinical impact of tests. This poses a
  challenge when reviewing medical tests.

31
Practice Question 4 (1 of 2)

• A causal chain diagram is used to
• Show possible sources of bias
• Show the links between the use of a test and the
  patient outcomes
• Help with decisionmaking based on a positive or
  negative test result
• Display the method used for the literature search

32
Practice Question 4 (2 of 2)

• Explanation for Question 4
• The correct answer is b. A causal chain diagram
  is used to show visually the links between the
  use of a test and patient outcomes.

33
Authors

• This presentation was prepared by Brooke
  Heidenfelder, Rachael Posey, Lorraine Sease, Remy
  Coeytaux, Gillian Sanders, and Alex Vaz, members
  of the Duke University Evidence-based Practice
  Center.
• The module is based on Chapter 1, Introduction to
  the Methods Guide for Medical Test Reviews. In
  Methods Guide for Medical Test Reviews. AHRQ
  Publication No. 12-EC017. Rockville, MD Agency
  for Healthcare Research and Quality June 2012.
  www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
  .

34
References (1 of 4)

• Agency for Healthcare Research and Quality. U.S.
  Preventive Services Task Force Procedure Manual.
  Rockville, MD Agency for Healthcare Research and
  Quality July 2008. 22-4. AHRQ Publication No.
  08-05118-EF. www.uspreventive servicestaskforce.or
  g/uspstf08/methods/procmanual.htm.
• Agency for Healthcare Research and Quality.
  Methods Guide for Effectiveness and Comparative
  Effectiveness Reviews. Rockville, MD Agency for
  Healthcare Research and Quality April 2012. AHRQ
  Publication No. 10(12)-EHC063-EF. Chapters
  available at www.effectivehealthcare.ahrq.gov/meth
  ods guide.cfm.
• Bravata DM, McDonald KM, Shojania KG, et al.
  Challenges in systematic reviews synthesis of
  topics related to the delivery, organization, and
  financing of health care. Ann Intern Med. 2005
  Jun 21142(12 Pt 2)1056-65. PMID 15968030.

35
References (2 of 4)

• Chalmers I, Hedges LV, Cooper H. A brief history
  of research synthesis. Eval Health Prof 2002
  Mar25(1)12-37. PMID 11868442.
• Fryback DG, Thornbury JR. The efficacy of
  diagnostic imaging. Med Decis Making 1991
  Apr-Jun11(2)88-94. PMID 1907710.
• Green DM, Swets JA. Signal detection theory and
  psychophysics. New York, NY Wiley 1966.
  Reprinted in 1988 with corrections and an updated
  topical bibliography (Los Altos, CA Peninsula
  Publishing).
• Ledley RS, Lusted LB. Reasoning foundations of
  medical diagnosis symbolic logic, probability,
  and value theory aid our understanding of how
  physicians reason. Science 1959 Jul
  3130(3366)9-21. PMID 13668531.

36
References (3 of 4)

• Matchar DB. Introduction to the Methods Guide for
  Medical Test Reviews. In Chang SM and Matchar
  DB, eds. Methods guide for medical test reviews.
  Rockville, MD Agency for Healthcare Research and
  Quality June 2012. p. 1.1-1.11. AHRQ Publication
  No. 12-EHC017. www.effectivehealthcare.ahrq.gov/
  medtestsguide.cfm.
• Mulrow C, Langhorne P, Grimshaw J. Integrating
  heterogeneous pieces of evidence in systematic
  reviews. Ann Intern Med. 1997 Dec
  1127(11)989-95. PMID 9412305.
• Siebert U. When should decision analytic modeling
  be used in the economic evaluation of health
  care? Eur J Health Econ. 20034(3)143-50.
• Tatsioni A, Zarin DA, Aronson N, et al.
  Challenges in systematic reviews of diagnostic
  technologies. Ann Intern Med. 2005 Jun 21142(12
  Pt 2)1048-55. PMID 15968029.

37
References (4 of 4)

• Whitlock EP, Orleans CT, Pender N, et al.
  Evaluating primary care behavioral counseling
  interventions an evidence-based approach. Am J
  Prev Med. 2002 May22(4)267-84. PMID 11988383.
• Yerushalmy J. Statistical problems in assessing
  methods of medical diagnosis, with special
  reference to X-ray techniques. Public Health Rep.
  1947 Oct 362(40)1432-49. PMID 20340527.