Retrospective Studies of Clinical Outcomes A Primer for Clinicians

1
Retrospective Studies of Clinical OutcomesA
Primer for Clinicians

• Marc D. Silverstein, MD FACP

2
Overview

• Role of retrospective studies in a research
  program
• Anatomy physiology of research
• Observational research designs
• Cross-sectional studies
• Cohort studies
• Case-control studies
• Human Subjects IRB Review

3
Objectives

• Describe 4 research designs
• Describe 3 threats to validity
• List advantages of retrospective studies
• List disadvantages of retrospective studies
• Understand requirements and processes for IRB
  review of retrospective studies

4
Research

• Definition
• Types of Research

5
Definition of Clinical ResearchNIH Directors
Panel, 1997

• Patient-oriented research
• Research conducted with human subjects (or on
  material of human origin such as tissues,
  specimens and cognitive phenomena) for which an
  investigator (or colleague) directly interacts
  with human subjects
• Mechanisms of human disease
• Therapeutic interventions
• Clinical trials
• Development of new technologies
• Epidemiologic and behavioral studies
• Outcomes research and health services research

6
Health Services Research

• Health services research is the multidisciplinary
  field of scientific investigation that studies
  how social factors, financing systems,
  organizational structures and processes, health
  technologies, and personal behaviors affect
  access to health care, the quality and cost of
  health care, and ultimately our health and
  well-being.
• Its research domains are individuals, families,
  organizations, institutions, communities, and
  populations.
• AcademyHealth

7
Outcomes Research

• Research on measures of changes in patient
  outcomes - patient health status and satisfaction
  - resulting from specific medical and health
  interventions
• Attributing changes in outcomes to medical care
  requires distinguishing the effects of care from
  the effects of the many other factors that
  influence patients health and satisfaction
• AcademyHealth

8
Patient Care and Outcomes Research
9
Physiology of Research

• Design the study
• Implement the study
• Make valid (causal) inferences

10
The Goal
11
Design Implementation
Design
Implement
Research Question
Actual Study
Study Plan
12
Causal Inferences
13
Errors May Occur in Design Implementation of
Research
Error
Error
Design
Implement
Infer
Infer
14
Errors May Occur in Making Inferences about
Internal or External Validity
Design
Implement
Error
Error
Infer
Infer
15
Errors May Occur Anywhere
Error
Error
Implement
Design
Error
Error
Infer
Infer
16
Threats to Validity

• Chance
• Bias
• Confounding

17
Threats to Validity

• Chance random error due to unknown sources of
  variation that distort sample and measurements in
  either directions
• Bias systematic error that distorts sample and
  measurements in one direction
• Confounding an external factor that is
  associated with a predictor variable and an
  outcome variable

18
Reducing Random Error

• Increase sample size
• Statistical analyses

19
Reducing Systematic Error (Bias)

• Population-based studies
• Inclusion and exclusion data
• Standardize measurement
• Train and certify observer
• Refine instrument
• Automate instruments
• Blinded measurements

20
Reducing Confounding

• Anticipate potential confounders
• Measure potential confounders
• Matching, restriction, stratification
• Multivariate analysis

21
I cannot give any scientist of any age better
advice than this the intensity of the conviction
that a hypothesis is true has no bearing on
whether it is true or not. P.B. Medawar
22
Anatomy of Research

• Research Question
• Significance
• Methods

23
Research Question
24
Characteristics of a Good Research Question

• Feasible
• Interesting
• Novel
• Ethical
• Relevant

25
It can be said with complete confidence that any
scientist of any age who wants to make important
discoveries must study important problems. P.B.
Medawar
26
Hypotheses

• Simple (versus complex)
• Specific (versus vague)
• In advance (versus after-the-fact)

27
Estimation

• In clinical studies the goal is often to estimate
  a risk of an outcome or the magnitude of impact
  on a clinical measurement

28
Observational Studies

• Strengths
• Intervention is not feasible or ethical
• Rapid efficient
• Existing data
• Less time
• Expenses are lower
• Case-control studies for rare events

• Limitations
• Do not permit true assessment of time sequence of
  factors and outcomes
• Subject to bias and confounding
• Limited power to study rare risk factors or rare
  outcomes (surveys cohort studies)

29
Study Designs

• Observational studies
• Cross sectional studies
• Cohort studies
• Case Control studies
• Experiments

30
Cross-Sectional Study Design
31
Cross Sectional Study
Population
Sample
32
Cross Sectional Study Example

• The Probability of Malignancy in Solitary
  Pulmonary Nodules
• Swensen, Silverstein, Ilstrup et al
• Arch Intern Med 1997 157 849

33
The Probability of Malignancy in Solitary
Pulmonary Nodules

• Can clinical and radiological SPN characteristics
  predict malignancy in SPNs
• Retrospective cohort at multi-specialty group
  practice
• New diagnosis of 4mm 30 mm solitary pulmonary
  nodule
• Radiological indeterminate SPN with no
  calcification on thin section CT
• Exclude patients with primary lung cancer or
  other cancer within 5 years

34
The Probability of Malignancy in Solitary
Pulmonary Nodules

• Outcomes determined by radiological follow-up for
  2 or more years, surgical diagnosis,
  transthoracic needle biopsy, bronchoscopy biopsy
  or washings
• Clinical characteristics (age, smoking, history
  of other cancer) and radiological characteristics
  (size, location, edge characteristics
  lobulation, spiculation, shagginess)
• Multivariate analysis with logistic regression
• Predictors developed in 2/3 random sample and
  validated in remaining 1/3 sample

35
DiscriminationROC Curve Area (0.83, 0.80)
36
Calibration Observed vs Predicted Probability
37
Threats to ValidityMalignancy in SPN, 1

• Large number of SPNs (629)
• Referral population
• Clinically relevant SPNs (5-30 mm)
• Excludes low risk (lt 4mm)
• Excludes high risk (gt 30 mm)
• No calcification on thin section CT (benign)
• Cancer diagnosis
• Cohort study for outcomes after 2 years
• Some SPNs indeterminate classification

38
Threats to ValidityMalignancy in SPN, 2

• Independent review single radiologist
• Swenson, Silverstein, Edell et al. SPN
  Clinical Prediction vs Physicians, Mayo Clin
  Proc, 1999
• Analysis
• Discrimination and calibration
• Independent sample for validation
• Distance to assess referral bias
• Included all SPNs
• Malignant vs (indeterminate benign)
• (Malignant indeterminate) vs benign

39
Cohort Study Design
40
Cohort Study Design
41
Prospective Cohort Study Design
The Present
The Future
42
Retrospective Cohort Study Design
The Past
The Present
43
Cohort Study Example

• Long-term Survival of a Cohort of Community
  Residents with Asthma
• Silverstein, Reed, OConnell et al
• N Eng J Med 1994331 1537

44
Long-term Survival of a Cohort of Community
Residents with Asthma

• Asthma mortality based on general US population
  death certificates with asthma listed as
  underlying cause of death
• Residents of Rochester, MN with first asthma
  diagnosis 1/1/1964-12/31/1983
• Explicit pre-defined criteria, review of all
  medical records from all providers of care
• Medical records and autopsy reports used to
  classify deaths as due to asthma or other
  conditions

45
Long-term Survival of a Cohort of Community
Residents with Asthma

• 2499 patients with definite or probable asthma
• Mean duration follow-up 14 years (range 0-29
  years)
• 140 deaths in 32,605 person-years of follow-up
• Survival not significantly different form
  expected
• Survival worse in asthmatics with other lung
  disease
• 4 of deaths in persons with asthma were due to
  asthma

46
Observed vs. Expected Survival
47
Survival in Asthma only vs Asthma and Other Lung
Disease
48
Threats to ValidityLong-term Survival in Asthma,
1

• Large population-based cohort (2499)
• Yunginger, Reed OConnell, A Community based
  study of the Epidemiology of asthma, Am Rev Resp
  Dis, 1992
• Asthma diagnosis
• Beard, Yunginger, Reed et al, Interobserver
  Variability in Medical Record Review An
  Epidemiological Study of Asthma, J Clin Epid,
  1992

49
Threats to ValidityLong-term Survival in Asthma,
2

• Asthma deaths
• 14 years of follow-up
• Small number of deaths (140)
• Classification of deaths
• Hunt, Silverstein, Reed et al. Accuracy of Death
  Certificate in a Population-Based Study of
  asthmatic Patients, JAMA, 1993
• Review of all death certificates autopsy
  reports
• 13 out of state

50
Case-Control Study Design
51
Case-Control Study Design
The Past
The Present
52
Case Control Study Example

• Risk Factors for Deep Vein Thrombosis and
  Pulmonary Embolism
• Heit, Silverstein, Mohr et al
• Arch Intern Med 2000 160 809-15

53
Risk Factors for Deep Vein Thrombosis and
Pulmonary Embolism

• Reported risk factors vary in magnitude and
  independence of each are unknown
• Population based nested case-control study
• Cases 625 Olmsted County residents with first
  lifetime VTE 1/1/1976-12/31/-1990
• Controls 625 Olmsted County residents without
  VTE
• Matched on age, sex, calendar year, and medical
  record number (duration of observation)

54
Risk Factors for Deep Vein Thrombosis and
Pulmonary Embolism

• Mean duration of medical records documentation
  reviewed 34.7 years
• 30 potential independent risk factors
• Conditional logistic regression to identify
  independent risk factors

55
Risk Factors for VTE
56
Threats to ValidityRisk Factors for VTE, 1

• Population-based inception cohort
• Definite VTE confirmed by imaging
• Antemortum and postmortum ascertainment of VTE
• Silverstein, Heit, Mohr et al Trends in the
  Incidence of DVT and PE, Arch Intern Med 1998
• Risk factors
• Large number of pre-specified risk factors
  analyzed
• Matching limits ability to analyze age and sex as
  risk factors
• Unable to analyze immobilization due to potential
  diagnostic suspicion bias
• Protein C had not been reported at time study was
  initiated

57
Threats to ValidityRisk Factors for VTE, 2

• Multivariate analysis
• Large number of potential confounders anticipated
  to control for confounding
• Bootstrap validation of multivariate analysis
• Age effect for risk due to varicose veins
• Treatment effect for risk due to cancer

58
Study Components

• Abstract
• Specific Aims
• Background significance
• Preliminary Data
• Methods
• Human Subjects
• References

59
Methods

• Design
• Setting
• Subjects
• Measurements
• Interventions
• Analysis Sample Size
• Interpretation Limitations

60
Human Subjects

• Risks
• Benefits
• Risks in Relation to Benefits
• Privacy Confidentiality
• Informed Consent
• Women, children, minorities, priority populations

61
IRB Review - Observational Studies

• Record review study (retrospective study)
• Observational study (prospective study)

62
IRB ReviewRecord Review Studies

• Risks
• Criminal or civil liability
• Damaging employability, financial status or
  reputation
• Ethical Issues
• Privacy
• Confidentiality
• Issues fro investigators
• Data collection
• Data security

63
Informed Consent

• Must be obtained, unless IRB approves waiver of
  informed consent
• Minimal risk
• Right and welfare of subjects not adversely
  affected by waiver of informed consent
• Research could not be practically carried out
  without the waiver

64
Category of IRB Review

• Exempt Studies
• Record review studies may be exempt form IRB
  review
• Determined by Chair of IRB
• Minimal risk
• Expedited Review
• Some IRBs allow expedited review for record
  review studies
• Determined by chair of IRB
• Publicly available data
• No personal identifiers and records can not be
  linked to subject
• Full IRB Review

65
Informed Consent

• Protect human subjects/volunteers
• Ensure that study subjects understand benefits
  and risks
• Provide potential subjects with all the
  information to reach a decision on whether or not
  to participate in a research study

66
Informed Consent Requirements

• Benefits
• Alternatives
• Confidentiality
• Compensation
• Medical treatment for injury
• Where to obtain further information
• Contact person
• Voluntary participation

67
Review
68
Objectives

• Describe 4 research designs
• Describe 3 threats to validity
• List advantages of retrospective studies
• List disadvantages to retrospective studies
• Understand requirements and processes for IRB
  review of retrospective studies