Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk factors of chronic diseases

1
Criteria to assess quality of observational
studies evaluating the incidence, prevalence, and
risk factors of chronic diseases

• Minnesota EPC
• Clinical Epidemiology Methods Centre, Ottawa
  Health Research Institute, Ottawa
• Tufts University Medical Center, Boston
• RTI International University of North Carolina,
  Chapel Hill
• Blue Cross and Blue Shield Association, Chicago
• Centers for Disease Control and Prevention,
  Atlanta
• Southern California EPC RAND Corporation, Santa
  Monica
• German Institute for Quality and Efficiency in
  Health Care, Köln, Germany
• Johns Hopkins University, Baltimore

2
Analytical tasks
3
Therapeutic vs. non therapeutic studies
PICOTS Non therapeutic studies Non randomized studies
Population General population Patients, health care recipients
Exposure (interventions, independent variables) Risk factors that are independent of health care (allocation irrelevant) Treatments allocation by investigators or health care professionals
Comparators Levels of risk Another trtm, usual care, standard care, previous trtm
Outcomes Incidence/prevalence of chronic conditions Morbidity, mortality, quality of life, adverse events after treatments
Time Length of exposure Length of trtm, followup off the trtm
Settings General population Health care settings
Comparability (selection bias) Not relevant Flaw
Intention-to treat Not relevant Flaw
4
Systematic literature review

• Systematic reviews and meta-analyses of
  observational nontherapeutic studies published in
  English in core clinical journals (145)
• Published tools (scales or checklists) for
  quality assessment of observational studies (96)
• Annotated bibliography of the methodological
  literature to identify biases and strategies to
  reduce bias in observational studies

5
Development of Checklists

• Separate evaluation of external and internal
  validity
• Exhaustive range of criteria
• Discriminating reporting and methodological
  quality
• Predefined major and minor flaws
• Study and hypotheses level analyses
• Automated reports
• Templates for the protocols of quality
  evaluations with predefined research specific
  quality standards
• Manual, help files, instructions

6
Collaboration with participating experts

• Criteria evaluation face and content validity
• Pilot reliability testing (10 randomly selected
  articles) overall, by topic, by article, by
  domains of external and internal validity, and by
  quality component
• Discriminant validity testing hypotheses of
  detected differences in quality and reporting vs.
  methodological quality

7
Finalizing checklists

• Checklist for studies of incidence or prevalence
  of chronic diseases (MORE)
• Checklist for studies of risk factors of chronic
  diseases (MEVORECH)
• Synthesis of evidence from the studies with major
  flaws
• Level A exclusion from synthesis
• Level B full review and quality abstraction
• Level C separate limited synthesis if major flaws
  detected

8
Quality criteria

• Incidence or prevalence studies
• Aim of study
• Study design
• External Validity
• 1. Sampling of the subjects by the investigators
• 2. Assessment of sampling bias
• 3. Estimation of sampling bias
• 4. Exclusion rate from the analysis
• 5. Sampling bias is addressed in the analysis
• 6. Subject flow
• Internal Validity
• 1. Source to measure outcomes
• 2. Definition of outcomes
• 3. Measurements of outcomes
• 4. Outcomes in race, ethnic, age, or gender
  subpopulations
• 5. Reporting of outcomes

• Risk factors studies
• Aim of study
• Objectives
• Study design
• External Validity
• 1. Sampling of the subjects by the investigators
• 2. Assessment of sampling bias
• 3. Estimation of sampling bias
• 4. Exclusion rate from the analysis
• 5. Sampling bias is addressed in the analysis
• 6. Subject flow
• Internal Validity
• 1.Source to measure outcomes
• 2. Definition of outcomes
• 3. Measurements of outcomes
• 4. Definition of the exposure
• 5.Measurements of the exposure
• 6. Confounding factors

9
Major flaws

• External validity
• Sampling in clinics for
• incidence in the general
• population
• Sampling of controls from different
• population as cases
• Response rate lt40
• Exclusion of gt10 of eligible subjects
• from the analyses
• Reasons to exclude from the analyses
• differ for exposed and not exposed

• Internal validity
• Severity of the outcome can be
• relevant but not assessed in the
• study
• Non valid methods were obtained to
• measure the outcomes, exposure, or
• confounding factors
• Different methods to measure
• exposure (risk factors, independent
• variable) in cases and controls
• No strategies to reduce bias

10
Minor Flaws

• Internal validity
• Outcomes proxy reported
• Obtained from medical records (data mining) or
  administrative database
• Inter-methods validation
• Subjective judgment of reliability
• Intensity/dose of exposure can be relevant but
  not assessed in the study

• External validity
• Random sampling restricted to geographic area
• Convenient sampling
• The authors did not assess sampling bias
• Sampling bias was not addressed in the analysis
• Sampling of controls from health care related
  sources

11
Example of the report
Article _________________
Evaluator _______________
External Validity
Not Reported
Addressing sampling bias Not reported
Sampling General population based Not reported
Major flaw
Exclusion rate from the analysis gt10
Internal Validity
Major flaw
Definition of incidence/prevalence Severity Can be relevant but not assessed in the study
Minor
Definition of incidence/prevalence Frequency of symptoms Can be relevant but not assessed in the study
12
Conclusions

• We propose two checklists for transparent and
  standardized quality assessment in systematic
  reviews and evidence-based reports
• Reasonable validity
• Poor random overall and good component
  reliability in a pilot testing of the articles
  from different topics

13
Future Research with Quality Assessment Using
Proposed Checklists

• Protocols of systematic reviews of nontherapeutic
  observational studies with justified definitions
  of research specific quality standards and
  preplanned reliability testing
• Synthesis of evidence incorporating quality
  assessment (sensitivity of the results to
  quality)
• The evaluation of the level of evidence from
  several observational nontherapeutic studies
• Quality assessment of primary studies should
  improve quality of systematic reviews and
  evidence-based reports