The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomized Studies in Meta-Analysis

1
The Newcastle-Ottawa Scale (NOS) for Assessing
the Quality of Nonrandomized Studies in
Meta-Analysis

• G. Wells, B. Shea, D. OConnell, J. Robertson,
  J. Peterson, V. Welch, M. Losos, P. Tugwell

2

• Development
• Applications
• Current Developments

3
Development Item Selection

• Newcastle quality assessment form
• Ottawa comprehensive list
• Panel review
• Critical review by experts

4
Development Grouping Items

• Cohort studies
• Selection of cohorts
• Comparability of cohorts
• Assessment of outcome
• Case-Control studies
• Selection of case and controls
• Comparability of cases and controls
• Ascertainment of exposure

5
Development Identifying Items

• Identify high quality choices with a star
• A maximum of one star for each h item within
  the Selection and Exposure/Outcome
  categories maximum of two stars for
  Comparability

6
(No Transcript)
7
(No Transcript)
8
Newcastle-Ottawa Quality Assessment Scale
Case-Control Studies

• Selection (4)
• Comparability (1)
• Exposure (3)
• A study can be awarded a maximum of one star for
  each numbered item within the Selection and
  Exposure categories. A maximum of two stars can
  be given for Comparability

9
Selection

• 1. Is the case definition adequate?
• a) yes, with independent validation ?
• b) yes, eg record linkage or based on self
  reports
• c) no description
• 2. Representativeness of the cases
• a) consecutive or obviously representative
  series of cases ?
• b) potential for selection biases or not
  stated
• 3. Selection of Controls
• a) community controls ?
• b) hospital controls
• c) no description
• 4. Definition of Controls
• a) no history of disease (endpoint) ?
• b) no description of source

e.g. ICD codes in database or self-report with
no reference to primary record or no description
gt1 person/record/time/process to extract
information, or reference to primary record
source such as x-rays or medical/hospital
records
10
Comparability

• 1. Comparability of cases and controls on the
  basis of the design or analysis
• a) study controls for ___________ (select
  the most important factor) ?
• b) study controls for any additional factor
  (This criteria could be modified to indicate
  specific control for a second important factor.)
  ?

11
Exposure

• 1. Ascertainment of exposure
• a) secure record (eg surgical records) ?
• b) structured interview where blind to
  case/control status ?
• c) interview not blinded to case/control
  status
• d) written self report or medical record
  only
• e) no description
• 2. Same method of ascertainment for cases and
  controls
• a) yes ?
• b) no
• 3. Non-Response Rate
• a) same rate for both groups ?
• b) non respondents described
• c) rate different and no designation

12
Newcastle-Ottawa Quality Assessment Scale
Cohort Studies

• Selection (4)
• Comparability (1)
• Outcome (3)
• A study can be awarded a maximum of one star for
  each numbered item within the Selection and
  outcome categories. A maximum of two stars can
  be given for Comparability

13
Selection

• 1. Representativeness of the exposed cohort
• a) truly representative of the average
  ___________ (describe) in the community ?
• b) somewhat representative of the average
  ___________ in the community ?
• c) selected group of users eg nurses,
  volunteers
• d) no description of the derivation of the
  cohort
• 2. Selection of the non exposed cohort
• a) drawn from the same community as the
  exposed cohort ?
• b) drawn from a different source
• c) no description of the derivation of the
  non exposed cohort
• 3. Ascertainment of exposure to implants
• a) secure record (eg surgical records) ?
• b) structured interview ?
• c) written self report
• d) no description
• 4. Demonstration that outcome of interest was
  not present at start of study
• a) yes ?

In the case of mortality studies, outcome of
interest is still the presence of a disease/
incident, rather than death that is a statement
of no history of disease or incident earns a star
14
Comparability

• 1. Comparability of cohorts on the basis of the
  design or analysis
• a) study controls for ___________ (select
  the most important factor) ?
• b) study controls for any additional factor (This
  criteria could be modified to indicate specific
  control for a second important factor.) ?

15
Outcome

• 1. Assessment of outcome
• a) independent blind assessment ?
• b) record linkage ?
• c) self report
• d) no description
• 2. Was follow up long enough for outcomes to
  occur
• a) yes (select an adequate follow up
  period for outcome of interest) ?
• b) no
• 3. Adequacy of follow up of cohorts
• a) complete follow up - all subjects
  accounted for ?
• b) subjects lost to follow up unlikely to
  introduce bias - small number lost - gt ___
  (select an adequate ) follow up, or description
  of those lost) ?
• c) follow up rate lt ___ (select an
  adequate ) and no description of those lost
• d) no statement

16
Applications

• Assess quality of nonrandomized studies
• Incorporate assessments in interpretation of
  meta-analytic results
• Design, content and ease of use

17
Long Term Hormone Replacement Therapy and
Coronary Heart Disease Events
18
Steps of a Cochrane Systematic Review

• Clearly formulated question
• Comprehensive data search
• Unbiased selection and abstraction process
• Critical appraisal of data
• Synthesis of data
• Perform sensitivity and subgroup analyses if
  appropriate and possible
• Prepare a structured report

19
Objective

• Is there a relationship between hormone
  replacement therapy and the incidence of coronary
  heart disease in postmenopausal women

20
Inclusion Criteria

• Types of studies
• case-control, cohort or cross-sectional studies
• Population
• postmenopausal women
• Intervention
• women exposed to hormone replacement therapy
  (estrogen or estrogen progesterone)
• ever, current, past
• Outcomes
• coronary heart disease (events)
• fatal, non-fatal, both

21
Steps of a Cochrane Systematic Review

• Clearly formulated question
• Comprehensive data search
• Unbiased selection and abstraction process
• Critical appraisal of data
• Synthesis of data
• Perform sensitivity and subgroup analyses if
  appropriate and possible
• Prepare a structured report

22
Search Strategy

• Electronic Search of
• MEDLINE (1966 to May 2000)
• Current Contents (to May 2000)
• Other Data Sources
• review of references cited in retrieved articles

23
Steps of a Cochrane Systematic Review

• Clearly formulated question
• Comprehensive data search
• Unbiased selection and abstraction process
• Critical appraisal of data
• Synthesis of data
• Perform sensitivity and subgroup analyses if
  appropriate and possible
• Prepare a structured report

24
Data Extraction

• 2 independent reviewers selected trials
• 2 independent reviewers extracted data using
  pre-determined forms
• study design
• population characteristics
• exposure to implants
• outcomes measures
• results
• differences resolved by consensus

25
Results

• 16 case-control or cross-sectional
• 14 cohort

26
Quantification of Effects

• Exposure (ever, current, past)
• Outcome (fatal, non-fatal, both)
• Effect estimates (EE)
• Relative Risk (RR)
• Odds Ratio (OR)
• Adjusted effect estimates
• Effects vs population, follow-up periods, etc.
  (homogeneity)

27
Steps of a Cochrane Systematic Review

• Clearly formulated question
• Comprehensive data search
• Unbiased selection and abstraction process
• Critical appraisal of data
• Synthesis of data
• Perform sensitivity and subgroup analyses if
  appropriate and possible
• Prepare a structured report

28
Cohort Star Template
Selection Comparability Outcome
Avila / 90
Bush / 87
Cauley / 97
Criqui / 98
Ettinger / 96
Folsom / 95
Grodstein / 96
Henderson / 91
Lafferty / 94
Lauritzen / 83
Petitti / 87
Sourander / 98
Wilson / 85
Wolf / 96
29
Case-Control Star Template
Selection Comparability
Exposure
Adam / 81
Beard / 89
Croft / 89
Grodstein / 97
Heckbert / 97
LaVecchia / 87
Mann / 94
Pfeffer / 78
Rosenberg / 76
Rosenberg / 80
Rosenberg / 93
Ross / 81
Sidney / 97
Szklo / 84
Talbott / 77
Thompson / 89
30
Adjusted Effect Estimates for Coronary Heart
Disease (All Events) (HRT Estrogen Current
Use)Case-Control Studies
Selection Comparability Exposure
31
Adjusted Effect Estimates for Coronary Heart
Disease (All Events) (HRT Estrogen Past
Use)Case-Control Studies
Selection Comparability Exposure
32
Adjusted Effect Estimates for Coronary Heart
Disease (All Events) (HRT Estrogen Ever
Use)Case-Control Studies
Selection Comparability Exposure
33
Adjusted Effect Estimates for Coronary Heart
Disease (All Events) (HRT Estrogen Progestin
Ever Use)Case-Control Studies
Selection Comparability Exposure
34
Adjusted Effect Estimates for Coronary Heart
Disease (All Events) (HRT Estrogen Current
Use)Cohort Studies
Selection Comparability Outcome
35
Adjusted Effect Estimates for Coronary Heart
Disease (All Events) (HRT Estrogen Ever
Use)Cohort Studies
Selection Comparability Outcome
36
Current Development Validity

• Face/content validity
• Criterion validity
• compare to more comprehensive scales
• compare to expert judgement
• Construct validity
• external criteria
• convergent validity
• divergent validity
• internal structure
• factorial validity

37
Current Development Reliability

• Inter-rater reliability
• Intra-rater reliability

38
Future Development Scoring

• Identify threshold score distinguishing between
  good and poor quality studies

39
The Newcastle-Ottawa Scale (NOS) for Assessing
the Quality of Nonrandomized Studies in
Meta-Analysis

• www.lri.ca
• NOS Quality Assessment Scales
• Case-control studies
• Cohort studies
• Manual for NOS Scales