Systematic Review of Prognostic Tests

About This Presentation

Title:

Systematic Review of Prognostic Tests

Description:

Systematic Review of Prognostic Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide – PowerPoint PPT presentation

Number of Views:152

Avg rating:3.0/5.0

Slides: 57

Provided by: AlexP173

Learn more at: http://www.baylorcme.org

Category:

more less

Transcript and Presenter's Notes

Title: Systematic Review of Prognostic Tests

1
Systematic Review of Prognostic Tests

Prepared for
The Agency for Healthcare Research and Quality
(AHRQ)
Training Modules for Medical Test Reviews Methods
Guide
www.ahrq.gov

2
Learning Objectives

Develop the topic and structure the systematic
review of a prognostic test.
Describe the similarities and differences between
the evaluation of diagnostic and prognostic tests
for systematic reviews.
Explain the time-dependent characteristics of
prognostic tests.
Perform appropriate statistical analyses based on
outcome probabilities.

Develop the review topic and framework.
Search for studies.
Select studies and assess quality.
Extract statistics to evaluate test performance .
Conduct meta-analyses of estimates of outcome
probabilities.

The review topic, analytic framework, and Key
Questions can be fundamentally different for
diagnostic and prognostic tests
Diagnostic tests determine whether a patient has
a disease at the time of the test.
A gold standard (i.e., the best available)
reference test is often used to determine true
disease presence or absence.
Prognostic tests predict a patients likelihood
of developing a disease or experiencing a future
medical event.
The reference test is the proportion of study
subjects who actually develop the condition
predicted by the test.

It is often useful to group test results by
implications for decisionmaking when structuring
reviews
Example 1
Structuring based on prognostic test categories
(low/intermediate/high risk) when different
treatments for each category are evaluated
Example 2
Structuring based on precision and accuracy of
outcome probabilities of categories when a
decision model is used as a framework

Considerations for the review summarized in the
general PICOTS framework

Population Clinical spectrum and other characteristics of the prognostic groups including the observed probabilities of the outcome being predicted
Intervention The prognostic test or assessment including all components exactly what the test measures and how it is done how clinical specimens are obtained, processed, and stored for testing exactly what is being predicted and how the test results are to be interpreted and used by test operators
Comparator Standard prognostic tests or assessments for predicting the same outcome.
Outcomes Time-dependent probabilities (time-to-event curves) of what is being predicted, changes or differences in predicted outcome probabilities or reclassification of patients into different prognostic groups, changes in patient care, the net effect of using the prognostic test on patient outcomes, and cost-effectiveness.
Timing The stage in the natural history of outcome development the prognostic test to be used the followup time covered by the prognostic test cover the percentage of patients who experience the outcome usually increases with time, thereby changing the performance characteristics of prognostic tests.
Setting Who will use the prognostic test how the test will be used what the applicable testing scenario is.
Rector TS, Taylor BC, Wilt TJ. Systematic review
of prognostic tests. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
7
Step 1 Developing the Review Topic and Framework
(4 of 4)

In some contexts, it is informative to
Categorize subjects as those who did/did not
experience the predicted outcome during a
specified time interval.
Consider what followup times are informative to
patients, clinicians, or policymakers.
Look back to categorize results of the test.
This step permits assessment of accuracy by
calculating sensitivity, specificity, and
predictive values.
Prognostic tests can be specifically used to
predict response to a treatment.
They predict beneficial/adverse responses to
treatment.
They are also referred to as predictive tests

Studies can relate to one or more categories
Proof of concept Is the test result associated
with a clinically important outcome?
Prospective clinical validation How accurately
does the test predict outcomes in different
patient cohorts, clinical practices, and
prognostic groups?
Incremental predictive value How much does the
new test change predictive probabilities and
increase the discrimination of patients who
did/did not experience the outcome of interest
within a specific time period?
Clinical utility Does the new assessment change
predicted probabilities enough to reclassify many
patients into different prognostic groups that
would be managed differently?
Clinical outcomes Would use of the prognostic
test improve patient outcomes?
Cost-effectiveness Do the improvements in
patient outcomes justify the additional costs of
testing and subsequent medical care?

The first four categories are most readily
addressed by large cohort studies and secondary
analyses of clinical trials.
For the last two categories, randomized control
trials (RCTs) are preferred.
RCTs are rare because of barriers including costs
and the time involved.
In the event that no RCTs are found, use a
decision model to focus on providing best
estimates of outcome probabilities.

There are no reliable, validated methods for
searching the literature for prognostic test
information.
Some strategies use key words and index terms in
studies meeting selection criteria.
Others use search terms such as incidence and
word roots such as prognos.
Terms describing the prognostic test and
condition/event to be predicted should be
included in the search.
Find similar or related article functions can
be useful.
A manual search of references will be needed.
The records of the regulatory agency for
submitted tests can be useful.
The Web site of test producer may be informative.

Unlike diagnostic tests, many prognostic tests
are incorporated into multivariable regression
models/ algorithms for prediction.
Many reports support only an independent
association of a variable with the patient
outcome.
It is difficult to find reports where the test
variable did not add significantly to a
multivariable regression model.
This introduces potential bias by failing to
uncover the lack of association/predictive value.
All studies including the variable should be
sought out.

When prognostic groups are defined by predicted
outcome probabilities
Search for decision analyses, guidelines, or
expert opinions that support outcome probability
thresholds used to define clinically meaningful
prognostic groups (i.e., groups that would be
treated differently in practice because of their
predicted outcome)
Ideally use randomized controlled trials of
interventions in patients selected on the basis
of the prognostic test.
This helps establish the rationale for using the
test to classify patients.
It is not always sufficient to evaluate this use
of the test.

Prognostic indicators vary substantially in
Study design
Subject inclusion criteria
Methods of measuring key variables
Methods of analysis (including definition of
prognostic groups)
Adjustment for covariates
Presentation of results
Reviewer access to patient-level data would allow
more uniform analyses to overcome these
difficulties.
When these data are lacking, following certain
questions can help.

Was the study designed to evaluate the new
prognostic test, or was it a secondary analysis
of data collected for other purposes?
Were the subjects somehow referred or selected
for testing? What was the testing scenario?
Was the clinical population clearly described
including the sampling plan, inclusion and
exclusion criteria, subject participation, and
the spectrum of test results? Did the sample
represent patients who would be tested in
clinical practice?
Did everyone in the samples have a common
starting point for followup with respect to the
outcome of interest, including any treatments
that could affect the outcome being predicted?

Was the prognostic test clearly described and
conducted using a standardized, reliable, and
valid method?
Was the test used and interpreted the same way by
all sites/studies, including any indeterminate
test results?
Were the results ascertained without knowledge of
the outcome?
Were investigators blinded to the test results?
How were previously established prognostic
indicators or other prognostic assessments
included in the study and analyses?
Was the outcome being predicted clearly defined
and ascertained using a standardized, reliable,
and valid method?
How complete was the followup of subjects, and
were losses to followup related to the test
results or the outcome being predicted?
Was the duration of followup adequate?

Were the data used to develop the prognostic
test?
Were the prognostic groups predefined based on
clinically meaningful decision thresholds for
predicted outcome probabilities?
Were the results externally validated using an
independent sample or internally validated using
bootstrap or cross-validation methods?
Were previously established prognostic tests that
were used as comparators fitted to the sample
data in the same manner as the potential new
prognostic test?
Were outcome predictions adjusted for any other
factors? Which ones? How?

Reviewers should explicitly state
inclusion/exclusion criteria
Test comparisons should use data from the same
cohort of subjects to minimize confounding
Within a study, the tests being compared should
be conducted at the same time.
This ensures a common starting point with respect
to the patient outcome being predicted.
Reviewers should note the starting point of each
study.

Prognostic test results and their interpretation
should be ascertained without knowledge of
outcomes to avoid ascertainment bias.
Investigators should be blinded to the results of
a test to avoid selective changes in treatment
that could affect the outcome being predicted.
Be aware of any previously established prognostic
indicators that should be included in the
comparative analysis of potential new tests.
Note any adjustments for covariates that could
make studies more or less comparable.

Fitting a new prognostic test to the sample data
(test development sample) by using these data to
define cutoffs or model relationships to outcomes
and estimate regression coefficients may
Be overly optimistic for estimated predicted
performance
Bias the comparison to an established prognostic
method that was not fitted to the sample data

Summary statistics must be appropriate for the
reviews Key Questions.
Example Hazard ratios from Cox regression
analyses or odds ratios from logistic regression
analyses for associations between tests and
outcomes
Address only the early phases in test development
Often do not discriminate between subjects who
eventually do or do not experience the outcome of
interest
Statistically significant associations
(odds/hazard ratios, relative risks) merely
indicate that more definitive evaluation of a new
predictor is warranted.
Due to these concerns, the questions that a
systematic review can answer by summarizing a
tests association with an outcome are limited.

Discrimination statistics
Indices of discrimination
Estimates of sensitivity, specificity, and area
under the receiver operating characteristic curve
are calculated at one particular time.
They can be calculated retrospectively and
compared when
A new prognostic indicator is added to a
predictive model
A prognostic test is compared to predictions made
by other methods
Retrospective indices of discrimination do not
Summarize predicted outcome probabilities
Directly address questions about the predictions
on the basis of a new prognostic test or its
impact on patient outcomes

Reclassification tables are better suited for
assessments of the clinical impact of prognostic
tests.
They are not as common or reported as often as
discrimination statistics.
When using discrimination statistics, recognize
that they change over time as more patients
develop the outcome being predicted.
Time-dependent measures of discrimination
statistics have been developed (e.g., Harrells
C-statistic) however, examples of reviews using
these measures were not found

Reclassification tables
For example
Patients are placed into prognostic groups based
on their Framingham cardiovascular risk scores.
Reclassification tables are then used to
determine how adding a prognostic test
reclassifies the patients into prognostic groups.
Ideally, the classification of outcome
probabilities into prognostic groups should be
based on outcome probabilities that will lead to
different courses of action.
If not, then reclassifications might not have
clinical utility that is, they may not make a
difference in patient care.

Sample Reclassification Table Based on Predicted
Outcome Probabilities

Grouped mortality probabilities estimated by the first prognostic test Grouped mortality probabilities estimated by the first prognostic test a new prognostic test Grouped mortality probabilities estimated by the first prognostic test a new prognostic test Grouped mortality probabilities estimated by the first prognostic test a new prognostic test
Grouped mortality probabilities estimated by the first prognostic test 0 to 0.10 gt 0.10 Total
0 to 0.10 Patients in prognostic group Mortality predictions using 1st test Mortality prediction using both tests Observed mortality 900 4.0 3.8 3.9 100 (10) 8.0 11.0 12.0 1000 4.4 - 4.7
gt 0.10 Patients in prognostic group Mortality predictions using 1st test Mortality prediction using both tests Observed mortality 100 (25) 15.0 9.0 10.0 300 17.0 19.0 19.0 400 16.5 - 16.8
Total Patients in prognostic group Mortality prediction using both tests Observed mortality 1000 4.3 4.5 400 17.0 17.2 1400 - 8.2
Adding the new test reclassified 10 of the 1000
people originally in the lower risk group and 25
of the 400 people in the higher risk group
Rector TS, Taylor BC, Wilt TJ. Systematic review
of prognostic tests. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
.
25
Step 4 Extracting Statistics To Evaluate Test
Performance (6 of 12)

Reclassification tables provide information
about
Observed outcome probabilities in each prognostic
group
Predicted probabilities
Drawbacks
Information is often limited to a single
follow-up time.
The precision of estimates may not be reported.
Differences between estimated probabilities and
observed outcomes for each prognostic group may
be analyzed by using the chi-square
goodness-of-fit test.
The results will not help determine if
differences in predicted and observed
probabilities are better when a new prognostic
test is added.

The net reclassification improvement statistic is
a summary statistic of separate reclassification
tables.
One table is for those who did experience the
outcome event within a particular time period.
The other table is for those who did not
experience the outcome event.
In a group experiencing the outcome of interest
(e.g., those who died), net improvement the
proportion of patients reclassified into a higher
probability group minus the proportion
reclassified into a lower probability group.
In a 2 x 2 table, this is an estimated change in
test sensitivity.

For those who do not experience an outcome (e.g.,
those who survived), the inverse is used net
improvement the proportion reclassified into a
lower-risk group minus the proportion
reclassified into the higher-risk group.
In a 2x2 table, this is the estimated change in
test specificity.
The net reclassification improvement (NRI)
statistic is the sum of net improvement in
classification in patients who did or did not
experience the outcome.
The integrated discrimination index (IDI) uses
mean changes in individual predicted
probabilities instead of net improvement in the
calculations above.

The continuous formulation version of the net
reclassification improvement (NRI) statistic
calculates the
Probability of predicted event among those who
have an increase in predicted probability after
new test
Probability of predicted event among those who
have a decrease in predicted probability after
new test
Event probability in the overall sample
With this version of the NRI statistic
The NRI statistic can be estimated by
time-to-event analysis, but the three
probabilities still represent only a single point
of followup.
The NRI statistic does not require clinically
meaningful prognostic categories, since not all
increases/decreases in probability prompt a
change in patient management.
The NRI statistic focuses instead on subjects
with a higher or lower predicted outcome
probability when a new test is used.

When pooling net reclassification improvement
(NRI) or integrated discrimination index (IDI)
estimates from different studies, the following
should not differ substantially
Characteristics of prognostic groups
Definition of outcome event
Overall probability of event
Followup time

Predictive values
Treatment decisions based on outcome
probabilities are often dichotomous.
For example, treat those at high risk and do
not treat those at low risk
If patients would be treated because a test
indicates they are at high risk, then the
observed time-dependent percentages of patients
developing the outcome (without treatment) are
positive predictive values.
That is, the proportion of those who have a
positive prognostic test result who end up
having the event.
If patients would not be treated because they are
at low risk, then 1 (observed outcome
probabilities) negative predictive values.
That is, the proportion of those who have a
negative prognostic test result who do not end
up having the event.

Predictive values (continued)
For a single point of followup, positive and
negative predictive values can be compared via
methods used for diagnostic tests.
Ratios of positive and negative predictive values
of two prognostic tests are often summarized
along with confidence intervals.
The regression model of Leisenring et al. (2000)
may be used to determine how patient
characteristics relate to relative predictive
values.
Time-to-event curves comparing predictive values
of two prognostic tests are available if
necessary.

Rector TS, Taylor BC, Wilt TJ. Systematic review
of prognostic tests. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Leisenring W, Alonzo T, Pepe MS. Biometrics
2000 Jun56(2)345-51. PMID 10877288.
32
Step 5 Meta-analysis of Estimates of Outcome
Probabilities (1 of 2)

Randomized controlled trials (RCTs) designed to
demonstrate net improvement in patient outcomes
and cost-effectiveness are the most definitive
level of evidence.
Many prognostic test studies do not provide this
level.
Systematic reviews can provide estimates of
outcome probabilities for decision models
instead.
Estimates from either RCTs or observational
studies can be used, provided that prognostic
groups are well characterized and similar.
Meta-analysis can provide more precise estimates
.
Additionally, meta-analysis of estimates of
outcome probabilities in a prognostic group
extracted from several studies can provide
insight into
Stability of the estimates
Whether variation is related to prognostic group
characteristics

Methods have been developed to combine estimates
of outcome probabilities from different studies.
Dears method (1994)
Uses a fixed-effects regression model
Arends method (2008)
Is similar to a DerSimonian-Laird random-effects
model
Is used when there is only one common followup
time for all studies/prognostic groups

Rector TS, Taylor BC, Wilt TJ. Systematic review
of prognostic tests. In Methods guide for
medical test reviews. Available at
www.effectivehealthcare.ahrq.gov/medtestsguide.cfm
. Arends LR, Hunink MG, Stijnen T. Stat Med 2008
Sep 3027(22)4381-96. PMID 18465839. Dear KB.
Biometrics 1994 Dec50(4)989-1002. PMID
7787011. DerSimonian R, Laird N. Control Clin
Trials 1986 Sep7(3)177-88. PMID 3802833.
34
Key Messages (1 of 2)

Methods used to conduct systematic reviews of
prognostic tests are not well established.
The intended use of the prognostic test should be
specified predicted probabilities need to be
classified into clinically meaningful groups that
are described in detail (including outcome
probabilities).
Many published reports focus on associations
between prognostic indicators and patient
outcomes (the first stage of development of
prognostic tests) such studies have limited
clinical value for a review.
Criteria for evaluating the quality of studies of
prognostic tests have not been firmly
established.
Reviewers can adapt criteria developed for
diagnostic tests.

Given differences between diagnostic tests
(current state of disease) and prognostic tests
(future state of disease), common evaluation
statistics used with diagnostic tests are not as
informative for prognostic tests.
These include point estimates of
sensitivity/specificity and area under the
receiver operating characteristic (ROC) curves.
The most pertinent summary statistics for
prognostic tests are
The time-dependent observed outcome probabilities
The closeness of the prognostic groups predicted
probabilities to observed outcomes
How the use of a new test reclassifies patients
into different groups and improves predictive
accuracy and overall outcomes
Methods for comparing and summarizing predictive
performance of prognostic tests need further
development and widespread use.

What is the relationship between predictive and
prognostic tests?
The terms are essentially synonymous.
Predictive tests are a category of prognostic
tests.
Prognostic tests are a category of predictive
tests.

37
Practice Question 1 (2 of 2)

Explanation for Question 1
The correct answer is b. Predictive tests, which
are a category of prognostic tests, can be used
to predict beneficial or adverse responses to
treatment.

38
Practice Question 2 (1 of 2)

Which of the following four categories of
prognostic tests is NOT most readily addressed by
large cohort studies and secondary analyses of
clinical trials?
Clinical outcomes
Clinical utility
Prospective clinical validation
Proof of concept
None of the above

39
Practice Question 2 (2 of 2)

Explanation for Question 2
The correct answer is a. Questions that pertain
to clinical outcomes or cost-effectiveness are
generally better answered by randomized
controlled trials. The other three categories
listed may often be addressed by less costly
cohort studies or secondary analyses of clinical
trials.

40
Practice Question 3 (1 of 2)

There are relatively few reports where the test
variable did not add significantly to a
multivariable regression model.
True
False

41
Practice Question 3 (2 of 2)

Explanation of Question 3
The correct answer is true. The relative paucity
of published findings in which a test variable
did not add significantly to a multivariable
regression model suggests that there may be
potential bias by failing to publish lack of
association or predictive value.

42
Practice Question 4 (1 of 2)

Prognostic indicators vary substantially in which
of the following
Study design
Subject inclusion criteria
Methods of measuring key variables
Adjustment for covariates
Presentation of results
All of the above

43
Practice Question 4 (2 of 2)

Explanation of Question 4
The correct answer is f. Wide variability in one
or more of these factors presents a challenge to
selecting studies and assessing quality. Reviewer
access to patient-level data would allow uniform
analyses to overcome these difficulties. When
lacking the ability to access patient-level data,
suggestions for judging the quality of individual
studies of prognostic tests can be followed.

44
Practice Question 5 (1 of 2)

Examples of indices of discrimination do not
include
95-percent confidence intervals
Estimates of sensitivity and specificity
Area under the receiver operating characteristic
(ROC) curve
All of the above

45
Practice Question 5 (2 of 2)

Explanation of Question 5
The correct answer is a. 95-percent confidence
intervals provide a measure of the precision of a
given statistic, but are not, in themselves, a
discrimination statistic.

46
Authors

This presentation was prepared by Brooke
Heidenfelder, Rachael Posey, Lorraine Sease, Remy
Coeytaux, Gillian Sanders, and Alex Vaz, members
of the Duke University Evidence-based Practice
Center.
The module is based on Rector TS, Taylor BC, Wilt
TJ. Systematic review of prognostic tests. In
Chang SM and Matchar DB, eds. Methods guide for
medical test reviews. Rockville, MD Agency for
Healthcare Research and Quality June 2012. p.
12.1-13. AHRQ Publication No. 12-EHC017.
Available at www.effectivehealthcare.ahrq.gov/medt
estsguide.cfm.

47
References (1 of 10)

Arends LR, Hunink MG, Stijnen T. Meta-analysis of
summary survival curve data. Stat Med 2008 Sep
3027(22)4381-96. PMID 18465839.
Altman DG. Systematic reviews of evaluations of
prognostic variables. BMJ 2001 Jul
28323(7306)224-8. PMID 11473921.
Altman DG, Riley RD. Primer an evidence-based
approach to prognostic markers. Nat Clin Pract
Oncol 2005 Sep2(9)466-72. PMID 16265015.
Cook NR. Use and misuse of the receiver operating
characteristic curve in risk prediction.
Circulation 2007 Feb 20115(7)928-35. PMID
17309939.
Cook NR. Statistical evaluation of prognostic
versus diagnostic models beyond the ROC curve.
Clin Chem 2008 Jan54(1)17-23. PMID18024533.
Cook NR, Ridker PM. Advances in measuring the
effect of individual predictors of cardiovascular
risk the role of reclassification measures. Ann
Intern Med 2009 Jun 2150(11)795-802. PMID
19487714.

48
References (2 of 10)

Dear KB. Iterative generalized least squares for
meta-analysis of survival data at multiple times.
Biometrics 1994 Dec50(4)989-1002. PMID
7787011.
DerSimonian R, Laird N. Meta-analysis in clinical
trials. Control Clin Trials 1986 Sep7(3)177-88.
PMID 3802833.
Earle CC, Pham B, Wells GA. An assessment of
methods to combine published survival curves. Med
Decis Making 2000 Jan-Mar20(1)104-11. PMID
10638543.
Feng Z, Prentice R, Srivastava S. Research issues
and strategies for genomic and proteomic
biomarker discovery and validation a statistical
perspective. Pharmacogenomics 2004
Sep5(6)709-19. PMID 15335291.
Fibrinogen Studies Collaboration. Measures to
assess the prognostic ability of the stratified
Cox proportional hazards model. Stat Med 2009 Feb
128(3)389-411. PMID 18833567.

49
References (3 of 10)

Fowkes FG, Murray GD, Butcher I, et al. Ankle
brachial index combined with Framingham Risk
Score to predict cardiovascular events and
mortality a meta-analysis. JAMA 2008 Jul
9300(2)197-208. PMID 18612117.
Groenveld HF, Januzzi JL, Damman K, et al. Anemia
and mortality in heart failure patients a
systematic review and meta-analysis. J Am Coll
Cardiol 2008 Sep 252(10)818-27. PMID 18755344.
Grundy SM, Cleeman JI, Merz CN, et al National
Heart, Lung, and Blood Institute American
College of Cardiology Foundation American Heart
Association. Implications of recent clinical
trials for the National Cholesterol Education
Program Adult Treatment Panel III guidelines.
Circulation 2004 Jul 13110(2)227-39. PMID
15249516.
Hall PA, Going JJ. Predicting the future a
critical appraisal of cancer prognosis studies.
Histopathology 1999 Dec35(6)489-94. PMID
10583572.

50
References (4 of 10)

Hlatky MA, Greenland P, Arnett DK, et al.
Criteria for evaluation of novel markers of
cardiovascular risk a scientific statement from
the American Heart Association. Circulation 2009
May 5119(17)2408-16. PMID 19364974.
Hayden JA, Cote P, Bombardier C. Evaluation of
the quality of prognosis studies in systematic
reviews. Ann Intern Med 2006 Mar
21144(6)427-37. PMID 16549855.
Ingelsson E, Schaefer EJ, Contois JH, et al.
Clinical utility of different lipid measures for
prediction of coronary heart disease in men and
women. JAMA 2007 Aug 15298(7)776-85. PMID
17699011.
Ingui BJ, Rogers MA. Searching for clinical
prediction rules in MEDLINE. J Am Med Inform
Assoc 2001 Jul-Aug8(4)391-7. PMID 11418546.

51
References (5 of 10)

Janes H, Pepe MS. Adjusting for covariates in
studies of diagnostic, screening, or prognostic
markers an old concept in a new setting. Am J
Epidemiol 2008 Jul 1168(1)89-97. PMID
18477651.
Janes H, Pepe MS, Gu W. Assessing the value of
risk predictions by using risk stratification
tables. Ann Intern Med 2008 Nov
18149(10)751-60. PMID 19017593.
Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Almost all
articles on cancer prognostic markers report
statistically significant results. Eur J Cancer
2007 Nov43(17)2559-79. PMID 17981458.
Kyzas PA, Loizou KT, Ioannidis JP. Selective
reporting biases in cancer prognostic factor
studies. J Natl Cancer Inst 2005 Jul
2097(14)1043-55. PMID 16030302.
Leisenring W, Alonzo T, Pepe MS. Comparisons of
predictive values of binary medical diagnostic
tests for paired designs. Biometrics 2000
Jun56(2)345-51. PMID 10877288.

52
References (6 of 10)

Mackillop WJ, Quirt CF. Measuring the accuracy of
prognostic judgments in oncology. J Clin
Epidemiol 1997 Jan50(1)21-9. PMID 9048687.
McShane LM, Altman DG, Sauerbrei W, et al
Statistics Subcommittee of the NCI-EORTC Working
Group on Cancer Diagnostics. Reporting
recommendations for tumor marker prognostic
studies (REMARK). J Natl Cancer Inst 2005 Aug
1797(16)1180-4. PMID 16106022.
Meigs JB, Shrader P, Sullivan LM, et al. Genotype
score in addition to common risk factors for
prediction of type 2 diabetes. N Engl J Med 2008
Nov 20359(21)2208-19. PMID 19020323.
Pencina MJ, D'Agostino RB. Overall C as a measure
of discrimination in survival analysis model
specific population value and confidence interval
estimation. Stat Med 2004 Jul 1523(13)2109-23.
PMID 15211606.

53
References (7 of 10)

Pencina MJ, DAgostino RB Sr, DAgostino RB Jr,
et al. Evaluating the added predictive ability of
a new marker from area under the ROC curve to
reclassification and beyond. Stat Med 2008 Jan
3027(2)157-72, discussion 207-12. PMID
17569110.
Pepe MS. Section 9.2 Incorporating the time
dimension. In The statistical evaluation of
medical tests for classification and prediction.
New York, NY Oxford University Press USA 2004.
p. 259-67.
Pepe MS, Feng Z, Janes H, et al. Pivotal
evaluation of the accuracy of a biomarker used
for classification or prediction standards for
study design. J Natl Cancer Inst 2008 Oct
15100(20)1432-8. PMID 18840817.
Pepe MS, Janes HE. Gauging the performance of
SNPs, biomarkers, and clinical factors for
predicting risk of breast cancer. J Natl Cancer
Inst 2008 Jul 16100(14)978-9. PMID 18612128.

54
References (8 of 10)

Pepe MS, Janes H, Longton G, et al. Limitations
of the odds ratio in gauging the performance of a
diagnostic, prognostic, or screening marker. Am J
Epidemiol 2004 May 1159(9)882-90. PMID
15105181.
Pepe MS, Zheng Y, Jin Y, et al. Evaluating the
ROC performance of markers for future events.
Lifetime Data Anal 2008 Mar14(1)86-113. PMID
18064569.
Poses RM, Cebul RD, Collins M, et al. The
importance of disease prevalence in transporting
clinical prediction rules. The case of
streptococcal pharyngitis. Ann Intern Med 1986
Oct105(4)586-91. PMID 3530079.
Rector TS, Taylor BC, Wilt TJ. Systematic review
of prognostic tests. In Chang SM and Matchar DB,
eds. Methods guide for medical test reviews.
Rockville, MD Agency for Healthcare Research and
Quality June 2012. p. 12.1-13. AHRQ Publication
No. 12-EHC017. Available at www.effectivehealthcar
e.ahrq.gov/medtestsguide.cfm.

55
References (9 of 10)

Ridker PM, Buring JE, Rifai N, et al. Development
and validation of improved algorithms for the
assessment of global cardiovascular risk in
women the Reynolds Risk Score. JAMA 2007 Feb
14297(6)611-9. PMID 17299196.
Ridker PM, Paynter NP, Rifai N, et al. C-reactive
protein and parental history improve global
cardiovascular risk prediction the Reynolds Risk
Score for men. Circulation 2008 Nov
25118(22)2243-51, 4p following 2251. PMID
18997194.
Riesterer O, Milas L, Ang KK. Use of molecular
biomarkers for predicting the response to
radiotherapy with or without chemotherapy. J Clin
Oncol 2007 Sep 1025(26)4075-83. PMID 17827456.

56
References (10 of 10)

Riley RD, Abrams KR, Sutton AJ, et al. Reporting
of prognostic markers current problems and
development of guidelines for evidence-based
practice in the future. Br J Cancer 2003 Apr
2288(8)1191-8. PMID 12698183.
Sinuff T, Adhikari NK, Cook DJ, et al. Mortality
predictions in the intensive care unit comparing
physicians with scoring systems. Crit Care Med
2006 Mar34(3)878-85. PMID 16505667.
Speight PM, Palmer S, Moles DR, et al. The
cost-effectiveness of screening for oral cancer
in primary care. Health Technol Assess 2006
Apr10(14)1-144, iii-iv. PMID 16707071.
Ware JH. The limitations of risk factors as
prognostic tools. N Engl J Med 2006 Dec
21355(25)2615-7. PMID 17182986.