Validating Disease Progression and CostEffectiveness Models: How to Build Credibility and Avoid Ugly

1
Validating Disease Progression and
Cost-Effectiveness Models How to Build
Credibility and Avoid Ugly Outcomes
Presented at Health Protection Research
Initiative Grantee Meeting, Atlanta, GA April
21, 2006 Presented by Tom Hoerger, RTI-UNC
Center of Excellence in Health Promotion Economics
P.O. Box 12194 3040 Cornwallis Road
Research Triangle Park, NC 27709Phone
919-541-7146 Fax [phone]
tjh_at_rti.org www.rti.org RTI International is
a trade name of Research Triangle Institute.
2
Why is Validation Necessary?
3
A Typical Cost-effectiveness Model for Chronic
Disease
4
Challenges We Face

• It works! It works!
• Deadlines, budgets
• Its hard work
• Relatively little in the way of formal direction

5
Some Terms

• Internal testing (first-order validation,
  reliability, verification) are equations
  programmed correctly, with no bugs?
• Internal (second-order) validation (calibration)
  are inputs and outputs consistent with data used
  to create the model?
• External (third-order) validation does model
  match other data, not explicitly used to create
  the model?
• Cross-validation does the model reach the same
  conclusions as other models?
• Predictive validity does the model make accurate
  predictions about future events?
• Face validity does it pass the smell test,
  intuition?

6
Strategies
7
Internal Testing

• Run the model more than once
• Markov model same results
• Agent-based model similar results
• Review the code
• Have someone else review the code
• Sensitivity analyses
• Set discount rate 0
• Apply extreme values
• Do the results make sense?

8
Internal Testing (contd)

• Look at model outcomes very critically,
  wholistically and by component
• Outcomes/complications
• QALYs, LYs
• Costs
• Design model with validation in mind
• Be skeptical
• Cost-saving intervention
• Productivity costs dominate

9
Internal Validation

• Individual model equations should fit the data
  used to derive them
• Collectively, the model equations should also fit
  the data
• Plot model outcomes

10
Second-Order Validation for a Key Complication
11
Second-Order Validation (All variables)
12
Advanced Internal Validation Simulate a Trial

• Create the trial population
• Run the model for the trial duration
• Compare the results statistically

13
External Validation

• Identify appropriate data sets, trials
• Compare the model results to the external data
  set or trial
• Will probably be worse fit than internal
  validation

14
Combined Chart 2nd- and 3rd-Order Validation
15
External Validation Costs

• Often very difficultand very important
• Compare to external data sets
• Predicted costs to published cost estimates
• Estimated hospital days to actual days
• Can sometimes be done for budget-impact analyses

16
Cross-Model Validation

• Useful for new models
• Are we in the ballpark?
• May be only source for cost validation
• Be able to explain differences
• Useful from a methodological standpoint
• Economic models of colorectal cancer screening
  (IOM, NCI)
• The Mt. Hood Challenge (diabetes models)

17
Face Validity Do People Believe in your Model?

• Clinical input
• Are the disease pathways correct?
• Have you used the right data, trials, etc.?
• Transparency
• Sensitivity Analyses
• Provide assurance that the model changes as
  expected
• Technical report
• Formal validation section
• Scientific review
• Publication
• Do people use the model for policy?

18
Conclusion

• Model validation is necessary
• Plan for it from the start
• To achieve excellence, we need to do a better and
  more systematic job validating our disease and
  cost-effectiveness models
• Strategies are available
• Presentation features relatively simple
  approaches
• Need to develop better, more advanced, and more
  systematic approaches

19
A Little Reassurance (Weinstein et al.)

• Models should never be regarded as complete or
  immutable. They should be repeatedly updated, and
  sometimes abandoned and replaced, as new evidence
  becomes available to inform their structure or
  input values...

20
References

• Weinstein MC, O'Brien B, Hornberger J, et al.
  Principles of good practice of decision analytic
  modeling in health care evaluation Report of the
  ISPOR Task Force on Good Research
  Practices-Modeling Studies. Value Health 2003
  69-17
• Palmer AJ et al (2004). Validation of the CORE
  Diabetes Model Against Epidemiological and
  Clinical Studies. Current Medical Research and
  Opinions. 20(Suppl. 1) S27-S40.

21
References

• Leonard Schlessinger and David M. Eddy.
  Archimedes a new model for simulating health
  care systemsthe mathematical formulation.
  Journal of Biomedical Informatics 35 (2002)
  3750.
• Philip Cooley, Disease Model Validation Issues,
  RTI.
• American Diabetes Association Consensus Panel,
  Guidelines for Computer Modeling of Diabetes and
  Its Complications. Diabetes Care, 27(9)
  2262-2265.

22
Example 1 The Overworked, Undertrained RA

• Compared our diabetes CE model to previous model
• Similar results for no intervention arm
• Much smaller intervention effect
• Big effect on incremental CE ratio
• Key intervention parameter based on nonlinear
  equation
• Result risk incr. from 10 incr. in HBA1c ?
  risk reduction from 10 risk reduction
• Cause(?) RA didnt realize the problem

23
Example 2 The Demanding Client

• Client changed model specification just prior to
  policy group meeting
• Restructured model (long, long weekend!)
• Policy group impressed with findings
• Client subsequently found model error
• Corrected results conflicted with initial policy
  recommendations