Analysis Alongside A Randomized Trial

1
Analysis Alongside A Randomized Trial

• Todd Wagner, PhD
• May 2009

2
Objectives

• At the end of the class, you should
• Understand how to set up your datasets
• Be familiar with analytical methods
• Want to hear the class on decision modeling

3
Dominance

• No need to calculate an incremental cost
  effectiveness ratio if
• Intervention is more effective and less expensive
  than control
• Intervention is more expensive and less effective
  than control
• But
• Check whether dominance exists within subgroups
• Does dominance persists after including
  uncertainty

4
Incremental Cost-Effectiveness Ratio (ICER)

• Calculate in the absence of dominance

CostEXP - CostCONTROL _____________________ QALYEX
P -QALYCONTROL
5
Cost Data

• Have costs consistent with the stated perspective
• Societal
• Health care utilization
• Patient costs
• Caregiver costs
• Intervention costs (direct plus indirect)

6
Common Hurdle

• Many of the parameters in the analysis are based
  on assumptions (e.g., wage rate, mileage costs)
• Consider whether these assumptions are biased
  toward/against the intervention
• Ideally want to include uncertainty
• One way sensitivity analysis
• Statistical methods (bootstrapping, Cooks D)

7
Another hurdle

• Include disease-related utilization or all health
  care utilization?
• How do you define disease-related?
• Recommend look at all utilization for the CEA

8
Labor Outcomes

• Productivity employment is not in the cost
  estimate
• Anyone remember why?
• If labor outcomes are important, still collect
  them but report them separately.

9
Dataset

• You need to create a long dataset
• You need to have an group indicator for
  experiment (1) and control (0)
• You need to have cost and outcome information
• You need subgroup identifiers

10
Data
11
Analysis

• . tabstat totcost followup if exp1, by(papres)
  stat(mean) format(3.2f)
• Summary statistics mean
• by categories of papres (initial pap results)
• papres totcost followup
• -------------------------------
• ASCUS/AGUS 347.31 0.57
• LGSIL 373.97 0.64
• HGSIL 404.72 0.87
• -------------------------------
• Total 355.45 0.61
• --------------------------------
• . tabstat totcost followup if exp0, by(papres)
  stat(mean) format(3.2f)
• Summary statistics mean
• by categories of papres (initial pap results)

12
Hypothesis Testing
. reg totcost exp Source SS
df MS Number of obs
348 -------------------------------------------
F( 1, 346) 292.64 Model
6728200.2 1 6728200.2 Prob gt F
0.0000 Residual 7954890.66 346
22991.0135 R-squared
0.4582 ------------------------------------------
- Adj R-squared 0.4567 Total
14683090.9 347 42314.3829 Root
MSE 151.63 ------------------------------
------------------------------------------------
totcost Coef. Std. Err. t
Pgtt 95 Conf. Interval ------------------
--------------------------------------------------
--------- exp 278.1664 16.26051
17.11 0.000 246.1845 310.1483
_cons 77.27982 11.62933 6.65 0.000
54.40675 100.1529 ----------------------------
--------------------------------------------------
Logistic regression
Number of obs 348
LR chi2(1)
28.42
Prob gt chi2 0.0000 Log
likelihood -226.30785
Pseudo R2 0.0591 --------------------
--------------------------------------------------
-------- followup Odds Ratio Std. Err.
z Pgtz 95 Conf. Interval ------------
-------------------------------------------------
---------------- exp 3.225974
.7243906 5.22 0.000 2.077419
5.009538 -----------------------------------------
-------------------------------------
13
Conclusion

• Intervention is more costly and more effective
• Next Steps
• Include uncertainty
• Check for subgroups and
• calculate the ICER

14
ICER Calculation

• Intervention Group
• papres totcost followup
• -------------------------------
• ASCUS/AGUS 347.31 0.57
• LGSIL 373.97 0.64
• HGSIL 404.72 0.87
• -------------------------------
• Total 355.45 0.61
• --------------------------------
• Control Group
• papres totcost followup
• -------------------------------
• ASCUS/AGUS 74.92 0.32
• LGSIL 73.98 0.30
• HGSIL 104.94 0.43

• ICER(355.45-77.28)
• (0.61-0.32)
• ICER982.1837
• Is that good or bad?

15
Uncertainty

• You need to calculate the confidence regions
  around this parameter estimate

Variable Reps Observed Bias Std.
Err. 95 Conf. Interval ---------------------
--------------------------------------------------
----- cerp2 1000 982.1837 24.04897
152.2913 683.336 1281.031 (N)
787.438
1369.206 (P)
787.3935 1366.588
(BC) ---------------------------------------------
--------------------------------
N normal, P percentile, BC
bias-corrected
16
Confidence Regions

• Ratios are complex to interpret

17
More effective and more expensive
Less effective and more expensive
More effective and less expensive
Less effective and less expensive
18
Acceptability Curves

• Acceptability curves show the information based
  on willingness to pay for the outcome.
• Shape of the curve is dependent on the
  bootsrapped estimates
• Allows decision makers with different thresholds
  to interpret the data

Source Henry Glick
19
Threshold Value forCost-Effectiveness Ratio
Variable Reps Observed Bias Std.
Err. 95 Conf. Interval ---------------------
--------------------------------------------------
----- cerp2 1000 982.1837 24.04897
152.2913 683.336 1281.031 (N)
787.438
1369.206 (P)
787.3935 1366.588
(BC) ---------------------------------------------
--------------------------------
N normal, P percentile, BC
bias-corrected

• Rule of thumb
• U.S. health care system adopts interventions
  with ratio of less than 50,000 (100,000) per
  Quality Adjusted Life Year.

20
Adopt this Pap Smear Intervention?

• Poll
• Yes
• No
• Dont know
• Dont care

21
Limitations with RCTs

• Proxy outcomes
• Length of follow-up
• Generalizability

22
Ideal Study

• RCT comparing behavioral intervention and usual
  care control
• Follow participants for life
• Advantages know all costs and all benefits
• Disadvantages expensive and possibly useless
  results at end of study

23
Usual Study

• Outcomes measured at end of study
• Use models to estimate lifetime costs and
  benefits in the CEA

24
Effectiveness

• Preferred metric for CEA is quality adjusted life
  years (QALYs)
• Most behavioral interventions use an
  intermediate outcome
• e.g., receipt of mammography
• Few behavioral interventions use QALYs because
  the study would have to be huge and/or very long
  in duration

25
CEA with Intermediate Outcomes

• Easy and sufficient for publication
• Hard to interpret ICER
• Cant compare two CEAs with different
  intermediate outcomes
• Cant compare CEA to other CEA from another
  clinical area
• Sometimes only feasible approach

26
Intermediate to QALYs

• Translate intermediate outcome to QALYs
• Either build a model de novo or use an existing
  model
• Requires a lot of resources
• Most useful, but most challenging

27
Lost in Translation

• Gap between ideal and usual study
• Models fill the gap
• Behavioral models have unique challenges
• Partial behavior change is missing from current
  models
• Definition people who progressed in their
  stage of change but did not successfully change
  their behavior at the end of the study

Wagner TH, Goldstein MK. Behavioral interventions
and cost-effectiveness analysis. Prev Med. Dec
200439(6)1208-1214.
28
Outcome Responsiveness

• Many trials use indirect QALY assessments (HUI,
  QWB, EQ-5D)
• These measures are often not responsive to the
  intervention effect at the end of the trial.
• How do you interpret the results when the main
  outcome shows an effect and the QALYs do not?

29
QALY Responsiveness

• Try to understand why QALYs were not sensitive
• Analyze data for subgroup effects
• Analyze other outcomes that may be sensitive to
  change.
• Disease specific quality of life
• Willingness to pay
• If you think there is an effect, ignore the
  p-value on the QALYs and model the data using the
  mean and variance estimates

30
Another Approach

• Our CEA did not leverage the trial data
• We extracted parameters, created confidence
  regions and then (hopefully) put these parameters
  into a decision model

31
Net Monetary Benefits

• What we did
• Collect data to estimate ?C and ?E
• Calculate an estimate
• ?C / ?E (ICER incremental cost-effectiveness
  ratio)
• Another option
• ? ?E ?C (INB incremental net-benefit)

Hoch JS, Briggs AH, Willan AR. Something old,
something new, something borrowed, something
blue a framework for the marriage of health
econometrics and cost-effectiveness analysis.
Health Econ. Jul 200211(5)415-430.
32
Data
33
NMB set up

• You have cost data and effectiveness data for
  each person
• Use this information in a regular regression
  framework
• You need to estimate ?
• ? is the WTP for the outcome
• Rerun the analysis for different ? values

34
NMB Limitations

• Based on a linear model
• Does not necessarily translate into non-linear or
  latent models
• I suspect this is possible if you estimate
  predicted probabilities for groups and then carry
  out the analysis for the groups.

35
Generalizability

• Recall that RCTs may not enroll a generalizable
  population
• For many RCTs in VA, you can compare participants
  to non-participants
• Generate propensity scores
• Use the propensity score as a weight in the RCT
  analysis place more weight on people who are a
  lot like the non-participants

36
Next talk

• May 27, 2009How Can Cost Effectiveness Analysis
  Be Made More Relevant to US Health
  Care?Paul Barnett, Ph.D.