Title: Analysis Alongside A Randomized Trial
1Analysis Alongside A Randomized Trial
- Todd Wagner, PhD
- May 2009
2Objectives
- At the end of the class, you should
- Understand how to set up your datasets
- Be familiar with analytical methods
- Want to hear the class on decision modeling
3Dominance
- No need to calculate an incremental cost
effectiveness ratio if - Intervention is more effective and less expensive
than control - Intervention is more expensive and less effective
than control - But
- Check whether dominance exists within subgroups
- Does dominance persists after including
uncertainty
4Incremental Cost-Effectiveness Ratio (ICER)
- Calculate in the absence of dominance
CostEXP - CostCONTROL _____________________ QALYEX
P -QALYCONTROL
5Cost Data
- Have costs consistent with the stated perspective
- Societal
- Health care utilization
- Patient costs
- Caregiver costs
- Intervention costs (direct plus indirect)
6Common Hurdle
- Many of the parameters in the analysis are based
on assumptions (e.g., wage rate, mileage costs) - Consider whether these assumptions are biased
toward/against the intervention - Ideally want to include uncertainty
- One way sensitivity analysis
- Statistical methods (bootstrapping, Cooks D)
7Another hurdle
- Include disease-related utilization or all health
care utilization? - How do you define disease-related?
- Recommend look at all utilization for the CEA
8Labor Outcomes
- Productivity employment is not in the cost
estimate - Anyone remember why?
- If labor outcomes are important, still collect
them but report them separately.
9Dataset
- You need to create a long dataset
- You need to have an group indicator for
experiment (1) and control (0) - You need to have cost and outcome information
- You need subgroup identifiers
10Data
11Analysis
- . tabstat totcost followup if exp1, by(papres)
stat(mean) format(3.2f) - Summary statistics mean
- by categories of papres (initial pap results)
- papres totcost followup
- -------------------------------
- ASCUS/AGUS 347.31 0.57
- LGSIL 373.97 0.64
- HGSIL 404.72 0.87
- -------------------------------
- Total 355.45 0.61
- --------------------------------
- . tabstat totcost followup if exp0, by(papres)
stat(mean) format(3.2f) - Summary statistics mean
- by categories of papres (initial pap results)
12Hypothesis Testing
. reg totcost exp Source SS
df MS Number of obs
348 -------------------------------------------
F( 1, 346) 292.64 Model
6728200.2 1 6728200.2 Prob gt F
0.0000 Residual 7954890.66 346
22991.0135 R-squared
0.4582 ------------------------------------------
- Adj R-squared 0.4567 Total
14683090.9 347 42314.3829 Root
MSE 151.63 ------------------------------
------------------------------------------------
totcost Coef. Std. Err. t
Pgtt 95 Conf. Interval ------------------
--------------------------------------------------
--------- exp 278.1664 16.26051
17.11 0.000 246.1845 310.1483
_cons 77.27982 11.62933 6.65 0.000
54.40675 100.1529 ----------------------------
--------------------------------------------------
Logistic regression
Number of obs 348
LR chi2(1)
28.42
Prob gt chi2 0.0000 Log
likelihood -226.30785
Pseudo R2 0.0591 --------------------
--------------------------------------------------
-------- followup Odds Ratio Std. Err.
z Pgtz 95 Conf. Interval ------------
-------------------------------------------------
---------------- exp 3.225974
.7243906 5.22 0.000 2.077419
5.009538 -----------------------------------------
-------------------------------------
13Conclusion
- Intervention is more costly and more effective
- Next Steps
- Include uncertainty
- Check for subgroups and
- calculate the ICER
14ICER Calculation
- Intervention Group
- papres totcost followup
- -------------------------------
- ASCUS/AGUS 347.31 0.57
- LGSIL 373.97 0.64
- HGSIL 404.72 0.87
- -------------------------------
- Total 355.45 0.61
- --------------------------------
- Control Group
- papres totcost followup
- -------------------------------
- ASCUS/AGUS 74.92 0.32
- LGSIL 73.98 0.30
- HGSIL 104.94 0.43
- ICER(355.45-77.28)
- (0.61-0.32)
- ICER982.1837
- Is that good or bad?
15Uncertainty
- You need to calculate the confidence regions
around this parameter estimate
Variable Reps Observed Bias Std.
Err. 95 Conf. Interval ---------------------
--------------------------------------------------
----- cerp2 1000 982.1837 24.04897
152.2913 683.336 1281.031 (N)
787.438
1369.206 (P)
787.3935 1366.588
(BC) ---------------------------------------------
--------------------------------
N normal, P percentile, BC
bias-corrected
16Confidence Regions
- Ratios are complex to interpret
17More effective and more expensive
Less effective and more expensive
More effective and less expensive
Less effective and less expensive
18Acceptability Curves
- Acceptability curves show the information based
on willingness to pay for the outcome. - Shape of the curve is dependent on the
bootsrapped estimates - Allows decision makers with different thresholds
to interpret the data
Source Henry Glick
19Threshold Value forCost-Effectiveness Ratio
Variable Reps Observed Bias Std.
Err. 95 Conf. Interval ---------------------
--------------------------------------------------
----- cerp2 1000 982.1837 24.04897
152.2913 683.336 1281.031 (N)
787.438
1369.206 (P)
787.3935 1366.588
(BC) ---------------------------------------------
--------------------------------
N normal, P percentile, BC
bias-corrected
- Rule of thumb
- U.S. health care system adopts interventions
with ratio of less than 50,000 (100,000) per
Quality Adjusted Life Year.
20Adopt this Pap Smear Intervention?
- Poll
- Yes
- No
- Dont know
- Dont care
21Limitations with RCTs
- Proxy outcomes
- Length of follow-up
- Generalizability
22Ideal Study
- RCT comparing behavioral intervention and usual
care control - Follow participants for life
- Advantages know all costs and all benefits
- Disadvantages expensive and possibly useless
results at end of study
23Usual Study
- Outcomes measured at end of study
- Use models to estimate lifetime costs and
benefits in the CEA
24Effectiveness
- Preferred metric for CEA is quality adjusted life
years (QALYs) - Most behavioral interventions use an
intermediate outcome - e.g., receipt of mammography
- Few behavioral interventions use QALYs because
the study would have to be huge and/or very long
in duration
25CEA with Intermediate Outcomes
- Easy and sufficient for publication
- Hard to interpret ICER
- Cant compare two CEAs with different
intermediate outcomes - Cant compare CEA to other CEA from another
clinical area - Sometimes only feasible approach
26Intermediate to QALYs
- Translate intermediate outcome to QALYs
- Either build a model de novo or use an existing
model - Requires a lot of resources
- Most useful, but most challenging
27Lost in Translation
- Gap between ideal and usual study
- Models fill the gap
- Behavioral models have unique challenges
- Partial behavior change is missing from current
models - Definition people who progressed in their
stage of change but did not successfully change
their behavior at the end of the study
Wagner TH, Goldstein MK. Behavioral interventions
and cost-effectiveness analysis. Prev Med. Dec
200439(6)1208-1214.
28Outcome Responsiveness
- Many trials use indirect QALY assessments (HUI,
QWB, EQ-5D) - These measures are often not responsive to the
intervention effect at the end of the trial. - How do you interpret the results when the main
outcome shows an effect and the QALYs do not?
29QALY Responsiveness
- Try to understand why QALYs were not sensitive
- Analyze data for subgroup effects
- Analyze other outcomes that may be sensitive to
change. - Disease specific quality of life
- Willingness to pay
- If you think there is an effect, ignore the
p-value on the QALYs and model the data using the
mean and variance estimates
30Another Approach
- Our CEA did not leverage the trial data
- We extracted parameters, created confidence
regions and then (hopefully) put these parameters
into a decision model
31Net Monetary Benefits
- What we did
- Collect data to estimate ?C and ?E
- Calculate an estimate
- ?C / ?E (ICER incremental cost-effectiveness
ratio) - Another option
- ? ?E ?C (INB incremental net-benefit)
Hoch JS, Briggs AH, Willan AR. Something old,
something new, something borrowed, something
blue a framework for the marriage of health
econometrics and cost-effectiveness analysis.
Health Econ. Jul 200211(5)415-430.
32Data
33NMB set up
- You have cost data and effectiveness data for
each person - Use this information in a regular regression
framework - You need to estimate ?
- ? is the WTP for the outcome
- Rerun the analysis for different ? values
34NMB Limitations
- Based on a linear model
- Does not necessarily translate into non-linear or
latent models - I suspect this is possible if you estimate
predicted probabilities for groups and then carry
out the analysis for the groups.
35Generalizability
- Recall that RCTs may not enroll a generalizable
population - For many RCTs in VA, you can compare participants
to non-participants - Generate propensity scores
- Use the propensity score as a weight in the RCT
analysis place more weight on people who are a
lot like the non-participants
36Next talk
- May 27, 2009How Can Cost Effectiveness Analysis
Be Made More Relevant to US Health
Care?Paul Barnett, Ph.D.