Title: The Analysis Of Clinical Trials With Clustering Effects Due To Treatment
1The Analysis Of Clinical Trials With Clustering
Effects Due To Treatment
- University of Manchester, UK
- Email chris.roberts_at_manchester.ac.uk
2Therapist Effect and Clinical Trials
- Outcome for patient treated by the same therapist
may be more similar than outcome for patients
treated by different therapists. - Therapist characteristics
- Experience
- Competence
- Empathy
- Patients clustered by therapist.
3Designs of Trials of Therapies
- Hierarchical Therapist Design.
- Therapist - Non Therapist Design.
- Stratified Therapist Design.
4Hierarchical Therapist Design
- Therapists nested within treatment
- Patients clustered by therapists
- Cluster Randomised Trial Therapist ? cluster
- e.g non-directive counselling (GP counsellors)
or cognitive-behaviour therapy (clinical
psychologist)
Treatment Effect
Therapist Variation
Covariates
Patient Variation
5Therapist - Non Therapist Design
Example No Drug vs Drug Therapy Similar to
comparison of group therapy with individual
therapy.
Treatment Effect
Therapist Variation
Patient Variation
Covariates
- Subjects in control arm are treated as clusters
of size 1
6Comparison of Trials With Clustering Due to
Randomisation or Treatment
7Comparison of Trials With Clustering Due to
Randomisation or Treatment
8Comparison of Trials With Clustering Due to
Randomisation or Treatment
9Comparison of Trials With Clustering Due to
Randomisation or Treatment
10Heteroscedasticity in a Hierarchical Therapist
Design
- In cluster randomised trial between arm
heteroscedasticity generally ignored.
Treatment Effect
Therapist Variation
Covariates
Patient Variation
Does heteroscedasticity matter or can one use a
random intercept model or equivalent.
11An Example RCT Of General Practitioners And
Nurse Practitioners In Primary Care
- Health professionals
- 20 nurse practitioners seeing a mean of 30
patients (range 30 to 36) - 71 general practitioners seeing a mean of 8
patients (range 2 to 29) - Patients Sample size
- 1280 patients approx 64 per Practice in 20
practices - Outcome measures
- Patient satisfaction
- Consultation process (length of consultation,
examinations, prescriptions, referrals), - Health service costs
12NP-GP TrialTreatment Effect- Patient Satisfaction
Choice of model effects treatment effect standard
errors
13Total Variance Equal In Both Arms
14Conclusions- Heterogeneity
- Fitting a random intercept model may lead to
under or over estimation of the intra-class
correlation coefficient if cluster size differ
systematically between arms - Test size for the treatment effect for a random
intercept model may therefore be biased . - Ideally, fit a heteroscedastic model.
- But this may be difficult in small samples sizes.
- If you fit the wrong model,
- you can expect to get the wrong answer.
15Conclusions- Heterogeneity
- Sample size for cluster randomised trials based
on a summary measures t-test - Heterogeneity in intra-class correlation
coefficients cluster size leads to
heteroscedasticity at the summary level. - Base sample size on Satterthwaites t-test
- Implemented in NQuery adviser
- Stata routine
16Unequal Randomization Group vs Individual Therapy
Moderate Effect Size 0.5 80 Power, 5 sig.
level ?? 0.05
17Cluster Membership
18Trial Non-directive Counselling or
Cognitive-Behaviour Therapy
- Participants 262 patients presenting with
depression or mixed anxiety depression. - Interventions Up to 12 sessions of therapy
- cognitive-behaviour therapy (12 psychologist)
- non-directive counselling (14 counsellors)
- Primary outcome measures Beck Depression
Inventory scores at 4 and 12 months. - Original published analysis ignored clustering by
therapist
19Numbers of Patients per Therapist
67 of patients treated by 5 therapists
13 of patients have no identified therapist
20Issues for Design Analysis
- No therapist information for some patients.
- Non-compliance?
- Grossly unequal cluster size.
21Analysis of Therapist Effects with Non-compliance
- Intended Therapist (IT)
- Subjects analysed according to the intended
therapist. - If the patient does not receive therapy, do they
have between therapist variation? - IT method mis-specifies the therapist random
effect. - Intended therapist may not be known.
- Actual Therapist (AT)
- Subjects analysed according to the actual
therapist (AT). - Non-compliant subjects assumed to have no
therapist random effect.
22Simulations Study of Non-compliance with
Treatment Related Clustering Variation
- Objectives Compare performance of Intended
Therapist (IT) and Actual Therapist (AT) analysis
strategies. - Assess bias in
- Treatment effect.
- Clustering effect of therapist.
- Data simulated for a trial with comparing
therapists with control.
23Simulations Study with All or Nothing Compliance
24Analysis Methods Compared
- Ignore Clustering.
- Using either Intended Therapist (IT) or Actual
Therapist (AT) Clustering - Random Effects ML
- GEE
- Robust Standard Error / Sandwich Estimators
- Analysis carried out in STATA
25Treatment Effect
26Treatment Effect Intended and Actual Therapist
27ITT with No Clustering
28Clustering- ML Estimation
29Maximum likehood Estimatorfor Actual Therapist
Analysis
Likelihood for therapists
Approximation for AT treatment effect estimate
30Treatment Effect ML GEE
31Treatment Effect Robust Estimators
32Intra-class Correlation for Therapist
33Estimated ICC Random Effects ML
34Estimated ICC
35Estimated ICC Random Effects Robust
36Summary
- Ignoring therapist will be non-conservative.
- Actual Therapist (AT) method does not perform
well for ML and GEE. - Intended Therapist (IT) method gives unbiased
estimates of treatment effect for ML GEE and
Robust estimation. - IT method can be non-conservative.
37Causal Analysis Methods
- Ignoring Therapist Clustering
- Maximum Likelihood Latent Class
- Instrumental variables regression
- Using either Intended or Actual Therapist
- Instrumental variables regression with Robust
Standard Error / Sandwich Estimators. - Using either Actual Therapist
- Maximum Likelihood Latent Class with random
effect.
38CACE Random Effects Mixture Model
Latent Class / Normal Mixture Model fitted to
control subjects
Therapist arm
- compliant
- non compliant
Control arm
39CACE Mixture Model
40Estimated ICC from CACE estimates
41CACE Random Effects Mixture Model
- Fitted using ML in Stata
- Covariates could be added for compliance
probability - What if compliance probability varies between
therapist? - This require a combined IT model for compliance
probability and AT model of outcome.
42Summary
- Ignoring clustering is non-conservative.
- Robust instrumental variable methods are unbiased
but may be conservative for both IT and AT
methods. - Mixture model methods performed well for AT
method. - Problems with convergence for mixture model with
IT method.
43Conclusions
- Therapist trials more complex than cluster
randomised trials if clustering is considered as
cluster size and intra-cluster correlation may
differ systematically between treatment arms. - Therapist effects must be considered in trial
design.
44Comparison of Trials With Clustering Due to
Randomisation or Treatment
45Additional/Left Over Slides
46Trial Non-directive Counselling or
Cognitive-Behaviour Therapy
Problems due to wide variation in cluster size
47Problems of Large Variation in Cluster Size
- ICC values for therapist likely to be close to
zero - Maximum cluster size places lower limit on ICC
-1/(max cluster size -1) causing non-convergence - Negative ICC (GEE or ML were negative variances
are allowed) and design effect (Robust) leading
to downwardly biased treatment effect standard
errors.
48References
- Heteroscedasticity and sample size discussed in
- Roberts C Roberts SA (2005) Design and analysis
of clinical trials with clustering effects due to
treatment Clinical Trials, 2152-162.
49Acknowledgements
- Steve Roberts Help with simulation work.
- Pete Bower Providing data from Trial of
Non-Directive Counselling or Cognitive-Behaviour
Therapy.
50Design Effect
Above 1 is conservative
51Design Effect
Above 1 is conservative
52Stratified Therapist Design
- Same health professional delivers both
interventions - Example CBT delivered face-to-face or by phone
- Similar to
- Multi-centre studyTherapist ? Centre/Study
- Meta-analysis Therapist ? Study
- Match pairs cluster randomised Therapist ? Match
pair
53Stratified Therapist Design
Patient Variation
Treatment Effect
Therapist Variation
Covariates
Therapist-Treatment Interaction
Patient Variation
Treatment Effect
Therapist Variation
Covariates
Treatment within Therapist
54IV Random Effects Model
55IV Random Effects Model
56Likelihood Equation For Random Intercept Model
Replace red bits by expected values for
different cluster size and heteroscedasticity
then maximize
57Modelling NP vs GPAdjusting for Age and Sex
58Comparison of observed with approximation
59Simulation Study
60Total Variance Equal In Both Arms
1000 simulation ML REML in R Solid line are
Guesstimates
61Total variance 25 smaller for smaller clusters
62Total variance 25 larger for smaller clusters