Participant Preferences in Randomised Controlled Trials: an example from education'

1
Participant Preferences in Randomised Controlled
Trials an example from education.

• Hannah Ainsworth
• Research Fellow
• Institute for Effective Education and York Trials
  Unit
• University of York

2
Why are participant preferences important in RCTs?

• Bias can result in an incorrect estimate of
  effect and lead to an intervention being found to
  be ineffective when it is actually effective, or
  effective when it is actually ineffective.
• The process of randomisation equalises all the
  possible unknown confounding variables between
  the arms of a trial and is therefore able to
  eliminate many forms of bias.
• However forms of bias can occur after
  randomisation, it is important to be aware of the
  sources of such bias and take steps to minimise
  them or deal with them adequately in the
  analysis.

3
Participant Preferences Possible Bias

• One source of bias after randomisation is
  participant preferences.
• When presented with different interventions
  participants often have a preference for which
  arm of a trial they would like to be in.
• Participants with a strong preference for the
  novel intervention, if unavailable outside the
  trial, will consent to randomisation.
• Participants who have no preference will consent
  to take part in the trial.
• Participants with weaker preferences for
  standard support will sometimes consent to
  randomisation.
• Participants with a strong preference for
  standard support often do not consent to go
  into a trial because they can get standard
  support without being in the trial and
  randomisation may mean they receive an
  intervention they dont want.

4
Participant Preferences Possible Bias

• These preferences can introduce bias if one of
  the interventions is only available within the
  trial and if the trial is an open trial, where
  participants know which arm they have been
  randomly allocated to.
• For example if participants are randomised to
  the intervention they did not want, they may
  consciously or unconsciously perform less well in
  their response to outcome measures in the
  knowledge that there was a better intervention
  available this is sometimes termed as resentful
  demoralisation.
• Or those who are randomised to the intervention
  they preferred may be more likely to complete
  follow up and report favourable outcomes.
• Whilst randomisation will ensure that
  participants with different preferences are
  spread equally among the arms of the trials, it
  cannot deal with the effect of participant
  preferences on the outcome measures after
  randomisation.

5
Theoretical Example

• Torgerson and Torgerson (2008 87-89) provide the
  following theoretical impact of preferences.
• The intervention has no intrinsic effect and both
  groups should score 10 on their post-test scores
  but participant preferences effect their scores.
• Those who preferred intervention A and received A
  scored greater at post-test than those who were
  indifferent.
• Those who preferred intervention A but received B
  feel demoralised and perform less well.
• In this example we could conclude that
  intervention A is more effective than B, despite
  the intervention having no intrinsic effects.

6
Examples from Health trials

• There are have been a number of trials in Health
  care which recorded the preferences of
  participants some of which have demonstrated an
  effect of preference on outcomes.
• SPRINTER trial - looked at the effect of a brief
  physiotherapy intervention compared with usual
  physiotherapy for neck pain
• The results were suggestive of an interaction
  between preference and outcome.

Klaber-Moffett et al. (2005) Randomised trial
of brief physiotherapy intervention compared with
usual physiotherapy for neck pain patients
outcomes, costs and patient preference British
Medical Journal 330 75.
7
Brewin-Bradley Design

• Patient preference design, comprehensive cohort
• Participants are asked before randomisation if
  they have a preference.
• Only those who do not have a preference are
  randomised.
• This removes the preference effects from the
  trial results.
• Non-randomised groups are also followed up to see
  what happens to people who receive the
  intervention they desire.

Brewin, C. R. and Bradley, C. (1989) Patient
Preferences and randomised clinical trials,
British Medical Journal 299 313-15
8
Participant Flow in the Brewin-Bradley Design
Eligible participants
Prefer usual care
Indifferent Randomise
Prefer Intervention
Receive usual care
Receive Intervention
Receive Intervention
Receive usual care
Assess outcomes
Assess outcomes
Assess outcomes
Assess outcomes
9
Problems with Brewin-Bradley Design

• Torgerson and Torgerson (2008) highlighted a
  number of limitations to this design
• Participants with preferences are not randomised,
  these arms of the trial are therefore exposed to
  selection bias. This makes the comparisons of
  these arms hazardous.
• If the proportion of participants with
  preferences are high this endangers recruitment.
• We dont know for sure that preferences will
  affect outcomes so by using this design we might
  be losing participants from randomisation.

10
Wennberg Design

• In the Wennberg design, all eligible participants
  are randomly allocated into two groups.
• One group is randomly allocated again to the
  intervention and control arms of the trial.
• In the other group the participants can choose
  which intervention they want to receive.

Luellen, J. K., Shadish, W. R. and Clark M. H.
(2005) Propensity scores an introduction and
experimental test, Evaluation Review 29 530-58
11
Participant Flow in the Wennberg Design
Assess Outcomes
Assess Outcomes
Assess Outcomes
Assess Outcomes
12
Problems with the Wennberg Design

• Torgerson and Torgerson (2008) noted that
• The Wennberg design still doesnt allow us to
  deal with the effects of preference
  satisfactorily.
• In the randomised group participants still have
  their preferences, which can still affect the
  outcome of comparison between those two groups.

13
Zelens Method

• Post-randomised consent design
• Participants are identified and randomised
  without their knowledge and consent.
• In the single consent method, consent is then
  sought from those allocated to the novel
  intervention. Those refusing consent are given
  access to usual care.
• In the double consent method participants
  randomised to either intervention or usual care
  can refuse and be given unhindered access to the
  alternative treatment.
• Intention to Treat Analysis must then be used,
  participants must remain in the group to which
  they were randomised regardless of whether they
  refused or not, to avoid selection bias.
• Zelens method avoids the disappointment of
  patients allocated to the control treatment and
  can therefore be used as an alternative to the
  patient preference design

Zelen, M. (1979) A new design for randomized
clinical trials, New England Journal of Medicine
300 1242-5
14
Participant Flow in Zelens MethodSingle Consent
Randomise all eligible participants
Control Group Usual Care
Intervention Group
Refuse Intervention Receive Usual Care
Accept Intervention Receive Intervention
ITT Assessment of Outcomes
Assess Outcomes
15
Problems with Zelens Method

• Ethically controversial especially in
  individually randomised trials because
  participants do not consent to randomisation.
• Unacceptable treatment dilution can occur.
• ITT analysis is often not used (Adamson et al
  2006) resulting in selection bias.

16
Fully Randomised Preference Trial

• Torgerson et al (1996) suggested using a fully
  randomised preference trial.
• Consent and preference are recorded before
  randomisation. All eligible participants are
  randomised.
• An estimate of the effect of the intervention
  among participants without a preference and with
  a preference can be calculated.

Torgerson, D. J., Klaber-Moffet, J. A. and
Russell, I. T. (1996) Patient preferences in
randomised trials threat or opportunity,
Journal of Health Services Research and Policy 1
194-7
17
Participant Flow in a Fully Randomised Preference
Trial
Ask preferences and randomise all eligible
participants
Intervention
Control group
Indifferent Receive usual Support
Preferred Intervention Receive usual support
Preferred usual support Receive usual support
Preferred usual support Receive intervention
Preferred Intervention Receive intervention
Indifferent Receive intervention
Assess Outcomes
Assess Outcomes
Assess Outcomes
Assess Outcomes
Assess Outcomes
Assess Outcomes
18
Problems with Fully Randomised Preference Trials

• As Torgerson and Torgerson (2008) noted
  participants who have a very strong preference
  for usual care may not consent to the trial.
• If the intervention was found to be effective we
  could never be sure that it would have the same
  effect among those who strongly preferred usual
  care.

19
An example from Education A Randomised Trial
of a Computer Software Package to Improve
Numeracy Skills Among Nursing Students.
20
A Randomised Trial of a Computer Software Package
to Improve Numeracy Skills Among Nursing Students.

• Background
• Previous research has highlighted an ongoing
  concern about numerical competence across
  healthcare professions.
• Many students who are enrolled on the Department
  of Health Sciences nurse training courses have
  difficulties with essential numeracy skills.
• The Department has invested a large amount of
  money into an online numeracy course known as
  Authentic World to supplement existing teaching.
• A recent Systematic Review (Torgerson et al 2003)
  of interventions to promote numeracy learning in
  adults found no trial of computer software aimed
  specifically at Student nurses and none of this
  package.
• Because this package has not been evaluated using
  a Randomised Controlled Trial we cannot be
  confident about its effect on the Numeracy skills
  of student Nurses.
• A Randomised Controlled Trial is therefore
  needed.
• Research question
• Does the use of a supplementary computer based
  numeracy training programme (Authentic World)
  increase numeracy skills among first year nursing
  students?

21
Participant Flow
139 First year student nurses and midwives
approached to take part in the trial
24 students did not consent to take part in the
trial
115 Students consented to take part in the
trial. Participants were asked to complete a
baseline questionnaire (preferences elicited) and
take a baseline numeracy test.
5 Baseline questionnaires were unavailable 9
Baseline scores were unavailable
101 Students randomised
Control Group (n 51) Usual Support from staff
Intervention Group (n 50) Access to the online
numeracy support package and usual support from
staff
1 Wanted Usual support
0 Wanted Usual support
28 Wanted Intervention
32 Wanted Intervention
21 Indifferent
19 Indifferent
22
Current results

• Participants who preferred computer support had
  lower baseline numeracy test scores than those
  who had no preference.
• Mean Difference -1.59 (p 0.06)
• Analysis can take into account this potential
  difference between the preference groups.

23
What about those who didnt consent?

• The fully randomised preference design is not
  able to deal with strong preferences for usual
  support.
• However in this trial we have been able to
  collect the baseline numeracy scores for those
  who did not consent to the trial.
• Analysis revealed that the mean test score of
  those who did not consent to take part in the
  trial was not statistically different from those
  who did consent.
• We were also able to ask the students why they
  didnt take part in the trial.
• The most common reason for not taking part was
  because the students had not been in class when
  the trial had been explained rather than they
  strongly preferred usual support and did not want
  to be randomised.
• One student however did not want to be
  randomised I wanted to see how well I did
  without the extra help!

24
The Trial Continues

• Participants randomised to receive the
  intervention have unlimited access for six
  months. All participants will then complete an
  outcome numeracy test.
• The analysis will take into account the possible
  difference between the preference groups, and
  will look to see whether the preference of the
  participants has altered the effect of the
  numeracy package.

25
Conclusion

• Participant preferences can introduce bias into a
  trial after randomisation.
• Using a fully randomised preference trial design
  is a simple way to take into account the possible
  effect of participant preferences.
• Where appropriate this design should be used.

26
References

• Adamson, J., Cockayne, S., Puffer, S. And
  Torgerson, D. J. (2006) Review of randomised
  trials using the post-randomised consent (Zelens
  ) design Contemporary Clinical Trials 27 305-19
• Brewin, C. R. and Bradley, C. (1989) Patient
  Preferences and randomised clinical trials,
  British Medical Journal 299 313-15
• Klaber-Moffett, J. A., Richmond, S., Jackson, D.,
  Coulton, S., Hahn, S., Farrin, A., Manca, a. and
  Torgerson, D. J. (2005) Randomised trial of
  brief physiotherapy intervention compared with
  usual physiotherapy for neck pain patients
  outcomes, costs and patient preference British
  Medical Journal 330 75.
• Luellen, J. K., Shadish, W. R. and Clark M. H.
  (2005) Propensity scores an introduction and
  experimental test, Evaluation Review 29 530-58
• Torgerson, C. J., Porthouse, J. And Brooks, G.
  (2003) A systematic review and meta-analysis of
  randomised controlled trials evaluating
  interventions in adult literacy and numeracy
  Journal of Research in Reading Vol 25, Issue 3,
  p. 234-255
• Torgerson, D. J., and Torgerson, C.J. (2008)
  Designing Randomised Trials in Health, Education
  and the Social Sciences Palgrave Macmillan
• Torgerson, D. J., Klaber-Moffet, J. A. and
  Russell, I. T. (1996) Patient preferences in
  randomised trials threat or opportunity,
  Journal of Health Services Research and Policy 1
  194-7
• Zelen, M. (1979) A new design for randomized
  clinical trials, New England Journal of Medicine
  300 1242-5