Title: Experimental Design How to organise a successful research project Including notes on sample size and
1Experimental DesignHow to organise a successful
research projectIncluding notes on sample size
and reliability
- Research Methods Lecture
- Bruce Lynn
- Director, MSc School of human Health and
Performance - March 09
2Experimental DesignHow to organise a successful
research project
- The initial idea
- Reading the literature
- Planning
- Doing
- Writing
3Generating the initial idea
- Something you always wanted to investigate
- Look at past projects and shamelessly borrow
ideas - Read around a subject of interest looking for
loose ends - Talk to one of the labs associated with the MSc
Programmes
4Reading the literature
- Essential
- Make sure project has originality
- Find out about the methods used in the field of
interest
5PlanningThe key to a successful research project
- Start early
- Decide what measures to use
- Who are the test population?
- What is being compared to what?
- Statistical issues
6What measure(s) to use?
- Questionnaires.
- Find suitable validated instruments.
- Try developing your own?
- Physiological measures
- Explore availability of equipment
- Investigate suitable analytical software
- Example
- Investigating heart rate variability.
- Find suitable heart rate monitoring equipment,
e.g. Polar - Find software package to analyse beat by beat
data - Decide which HRV parameter(s) are appropriate
7Who are the test population?
- Biological investigations, e.g. animal studies,
or studies of tissue properties, or cadaver
anatomy studies - Healthy volunteer studies
- Patient studies
- All are regulated re ethics. In particular
patient studies need Central (NHS) ethical
clearance (NRES) - Recruitment for human studies
- Posters
- Letters or e-mails
- Will subjects be paid if so who funds this?
- Subject information sheets
- Will enough subjects be available within the time
period available for the project?
8What is being compared with what?Experimental
designs (1)
- Example
- A double blind random controlled trial (RCT)
- Prospective subjects are checked out in advance
of intervention - Controlled. 2 (or more) groups run in parallel.
One receiving an ineffective treatment that will
appear equivalent to the real treatment, thus
controlling for any placebo effect. This is
often the hardest part of any design. - Randomisation. Must not select subjects. More
info in a minute. - Subject blinds. Can be impossible. E.g. testing
health education literature. - Observer blinds. Nearly always possible. Need an
assistant to organise this. May not be necessary
if outcome measure is very objective, e.g. blood
count. - Dropout rates. Must be monitored. Differential
dropout from control or treatment group can
introduce bias.
9What is being compared with what?Experimental
designs
- Other designs
- Crossover trials. Control group subsequently
receives treatment and vice versa. Each subject
own control good feature, but carry-over of
effects is a problem. - Non-experimental designs surveys
- Longitudinal. Following same subjects for period
of time. - Cross-sectional. Looking at sample of
population at one time point. - Retrospective. Looking back at health records
of patients.
10More on randomisation
Some random numbers from a published table 03 47
43 73 86 36 96 47 36 61 97 74 24 67 62 42 81
14 57 20 16 76 62 27 66 56 50 26 71 07 12 56 85
99 26 96 96 68 27 31 55 59 56 35 64 38 54 82
46 22
If you were using odd for treatment and even for
control, then note long runs of up to 8 all the
same. When n is small, these occasional long
runs can upset the balance of the study
- So pure random sequences can cause problems.
Need to remove long runs. Also need all numbers
to appear within any given sequence the same
number of times for this use random
permutations. - Often useful to ensure same number of treatment
(T) and control (non-treatment, NT) within blocks
of subjects. So every 10, make sure you have
equal T/NT even in a design for 30. Means if you
have to stop short, still have roughly equal T
and NT - Often need to ensure balance by randomising
separately for male/female, old/young.
severe/less severe etc as appropriate. Make sure
you have extra random allocations in sub-groups
if not sure of numbers, e.g. for M/F
11Statistical issues how many subjects
- Key statistical issue in the planning phase is
whether enough subjects can be studied to get a
clear result. - Involves the question of statistical power.
- Need to know 4 things
- ? The criterion for rejecting the null
hypothesis, normally 5 - The criterion for missing a possible result,
normally 10 or 20 (NB 100-?, is the Power
of the study) - The standard deviation of the outcome measure. Or
if using counted data or another non-continuous
measure, then the appropriate error measure - d The extent of change to be detected.
- For a comparison of 2 groups with same n,
formulae for required n is - n 16?2/d2
12Statistical issues how many subjects
- Example (From Statistics at Square One,
http//www.bmj.com/collections/statsbk/13.dtl) - BP Trial, comparing 2 groups. Want to find 5mm Hg
change if it occurs standard deviation in
population is 10mm Hg. - n 161010/55 64
- If we decide only a 10mm Hg change is of
significance to us, then - n 161010/1010 16
- So sample size needed depends very much on
criterion we set. - Obviously, the smaller the standard deviation in
the measure/population, the smaller n will be.
How do we find out s.d. in advance of the study!!
Can find literature values. Or do a small pilot
study. - Also, if we set a tougher criterion for rejecting
the null hypothesis, e.g. 1, then n rises. - A more detailed discussion of this issue, and the
formulae for other types of design and measure,
is in Kirkwood, Essentials of Medical Statistics,
Chapt 26
13Sample size for standardised difference at power
80 or 90
- This is for comparing an average result against a
fixed criterion or for paired data where SD is
SD(diff) for comparing 2 groups need to double n
in each group and use SD(error) i.e. RMS(within)
from ANOVA or SD(diff)/sqrt(2). - Difference you wish to detect is given in
standardised form, i.e. difference/standard
deviation. e.g Diff to detect in actual units
3, SD of repeated measurements in population
being studied 5, then "standardised" diff to
detect is 3/5 0.6
Note the very large effect of the difference you
choose to set as your criterion. Note also that
this is the ratio of the actual difference and
the error SD. So if you can reduce errors (by
better technique etc), can often greatly reduce
number needed for study.
14Sources of error
- Systematic leading to bias
- Random
- Bias is the bigger problem. Can have apparently
clear result that is wrong. Bias may be caused
by inadequate randomisation, or not ensuring good
match between T and NT samples, or inadequate
technique (e.g. not maintaining calibration of
equipment during study) - Random error will not lead to false positives
but will of course lead to more false negatives.
Important to spend time optimising methods. Also
to consider using only a relatively homogeneous
population, e.g. restrict to young males.
15Reliability statistics
- Need to know how reliable a measurement is.
- Is it repeatable, e.g on test-retest.
- Also if comparing groups, how much variation
within the group? - Comparison with other methods the calibration
problem.
16Reliability statistics
- Repeatability
- 1. Is it repeatable, e.g on test-retest.
- 2. Comparing groups, how much variation within
the group? - Need this information to compare different
methods - Importantly need it for sample size estimation
- Can just take a lot of measurements and calculate
SD - Often have pairs of measurements. The SD(diffs),
the SD of the differences between the values in
each pair, will be SD21/2 - See Reliability.doc and the example on
Reliability.xls, sheetTest-retest on WebCT or
at http//www.archway.ac.uk/Activities/Departments
/SHHP/current/Res_Methods/resmet1.htm
17Reliability statistics
- Comparison of new method with a standard method
- Sometimes referred to as validity, but this term
also covers wider aspects of any test. - Common approach is to measure same subjects with
both methods - Then can do SD(diffs) as for test-retest
comparisons - Often to methods are plotted as an XY plot, with
correlation coefficient, r used to assess
reliability - This is bad practice, r depends on range of X
values as well as errors. So more variable
methods tested on a wide range of subjects can
look better than a less variable methods tested
over a more limited range. - See Reliability.doc and the example on
Reliability.xls, sheet Comparison on WebCT or
at http//www.archway.ac.uk/Activities/Departments
/SHHP/current/Res_Methods/resmet1.htm
R 0.85 SD(diffs) 5.5ml/min.kg
18Doing the studyThe fun bit!
- Usually need to run initial trials to learn
methods, establish reliability etc - Then plan the main data collection. Try to keep
things standardised - If unexpected observations occur, then DO FOLLOW
THEM UP - Back up data. Keep back ups in a separate place
from the main data storage. Hard disc crashes do
occur!!
19Sorting out and analysing the data
- Excel is useful.
- Be sure to graph data before you analyse.
- Visualising the data usually indicates the real
trends your eyes are good at statistics! - Look at average trends normalise data where
appropriate. - For most regression, ANOVA or non-parametric
tests you will need a statistical package. SPSS
is on the cluster machines but is rather user
unfriendly, so expect to spend some time finding
your way around.
20Resources
- Statistics at square one, last chapter. Good
introduction to statistical issues.
http//www.bmj.com/collections/statsbk/ - Kirkwood, B.R. Essentials of Medical Statistics,
Chapts 21-26 - Bland, M. An Introduction to Medical Statistics,
Chapt 15, Reliability measures Chapt 18, smaple
size - Field, A. Discovering Statistics using SPSS
Detailed on regression good explanation of SPSS
computer package - Mackenzie, A. Mathematics and Statistics for
Life Scientists, Bios Instant Notes, 2005 mostly
maths - Random permutations
- http//calculators.stat.ucla.edu/perm.php
- http//www.webcalculator.co.uk/statistics/rpermut
e3.htm - NRES (National Research Ethics Service)
- http//www.nres.npsa.nhs.uk/