Pitfalls in Research

1
Pitfalls in Research

• Professor Alastair Chambers
• University of Aberdeen

2
Declarations of Interest

• Editor of Anaesthesia
• Reviewer for BJA, Pain, EJP, Palliative Care
  Journals
• Reviewer of grant applications - AAGBI, NIAA,
  HTA, various RD
• Currently researching chronic post mastectomy
  pain and community PMPs
• Example problems have been anonymised and
  disguised

3
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

4
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

5
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

6
Research?

• Why bother?

7
Research - why do it?

• Why not just do an audit project
• Less problem with ethics
• Anyone can do an audit project (and many do)
• Finished more quickly
• More difficult to get published
• May improve local practice - but not more

8
Research - why do it?

• Commercial Research
• act as clinical technician and get paid for it
• work with company to design and run project

9
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

10
Research in Pain - Where do you start?

• Initial thoughts
• Literature Review
• Be very wary of reports of animal work

11
Problems with animals
12
Problems with animal research

• Efficacy - what is being measured in behaviour?
• Many side effects - dizziness?
• Validation is often ignored
• Automated measures
• Irrelevant tests
• Doses and treatments

13
Where do you start?

• Initial thoughts
• Literature Review
• Be very wary of reports of animal work
• Formulate RESEARCH QUESTION or HYPOTHESIS TO TEST
• AND WRITE IT DOWN!

14
I want to find out whether morphine is more
effective than paracetamol for patients on my
gynae list.

• I wish to compare the effect of 10mg morphine
  against 2G paracetamol administered intravenously
  30 minutes prior to the end of intermediate
  gynaecological surgery.

15
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

16
Difficulties with protocol (1)

• Measuring the subjective
• Difficulty getting homogeneous population
• Ethics
• Time
• Changing practice

17
Measurement Issues (1)Visual Analogue Scale

_________________________________
No Pain
Severe Pain
18
Visual Analogue Scale
No Pain
Severe Pain
_________________________________
19
Visual Analogue Scale
No Pain
Severe Pain
_________________________________
20
Measurement Issues (2)

• Unvalidated Instruments
• Inappropriate Instruments
• Lack of blinding

21
Difficulties with protocol (2)

• Difficulty getting homogeneous population
• Ethics
• Time - contemporaneous/recall bias
• Changing practice

22
Difficulties with protocol (3)

• Recruitment NEVER as good as expected
• Protocol violations
• Research designers need to be familiar with
  clinical practice

23
Methodology

• Questionnaires - the easy option?

24
Questionnaires

• Usually poorly thought out
• Usually poorly set out
• Usually poor response
• Analysis poor or worse
• Results not robust or reliable

25
Questionnaires

• Confidential better than Anonymous
• True random sample better than large sample
• Use of validated instruments vital
• Remember ethics might be essential

26
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

27
Purpose of Statistics

• Provide description
• Make inferences about patterns/differences
  between data sets

28
Inferential Statistics

• Comment on likelihood of variation NOT arising by
  chance
• Never use statistics in isolation result must
  be plausible
• Statistical design must be robust

29
(No Transcript)
30
(No Transcript)
31
Problems with Statistics

• Power of study
• Usually calculated in terms of primary outcome
• Therefore may be inadequate for other items

32

• No serious complications were observed in any of
  the 20 patients studiedIt is safe to administer
  this treatment.
• If nothing goes wrong, is everything all right?
  Interpreting zero numerators
• Hanley Lippman-Hand JAMA 1983 249 1743-5
• Gambling with ethics? A statistical note on the
  Poisson (binomial) distribution
• Pandit J, Anaesthesia, 2008 63 1163-6

33
Randomisation

• THE most important feature of sampling
• Increasing size will not compensate for poor
  randomisation
• Every object has an equal chance of inclusion

34
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

35
ETHICAL ISSUES

• Is it really necessary? For a study on
  mannikins???
• If the LREC say its ok, then it must be
  ethically ok!
• Difficult editors or editors with difficulty?

36
ANAESTHESIAInstructions to Authors

• Studies of novel treatments, in particular
  drug studies where the agent used is given via
  unlicensed routes (especially spinal and
  epidural), may have received approval from the
  Research Ethics Committee or equivalent, but the
  Editorial Board is likely to reject such studies
  if it considers that the risks posed outweigh the
  potential benefits. Such a conclusion is more
  likely to be reached if the drug in question is
  not widely used in routine practice (as evidenced
  by inclusion in standard textbooks), if the study
  participants are especially vulnerable (e.g.
  children, women in labour), if there are
  questions over consent, or if only modest
  improvements in outcome are expected where other,
  well established methods already exist.

37
Other Pitfalls

• Not asking for advice
• Asking for advice
• Not piloting or assessing feasibility
• Not taking statistical advice
• Not having enough help/assistance
• Having too many assistants/helpers

38
Issues to consider

• Get things in the right order!
• Why are you doing this?
• What are you doing?
• Protocol
• Pilot
• Statistics
• Ethics
• Finishing it off

39
Publications with trainees

• Parris MR Chambers WA Effects of the addition
  of potassium to prilocaine or bupivacaine
  studies on axillary nerve blockade Br J Anaesth,
  1986 58 297-300
• MacLean D, Chambers WA, Tucker GT Wildsmith JAW
  Plasma levels of prilocaine following three
  techniques of brachial plexus block Br J Anaesth,
  1987 60 136-40
• Ronald AL, Ramayya GP Chambers WA An
  inexpensive method of capturing oximetry data Br
  J Anaesth, 1992 69 105-7
• Ronald AL, Docherty D, Broom J Chambers WA
  Morphine absorption following spinal anaesthesia
  Anesth Analg (Cleve), 1993 76 778-82
• Byers GF, Parkin DE, Pinion S Chambers WA Fluid
  absorption during endometrial resection
  Gynaecological Endoscopy, 1993 2 21-3
• Power I, Bowler GMR, Pugh GC Chambers WA
  Ketorolac as a component of balanced analgesia
  after thoracotomy Br J Anaesth, 1994 72 224-6
• Martin PD Chambers WA Teaching airway
  management in anaesthetised patients Scottish Med
  J, 1994 39 111-3
• Smith BH, Read JRM, Grimshaw JM, Watt BG
  Chambers WA Researching chronic pain
  identification of a community based sample The
  Pain Clinic, 1996, 73-6

40

• Smith BH, Penny KI, Purves AM, Munro C, Grimshaw
  J, Chambers WA Smith WC The Chronic Pain Grade
  Questionnaire validation and reliability in
  postal research Pain, 199771 141-7
• Richards JT, Read JRM Chambers WA Extradural
  anaesthesia as a method of pre-emptive analgesia
  for abdominal hysterectomy Anaesthesia, 1998
  53 296-8
• Hunter JD, Penny KI Chambers WA Minor morbidity
  after day case surgery
• Scottish Med J, 1998 43 54-6
• Purves A, Munro C, Smith BH, Grimshaw J, Wilson
  B, Smith WCS Chambers WA Defining chronic pain
  for epidemiological research assessing a
  subjective definition. The Pain Clinic, 1998 10
  139 47
• Smith WC, Bourne D, Squair J, Phillips DO,
  Chambers WA. A retrospective cohort study of
  post mastectomy pain syndrome. Pain 1999 83
  91-95
• McCartney CJL Chambers WA Review Coeliac
  plexus block Current Anaesthesia Critical Care
  2000 6 101-105
• McCartney CJ, Cooper G, Smith WCS Chambers WA.
  Pain and disability following lower limb
  amputation - a quantitative and qualitative
  study. The Pain Clinic, 2000 12 36-41

41

• McCartney CJL Chambers WA. Central Neuraxial
  Techniques for Cancer Pain Current Anaesthesia
  Critical Care 2000 11 166 172
• Kumar N, Chambers WA Tunnelling epidural
  catheters a worthwhile exercise?Anaesthesia.
  200055 625-6.
• Chambers WA, Ferguson K, Prescott GJ Continuing
  medical education by anaesthetists in scotland
  activities, motivation and barriers. Anaesthesia.
  200055 1192-7.
• Kumar RN, Chambers WA, Pertwee RG Pharmacological
  actions and therapeutic uses of cannabis and
  cannabinoids. Anaesthesia. 2001 56 1059-68
• Poobalan AS, Bruce J, King PM, Chambers WA,
  Krukowski ZH, Smith WCS Chronic pain and quality
  of life following open inguinal hernia repair
  British J Surgery. 2001 88 1122-1126
• Linklater GT, Leng ME, Tiernan EJ, Lee MA,
  Chambers WA Pain management services in
  palliative care a national survey. Palliat Med.
  2002 16 435-9.
• Bruce J, Chambers WA Questionnaire surveys.
  Anaesthesia. 2002 57 1049-51
• Poobalan AS, Bruce J, Smith WCS, King PM,
  Krukowski ZH, Chambers WA
• A review of chronic pain following inguinal
  herniorraphy Clin J Pain. 2003 19 48-54

42

• Haetzman M, Elliott AM, Smith BH, Hannaford P,
  Chambers WA Chronic pain and the use of
  conventional and alternative therapy. Fam Pract.
  2003 20 147-54
• Bruce J, Drury N, Poobalan AS, Jeffrey RR, Smith
  WCS, Chambers WA The prevalence of chronic chest
  and leg pain following cardiac surgery a
  historical cohort study Pain. 2003
  Jul104(1-2)265-73
• Bruce J, Poobalan AS, Smith WCS, Chambers WA
  Quantitative assessment of chronic post surgical
  pain using the Magill Questionnaire. Clinical J
  Pain. 2004 20(2) 70-5
• Macdonald L, Bruce J. Scott NW, Smith WC,
  Chambers WA Long-term follow-up of breast cancer
  survivors with post-mastectomy pain syndrome. Br
  J Cancer. 2005 Jan 3192(2)225-30
• McDermott ME, Smith BH, Elliot AM, Bond CM,
  Hannaford PC Chambers WA. The use of medication
  for chronic pain in primary care, and the
  potential for intervention by a practice-based
  pharmacist. Fam Pract. 2006 Feb23(1)46-52.
• Saqr L, Chambers WA Preventing excessive
  pre-operative fasting national guideline or
  local protocol? Anaesthesia. 2006 Jan61(1) 23-5
• McCartney CJL Chambers WA Preoperative
  analgesicswhat is wrong with the oral route?
  Anaesthesia, 1998 53 607
• McCartney CJL, Herriot R, Chambers WA Bilateral
  hand oedema related to Acupuncture Pain, 2000
  84429-430
• Lee MA, Leng ME, Tiernan EJ, Chambers WA A simple
  method of using epidural analgesia in palliative
  medicine. Palliat Med. 2001 15 347-8

43
(No Transcript)
44
Why present data?

• Makes the paper longer
• Nice graphs break up boring text
• Everyone will believe what you say

45
Why you really should present data

• To allow reader to check what you say
• To put it in context
• To clarify or expand on what you have written

46
Errors in presentation

• using standard errors for descriptive
  information
• presenting means of continuous data without
  any indication of variability
• presenting results of statistical analysis
  solely as a p-value

47
Examples of precision

• P 10-54
• P 0.5254
• R 0.99299
• ?2 0.7264
• 86.95 of cases

48
Even more precise

• Relationship of birth weight (BWt) to chest
  circumference (CC) and mid-arm circumference (AC)
• BWt -3.0983527 0.142088CC 0.158039AC
• Predicts birth weight to the nearest 1/10000g!!

49
Misleading features on graphs

• Lack of true zero on vertical axis
• Change of scale in the middle of an axis
• Three dimensional effects
• Failure to show coincident points in a scatter
  diagram
• Showing a fitted regression line without scatter
  diagram of raw data
• Superimposing graphs with different vertical
  scales
• Plotting means without any indication of
  variability

50
(No Transcript)