How to do Cochrane systematic reviews: Introductory workshop

1
How to do Cochrane systematic reviews
Introductory workshop

• Dr Jeremy Grimshaw
• Canadian Cochrane Network and Centre
• University of Ottawa

2
Outline of the course

• Introductions
• Introduction to systematic reviews
• Introduction to The Cochrane Collaboration
• Steps in doing a Cochrane review
• Coffee Break
• Advanced methods
• Appraising the risk of bias
• Locating and selecting studies for your
  systematic review

3
Introductions

• Name
• Background
• Previous experience of systematic reviews
• What you want to get out of the training?

4
Session 1 Outline

• Introduction to systematic reviews
• Introduction to The Cochrane Collaboration
• Rationale for developing a protocol
• Format of a protocol
• Resources to support reviewers

5
Problems of information management in healthcare

• Over 20,000 health journals published per year
• Published research of variable quality and
  relevance
• Research users (consumers, health care
  professionals and policy makers) often poorly
  trained in critical appraisal skills
• Average time professionals have available to read
  lt1 hour/week

6
Systematic reviews

• Systematic reviews use rigorous scientific
  methods to identify, assess and synthesise the
  worldwide available evidence

7
Systematic reviews

• A review of a clearly formulated question that
  uses systematic and explicit methods to identify,
  select, and critically appraise relevant
  research, and to collect and analyse data from
  the studies that are included in the review.
• Cochrane Collaboration (2005) Glossary of Terms
  in The Cochrane Collaboration

8
Meta-analysis

• The use of statistical techniques in a systematic
  review to integrate the results of included
  studies.
• Cochrane Collaboration (2005) Glossary of Terms
  in The Cochrane Collaboration

9
Meta-analysis
10
Systematic reviews

• Systematic reviews involve
• stating the objectives of the research
• defining eligibility criteria for studies to be
  included
• identifying (all) potentially eligible studies
• applying eligibility criteria
• assembling the most complete dataset feasible
• analysing this dataset, using statistical
  synthesis and sensitivity analyses, if
  appropriate and possible
• preparing a structured report of the research.

11
Systematic reviews

• Reduce bias
• Reduce random error
• Explore variability
• Provide reliable basis for making decisions
• Inform and influence future research

12
Traditional vs systematic reviews

• Traditional
• lack rigor
• methodology not transparent
• different reviewers reach different conclusions
• become out of date

• Systematic
• scientific rigor to minimise bias
• explicit and reproducible methodology
• regularly updated (Cochrane)

13
The Cochrane Collaboration

• It is surely a great criticism of our profession
  that we have not organised a critical summary, by
  specialty or subspecialty, adapted periodically,
  of all relevant randomized controlled trials.
• Archie Cochrane, 1979

14
The Prototype - Effective Care in Pregnancy and
Childbirth
15
The Cochrane Collaboration

• The Cochrane Collaboration is an international
  organization that aims to help people make
  well-informed decisions about healthcare by
  preparing, maintaining, and promoting the
  accessibility of systematic reviews of the
  effects of healthcare interventions.

16
Structure of The Cochrane Collaboration
Centres
Review Groups
Steering Group
Consumer Network
Fields
Methods Groups
17

• Cochrane Centres

• Promote awareness and support of The Cochrane
  Collaboration on a geographical basis
• support individual review authors
• support review groups in that area
• provide a unique function e.g. software
  development, consumer support
• link to government and other agencies
• not a production house for reviews

18
Canadian Cochrane Centre staff
19

• Cochrane Review Groups

• International and multidisciplinary
• Produce reviews
• Focused around health problems
• 51, covering most of health care
• Editorial base facilitates process

20
Cochrane Review Groups

• Worldwide there are over 11,000 review authors
  working with 51 registered review groups
• 5 review groups have their editorial base in
  Canada
• Back Group
• Effective Practice and Organisation of Care Group
  (EPOC)
• Hypertension Group
• Inflammatory Bowel Disease and Functional Bowel
  Disorders Group
• Musculoskeletal Group

21

• Cochrane Fields

• Represents a population, group, or type of care
  that overlaps multiple review group areas
• E.g., primary care, health promotion and public
  health, health care of older people
• Child Health Field and the Health Equity Field
  are located in Canada
• Ensures that their priorities are reflected in
  the work of review groups

22

• Cochrane Methods Groups

• Develops methodology and advise the Collaboration
  on how the validity and precision of systematic
  reviews can be improved
• Examples Statistical Methods, Non-Randomized
  Studies, Information Retrieval
• Reporting Bias Methods Group is located in Canada

23
Cochrane Consumer Network

• Facilitates the dissemination of information to
  patients, their families, friends and advocates
  throughout the world
• Plays an integral part to all activities within
  The Cochrane Collaboration
• Supports the role of consumers within The
  Cochrane Collaboration
• Liz Whamond (NB) current chair of The Cochrane
  Consumer Network

24
Cochrane reviews

• Cochrane reviews are conducted using standardised
  rigorous approaches including detailed critical
  appraisal of all studies
• Cochrane reviews are reported using standardised
  structure including abstract and consumer summary

25
Steps of a Cochrane systematic review

• STEP 1 formulate the problem and register the
  title with CRG
• STEP 2 write protocol, submit for peer review
  and publish
• STEP 3 locate and select studies
• STEP 4 critically appraise studies for risk of
  bias
• STEP 5 collect data
• STEP 6 analyse and present results
• STEP 7 interpret results and write review
• STEP 8 improve and update review

26
Title registration

• Decide which Cochrane CRG would be responsible
• relevant CRG may be obvious
• most groups are disease (or organ) focused
• if unsure which CRG, contact CCN/C for advice
• Information on CRGs
• The Cochrane Library
• www.cochrane.org
• Contact CRG directly
• Complete title registration form (from CRG)

27
Tips for titles

• Titles for Cochrane reviews have a set format
• Format intervention for problem in category
• Caffeine for daytime sluggishness in adults
  (hypothetical for today)
• Keep it short

28
Why have a protocol?

• Systematic reviews are scientific research
• Plan methods
• Reduce bias
• Access to peer review
• Avoid duplication of effort
• Rest of review follows
• focus for thinking about review
• planning and allocating tasks

29
Format of a Cochrane protocol

• Standard format
• to help review authors be systematic
• to help people reading the reviews to find
  information quickly

30
Format of a Cochrane protocol

• Title/review authors
• Background
• Objectives
• Selection criteria
• Search strategy
• Methods

31
Background, objectives and selection criteria
32
Background (Handbook Section 3)

• Description of the condition and its significance
• Description of the intervention and its role in
  practice
• How the intervention might work
• Why it is important to do the review

33
Objectives (Handbook Section 3)

• Follow naturally from the background
• What are the questions?
• does it work?
• is the effect consistent or variable?
• Questions should be clearly framed

34
Lumping and splitting

• Systematic review question can be framed broadly
  or narrowly
• Effects of audit and feedback to improve chronic
  diseases across all health care settings and
  professionals
• Effects of audit and feedback to improve chronic
  diseases within primary care
• Effects of audit and feedback to improve diabetes
  care within primary care

35
Lumping and splitting

• The lumping rationale - systematic reviews aim
  to identify the common generalisable features
  within similar interventions, minor differences
  in trial design are not important.
• Lumped reviews allow generalisability and
  consistency of findings to be assessed across
  wide range of settings and populations this
  reduces risk of bias or chance results
• The lumping principle is that the results of
  two interventions should be combined unless there
  are good grounds to believe they will have
  opposing effects.

36
Lumping and splitting

• The splitting rationale is that it is only
  appropriate to combine trials which are very
  similar in design, patient selection,
  intervention characteristics and outcome
  recording.
• Split reviews avoid combining apples and
  oranges
• Reviews can be split by participants,
  interventions or outcome
• Very narrowly focused reviews are de facto
  subgroup analyses

37
Lumping and splitting

• Practical considerations
• Lumped reviews
• Logistically challenging (large number of
  included studies)
• Analytically challenging
• Heterogeneity expected
• Interpretation maybe challenging seeing the
  woods for the trees
• Split reviews
• Easier, quicker, cleaner

38
Lumping and splitting

• Effects of audit and feedback (85 studies)
• Effects of audit and feedback to improve chronic
  diseases across all health care settings and
  professionals (18 studies)
• Effects of audit and feedback to improve chronic
  diseases within primary care (14 studies)
• Effects of audit and feedback to improve diabetes
  care within primary care (3 studies)

39
Selection criteria (Handbook Section 4)

• Follow naturally from the objectives
• type of participants
• type of interventions
• type of outcomes
• type of studies

40
Purpose of selection criteria

• Help you design the search
• Selecting studies
• Start thinking about the analysis
• One way to minimise bias

41
Small group exercises
42
Writing the methods and completing the protocol
(Handbook sections 6,7,8)
43
Purpose of methods (Handbook sections 6,7,8)

• Plan what you will do before you start
• Divide work among review authors and establish
  timeline
• Minimise bias
• Start thinking about the analysis

44
Methods (Handbook sections 6,7,8)

• Inclusion of trials
• - how?
• - by who?

45
Methods(Handbook sections 6,7,8)

• Assessment of risk of bias
• - type of assessment
• - by who?
• - how to incorporate in the review?

46
Methods (Handbook sections 6,7,8)

• Data extraction
• - what will be extracted?
• - by who?
• - how to deal with missing data?

47
Methods (Handbook sections 6,7,8)

• Data synthesis
• - what type of synthesis to do?
• - what summary measures to use?
• - what comparisons to make?
• plans to investigate for differences in study
  results (heterogeneity) or potential for missing
  studies (publication bias)
• plans for subgroup or sensitivity analyses?

48
Writing your protocol

• Make sure all sections are completed
• - title/review authors, background, objectives,
  selection criteria, search strategy, methods
• Dont forget the other bits and pieces
• - contact details of review authors,
  acknowledgements, conflict of interest, sources
  of support, references, additional tables

49
Resources for Cochrane reviewers

• The Cochrane Handbook
• Open learning support
• Revman
• Support from CRGs
• Trial search co-ordinators
• Review group co-ordinators
• Contact editor
• Support from CCN/C
• Training
• Whatever else we can do!

50
After protocol is complete

• Submit to CRG for editorial process
• Internal and external peer review 1-2 months
• May receive critical comments to respond to
• Protocol published on The Cochrane Library
• Reward yourself then start the review
• Note - if you need to change your plan report
  it and justify it (be transparent)

51

• COFFEE!

52
Session 2 Outline

• Locating and selecting studies for your
  systematic review
• Appraising the risk of bias

53
Locating and selecting studies for your
systematic review
54
Why is a rigorous approach to searching important

• To achieve up-to-date, relevant, unbiased
  systematic reviews of the effects of
  interventions
• To overcome problems with limited searching
• bias
• lack of precision and generalisability

55
Finding as much as possible and being strategic

• Use multiple sources and start with the highest
  yield
• Electronic databases
• Cochrane Review Group register
• Cochrane Central Register of Controlled Trials
  (CENTRAL)
• MEDLINE
• EMBASE
• Other databases depending on your topic
• Reference lists of trials and reviews
• Handsearching journals and conferences
• Personal communication

56
Designing a search strategy

• Work with TSC of review group
• Formulate your question (PICO)
• Identify your search terms
• Put it all together

57
Formulate the question

• Translate the clinical problem into a structured
  question and identify the key concepts
• Participants
• Interventions
• Outcomes
• Studies
• Develop search terms (commonly based on
  participants, interventions, study design)

58
Searching tip

• Searching is often a matter of trying terms and
  seeing what result you get
• Find additional terms from the citations you find
  that are helpful how are they indexed what are
  the keywords used?

59
Documenting your search

• Important for you, so you can remember what
  youve done
• Important for users of your review
• Document
• what you searched
• when you searched
• how you searched

60
Additional points

• No right time to stop searching be guided by
  returns and resources
• Ideas for search terms from citations you know
  of try them and check returns
• Use a referencing database to manage citations

61
Minimising bias when searching

• Dont limit search on
• language
• source (which journal, database etc)
• publication status (full article, conference
  abstract, unpublished etc)
• publication date

62
Inclusion and exclusion of studies

• Develop and pilot a form based on your selection
  criteria
• Scan and retrieve possibly relevant studies
• Decide if studies meet inclusion criteria and
  record reasons for exclusion
• How reproducible are the decisions?
• Done by two independent review authors
• Beware of duplicates

63
Appraising the risk of bias
64
What is bias?

• Bias is systematic (non-random) error arising
  from the design or conduct of a trial
• Previously termed methodological quality
  updated to susceptibility to bias or risk of
  bias

65
Bias vs Quality
Bias
Quality
Systematic error influencing trial estimate of
the true effect of an intervention
Based on the conduct of the trial did trialists
go to sufficient lengths to minimise potential
bias?
66
Why assess susceptibility to bias?

• To determine whether biases could have affected
  the results
• Studies with less susceptibility to bias better
  estimate the truth about the effect of an
  intervention
• Biased primary studies that provide misleading
  results could generate misleading systematic
  reviews

67
(No Transcript)
68

• Which biases make a difference to trial results?

69
(No Transcript)
70
(No Transcript)
71
Target Population
Random sequence Allocation concealment
Selection
Performance
Detection
Attrition
72
Target Population
Selection
Performance
Detection
Attrition
73
Selection bias

• systematic differences in participant
  characteristics at the start of a trial

74
Selection bias

• Occurs during allocation into comparison groups
• Avoiding selection bias
• random unpredictable sequence
• allocation concealment (protection of the
  sequence most important)

75
Selection bias

• Adequate allocation concealment
• central computer randomisation
• on-site computer from which assignment can only
  be determined after entering patient data
• serially numbered, sealed, opaque envelopes
• Inadequate allocation concealment
• non opaque envelopes
• odd or even date of birth or medical record
  number

76
Target Population
Selection
Performance
Detection
Attrition
77
Performance bias

• Occurs during the intervention phase
• Systematic differences, other than the
  intervention being investigated, in the treatment
  of the two groups
• Participant expectations influencing their
  response to treatment
• Avoiding performance bias
• - blinding the provider
• - blinding the participant

Schulz KF Grimes DA 2002 Lancet
78
Allocation concealment vs blinding
79
Target Population
Selection
Performance
Detection
Attrition
80
Detection bias

• Occurs during the follow up phase
• Systematic differences in the outcome assessment
  between groups
• Avoiding detection bias
• - blinding the outcome assessor
• - blinding data analyst/investigator?

81
Target Population
Selection
Performance
Detection
Attrition
82
Attrition bias

• Occurs over the duration of follow up
• Systematic differences in the loss of
  participants to follow up between groups
• Completeness of follow up
• - participants lost to follow up/ not included
  in outcome assessment, could be different from
  those who remained in the trial
• avoiding attrition bias
• - describe proportion of participants lost to
  follow-up
• - use intention-to-treat analyses

83
Attrition bias

• Intention-to-treat (ITT) analysis
• - once randomised, analysed
• - regardless of whether participants
• received the intervention
• deviated from the protocol
• withdrew from the study
• - maintains randomised allocation of participants

84

• What is the evidence for these biases influencing
  trial results?

85
Empirical evidence that bias affects results

• Selection bias
• Inadequate/unclear allocation concealment
  associated with exaggerated treatment effect
  (average 29 increased benefit)1,2,3,4
• Performance bias
• Open trials (no blinding) exaggerate benefits
  of treatment (average 14 increased benefit)
  1,2,3,4

• Schulz et al, JAMA 1995
• Moher et al, Lancet, 1998
• Kjaergard et al, Ann Intern Med, 2001
• Egger et al, HTA, 2003
• Juni et al, JAMA, 1999

86
Empirical evidence that bias affects results

• Detection bias
• Open outcome assessment exaggerated treatment
  effects (171 and 355)
• Attrition bias
• Handling of dropouts did not affect results1,5
  however, compared whole group differences (not
  between group differences). Not an exclusion but
  important to report.
• Schulz et al, JAMA 1995
• Moher et al, Lancet, 1998
• Kjaergard et al, Ann Intern Med, 2001
• Egger et al, HTA, 2003
• Juni et al, JAMA, 1999

87
Assessing susceptibility to bias

• How do we assess susceptibility to bias?
• - Scales
• - Checklists

88
Beware of quality scales.

• Different scales are available (gt50)
• - vary in dimensions covered, size, complexity
• - some provide composite numerical score
• Reliability and validity of many scales not
  established
• May lead to different conclusions regarding the
  susceptibility to bias of the same trial (Juni
  1999)

89
Treatment estimates from high-quality trials
Treatment estimates from low-quality trials
Same group of trials divided into high and low
quality using 25 different Quality Assessment
Scales
Juni P, Witschi A, Bloch R, Egger M. The hazards
of scoring the quality of clinical trials for
meta-analysis. JAMA 1999282(11)1054-60
90
How to assess susceptibility to bias

• Component approach assessment against individual
  criteria

91
Assessing susceptibility to bias in reviews

• Was the allocation sequence adequately generated
  (randomised)?
• Was the allocation adequately concealed?
• Was there blinding of participants, personnel
• Was there blinding of outcome assessors?
• Were dropouts and exclusions adequately
  addressed?
• Other (including selective outcome reporting and
  other criteria for addressing topic-specific or
  design-specific biases)

92

• How do we incorporate susceptibility to bias in
  meta-analysis?

93
Incorporating susceptibility to bias in
meta-analysis

• Exclude trials with high susceptibility to bias
• - critical a priori cutoff for inclusion (e.g.
  only trials with adequate allocation concealment
  or blinding)
• - may be problematic if few trials
• Sensitivity analysis
• - analyse separately above and below a certain a
  priori level and compare results (does it change
  conclusions?)
• - for instance, analyse the impact of allocation
  concealment, blinding, use of ITT analysis, on
  results

94
Example of sensitivity analysis
95
But beware

• Scant empirical evidence about which biases are
  important
• Reporting of susceptibility to bias is not always
  the same as the susceptibility to bias in the
  conduct of the trial
• - allocation concealment unclear in 87 of
  trials as per published report (Pildal et al,
  2005)
• - allocation concealment unclear in about 80 of
  trials (Middleton 2001)
• CONSORT statement (1994, 2001) www.consort-stateme
  nt.org
• Possible discrimination against older trials

96
Important points for your review

• Plan criteria and how you will incorporate
  susceptibility to bias at the protocol stage
• Assessment undertaken once trials are included
  and prior to data extraction
• 2 independent review authors