Systematic Review: Workshop 1 26 July 2007 - PowerPoint PPT Presentation


PPT – Systematic Review: Workshop 1 26 July 2007 PowerPoint presentation | free to download - id: 686e24-NWY5Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Systematic Review: Workshop 1 26 July 2007


Workshop 1 26 July 2007 Chris Bridle, PhD, CPsychol Institute of Clinical Education Warwick Medical School University of Warwick – PowerPoint PPT presentation

Number of Views:29
Avg rating:3.0/5.0
Slides: 46
Provided by: www2Warwi9
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Systematic Review: Workshop 1 26 July 2007

Systematic ReviewWorkshop 126 July
Chris Bridle, PhD, CPsychol Institute of
Clinical Education Warwick Medical School
University of Warwick Tel 44(24) 761 50222
Workshop Overview
  • Systematic review 70 mins
  • Break 15 mins
  • Programme structure 10 mins
  • Protocol development 45 mins
  • Questions -

Why is research synthesis needed?
  • 2 million articles published in 20,000 health
    journals pa
  • Equivalent to a pile of paper 900 metres high
  • If you tried to stay current by reading two
    articles each day, within 1 year you would fall
    55 centuries behind!
  • Put another way, if you try to read everything of
    possible relevance, you would have to read 5,500
    articles per day.
  • Seeing is believing ...

What burns me up is that the answer is right
here somewhere, staring us in the face if
only Chris B was here!
Expert Advice?
What and When?
  • What? Thalidomide
  • When? Before 1960
  • The evidence was there, but the methods for
    making it accessible and meaningful were not

Treatment ltFavoursgt Control (Fewer ltAdverse
Eventsgt More)
  • Dont forget the slides handout!

Why we need research synthesis
  • Vast quantity of research ?
  • Single trials rarely definitive ?
  • Hidden effects ?
  • Research and practice built upon ?
    biased /or unfounded assumption
  • Need for research synthesis ?
  • How can research evidence be
  • synthesized?

Chalk and Cheese
You Decide
  • Traditional Review
  • Take a simmering topic, extract the juice of one
    argument, add the essence of one filling cabinet,
    sprinkle liberally with your own publications and
    sift out the work of noted detractors
  • A non-scientific method prone to bias in
    evidence retrieval, evaluation, synthesis and
  • Systematic Review
  • A review of the evidence on a clearly formulated
    question that uses systematic and explicit
    methods to identify, evaluate and synthesise
    relevant primary research
  • A scientific methodology that is reproducible and
    amenable to rigorous evaluation

Formulate Review Question
Key stages in conducting a systematic review
Develop Protocol
Search, Identify Select Evidence
Data extraction
Quality Assessment
Data Synthesis Interpretation
Conclusions Recommendations
Write Review Report
Formulating a Review Topic / Question
  • Example question
  • Effects of interventions for overweight children
  • This is a bad question!
  • It is much too broad, would take years to answer,
    and results unlikely to be meaningful
  • Needs to be narrower, i.e. more focussed
  • What effects?
  • Effects of what exposure?
  • Among who?
  • Compared to what / who? Etc., etc., etc.

Focussing the Review
  • Answerable questions include
  • A description of the population P
  • An identified intervention I
  • An explicit comparison C
  • Relevant outcomes O

Overweight children Community
walking Orilstat Weight loss
Community walking versus orilstat for weight loss
among overweight children
PICO Components
  • Identify the PICO components
  • Effectiveness of homeopathy
  • Stage-based interventions for smoking cessation
  • Psychosocial interventions for prevention and
    treatment of childhood obesity
  • Effectiveness of atypical antipsychotics for
    treatment of mania in bipolar disorder

The Review Protocol
  • The review protocol is the first major milestone
    of any systematic review
  • Provides a rigid, well-specified plan for how
    each stage of the review will be conducted a
  • Helps to avoid or minimise bias at each stage of
    the review but only if followed rigidly
  • Should be sent for external peer review, e.g.
    advisory group comprising researchers,
    practitioners, users, etc.
  • Should be published, e.g. project webpage,
    research register, or peer-reviewed journal

Comprehensive Literature Search
  • Necessary to reduce bias
  • Publication, language, and geographical
  • Search media
  • Electronic, manual, and personal
  • Publication type
  • Journal, conference, dissertation, grey, and

From question to search terms
  • For each PICO component identify and record
  • Abbreviations, e.g. CBT, TTM, MS, CFS
  • Synonyms, e.g. CBT - Theory or Therapy
  • Related terms cognitive theory/therapy, behavior
  • Transatlantic differences, e.g. pressure sores /
    decubitus ulcers
  • Spelling, e.g. behavior or behaviour
  • Set limits?
  • Publication, e.g. published language date, etc.
  • Study, e.g. sample size, follow-up time, type of
    outcome assessment, etc.
  • Evidence reliability, e.g. study design specific
  • Hierarchy of evidence

Systematic Review
Experimental Study
Potential For Bias - -
Cohort Studies
Case-Control Studies
Observational Studies
Case Reports
Cross Sectional Surveys
You, and me
Expert Opinion
Drunk Bob
Question Evidence Relationship
  • Question Type Relevant Evidence
  • Intervention - SR, RCT, Cluster RCT
  • Prognosis - Cohort study
  • Etiology - Cohort, case-control
  • Harm - Cohort, case-control
  • Diagnosis - Cross-sectional, case-control
  • Experience - Qualitative research

Building a Search Strategy
  • Record everything!
  • Search history, e.g. database, host and date
  • Search strategy, e.g. terms and combinations
  • Search results
  • Design a search-term table
  • Enter search terms in relevant column
  • Combine search terms for each column with or
  • Combine column results with and less
  • See example

Preparing to be flooded
  • How will you manage the many (thousands) of
    references and abstracts your search has
  • Bibliographic software
  • Endnote
  • RefWorks
  • ProCite
  • RevMan

Study Selection
  • What references to retrieve?
  • Study information limited to Abstract or Title
  • Assess potential relevance of studies against
    inclusion / exclusion criteria (i.e. P.I.C.O)
    retrieve relevant full text papers
  • What studies to include?
  • Use full text papers to assess the relevance of
    studies against the study inclusion / exclusion
    criteria - PICO
  • The remaining papers are those relevant to your
  • Keep precise records from start to finish
  • Account for all references (studies)

Example Flow chart of study papers through a
reviewStudy papers are to a review, what
participants are to a trial
Data Extraction
  • Recording of information presented in primary
  • Ranges from superficial reporting to systematic
  • Systematic extraction / recording is important
    for data synthesis and interpretation
  • Important to strike the right balance between
    reporting / recording too much or too little
  • Determined by the nature of the review and its
    intended use / audience

Extracting Study Data
  • Important data to extract
  • Participants demographic / disease
  • Intervention description of intervention and
  • Analysis statistical method, number of Ps, ITT
  • Results be selective and sensible, e.g. primary
  • Authors conclusions word-for-word reporting of
  • Reviewers comments validity of results,
    calculation of summary (effect) statistic, etc.

Data Extraction Example
Participants Intervention Results Conclusion / Comment
N Randomised 290 (I150, C140) Age m43. Gender 30 female Type UK Community (Patient) Recruitment Non-smoking related attendance at GP surgery I 3 x 30 min weekly stage-based, group MI with take-home intervention pack. C GP advice Provider GP Nurse SettingSurgery Follow-up2 month Outcome Abstinence 3 wks, self-report Dropout 82 (I53, C29) N Analysed 208 (I97, C111) Abstinence 31 (I19, C12) (plt0.05) Reviewer ITT OR1.54 (95 CI, 0.63 to 4.29) Brief, stage-based MI with take-home material is an effective smoking cessation intervention. Comment High attrition (I, OR2.09) and ns effect with ITT Unknown inclusion criteria Unclear tailoring (group-MI) Follow-up too short 80 relapse/failure to 12 months Authors conclusions are inconsistent with data
Tabulated Data Example
  • Tabulated data are usually presented in the text
    of the report / paper

Study Ps Exposure Context Control Outcome (Abstinence) Summary effect OR (95CI) Validity
Smith, 2003 290, UK GP patients Group MI written advice Nurse, GP surgery, 3 pw Usual care Self-report at 2 months 1.54 (0.63,4.29) Poor
Jones, 2004 600, UK community Group MI Researcher, community centre, 2 pw No intervention Biochemical validation at 12 months 1.03 (0.33,1.22) Good
Davis, 2005 100, UK students Stage-based Written material No intervention Self-report at 2 months 2.54 (1.33,4.89) Poor
Scott, 2006 60, UK GP patients Individual MI Counsellor, home visit, 1pw No intervention Self-report at 1 month 1.87 (1.12,3.19) Poor
Critical Appraisal
  • Estimates potential for bias
  • Extent to which results attributable to factors
    other than named intervention
  • Independent application of a priori criteria
  • Criteria depend on study design
  • Common criteria for RCTs
  • Adequate randomisation Concealed allocation a
    priori sample size calculation Blinding
    Baseline comparability Adequate follow-up
    Intention-to-treat analysis

Importance of Criteria
Quality Item Overestimation of intervention effect
Flawed randomisation 41
Unclear randomisation 30
Lack of blinding 17
No a priori sample size calculation 30
Failure to use ITT analysis 25
Trials with poor quality of reporting 20
Khan et al (1995) Moher et al (1998)
Uses of Quality Assessment
  • Determine the strength of conclusions
  • A threshold for inclusion
  • Potential source of heterogeneity
  • As weights in statistical analyses unwise!
  • Recommendations for improving future research

Data Synthesis
  • Quantitative
  • statistically combine results of several studies
    into a single numerical estimate of effect, e.g.
  • Only used for (some) quantitative data
  • Qualitative
  • a narrative summary and synthesis of primary
  • Used for both quantitative and qualitative data

Selecting a Method of Synthesis
  • Decision based on degree of heterogeneity
  • Variation within the group of included studies
  • Types of heterogeneity
  • Clinical participants, interventions and
    outcomes, e.g. age, intervention intensity,
    outcome definition, etc.
  • Methodological how studies were conducted, e.g.
    study design, study quality
  • Statistical measured intervention effect you
    should calculate a summary statistics for each
    study yourself

Workplace exercise intervention for mild
Assessing Statistical Heterogeneity
  • Visual inspection
  • Overlap of CIs
  • Chi-square statistic
  • Significant plt.01
  • Power, too much, too little
  • Statistic Vs degrees of freedom
  • Meaningless question
  • Is heterogeneity present? Of course it is!
  • We want to know how much heterogeneity is
    present, where, and is it important?

Dealing With Heterogeneity
  • Subgroup analyses
  • Comparison of separate meta-analyses based on
    subgroups of the studies
  • Sensitivity analyses
  • Repeating meta-analyses with amended criteria to
    determine robustness of review findings

Subgroup analyses Mortality results of
zidovudine trials in HIV, stratified by infection
stage (early or late).
Sensitivity analyses Case-control studies
relating residential exposure to electromagnetic
fields to childhood leukaemia, stratified by
methodological quality.
  • Subgroup and sensitivity analyses need to
    interpreted with caution
  • Strong conclusions are often drawn from subgroup
    / sensitivity analyses misleading and dangerous
  • Results based on indirect comparisons
  • Any differences are observational in nature
  • Prone to bias and confounding

Narrative Synthesis
  • Used for both quantitative and qualitative data
    but no formal method of synthesis
  • Studies meaningfully grouped and explored for
    potential sources of effect heterogeneity
  • Participants age, gender, ethnicity, SES
  • Interventions setting, provider, intensity,
  • Outcomes definition, measured, collection,
  • Methods quality, design, sample size, protocol

  • Must be based solely on the evidence reviewed
  • Should highlight implications for research,
    practice, policy, decision makers, etc.
  • May be graded according to the strength of the
  • May indicate a time period within which the
    findings of the review are relevant

Core Systematic Review Stages
Formulate Review Question
Advisory Group
Scoping Search
Develop Protocol
Publish Protocol
Independently Assessed
Search, Identify Select Evidence
Reasons for Exclusion
Check for accuracy
Data extraction
Review Processes - Inputs
Review Outputs
Independently Assessed
Quality Assessment
Investigate Heterogeneity
Data Synthesis Interpretation
Advisory Group Review Drafts
Conclusions Recommendations
Develop Research Bid
Publish Review
Write Review Report
Critical Appraisal of SRs
  • Question Is the review question clear, specific
    and answerable?
  • Search Have attempts to identify relevant
    evidence been sufficiently comprehensive?
  • Evaluation Have included studies been critically
    appraised against appropriate criteria?
  • Synthesis Is the method of synthesis appropriate,
    and have potential sources of heterogeneity been
  • Conclusions Do conclusions reflect both the
    quality and quantity of evidence?
  • Process Is the review process independent and
    transparent so as to limit potential bias?

  • Systematic reviews are important
  • Meet the need for research synthesis
  • Use scientifically rigorous methods
  • Provide evidence-based answers to meaningful
  • Identify gaps and weaknesses in evidence base
  • Improve research quality
  • Enhance effectiveness of clinical practice
  • Strengthen bids for research funding

Good Guidance
  • Centre for Reviews Dissemination Report 4
  • Cochrane Collaboration Reviewers Handbook
  • Cochrane Behavioral Medicine Recommended paper
  • http//
  • WMS Systematic Review Training Materials

  • Any questions?
  • Now, or later
  • Chris Bridle PhD, CPsychol
  • Institute of Clinical Education
  • Warwick Medical School
  • University of Warwick
  • Tel (024) 761 50222
  • Email
  • Web

Programme Overview
Date Review Workshop Intervening Tasks Objectives
July Planning Developing a review protocol Formulating a review question / objective Methods for conducting the review Develop protocol with completed background, specified objective/question Outline methods. Submit draft for assessment, amend and submit to expert for comment.
Sept Conducting Search strategy Data extraction Quality assessment (Data synthesis Heterogeneity Interpretation) Update protocol methods (search, extraction template, assessment criteria) and submit. Conduct review as specified in approved protocol, submit draft review report (data synthesis plan).
Oct Reporting Report structure Manuscript guidelines Dissemination strategy Targeted publication (Data synthesis) Synthesise data Write draft report Identify peer-review jrnl Obtain manuscript guidelines. Submit draft report and dissemination strategy for internal review and respond to feedback.
Nov Disseminating Publishing the review Target audiences Dissemination media Responding to peer-review Medical education Present completed review to peers Reflect on programme experience / evaluation feedback. Write full report for internal peer-review, submit for publication and go to the pub to celebrate.