What is a C2 Review Protocol?

1
What is a C2 Review Protocol?

• Presented by
• Harris Cooper
• University of Missouri-Columbia
• Larry V. Hedges
• University of Chicago

2
A Review Protocol is a document that

• Sets out the reviewers intentions with regard to
  the topic and the methods to be used in carrying
  out a proposed review
• Is meant for inclusion in the Campbell Database
  of Systematic Reviews

3
How were the Guidelines for a C2 Review Protocol
Established?
4
The C2 Protocol Guidelines were

• Drafted by the Methods Working Group at the
  request of the Steering Committee using the
  Cochrane Collaboration Protocol Guidelines and
  other sources as guides
• Reviewed (via e-mail) by attendees of the C2
  Inaugural Meeting, sent to approximately 80
  people, 7 of whom provided feedback
• Reviewed by the Steering Committee
• Approved by the Steering Committee (December 22,
  2000)

5
Why are C2 Review Protocols Necessary?

• Preparing a review is a complex process that
  comprises many judgments and decisions.
• The methods to be used should be established
  beforehand (to the extent possible) because
• The studies to be included are usually identified
  after they have been completed
• The results of many of these studies may already
  be known to the reviewers

6
Why are C2 Review Protocols Necessary?

• Therefore, it is important to make the review
  process as well-defined, systematic, and unbiased
  as possible while maintaining a practical
  perspective.
• Requiring detailed protocols is one way to
• Allow the editorial referees to provide guidance
  and advice
• Prevent problems from occurring during the review
  process
• Ensure that final products will meet the
  standards of the Campbell Collaboration

7
What Should a C2 Protocol Contain?
8
A protocol for a Campbell Review should consist
of the following sections

1. Cover Sheet
2. Background for the Review
3. Objectives for the Review

9
A protocol for a Campbell Review should consist
of the following sections

• Methods
• Criteria for inclusion and exclusion of studies
  in the review
• Search strategy for identification of relevant
  studies
• Description of methods used in the component
  studies
• Criteria for determination of independent
  findings
• Details of study coding categories
• Statistical procedures and conventions
• Treatment of qualitative research

10
A protocol for a Campbell Review should consist
of the following sections

1. Timeframe
2. Plans for Updating the Review
3. Acknowledgements
4. Statement Concerning Conflict of Interest
5. References
6. Tables

11
What Should a C2 Protocol Cover Sheet Contain?
12
The cover sheet of the protocol should include

• The title of the review
• The names of the reviewers
• Contact information for the lead reviewer
• Sources of support

13
What should a C2 Protocol Background Section
Contain?
14
The background section of the protocol should
present

• An overview of the theoretical, conceptual,
  and/or practical issues surrounding the research
  problem
• A general description of prior reviews, the
  controversies these reviews have created or left
  unresolved, and which of these will be the focus
  of the new review effort

15
The background section

• Sets out the context of an already formed body of
  knowledge
• Provides the rationale for the review
• Explains why the questions being asked are
  important
• Sets the stage for the empirical results that
  follow

16
The background section should contain

• A conceptual discussion of the research problem
• A brief overview of the research question
  including its theoretical, practical, and
  methodological history
• The qualitative and historical debates
  surrounding the research question
• A discussion of previous reviews of the research
  topic

17
The background section should answer questions
such as

• From where does the problem, approach, and/or
  intervention in the research come?
• Do debates exist surrounding the meaning of the
  problem or utility of the intervention?
• Do theories predict how the major variables
  involved in the review will be related to one
  another?
• Do different theories or philosophies of
  treatment yield conflicting predictions?

18
What Should a C2 Protocol Objectives Section
Contain?

• All Campbell Collaboration Reviews are undertaken
  to gather, summarize and integrate empirical
  research so as to help people understand the
  evidence.

19
Within this overarching framework, reviews can,
for example, be meant to

• Produce general statements about relationships
  and treatment effects through the synthesis of
  individual study results
• Find reasons for conflicting evidence
• Explain variations in practice

20
Within this overarching framework, reviews can,
for example, be meant to

• Answer questions, using variations in studies,
  that could not have been answered in the
  individual component studies
• Review the evidence on the subjective experience
  of an intervention
• Build connections between related areas of
  research

21
In setting out the objectives, reviewers should
keep in mind that Campbell Reviews must

• Help people make practical decisions about social
  behavioral interventions and public policy
• Address the choices (practical options) people
  face when deciding about whether or not to adopt
  a policy or practice
• Relate to outcomes that are meaningful to people
  making decisions about public policy

22
What should a C2 Methodology Section Contain?

• The methods section should describe operationally
  how the review will be conducted.

23
What should a C2 Methodology Section Contain?

• The methods sections of a C2 review will differ
  considerably from that of a primary research
  study.

24
What should a C2 Methodology Section Contain?

• Most methods section will need to address several
  separate issues
• Criteria for inclusion and exclusion of studies
  in the review
• Search Strategy for identification of possibly
  relevant studies
• Description of methods used in primary research

25
What should a C2 Methodology Section Contain?

• Criteria for determination of independent
  findings
• Details of study coding strategies
• Statistical procedures and conventions that may
  be used
• How qualitative research will be treated in the
  review

26
How should a C2 Protocol Address Questions of
Study Inclusion/Exclusion?

• The protocol should explain the criteria that
  will be applied to determine relevance of studies
  uncovered by the search.
• It is critical that these criteria be explicated
  in advance.

27
How should a C2 Protocol Address Questions of
Study Inclusion/Exclusion?

• This section of the protocol should answer
  question such as
• What characteristics of studies will be used to
  determine if a study was relevant to the topic of
  interest?
• What characteristics of studies will lead to
  exclusion?

28
How should a C2 Protocol Address Questions of
Study Inclusion/Exclusion?

• Will decisions be based on
• Report tile?
• Abstracts?
• Full Reports?
• Who will make the relevance decisions?
• How will reliability of relevance decisions be
  assessed?

29
How should a C2 Protocol Address Questions of
Study Inclusion/Exclusion?

• Examples of studies that would be included and
  excluded should be given.

30
How should a C2 Protocol Address Questions of
Study Inclusion/Exclusion?

• C2 reviews can include evidence from studies of
  implementation. This evidence can derive from a
  broad range of qualitative and quantitative
  evidence.

31
How Should a C2 Protocol Address Questions of
Search Strategy?

• Reviewers should present details of their search
  strategy, such as
• Reference databases used
• Hand searches of specific journals
• Personal contacts

32
How Should a C2 Protocol Address Questions of
Search Strategy?

• It should also include a rationale for the search
  strategy, bearing in mind that the ultimate goal
  is to reduce the differences between the body of
  retrieved studies and those that go undiscovered.

33
How Should a C2 Protocol Address Questions of
Search Strategy?

• Reviewers need to report the keywords, years
  searched, and search strategy used to guide the
  search of reference databases and bibliographies.

34
How Should a C2 Protocol Address Questions of
Search Strategy?

• Reviewers also need to indicate mechanisms they
  will use to retrieve documents, especially
  unpublished ones.

35
How Should a C2 Protocol Address Questions of
Search Strategy?

• As far as possible, evidence considered should
  not be restricted by nationality of investigators
  or by language unless there is a good
  justification to do so.

36
How Should a C2 Protocol Address Descriptions of
Primary Research Methods

• The protocol should describe the methods most
  commonly used in the primary research covered in
  the review.

37
How Should a C2 Protocol Address Descriptions of
Primary Research Methods

• This should focus more on designs actually
  realized than on theory, e.g.
• Participant sampling procedures
• Research designs
• Measurement methods

38
How Should a C2 Protocol Address Descriptions of
Primary Research Methods

• Reviewers should identify a few studies that
  illustrate the methods used in primary research
  and present the details of these studies.

39
How Should a C2 Protocol Address Criteria for
Determining Independence on Findings?

• Reviewers need to describe how they will handle
  studies that produce multiple findings of
  effectiveness based on the same data.

40
How Should a C2 Protocol Address Criteria for
Determining Independence on Findings?

• This can happen when
• Several types of outcome are measured on the same
  subjects
• The same outcome is measured at several points in
  time
• Several treatment groups are compared to the same
  control group
• Several different studies (publications) use data
  from the same people

41
How Should a C2 Protocol Address Criteria for
Determining Independence on Findings?

• If more than one outcome is used from the same
  study (or from different studies using the same
  people) reviewers need to explain how they will
  assure that the different outcomes will be
  statistically independent.

42
How should a C2 Protocol Describe Coding
Categories?

• Characteristics of studies that will be coded and
  examined for potential use as moderators of study
  outcomes should be described.

43
How should a C2 Protocol Describe Coding
Categories?

• All retrieved characteristics should be
  mentioned, even if they are not eventually used.

44
How should a C2 Protocol Describe Coding
Categories?

• If some outcomes or moderators are excluded, a
  rationale should be given for that exclusion.

45
How should a C2 Protocol Describe Coding
Categories?

• Information about quality assurance of coding
  (e.g., coding reliability assessment and
  monitoring) should be included in this section.

46
How Should a C2 Protocol Describe Statistical
Procedures?

• This section should describe the procedures and
  conventions used to carry out the quantitative
  analysis of results.

47
How Should a C2 Protocol Describe Statistical
Procedures?

• It should explain
• The effect size measure used
• Any adjustments that are made to effect sizes to
  reduce bias
• The techniques that will be used to combine
  evidence
• How missing data will be handled

48
How Should a C2 Protocol Describe Statistical
Procedures?

• How statistics describing the overall literature
  will be presented
• What techniques will be used to assess
  variability of results
• What techniques will be used to explain variation
  in results
• What sensitivity analyses will be carried out

49
How Should a C2 Protocol Describe Statistical
Procedures?

• A rational for each of the choices above should
  be given.

50
How Should a C2 Protocol Describe Statistical
Procedures?

• If no quantitative synthesis will be carried out,
  rationale should be given for the alternative
  technique.

51
How should a C2 Protocol Address Treatment of
Qualitative Research?
52
Qualitative Studies Can Be Part of C2 Reviews

• Qualitative studies can assist in
• Defining interventions more precisely
• The choice of relevant outcome measure
• The development of valid research questions
• Interpretation heterogeneous results

53
Qualitative Studies Can Be Part of C2 Reviews

• When a review contains relevant qualitative
  research, reviewers should operationally describe
  for qualitative studies
• Criteria for inclusion and exclusion of
  qualitative studies
• Methods used in these primary research studies
• Criteria for determining independent findings
• Characteristics of included studies in the same
  detail as they do for quantitative research.

54
What Should a C2 Protocol Timeframe Contain?
55
Examples of some benchmarks to be used in setting
targets are the anticipated dates for completion
of

• Searches for published and unpublished studies
• Pilot testing of inclusion criteria
• Relevance assessments
• Pilot testing of study codes and data collection
• Extraction of data from research reports
• Statistical Analysis
• Preparation of report

56
What should a C2 Protocol Timeframe Contain?

• Reviewers together with the editors of their
  Collaborative Review Group, must determine and
  appropriate timeframe for a specific review.
• Targets may vary widely from review to review
  depending on its scope and complexity, as well as
  the resources available.

57
What Should a C2 Protocol Plan for Updating the
Review Contain?
58
Updating Plans should include

• Specifications for how the review, once
  completed, will be updated
• Information on who will be responsible for
  updates
• The frequency with which updates can be expected

59
What Should a C2 Protocol Acknowledgements
Section Contain?

• Acknowledgment should be made of all individuals
  contributing to the preparation of the protocol
  who were not listed on the cover sheet.

60
What Should a C2 Protocol Statement Concerning
Conflict of Interest Contain?

• It is a matter of Campbell Collaboration policy
  that direct funding from a single source with a
  vested interest in the results of the review is
  not acceptable.

61
Reviewers should report any conflict of interest
capable of influencing their judgments including

• Personal
• Political
• Academic
• Financial

62
Campbell Reviews should be free of any real or
perceived bias introduced by

• The receipt of any benefit in cash or kind
• Any hospitality
• Any subsidy derived from any source that may have
  or be perceived to have an interest in the
  outcome of the review
• Financial conflicts of interest cause the most
  concern. They can and should be avoided, but
  must be reported if there are any.

63
Campbell Reviews should be free of any real or
perceived bias

• It is impossible to abolish conflict of interest,
  since the only person who does not have some
  vested interest in a subject is somebody who
  knows nothing about it.

64
Campbell Reviews should be free of any real or
perceived bias

• Disclosing a conflict of interest does not
  necessarily reduce the worth of a review and it
  does not imply dishonesty.

65
Campbell Reviews should be free of any real or
perceived bias

• Reviewers should include statements in their
  protocol about potential conflicts even when they
  are confident that their judgments will not be
  influenced.

66
Campbell Reviews should be free of any real or
perceived bias

• Editors may decide that disclosure is not
  warranted or they may decide that readers should
  know about such a conflict of interest so that
  they can make up their own minds about how
  important it is.
• Decisions about whether or not to publish such
  information should be made jointly by reviewers
  and editors.

67
What Should a C2 Protocol References Section
Contain?

• The protocol should include complete references
  of all cited works.
• References and other stylistic considerations
  should follow the guidelines of the American
  Psychological Association
• American Psychological Association. (1994).
  Publication Manual of the American Psychological
  Association (4th Ed.). Washington, DC Author

68
What Should a C2 Protocol Tables and Figures
Contain?

• Any tables and figures included in the protocol
  should appear after the text.

69
How is a C2 Protocol Registered?

• Once a proposed protocol has been completed it
  should be sent to the appropriate Collaborative
  Review Group editors.

70
How is a C2 Protocol Registered?

• When the editors are satisfied that the protocol
  meets the standards of the Campbell Collaboration
  they will include it in the Review Groups module
  for publication in the Campbell Database of
  Systematic Reviews.
• Publishing protocols may encourage interested
  parties to contact the reviewers and may
  discourage others from undertaking a review on
  the same topic

71
Editors and reviewers should not include a
protocol in a module unless there is a firm
commitment

• To complete the review within a reasonable
  timeframe
• To keep the review up-to-date once it is completed

72
What Happens if the Approved C2 Protocol
Undergoes Changes?

• While every effort should be made to adhere to a
  predetermined protocol, it is recognized that
  this is not always possible or appropriate.
• Changes in the protocol should not be made on the
  basis of how they affect the results of the
  review.
• As a rule, when possible analyses should be
  performed to show the effect of the change on the
  results of the review.

73
How should a C2 Protocol be cited?

• When the protocol is converted into a full
  review, the fact that this review was preceded by
  a published protocol should be noted.

74
How should a C2 Protocol be cited?

• It is Campbell Collaboration policy that
  protocols that have not been converted into full
  reviews within
• TWO YEARS
• will be withdrawn from the Campbell Database of
  Systematic Reviews.

75
What Other Sources of Assistance Can Help Develop
a C2 Protocol?

• In addition to the guidelines presented above,
  prospective reviewers will find sound advice for
  proposing and conducting Campbell Reviews in the
  following works
• Clark M. Oxman A. D. (Eds.).(2000). Cochrane
  reviewers handbook Version 4.1. In Review
  Manager (RevMan) Computer program. Version 4.1
  Oxford, England The Cochrane Center. (Available
  on-line at http//www.cochrane.org/cochrane/hbook
  .htm ).

76
What Other Sources of Assistance Can Help Develop
a C2 Protocol?

• In addition to the guidelines presented above,
  prospective reviewers will find sound advice for
  proposing and conducting Campbell Reviews in the
  following works
• Cooper, H. Hedges, L. V. (Eds.). (1994). The
  handbook of research synthesis. New York Russell
  Sage Foundation.

77
What Other Sources of Assistance Can Help Develop
a C2 Protocol?

• In addition to the guidelines presented above,
  prospective reviewers will find sound advice for
  proposing and conducting Campbell Reviews in the
  following works
• National Health Service Centre for Reviews and
  Dissemination. (2000). Undertaking systematic
  reviews of research on effectiveness. York,
  England University of York. (Available at
  http//www.york.ac.uk/inst/crd/report4.htm ).

78
What Other Sources of Assistance Can Help Develop
a C2 Protocol?

• Reviewers may contact the Campbell Secretariat
  for further guidance in preparing Campbell Review
  protocols.