University of Oxford Continuing Education Qualitative Research and Systematic Reviews Workshop: ORGA

1
University of Oxford Continuing
EducationQualitative Research and Systematic
Reviews Workshop ORGANISING AND MANAGING
QUALITATIVE DATA IN QUALITATIVE SYNTHESIS

• Study Selection Critical Appraisal and Inclusion
  and Exclusion Criteria

2
This presentation

• Inclusion and Exclusion Criteria
• Critical appraisal and
• The Debates Surrounding Critical Appraisal and
  the Exclusion/Inclusion of Studies.

3
Evidence for health policy and practice
4
The Evidence Interests of Clinicians
5

• Clinicians need information up to 60 times per
  week (twice per every three patients), and it
  could affect eight decisions per day
• D. G. Covell, G. C. Uman and P. R. Manning.
  Information Needs in OfficePractice. Are they
  being met? Annals of Internal Medicine 1985,103,
  pp. 596-9.

6

• Self-reports from Oxford in 1997 showed that
  while medical students might spend 60 mins per
  week reading professional journals, house
  officers spent none, and even consultants spent
  as little as 30
• D. L. Sackett. Surveys of self-reported reading
  times of consultants in Oxford, Birmingham,
  Milton Keynes, Bristol, Leicester, and Glasgow.
  In W. M. C. Rosenberg et al, Evidence-based
  Medicine. Churchill Livingstone, 1997.

7
Clinicians and policy makers express a need for

• Concise summaries of the best available appraised
  evidence that addresses their evidence interests
• To inform (not direct) services/practices

8
The SR Process

• Quantitative Reviews
• Question
• Inclusion Criteria
• Search Strategy
• Critical Appraisal
• Extraction
• Synthesis

• Qualitative Reviews?
• ?
• ???
• ?
• ???
• ?
• ?

9
Inclusion Criteria

• Naturally flow from well constructed review
  questions

10

• Effects
• Population
• Intervention
• Comparison
• Outcome

• Feasibility, Appropriateness, Meaningfulness
• Setting
• Perspective
• Intervention/Phenomena of Interest
• (Comparison)
• Evaluation
• Adapated from Booth A. Using research in
  practice Australian supermodel? A practical
  example of evidence-based library and information
  practice. Health Information and Libraries
  Journal 2006 23 69-72

11
Critical Appraisal
12

• Critically appraising studies and excluding those
  judged to be of poor quality is a well accepted
  step in the systematic review process.
• Critical appraisal aims at ensuring that
  systematic reviews draw their conclusion from
  good-quality research evidence and is one of the
  characteristics of systematic reviews that
  differentiates them from literature reviews
• There is, however, disagreement amongst those who
  conduct systematic reviews of effects on the
  degree to which critical appraisal improves the
  rigour of reviews. Edwards et al (1998) for
  example, suggest that reviews may underestimate
  the evidence if they only include those of a
  certain methodological quality
• Edwards, A.G., Russell, I.T. and Stott, N.C.
  (1998) Signal versus noise in the evidence
  basefor medicine an alternative to hierarchies
  of evidence?, Family Practice. 15 4, 319-22

13
Applying critical appraisal criteria to
qualitative studies is a contentious practice!

• Dixon-Woods et al (2004) argue that Some means
  of appraising qualitative research is needed if
  it is to contribute appropriately to systematic
  reviews
• Attree and Milton 2006 argue that Critical
  appraisal must be central to research syntheses,
  thus enabling reviewers to make only good-quality
  qualitative evidence accessible for policy makers
  and practitioners
• Dixon-Woods, M., Shaw, R. L., Agarwal, S. and
  Smith, J.A. (2004a) The problem of appraising
  qualitative research, Quality and Safety in
  Health Care.133, 223-5
• Attree, P. and Milton, B. (2006) Critically
  appraising qualitative research for systematic
  reviews defusing the methodological cluster
  bombs Evidence Policy. 21, 109-26

14
Applying critical appraisal criteria to
qualitative studies is a contentious practice!

• Garratt and Hodkinson (1998) argue, however, that
  it is illogical to attempt to predetermine a
  fixed set of criteria to appraise qualitative
  research, as the meaning of research only emerges
  in the interaction between the findings and the
  critical reader.
• Others argue that important understandings
  emerging from qualitative studies judged to be of
  poor quality may contribute to theorising and
  that, therefore, quality should not be used as a
  criterion to exclude studies in qualitative
  reviews
• Garratt, D. and Hodkinson, P. (1998) Can there
  be criteria for selecting research criteria? A
  hermeneutical analysis of an inescapable
  dilemma, Qualitative Inquiry. 4 4, 515-39

15
Popay, Rogers and Williams (1998) suggest that
the development of standards for assessing
evidence from qualitative research is both
possible and desirable.Popay, J., Rogers, A.
and Williams, G. (1998) Rationale and standards
for the systematicreview of qualitative
literature in health services research,
Qualitative Health Research. 8 3, 341-51
16
Popay, Rogers and Williams (1998) provide the
following as a guide to common standards

• evidence of responsiveness to social context and
  flexibility of design
• evidence of theoretical or purposeful sampling
• evidence of adequate description
• evidence of data quality
• evidence of theoretical and conceptual adequacy
  and
• potential for assessing typicality.

17
Dixon-Woods et al (2004)suggest that

• Some means of appraising qualitative research is
  needed if it is to contribute appropriately to
  systematic reviews.
• Proposals for criteria that might define high
  quality qualitative research have proliferated,
  but sometimes do not overlap or are difficult to
  operationalise.
• A minimal set of prompts is proposed to help cue
  attention to the range of dimensions of
  qualitative research that require appraisal.
• There is a need for additional criteria that
  recognise the diversity of study designs and
  theoretical perspectives in qualitative research,
  and to distinguish between minor errors and fatal
  flaws.
• The measurement of all aspects of quality of
  qualitative research will remain difficult.

18
Dixon-Woods et al (2004)suggest the following
criteria

• Are the research questions clear?
• Are the research questions suited to qualitative
  inquiry?
• Are the following clearly described?
• sampling
• data collection
• analysis
• Are the following appropriate to the research
  question?
• sampling
• data collection
• analysis
• Are the claims made supported by sufficient
  evidence?
• Are the data, interpretations, and conclusions
  clearly integrated?
• Does the paper make a useful contribution?

19
Pearson (2004) takes the view that a transparent
approach to appraising qualitative research -
sensitive to the nature of the qualitative
research and its basis in subjectivity - is
central to its ongoing credibility,
transferability and theoretical potential.
Pearson A. (2004) Balancing the Evidence
Incorporating the Synthesis of Qualitative Data
into Systematic Reviews. JBI Reports245-64
20
Pearson (2004) suggests the following framework
for appraising qualitative research

• Congruity between the philosophical/theoretical
  position adopted in the study and the study
  methodology the study methods the
  representation of the data and the
  interpretation of the results
• The degree to which the biases of the researcher
  are made explicit and
• The relationship between what the participants
  are reported to have said/done and the
  conclusions drawn in analysis.

21
(No Transcript)
22
The debates

• The ongoing, and frequently fierce, contemporary
  debates surrounding the place critically
  appraising or assessing the quality of -
  qualitative studies can only be understood by
  situating the various viewpoints within the
  traditions they are grounded in and locating the
  points of difference between these positions
  within the broader evidence based health care
  discourses.
• Qualitative researchers who support the notion
  that qualitative research findings represent
  evidence and the development of approaches to
  systematically review qualitative findings that
  address specific phenomena of interest or
  questions have yet to reach consensus on a common
  process and there is continuing, vigorous debate
  surrounding these issues.
• As Dixon-Woods et al (2004) states qualitative
  research is not a unified field thus,
  qualitative researchers are grounded in diverse
  research traditions and the field is
  characterised by ongoing debate.

23
fit for purpose

• Much of the work to-date on qualitative research
  review is generated by scholars from the social
  sciences whose interests lie more in
  understanding and theorising the social world and
  individual experience. Given that evidence based
  practice is commonly associated with decision
  making by health and social care practitioners,
  Newman et al (2006) question the fit of
  qualitative research with the evidence-based
  practice model. They suggest that
  Conceptualising qualitative clinical questions
  and the corresponding answers or products in this
  way raises the question of whether qualitative
  research should fit within the evidence-based
  practice decision making framework at all.
• Newman, M., Thompson, C., and Roberts, A.P.
  (2006) Helping practitioners understand the
  contribution of qualitative research to
  evidence-based practice. Evidence-Based Nursing
  94-7

24

• Social scientists interests in the systematic
  review of qualitative research findings focus on
  those suggested by Booth (2001)
• 1 For the generation of models and theories
• 2 For the subsequent validation and gauging of
  the empirical support for theories
• In methodological reviews as a means of
  determining the existence of certain bodies or
  "schools" of thought.
• To chart the development of concepts
  longitudinally in the literature over a period of
  time
• To complement the findings and interpretation of
  quantitative systematic reviews, for example in
  looking at issues around patient acceptance or
  compliance.
• As a means of identifying significant domains or
  attributes to enable the development of prototype
  instruments or scales.

25

• Health and social care practitioners, policy
  makers and managers look to systematic reviews of
  qualitative evidence to provide summarised
  information on which to base decision making.
  Their interests focus on the more pragmatic
  matters suggested by Lavis et al (2005)
• Rigorous reviews that are potentially
  reproducible, though generally researchers are
  assumed to know their business
• Trustworthy, transparent methods
• Relevant, up-to-date answers to their questions
  in their context/population
• Accessible presentation of findings with clear
  messages
• Timeliness
• Information about risks (harms) as well as costs
  benefits, preferably by population sub-groups
• Some indication of uncertainty associated with
  estimates
• Lavis J, et al. Towards systematic reviews that
  inform health are management and policy making.
  JHSRP 200510(suppl 1) 35-48

26
The purpose of the process

• Clarifying the purposes of qualitative
  meta-synthesis may well facilitate consensus on
  appropriate processes. Qualitative reviewers with
  a clinical or health background and who see the
  purpose of qualitative reviews as informing
  clinical decision making do not generally oppose
  adapting the current Cochrane Collaboration
  approach in ways that sensitive to the
  qualitative approach to inquiry. There is a
  growing number of published systematic reviews
  that follow all of the steps of the Cochrane
  Collaboration approach modified to accommodate
  agreed criteria for inclusion criteria, critical
  appraisal, data extraction and data synthesis
• Such an approach is not supported by reviewers
  whose interests lie more in the development of
  theory or in illuminating a phenomena (albeit as
  a way of assisting practitioners, policy makers
  and planners to gain insights to enhance decision
  making).

27
the debate continues!!