Issues and challenges around integrating qualitative and quantitative research in systematic reviews

1
Issues and challenges around integrating
qualitative and quantitative research in
systematic reviews

• Angela Harden, Professor of Community and Family
  Health
• InCQuiRES, 4th September, 2013

2
INSTITUTE FOR HEALTH AND HUMAN DEVELOPMENT

• A centre for research, innovation and community
  engagement to promote health and wellbeing.
• Staff from a range of health and social science
  disciplines.
• Includes collaborative programme of research with
  Barts Health NHS Trust focused on women, children
  and young people

3
Overview of recent SR projects
Title Funder Main collaborating institutions
The effects of schools and school-environment interventions on health NIHR LSHTM, UEL, IOE
Can specific approaches to community engagement help to reduce inequalities in health? NIHR IOE, UEL, LSE
A meta-narrative review of conceptualisations and meanings of community within and across research traditions AHRC UEL
Facilitating access to antenatal care for women in India, Pakistan and Bangladesh EUFP7 UEL, QMUL other partners
4
Acknowledgements

• Colleagues at the EPPI-Centre, Social Science
  Research Unit, Institute of Education
• Co-convenors of the Cochrane Qualitative Research
  Methods Group

5
Outline

• Drivers and challenges of integration
• Review of approaches and methods
• Recent developments
• Future challenges and issues

6
Drivers for integration

• Greater recognition of the value of qualitative
  research in evidence-based policy
• Empty reviews
• Public perspectives and experiences
• Systematic reviews of complex interventions
• Issues of process and implementation
• Extension of evidence-based health care to other
  areas of public policy
• Growing tradition for integration in primary
  research (mixed methods research)
• Dedicated research funding for methodological
  research
• Establishment of dedicated methods groups (e.g.
  Cochrane Qualitative Methods Research Group)

7
(No Transcript)
8
Differences in Male Life Expectancy within a
small area in London
Travelling east from Westminster, every two tube
stops represent over one year of life
expectancy lost Data revised to 2004-08
Male Life Expectancy 73.6 (CI 71.9-75.2)
Canning Town
Male Life Expectancy 78.5 (CI 75.5-81.6)
Westminster
Canary Wharf
London Bridge
River Thames
Canada Water
North Greenwich
Bermondsey
Waterloo
Southwark
Electoral wards just a few miles apart
geographically have life expectancy spans varying
by years. For instance, there are eight stops
between Westminster and Canning Town on the
Jubilee Line so as one travels east, every two
stops, on average, mark over a year of shortened
lifespan. 1
London Underground
Jubilee Line
1 Source Analysis by London Health Observatory
of ONS and GLA data for 2004-08. Diagram produced
by Department of Health
9
Mega-events and urban regeneration

• Examples of complex interventions with multiple
  anticipated outcomes
• Regeneration - a set of activities to reverse
  economic, social and physical decline
• Housing stock, service improvements, physical
  environment, creation of socially mixed
  population

10
McCartney et al. BMJ 2010 340c2369
11
McCarthy et al. (2010) conclude.

• There is little evidence that major multi-sport
  events held between 1978 and 2008 delivered
  health or socio-economic benefits for the
  population of the host country..
• Future events such as the 2012 Olympic Games and
  Paralympic Games, or the 2014 Commonwealth Games,
  cannot be expected to automatically provide
  benefits. Until decision makers include robust,
  long term evaluations as part of their design and
  implementation of events, it is unclear how the
  costs of major multi-sport events can be
  justified in terms of benefits to the host
  population.

12
Unanswered questions

• What happens during implementation?
• How do the various actors involved experience
  mega-sporting events and the associated
  regeneration?
• What are the pathways to impact?
• Do variations in the above relate to outcomes?

13
Challenges of integration

• New and evolving field
• Scale of task
• Paradigm wars
• Lack of training and expertise
• Few worked examples
• Infrastructure in development

14
Approaches and methods in 2004

• Narrative summary
• Thematic analysis
• Grounded theory
• Meta-ethnography
• Meta-study
• Miles and Huberman's data analysis techniques
• Content analysis
• Case survey
• Qualitative comparative analysis
• Bayesian meta-analysis

Qualitising
2004
Quantitising
15
Approaches and methods 2013
16
Origins of approaches and methods
Method Developed by/Exemplars Context and purpose
Narrative synthesis Popay et al. (2005) Cochrane remit to examine issues of process, implementation and experience
Mixed methods synthesis Thomas, Harden et al. (2004) Harden and Thomas (2005) Informing policy to promote childrens health
Bayesian synthesis Roberts et al. (2002) Factors that affect the uptake of childhood immunisation
Critical interpretive synthesis Dixon-Woods et al. (2006) Access to healthcare for vulnerable groups
Meta-narrative Greenhalgh et al. (2005) To review research on diffusion of innovation to inform healthcare policy
Realist synthesis Pawson (2006) To develop and test theories of change underpinning complex policy interventions
17
In focus Mixed methods synthesis

• Recent examples
• Teenage pregnancy and social disadvantage (Harden
  et al., 2009)
• Knowledge and information needs of young people
  with epilepsy (Lewis et al., 2010)
• Behavioural interventions for weight management
  in pregnancy (Campbell et al., 2011)
• The effects of schools and school environment
  interventions (Bonell et al., 2013)

• Aim is to generate and test theory from diverse
  body of literature
• Exhaustive search, review questions, inclusion
  criteria and quality assessment largely specified
  a priori
• Each review typically has three syntheses
• Statistical meta-analysis
• Thematic synthesis
• Cross-study synthesis

18
Mixed methods systematic reviews

• Working definition
• Combining the findings of qualitative and
  quantitative studies within a single systematic
  review, in order to address the same, overlapping
  or complementary review questions
• (Harden and Thomas, 2010)

19
Three ways in which reviews are mixed.

• The types of studies included and hence the type
  of findings to be synthesised (i.e. qualitative/
  textual and quantitative/numerical)
• The types of synthesis method used (e.g.
  statistical meta-analysis and qualitative
  synthesis)
• The mode of analysis theory testing AND theory
  building

20
(No Transcript)
21
Review questions

• What is known about the barriers to, and
  facilitators of, healthy eating amongst children?
• Do interventions promote healthy eating amongst
  children?
• What are childrens perspectives on healthy
  eating?
• What are the implications of the above for
  intervention development?

22
REVIEW PROCESS
Searching, screening and mapping
Focus narrowed to fruit veg
Synthesis 2 Qualitative studies (n8) 1. Quality
assessment 2. Data extraction 3. Thematic
synthesis
Synthesis 1 Trials (n33) 1. Quality
assessment 2. Data extraction 3. Statistical
meta-analysis
Synthesis 3 Trials and qualitative studies
23
Synthesis 3 Across studies
Childrens views Trials Trials
Recommendation for interventions Good quality Other
Do not promote fruit and vegetables in the same way 0 0
Brand fruit and vegetables as an exciting or child-relevant product, as well as a tasty one 5 5
Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health 5 6
24
Synthesis 3 Across studies
Increase (standardised portions per day) in
vegetable intake across trials
Little or no emphasis on health messages
25
Recent development

• Candy et al. (2013) Using qualitative evidence on
  patients views to help understand variation in
  effectiveness of complex interventions
• Used qualitative comparative analysis to identify
  pathways to effectiveness
• Worked example - improving adherence to drug
  therapy

26
Recent developments
27
In focus Bayesian synthesis

• Recent examples
• Factors affecting the uptake of childhood
  immunisation (Roberts et al., 2002)
• Factors that influence adherence to HIV
  medication regimes
• (Vollis et al., 2009)
• (Crandell et al., 2011)

• Aim is to test theory
• Findings from qualitative and quantitative
  research are fused
• Only three worked examples to date
• Variation in weight given to the qualitative
  evidence
• Synthesis product is a set of weighted factors
  associated with/predicting the phenomenon under
  review

28
Factors that influence adherence to HIV
medication (Crandell et al., 2011)
29
In focus Critical Interpretive Synthesis

• Examples
• Access to health care for vulnerable groups
  (Dixon-Woods et al. 2006)
• Use of morphine to treat cancer related pain
  (Flemming, 2009)
• Nurses response to suicide and suicidal patients
  (Talseth and Gilje, 2011)

• Aim is to generate theory from large and diverse
  body of literature
• Literature itself is an object of scrutiny
  (critical)
• Comprehensive search to identify sampling frame
• Purposive and theoretical sampling
• Analysis leads to generation of synthetic
  constructs and a synthesising argument

30
Access to health care for vulnerable groups
(Dixon-Woods et al. 2006)

• Based on 119 papers
• Interpretive qualitative analysis of diverse
  types of studies
• Conceptual and methodological problems with
  measures of health service utilisation
• Synthesising argument organised around a set of
  central concepts (e.g. navigation, adjudications)
  with the synthetic construct of candidacy at
  the core

31
Access to health care for vulnerable groups
(Dixon-Woods et al. 2006)

• Core construct candidacy
• Candidacy describes the ways in which people's
  eligibility for medical attention and
  intervention is jointly negotiated between
  individuals and health services.candidacy is a
  dynamic and contingent process, constantly being
  defined and redefined through interactions
  between individuals and professionals, including
  how "cases" are constructed.

• Central concepts
• Identification of candidacy
• Navigation
• Permeability of services
• Appearances at health services
• Adjudications
• Offers and resistance
• Operating conditions and local production of
  candidacy

32
In focus Meta-narrative review

• Recent examples
• Spread and sustainability of innovations in
  health service delivery and organisation
  (Greenhalgh et al., 2005)
• Understanding the use of electronic patient
  records in health care organisations (Greenhalgh
  et al., 2009)
• See also the
• Realist and meta-narrative evidence synthesis
  evolving standards project (RAMASES) (Greenhalgh
  et al., 2011 Wong et al., 2013)

• Aim is to make sense of and understand diverse
  bodies of literature and their findings
• Literature itself is an object of scrutiny
  (critical)
• Searching is iterative, snowballing a key
  technique
• Analysis leads to production of a set of
  meta-narratives (storylines of research)

33
A meta-narrative approach

• The influence of Kuhns paradigms (1962) and
  the makings of the first meta-narrative
• The essential technique is interpretive synthesis
  exploring distinct research traditions, each with
  its own meta-narrative
• Methods of unpacking the meta-narrative
  exploratory methods expert consultations
  snowballing database searching

34
Stages of a meta-narrative review (from
Greenhalgh et al., 2009)
35
Meta-narratives identified in the electronic
patient records review (from Greenhalgh et al.,
2009)
36
Comparing approaches and methods
Method Idealist realist continuum Deconstruct body of literature? Mixed methods lens Other characteristics
Mixed methods synthesis Realist No Complementary strengths stance and Dialectical stance Little iteration in methods Synthetic product aims to directly address policy
Bayesian synthesis Realist No Alternative or single paradigm stance Little iteration in methods Synthetic product aims to directly address policy
Critical interpretive synthesis Idealist Yes Alternative or single paradigm stance Iterative approach key Synthetic product requires interpretation
Meta-narrative Idealist Yes Dialectical stance Iterative approach key Synthetic product requires interpretation
37
Future challenges

• More worked examples are key
• Focus on methods and tools for the actual
  integration
• Enhancing transparency
• Establishing rigour
• Further conceptual work to illuminate points of
  difference, strengths and weaknesses, fit for
  purpose (e.g. aggregation and configuration
  Sandelowski et al., 2011 Gough et al., 2012)
• Learning from, and contributing to, the mixed
  methods literature for primary research.
• Critical appraisal and reporting standards

38

• Thank you!
• a.harden_at_uel.ac.uk