Better Improvement Research Resources download from: http:homepage'mac'comjohnovrFileSharing2'html

1
Better Improvement ResearchResources download
fromhttp//homepage.mac.com/johnovr/FileSharing2
.html

• John Øvretveit,
• Director of Research, Professor, Karolinska
  Medical Management Centre Sweden and Professor of
  Health Management, Faculty of Medicine, Bergen
  University

2
Recognition of AHRQ researchersYou are making
a difference

• Just some achievements
• Shojania ed 2001 700 page review of safety
  interventions
• Quality and safety indicators
• Culture survey
• Team STEPS other tools
• Innovations exchange

3
Achievements

• Notable research funded by AHRQ
• Closing the quality gap series http//www.ahrq.gov
  /clinic/epc/qgapfact.htm
• Henriksen K, Battles JB, Marks ES, Lewin DI,
  editors. Advances in patient safety from
  research to implementation. Vol. 1, Vol2. Vol 3
  implement Vol4 AHRQ Publication No. 05-0021-1.
  Rockville, MD Agency for Healthcare Research and
  Quality Feb. 2005.
• http//www.ncbi.nlm.nih.gov/books/bv.fcgi?ridaps.
  part.1
• Partnerships in Implementing Patient Safety
  (PIPS) grants
• REAIM studies (eg Magid et al 2008)

4
Acknowledge also

• QUERY series , Mittman et al, eg Yano 2008

5
Achievements

• Questions are the answer

6
Shown excellence, but now challenges

• 1 Is it effective? (for many types of QSI)
• 2 Why? causal model
• 3 Who cares anyway? - More useful to research
  users
• 3a How to implement it?
• 3b Researcher-user interaction use knowledge
  translation res/K to shape question and enable
  users to use
• Exciting opportunity for research innovation
• But silos

7
My subject interventions to providers/organisatio
ns, not patientsevaluating non-standardisable
complex interventions and implementation
strategies

• Not Treatments BBs after AMI (beta-blockers
  after myo cardial infarc tion)
• But
• Intervention to get BBs given appropriately (eg
  Education, guidelines, CDS, audit)
• Intervention to spread CDM eg Breakthrough
  collaborative
• RRT (or CRM)
• Development programme to lead improvement
• P4P for QS
• Accreditation benefits for costs compared to
  alt?

8
Distinguish
9
Themes

• Horses for courses
• Match method to question and type of QSI
• More flexibility and innovation
• Its not the camera, but whats behind and in
  front that makes a quality picture
• Its not the intervention, but the context and the
  beneficiaries that makes the impact

10
More complex more dependent on context for
implementation

• How

Evaluation Method gt How context dependent is
the intervention? More complex more dependent
on context for implementation
11
Next 4 challenges and resolutions

• Useful research
• Efficacy
• Effectiveness/generalisation
• Translation
• Examples RRT CRM Transition interventions
  Accreditation.

12
Summary
13
1 challenge decision makers information needs

• Go/not go decision pilot, full-scale?
• Implementers guidance adapt and progress it?
• Install update?
• Needs useful credible information, now!, about
• Costs, savings, benefits, risks for our
  organisation
• Implementation to maximise success
• Dont even think about it unless.
• Utility not purity Good enough validity some
  attention to bias
• Researcher response?
• No compromise publication and promotion

14
1 challenge decision makers information needs

• "Many QIs have small to moderate effect"
• Research design limitations?
• Does quantitative RCT/CT design
• a) fail to measure enough intermediate or
  ultimate outcomes?
• b) obscure extremes, where context important?
• c) require prescribed implementation, when
  iterative adaption necessary?

15
1 challenge decision makers information needs

• Resolution by decision-makers
• Hierarchy of evidence
• 1)Face validity/make sense? - Try it on a small
  scale
• 2)Steve or Janes experience in Kansas
• 3)IHI practitioner reports O1 gt I gt O2 data
  (BeforegtInterventiongtAfter)
• 4)Published practitioner-scientist study
• 5)High-church medical journal publication
• Proportionality of proof cost/ease, risk,
  benefit

16
2 Challenge Efficacy proof

• Does it work anywhere?
• Maximise certainty of attribution of outcomes to
  intervention
• Causal assumptions why/how does it work?
• Resolutions
• Paradigm O1 gt I gt O2 quantitative
  experimental black box
• Is there are difference?
• Better
• O1 gt I gt O2 Bigger difference?
• O1 gt ? gt O2
• Other explanations for difference?
• Control, randomise, compare, hygiene to avoid
  contamination by confounders

17
Disconnect between

• A Linear sequential intervention outcome
  assumptions underlying research designs and
  explanation and
• B Sophisticated systems understanding of causes
• Outcomes the result of a number of causes
• Causes interact with each other and with
  influences outside the boundary of the system
• Eg Senge Architypes (latent predisposing
  factors/active cause) ref Anderson et al 2005

18
2 Resolutions to increase proof of Efficacy

• Strengths
• v specifiable, controllable interventions like
  drug
• v Unchanging, control known confounders and
  randomise-out others, 2/3 measures all you need
• Limitations
• Absence of above. Works for whom? - Multiple
  perspectives. Unintended consquences study more
  outcomes
• Decision makers translation info they need in
  addition

19
2 Resolutions to increase proof of Efficacy

• Strengthening
• Parallel process evaluation
• Reporting ("SQUIRE" etc)
• (labels for what implemented, not the brand)
• Attribution steriods for observational studies
• (sensitivity analyses to assess results
  Propensity score (Johnson et al 2006) and
  instrumental variable (Harless and Mark 2006)
  methods

20
3 Challenge effectiveness research for
generalisation

• Effectiveness in different situations?
• Issues
• Many interventions sensitive to context
• Implementable only if changed to suit context
• Evolve in interaction with changing context -
  journey/story
• Ie
• efficacy guarantee violated by user adaption of
  some interventions
• For others guarantee failure if you do not adapt
• Or buy installation and 3 year guarantee

21
3 Resolutions generalisable effectiveness
research

• R1 Maintain paradigm Pragmatic trials
• Minimise loss of attribution with Time series,
  Step-wise wedge, SPC (but increase cost and time)
• Some v for routine practice feedback
• Generalisable to similar situations and
  interventions
• Add more situations and variations of the
  intervention
• Compare many pragmatic trials and assess what
  works best where
• Invite trails in X situations?
• Improve reporting (standardise and details)
• - ve no answer to why?
• explanation helps adapt, and contributes to
  science
• .
• .

22
3 Resolutions context sensitive generalisable
effectiveness research

• R2 Case study research
• v Describes intervention as it evolves context
  helpers and hinderers
• v Assesses intermediate changes
• v Links these to ultimate patient/cost outcomes,
  if possible
• Multiple case study in selected situations (eg
  Dopson 2002)
• NEXT What we have learned in doing this research

23
What we have learned in doing this research

• The research
• 12 Action evaluation case studies of innovation
  implementation in Swedish health care
• variety of research into practice
  implementation and change studies

24
L2 Distinguish

• Safer clinical practices
• Changed providers behaviour reduce adverse
  events?
• Safer organisation and processes
• support changes in provider behaviour and address
  latent causes
• Implementation actions to achieve the above
• at team, organisation, system and national
  levels
• External context helpers and hinders
• (is a MET/RRT a safe clinical practice or a
  "safer organisation or process" change, or both?)

The seed
Planting
Soil climate
25
Sodertalje innovation established 1996-1999
Context factors help and hinder implementation at
different times
Government policy helps planning


Planning Establishment Further actions
and types of coordination created
Combined client/patient care planning system
Development of systems procedures and sub
structuresgtgtgtgtgtgtgtgtgtgtgtgt
One management group with representatives from
the county council and municipal care
Result Innovation content three shared
rehabilitation units with common decision making
forum
Actions planning and pre- implementation
Results Consequences for personnel 1996 2000 20
04 2008
Results changes to organisation 1996 2000 2004
2008
Results Consequences for patients/clients 1998 2
002 2004 2008
1994 1995 1996 1997 1998 1999 2000 2001
2002 2003 2004 2005 2006 2007 2008 2009
26
L3 Theory essential - of intervention pathway to
outcomes

• To decide which data to gather
• To provide explanations to test
• To give implementers to help them adapt.
• (Program theory Weiss 1972, 1997 RogFournier
  1997 Logic Model Wholey 1979 Theory-driven
  evaluation Chen 1990, Sidani Braden 1998
  realist evaluation Henry et al 1998, Pawson
  Tilley 1997 Theories Grol et al 2007)

27
L4 Action evaluation learning cycle

• Feedback findings during implementation
• and for science
• Assess effect of researcher on implementation and
  results
• Helps develop intervention during the
  implementation journey
• Increases cooperation and access to data
• Partnership, but distinct roles
• Study how implementers use knowledge and help use
  more

28
4 Challenge use faster, wider

• Demand? - Real men dont need research
• Supply? - Real researchers dont write exec
  summaries
• Make sure unusable and throw over the fence
  delivery
• Closing the research/practice gap

29
Translation in QSI HSR

• EvidencegtTestgtPackage
  UsergtAdaptgtImplement/Adjust
• Development Translation 1 Implementation
  Translation 2
• (intervention development and testing)
  (adoption/ spread)
• What is the intervention?
• Where do you draw the boundary?

30
4 Resolutions our experience

• Use KT/KM literature what works?
• Content accessibility and relevance
• Service implications many examples 320Appx
  reports ghost writers and mediator authors
• Engage emotionally patient describes experience
  or video
• Process interact with users at each stage
• Structure forums, networks, joint appointments,
  brokers

31
Summary
32
Questions

• Efficacy and causality
• System thinking in research - causality
  explanations and data gathering
• Always trade off between internal/external
  validity?
• Generalisable effectiveness research
• Journey/story approach unique?
• Use faster, quicker
• Extend researcher role?
• Increase demand?
• Effect of action role?