The dependent variable in knowledge translation research: What should we measure How should we measu

1
The dependent variable in knowledge translation
research What should we measure? How should we
measure it?
Carole Estabrooks PhD, RN Janet Squires PhD
Student, RN Lars Wallin PhD, RN Petter Gustavsson
PhD
KU07 Conference Stockholm. Sweden August 16th,
2007
2
Outline

• The need for a measure
• Measures to date
• The Canada/Sweden measurement study
• How should we go about measuring RU?
• Introduction to group discussions
• What does the theory tell us to measure?

3
The need for a measure
CE
4
Why measure research use instead of patient
outcomes when assessing KT interventions?

• Not instead in addition
• However, if we do not measure research use, it is
  difficult to claim that the intervention tested
  to increase research use was effective
• And, if we only measure patient outcomes.it is
  difficult to determine if an improvement in such
  outcomes is attributable to the intervention to
  increase research use
• Finally, working in organizations, it is we argue
  useful to develop and use measures of KT that
  reflect use at unit (possibly team) and facility
  levels, in addition to individual levels

5
What is Research Utilization?

• Types of Research Utilization
• Instrumental
• Conceptual
• Symbolic/Persuasive

6
Measures to date

• Nurses Allied Health Professionals

JS
7
Nurses and Allied Health Professionals

• Review of nursing and allied literature
• Estabrooks, Wallin, Milner. (2003). ).
  Measuring knowledge utilization in health care.
  International Journal of Policy Evaluation
  Management, 1(1), 3-36.
• Purpose To assess the state of measurement
  science specific to the concept of research
  utilization

8
In that review the group assumed some
requirements of robust measurement instruments

• Substantive (content) theory
• Measurement theory
• Minimal self-report social desirability effects
• Unequivocal scale
• Acceptability within the research field

9
The review

• Search Strategy
• Published reports of research use
• CINAHL and Medline electronic databases
• Search Results
• 42 publications
• 3 commonly used multi-item measures
• Multiple single item measures

JS
10
Findings

• Existing Measures
• Common Multi-item Measures
• Nursing Practice Questionnaire (N3)
• Nursing Practice Questionnaire (Modified) (N6)
• Research Utilization Questionnaire (N7)
• Edmonton Research Orientation Survey (N3)
• Other multi-item measures (N3)
• Single item measures (N20)

11
Multi-Item measures

• NPQ (Brett 1987)
• Based of Rogers innovation-decision process
  theory, reliability limited to chronbach alpha,
  content validity assumed
• RUQ (Champion Leach 1989)
• reliability limited to chronbach alpha, limited
  validity assessments, operational defn of use
  unclear
• EROS (Pain, Hagler, Warren 1996)
• reliability limited to chronbach alpha, construct
  validity assessment, operational defn of use
  unclear, recall bias

12
Single Item Measures

• 20 studies used single-item measures
• Similar limitations to multi-item but also
• Different terminology (makes it difficult to
  compare studies)
• Array of different scoring methods and scaling
  assumptions
• Recall bias

13
Measures to date

• Medicine

LW
14
Guideline literature

• Based on Grimshaw et al (2004). Effectiveness and
  efficiency of guideline dissemination and
  implementation strategies. Health Technology
  Assessment, 8, 1-72.
• 235 studies were reviewed
• Data extracted on two main measures
• Process of care results
• Outcome of care results
• Not described how data was collected, exemplified
  with charts and self-reports
• 225/235 studies used process of care measures
• 50/235 studies used outcome of care measures

15
Guideline literature

• Examples of measures
• Process of care
• of patients blood pressure measured
  (hypertension screen)
• of patients smoking status recorded
• of patients documented nutritionist consult
• Rating of time spent counseling (physician
  questionnaire)
• Outcome of care
• Systolic blood pressure
• of patients stopped smoking at 12 month
• of patients satisfied with care
• Sickness Impact Scores at 6 month
• of patients surviving at 1-year

16
General Problems Across Measures
Regardless of the measure.

• Measuring the construct of research use
• Poor construct clarity
• Lack of theoretical framing
• Presumption of linearity
• Common assumptions held about research use
• Methodological
• Lack of measurement theory underpinning
• Lack of psychometric assessment
• Unit of analysis - ecological fallacies
• Retrospective survey design dependent on
  self-report and recall
• Operationalization of use unclear
• Absence of longitudinal work
• Scaling

JS
17
Update of the 2003 measurement review repairing
prior sins

• Expanded Search
• Databases HAPI, CINAHL, MEDLINE, EMBASE,
  SCOPUS, OCLC Papers First, OCLC WorldCat, ABI
  Inform, Web of Science, Sociological Abstracts
• Hits 23,065
• Abstracts Screened 11,810
• Articles pulled n105
• Need more information n497
• Hand search
• Cross references

18

• Some notes on
• Experiences with a single item measure

CE
19
Research Utilization Studies (1999-2003)
Range 1-7
20
Alberta Nurse Survey (2002)
One-Way ANOVA P-value lt0.001
21
CIVMIL Study (1999, 2005)
Gamma (ordinal test) statistic P-value lt0.05
22
AKUTE Study
One-Way ANOVA P-value lt0.05
23
Research use in nursing practice in relation to
educational, individual and work contextual
factors(Sweden)

• Data collected within the LANE project
  (Longitudinal analysis of Nursing Education)
• Nationwide survey, prospective longitudinal
  design, 2002-2008
• Two cohorts of nursing students
• Instrumental, conceptual and symbolic research
  use at 1, 2, 3 and 5 years after graduation
• Aims To identify predictors and to study the
  variability over time of nurses research use

LW
24
(No Transcript)
25

• The current measurement study
• Canada/Sweden

CE
26
Developing a Valid and Reliable Measure of
Research Utilization(Ref Belfast KU04)
Purpose To develop and assess an instrument that
measures research utilization among health
professionals
The Canada ARM CIHR (2005-2007) Investigators
Estabrooks, Wallin, Scott-Findlay,
Profetto-McGrath, Harley
The Swedish ARM Karolinska Institute
supported Investigators Wallin, Strandberg,
Kajermo-Nilsson, Gustavsson
27
Measurement Study

• Planned outcomes
• A set of definitions for research utilization
• A finalized version of the research utilization
  concept model previously developed by the team
  and expert panel
• A preliminary set of indicators of research
  utilization from which to develop items for the
  instrument
• An instrument

28
The Approach

• Construct clarity ? Series of focus groups with
  three sets of experts
• Local (research team) expert panel
• International panel
• Managers and educators
• Providers (RNs/RPNs, LPNs/aides)
• Indicator and item development ? Series of focus
  groups with two sets of experts
• Managers and educators
• Providers of nursing care (RNs/RPNs, LPNs/Aides)
• Develop and pilot test the draft instrument(s)

29
Measurement Study

• International Panel
• Jo Rycroft-Malone (UK)
• Carl Thompson (UK)
• Marita Titler (US)
• Anne Sales (US)
• Judith Ritchie (Canada)
• Jo Logan (Canada)
• Nancy Edwards (Canada)
• Donna Ciliska, (Canada)

30
Action in Practice
Concept Map (Study Team)
Action
Thinking (i.e., Clinical Decision
Making Clinical Reasoning )
Instrumental Use
Task completion
Research Use
Instrumental Use
Task completion Relational Assessment Screening
Research Non-Use
Non-use
A. Informed (non-defensible)
Non-use
B. Informed (defensible)
Non-use
Uninformed
31
Research utilization
HI
Conceptual
Instrumental
HI
LO
LO
32

• Findings Canadian ARM

JS
33
Focus Groups Round 1 Themes Identified
34
Terminology

• Majority related to instrumental research use
• No delineation between low and high conceptual
  use
• Research Utilization not commonly used
• Terms commonly used included
• Evidence-based medicine (practice)
• Research-based knowledge
• Based on research
• Based on clinical evidence

35
Definitions

• Research Use
• Putting something into practice
• A personal choice or judgement
• Conceptual Research Use
• Thinking, critical thinking
• Instrumental Research Use
• Action, implementing, observable

36
Examples

• Examples increased in concreteness from
  manager/educator to RN/RPN to LPN/aide
• Most examples r/t instrumental research use
• Gradient from conceptual to instrumental
• Examples common to groups
• Talking/discussing
• Following policies and procedures/protocols
• Specific conditions
• e.g. wound care, Braden scale

37
Indicators

• Following research-based policies and
  procedures/protocols
• Asking questions
• Peer-to-peer interactions
• Reading research literature
• Bringing in research articles
• Self report
• Observation
• Chart documents

38
Focus Groups Round 2

• Validation for IRU schema developed based on
  round 1 analysis
• Self-report
• Questionnaire
• Interview (healthcare provider, manager)
• Observation
• Chart audit

39

• Findings Swedish ARM

LW
40
Terminology

• Research Utilization not a common term (but might
  work)
• More accepted than evidence-based practice -
  trendy concept

41
Definitions

• Research Use
• Basing nursing on scientific findings
• Working according to appraised and compiled
  information
• Adopting a critical approach to ones work
• Conceptual Research Use
• learning through critical reflection/thinking
• Instrumental Research Use
• Active process

42
Examples

• Most examples r/t instrumental research use
• Adherence to research-based routines/guidelines
• Using systematic assessment instruments
• Quality management work
• All groups mentioned application of concrete
  findings r/t procedures and treatments
• Insulin Injection technique
• S/C injection of sterile water for labour pain

43
Indicators

• Felt instrumental and conceptual research use
  were observable
• Following research-based routine/guidelines
• Updating routine/guidelines
• Observation
• of specific practices, using systematic
  assessment instruments, a structured nursing
  working method
• Chart documents
• Using internet or intranet to seek information
• Providing rationale for ones actions
• Peer-to-peer interactions

44
Similarities between Canadian and Swedish
findings

• Instrumental
• Adherence to research-based policies/protocols
• Easier to talk about specific practices
• Examples increased in concreteness from
  manager/educator to RN to LPN/Aide
• Conceptual
• A progression of conceptual to instrumental
• Equated with thinking/critical thinking

LW
45
Learnings Directions

• Participants not at all accustomed to discussing
  research utilization
• Consequently, focus groups not an optimal method
• Much confusion about conducting research vs. use
  of research
• Can we distinguish conceptual RU from
  instrumental RU? RU often viewed as a process
  with similarities to the process of EBP. Is
  conceptual use necessarily a precursor to
  implementation? What about implementation without
  understanding?

46
Learnings Directions

• Critical reflection viewed as an important
  attribute to RU. At the same time participants
  claimed that much of RU was unreflected.
• Enhanced awareness of the knowledge base for
  clinical practice - a strategy for RU? How to
  measure RU if practitioners are not aware of if
  they are using research?
• Adaptation to patients perceptions and
  preferences important and contributes to make the
  concept of RU vague. Consequences for measuring
  RU?

47
Next Steps

• Systematic review (update) of instruments to
  measure research use (in-progress)
• Manuscript on Canada/Sweden findings re
  instrumental use (in-progress)
• Concept analysis (literature based) on conceptual
  use (in-progress, Canada)
• Submit grant applications 2008
• Develop and pilot instrumental measures
• Construct clarity re conceptual use (concept
  analysis using ethnoscience)
• Eventually development of a measure of CRU

CE
48
Psychometric theory
PG
49
Sources
50
Manifest Observed responses
Latent, Not manifest
51
To a great extent
To a moderate extent
To a little extent
To no extent
52
i1 i2 i3 i4
53
Strength
i1 i2
54
Location
55

• an instrument is valid for measuring an attribute
  if and only if
• (1) the attribute exists and
• (2) variations in the attribute causally produce
  variations in the outcomes of the measurement
  procedure

56
Classical Test Theory Strength
i1 i2 i3 i4
57
Item Response Theory Location
58
Respondents Items
Direction of increasing X
Respondents with high X
Item indicating highest level of X
Item indicating higher level of X
Respondents with midrange X
Item indicating lower level of X
Item indicating lowest level of X
Respondents with low X
Direction of decreasing X
59
The construct

• The research-practice gap
• Barriers to research use
• Inability to obtain research findings in the area
  of interest
• Understanding, Resistance, Relevance, Agreement
  with conclusions
• Time constraints
• Costs
• Rewards

60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
10 to 60
65
Unidimensional?

• Measures of model fit
• Absolut fit
• SRMR0.09 (0.07)
• Parismony fit
• RMSEA0.08 (0.05)
• Comparative fit
• CFI0.93 (0.95)
• not unidimensional!

66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
Conclusions

• Location
• Process
• building an hierarchy

• Strength
• Variance
• Repeated measurement

70
Respondents Items
Construct Map
Direction of increasing X
Respondents with high X
Item indicating highest level of X
Item indicating higher level of X
Respondents with midrange X
Item indicating lower level of X
Item indicating lowest level of X
Respondents with low X
Direction of decreasing X
71

• Relevant Theories for
• Research Utilization Studies

72
Theory of Planned Behaviour
Attitudes
Subjective Norms
Behavioural Intention
Behaviour
Perceived Behavioural Control
Ajzen Madden, (1986), Journal of Experimental
Social Psychology, 22, 453
73
Promoting Action on Research Implementation in
Health Services (PARIHS)
Successful implementation is a function of
evidence, context and facilitation
Context Culture Leadership Evaluation

Context


Facilitation
Evidence
Purpose Role Skills
Research Clinical experience Patient
experience Local data
Evidence-based practice
High
Kitson et al, 1998 Rycroft-Malone et al,
2002 Rycroft-Malone et al, 2004
Low
74
Diffusion of Innovation

• Elements
• The Innovation
• Communication Channels
• Time
• Social System

• Stages
• Awareness
• Persuasion
• Decision
• Implementation
• Adoption

1931 - 2004

• Influencing factors
• Attributes of the innovation
• Individual
• Organizational

75
(No Transcript)
76
Group Work What does the theory tell us to
measure?

• Crucial concepts in each theory? What is the
  dependent variable in the theory?
• Focusing on one theory what concepts are
  important to measure? How do these concepts
  relate to research use? Are they a measure of
  research use?
• How would you measure research use according to
  the theory? What measurement approach would you
  take?
• Is any one theory adequate to guide the
  measurement of research use?