The dependent variable in knowledge translation research: What should we measure How should we measu - PowerPoint PPT Presentation

1 / 76
About This Presentation
Title:

The dependent variable in knowledge translation research: What should we measure How should we measu

Description:

Why measure research use instead of patient outcomes when assessing ... More accepted than evidence-based practice - 'trendy' concept. Definitions. Research Use ... – PowerPoint PPT presentation

Number of Views:142
Avg rating:3.0/5.0
Slides: 77
Provided by: HCh68
Category:

less

Transcript and Presenter's Notes

Title: The dependent variable in knowledge translation research: What should we measure How should we measu


1
The dependent variable in knowledge translation
research What should we measure? How should we
measure it?
Carole Estabrooks PhD, RN Janet Squires PhD
Student, RN Lars Wallin PhD, RN Petter Gustavsson
PhD
KU07 Conference Stockholm. Sweden August 16th,
2007
2
Outline
  • The need for a measure
  • Measures to date
  • The Canada/Sweden measurement study
  • How should we go about measuring RU?
  • Introduction to group discussions
  • What does the theory tell us to measure?

3
The need for a measure
CE
4
Why measure research use instead of patient
outcomes when assessing KT interventions?
  • Not instead in addition
  • However, if we do not measure research use, it is
    difficult to claim that the intervention tested
    to increase research use was effective
  • And, if we only measure patient outcomes.it is
    difficult to determine if an improvement in such
    outcomes is attributable to the intervention to
    increase research use
  • Finally, working in organizations, it is we argue
    useful to develop and use measures of KT that
    reflect use at unit (possibly team) and facility
    levels, in addition to individual levels

5
What is Research Utilization?
  • Types of Research Utilization
  • Instrumental
  • Conceptual
  • Symbolic/Persuasive

6
Measures to date
  • Nurses Allied Health Professionals

JS
7
Nurses and Allied Health Professionals
  • Review of nursing and allied literature
  • Estabrooks, Wallin, Milner. (2003). ).
    Measuring knowledge utilization in health care.
    International Journal of Policy Evaluation
    Management, 1(1), 3-36.
  • Purpose To assess the state of measurement
    science specific to the concept of research
    utilization

8
In that review the group assumed some
requirements of robust measurement instruments
  • Substantive (content) theory
  • Measurement theory
  • Minimal self-report social desirability effects
  • Unequivocal scale
  • Acceptability within the research field

9
The review
  • Search Strategy
  • Published reports of research use
  • CINAHL and Medline electronic databases
  • Search Results
  • 42 publications
  • 3 commonly used multi-item measures
  • Multiple single item measures

JS
10
Findings
  • Existing Measures
  • Common Multi-item Measures
  • Nursing Practice Questionnaire (N3)
  • Nursing Practice Questionnaire (Modified) (N6)
  • Research Utilization Questionnaire (N7)
  • Edmonton Research Orientation Survey (N3)
  • Other multi-item measures (N3)
  • Single item measures (N20)

11
Multi-Item measures
  • NPQ (Brett 1987)
  • Based of Rogers innovation-decision process
    theory, reliability limited to chronbach alpha,
    content validity assumed
  • RUQ (Champion Leach 1989)
  • reliability limited to chronbach alpha, limited
    validity assessments, operational defn of use
    unclear
  • EROS (Pain, Hagler, Warren 1996)
  • reliability limited to chronbach alpha, construct
    validity assessment, operational defn of use
    unclear, recall bias

12
Single Item Measures
  • 20 studies used single-item measures
  • Similar limitations to multi-item but also
  • Different terminology (makes it difficult to
    compare studies)
  • Array of different scoring methods and scaling
    assumptions
  • Recall bias

13
Measures to date
  • Medicine

LW
14
Guideline literature
  • Based on Grimshaw et al (2004). Effectiveness and
    efficiency of guideline dissemination and
    implementation strategies. Health Technology
    Assessment, 8, 1-72.
  • 235 studies were reviewed
  • Data extracted on two main measures
  • Process of care results
  • Outcome of care results
  • Not described how data was collected, exemplified
    with charts and self-reports
  • 225/235 studies used process of care measures
  • 50/235 studies used outcome of care measures

15
Guideline literature
  • Examples of measures
  • Process of care
  • of patients blood pressure measured
    (hypertension screen)
  • of patients smoking status recorded
  • of patients documented nutritionist consult
  • Rating of time spent counseling (physician
    questionnaire)
  • Outcome of care
  • Systolic blood pressure
  • of patients stopped smoking at 12 month
  • of patients satisfied with care
  • Sickness Impact Scores at 6 month
  • of patients surviving at 1-year

16
General Problems Across Measures
Regardless of the measure.
  • Measuring the construct of research use
  • Poor construct clarity
  • Lack of theoretical framing
  • Presumption of linearity
  • Common assumptions held about research use
  • Methodological
  • Lack of measurement theory underpinning
  • Lack of psychometric assessment
  • Unit of analysis - ecological fallacies
  • Retrospective survey design dependent on
    self-report and recall
  • Operationalization of use unclear
  • Absence of longitudinal work
  • Scaling

JS
17
Update of the 2003 measurement review repairing
prior sins
  • Expanded Search
  • Databases HAPI, CINAHL, MEDLINE, EMBASE,
    SCOPUS, OCLC Papers First, OCLC WorldCat, ABI
    Inform, Web of Science, Sociological Abstracts
  • Hits 23,065
  • Abstracts Screened 11,810
  • Articles pulled n105
  • Need more information n497
  • Hand search
  • Cross references

18
  • Some notes on
  • Experiences with a single item measure

CE
19
Research Utilization Studies (1999-2003)
Range 1-7
20
Alberta Nurse Survey (2002)
One-Way ANOVA P-value lt0.001
21
CIVMIL Study (1999, 2005)
Gamma (ordinal test) statistic P-value lt0.05
22
AKUTE Study
One-Way ANOVA P-value lt0.05
23
Research use in nursing practice in relation to
educational, individual and work contextual
factors(Sweden)
  • Data collected within the LANE project
    (Longitudinal analysis of Nursing Education)
  • Nationwide survey, prospective longitudinal
    design, 2002-2008
  • Two cohorts of nursing students
  • Instrumental, conceptual and symbolic research
    use at 1, 2, 3 and 5 years after graduation
  • Aims To identify predictors and to study the
    variability over time of nurses research use

LW
24
(No Transcript)
25
  • The current measurement study
  • Canada/Sweden

CE
26
Developing a Valid and Reliable Measure of
Research Utilization(Ref Belfast KU04)
Purpose To develop and assess an instrument that
measures research utilization among health
professionals
The Canada ARM CIHR (2005-2007) Investigators
Estabrooks, Wallin, Scott-Findlay,
Profetto-McGrath, Harley
The Swedish ARM Karolinska Institute
supported Investigators Wallin, Strandberg,
Kajermo-Nilsson, Gustavsson
27
Measurement Study
  • Planned outcomes
  • A set of definitions for research utilization
  • A finalized version of the research utilization
    concept model previously developed by the team
    and expert panel
  • A preliminary set of indicators of research
    utilization from which to develop items for the
    instrument
  • An instrument

28
The Approach
  • Construct clarity ? Series of focus groups with
    three sets of experts
  • Local (research team) expert panel
  • International panel
  • Managers and educators
  • Providers (RNs/RPNs, LPNs/aides)
  • Indicator and item development ? Series of focus
    groups with two sets of experts
  • Managers and educators
  • Providers of nursing care (RNs/RPNs, LPNs/Aides)
  • Develop and pilot test the draft instrument(s)

29
Measurement Study
  • International Panel
  • Jo Rycroft-Malone (UK)
  • Carl Thompson (UK)
  • Marita Titler (US)
  • Anne Sales (US)
  • Judith Ritchie (Canada)
  • Jo Logan (Canada)
  • Nancy Edwards (Canada)
  • Donna Ciliska, (Canada)

30
Action in Practice
Concept Map (Study Team)
Action
Thinking (i.e., Clinical Decision
Making Clinical Reasoning )
Instrumental Use
Task completion
Research Use
Instrumental Use
Task completion Relational Assessment Screening
Research Non-Use
Non-use
A. Informed (non-defensible)
Non-use
B. Informed (defensible)
Non-use
Uninformed
31
Research utilization
HI
Conceptual
Instrumental
HI
LO
LO
32
  • Findings Canadian ARM

JS
33
Focus Groups Round 1 Themes Identified
34
Terminology
  • Majority related to instrumental research use
  • No delineation between low and high conceptual
    use
  • Research Utilization not commonly used
  • Terms commonly used included
  • Evidence-based medicine (practice)
  • Research-based knowledge
  • Based on research
  • Based on clinical evidence

35
Definitions
  • Research Use
  • Putting something into practice
  • A personal choice or judgement
  • Conceptual Research Use
  • Thinking, critical thinking
  • Instrumental Research Use
  • Action, implementing, observable

36
Examples
  • Examples increased in concreteness from
    manager/educator to RN/RPN to LPN/aide
  • Most examples r/t instrumental research use
  • Gradient from conceptual to instrumental
  • Examples common to groups
  • Talking/discussing
  • Following policies and procedures/protocols
  • Specific conditions
  • e.g. wound care, Braden scale

37
Indicators
  • Following research-based policies and
    procedures/protocols
  • Asking questions
  • Peer-to-peer interactions
  • Reading research literature
  • Bringing in research articles
  • Self report
  • Observation
  • Chart documents

38
Focus Groups Round 2
  • Validation for IRU schema developed based on
    round 1 analysis
  • Self-report
  • Questionnaire
  • Interview (healthcare provider, manager)
  • Observation
  • Chart audit

39
  • Findings Swedish ARM

LW
40
Terminology
  • Research Utilization not a common term (but might
    work)
  • More accepted than evidence-based practice -
    trendy concept

41
Definitions
  • Research Use
  • Basing nursing on scientific findings
  • Working according to appraised and compiled
    information
  • Adopting a critical approach to ones work
  • Conceptual Research Use
  • learning through critical reflection/thinking
  • Instrumental Research Use
  • Active process

42
Examples
  • Most examples r/t instrumental research use
  • Adherence to research-based routines/guidelines
  • Using systematic assessment instruments
  • Quality management work
  • All groups mentioned application of concrete
    findings r/t procedures and treatments
  • Insulin Injection technique
  • S/C injection of sterile water for labour pain

43
Indicators
  • Felt instrumental and conceptual research use
    were observable
  • Following research-based routine/guidelines
  • Updating routine/guidelines
  • Observation
  • of specific practices, using systematic
    assessment instruments, a structured nursing
    working method
  • Chart documents
  • Using internet or intranet to seek information
  • Providing rationale for ones actions
  • Peer-to-peer interactions

44
Similarities between Canadian and Swedish
findings
  • Instrumental
  • Adherence to research-based policies/protocols
  • Easier to talk about specific practices
  • Examples increased in concreteness from
    manager/educator to RN to LPN/Aide
  • Conceptual
  • A progression of conceptual to instrumental
  • Equated with thinking/critical thinking

LW
45
Learnings Directions
  • Participants not at all accustomed to discussing
    research utilization
  • Consequently, focus groups not an optimal method
  • Much confusion about conducting research vs. use
    of research
  • Can we distinguish conceptual RU from
    instrumental RU? RU often viewed as a process
    with similarities to the process of EBP. Is
    conceptual use necessarily a precursor to
    implementation? What about implementation without
    understanding?

46
Learnings Directions
  • Critical reflection viewed as an important
    attribute to RU. At the same time participants
    claimed that much of RU was unreflected.
  • Enhanced awareness of the knowledge base for
    clinical practice - a strategy for RU? How to
    measure RU if practitioners are not aware of if
    they are using research?
  • Adaptation to patients perceptions and
    preferences important and contributes to make the
    concept of RU vague. Consequences for measuring
    RU?

47
Next Steps
  • Systematic review (update) of instruments to
    measure research use (in-progress)
  • Manuscript on Canada/Sweden findings re
    instrumental use (in-progress)
  • Concept analysis (literature based) on conceptual
    use (in-progress, Canada)
  • Submit grant applications 2008
  • Develop and pilot instrumental measures
  • Construct clarity re conceptual use (concept
    analysis using ethnoscience)
  • Eventually development of a measure of CRU

CE
48
Psychometric theory
PG
49
Sources
50
Manifest Observed responses
Latent, Not manifest
51
To a great extent
To a moderate extent
To a little extent
To no extent
52
i1 i2 i3 i4
53
Strength
i1 i2
54
Location
55
  • an instrument is valid for measuring an attribute
    if and only if
  • (1) the attribute exists and
  • (2) variations in the attribute causally produce
    variations in the outcomes of the measurement
    procedure

56
Classical Test Theory Strength
i1 i2 i3 i4
57
Item Response Theory Location
58
Respondents Items
Direction of increasing X
Respondents with high X
Item indicating highest level of X
Item indicating higher level of X
Respondents with midrange X
Item indicating lower level of X
Item indicating lowest level of X
Respondents with low X
Direction of decreasing X
59
The construct
  • The research-practice gap
  • Barriers to research use
  • Inability to obtain research findings in the area
    of interest
  • Understanding, Resistance, Relevance, Agreement
    with conclusions
  • Time constraints
  • Costs
  • Rewards

60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
10 to 60
65
Unidimensional?
  • Measures of model fit
  • Absolut fit
  • SRMR0.09 (0.07)
  • Parismony fit
  • RMSEA0.08 (0.05)
  • Comparative fit
  • CFI0.93 (0.95)
  • not unidimensional!

66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
Conclusions
  • Location
  • Process
  • building an hierarchy
  • Strength
  • Variance
  • Repeated measurement

70
Respondents Items
Construct Map
Direction of increasing X
Respondents with high X
Item indicating highest level of X
Item indicating higher level of X
Respondents with midrange X
Item indicating lower level of X
Item indicating lowest level of X
Respondents with low X
Direction of decreasing X
71
  • Relevant Theories for
  • Research Utilization Studies

72
Theory of Planned Behaviour
Attitudes
Subjective Norms
Behavioural Intention
Behaviour
Perceived Behavioural Control
Ajzen Madden, (1986), Journal of Experimental
Social Psychology, 22, 453
73
Promoting Action on Research Implementation in
Health Services (PARIHS)
Successful implementation is a function of
evidence, context and facilitation
Context Culture Leadership Evaluation

Context


Facilitation
Evidence
Purpose Role Skills
Research Clinical experience Patient
experience Local data
Evidence-based practice
High
Kitson et al, 1998 Rycroft-Malone et al,
2002 Rycroft-Malone et al, 2004
Low
74
Diffusion of Innovation
  • Elements
  • The Innovation
  • Communication Channels
  • Time
  • Social System
  • Stages
  • Awareness
  • Persuasion
  • Decision
  • Implementation
  • Adoption

1931 - 2004
  • Influencing factors
  • Attributes of the innovation
  • Individual
  • Organizational

75
(No Transcript)
76
Group Work What does the theory tell us to
measure?
  • Crucial concepts in each theory? What is the
    dependent variable in the theory?
  • Focusing on one theory what concepts are
    important to measure? How do these concepts
    relate to research use? Are they a measure of
    research use?
  • How would you measure research use according to
    the theory? What measurement approach would you
    take?
  • Is any one theory adequate to guide the
    measurement of research use?
Write a Comment
User Comments (0)
About PowerShow.com