A Process Approach to Outcome Measurement

1
A Process Approach to Outcome Measurement

• This presentation will probably involve audience
  discussion, which will create action items. Use
  PowerPoint to keep track of these action items
  during your presentation
• In Slide Show, click on the right mouse button
• Select Meeting Minder
• Select the Action Items tab
• Type in action items as they come up
• Click OK to dismiss this box
• This will automatically create an Action Item
  slide at the end of your presentation with your
  points entered.

2
A Process Approach to Outcome Measurement

• Topics covered in this presentation
• Design of the evaluation
• Description of the participants
• Knowledge findings
• Participant learning styles
• Instructor teaching styles
• Intention to change practice findings
• Motivation to change findings
• Lessons learned

3
The Study

• These data were developed at the 2006 Optimal
  Management of HIV Conference
• My collaborators, Dr. Harold Kessler and Michael
  Saag, and I are in the process of analyzing these
  data for publication
• The analyses I report here are preliminary are
  completed only for the purpose of these
  discussions

4
Two Elements Participant Descriptors and
Process Measurement
5
Data gathered

• Learning style
• Preference indicated by degree of endorsement of
  one or the other end of a dichotomy
• Example
• I prefer
• Hands-on learning experience ...Learn
  ing through thinking and reasoning
• Learning through simulation Learning
  through lectures

6
Data gathered (continued)

• Demographics
• Questions on age specialty and Board status
  delivered on the Learning Style questionnaire
• Motivation to change
• Motivation indicated by a modified standard scale
  of motivation to change
• Example
• Its important to use the new approaches Ive
  learned Strongly Agree - Strongly
  Disagree

7
Data gathered (continued)

• Pre-Post Tests
• Knowledge
• Simple multiple choice knowledge based tests, one
  question per presenter per test, rotated to
  prevent order bias
• Example
• Which of the following antiretrovirals is the
  least likely to cause DSPN
• indinavir
• zalcitabine
• stavudine
• didanosine
• Laminvudine

8
Data gathered (continued)

• Pre-Post Tests (continued)
• Intent to change
• Procedural intent questions provided prior and
  post the session
• Example
• Pre
• When treating HIV patients with PCP I use
  Strongly Agree - Strongly Disagree
• TMP-SMX or TMP-dapsone
• Post
• When treating HIV patients with PCP I
  Strongly Agree-Strongly Disagree
• intend to use.

9
Data gathered (continued)

• Teaching style
• 3-5 observers rated each session using simple
  descriptors of the presentation using agreement
  or disagreement with the description
• Example
• The Presentation focused on clinical application
  Strongly Agree - Strongly
  Disagree
• The presenter focused on the underlying science
  of medicine Strongly Agree - Strongly
  Disagree

10
Description of the participants

• Of the total participants in the program,
  analysis is focused on the 62 complete datasets,
  these participants provided data for all three
  days both in the morning and afternoon
• While it is possible that they differed in some
  systematic way from the rest of the participants,
  complete data will be necessary for the ultimate
  analysis

11
Results Participants were generally middle aged

• The participants had an average age of 49
• The youngest was 29 and the oldest 73
• The median age was 47 indicating a slight skew to
  the younger side of 49.

12
The average participant graduated from medical
school in 1984

• The youngest participant graduated in 2003
• The oldest participant graduated in 1958

13
Most participants were in infectious disease or
general internal medicine
Specialty Percentage
Family Medicine 36
Infectious Disease 24
Internal Medicine 28
Missing 6
Pediatrics 4
Psychiatry 2
14
Most of the participants were ABMS Board
Certificants

• 62 of participants reported Certificant status
  for a Board
• 37 were not members of a Board or did not
  respond.

15
The most frequent Boards of participants were
Infectious Disease or Internal medicine
Category Relative frequency per category ()
AAHIVS 3.3
Family Medicine 33.3
Infectious Disease 26.6
Internal Medicine 30.0
Pediatrics 3.3
Psychiatry 3.3
16
The participants improved their performance

• Participants improved their performance from pre
  to post measures
• The improvement was approximately 10.
• The improvement was significant p lt .001.

17
17
Participants also showed an expectation of
changing practice patterns

• Intention or expectation of changing practice
  patterns was measured by asking the likelihood of
  a particular practice being adopted
• The likelihood was measured prior to the session
  (Morning) and after the session (Afternoon).
• The stated likelihood increased from 3.4 to 4 on
  a 5 point scale.
• The effect was assessed and the increase in
  likelihood is significant plt.001.

18
Learning style was assessed a modification of a
scale developed by Kolb

• The approach to learning style that was employed
  in this study was a modified version of Kolbs
  learning style inventory.
• We employed his core scale, adapting it from a
  dichotomy to a numeric format with a Kolb
  descriptor on each end of the scale.

19
Teaching style was assessed by an observer
assessment scale

• The items were selected to reflect the teaching
  characteristics for the Kolb learning styles
• During each presentation, 3-5 observers assessed
  the style of each presenter.

20
Motivation to change

• Participants were asked to endorse 30 statements
  of change commitment
• The statements were modified from the original
  focus to reflect motivation and readiness to
  learn new information and make changes in
  clinical practice patterns.

21
A Process Approach to Outcome Measurement

• Outcome measurement
• Direct assessment/measurement of actual practice
  change or improvement in patient health status is
  difficult
• Focus is on valid secondary measures such as
• Self-report or,
• Vignettes.
• The core question for CME measurement
  professionals is the validity of such secondary
  measures.
• Assessment validity is typically based on a
  combination of
• Face validity (is it reasonable)
• Construct validity (does it render a measurement
  correlated with some other measure deemed valid)
  and,
• Predictive validity.

22
First we looked at the acquisition of knowledge

• We used the teaching style judgment questions as
  indicators of an underlying set of predominate
  teaching styles
• We then used the averaged teaching style scores
  for each instructor as predictors of knowledge
  acquisition of the participants.

23
Teaching and Knowledge Acquisition

• We looked at the relationship between Teaching
  Style and Knowledge
• We found a significant relationship between the
  two measures
• The unobserved variable Teaching Style explains
  approximately 25 of the variance in Knowledge.

24
We looked at the formation of an intent to change

• First, we used the learning style questions as
  indicators of an underlying set of learning
  preferences
• We then used those preferences as predictors of
  intent to change practice patterns

25
Learning Style, Knowledge and Intent to Change
Practice

• We looked at the relationship between Learning
  Style, Knowledge and Intent to change practice
  patterns
• We found a significant relationship between the
  two causal measures and Intent
• The unobserved variable Learning Style explains
  approximately 13 of the variance in Intent
• The variable Knowledge explains approximately 55
  of the variance in Intent.

26
Self report of practice change

• First we looked at the relationship between
  Motivation to change and reported practice
  change
• Motivation to change was assessed using a series
  of readiness for change items
• Change items were taken as indicators of an
  underlying state of change readiness.

27
Motivation and Intent to Change Practice and
Self-reported Practice

• We looked at the relationship between Motivation
  to change, Intent to change practice patterns and
  self-report of practice patterns at a 12 month
  follow-up
• We found a significant relationship between the
  two causal measures and self-report practice
  change
• We also found a relationship between motivation
  and intent to change.

28
A Path Model of Cognitive Elements of CME Efficacy

• We looked at the relationships among a number of
  variables as they relate to practice change. We
  found
• Teaching Style has a path to Knowledge
• Knowledge and Learning Style have paths to Intent
  to Change
• Motivation to Change and Intent to Change have
  paths to Self-reported Practice Change.

29
Discussion

• All of these variables are related in a causal
  manner with self-reported practice change
• Each variable has a theoretical reason to be
  thought of as causally related to practice
  change
• These data suggest that there are multiple
  variables that could provide a meaningful
  estimate of the degree of efficacy of the CME
  program.