Evaluation and clinical informatics

1
Evaluation and clinical informatics

• Annette L. Valenta, DrPH
• Professor and Department Head
• www.bhis.uic.edu
• VA Cyberseminar Series
• June 20, 2006

2
What is medical informatics

• Deals with biomedical information, data, and
  knowledgestorage, retrieval, and optimal use for
  problem-solving and decision-making (Shortliffe
  et al., 2001)
• Covers the computational and informational
  aspects of processes and structures

3
What is clinical informatics

• Subset of medical informatics concerned with
  clinical practice
• Represents for medical informatics what clinical
  epidemiology is for epidemiology (Degoulet
  Fieschi, 1997)

4
What does the discipline do?

• Extracts information from clinical data
• Provides the relevant information to those who
  need it
• Produces new knowledge
• Borrows methods and tools from domains other than
  medical research

5
What is informatics evaluation?

• Systematic application of social research to
  judge/improve an informational resources design
  and implementation (Rossi Freeman, 1989)
• Process of describing implementation of a
  resource and judging its merits and worth
  (Guba Lincoln, 1981)
• Leads to the settled opinion that something about
  an informational resource is the case, leading to
  a decision to action (House, 1980)

6
Why do evaluations?

• Promotional Reassure systems are safe and
  beneficial
• Scholarly To obtain a deeper understanding of
  technology structure/function and effects on
  practice
• Pragmatic Otherwise, developers will never know
  what worked and did not work
• Ethical To justify its safety in practice to
  justify its cost/benefit
• Medicolegal Developers should collect accurate
  information to assure safety and effectiveness
  for the purpose

7
Evaluation vs. research

• Agenda set by client
• Immediate impact required
• Seek out unanticipated result/ comprehensive
• Multi-sourced data

• Agenda set by researcher
• No immediate impact required
• Exclude extraneous variables or not controlled
• Single data collection paradigm

8
Evaluation characteristics

• Tailor the study to the problem
• Collect data useful for decision-making
• Look for intended and unintended effects
• Study while under development and once
  implemented
• Study in lab and in field

9
Evaluation characteristics (Contd)

• Take environment into account
• Let key issues emerge over time
• Use diverse methods

10
Evaluation typologies

• Objectivist
• Logical-positivist, linear
• Hypothesis driven
• Attributes of interest are properties of resource
• Agree on gold standard of outcome
• Numerical measurement
• Can prove worth beyond reasonable doubt
• Hypothetico-deductive confirmation of theory

• Subjectivist
• Intuitionist-pluralist, nonlinear
• Issues driven
• What is observed about a resource depends on the
  observer
• Merit and worth explored in context
• Verbal descriptions are useful
• An exercise in argument, not demonstration
• Inductive generation of hypotheses

11
Objectivist approaches

• Comparison-based
• Objectives-based
• Decision facilitation
• Goal-free

12
Steps

• Negotiation of questions to be answered
• Agreement as to methods
• Investigation
• Report

13
Limitations of scientific method

• That theories make real world sense
• Cannot test relevance of hypotheses
• Cannot test whether proper variables have been
  operationalized
• Individual differences always allocated to random
  error

14
Evaluation typologies

• Objectivist
• Logical-positivist, linear
• Hypothesis driven
• Attributes of interest are properties of resource
• Agree on gold standard of outcome
• Numerical measurement
• Can prove worth beyond reasonable doubt
• Hypothetico-deductive confirmation of theory

• Subjectivist
• Intuitionist-pluralist, nonlinear
• Issues driven
• What is observed about a resource depends on the
  observer
• Merit and worth explored in context
• Verbal descriptions are useful
• An exercise in argument, not demonstration
• Inductive generation of hypotheses

15
Subjectivist approaches

• Quasi-legal
• Art criticism
• Professional review
• Responsive/illuminative

16
Responsive/illuminative approach

• Investigators immersed in the environment
• Collect data primarily through observations,
  interviews, review of documents
• Data collection plans evolve as experience
  accumulates adjust future aspects based on
  information obtained
• Reports tend to be narrative in nature
• Can be performed before, during, or after
  introduction of information resource

17
Steps

• Negotiation of ground rules (orienting questions)
  of study
• Immersion into environment
• Initial data collection to focus questions
• Iterative loop of data collection, analysis and
  reflection, checking details with participants in
  study, reorganization for next data collection
  loop
• Preliminary report that could feed back into
  sharpening of the study findings
• Final report

18
Data collection

• Pure observation or participant observation
• Interviews
• Formal - unstructured, semi structured,
  structured
• Informal spontaneous discussions
• Document/artifact analysis
• Anything else useful

19
When introducing information resources

• Anticipated effects
• Transient effects
• Unanticipated effects

20
Anticipated effects

• Decision-making
• Operations
• Quality of information
• Organizational structure
• Personnel's attitudes
• Staffing
• Costs of operation and information processing

21
Unanticipated effects

• Communication
• Care
• Context
• Cost
• (organizational) Control

22
Evaluation plan

• Focus on variety of technical, economic,
  organizational concerns
• How are processes affected by info resource?
• Use multiple methods
• Be modifiable
• Be longitudinal
• Be formative as well as summative

23
Evaluation plan

• Focus on variety of technical, economic,
  organizational concerns
• Use multiple methods
• Scaled response, open-ended questions,
  observations, document analysis, logs
• Be modifiable
• Be longitudinal
• Be formative as well as summative

24
Evaluation plan

• Focus on variety of technical, economic,
  organizational concerns
• Use multiple methods
• Be modifiable
• While keeping in mind scientific rigor
• Be longitudinal
• Be formative as well as summative

25
Evaluation plan

• Focus on variety of technical, economic,
  organizational concerns
• Use multiple methods
• Be modifiable
• Be longitudinal
• Org change happens over time
• Be formative as well as summative

26
Evaluation plan

• Focus on variety of technical, economic,
  organizational concerns
• Use multiple methods
• Be modifiable
• Be longitudinal
• Be formative as well as summative

27
Data analysis

• Generally collate individual statements and
  observations by theme and source
• Triangulation checks on veracity
• Closure, saturation, convergence
• Verification

28
Data analysis

• Generally collate individual statements and
  observations by theme and source
• Triangulation checks on veracity
• Look across different types of information
  (observation, interview, document analysis) for
  consistent picture of theme
• Closure, saturation, convergence
• Verification

29
Data analysis

• Generally collate individual statements and
  observations by theme and source
• Triangulation checks on veracity
• Closure, saturation, convergence
• If hear nothing new throughout different
  approaches, likely exhausted the views available
• Verification

30
Data analysis

• Generally collate individual statements and
  observations by theme and source
• Triangulation checks on veracity
• Closure, saturation, convergence
• Verification
• By individuals external to study and by
  participants themselves

31
In ANY form of evaluation

• Evidence collected with great care
• Evidence compiled, interpreted, reported in
  enough detail to permit replication of study by
  another
• Records are kept available for audit
• Depend on theories for interpretation and use of
  pertinent empirical literature

32
Q-methodology

• Provides systematic means to examine and reach
  understanding about it seems to me or in my
  opinion
• Enables respondents to model their viewpoints
  (most characteristic - least characteristic of my
  viewpoint) on matters of subjective importance
• Uses medium of Q-sort

33
The process of Q creating an instrument
Collation Triangulation Closure, saturation,
convergence Verification
34
The process of Q administering an instrument
Plus
Condition of Instruction
Which issues would be important or not so
important to you
Plus
c 2006 Annette L. Valenta
35
The process of Q analyzing the data
Individual Q-sorts
PQ Method v 2.11 SAS, SPSS
Factor analysis correlation
Factors representing a groups shared perspective
c 2006 Annette L. Valenta
36
ExampleStudents views of web-based instruction
Plus
Condition of Instruction
Which issues are important not so important to
you when thinking about the application of
web-based technology to course instruction?
Plus
c 2006 Annette L. Valenta
37
Factor Matrix With an X Indicating a Defining
Factor   Loadings   QSORT
1 2 3 4 5
1 subjo1 .0496 .9162X .1197
-.0187 .1675 2 subjo2 .1788
.2365 .5749X -.0032 -.3106 3 subjo3
.2636 .6071X -.1687 .3245
.0358 4 subjo4 -.1906 .8137X
-.1253 -.0146 .0214 5 subjo5
.0288 .2879 .1061 .6798X -.1023
6 subjo6 .2344 -.2540 .5971
.2512 -.4465 7 subjo7 .1233
.5082X -.0470 .1828 .0166 8 subjo8
.1517 .3772 -.1113 .1523
.5993X 9 subjo9 .0603 .8206X
.2762 .1692 .2918 10 subjo10
.0420 -.0345 .0781 .4148X .1710
38
Normalized Factor Scores -- For Factor 1  
No. Statement
No. Z-SCORES 19 Requires
active learning and initiative
19 1.758 9 Provides flexible time
management 9
1.572 18 Requires self-discipline
18 1.327 14
Requires basic skills in computer troubleshooting
14 1.043 12 Can work at home
when I want 12
.933 21 Can learn at my own pace
21 .889 20
You'll sure learn to use the Internet
20 .758 11 Saves travel
time
11 .499 6 Less discussion with
participants 6
.148 2 Fewer subtleties in teaching -
instructor observation, 2 .139
17 No set class time
17 .013 5 Less
informal learning - side comments by teacher and
stu 5 -.019 4 Less enrichment from
other perspectives 4
-.119 1 Less sense of self assessment in
comparison to others 1 -.127 22
Saves commuting cost
22 -.232 3 Fewer
opportunites to meet now people - social
interactio 3 -.350 10 Potential
interference with work obligations
10 -.518 7 Sometimes hard to find
quiet time at home or school 7
-.769 13 Trouble getting access to Internet at
home 13 -1.087 8
Sometimes computer time hard to get at home
8 -1.118 23 Can work in your
bathrobe 23
-1.416 16 Access to Internet only through
work 16 -1.529 15
Must pay home phone line costs
15 -1.795
3, 2s
0
-2, -3s
c 2006 Annette L. Valenta
39
Factor Q-Sort Values for Each Statement  

Factor Arrays   No. Statement

No. 1 2 3 4 5 1
1. Less sense of self assessment in comparison to
others 1 0 0 -2 -1
-1 2 2. Fewer subtleties in teaching -
instructor observation, 2 0 3
1 1 1 3 3. Fewer opportunities to
meet now people - social interactio 3 -1
1 -1 2 2 4 4. Less
enrichment from other perspectives
4 0 2 -1 0 1 5
5. Less informal learning - side comments by
teacher and stu 5 0 1 -1 1
0 6 6. Less discussion with participants
6 1 3 0
0 1 7 7. Sometimes hard to find
quiet time at home or school 7 -1
0 1 -2 -1 8 8. Sometimes
computer time hard to get at home
8 -2 -1 0 -3 0 9 9.
Provides flexible time management
9 3 1..
c 2006 Annette L. Valenta
40
Factor 1 Time and Structure in Learning
Most important
Neutral
Most Unimportant
41
Example Physicians views of IT

• What do you think about the use of information
  technologies in health care? What do you like or
  dislike? What do you see as advantages or
  disadvantages?

42
Example Physicians views of IT
Plus
Condition of Instruction
What do YOU think about the use of IT in health
care? What do you like or dislike? What do you
see as advantages or disadvantages?
Plus
c 2006 Annette L. Valenta
43
Statement scores by factors/opinion types
Selected Statements 1 2 3 4 5 6
3. Can I take full advantage of computer? -2 4 1 4 2 0
4. Confidentiality and security -3 -1 4 4 3 2
5. Its useful to print out patient education info 2 4 3 2 1 -1
7. Assessing performance is best done by observing -1 0 2 3 4 0
8. Physician knowledge and critical thinking will decrease -4 0 -2 -4 3 -4
9. It will improve communication 1 3 4 1 0 4
10. Useful to obtain eligibility data 1 1 1 1 0 4
11. Will use computers only when voice-activated -4 -4 -4 -4 -3 -1
c 2006 Annette L. Valenta
44
Opinion types

• Full-Range Adopters
• Skills-Concerned Adopters
• Technology-Critical Adopters
• The Independently-Minded and Concerned
• The Inexperienced and Worried
• The Business-Minded and Adaptive

45
References

• Anderson JG, Aydin CE, editors. Evaluating the
  organizational impact of healthcare information
  systems Methods and applications. 2nd ed. New
  York Springer ScienceBusiness Media, 2005.
•  Anderson JG, Aydin CE, Jay SJ, editors.
  Evaluating healthcare information systems
  Methods and applications. Thousand Oaks Sage
  Publications, 1993.
•  Dennis, K. E. (1986). Q methodology
  Relevance and application to nursing research.
  Advances in Nursing Science, 8(3), 6 - 17.
• Friedman CP, Wyatt JC. Evaluation methods in
  biomedical informatics. 2nd ed. New York
  Springer ScienceBusiness Media, 2006.
• Friedman CP, Wyatt JC. Evaluation methods in
  medical informatics. New York Springer-Verlag,
  1997.
• Guba EG, Lincoln YS. Fourth generation
  evaluation. Newbury Park Sage Publications,
  1989.
• McKeown, B., Thomas, D. (1988). Q-methodology
  (Vol 66, Series on Quantitative Applications in
  the Social Sciences). Newbury Park Sage
  Publications. 

46
Questions?