Title: JEM: A Motivational Model of Evaluation for Information Environments
1JEM A Motivational Model of Evaluation for
Information Environments
- Jeff Stanton, Ping Zhang, Gisela von Dran
- Syracuse University
2Information Environments (IE)
- An Information Environment (IE) is an artifact
that encompasses information content and
organization or structure in support of access to
information content (Davern, Teeni, Moon 02) - Examples are
- Websites, Web pages, databases, palm PDAs (info
access), smart-phones (address book, access to
other sources), kiosks (interface), handheld GPS
mapping systems (map, info, control), Information
Systems - Excludes
- The technical infrastructure (e.g. hardware,
supporting software, network, connection, etc.)
of an artifact - Other artifacts that may have an user interface
but not for information seeking purposes
(dishwasher, a cars dashboard) - Supports goal-driven (information seeking)
activities
3Problems w/ Existing Studies
- Many studies lack theoretical foundations for
user evaluation of IEs - For those that are theoretically based (80
empirical studies in IS literature), - Most are generally about beliefs and attitude
changes,and their impact on subsequent intention
and use - Few on the affect changes of user experience
evaluation
4Individual Level IS Evaluation Dominating
Theories, Constructs, Emphases
5Objectives of the Study
- To understand the affective experience a user has
when using and evaluating an IE. - To synthesize current theories from behavioral
science literature into a workable
process-oriented framework of user evaluation - To use the framework to gain a better
understanding of processes underlying user
evaluation of an artifact as well as the likely
future behavior toward that artifact - After testing and pruning, to use the framework
as a guide to evaluation and assessment for IEs - To use good, theory-driven evaluation tools to
provide guidance to IE designers about how to
improve the experience of users
6Basic Working Assumptions
- A given users evaluation of an IE is a function
of both - The artifact itself, and
- The person who is using it.
- A users ultimate judgments about an IE accrue
neither solely from person characteristics nor
solely from the characteristics of the IE but
from the intersection of the two that occurs as
the user uses the artifact to conduct and
possibly complete some goal oriented tasks.
7JEM Joint Evaluation Model
- Jointly scaling artifact and person
- Artifact brings observable features to the
interaction - Static visible without interaction (color,
layout of a web page) - Dynamic aspects of performance (loading time,
search results, operation of controls) - Person brings both stable and transient
characteristics to the interaction - Stable general /- outlook (affectivity), prior
experience with and knowledge of similar
artifacts - Transient specific goals for using the artifact,
mood (state affect)
8Our Strategy for Framework Synthesis
- Try to shift the focus from attitudes (cold) to
emotions (warm) by integrating Regulatory Focus
Theory - Include the notion of discrepancy or matching or
confirmation of expectations and reality by
incorporating elements of Expectation-Confirmation
Theory - Describe a process model that provides details
about what happens during the actual process of
evaluation (rather than simply post-evaluation)
9Existing Theories Part I Regulatory Focus
Theory (RFT)
- Higgins 1997, 1998 Brockner Higgins 2001
developed an emotion-centered motivation theory
that overarches several popular precursor
theories - Goal setting theory, expectancy-valence theory,
behavioral decision theory - RFT posits two self-regulatory (internal)
systems through which individuals exert control
over their emotional state via behavior - Promotion focused Exert effort towards achieving
an ideal goal or standard - Cheerful when goal achieved, dejected when goal
not achieved - Prevention focused Exert effort towards
vigilance and loss avoidance - Quiescent when goal achieved, agitated when goal
not achieved - Individual difference situational component
- Stable trait Predisposition toward promotion or
prevention focus - Situational factors Shift regulatory focus
depending upon whether one perceives
situationally specific outcomes as potential
gains or losses - Extensive psychological literature supporting
theory predictions
10Existing Theories Part II Expectation-Confirmati
on Theory
- Expectation confirmation/disconfirmation from the
customer satisfaction literature on consumer
satisfaction post-purchase behavior - E.g., Cardozo (1965), Anderson (1973), Oliver
(1980, 93), Dabholkar (2000) - Bhattacherjee 01 IS Continuance
- Key constructs initial expectation, initial
consumption/use and perceptions about
performance, confirmation, satisfaction,
continuing intention. - Consistent with Fishbein/Ajzen Theory of Reasoned
Action with the addition of an attitude
discrepancy construct
11Pre-Evaluation Phase
Evaluation Phase
Post-Evaluation Phase
Similar Artifacts
This Artifact
ApproachIntentions
Artifact Confirmation
Artifact expectation
Circumplex Evaluation
Regulatory Focus
Quiescence
GoalProgress
Elation
Goal Cognition
AffectChange
State Trait Affect
AvoidanceIntentions
12Pre-Evaluation Phase
- Conditions that exist prior to the users
experience with the artifact - Prior experience with similar artifacts
- Regulatory focus Whether ones purpose in use of
the artifact is pursuit of a gain or avoidance of
a loss. - Goal cognition Specifically what the user hopes
to achieve in using the artifact and something
about his/her strategy or plan for achieving it. - Affect (Emotion) The users emotional and mood
status just prior to the use of the artifact
(important to establish a baseline for assessing
subsequent change)
13Evaluation Phase
- What occurs during the time the user uses the
artifact - Three processes (with iterations)
- The user, guided by previous experience with
similar artifacts, makes sense of the artifact by
understanding its features, then compare the
perceived performance with expectation - Meanwhile, the user attempts to pursue a
particular goal, thus cognitively evaluating
whether s/he is achieving that goal - The users hedonic (emotional or affective) state
changes as a result of making progress toward
that goal and also a result of making sense of
the artifact - We suggest that reaching a goal or making
progress toward a goal has a positive influence
on hedonic state as does recognizing the presence
of a desired feature that would facilitate goal
progress
14Circumplex Affect
- Following recent psychological research on the
structure of affect (e.g., Watson and Tellegen) - Also consistent with RFT
- Affect/Emotion has two distinct and orthogonal
dimensions to it - Elation The positive/negative dimension, where
one falls on the cheerfulness-dejectedness
dimension - Quiescence The activated/inactivated dimension,
where one falls on the quiescence-agitation
dimension
15Post-Evaluation Phase
- The user reaches a final hedonic state as a
result of his or her experience with the artifact - A memory of this hedonic state will guide his/her
future behavioral intentions and thereby
influence subsequent behavior - Because RFT and Circumplex describe two distinct
emotive conditions, the behavioral intentions may
be separable into - Approach intentions
- Avoidance intentions
16Examples Using Websites
Person A Person B
Artifact expectation Normal search engines Normal info intensive sites
Regulatory focus Pursue a gain Avoid a loss
Goal Find best airfare for vacation Find info on breast cancer
Affect trait Positive Positive
Affect state Happy, calm Sad, agitated
Artifact confirmation Below expectation Equal to or higher than expectation
Goal progress Not be able to find a decent fare during a reasonable time A comprehensive coverage
Affect change Yes Yes
Elation affect Dejected Sad
Quiescence affect Agitated Calm
Intention Weak approach strong avoidance user will actively seek for alternatives and criticize it when prompted Strong approach and weak avoidance user will actively and frequently reuse or recommend it if opportunities arise
17Examples (contd) Affect Changes
Person A Bad Travel Site
Person B Good Breast Cancer Site
Happy
Happy
Start
Elation
Elation
End
End
Start
Sad
Sad
Agitated
Calm
Agitated
Calm
Quiescence
Quiescence
18Preliminary Propositions (1/2)
- A user, dependent upon the goal (or mood at the
time of evaluation) may evaluate the same IE
differently at different times. - The users hedonic state can be improved during
the evaluation process by facilitating goal
achievement (or matching the artifact toward the
expectation). - Positive changes in Elation component of the
hedonic state will increase the likelihood of
subsequence approach behavior may not strongly
affect subsequent avoidance behavior. - Changes in Quiescence (Activation) component of
the hedonic state toward the agitated end will
increase the likelihood of subsequent avoidance
behavior, but will not strongly affect subsequent
approach behavior.
19Preliminary Propositions (2/2)
Positive Elation Negative Elation
High Activation Strong approach and strong avoidance user will actively and frequently reuse or recommend IE in an effort to avoid lost opportunities. Weak approach and strong avoidance user will actively seek alternatives and criticize IE in order to help prevent own or others losses due to wasted time or frustration.
Low Activation Strong approach and weak avoidance intentions user may use, revisit, or recommend IE if opportunity arises. Weak approach and weak avoidance user will forget about IE or criticize it if prompted.
20Pilot Study
- Designed to explore the salience of various IE
features - E.g., how important were the navigational
features versus the credibility of the
information versus the layout, and so forth - 150 working people in Spring 2002 a pre-measure
and a post-measure of mood using Watson and
Tellegens Positive Affect, Negative Affect
Survey (PANAS) - We found that a brief use of a web-based
encyclopedia showed changes in both positive and
negative affect - Additionally we found that both person variance
and artifact variance influenced affect
21Potential Contributions
- With more testing, we may have a process model
that elucidates the processes underlying user
evaluation of an artifact as well as the likely
future behavior toward that artifact - The model focuses on the hedonic experience a
user has during goal driven use of an IE. - The model tries to clarify constructs and their
relationships, as well as provide a model-driven
basis for measurement
22Potential Practical Contributions
- May contribute to efforts to change and improve
the design of IEs identify and cater to users
potential motivations of using the IEs and
improve their emotional experiences of using IEs - May provide useful strategies for understanding
and predicting why user evaluations of IEs change
over time, why users do not agree among
themselves about the quality of an IE, and how
strategies/contexts of evaluation may influence
the outcomes
23Limitations
- A preliminary conceptual model needs empirical
support - Quite complex needs trimming in order to include
just the most important mechanisms and constructs - Did not specifically consider repeated use of the
same IE and the evaluation of it - Does not reflect the social context of artifact
evaluation - May have no applicability to evaluation of other
kinds of products or user interfaces
24Future Research
- Operationalize the JEM model
- Develop research propositions
- Develop specific hypotheses for empirical testing
- Complete a preliminary verification
- Revise the model as needed
- Confirm the revised model
- Publish validated assessment tools