Chapter 12 Observing Users

1
Chapter 12 Observing Users

• Youlan Hu
• Xiaoqian Hu
• Nov 15, 2002

2
Outline

• What and when to observe
• Approaches to observation
• How to observe(in lab and in field)
• Data collection techniques
• Indirect observation
• Analyzing, interpreting and presenting data

3
What and When to Observe

• Observation is valuable any time during design
  stages
• Help understand users need in early design
• Find out whether it meets users needs during
  later refinement of a product
• Goals and questions determine the paradigm and
  techniques used
• Provide a focus for observation
• Should guide all evaluation studies

4
Approaches to Observation

• Quick and dirty observation
• Any where, any time
• Find out whats happening quickly and with little
  formality
• Observation in usability test
• Prototype/products
• In controlled setting
• Observation in field studies
• Early on
• Real-world test of prototype/products

5
Type of Observation
6
Observers

• Outsiders
• In lab or field studies
• Recording is continuous
• Data analysis and observation almost simultaneous
• More objective than participant observation
• Participant observation

7
Participant Observation

• Field studies
• Must get co-operation of people observed
• Informants are useful
• Recording might be interrupted
• Data analysis is continuous
• Interpretive technique
• Questions get refined as understanding grows
• Reports usually contain examples

8
In Lab Versus In Field?

• In field
• Focus on context in which individuals perform
  their work
• More realistic
• Harder to control other factors

• In lab
• Focus on the details of what individuals do
• Easier to control
• Easier to replicate
• more objective

9
How to Observe

• Direct observation
• In same room
• Can be intrusive
• Only see it one time
• Notes usually incomplete
• May use 1-way mirror
• Help get informal and direct information

• Indirect observation
• More distance
• Reduces intrusiveness, but doesnt eliminate it
• Gives archival record
• Data has to be synchronized and managed
• Need time to analyze data

10
Direct Observation in controlled environments

• Issues before observation
• Decide what laboratory to do the tests
• Set up appropriate equipment
• Test equipment to make sure it works
• Provide an informed consent form for users to
  read and sign at the beginning of the study

11
Simple Observation Method

• User is given the task, and the evaluator just
  watches the user
• Problemdoes not give insight into the users
  decision and attitude

12
Think-aloud Method

• Subjects are asked to say what they are
  thinking/doing
• Gives insight into what the user is thinking
• Very widely used in industry
• Potential problems
• Can be awkward
• for subject
• Think aloud
• may modify the
• way users perform
• their task.

13
Constructive Interaction Method

• Join pairs of participants to work together
• Perhaps have one person be semi-expert and one be
  novice
• Remove awkwardness of individual think-aloud
• Provides insight
• into thinking
• process of both
• beginner and
• intermediate.

14
How to observe in the field

• Need to consider in advance
• State initial study goal and question clearly
• Select a framework to guide activity in the field
  
• Decide how to record events
• Be prepared to go through notes and other records
  as soon as possible after each evaluation session
• Highlight and separate personal opinion from what
  really happens

15
How to observe in the field(contd)

• Need to consider in advance
• Refine goals and questions while observing
• How to gain acceptance and trust of those
  subjects
• How to handle sensitive issues
• Consider working as a team
• Whether and how to involve informants
• How to analyze the data
• Plan to look at situation from different
  perspectives

16
Frameworks to Guide Field Observation

• Why use frameworks
• Organize/structure and focus observation
• Simple framework to capture context
• The person. Who?
• The place. Where?
• The thing. What?
• The Goetz and Lecompte(1984) framework
• Who is present?
• What is happening?
• When does the activity occur?
• Where is it happening?
• Why is it happening?
• How is the activity organized?

17
The Robinson Framework

• Space. What is the physical space like?
• Actors. Who is involved?
• Activities . What are they doing and why?
• Objects. What objects are present?
• Acts. What are specific individuals doing?
• Events . What kind of event is it?
• Goals. What are they trying to accomplish?
• Feelings. What is the mood of the group and
  individuals?

18
Dilemmas

• When should I stop observing
• When see similar patterns of behavior being
  repeated
• When finish listening to all the main stakeholder
  groups and understand their perspectives
• How can I adapt ethnography to fit the
  development process
• Preparation
• Field study
• Analysis
• Reporting

19
Indirect Observation

• When direct observation is not possible
• Tracking Users Activities without intruding
• Diaries
• Interaction Logging

20
Diaries

• Track what users did, when they did it , and what
  they thought about their interactions with the
  technology
• Useful when users are scattered and unreachable
• internet and web evaluations
• Templates can also be created online to
  standardize entry format

21
Diaries

• Advantages
• Inexpensive
• No special equipment or expertise needed
• Suitable for long-term studies

• Disadvantages
• Rely on participants being reliable and
  remembering to complete the diaries
• Often incentives are needed and the process has
  to be straightforward and quick

22
Interaction Logging

• For example, you want to track how long people
  stayed at a site, which areas they visited, where
  they came from, and where they went next
• Web page (visitors) counter
• Server logs
• can also raises ethical concerns that need
  careful consideration

23
Interaction Logging

• Advantages
• Unobtrusive
• Large volumes of data can be logged automatically

• Disadvantages
• Ethical concerns
• Powerful tools are needed to analyze the logs
  (WebLog)

24
Data Collection

• Notes plus still camera
• Audio plus still camera
• Video

25
Notes Camera

• Advantages
• Unobtrusive
• Less technical and flexible
• inexpensive
• Easy to transcribe

• Disadvantages
• Writing speed limited
• Difficult to write and observe at the same time
• Rely on the note-takers

26
Audio Camera

• Advantages
• less obtrusive than video
• Less expensive
• Flexible
• Good for recording thinking aloud activities

• Disadvantages
• Lack of visual record (user actions missing)
• Difficult to transcribe the data
• Have to change tapes and position microphone

27
Video

• Advantages
• Capture both visual and audio
• High reliability
• Permanent original record can be revisited

• Disadvantages
• More expensive
• Can be intrusive
• Easy to miss other things going on outside of the
  camera view
• need more than one camera (subject screen)
• Need positioning and focusing camera lens
• Analysis of video data can be time-consuming

28
Dilemma Observing Without Being Seen

• Shall we tell users they are being observed?
• Tell users may react or change their behaviors
• Not tell raising ethical issues
• Depends on the context
• How much personal information to be collected
• How the information will be used

29
Analyzing, Interpreting, and Presenting the Data

• Qualitative analysis to tell the story
• interpret what was observed
• Qualitative analysis for categorization
• content analysis
• Quantitative data analysis

30
Qualitative Analysis to Tell a Story

• Review the data identify key themes, make
  collections
• Record the themes with examples
• Record the date and time for each session
• Check your understanding with people you
  observed
• Iterate this process
• Report your findings to development team

31
Qualitative Analysis for Categorization

• Looking for incidents or patterns
• Analyzing data into categories
• Analyzing discourse

32
Looking For Incidents Or Patterns

• Look for critical incidents, such as times when
  users were obviously stuck
• comments, silence, or looks of puzzlement
• Theory may also be used to guide the study
• help to focus on relevant incidents
• Use Tools to record, manipulate and search the
  data
• NIDIST
• Observer video-pro tool

33
Analyzing Data Into Categories

• Content analysis fine grain way of analyzing
  video data
• determine a meaningful set of mutually exclusive
  categories
• decide on the appropriate granularity
• train a second person, both will analyze
• labor-intensive and time consuming

34
Analyzing Discourse

• Focus on the dialog meaning of what is said,
  rather than the content
• Strongly interpretive, pay great attention to
  context
• E.g. Analyzing discourse on the internet (Chat
  rooms, bulletin boards) has started to influence
  designers understanding about users need in
  these environments
• how to start a conversation
• how turn-taking is structured

35
Quantitative Data Analysis

• Data from video and interaction logs are
  annotated
• Typically this data is analyzed and treated
  statistically
• Usually presented as values, tables, charts and
  graphs

36
Summary of data analysis

• Flag events in real time and examine them in more
  detail later
• Fine-grained analyses can be very time-consuming
• Identifying key events is an effective approach

37
Feeding the Findings Back Into design

• Written reports (overview details)
• Verbal presentation
• Companied by anecdotes, quotations, pictures and
  video clips
• Both qualitative and quantitative analysis useful
• quantitative analysis is used less often and
  depends on goal