UBICOMP 2002 W2: User Centred Evaluations for Ubiquitous Computing Systems: Best Known Methods

1
UBICOMP 2002W2 User Centred Evaluations for
Ubiquitous Computing Systems Best Known Methods

• Scenario Evaluations for Ubiquitous Computing -
  Stories Come True
  
• VTT Information Technology, Human-Centred Design,
  Tampere, Finland

2
One ordinary morning in the near future

• You live in an apartment at the old town of
  Gothenburg, Sweden. It is an ordinary autumnal
  Tuesday morning. Your electronic calendar has
  noticed that you have to be at the office by 9,
  because you have a meeting at 9.15. Your car's
  travelling program has sent a message to
  calendar, that the trip from your apartment to
  the office takes 20 minutes. The calendar has
  sent a message to the alarm system, which actives
  wake-up at 8 o'clock.
• First there are silent voices, birds are singing
  softly in the background. Your favourite bird is
  finch. Little by little the curtains are opened,
  lights turn brighter and the finch's song becomes
  louder. As the awakening continues, the room
  fills with coffee smell and with the scent of
  coffee and with sounds of morning routines.
  Gradually you start to be aware of your
  surroundings and sit up at the side of the bed.
  You notice that you have a morning meeting and
  you go to the shower...

3
SCENARIO METHOD - definitions - what?

• Scenarios stories or descriptions of possible
  futures.
  
• Story of the usage of a service or product.
• Users evaluate scenarios made by others, or
• Users build scenarios themselves.

4
SCENARIO METHOD - definitions - why? ( also
benefits)

• Scenarios help potential users to understand
  imagined applications and services of new
  technologies.
  
• Scenarios treat technology quite flexibly
  technology can either be described in detail or
  the focus can be more on the context of use
  
• Scenarios make ideas more concrete for the
  purpose of analysis and communication.

5
SCENARIO METHOD - definitions - when?
Human-Centred Design Process
Literature study Analysis/evaluation of correspon
ding products
Observation, interviews
Initial context of use Initial user requirements
New and revised requirements
Scenarios Paper protos Mockups
Design requirements
Evaluation
Revised design
Design and implementation
Prototypes
Easy to use and useful product
6
The goals of the KEN project
The main goal is to make sure that NAVI projects
take into account user needs and preferences as
well as ethical issues in the design.
Results 1) usability guidelines and tools 2) us
ability criteria 3) trade descriptions for person
al navigation products 4) ethical guidelines 5)
a study on usage cultures in Finland and in other
countries 6) guidelines for internationalisation
and localisation 7) international network with us
ability research organisations
7
Purpose of scenario evaluations

• study how credible and useful potential
  customers considered the scenarios made by
  experts. SWOT analysis.
  
• identify needs of the user groups for navigation
  services and let the participators innovate new
  navigation services
  
• help people to understand the idea of personal
  navigation and then to elicit the ideas,
  attitudes, opinions and needs of different user
  groups with regard to navigation services and
  products.
• timing spring 2001

8
Tarzan is already at the gates of the zoo when
his navigator rings up. Tarzan has allowed
himself to be located and his friend Clayton has
noticed in his device that Tarzan has arrived to
town. So Clayton rings his friend and suggests
meeting with Tarzan as he is nearby.....
Tarzan schedules to meet with Clayton in the
afternoon. At the same time Tarzan allows
advertisers to send word to him about nearby
flower shops and their offerings because he wants
to buy flowers for Jane.
9
Future Home -scenario evaluation

• Future Home is a project which was set up to
  define a home network concept that will be based
  on short range wireless technologies.
  
• The beginning of the scenario evaluation
• At first every project group member gave their
  own idea about what Future Home is.
  
• We received 14 scenarios. We analysed them and
  found several overlaps. We stripped all scenarios
  into a flow of events and produced 5 scenarios
  Services, Nomadic users, Monitoring and
  controlling the system, Entertainment and spare
  time, Communication.
• Now the problem was, that it was not realistic to
  implement some of the scenarios during this
  project. Also there were far too many features
  presented.
• The conclusion was to set up a vote in order to
  reduce the features One partner / one vote.
  There were problems with this.
  

10
Future Home -Scenario evaluation continues

• Our goal was still to find possible features and
  to build up final scenarios in order to clarify
  what the system prototype will do.
  
• Other goal was to get the potential users'
  viewpoint in order to produce user requirements
  later on. How users accept the general concept of
  Future Home and Ubicomp.
• Also real users feedback may affect the technical
  requirements.
  
• Interviews users, provides, usability
• 20 people were interviewed during February 2002,
  10 in Finland and other ten in Germany.
  
• The 5 scenarios were sent to them beforehand and
  each interview took approximately an hour. The
  users were interviewed at their homes usually in
  the evening.
• Concentrated on following issues ...

11
Scenario material

• Demonstration video was presented from the
  Internet
  
• http//cooltown.hp.com/cooltownhome/cooltown-video
  .asp
  
• We had visualiations about the possible terminal
  devices made by Nokia, they were presented as the
  scenario was discussed with the user.
  
• The scenario stories were posted or given to the
  users a week before the actual interview.
  
• About the interviews
• All interviews were made at the users home or at
  the office, either in the evening after work or
  at week-end.
  

12
Future Home -Scenario evaluation results

• most wanted vs. rejected features were listed.
• The results were reported to the project group
  and they were also added to a Deliverable
  -paper.
  
• A requirement specification was written and the
  scenarios were presented in a new formal table
  form.

13
Evaluating scenario evaluations - problems
scenario building - terminology - coherent
/ uniform structure of scenarios, e.g. flow of
events, length of scenarios, actors
- potential user participation
qualitative nature of the evaluations
- many interpretations are possible from the same
material - how to ensure that different evaluator
s are doing the same (that results may be
compared) structured interviews, planning
and rehearsing together? - leading group discussi
ons requires training and special skills
- planning, doing and analysing could be time
consuming - results are qualitative answering to
the question why and not how many - quantitative
methods are also needed presenting the results
- to peer groups - how to ensure that the res
ults of the evaluations (user requirements and
attitudes) are put to use later on
14
Evaluating scenario evaluations -development ideas
1. Could it be possible to collect some valid
quantitative material, e.g. with web forms?
2. More active user participation in scenario
building - more ethnography diaries, participan
t observation - more mock-up prototyping with the
users in authentic settings - how to ensure user
commitment and motivation money?
3. Scenario evaluations more interaction with
users - users own habits and experiences, use of
senses, interactive scenarios......