Evaluation and Usability

1
Evaluation and Usability

• Jon Ivins

2
Overview

• This lecture continues with the topics of
  usability and evaluation
• Examples of these are given
• Some techniques for requirements analysis and
  usability evaluation are introduced
• User needs are considered more fully in the next
  lecture

3
What are they?

• Evaluation is
• a set of concepts which define qualities to be
  tested
• procedures to be carried out
• means of interpreting the results
• Usability are the goals for evaluations

4
Why bother with usability testing?

• Benchmark testing to measure quality
• ISO 9000 13000 series see page 42
• Kitemarks
• targets set by market research
• Diagnostic
• find errors in design
• Compare two, or more, designs

5
Are there any standards?

• ISO publish 9000 13000 series
• Company driven
• Macintosh Microsoft publish guidelines for
  future products
• In-house e.g. web pages, intranet
• De-facto
• Windows dominates the PC market, Sun for UNIX, etc

6
What can we measure? (1)

• Utility
• how good is the interface at helping user do a
  task? (task fit)
• Can use prototypes / mock-ups with users
• not easy to measure
• 95 of users are satisfied with the navigation
• Coverage (how much of the application is used)
• frequency of facilities used by x of users
  within a given time period

7
What can we measure? (2)

• Usability
• Effectiveness
• error rates, task completion times, facilities
  usage
• Learnability
• error rates fall over time, fall in task
  completion time, increase in knowledge of
  facilities
• Memorability
• recall of task and facilities after a long time
• Attitude satisfaction
• very subjective

8
Examples

• Effectiveness
• 95 of users complete task X within 10 minutes
  with an error rate of less than 2
• Learnability
• 95 of users learn to use the commands to
  complete Task X within 20 minutes of training
• Memorability
• 95 of users should recall and describe 90 of
  commands after 7 days
• Attitude satisfaction
• 95 rate overall systems as 5 on a 7 point scale

9
Data Collection Techniques

• observation
• think-aloud protocols
• interviews
• monitoring
• questionnaires
• covered in first year and reviewed at start of
  this
• Focus groups a relatively new idea, currently
  fashionable

10
Procedures for benchmark evaluation

• set usability goals
• develop prototype
• design evaluation tests
• THE HARD PART!!!!!!
• select users
• collect results
• analyse results

11
Procedures for diagnostic evaluation

• Expert judgement
• guidelines checklists e.g ISO 9421
• walkthroughs
• design evaluation tests
• select users
• test
• analyse

12
Are there any problems?

• IT moves rapidly but Standards Organisations do
  not
• Are the guidelines being used valid?
• Is there paralysis by analysis?
• Guidelines can be too prescriptive or too general
  to be any use
• Following guidelines does not guarantee usability
  but it does help!

13
Are there any problems cont?

• Concentrating on usability per se is pointless
  unless
• The needs of the user have been identified
  correctly
• The designer(s) have clearly thought out HOW the
  user needs will be met

14
User Needs

• Will be covered in the next lecture
• However, a key issue for multimedia designers is
• WHO ARE THE USERS?
• The people who pay for the work?
• The people who will use the application?
• Other interested parties?
• Sometimes the answer is not obvious!

15
Identifying Users

• Peuple recommends conducting a stakeholder
  analysis to do this p52 on
• Once the stakeholders are identified, we can
  identify the characteristics of each.
• The designers can then write a draft functional
  specification
• This leads on to the interface design
  specification
• Now, usability can be considered.
• CAN YOU SEE A POTENTIAL PROBLEM(S) WITH THIS
  APPROACH?

16
Summary

• We have looked at
• what are evaluation and usability?
• why bother with them?
• what can be measured?
• techniques to measure them
• procedures to conduct tests