Software for Metrology

1
Formalising usability as part of ISO standards
Nigel Bevan nbevan_at_usability.serco.com www.usabil
ity.serco.comwww.usabilitynet.org Serco
Usability Services UK
2
Topics

• What is usability?
• Definitions of usability? ISO 9126-1 and ISO
  9241-11
• Specifying usability as quality in use ISO
  9126-4, the Common Industry Format and ISO 25030
• Usability of everyday products ISO 20282
• User interface usability ISO 9241
• User centred design process ISO 13407
• Usability capability ISO TR 18529

3
What is usability?

• what do you think usability is ......?

4
IBM

• IBM's focus on ease of use has led to technology
  breakthroughs and advances in design research
• For developers and manufacturers, the advantages
  of creating usable products far outweigh the
  costs
• Every dollar invested in ease of use returns 10
  to 100
• http//www.ibm.com/ibm/easy/

5
Microsoft

• The Usability Group is an integral part of the
  product design process at Microsoft
• Over 120 usability engineers
• Over 25 usability labs
• http//www.microsoft.com/usability/

6
Why is usability important?

• Focus on user and organisational needs
• reduce development times
• less training and support and documentation is
  required
• Improve productivity
• simpler interface, fewer user errors
• Improve the competitive edge
• increasing expectations for ease of use
• increasing usability of competitive products
• high profile of usability in advertising
• Improve the quality of life
• less stress, users are more satisfied
• lower staff turnover
• Health and safety legislation
• European Directive on Display Screen Equipment

7
Case study Hewlett Packard

• User-centred design methods were applied to
  redesign software used for identifying network
  problems
• Old New
• time to finish task 9.4 min 4.1 min
• problems identified 16 78
• average length of call 30 min 10 min
• size of manual 25 pages 4 pages
• people needing the manual 53 3
• user satisfaction rating 3.5 6.8
• In addition to the benefits to customers, HP
  recovered their costs in 18 months

8
ISO/IEC 9126 Software Product Quality Model
functionality
reliability
accuracy suitability interoperability security
maturity fault tolerance recoverability availabili
ty
efficiency
usability
understandability learnability operability
time behaviour resource utilisation
maintainability
portability
adaptability installability co-existence replaceab
ility
analysability changeability stability testability
9
ISO/IEC 9126-1 Definitions

• Functionality
• The capability of the software product to provide
  functions which meet stated and implied needs
  when the software is used under specified
  conditions.
• Reliability
• The capability of the software product to
  maintain a specified level of performance when
  used under specified conditions
• Usability
• The capability of the software product to be
  understood, learned, used and attractive to the
  user, when used under specified conditions.
• Efficiency
• The capability of the software product to provide
  appropriate performance, relative to the amount
  of resources used, under stated conditions.
• Maintainability
• The capability of the software product to be
  modified. Modifications may include corrections,
  improvements or adaptation of the software to
  changes in environment, and in requirements and
  functional specifications.
• Portability
• The capability of the software product to be
  transferred from one environment to another.

10
ISO 9241-11 Guidance on Usability
of use.
11
ISO/IEC 9126-1 Software Product Quality Model
functionality
reliability
accuracy suitability interoperability security
maturity fault tolerance recoverability availabili
ty
efficiency
usability
understandability learnability operability attract
iveness
time behaviour resource utilisation
maintainability
portability
adaptability installability co-existence replaceab
ility
analysability changeability stability testability
12
Quality in Use

• ISO/IEC 9126-1 (2001) Software Engineering -
  Product quality - Part 1 Quality model
• quality in use
• the capability of the software product to enable
  specified users to achieve specified goals with
  effectiveness, productivity, safety and
  satisfaction in specified contexts of use
• Quality in use is the users view of the quality
  of a system containing software, and is measured
  in terms of the result of using the software,
  rather than properties of the software itself
• Quality in use measures the combined effect of
  ease of use, functionality, efficiency and
  reliability.

13
Approaches to achieving usability
quality in use
organisational capability
process quality
product quality
development process
effect of the product
life cycle processes
product
usability in context
user centred process
interface and interaction
usability capability
ISO 9241-11 ISO 14598-1 ISO/IEC 9126-1 ISO/IEC
9126-4 ISO 20282-2
ISO 13407
ISO 9241 parts 10, 12-17 ISO/IEC 9126-2/3
ISOTR 18529
14
Approaches to achieving usability

• Design for quality in use
• ISO 9241-11, ISO 9126-4 effectiveness,
  efficiency, (safety), satisfaction
• ISO 20282-2 Usability of everyday products
• Design the user interface
• ISO 9241 parts 12 to 17 Ergonomic principles
• Use user-centred methods in development
• ISO 13407 Human centred design processes
• Establish organisational processes
• ISO TR 18529 Human-centred lifecycle process
  descriptions

15
What can be specified and measured?
requirements
product
requirements
evaluation
Quality in usemetrics ISO/IEC 9126-4
User requirements
Deliver
Test user goal
Externalusability metrics ISO/IEC 9126-2
Functional requirements
Prototype
Test user function
Internalusability metrics ISO/IEC 9126-3
Detailed specification
Code
inspection
16
1. Design for quality in use

• Usability/quality in use measures
• Effectiveness and efficiency
• MUSiC Performance Measurement Method
• http//www.usability.serco.com/publs.html
• Satisfaction
• SUMI questionnaire
• www.ucc.ie/hfrg/questionnaires/sumi
• Reporting usability
• Common industry format for usability test results
• includes intended context of use and actual
  context of evaluation
• www.nist.gov/iusr

17
ISO/IEC CD 25030 Software quality requirements
and evaluation Quality requirements

• Requirements can be categorised as
• functional requirements
• quality requirements
• other non-functional requirements
• Software quality requirements are categorised
  using the ISO/IEC 9126-1 software quality model
  into requirements for
• quality in use (the effectiveness, productivity,
  safety and satisfaction with the software applied
  in its context of use)
• external quality (the behaviour of the finished
  software product)
• internal quality (static properties during
  development, such as adherence to coding
  standards, and meeting size and complexity
  thresholds)

18
Requirements life cycle model
Formalisation
Definition
Evaluation
Functional requirements
Quality requirements
Developed system
Quality in use
Measurable requirements
Measured quality
Stated requirements
External quality
Stake-
holder
Measured quality
Measurable requirements
Stated requirements
needs
Internal quality
Measurable requirements
Stated requirements
Measured quality
Other non-functional requirements
19
Quality requirements specification

• Define quality requirements
• Identify stakeholders
• end users, developers, producers, trainers,
  maintainers, disposers, acquirer, supplier
  organisations and regulatory bodies
• Elicit requirements from stakeholders
• Quality in use, External and Internal Quality
• Analyse the set of requirements
• Resolve problems
• Confirm requirements
• Record requirements
• Formalise quality requirements
• Specify target values for measures
• Demonstrate traceability
• Maintain requirements

20
Elicit requirements from stakeholders

• Include the needs, wants, desires, expectations
  and perceived constraints
• Identify requirements
• Quality in use
• External
• Internal
• Identify constraints
• Define scenarios of use

21
Quality in use

• Extent to which a product used by specified users
  meets their needs to achieve specified goals with
  effectiveness, productivity and satisfaction in
  specified contexts of use
• Effectiveness accuracy and completeness
• Productivity the resources expended by users in
  relation to the effectiveness achieved

22
Why is quality in use important?

• It is about meeting business and user needs
• Effectiveness success in achieving goals
• Productivity staff time
• Satisfaction willingness to use the system

23
Quality in use context of Use

• The usability of a product is affected not only
  by the features of the product itself but also by
  its Context of Use
• Context is the characteristics of
• the users of the product
• the tasks they carry out
• the technical, organisational and physical
  environmentin which the product is used
• the date and time when the product is being used

24
Example A bank ATM

• Context description
• The users
• The user characteristics
• The tasks users perform
• The technical environment(hardware and software
  tosupport system)
• The physical environment
• The social or organisational environment

25
New design features to meet contextual needs

• Recess for wheelchair access.
• Speech output for visuallyimpaired users.
• Customisation features forrapid access.
• Finger print for identification.
• Visor appears during sunny weather.
• Buttons light during darkness.
• Alarm button for security alert.

26
Efficiency mean transaction times
27
User satisfaction SUMI questionnaire
60
50
40
30
New System
20
Old System
10
0
Global
Efficiency
Affect
Helpfulness
Control
Learnability
28
The danger of not setting quality in use
requirements

• New software for issuing UK passports
• Installed in passport issuing offices
• Took operators twice as long
• Caused delays of up to 3 months in obtaining a
  passport
• Huge cost of additional clerical staff

29
Example of quality in use requirements

• User performance
• all data entry clerks will be able to complete
  the task with at least 95 accuracy in under 10
  minutes
• User satisfaction
• the mean score on the SUMI scale will be greater
  than 50 More information on quality in use
  requirements

30
Common Industry Format usability tests

• NIST initiative
• National Institute of Standards and Technology
• Objective raise the profile of usability in
  procurement
• Suppliers provide standard test reports to
  purchasers
• Suppliers include
• IBM, Microsoft, HP, Sun, Kodak, Oracle, Compaq
• Purchasers include
• Boeing, Northwest Mutual Life, State Farm
  Insurance, Fidelity
• Reports provided in confidence
• Could permit comparisons
• Agreed as US standard ANSI/NCITS 354, to be
  submitted to ISO
• http//www.nist.gov/iusr

31
Report format - technical

• Test objectives
• Participants
• Total number and user groups
• Key characteristics and capabilities expected
• How selected with the key characteristics and
  capabilities?
• Context of evaluation
• Task scenarios and task data used
• Any performance criteria used
• Physical and computer environment

32
Report format - results

• Effectiveness
• completely and correctly achieving their goals
• Separate data where assistance was given
• Efficiency
• Mean time taken to complete the task
• Standard deviation (and standard error)
• Satisfaction
• Optional own scales or standard questionnaire

33
ISO 20282 Usability of everyday products Part
1 Universal user profile

• Design should take account of
• Strength and Biomechanical Abilities
• Handedness
• Body Dimensions.
• Visual Abilities.
• Auditory Abilities
• Cognitive Abilities
• Language and literacy
• Culture
• Age
• Gender

34
ISO WD 20282-2 Test method

• A summative test method for measuring usability
  based on
• ISO 9241-11 (Guidance on usability)
• ISO 9126-4 (Quality in use metrics)
• Common Industry Format for usability test reports
• Identify intended context of use and user groups
• Measures
• Success rate (ease of operation)
• Task time (efficiency)
• Satisfaction scale (being developed)

35
Approaches to achieving usability
product quality
quality in use
organisational capability
process quality
development process
effect of the product
life cycle processes
product
usability in context
user centred process
interface and interaction
usability capability
ISO 9241-11 ISO 14598-1 ISO/IEC 9126-1 ISO/IEC
9126-4 ISO 20282-2
ISO 13407
ISO 9241 parts 10, 12-17 ISO/IEC 9126-2/3
ISOTR 18529
36
2. Design the user interfaceISO 9241 Ergonomic
requirements for office work with visual display
terminals

• Parts 10 to 17 give guidance on software
• 10 Dialogue principles
• 11 Guidance on usability
• 12 Presentation of information
• 13 User guidance
• 14 Menu dialogues
• 15 Command Language dialogues
• 16 Direct Manipulation dialogues
• 17 Form-filling dialogues

37
Approaches to achieving usability
process quality
quality in use
organisational capability
product quality
development process
effect of the product
life cycle processes
product
usability in context
user centred process
interface and interaction
usability capability
ISO 9241-11 ISO 14598-1 ISO/IEC 9126-1 ISO/IEC
9126-4 ISO 20282-2
ISO 13407
ISO 9241 parts 10, 12-17 ISO/IEC 9126-2/3
ISOTR 18529
38
3. ISO 13407 Human centred design process for
interactive systems
1. Plan the human centred process
Meets requirements
2. Specify the context of use
5. Evaluate designs against user requirements
3. Specify user and organisational requirements
4. Produce design solutions
39
ISO 13407 Scope

• Computer-based interactive systems
• The detailed recommendations are applicable to
  computer hardware and software
• Can apply the same principles to the design of
  any system for use by humans
• A pen
• A chair
• A meeting room
• A conference

40
TRUMP Trial Usability Maturity Process

• EU-funded trial application of user-centred
  design methods developed in previous research
  projects (INUSE and RESPECT)
• Serco apply the methods
• Lloyds Register Usability Maturity Assessment
• Inland Revenue/EDS - IT for 60,000 staff
• RAD methodology
• Israel Aircraft Industries - aerospace systems
• traditional methodology

41
Typical approach to usability
feasibility
requirements
design
implement
release
user and task analysis
usability test
Usability group
prototyping
42
TRUMP approach ubiquitous usability
PlanProcess
SpecifyContext of Use
DesignSolutions
SpecifyRequirements
Evaluate against Requirements
System lifecycle
feasibility
requirements
design
implement
release
4. Usability requirements 5. Evaluate existing
system
10. Collect feedback
6. Prototyping 7. Style guide
8. Evaluation 9. Usability testing
www.usability.serco.com/trump/ucdmethods
43
Conventional design processHow do we incorporate
user centred design?
requirements
evaluation
product
User requirements
Deliver
Functional requirements
Prototype
Function test
Detailed specification
Code
44
5. Evaluate existing system
1. Plans
Context used for evaluation
2. Intended context of use
3. Intended scenarios of use
Scenarios used for evaluation
requirements
evaluation
product
4. Quality in use requirements
User requirements
6.Mock up
Use in context
4. External usability requirements
Function test
Functional requirements
6. Paper prototype
Prototype
6. Paper specification
4. Internal usability requirements
Detailed specification
Code
45
Methods feasibility

• 1. Stakeholder meeting
• A half-day meeting to identify and agree on the
  role of usability, broadly identifying the
  intended context of use and usability goals, and
  how these relate to the business objectives and
  success criteria for the system.
• 2. Context of use
• A half-day workshop to collect and agree
  detailed information about the intended users,
  their tasks, and the technical and environmental
  constraints.
• 3. Scenarios of use
• A half day workshop to document examples of how
  users are expected carry out key tasks in a
  specified contexts, to provide an input to design
  and a basis for subsequent usability testing.

46
Methods requirements

• 4. Evaluate an existing system
• Evaluate an earlier version or competitor system
  to identify usability problems and obtain
  measures of usability as an input to usability
  requirements.
• 5. Usability requirements
• A half-day workshop to establish requirements
  for effectiveness, efficiency and satisfaction
  for the user groups and tasks identified in the
  context of use analysis and in the scenarios.

47
Methods design

• 6. Paper prototyping
• Evaluation by users of quick low fidelity
  prototypes (using paper or other materials) to
  clarify requirements and enable draft interaction
  designs and screen designs to be rapidly
  simulated and tested.
• 7. Style guide
• Identify, document and adhere to industry,
  corporate or project conventions for screen and
  page design.

48
Methods evaluation

• 8. Evaluation of machine prototypes
• Informal usability testing with 3-5
  representative users carrying out key tasks to
  provide rapid feedback on the usability of
  prototypes.
• 9. Usability testing
• Formal usability testing with 8 representatives
  of a user group carrying out key tasks to
  identify any remaining usability problems and
  evaluate whether usability objectives have been
  achieved.
• 10. Collect feedback from users
• Collect information from sources such as
  usability surveys, help lines and support
  services to identify any problems that should be
  fixed in future versions.

49
Cost benefits at IAI

• Development cost/benefits
• A all the methods used (except Style Guide)
  resulted in savings in development costs of
  between 5K and 70K for each method, with a
  total saving of 330K.
• The cost of using the methods was only 22K,
  giving a cost-benefit ration of 115.
• Sales cost/benefits
• Increased sales because customers would be more
  likely to buy the product were estimated to be
  400K
• Support cost/benefits
• Reduced costs of training and support were
  estimated at 50K.
• Overall cost benefits
• The overall costs of the maturity assessments and
  use of methods was 27K. The total estimated
  savings and increased sales is 780K, giving a
  cost-benefit ration of 129.

50
Trial Application at IAI Bottom Line

• Positive Feedback from Participants
• A Definite Improvement in the Development Process
  
• Very Cost Effective and Low Cost
• Mostly Intuitive however tailoring sometimes
  required
• Expert Guidance needed in few techniques
• LAHAV decided to incorporate TRUMP techniques in
  its standard development process

51
IAI Conclusions

• Low cost simple methods for improving systems
  usability are here
• The methods can be easily integrated into any
  software development process
• The responsibility for quality in use should be
  transferred to development organization
• The Methods and supporting means are available on
  the Web
• No more Excuses

52
Approaches to achieving usability
organisational capability
quality in use
process quality
product quality
development process
effect of the product
life cycle processes
product
usability in context
user centred process
interface and interaction
usability capability
ISO 9241-11 ISO 14598-1 ISO/IEC 9126-1 ISO/IEC
9126-4 ISO 20282-2
ISO 13407
ISO 9241 parts 10, 12-17 ISO/IEC 9126-2/3
ISOTR 18529
53
4. Establish usability capability ISO TR 18529
Human-centred lifecycle process descriptions

• HCD.1 Ensure HCD content in system strategy
• HCD.2 Plan and manage the HCD process
• HCD.3 Specify the user and organisational
  requirements
• HCD.4 Understand and specify the context of use
• HCD.5 Produce design solutions
• HCD.6 Evaluate designs against requirements
• HCD.7 Introduce and operate the system

54
Process improvement overview

• Identify needs for usability process improvement
• Use the usability maturity model in ISO TR 18529
  to assess the current capability of each
  organisation
• Make the identified improvements to the software
  development processes
• Introduce simple user-based methods implementing
  ISO 13407
• Reassess the usability capability the
  organisation
• Assess the extent of the improvement
• Identify the cost-benefits of the improvements
• Integrate the methods into the documented
  processes

55
Usability Maturity Model HCD.2 Plan and manage
the HCD process

• Consult stakeholders.
• Identify and plan user involvement.
• Select human-centred methods and techniques.
• Ensure a human-centred approach within the
  project team.
• Plan human-centred design process.
• Manage human-centred process.
• Champion human-centred activities.
• Provide support for human-centred design

56
Usability Maturity Model HCD.3 Specify the user
and organisational requirements

• Clarify and document the system goals.
• Define stakeholders.
• Assess risk to stakeholders.
• Define the system.
• Generate the user and organisational
  requirements.
• Set usability objectives.

57
Usability Maturity Model HCD.4 Understand and
specify the context of use

• Identify and document
• users tasks.
• significant user attributes.
• organisational environment.
• technical environment.
• physical environment.

58
Usability Maturity Model HCD.5 Produce design
solutions

• Allocate functions
• Produce composite task model
• Produce system design
• Use existing knowledge to develop design
  solutions
• Specify system
• Develop prototypes
• Provide user training
• Provide user support

59
Usability Maturity Model HCD.6 Evaluate designs
against requirements

• Specify and validate context of evaluation.
• Evaluate early prototypes in order to define the
  requirements for the system.
• Evaluate prototypes in order to improve the
  design.
• Evaluate the system in order to check that the
  system requirements have been met.
• Evaluate the system in order to check that the
  required practice has been followed.
• Evaluate the system in use in order to ensure
  that it continues to meet organisational and user
  needs.

60

• United Kingdoms tax collection department
• Employ over 60,000 staff in more than 600 local
  offices
• Partnership with EDS for software development now
  running for more than 5 years
• JAD Joint Application Development
• RAD Rapid Application Development

61
Usability capability assessment

• Formal CMM/SPICE style assessment by Lloyds
  Register
• Interviewed 13 stakeholders in twelve 3-hour
  sessions over one week.
• A conventional software process assessment
  procedure based on SPICE (ISO 15504) was used.
• Produced a detailed profile and rich information
  about where improvements would be beneficial.
• At a feedback meeting on the last day, it
  provided the basis for an agreed set of
  improvement activities
• A second assessment a year later showed
  significant improvements

62
Inland Revenue/EDS Conclusions

• Significant improvement in all areas of usability
  maturity
• IR/EDS now strong in the core areas of context,
  requirements, design and evaluation
• Evaluation activity rated as outstanding
• The methods are very cost effective and will be
  incorporated into the documented development
  process
• A step change in the awareness and practice of
  human centred design within IR/EDS.

63
IAI LAHAV Background

• Expertise in Military Aircraft Avionics Upgrade
  Programs
• Customers Worldwide
• User needs addressed by a group of Pilots
• Trial Application PC-based Mission Planning
  system

64
Usability capability assessment workshop

• Process improvement workshop
• One-day workshop
• Review each process and activity in ISO TR 18529
  in detail
• Score each process as
• Not, Partly, Largely or Fully performed
• Note whether the process is managed
• a documented part of normal procedures
• Establish a target level for each process on the
  same scales
• A second assessment a year later showed
  significant improvements

65
Trial Application at IAI Bottom Line

• Positive Feedback from Participants
• A Definite Improvement in the Development Process
  
• Very Cost Effective and Low Cost methods
• LAHAV decided to incorporate TRUMP techniques in
  its standard development process

66
Process improvement conclusions

• The usability maturity model was a valuable tool
  for identifying needs for process improvement
• Inland Revenuesummative assessment requiring
  three person weeks effort
• IAI a simpler formative one-day assessment
• A very promising way to make major process
  improvements
• www.usability.serco.com/trump

67
ISO 13407 Conclusions

• ISO 13407 and ISO TR 18529 provide an
  internationally-agreed description of good
  practice in user-centred design
• The content can be used to specify or evaluate
• How usability should be incorporated into the
  development process
• Whether a project has applied user centred design
• Organisational capability in user-centred design
• Basis for process improvement
• Appropriate content for education and training
  courses in user centred design
• Necessary areas of professional competence
  professional accreditation

68

• www.usabilitynet.org