Usability Evaluation Issues in Commercial and Research Systems - PowerPoint PPT Presentation


PPT – Usability Evaluation Issues in Commercial and Research Systems PowerPoint presentation | free to download - id: 1d7b9a-ZDc1Z


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Usability Evaluation Issues in Commercial and Research Systems


To make proper selection, one must know the range and properties of criteria available ... Wizard-of-Oz (early-middle) Field test (late) ... – PowerPoint PPT presentation

Number of Views:11
Avg rating:3.0/5.0
Slides: 13
Provided by: cars155
Learn more at:


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Usability Evaluation Issues in Commercial and Research Systems

Usability Evaluation Issues in Commercial and
Research Systems
  • Laila Dybkjær, Niels Ole Bernsen NISLab,
    University of Southern Denmark
  • Hans Dybkjær SpeechLogic, Prolog Development
    Center A/S
  • ASIDE 2005-11-10 COST Workshop, Ålborg University
  • Slides available at www

Its all about design usable design ...
Users, prompts, modalities, media, ...
Usability in academia and industry
  • Academia New and challenging?
  • Focus on advanced systems and new knowledge
  • Industry Cost? ROI? Market? Customers?
  • Focus on state-of-the-art and functionality
  • But they have much to learn from each other
  • Lots of research results to streamline for
  • Lots of simple systems questions open to
  • Gap EU research visions vs. industrial reality

What can they learn from each other?
Three examples
System Traffic FAQ NICE HCA
Task / domain Road traffic information Holiday allowance information H. C. Andersens life and fairytales edutainment
Purpose Commercial Commercial Gov. support Research
I/O Speech Speech Speech, gesture, 3D graphics
Language Da 50 words Da 500 words En 2000 words
Target Car drivers All employees Children 10-18
Who built it PDC PDC, NISLab NISLab and 4 other EU partners
Wide range in purpose and complexity
Cost and complexity
Months Traffic 400 hours Simple FAQ 4000
hours Complex NICE HCA 40000 hours Very complex
F 2005
F2 2002
P2 2005
Academic focus Prototypes Industry focus
Final systems
Usability evaluation criteria
  • Difficult to select right criteria for given
  • Is purpose to compare, to investigate, or to
    define contract?
  • To make proper selection, one must know the range
    and properties of criteria available
  • Many usability criteria vaguely defined
  • E.g. adequacy or sufficiency of
  • Quantifiability often missing
  • Subjective or qualitative evaluation
  • New system types require new criteria
  • Must be clearly defined and operationalised

Standards may emerge, but new needs keep coming
Core usability evaluation criteria
System Criteria Criteria
Traffic Interaction problems Correctness Task and domain completeness
FAQ Interaction problems Correctness Transaction success Task and domain completeness
NICE HCA Conversation success Naturalness Reasoning capabilities Ease of use Error handling Scope of user modelling Entertainment and education value User satisfaction
Clearly different focus in academia and industry
Usability evaluation methods
  • Which one to choose depends e.g. on
  • Evaluation purpose
  • Resources (who, time, money)
  • Stage of development process
  • Examples of methods
  • Walkthrough (early)
  • Focus groups (early, but ok any time)
  • Wizard-of-Oz (early-middle)
  • Field test (late)
  • Heuristic evaluation (best early but also ok
  • User interviews and questionnaires (any time)

Many current practice methods!
Usability evaluation methods
System Methods
Traffic Walkthrough (using DialogDesigner) Semi-formal WOZ (using DialogDesigner) In-house scenario-based test Expert evaluation of domain information
FAQ Walkthrough (manually) In-house and external scenario-based test Questionnaire on web Monitored scenario-based lab tests Field data analysis Expert evaluation of domain information
NICE HCA WOZ in schools and in museum Lab-test of first and second prototype Post-lab-test interviews
Industrial systems need broad range of methods
Data and analysis
System Data Analysis
Traffic Logfiles Problem identification via observation and feedback Log-based analysis of problems
FAQ Logfiles trans-criptions trans-action annotations questionnaires Problem identification via ob-servation, feedback from users and domain experts, and ana-lysis of transcribed dialogues
NICE HCA Logfiles trans-criptions topic annotation Eng-lish evaluation interviews Analysis of WOZ and lab test data for design input analysis of lab tests and interviews to get users opinion and develop new criteria
Research ? more data and analysis needed
Electronic model ? IT tools possible
  • Sketch, prompt design and recording,
    walk-through, WOZ, document, test, formal
    properties (coherence, well-formedness, ...),
    ..., you name it

A lot of knowledge and theory can be made
Academia and industry do meet ...
  • Industrial actions and challenges
  • Optimise existing processes
  • Automation (transcription support, annotation,
  • Use known results and theories
  • Unknown effects of new technology
  • Academic challenges
  • Highly sophisticated technology
  • New factors to analyse, define, and measure
  • On-line adaptivity to users skills, expertise,
  • Investigate troubles with simple systems

... even though they are also different beasts