Digital Library Evaluation

1
Digital Library Evaluation

• Approaches, Criteria, Methodologies

2
Definition of the term evaluation

• Means assessing performance or value of a
  system, process (technique, procedure ..),
  product or policy. (Saracevic, 1995)
• The process of identifying and collecting data
  about specific services or activities,
  establishing criteria by which their success can
  be assessed, and determining both the quality of
  the service or activity and the degree to which
  the service or activity accomplishes stated goals
  and objectives. ( Harter and Hert, 1997)

3
What is DL evaluation?

• Marchionini (2000) believes that evaluating
  digital libraries is a bit like judging the
  success of a marriage where much depends on how
  successful the partners are as individuals as
  well as the emergent conditions made possible by
  the union.

4
Evaluation as research vs. product evaluation ?

• It is important to distinguish evaluation as a
  research process from evaluation in the product
  testing and system efficiency sense.
• However, evaluation research considers the
  interactions of complex phenomena including
  people and reaches conclusions through chains of
  inferences supported by data rather than direct
  measurement.

5
Why is DL evaluation important?

• Digital libraries need to be evaluated as systems
  and as services to determine how useful, usable,
  and economical they are.
• Results of evaluation studies can provide
  strategic guidance for the design and deployment
  of future systems

6
Digital library evaluation literature

• Two general types
• Literature on concepts, metrics, models and
  approaches or methodologies
• Literature that reports evaluative studies and
  provides qualitative and/or quantitative data

7
Various approaches and contexts

• Digital libraries are multifaceted,
  multidimensional social, technical, cultural and
  institutional systems and as a result have been
  evaluated from a wide range of perspectives.
• Context of evaluation system-centred, human
  centred, usability-centred, ethnographic approach

8
Evaluation approaches

• Information retrieval (TREC)
• HCI usability testing
• Human-centred information needs and search
  behaviour, user interaction with information

9
Elements for the evaluation of DLs

• Digital collections
• Selection, gathering, holdings
• Organization, structure, storage
• Interpretation, representation, metadata
• Management
• Preservation and persistence
• Access
• Physical networks
• Interfaces, interaction
• Search, retrieval

• Services
• Availability
• Assistance and referral
• Use, users, communities
• Security, privacy, policies, legal aspects
• Staff and operations
• Costs and economics
• Integration, cooperation with other resources,
  libraries and services

Saracevic, 2000
10
Types of evaluation

• Formative evaluation begins at the initial
  stages of a development project to establish
  baselines on current operations, set goals, and
  determine desired outcomes.
• Summative evaluation takes place at the end of a
  project to determine if the intended goals were
  met.
• Iterative evaluation takes place throughout a
  project, beginning in the earliest design and
  development stages.
• Comparative evaluation requires standardized
  measures that can be compared across systems.
  Communities can identify and validate measures.

11
DL evaluation criteria (Saracevic, 2004)

• Usability content, format, process, overall
  satisfaction
• System features system performance such as
  response time, functionality, algorithm
  performance
• Usage usage patterns, usage statistics, use of
  materials, what uses for what purposes
• Ethnographic and other criteria anthropological,
  cultural, linguistic and impact implications and
  issues

12
Xie, H. I. (2006). Evaluation of digital
libraries Criteria and problems from users'
perspectives. Library Information Science
Research, 28(3), 433-452.
13
Methodology

• surveys, including direct questionnaires and
  online surveys
• focus groups and structured interviews
• observations
• case studies
• records and usage analysis
• documents, meeting, communication analysis
  (anthropology)
• economic analysis
• transaction log analyses
• think aloud, cognitive walk-through
• eye tracking

14
Examples of evaluation studies

• Perseus Digital library
• Usability inspection of NCSTRL
• Usability study of digital libraries ACM,
  IEEE-CS, NCSTRL,NDLTD

15
Perseus Digital libraryEvaluation goals

• Access to large volumes of multiple media source
• Freedom self-directed access and use,
• Collaboration among learners and teachers.

16
Preseus DL Evaluation objects and data
collection methods

• Evaluation objects
• Learners,
• Teachers,
• The technical system,
• The content
• Data collection methods
• Observations,
• Interviews,
• Document analysis,
• Learning analysis.

17
Usability inspection of NCSTRL

• Three evaluators one fully familiar, one new to
  NCSTRL and one expert in the general area of
  digital libraries
• Co-discovery technique
• Task-driven
• Simple and advanced search facilities, browsing,
  submit reports

Hartson et al. (2004)
18
NCSTRL Usability inspection results

• Problems with consistency the terms Group and
  archives were used interchangeably to
  institutions and collections belonging to
  institutions
• Problems with wording Discovery date, Submit
  to CoRR
• The term hit rather than more descriptive terms
  
• Tabs look like buttons but do not behave like
  buttons
• No user control of result display
• No query refinement facility
• The browsing option is only available by
  institution not author, title etc.

19
Usability study of digital libraries ACM,
IEEE-CS,NCSTRL,NDLTD

• Four digital libraries
• Four tasks for each library
• 48 computer science students
• Video camera, screen capture, search logs
• Evaluation criteria search features, search
  time, search errors

(Kengeri et al. 1999)
20
(No Transcript)
21
Selected results

• Some users commented that the search interfaces
  were too complicated and distracting.
• ACM digital library provided too many search
  options.
• Users wanted to search by keyword, author, or
  title, and where appropriate by year, journal,
  and volume.
• NDLTD online instructions were not easy to
  understand
• In NDLTD the difference between search and browse
  was not obvious

22
Eye tracking study

• Using eye tracking to evaluate alternative search
  results interfaces (Rele Duchowski, 2005)
• Eye movements and ocular behaviour
• fixations
• saccades
• pupil dilation
• scan paths

23
Eye trackers
24
(No Transcript)
25
(No Transcript)
26
p