DL Classification

1
DL Classification Evaluation
A Publishers View of the Work of the DELOS
Evaluation Forum
2
DELOS Network of Excellence

• Digital Library Evaluation Forum
• DL Test Suite Work Group
• Norbert Fuhr, U. Darmstadt
• Preben Hansen, Swedish Inst. Comp. Sci.
• Sarantos Kapidakis, Ionian U.
• Laszlo Kovacs, MTA SZTAKI
• Michael Mabe, Elsevier Science/City University
• Andras Micsik, MTA SZTAKI
• Pasquale Savino, IEI-CNR
• Ingeborg S?lvberg, Norwegian U. Sci. Techn.

3
Overview

• Applying Strategic Marketing
• Market Segmentation and Classification
• DELOS Evaluation Forum
• market-based classification
• metrics and evaluation
• development of the Meta-Library
• Helping shape the future agenda

4
Strategic Marketing DL Evaluation

• How can these possibly be related?
• Business is customer-centred
• the customer is king
• First Commandment of Marketing
• WHO is buying WHAT and WHY

• Information is user-centred
• the user is king
• First Commandment of Info-Business

WHO is using WHAT and WHYWHO is buying WHAT
and WHY
5
Strategic Marketing Cycle
Segment Market
Decide on Benchmarks
Annual Planning Cycle
Evaluate Performance
Take Actions
Analyse Outcomes
6
Market segmentation

• Most important aspect of strategic marketing
• Market segments must be
• highly similar within
• distinct from other segments
• described in terms of use or purchase
• reachable
• Accuracy of segmentation affects success TQM and
  thus of the business

7
Market Segmentation
WHO is using WHAT and WHY

• WHOdemographic socioeconomicbrand
  loyaltyheavy/light userlifestyle

• WHAT( HOW)productplacepriceoutletservices

• WHYbenefitsattitudesperceptionspreferences

Customers dont buy products they seek to
acquire benefits
8
Publishing Example Information Pyramid
Research
Business-to-Business
Professional
Mass Market
9
DL Segmentation Classification
WHO is using WHAT, HOW and WHY

• WHO
• users
• demographics
• interests (subject)
• approaches

• WHAT
• data/collection
• description
• management

• HOW
• technology
• user techn.
• info. tools
• system techn.
• document techn.

• WHY
• uses (purpose)
• encounter type

WHO and WHY predetermine WHAT and HOW
10
Generalised Schema for a Digital Library
11
Market Segmentation and DL Metrics

• A market segment must be
• highly similar within
• distinct from other segments
• described in terms of use or purchase
• reachable
• A DL evaluation metric
• independent of other metrics
• relevant
• measurable

12
Proposed DL Classification and Metrics

• Classification
• Users/uses
• user/uses
• internal, general, education, professional,
  research
• domain (subject area)
• info encounter
• object seeking, browsing
• purpose
• consume, analyse, synthesize

• Metric
• Users/uses
• user/uses
• number
• distribution
• domain
• distribution
• info encounter
• distribution
• purpose
• distribution

13
Proposed DL Classification and Metrics

• Classification
• Data/Collection
• content
• none/partial/full
• audio, video, text
• 2D/3D
• meta-content
• biblio
• indexing/thesaurus /classification
• citation
• management
• rights, work flow, user management, maintenance

• Metric
• Data/Collection
• content
• diversity, age, size, quality (white/grey
  literature)
• meta-content
• media
• level of detail
• management
• doc. age, growth rate, immediacy, completeness,
  maintenance intervals

14
Proposed DL Classification and Metrics

• Classification
• technology
• user technology
• doc. creation, disclosure, interface, browsing,
  search, printing, group/individual
• information access
• retrieval, navigation, filtering, extraction,
  text mining
• systems structure tech.
• repository, transport models
• document technology
• doc. model, format

• Metric
• technology
• user technology
• information access
• efficiency
• effectiveness
• systems structure tech.
• document technology

15
Evaluation of the Scheme

• Creation of a questionnaire from the
  classification scheme
• Questionnaire A
• What DLs are currently available that can be used
  as test collections
• Questionnaire B
• What test collections will be needed in the
  future
• methodology open web-based survey tool designed
  by SZTAKI Kovacs Micsik
• survey announced via mailing lists
• Outcome
• 3-4 response, 70 from research domain
• scheme appropriate but wording of questions
  difficult

16
Development of the MetaLibrary

• Extensible survey database
• Registers and classifies each submitted DL
  collection
• Allows identification of missing research areas
  or test collections
• Potentially could set the agenda for future
  research
• Analogy with standard marketing strategy

17
Market Extension
18
Conclusions Next Steps

• DL evaluation needs a broad and user-centred view
• new DL evaluation scheme
• a hierarchy of user-based metrics
• meta library of DL collections
• http//www.sztaki.hu/delos_wg21/metalib/
• extension of the meta library
• agenda setting for DL research
• my e mail m.mabe_at_elsevier.co.uk