Proposal for a Preliminary Evaluation Study of the JerseyCat ILL System

1
Proposal for a Preliminary Evaluation Study of
the JerseyCat ILL System
2
JerseyCat

• no charge to libraries for the service.
• includes a union catalog of holdings from the
  small to medium libraries that are under 100,000
  volumes.
• system also provides access to periodical titles
  through the NJ Union List of Serials.

3
JerseyCat

• uses the NISO Z39.50 protocol to establish a
  virtual union catalog from the electronic
  catalogs of member libraries, and also uses
  Z39.50 to search the catalogs of 50 other
  participating libraries
• system has been operational since July 2002
• 700 participating libraries, and over 100,000
  items were requested in 2004

4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
Relevant Research

• Federated Searching
• Use and evaluation of OPACs
• IR Evaluation

10
Federated Search

• one common feature is a single search access to
  multiple resources
• major anticipated benefit of federated searching
  is usability
• The search interface can provide a single point
  of access to the librarys catalog, subscription
  databases, and selected Internet resources.
• single interface would allow new or occasional
  users to quickly search a variety of quality
  resources, without the need to know the specific
  details of the resources.

11
Federated Search

• There are also some problems with federated
  searching
• the lack of relevancy ranking for results
• limits on the ability to integrate database
  products from different vendors
• many federated search interfaces do not provide
  de-duplication of records

12
Federated Search

• Similarity of most OPACs make them ideal for a
  federated search engine
• Underlying structure provided by MARC records
• Uniform access points author, title, subject
  available for all data representations

13
Matthews, Lawrence, and Ferguson (1983)

• national survey of OPAC users
• measured users perceptions of success and
  satisfaction
• found that positive perceptions were related to
  frequency of use
• demographic factors such as age, sex, and
  academic status were found to have only an
  indirect effect on perceived success and
  satisfaction

14
Tolle (1985)

• monitored system performance is to discover
• the extent to which systems are used
• user patterns while conducting an information
  search
• used transaction logs to capture system activity
• used these logs to create a taxonomy of user
  commands and system responses

15
Figure 4. Transition Probability MatrixPrimary
Mapping 3rd Order Context Validated
LC(from Tolle (1985) p. 200)
16
Figure 5. State Transition Diagram from ERR
StatePrimary Mapping 1st Order LC(from
Tolle (1985) p. 201)
17
Borgman (1985, 1986, 1996)

• users have problems with
• the mechanical aspects of searching
• the conceptual aspects of searching
• some systems are easier to use than others
• have found it difficult to isolate the variables
  responsible for the differences

18
Borgman (1985, 1986, 1996)

• people have problems searching both online
  catalogs and other types of retrieval systems
• the problems may be related to individual
  characteristics and background or to design
  features
• identification of the existence of problems did
  not automatically lead to identification of the
  sources of the problems

19
Borgman (1985, 1986, 1996)

• OPACs are designed to answer questions
• in the short-term, improved training and
  documentation that is based on information-seeking
  behavior will help make OPACs easier to use
• OPAC use of library patrons (novice users) is
  well-studied, but there has been little study of
  OPAC use by librarians and skilled users

20
Other Research Findings

• a users goal is an important component of the IR
  interaction
• categorization of goals can form the basis for
  determining appropriate retrieval strategies and
  ranking algorithms which might improve the
  information retrieval process for users

21
Other Research Findings

• operational definition of relevance is whether
  the users of the systems are meeting with success
  in their information-retrieval activities
• task can be used as the basis for a
  non-interactive evaluation of an IR system

22
Research Proposal

• RQ1 How efficient is JerseyCat as an IR system?
• RQ2 Are user tasks an appropriate way to study
  user/system interaction?
• RQ3a What variables of system/user interaction
  can be identified from an examination of the
  system logfiles?

23
Research Proposal

• RQ3b Which of these variables provide unique
  information for studying system/user interaction?
• RQ4 Is there a relationship between user
  perceptions of system effectiveness and the
  measures of effectiveness derived from the
  quantitative analysis?

24
Research Proposal

• examine JerseyCat logfiles
• the logfile data will be examined to identify
  relevance variables, session variables, search
  variables, display variables, error variables,
  and help variables

25
Research Proposal

• Some of the possible variables include
• Number of queries per task
• Number of queries per session
• Number of requests per session
• Number of queries that return results
• Number of queries that return error messages or
  no results
• Usage patterns (how often, how long)
• Relationships between usage patterns and number
  of requests

26
Research Proposal

• data from the logfiles will be analyzed
  quantitatively following the models suggested by
  Tolle (1996) and by Cooper and Chen (2001)
• staff at this library will also be observed and
  interviewed to provide qualitative data to
  supplement the data from the logfiles

27
Research Proposal

• the success of this study depends on finding an
  effective way to categorize and analyze the
  logfiles
• however, if successful, this study will provide
  information to conduct a larger study that
  includes the other libraries that use this system

28
Research Proposal

• providing a large-scale examination of the use of
  a federated search interface
• extending the research on OPACs by examining how
  they are used by experts

29
Selected References

• Borgman, C.L. (1987, Spring/Summer). The study
  of user behavior on information retrieval
  systems. SIGCUE Outlook, 35-48.
• Borgman, C.L. (1986). Why are online catalogs
  hard to use? Lessons learned from information
  retrieval studies. Journal of the American
  Society for Information Science, 37(6), 387-400.
• Borgman, C.L. (1996). Why are online catalogs
  still hard to use? Journal of the American
  Society for Information Science, 47(7), 493-503.

30
References

• Boss, R.W. (2002). TechNotes Library web
  portals. Retrieved October 6, 2004, from
  http//www.ala.org/ala/pla/plapubs/technotes/libra
  rywebportals.htm.
• Hert, C.A. (1996). User goals on an online
  public access catalog. Journal of the American
  Society for Information Science and Technology,
  47, 504-518.

31
Selected References

• Library of Congress Portals Applications Issues
  Group (LCPAIG). (2003). List of portal
  applications functionalities for the Library of
  Congress First draft for public comment, July
  15, 2003. Retrieved October 6, 2004, from
  http//www.loc.gov/catdir/lcpaig/
  portalfunctionalitieslist4publiccomment1st7-22-03r
  evcomp.pdf.
• Matthews, J.R., Lawrence, G.S., Ferguson, D.K.
  (1983). Using Online Catalogs A Nationwide
  Survey. New York Neal-Schuman.

32
Selected References

• Reid, J. (2000). A task-oriented non-interactive
  evaluation methodology for information retrieval
  systems. Information Retrieval, 2, 115-129.
• Tolle, J.E. (1985). Performance measurement and
  evaluation of online information systems.
  Proceedings of the 1985 ACM Computer Science
  Conference Agenda for Computing Research The
  Challenge for Creativity, 1985 March 12-14,
  196-203.