Unbundling the ILS @ NCSU: implementation of an e-commerce search solution

1
Unbundling the ILS _at_ NCSU implementation of an
e-commerce search solution

• Emily Lynema
• Andrew K. Pace
• North Carolina State University Libraries
• LITA 2006 National Forum

2
Or better yet

• Endeca implementing a faceted search solution
  for the library catalog

3
Agenda

• The Context
• Next Gen Search Tools vs. OPAC Problems
• Local Implementation
• Why, What and How?
• Challenges Encountered
• Assessment
• Usage Statistics
• Usability Testing
• The Future

4
The Context
5
Online Catalogs
"Most integrated library systems, as they are
currently configured and used, should be removed
from public view." - Roy Tennant, CDL
6
Next gen search tools

• Proving that its possible to improve the search
  experience beyond the functionality that
  traditional OPACs have supported.

7
NextGen Library Search Tools

• WorldCat.org (Beta)
• RedLightGreen (RLG), subsumed by WorldCat
• OCLC Fictionfinder
• Vivisimo clustered search (Serials Solutions, Ex
  Libris)
• Aquabrowser visual context
• Endeca ProFind
• Innovative Interfaces OPAC Pro and Encore
• Ex Libris Primo
• Polaris, AJAX-Enabled OPAC
• SirsiDynix Enterprise Portal System, FAST
• Talis, et al?Web Services
• EBSCO Research Databases
• Georgia PINES, Koha, and the Library 2.0
  Bandwagon
• And of course the entire commercial web

8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
Existing catalogs are hard to use

• Known item searching works pretty well
  (sometimes), but
• Lots of topical searches and poor subject access
• keyword gives too many or too few results leads
  to general distrust among users
• authority searching is under-utilized and
  misunderstood
• Relevance system sort order
• Impossible to browse the collection
• Unforgiving on spelling errors, stemming
• Response time doesnt meet expectations of
  web-savvy users

15
Valuable metadata is buried

• Subject headings are not leveraged in keyword
  searching
• they should be browsed or linked from, not
  searched
• Data from the item record is not leveraged
• should be able to easily filter based on users
  changing requirements using item type, location,
  circulation status, popularity

16
Whats the big picture?

• Improve the quality of the library catalog user
  experience
• Exploit our existing authority infrastructure
  (aka make MARC data work harder)
• Build a more flexible catalog tool that can be
  integrated with discovery tools of the future.

17
What is Endeca?

• Software company based in Cambridge, MA
• Search and information access technology provider
  for a number of major e-commerce websites
• Developers of the Endeca Information Access
  Platform

18
Why Endeca?

• Customized relevance ranking of results
• Better subject access by leveraging available
  metadata (including item level data!) through
  facets
• Improved response time
• Enhanced natural language searching through spell
  correction, etc.
• Browse

19
Local Implementation
20
Demo
21
Relevance ranking

• Based on locally customizable algorithm
• Most relevant query as entered
• For multi-term searches phrase match
• Field match
• title match more relevant than notes match
• Other factors
• number of fields matched
• weighted frequency (tf/idf)
• static ordering (publication date, circulation
  stats)

22
Faceted browse

• Combine search and browse in single interface
  (Guided Navigation)
• Filter results across multiple facets
• Remove facets in any order

23
Facet refinements

• Availability
• Author
• Library
• Format
• Language
• New

• LC Classification
• Subject Topic
• Subject Genre
• Subject Region
• Subject Era

24
True browse

• Regain ability to browse catalog without entering
  any search terms

25
Added search tools

• Automatic spell correction
• Did you mean suggestions
• Automatic stemming

26
The nitty gritty

• Endeca co-exists with SirsiDynix Unicorn ILS and
  Web2 online catalog
• Endeca handles keyword search
• Web2 handles authority search and detail page
  display
• Endeca indexes MARC records exported nightly from
  Unicorn
• Endeca discovery portion of the ILS

27
Technical overview
Information Access Platform
NCSU exports and reformats
Data Foundry
MDEX Engine
Parse text files
Raw MARC data
Indices
Flat text files
HTTP
HTTP
NCSU Web Application
28
Technical overview
Offline - Nightly
NCSU exports and reformats
Data Foundry
MDEX Engine
Parse text files
Raw MARC data
Indices
Flat text files
HTTP
HTTP
NCSU Web Application
29
Technical overview
Always Online
NCSU exports and reformats
Data Foundry
MDEX Engine
Parse text files
Raw MARC data
Indices
Flat text files
HTTP
HTTP
NCSU Web Application
30
Implementation team

• Seven member team
• 5 IT/DLI staff, 1 cataloging librarian, 1
  reference librarian
• As a team functional requirements, metadata,
  interface issues (total of 40-60 hours)
• Java-trained IT librarian (40 hrs/wk for 14
  weeks)
• IT project manager (10 hours/wk for 20 weeks)
• Timeline
• License / negotiation Spring 2005
• Software acquisition Summer 2005
• Implementation Aug 2005 to Jan 2006

31
Local decision points

• Identifying appropriate facets

32
(No Transcript)
33
Local decision points

• Identifying appropriate facets
• Designing the user interface

34
1. Availability
2. Library of Congress Classification

1. Subject Topic
2. Subject Genre
3. Format
4. Library
5. Subject Region
6. Subject Era
7. Language
8. Author

35
Local decision points

• Identifying appropriate facets
• Designing the user interface
• Integrating authority searching and Endeca
  keyword searching

36
Pre-Endeca Catalog Search

• 6 search tabs
• 14 radio buttons
• 1-4 drop down boxes
• Title begins with search default

37
Post-Endeca catalog search

• 3 search tabs
• No radio buttons
• 2 search boxes
• Keyword search default

Endeca keyword
Web2 authority
38
Local decision points

• Identifying appropriate facets
• Designing the user interface
• Integrating authority searching and Endeca
  keyword searching
• Creating the relevance ranking algorithm for each
  field index

39
Special challenges encountered

• ILS data with MARC-8 encoding gt Text data with
  UTF-8 encoding
• Data consistency between ILS and Endeca catalog
  indexes (updates!)
• Data issues revealed by exposing metadata (ex
  subject headings) in facets

40
Assessment
41
Usage statistics
42
Usage statistics
43
Usage statistics
44
Usage statistics
45
Usability testing

• 10 undergraduate students
• 5 with new Endeca-based interface
• 5 with old catalog interface
• Identical searching tasks
• Data collected
• Task difficulty/failure
• Task duration

46
Usability testing
47
Usability testing
48
Usability testing

• For students, relevance ranking is key.
• March 2006 13 continue to page 2
• Faceted browsing is intuitive, even for students
  who dont use it.
• Beware of library jargon
• keyword anywhere, keyword in subject
• User behavior is influenced by previous
  experience.

49
Relevance

• Are search results in Endeca more likely to be
  relevant to a users query than search results in
  old OPAC?
• 100 topical user searches from 1 month in Fall
  2005
• How many of top 5 results relevant?
• 40 relevant in Web2 OPAC 31 no hits
• 68 relevant in Endeca catalog 12 no hits

50
The Future
51
Future directions

• Experiment with FRBR search/display through
  partnership with OCLC.
• Update circulation status throughout the day.
• Integrate catalog w/other tools through web
  services
• OpenSearch, RSS
• Enrich catalog through external web services
• book jackets, reviews, etc. Amazon/OCLC
• Build modular shopping cart functionality.
• Use Endeca to index local collections.

52
From the Calhoun report

• "If one accepts the premise that library
  collections have value, then library leaders must
  move swiftly to establish the catalog within the
  framework of online information discovery systems
  of all kinds. Because it is catalog data that has
  made collections accessible over time, to fail to
  define a strategic future for library catalogs
  places in jeopardy the legacy of the world's
  library collections themselves. For this reason,
  the option of rejecting library catalogs is not
  considered in this report."

53
So what? Its still just a catalog
Serials
AI / FT DBs
Metasearch
ERM Systems
GS
Digital Repositories
Guided Navigation
Catalog
Web
IR
Legacy ILS
54
Strong to our finish

• Too often, we have an "eat your spinach" message
  about the library come to the library, it is
  good for you.
• Lorcan Dempsey, OCLC

55
Moving in a new direction
OLD SEARCH MODEL
NEW SEARCH MODEL
56
Things to read

• Rethinking how we provide bibliographic services
  for the University of California by the
  Bibliographic Services Task Force
• http//libraries.universityofcalifornia.edu/sopag
  /BSTF/Final.pdf
• The Changing nature of the catalog and its
  integration with other discovery tools by Karen
  Calhoun
• http//www.loc.gov/catdir/calhoun-report-final.pd
  f
• The Changing nature of the catalog and its
  integration with other discovery tools A
  Critical review by Thomas Mann
• http//www.guild2910.org/AFSCMECalhounReviewREV.p
  df
• A Next Generation Catalog, Eric Morgan
• http//dewey.library.nd.edu/morgan/ngc/
• Metadata Research Center, SILS
• http//ils.unc.edu/mrc/
• University of Rochester eXtensible Catalog
• http//www.extensiblecatalog.info/
• Toward a 21st Century Catalog, ITAL, Sept. 2006,
  Antelman, Lynema, and Pace
• http//www.lib.ncsu.edu/endeca/publications/antel
  man_lynema_pace.pdf

57
Thanks

• NCSU project site
• http//www.lib.ncsu.edu/endeca
• Andrew K. Pace
• Head, Information Technology
• andrew_pace_at_ncsu.edu
• Emily Lynema
• Systems Librarian for Digital Projects
• emily_lynema_at_ncsu.edu