Once a site is done, most people are not eager to hav

1
Putting Evidence-based Practice to Work
Usability Testing at Northwestern University
Library
Frank CervoneAUL for Information
TechnologyMonday October 29, 2007Internet
Librarian 2007
2
Overview

• The problem of web site design
• Using more formal methods
• Formal methods in practice
• Outcomes

3
The problems of web site design

• Web development/HCI is an intricate mix of
  technology and design
• While there are exceptions, the majority of
  librarians havent been trained in HCI
• Gaps in understanding the significant differences
  between the on-line and in-person experience
• Once a site is done, most people are not eager to
  have others suggest adjustments or changes,
  particularly to those things they know are
  right

Liberally adapted from Problems with in-house
website development by The IT Guyhttp//www.new
fangled.com/contentmgr/showdetails.php/id/7441
4
Web site design and organizational responsiveness

• The more an organization depends upon its publics
  for achieving its mission, the more it should
  employ dialogic features into its Web site design
• The challenge is to move the tasks associated
  with Web site design and maintenance away from
  being a "B-list" job to being an imperative for
  the survival of highly stakeholder-dependent
  organizations
• Kent, M. L., Taylor, M., and White, W. J. (2003).
  The relationship between Web site design and
  organizational responsiveness to stakeholders.
  Public Relations Review, 29(1), pp. 63-77.

5
Which leads to

• Evidence-based information practice
• an approach to information practice that
  promotes the collection, interpretation, and
  integration of valid, important, and applicable
  user-reported, librarian-observed, and
  research-derived evidence

Andrew BoothSenior Lecturer in Evidence-based
Healthcare InformationSchool of Health and
Related ResearchUniversity of Sheffield
6
What is evidence-based librarianship?

• Data provides the primary evidence for making
  decisions
• Not anecdotal stories
• Not common sense
• Evaluation occurs early in the process
• How is this different from what often happens
  now?
• Decisions are made based on
• Beliefs of what is needed
• Frequently, these beliefs are inherently biased
• Assumptions, anecdotal evidence, and preferences
• Evaluation, if it occurs, happens afterward

7
Evidence-based professional practice

• Derived from the evidence-based model of medicine
• Fundamental precepts
• Study the phenomenon
• Contrast results to other studies of the same or
  related phenomena
• Combine results to better understand the
  phenomenon at hand

8
Evidence-based practice process
Redefine the problem
Define the problem
Evaluate change
Find the evidence
Apply results of the evaluation
Evaluate the evidence
9
Defining the problem

• Very similar to concepts used in user-centered
  design
• Five points
• Yes, theres an acronym - SPICE
• SETTING
• Where is this being used what is the context?
• POPULATION
• Who are the (potential) users?
• INTERVENTION
• What is being done to/for them?
• COMPARISON
• What are the alternatives?
• EVALUATION
• What does success mean?

10
Levels of evidence

• Systematic review of a wide-body of rigorous
  studies
• Systematic review of a wide-body of less rigorous
  studies
• Randomized controlled trials
• Controlled-comparison studies
• Cohort studies
• Descriptive surveys
• Case studies
• Decision analysis
• Qualitative research (focus groups, ethnography,
  historic, Delphi techniques, interviews)
• Surveys, audits

Adapted from Eldredge, J. (2000). Evidence-based
librarianship An overview. Bulletin of the
Medical Library Association, 88(4). pp
289-302. Online at http//www.pubmedcentral.nih.go
v/articlerender.fcgi?artid35250
11
Which brings us to the Northwestern story

• First usability test in 2001
• Focused in on Electronic Resources
• A home-grown resource finder
• Investigation and questions based on anecdotal
  evidence with ER interface
• Learning experience
• Provided model for organization
• Whetted peoples appetite for usability testing

12
The second test Searching in NUcat

• Began summer 2002
• How are people using Voyager?
• Are there discernable patterns?
• Most importantly are there patterns of errors?
• Can we do something to fix that?
• What we did
• Looked at searching patterns in our Voyager
  system
• Used a more rigorous research approach
• Based on grounded theory
• Simple data mining techniques

13
And in the process
Searches by type
14
dispelled some myths
No hits within searches by type
15
Which lead to our tackling a bigger problem

• Existing site did not work especially well for
  the Library's various user communities
• Both students and faculty expressed frustration
  with the web site
• Ability to navigate the site
• Overall look and feel
• Problem areas determined by gathering information
  in focus groups, online surveys, interaction with
  librarians and staff
• Hours and locations
• Collections
• Electronic resources

16
Moving to a more evidence-based process

• Restructuring of the web development process
• Creation of a active web group
• Library Web Advisory Group
• Representation from all divisions of the Library
• Reference/Instruction
• Collection management
• Bibliographers
• Branch libraries
• Technical Services
• Library Public Relations
• Digital Library Systems Development
• Library Information Technology Support Services
• Web Communications Group at University Relations

17
Induction process

• Required reading list
• Extensive list of supplemental readings
• Required training
• In usability
• Conducting a usability test
• Other soft skills
• Evidence-based procedures
• Required participation in design and testing
• Usability test moderator
• Usability test observer
• Transcriber
• Missionary/representative

18
Current results

• Overall, site usability has improved
• And we can provide it with statistical measures!
• People find resources more quickly and easily
• But there is still an unfortunate amount of
  confusion
• Debates are less rancorous about how to proceed
• We can always go back to the data
• We can always go back to the studies
• Easier to develop a strategy for incremental
  improvements over time
• No longer locked into a tight academic schedule

19
Remaining issues

• Jargon
• I dont know what
• Metasearch
• E-journals
• Virtual reference
• Electronic resources
• means!
• Why should I go here? I can find everything in
  Google (or Google Scholar)
• To some degree, this is true

20
The big issue

• Designing for a world where they dont even come
  to the library web site
• When the day comes that Google Scholar or
  Microsoft Academic Live do provide a universal
  index
• Pluggable modules for the University portal

21
Thanks

• Frank Cervone
• Assistant University Librarian for Information
  Technology
• Northwestern University Library
• 1970 Campus Drive
• Evanston, IL 60208
• email f-cervone_at_northwestern.edu
• AIM/YAHOO hfcervone
• Skype frank.cervone