Title: Once a site is done, most people are not eager to hav
1Putting Evidence-based Practice to Work
Usability Testing at Northwestern University
Library
Frank CervoneAUL for Information
TechnologyMonday October 29, 2007Internet
Librarian 2007
2Overview
- The problem of web site design
- Using more formal methods
- Formal methods in practice
- Outcomes
3The problems of web site design
- Web development/HCI is an intricate mix of
technology and design - While there are exceptions, the majority of
librarians havent been trained in HCI - Gaps in understanding the significant differences
between the on-line and in-person experience - Once a site is done, most people are not eager to
have others suggest adjustments or changes,
particularly to those things they know are
right
Liberally adapted from Problems with in-house
website development by The IT Guyhttp//www.new
fangled.com/contentmgr/showdetails.php/id/7441
4Web site design and organizational responsiveness
- The more an organization depends upon its publics
for achieving its mission, the more it should
employ dialogic features into its Web site design - The challenge is to move the tasks associated
with Web site design and maintenance away from
being a "B-list" job to being an imperative for
the survival of highly stakeholder-dependent
organizations -
- Kent, M. L., Taylor, M., and White, W. J. (2003).
The relationship between Web site design and
organizational responsiveness to stakeholders.
Public Relations Review, 29(1), pp. 63-77.
5Which leads to
- Evidence-based information practice
- an approach to information practice that
promotes the collection, interpretation, and
integration of valid, important, and applicable
user-reported, librarian-observed, and
research-derived evidence
Andrew BoothSenior Lecturer in Evidence-based
Healthcare InformationSchool of Health and
Related ResearchUniversity of Sheffield
6What is evidence-based librarianship?
- Data provides the primary evidence for making
decisions - Not anecdotal stories
- Not common sense
- Evaluation occurs early in the process
- How is this different from what often happens
now? - Decisions are made based on
- Beliefs of what is needed
- Frequently, these beliefs are inherently biased
- Assumptions, anecdotal evidence, and preferences
- Evaluation, if it occurs, happens afterward
7Evidence-based professional practice
- Derived from the evidence-based model of medicine
- Fundamental precepts
- Study the phenomenon
- Contrast results to other studies of the same or
related phenomena - Combine results to better understand the
phenomenon at hand
8Evidence-based practice process
Redefine the problem
Define the problem
Evaluate change
Find the evidence
Apply results of the evaluation
Evaluate the evidence
9Defining the problem
- Very similar to concepts used in user-centered
design - Five points
- Yes, theres an acronym - SPICE
- SETTING
- Where is this being used what is the context?
- POPULATION
- Who are the (potential) users?
- INTERVENTION
- What is being done to/for them?
- COMPARISON
- What are the alternatives?
- EVALUATION
- What does success mean?
10Levels of evidence
- Systematic review of a wide-body of rigorous
studies - Systematic review of a wide-body of less rigorous
studies - Randomized controlled trials
- Controlled-comparison studies
- Cohort studies
- Descriptive surveys
- Case studies
- Decision analysis
- Qualitative research (focus groups, ethnography,
historic, Delphi techniques, interviews) - Surveys, audits
Adapted from Eldredge, J. (2000). Evidence-based
librarianship An overview. Bulletin of the
Medical Library Association, 88(4). pp
289-302. Online at http//www.pubmedcentral.nih.go
v/articlerender.fcgi?artid35250
11Which brings us to the Northwestern story
- First usability test in 2001
- Focused in on Electronic Resources
- A home-grown resource finder
- Investigation and questions based on anecdotal
evidence with ER interface - Learning experience
- Provided model for organization
- Whetted peoples appetite for usability testing
12The second test Searching in NUcat
- Began summer 2002
- How are people using Voyager?
- Are there discernable patterns?
- Most importantly are there patterns of errors?
- Can we do something to fix that?
- What we did
- Looked at searching patterns in our Voyager
system - Used a more rigorous research approach
- Based on grounded theory
- Simple data mining techniques
13And in the process
Searches by type
14dispelled some myths
No hits within searches by type
15Which lead to our tackling a bigger problem
- Existing site did not work especially well for
the Library's various user communities - Both students and faculty expressed frustration
with the web site - Ability to navigate the site
- Overall look and feel
- Problem areas determined by gathering information
in focus groups, online surveys, interaction with
librarians and staff - Hours and locations
- Collections
- Electronic resources
16Moving to a more evidence-based process
- Restructuring of the web development process
- Creation of a active web group
- Library Web Advisory Group
- Representation from all divisions of the Library
- Reference/Instruction
- Collection management
- Bibliographers
- Branch libraries
- Technical Services
- Library Public Relations
- Digital Library Systems Development
- Library Information Technology Support Services
- Web Communications Group at University Relations
17Induction process
- Required reading list
- Extensive list of supplemental readings
- Required training
- In usability
- Conducting a usability test
- Other soft skills
- Evidence-based procedures
- Required participation in design and testing
- Usability test moderator
- Usability test observer
- Transcriber
- Missionary/representative
18Current results
- Overall, site usability has improved
- And we can provide it with statistical measures!
- People find resources more quickly and easily
- But there is still an unfortunate amount of
confusion - Debates are less rancorous about how to proceed
- We can always go back to the data
- We can always go back to the studies
- Easier to develop a strategy for incremental
improvements over time - No longer locked into a tight academic schedule
19Remaining issues
- Jargon
- I dont know what
- Metasearch
- E-journals
- Virtual reference
- Electronic resources
- means!
- Why should I go here? I can find everything in
Google (or Google Scholar) - To some degree, this is true
20The big issue
- Designing for a world where they dont even come
to the library web site - When the day comes that Google Scholar or
Microsoft Academic Live do provide a universal
index - Pluggable modules for the University portal
21Thanks
- Frank Cervone
- Assistant University Librarian for Information
Technology - Northwestern University Library
- 1970 Campus Drive
- Evanston, IL 60208
- email f-cervone_at_northwestern.edu
- AIM/YAHOO hfcervone
- Skype frank.cervone