Tweaking the Dashboard: Validating Academic Excellence Quality Indicators

1
Tweaking the Dashboard Validating Academic
Excellence Quality Indicators

• Susan J. Beck
• susan.beck_at_rutgers.edu
• Rutgers, the State University of New Jersey
• August 23, 2005

2
The Impact and Outcomes of Library Services

• 6th Northumbria International Conference on
  Performance Measures in Libraries and Information
  Services

3
Assessment at Rutgers University Libraries

• The Dashboard Approach as part of the Balanced
  Scorecard was selected to monitor organizational
  performance.
• Cabinet Developed the Academic Excellence Quality
  Indicators
• The University Librarian assigned the LAC to
  Validate the Indicators

4
(No Transcript)
5
The Process Used

• LAC conducted focus groups with selected user
  groups
• Faculty Students
• Librarians and Library Staff
• University Administration
• Alumni, and
• New Jersey Citizens

6
Results

• Indicators were validated or adapted
• based on user feedback.
• the study provided valuable qualitative data
  about
• Rutgers quality indicators
• The wants and needs of users, and
• the performance of the library.

7
Findings

• By Stakeholders

8
Faculty Role of interview participants

• The RUL Library Advisory Committee, a body of
  faculty from all three campuses who act as a
  sounding board for the university librarian
  represented the faculty

9
Faculty Quality indicators evaluated

• Responsive Collections in All Formats
• Easy Access to RUL Collections
• Environment Conducive to Research and Study
• Service Responsiveness of Staff/Librarians
• Information Literacy Attainment

10
Faculty Assessment, Concerns, Recommendations

• Group faculty graduate students together as one
  user group which represents the concept of
  researchers and have undergraduates as a separate
  group.
• Add criteria that measure the facultys
  perception of the librarys ability to plan for
  future faculty needs, disciplinary changes, and
  technological advancements.
• Measure faculty perceptions of the importance of
  the library to the university in terms of
  receiving adequate funding.

11
Faculty Importance of Indicators

• 1. Easy access to Rutgers university library
  collections
• 2. Responsive collections in all formats
• 3. Ease of use of digital information
• 4. Access to collections elsewhere
• 5. Service responsiveness of staff
  librarians
• 6. Information literacy attainment
• 7. Environment conducive to research study

12
Students Roles of Interview Participants

• Student Organization Presidents

13
Students Quality Indicators Evaluated

• Responsive Collections in All Formats
• Easy Access to RUL Collections
• Environment Conducive to Research and Study
• Service Responsiveness of Staff/Librarians
• Information Literacy Attainment

14
Students Assessment, Concerns, Recommendations

• Indicators validated.
• Indicators not changed or modified.
• These sound good to us.
• Pleased that the libraries consulted them.
• Desired better facilities and library use of the
  universitys debit card.
• Excited about the prospect of the libraries
  participation in LibQUAL.

15
Students Importance of Indicators

• No modifications
• No additions

16
Librarians and Staff Roles of Interview
Participants

• Self selected Library Faculty and Staff plus one
  administrative librarian from all campuses 

17
Librarians and Staff Quality Indicators Evaluated

• Adequate Institutional Support
• Access to Learning Development Activities
• Physical Work Environment
• Organizational Climate

18
Librarians and Staff Assessment, Concerns,
Recommendations

• Job satisfaction
• What is the public perception of the library?
• Under Adequate Institutional Support, add staff
  size sufficiency measures

• How do librarians add value?
• Easy access to RUL collections mention
  cataloging.
• Add evaluation of web subject guides
• How is the library used?

19
Librarians and Staff Importance of Indicators

• Organizational climate
• Institutional support
• Access to appropriate computing equipment
• Recognition
• Access to learning development activities
• Personal safety
• Systems availability
• Physical plant security

20
Campus Provosts Role of Interview Participants

• Principal Administrative Officer
• for the individual campuses representing the
  university administration perspective

21
Campus Provosts Quality Indicators Evaluated

• Reputation
• Leadership
• Service

22
Campus Provosts Assessment, Concerns,
Recommendations

• Faculty and Student satisfaction
• If you rate high on the library indicators
  developed for our faculty and students than I am
  happy with your performance.
• Align the librarys goals to support the
  universitys strategic priorities and goals.
• Integrate into the academic life of each campus
• participate in student life and learning
  activities.

23
Campus Provosts Assessment, Concerns,
Recommendations, Continued

• Monitor ARL Salary Statistics to ensure
  competitive salaries for recruitment.
• Reputation Not interested in ARL Rankings.
  They reflect financial support only. Provosts
  value effectiveness.
• Monitor electronic resource use by local business
  nonprofit communities.

24
Campus Provosts Relative Importance of
Indicators

• 1.      Faculty and Student satisfaction
• 2.      Competitive Salaries for Librarians

25
The Alumni Office Roles of Interview Participants

• These individuals represent the Alumni
  Perspective
• Director of Alumni Relations
• Manager of Membership and Marketing Alumni
  Communications
• Receptionist Alumni Office

26
The Alumni Office Quality Indicators Evaluated

• Alumni access to resources

27
The Alumni Office Assessment, Concerns,
Recommendations

• Desire remote access to our online resources
• Should be able to use the librarys research
  materials for their current jobs
• Alumni need to be more aware of the
• Librarys resources
• Librarys web page
• Their library privileges

28
The Alumni Office Assessment, Concerns,
Recommendations

• Offered to publish an alumni magazine article
  about the libraries
• linking to a library survey about alumni views
• Recommended how to survey their members
• Offered to track individuals for focus groups,
  including
• Those contacting them about their library
  privileges
• Those requesting remote access privileges

29
The Alumni Office Importance of Indicators

• 1.      Access to materials
• 2.      Awareness of services and resources

30
State of New Jersey Roles of Interview
Participants

• State Librarian and Library Development Bureau
  Consultant represent
• New Jersey Citizens
• Statewide library community
• Executive Director of NJLA represents the
  statewide librarian community.

31
State of New Jersey Quality Indicators Evaluated

• Service

32
State of New Jersey Assessment, Concerns,
Recommendations

• Citizens unaware of university library use
  privilege
• Increase awareness of library services to NJ
  citizens
• Advocate stronger RUL librarian collaboration
  with the State Library and NJLA.
• Data gathering recommendations survey government
  entities, clubs, state organizations and
  individuals.

33
State of New Jersey Importance of Indicators

• 1. Awareness of RUL Services and Collections
• 2.  Collaboration with the State Library NJLA

34
Conclusions

• Criteria validated or refined
• Dashboard not yet implemented
• Some elements of assessment are in place
• Participated in LibQual in Spring 2005
• Some data elements collected historically
• Other data elements identified
• Some data gathering instruments to be designed

35
Caution

• Development/validation process of the Dashboard
  is time-consuming, although worthwhile and
  interesting.
• Dashboard approach was introduced October 2002.
• Validation began in September 2004
• It is now August 2005

36
Momentum

• Such a long process may reduce assessments
  momentum
• The pace of our process is partially a result of
  being implemented by a Committee rather than an
  Assessment Office
• The Committee members are busy Library Faculty
  and Administrators with many responsibilities

37
An Alternative Structure

• The effort needed for timely implementation of a
  Dashboard,
• at a large institution such as Rutgers,
• may justify a library assessment office

38
Evolution

• The author believes that Rutgers process has
  moved from
• a Balanced Scorecard Dashboard
• toward the Performance Prism developed at
  Cranfield University
• The implications of this shift have yet to be
  explored.

39
The Performance Prism

• The Performance Prism considers
• the wants and needs of the stakeholders and
• also what the organization wants and needs from
  its stakeholders.

40
The Prism Address

• Centre for Business Performance
• School of Management
• Cranfield University
• www.som.cranfield.ac.uk
• www.som.cranfield.ac.uk/som/research/centres/cbp/p
  roducts/prism.asp
• available on August 14, 2005

41
For Further Information

• Academic Excellence Quality Indicators
• http//www.libraries.rutgers.edu/rul/staff/groups/
  assessment_com/reports.shtml
• Will direct you to the Library Assessment
  Committee Reports page.
• Once there scroll down to Academic Excellence
  Quality Indicators
• You will find links to several documents from
  LAC, related to the Quality Indicators