Assessment of Student Learning From Reference Service

1
Assessment of Student Learning From Reference
Service

• Jill Gremmels and Karen Lehmann
• Wartburg College
• Waverly, Iowa
• RUSA Reference Research Forum 2005 Chicago, June
  25, 2005

2
Survey of the Literature on Reference Evaluation

• Overview by Green Peach (2003)
• Reference as a teaching learning activity
• Review essays
• Bunge (1994), Smith (1991), Campbell (2000)
• Traditional techniques
• Tally questions, such as length, time, date, etc.
• User interviews (Mendelsohn, 1997)
• Focus groups (Massey-Burzio, 1998)
• Survey/Observation (Norlin, 2000)

3
Category 1 Reference Evaluation Literature

• 55 percent school
• Hernon McClure (1986)
• Reference service evaluated on accuracy of
  responses to predetermined questions.
• Questions come from unobtrusive people posing
  as patrons.
• Critics Reference is more than right and wrong
  answers.
• Critics If reference is communication, delivery
  can be as important as answers.

4
Category 2 Reference Evaluation Literature

• Interpersonal communication
• Durrance (1989 1995) Jardine (1995)
• More complex model.
• Focuses on willingness of patrons to return to
  the same staff member for future needs.
• Critics Wrong answers are still wrong, even
  from a librarian you liked.

5
Category 3 Reference Evaluation Literature

• User satisfaction conditions of the reference
  transaction
• Bunge Murfin (1995) Stalker Murfin (1995)
• Based on Wisconsin-Ohio Reference Evaluation
  Program.
• 2 part evaluation form patron and librarian.
• Allows variables ( of resources used, is library
  busy, subject area of inquiry, training of
  librarian).
• Externally validated assessment instrument (can
  match to other libraries).

6
Shortcomings of all 3

• Correctness
• Dont reflect reality of academic reference work.
• Performance
• Approaches dont offer proof of teaching activity
  of librarians.
• Assessment
• Dont connect to learning outcomes.

7
Assessment Applied to Learning

• Outcomes Assessment is
• Knowing what you are doing.
• Knowing why you are doing it.
• Knowing what students are learning as a result.
• Changing because of the information.
• Debra Gilchrist, Pierce Colleges, Lakewood and
  Puyallup, WA

8
History of Assessment in Higher Education (Ewell)

• AAHE Assessment Conferences ran 1985-2004.
• Assessment is still not a culture of use
  (Ewell, 2002), embedded into lives and work.
• Dueling agendas institutional change vs.
  accountability for results.
• Summative or formative?
• Basic (and still unanswered) question
• Is there any real evidence that assessment
  actually promotes better learning?

9
Tying Academic Libraries into Campus Assessment

• Quantitative vs. qualitative
• By Services
• Adams (1996) Wallace (2001)
• By Learning
• Iannuzzi (1999) Breivik (1998) Boyer Commission
  Report
• Gremmels Ruediger (2003) Lopez (2002)
• If information literacy library culture is
  strong, it will influence campus culture.

10
Building Foundations for Assessment

• Reference Course-Related Credit Courses
• Information Literacy Outcomes
• Information Literacy Definition
• 
  Mission Debra Gilchrist
• Pierce College
  Philosophy

11
Reference Mission Statement

• Vogel Librarys mission is to educate
  information-literate lifelong learners. We
  strive to make each reference encounter an
  educational experience that reinforces
  information literacy concepts by building upon
  prior instruction and giving further
  opportunities for guided practice.

12
The Teaching Role of Reference Can it be
Assessed?

• Elmborg (2002)
• Librarians need pedagogy for the reference desk.
• This implies reference is a form of teaching.
• Academic librarians should approach reference
  transactions as academic conferences where
  teaching and learning occur.
• Green Peach (2003)
• Assumed this connection and wanted to validate
• it for purposes of librarian evaluation.

13
The Wartburg Study

• Wanted to know WHAT the student learned.
• Does reference reinforce classroom instruction?
• Based on outcomes (Info Lit Competency Standards)
• They set our agenda
• Create our vision
• Focus our teaching
• Provide our common purpose
• Guide our students
• Debra Gilchrist, Pierce Colleges, Lakewood and
  Puyallup, Washington

14
Methodology

• Paired and numbered surveys.
• Librarian and student both answered after a
  reference encounter.
• Questions developed by reference librarians.
• Based on information literacy concepts taught in
  the classroom.

15
Methodology

• Teaching intentions vocabulary
• choosing good search terms
• database selection
• search strategy
• evaluating information
• how to use a specific tool
• other (specify)
• No limits on taking survey multiple times.

16
Methodology

• Student questionnaire Spring 2003
• Tried to tie reference encounters to prior
  instruction.
• 143 returned (85 response rate).
• Librarian questionnaire Spring 2003
• Six choices for instructional outcomes. Only one
  answer allowed.
• 169 returned.

17
Methodology

• 2003-04 Academic Year
• Student questionnaire
• Added demographic questions
• Asked student to describe instruction and assign
  a category
• 121 returned (78 response rate).
• Librarian questionnaire
• Only change was allowing multiple answers if
  ranked.
• 156 returned.

18
Methodology

• January-May 2003
• 141 useable responses (143 returned).
• Four librarians administered the survey.
• 3 of the 4 participate in ILAC classroom
  instruction.

• 2003-2004 Academic Year
• 121 useable responses
• Same four librarians administered the survey.

19
Data Analysis First Survey

• Surveys re-paired.
• Answers entered into spreadsheet and narrative
  list and tallied.
• Two librarians independently judged congruence as
  related, not related, or
  inconclusive. High inter-rater reliability.
• Discussed and resolved differences.

20
Data Analysis Second Survey

• Surveys re-paired.
• Answers entered into spreadsheet and Qualrus
• Category matches identified.
• Researchers independently analyzed forms for
  strong, acceptable or no description
  matches. Inter-rater reliability high.
• Discussed and resolved differences.

21
Description Match Examples

• Student how to use the catalog
• Librarian iPac to find music CDs
• Student how to cite CQ Researcher
• Librarian citation with CQ

• Student how to find literary criticism
• Librarian Literature Resource Center and
  Contemporary Literary Criticism

22
Non-Match Examples

• Student Showed me different places I could look
  for the information I was seeking.
• Librarian Oxford Reference Online defining
  "liberal arts"

• Student He told me some important information
  about companies on websites recommended by the
  college.
• Librarian LexisNexis Business and Business
  Source Elite

23
Demographics
24
Q. 1 Did the librarian who just helped you
teach you anything while answering your question?
25
Q. 3 Did a librarian meet with your class and
teach your class how to find information for this
assignment?
26
Q. 4 Did what the librarian taught you relate to
or build on anything a librarian taught your
class?
27
Q. 5 Did what the librarian taught relate to or
build on a previous lesson?
28
Students made the link sometimes.
29
Link stronger with tools
30
What We Learned

• Students understand reference to be an
  instructional activity.
• Information Literacy instruction does seem to
  yield results.
• Reference helps students practice and reinforce
  information literacy knowledge.

31
Reference Applications

• Began a three-tier reference model.
• 1--Frontline Tool instruction on demand
• 2--Backup Concepts on call
• 3--In-Depth Consultations as scheduled
• All staff participate.

32
Reference Applications

• Focus on reference as guided practice.
• Embrace role as facilitator.
• Be more explicit about links to classroom
  instruction.
• Brief questions about process and concepts.
• Graphic organizers.

33
Reference Applications

• Have declined to participate in consortial
  virtual reference project.
• Beginning a campaign to promote consultations
• PSA
• Faculty awareness
• New librarian will create online tutorials.

34
Classroom Applications

• Continue to seek information literacy
  opportunities in classes.
• especially as survey data reveal which
  courses students are working on.
• Have all staff sit in on information literacy
  lessons.
• so they know what is being taught and
  can reinforce better.

35
Further Research

• Other academic libraries adapt our form and do
  their own assessments?
• Assess consultations at Wartburg

36
Contact Information
Jill Gremmels [phone] jill.gremmels_at_wartbur
g.edu
Karen Lehmann [phone] karen.lehmann_at_wartbur
g.edu