Assessing our Assessment: Failures and Successes at UT-Austin

1
Assessing our Assessment Failures and Successes
at UT-Austin

• AJ Johnson, Michele Ostrow,
• Meghan Sitar
• Library Instruction Services
• University of Texas Libraries

2
FAILURE 1Didnt ask for help at the beginning
http//www.flickr.com/photos/inju/246717376/
3
FAILURE 2Didnt recognize the difference between
faculty and librarian culture
http//fc04.deviantart.com/fs16/f/2007/160/7/f/Wil
liamSnape2_by_WilliamSnape.jpg
http//www.hermionegrangerswand.com/
4
FAILURE 3Didnt anticipate low response rates
www.houseofzing.com/book/images/ignore.jpg
http//www.houseofzing.com/book/images/ignore.jpg
5
FAILURE 4Didnt consider how faculty view
librarians
http//www.flickr.com/photos/kaishin/1660238046/
http//www.flickr.com/photos/bibliona/238979054/
6
FAILURE 5 Didnt hire an army of instruction
librarians and statisticians
"United States Army, The." Online Photograph.
Encyclopædia Britannica Online. 22 May 2008
 lthttp//www.britannica.com/eb/art-82331gt.
7
FAILURE 6 Didnt pre-test
8
FAILURE 7Still didnt pre-test
9
FAILURE 7Still didnt pre-test

10
FAILURE 8 Designed our post-test around faculty
characteristics

11
FAILURE 8 Designed our post-test around faculty
characteristics

12
FAILURE 9Didnt match pre-tests to post-tests
13
FAILURE 9Didnt match pre-tests to post-tests
14
FAILURE 10 Project design didnt reflect
flexibility of our instruction programs
15
FAILURE 10 Project design didnt reflect
flexibility of our instruction programs
16
FAILURE 11 Didnt define a control group
No controle (in control) by renatotarga on
Flickr, http//www.flickr.com/photos/rtarga/22377
6150/
17
FAILURE 12 Thought writing questions would be
the easy part

• How would you . . .
• What three things would you . . .
• If you had to choose . . .
• Why does the . . .
•  
• ARGH!

18
FAILURE 13Didnt base questions on learning
outcomes
19
FAILURE 13Didnt base questions on learning
outcomes

• OLD Question - You found an article in a
  database but it isnt full-text. How do you find
  the full text of the article somewhere else?
  (Circle all that apply)
• Search the Library Catalog. Use the article
  title as your search terms.
• Search the Library Catalog. Use the title of the
  journal, magazine or newspaper as your search
  terms.
• Follow the FIND IT UT link

20
FAILURE 13Didnt base questions on learning
outcomes

• NEW Learning Outcome - Students will be able to
  determine where scholarly material will be found.
•  Question - You are doing research on border
  security along the U.S. Mexico border. Choose the
  BEST place to find a research study by a
  political science professor.
• Library article database
• www.google.com
• Published encyclopedia
• Dont know

21
FAILURE 14 Forced to use multiple choice tool
to measure critical thinking skills
22
FAILURE 14 Forced to use multiple choice tool
to measure critical thinking skills

• Old
• When evaluating web sites to see if you should
  use them in your research, which factors should
  you consider? (Choose all that apply)
• Site was on the first page of results in Google
• Author expertise
• Page design
• Bias
• Dont know

23
FAILURE 14 Forced to use multiple choice tool
to measure critical thinking skills

• Newer
• You find a web page about media mergers and know
  nothing about the author. What would be your
  best clue as to whether the page was a reliable
  source to use for your paper?
• date the page was last updated
• the URL (http//www...)
• the quality of the pages graphic design
• the advertisements on the page
• dont know

24
FAILURE 14 Forced to use multiple choice tool
to measure critical thinking skills

• Newest
• You are doing research on the effect of illegal
  immigration on border violence. You find a
  website from an organization called The Illegal
  Immigration Research Coalition, which includes
  statistics on incidents of border violence. List
  at least three criteria you would use to evaluate
  this site for bias and credibility.

25
FAILURE 15Created questions with more than one
right answer

• 1. Google searches will find information from
  library databases
• True False Dont know
•  
• 2. Choose the BEST place to find a recent report
  from a media watchdog organization
• library article database
• www.google.com
• published encyclopedia
• don't know

26
FAILURE 16Created trick questions

• You are doing Internet research about border
  security. Choose the domain of a website most
  likely to contain a U.S. presidential candidate's
  stance on immigration issues
• .org
• .edu
• .gov
• .com
• don't know

27
FAILURE 17Questions didnt mean the same thing
to students as they did to us
28
FAILURE 18Didnt define success before we
started
http//www.easywebcreator.com/applications/WebBlog
Manager/inc_webblogmanager.asp?CategoryID227Item
ID1061actionrefer
29
FAILURE 19Didnt identify all the variables
http//www.flickr.com/photos/
jdawg/172131301/
30
FAILURE 20Cant really use the data
31
FAILURE 20Cant really use the data
32
FAILURE 20 Cant really use the data
33
Group ActivityWriting Assessment Questions
Tied to Student Learning Outcomes
34
Student Learning Outcomes

• 1. Students will be able to explore general
  information sources to increase familiarity with
  a topic.

35
Student Learning Outcomes

• 2. Students will be able to identify the purpose
  and audience of popular and scholarly resources.

36
Student Learning Outcomes

• 3. Students will be able to differentiate between
  the types of sources cited and understand the
  elements of a citation.

37
Student Learning Outcomes

• 4. Students will be able to differentiate between
  searching by keyword and by subject.

38
Student Learning Outcomes

• 5. Students will be able to distinguish between
  the free Web and online, fee-based library
  resources.

39
Presentation and Questions Created by
Participants available at

• http//www.lib.utexas.edu/services/instruction/ost
  row.html