Strategies, Tips, and Tools for Facilitating Learning Outcomes Assessment in Student Services

1
Strategies Tips and Tools for Facilitating
Learning Outcomes Assessment in Student Services
Student Learning Outcomes
Jerry Rudmann Irvine Valley CollegeFebruary
2008
2
Overview - Student Services

• Fine-tuning assessment
• Tips for writing survey items
• Focus groups
• Helpful technology tools
• Clickers - promote active learning and record SLO
  information
• Rubric generators - a way to measure most
  anything
• PDF Acrobat forms - autoscoring and recording
  student input
• Portfolios - making students responsible and
  reflective
• Scanning - some ideas
• Tracking software - organizing all this stuff
• Several options / strategies for making SLOs
  meaningful
• SSO versus SLO
• Problem focus
• Less is better
• Use what you already have
• Think of SLOs in the context of student
  development
• Qualitative assessment in OK
• Other

3
Some Options / Strategies for Making SLOs
Meaningful

• Address robust SLOs (overarching outcomes)
• Problem focus
• Less is better
• Share SLOs with students
• Use what you already have
• Think of SLOs in the context of student
  development
• Qualitative assessment is OK
• SSOs vs. SLOs

4
General Tip 1 Problem Focus Approach

• What competencies do students have difficulty
  mastering
• Focus SLO activities on problem areas.

5
General Tip 2 Keep It Simple But Meaningful

• Corollary - Often less is better.

6
General Tip 3 Student Development Approach

• Student development
• Academic self-efficacy (Bandura)
• Academic self-regulation
• Campus involvement (Astin)
• Mentoring professor studies
• Student Services DO help student success

7

• Surveys

8
Surveys - SLO Uses

• Students self-rate their competencies on program
  or college level learning outcomes.
• Students satisfaction with various student
  services.

9
Types of Questions

• Open-ended respondents answer in own words
• Closed-ended respondent limited to a finite
  range of choices

10
Types of Questions

• Open-ended
• Flexible
• Hard to code answers
• Good for preliminary work to finalize a survey
• Closed-ended
• Easier to code answers process and analyze
• Hard to write good closed-ended items

11
Item Format

• Visual Analogue Scale
• Food in the cafeteria is
• Poor_ _ __ _ _ _ _ _ _ _ _ _ _ _ _ _Excellent
• Likert Scale
• Food in the cafeteria is outstanding!
• SD D N A SA
• (Strongly Agree) (Disagree) (Neutral)
  (Agree) (Strongly Agree)
  

12
Nine tips for designing and deploying a survey

• Dont call it a survey
• Provide a carefully worded rationale or
  justification at the beginning
• Group items by common format
• Start with more interesting items
• Put demographic items last
• Mix in negative wording to catch acquiescence
  (aka response set)
• Automate scoring when possible
• If asking for sensitive information use
  procedures designed to assure anonymity
• Always always always pilot test first

13
Survey Administration Methods

• Face to Face
• Written
• Group administration
• Mail
• Computerized http//research.ccc.cccd.edu
• Password protected
• Validation rules
• Branching and piping
• Telephone

14
Focus Groups
Focus groups can be especially insightful and
helpful for program and institutional level
learning outcome assessment. Have your college
researcher provide some background materials.
Focus Groups A Practical Guide for Applied
Research By Richard A. Krueger Mary Anne
Casey The RP Group sponsored several drive in
workshops over the last few years.
15
Goal for This Section

• Technology Uses
• Technology ToolsExpected Outcome Be able to
  select and use technology-based approaches to
  assess student learning outcomes

16
Assessment Challenges

• Engaging Students in Self-Evaluation
• More Efficient Assessment

17
Some Technology Tools

• Online Rubric Builders
• eLumen (SLO Assessment/Tracking)
• Classroom Responders (Clickers)
• Scannable and Online Tests and Surveys
• ePortfolios
• Adobe Acrobat Forms
• Excel Spreadsheets

18
Rubrics

• Way to measure the heretofore immeasurable
  products and performances.
• A rubric breaks the assessment into important
  components.
• Each component rated along a scale well-labeled
  scale.

19
Lets Develop an Assessment Rubric for a
Resume
20
Chocolate Chip Cookie Rubric
Chocolate Chip Cookie Rubric
21
Rubrics are Good!

• Facilitate staff dialogue regarding satisfactory
  performance.
• Create a more objective assessment.
• Make expectations more explicit to the student.
• Encourage metacognitive skill of self-monitoring
  own learning.
• Facilitate scoring and reporting of data.

22
Online Discussion Rubric
http//www.uas.alaska.edu/sitka/IDC/resources/onli
neDiscussionRubric.pdf
23
Design Your Own Rubric

• Please work in groups and use the worksheet in
  your packet to design a scoring rubric for
  assessing one of the following
• Coffee shops
• Syllabi
• Customer service at retail stores
• Grocery stores
• Online courses

24
Online Rubric Builders

• Rubrics to guide and measure learning
• Tools
• Rubistar http//rubistar.4teachers.org
• Landmark Rubric Machine http//landmark-project.co
  m/rubric_builder

25
Rubistar
Rubistar Art History Rubric
26
Rubric Builder Screen Shot
27
Adobe Acrobat Forms

• Make form using MS Word
• Import form and save as PDF form
• Adjust the fields
• Add fields to tally sub scores and total scores

28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
How Do You Report Results
32
eLumen to Assess SLOs

• Reduce Time Spent Creating Reports
• Assess Course Program and/or Degree-Level
  Outcomes
• Share Assessment Rubrics Across Classes and
  Programs
• View Individual or Aggregated Results
• Use Online or Offline

http//www.elumen.info
33
Use Online or Offline
34
Criterion-Based Assessment

• Rubrics are attached to each SLO

Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
35
Rubrics Describe Criteria

• Writes prose clearly

Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
36
Library of Degree-Level SLOs
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
37
And Rubrics Link to SLOs
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
38
Science and Gen Ed SLOs/Rubrics
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
39
Scorecard for All Students in the Course
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
40
Class Scores by Student
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
41
Aggregated Data for Course
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
42
Course Aggregates by Program
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
43
Classroom Responders

• Engage students
• Monitor student understanding
• Quickly and easily collect and store assessment
  data
• Use publisher item banks or create your own

44
Renaissance Classroom Response System
45
Most valuable tip is

• Finding ways to use technology to make SLO and
  SSO assessment easier and more efficient
• Concentrating SLO work on skills students have
  difficulty mastering
• Building SLOs around student development
  (self-efficacy goal clarity etc.)

46
Renaissance Learningfor clicker training
resources
http//www.renlearn.com
47
Scanning Technology

• A way to gather survey input from students
• A way to test students knowledge

http//www.scantron.com and http//www.renlearn.co
m
48
Surveys and Tests

• Online or Scannable
• Surveys
• Pre and post surveys of student self evaluation
  of progress
• Gather stakeholder (faculty business community
  leaders advisory groups) input on expected
  learning outcomes
• Student satisfaction with service (SSO)
• Quizzes/Tests
• Practice and graded

49
Some Survey Software Options

• Scannable surveys and quizzes - Optical Mark
  Reader by Remark (OMR Remark)
• Need software and a Fujitsu scanner
• Use word processor to create scannable bubble-in
  surveys or answer sheets.
• Produces item analysis output.
• Online survey tools
• eListen (Scantron Co.)
• SelectSurvey.net http//www.classapps.com/SelectSu
  rveyNETOverview.asp

50
Excel Spreadsheets

• Example of autoscoring and record keeping in a
  Japanese Program.

51
ePortfolios

• Advantages
• Document artifacts of learning
• Support diverse learning styles
• Authentic assessment
• Course program or degree-level tracking
• Job skill documentation
• Proprietary or Open Source
• ePortfolio and Open Source Portfolio

52
ePortfolio.org Assessment
Lock Assignments after submission Random
selection of assignments by learning
objective Anonymity of the student who produced
the assignment and the instructor Access to the
work and the scoring rubrics Reports aggregate
scores generate frequencies/means Ability to
download raw data which can be analyzed in
another format
http//www.eportfolio.org
53
Open Source Portfolio

• Aligned with Sakai
• Admins or Faculty can structure and review work
• Learning matrix documents levels of work

http//www.osportfolio.org
54
Resources

• eListen http//www.elisten.com
• eLumen http//www.elumen.info
• ePortfolios
• ePortfolio.org http//eportfolio.org
• Open Source Portfolio http//www.osportfolio.org/
  
• For others see EduTools ePortfolio product
  comparison http//eportfolio.edutools.info/item_l
  ist.jsppj16
• Online Rubric Builders
• Rubistar http//rubistar.4teachers.org
• Landmark Rubric Machine http//landmark-project.c
  om/rubric_builder/index.php
• Coastline Rubric Builder http//rubrics.coastline
  .edu
• Remark Survey Software http//www.principiaproduc
  ts.com/web/index.html
• Renaissance Classroom Responders
  http//www.renlearn.com/renresponder/
• SelectSurvey.Net http//www.classapps.com/SelectSu
  rveyNETOverview.asp

55
Contact Info Acknowledgements

• Dr. Jerry Rudmann
• Professor of PsychologyIrvine Valley
  Collegejrudmann_at_ivc.edu
• Much of thisshow was adapted (with the
  express written permission) from Pat Arlington
  Instructor/Coordinator Instructional
  ResearchCoastline Community Collegeparlington_at_co
  astline.edu

56
Development of New SLO Measures Procedure
Findings Conclusions and Recommendations from a
Recent Exploratory Study
57
Purpose of the Study

• The study was designed to explore whether
  assessment tools used to measure cognitive
  variables -- e.g. goal clarity self-efficacy --
  could serve as learning outcome measures in
  Student Services.

58
The Spark for this Study

• The need for truly appropriate and really useful
  assessment measures in Student Services.
• Ideas generated by interviews with counselors.

59
Possible Relationships

• Students Academic Outcomes
• Short term outcomes
• Semester GPA
• Units earned
• units completed
• Return next semester
• Long range outcomes
• GPA
• Units earned
• Certificate degree and/or transfer

• Attributes of New Students
• Confidence level
• Goals
• Motivation
• Study skills and habits

60
Possible Relationships

• Students Academic Outcomes
• Short term outcomes
• Semester GPA
• Units earned
• units completed
• Return next semester
• Long range outcomes
• GPA
• Units earned
• Certificate degree and/or transfer

• Attributes of New Students
• Confidence level
• Goals
• Motivation
• Study skills and habits

Student Services College success
courses Academic counseling Career Center
presentations Career counseling Career
course Club team chorus band student
government or other form of social
connectedness Formal and informal recognition for
progress Non-academic counseling Transfer Center
programs Peer advisors Tutoring center University
tours
61
Procedure

• Counselor interviews (preliminary brainstorming)
• Literature survey for promising assessments tools
• Recruitment presentations at Region 8 DSPS and
  EOPS meetings
• A website was created having all assessments
  online

62
Study Website
63
Measures We Tried

• Academic and Career Goal Clarity
• Academic Self-Efficacy
• Dispositional Hope
• Self-Regulation
• Optimism
• Positive Affect
• Negative Affect

64
Summary and Examples of Measures Found Most
Useful in This Study

• Academic Self-Efficacy
• Beliefs about ones capabilities to learn or
  perform at designated levels. Compared with
  students who doubt their learning capabilities
  those who feel efficacious for learning or
  performing a task participate more readily work
  harder persist longer when they encounter
  difficulties and achieve at a high level.
• I know how to schedule my time to accomplish
  tasks.
• I know how to study to perform well on tests.
• Academic Self-Regulation
• Confidence in ability to perform various academic
  tasks.
• I can take notes of class instruction.
• I know how to use the library to get information
  for assignments.
• Academic and Career Goal Clarity
• Measures clarity of immediate and long range
  academic plans and extent to which student has
  career clarity.
• I have worked with a counselor to develop a plan
  listing the courses I need to complete my lower
  division coursework.
• I have decided on an academic major.
• I am familiar with the daily work routine for
  people working in my desired career.

65
Participation

• DSPS study
• Seven colleges
• Students
• Pre-test - 142
• Post-test -127
• EOPS study
• Six colleges
• Students
• Pre-test - 276
• Post-test - 154

66
Descriptive Statistics from the DSPS Study
67
Intercorrelations Among Scales and Academic
Outcomes (DSPS study)
68
Predicting Academic Outcomes (Based on
Correlation Matrix and Stepwise Regression
Analyses)
69
Impact of Services on Student Outcomes
70
DSPS Study Self Regulation Receipt of
Counseling and Semester GPA
Low SR - Yes Counseling 32 Low SR - No
Counseling 31 High SR - Yes Counseling
20 High SR - No Counseling 20 (N 103)
71
DSPS Study Self Regulation Receipt of
Counseling and Percentage of Units Earned for
Units Attempted
Low SR - Yes Counseling 32 Low SR - No
Counseling 32 High SR - Yes Counseling
20 High SR - No Counseling 20 (N 103)
72
EOPS Study Self Regulation Peer Advising and
Semester GPA
73
Changes in Goal Clarity and Receipt of Transfer
Assistance
74
Changes in Goal Clarity and Receipt of Peer
Advisement
75
Limitations of Study

• Lack of random selection and assignment to
  treatments
• Self-selection bias
• Results are correlational not causal
• Data are an aggregate from the participating
  colleges but there may be significant
  differences among colleges procedures services
  personnel etc.

76
Thoughts

• The instruments are
• Inexpensive easy to complete and score
• Can help identify at risk students
• Can help formulate appropriate ways to assist
  students
• Gain scores derived from pre to post-test
  assessments can be useful
• Use as SLO assessment instruments that are good
  matches to the services provided within Student
  Services

77
Recommendations

• Map your services to the constructs measured by
  these instruments.
• Develop new interventions where none currently
  exist.
• Create an assessment referral system.

78
The Three Instruments Found Most Useful

• Academic and Career Goal Clarity (Tucker
  Rudmann 2006)
• Measures overall clarity and sub-components
  of goal clarity
• Academic Self-Efficacy (Chemers Hu Garcia
  2001)
• Measures confidence in reaching positive
  academic outcomes
• Efficacy for Self-Regulated Learning (Zimmerman
  Bandura Marinez-Pons 1992)
• Measures confidence in ones ability to manage
  and regulate academic tasks students face in
  college

79
Goal Clarity Instrument Structure
80
One Interpretation
81
Converting to Adobe Acrobat Interactive Form
82
Current Potential Services for Enhancing These
Important Student Learning Outcomes
83
Research Team

• Jerry Rudmann PhD
• Professor of Psychology
• Irvine Valley College
• jrudmann_at_ivc.edu
• Kari Tucker PhD
• Professor of Psychology
• Department Chair
• ktucker_at_ivc.edu
• Shañon Gonzalez MA
• Coastline College
• Research Assistant III
• sgonzalez_at_coastline.edu

84
Four Sources of Efficacy Beliefs

• Mastery experiences -- Outcomes interpreted as
  successful raise efficacy those interpreted as
  failures lower efficacy
• Vicariously success or failure of models
• Verbal persuasions by others positive or
  negative appraisals by others
• Physiological states (e.g. anxiety stress
  arousal fatigue mood) act as information about
  efficacy beliefs and can raise or lower efficacy