Title: Strategies, Tips, and Tools for Facilitating Learning Outcomes Assessment in Student Services
1Strategies Tips and Tools for Facilitating
Learning Outcomes Assessment in Student Services
Student Learning Outcomes
Jerry Rudmann Irvine Valley CollegeFebruary
2008
2Overview - Student Services
- Fine-tuning assessment
- Tips for writing survey items
- Focus groups
- Helpful technology tools
- Clickers - promote active learning and record SLO
information - Rubric generators - a way to measure most
anything - PDF Acrobat forms - autoscoring and recording
student input - Portfolios - making students responsible and
reflective - Scanning - some ideas
- Tracking software - organizing all this stuff
- Several options / strategies for making SLOs
meaningful - SSO versus SLO
- Problem focus
- Less is better
- Use what you already have
- Think of SLOs in the context of student
development - Qualitative assessment in OK
- Other
3Some Options / Strategies for Making SLOs
Meaningful
- Address robust SLOs (overarching outcomes)
- Problem focus
- Less is better
- Share SLOs with students
- Use what you already have
- Think of SLOs in the context of student
development - Qualitative assessment is OK
- SSOs vs. SLOs
4General Tip 1 Problem Focus Approach
- What competencies do students have difficulty
mastering - Focus SLO activities on problem areas.
5General Tip 2 Keep It Simple But Meaningful
- Corollary - Often less is better.
6General Tip 3 Student Development Approach
- Student development
- Academic self-efficacy (Bandura)
- Academic self-regulation
- Campus involvement (Astin)
- Mentoring professor studies
- Student Services DO help student success
7 8Surveys - SLO Uses
- Students self-rate their competencies on program
or college level learning outcomes. - Students satisfaction with various student
services.
9Types of Questions
- Open-ended respondents answer in own words
- Closed-ended respondent limited to a finite
range of choices
10Types of Questions
- Open-ended
- Flexible
- Hard to code answers
- Good for preliminary work to finalize a survey
- Closed-ended
- Easier to code answers process and analyze
- Hard to write good closed-ended items
11Item Format
- Visual Analogue Scale
- Food in the cafeteria is
- Poor_ _ __ _ _ _ _ _ _ _ _ _ _ _ _ _Excellent
- Likert Scale
- Food in the cafeteria is outstanding!
- SD D N A SA
- (Strongly Agree) (Disagree) (Neutral)
(Agree) (Strongly Agree)
12Nine tips for designing and deploying a survey
- Dont call it a survey
- Provide a carefully worded rationale or
justification at the beginning - Group items by common format
- Start with more interesting items
- Put demographic items last
- Mix in negative wording to catch acquiescence
(aka response set) - Automate scoring when possible
- If asking for sensitive information use
procedures designed to assure anonymity - Always always always pilot test first
13Survey Administration Methods
- Face to Face
- Written
- Group administration
- Mail
- Computerized http//research.ccc.cccd.edu
- Password protected
- Validation rules
- Branching and piping
- Telephone
14Focus Groups
Focus groups can be especially insightful and
helpful for program and institutional level
learning outcome assessment. Have your college
researcher provide some background materials.
Focus Groups A Practical Guide for Applied
Research By Richard A. Krueger Mary Anne
Casey The RP Group sponsored several drive in
workshops over the last few years.
15Goal for This Section
- Technology Uses
- Technology ToolsExpected Outcome Be able to
select and use technology-based approaches to
assess student learning outcomes
16Assessment Challenges
- Engaging Students in Self-Evaluation
- More Efficient Assessment
17Some Technology Tools
- Online Rubric Builders
- eLumen (SLO Assessment/Tracking)
- Classroom Responders (Clickers)
- Scannable and Online Tests and Surveys
- ePortfolios
- Adobe Acrobat Forms
- Excel Spreadsheets
18Rubrics
- Way to measure the heretofore immeasurable
products and performances. - A rubric breaks the assessment into important
components. - Each component rated along a scale well-labeled
scale.
19 Lets Develop an Assessment Rubric for a
Resume
20Chocolate Chip Cookie Rubric
Chocolate Chip Cookie Rubric
21 Rubrics are Good!
- Facilitate staff dialogue regarding satisfactory
performance. - Create a more objective assessment.
- Make expectations more explicit to the student.
- Encourage metacognitive skill of self-monitoring
own learning. - Facilitate scoring and reporting of data.
22Online Discussion Rubric
http//www.uas.alaska.edu/sitka/IDC/resources/onli
neDiscussionRubric.pdf
23Design Your Own Rubric
- Please work in groups and use the worksheet in
your packet to design a scoring rubric for
assessing one of the following - Coffee shops
- Syllabi
- Customer service at retail stores
- Grocery stores
- Online courses
24Online Rubric Builders
- Rubrics to guide and measure learning
- Tools
- Rubistar http//rubistar.4teachers.org
- Landmark Rubric Machine http//landmark-project.co
m/rubric_builder
25Rubistar
Rubistar Art History Rubric
26Rubric Builder Screen Shot
27Adobe Acrobat Forms
- Make form using MS Word
- Import form and save as PDF form
- Adjust the fields
- Add fields to tally sub scores and total scores
28(No Transcript)
29(No Transcript)
30(No Transcript)
31How Do You Report Results
32eLumen to Assess SLOs
- Reduce Time Spent Creating Reports
- Assess Course Program and/or Degree-Level
Outcomes - Share Assessment Rubrics Across Classes and
Programs - View Individual or Aggregated Results
- Use Online or Offline
http//www.elumen.info
33Use Online or Offline
34Criterion-Based Assessment
- Rubrics are attached to each SLO
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
35Rubrics Describe Criteria
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
36Library of Degree-Level SLOs
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
37And Rubrics Link to SLOs
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
38Science and Gen Ed SLOs/Rubrics
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
39Scorecard for All Students in the Course
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
40Class Scores by Student
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
41Aggregated Data for Course
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
42Course Aggregates by Program
Excerpted from eLumen A Brief Introduction by
David Shupe July 2007
43Classroom Responders
- Engage students
- Monitor student understanding
- Quickly and easily collect and store assessment
data - Use publisher item banks or create your own
44Renaissance Classroom Response System
45Most valuable tip is
- Finding ways to use technology to make SLO and
SSO assessment easier and more efficient - Concentrating SLO work on skills students have
difficulty mastering - Building SLOs around student development
(self-efficacy goal clarity etc.)
46Renaissance Learningfor clicker training
resources
http//www.renlearn.com
47Scanning Technology
- A way to gather survey input from students
- A way to test students knowledge
http//www.scantron.com and http//www.renlearn.co
m
48Surveys and Tests
- Online or Scannable
- Surveys
- Pre and post surveys of student self evaluation
of progress - Gather stakeholder (faculty business community
leaders advisory groups) input on expected
learning outcomes - Student satisfaction with service (SSO)
- Quizzes/Tests
- Practice and graded
49Some Survey Software Options
- Scannable surveys and quizzes - Optical Mark
Reader by Remark (OMR Remark) - Need software and a Fujitsu scanner
- Use word processor to create scannable bubble-in
surveys or answer sheets. - Produces item analysis output.
- Online survey tools
- eListen (Scantron Co.)
- SelectSurvey.net http//www.classapps.com/SelectSu
rveyNETOverview.asp
50Excel Spreadsheets
- Example of autoscoring and record keeping in a
Japanese Program.
51ePortfolios
- Advantages
- Document artifacts of learning
- Support diverse learning styles
- Authentic assessment
- Course program or degree-level tracking
- Job skill documentation
- Proprietary or Open Source
- ePortfolio and Open Source Portfolio
52ePortfolio.org Assessment
Lock Assignments after submission Random
selection of assignments by learning
objective Anonymity of the student who produced
the assignment and the instructor Access to the
work and the scoring rubrics Reports aggregate
scores generate frequencies/means Ability to
download raw data which can be analyzed in
another format
http//www.eportfolio.org
53Open Source Portfolio
- Aligned with Sakai
- Admins or Faculty can structure and review work
- Learning matrix documents levels of work
http//www.osportfolio.org
54Resources
- eListen http//www.elisten.com
- eLumen http//www.elumen.info
- ePortfolios
- ePortfolio.org http//eportfolio.org
- Open Source Portfolio http//www.osportfolio.org/
- For others see EduTools ePortfolio product
comparison http//eportfolio.edutools.info/item_l
ist.jsppj16 - Online Rubric Builders
- Rubistar http//rubistar.4teachers.org
- Landmark Rubric Machine http//landmark-project.c
om/rubric_builder/index.php - Coastline Rubric Builder http//rubrics.coastline
.edu - Remark Survey Software http//www.principiaproduc
ts.com/web/index.html - Renaissance Classroom Responders
http//www.renlearn.com/renresponder/ - SelectSurvey.Net http//www.classapps.com/SelectSu
rveyNETOverview.asp
55Contact Info Acknowledgements
- Dr. Jerry Rudmann
- Professor of PsychologyIrvine Valley
Collegejrudmann_at_ivc.edu - Much of thisshow was adapted (with the
express written permission) from Pat Arlington
Instructor/Coordinator Instructional
ResearchCoastline Community Collegeparlington_at_co
astline.edu
56Development of New SLO Measures Procedure
Findings Conclusions and Recommendations from a
Recent Exploratory Study
57Purpose of the Study
- The study was designed to explore whether
assessment tools used to measure cognitive
variables -- e.g. goal clarity self-efficacy --
could serve as learning outcome measures in
Student Services.
58The Spark for this Study
- The need for truly appropriate and really useful
assessment measures in Student Services. - Ideas generated by interviews with counselors.
59Possible Relationships
- Students Academic Outcomes
- Short term outcomes
- Semester GPA
- Units earned
- units completed
- Return next semester
- Long range outcomes
- GPA
- Units earned
- Certificate degree and/or transfer
- Attributes of New Students
- Confidence level
- Goals
- Motivation
- Study skills and habits
60Possible Relationships
- Students Academic Outcomes
- Short term outcomes
- Semester GPA
- Units earned
- units completed
- Return next semester
- Long range outcomes
- GPA
- Units earned
- Certificate degree and/or transfer
- Attributes of New Students
- Confidence level
- Goals
- Motivation
- Study skills and habits
Student Services College success
courses Academic counseling Career Center
presentations Career counseling Career
course Club team chorus band student
government or other form of social
connectedness Formal and informal recognition for
progress Non-academic counseling Transfer Center
programs Peer advisors Tutoring center University
tours
61Procedure
- Counselor interviews (preliminary brainstorming)
- Literature survey for promising assessments tools
- Recruitment presentations at Region 8 DSPS and
EOPS meetings - A website was created having all assessments
online
62Study Website
63Measures We Tried
- Academic and Career Goal Clarity
- Academic Self-Efficacy
- Dispositional Hope
- Self-Regulation
- Optimism
- Positive Affect
- Negative Affect
64Summary and Examples of Measures Found Most
Useful in This Study
- Academic Self-Efficacy
- Beliefs about ones capabilities to learn or
perform at designated levels. Compared with
students who doubt their learning capabilities
those who feel efficacious for learning or
performing a task participate more readily work
harder persist longer when they encounter
difficulties and achieve at a high level. - I know how to schedule my time to accomplish
tasks. - I know how to study to perform well on tests.
- Academic Self-Regulation
- Confidence in ability to perform various academic
tasks. - I can take notes of class instruction.
- I know how to use the library to get information
for assignments. - Academic and Career Goal Clarity
- Measures clarity of immediate and long range
academic plans and extent to which student has
career clarity. - I have worked with a counselor to develop a plan
listing the courses I need to complete my lower
division coursework. - I have decided on an academic major.
- I am familiar with the daily work routine for
people working in my desired career.
65Participation
- DSPS study
- Seven colleges
- Students
- Pre-test - 142
- Post-test -127
- EOPS study
- Six colleges
- Students
- Pre-test - 276
- Post-test - 154
66Descriptive Statistics from the DSPS Study
67Intercorrelations Among Scales and Academic
Outcomes (DSPS study)
68Predicting Academic Outcomes (Based on
Correlation Matrix and Stepwise Regression
Analyses)
69Impact of Services on Student Outcomes
70DSPS Study Self Regulation Receipt of
Counseling and Semester GPA
Low SR - Yes Counseling 32 Low SR - No
Counseling 31 High SR - Yes Counseling
20 High SR - No Counseling 20 (N 103)
71DSPS Study Self Regulation Receipt of
Counseling and Percentage of Units Earned for
Units Attempted
Low SR - Yes Counseling 32 Low SR - No
Counseling 32 High SR - Yes Counseling
20 High SR - No Counseling 20 (N 103)
72EOPS Study Self Regulation Peer Advising and
Semester GPA
73Changes in Goal Clarity and Receipt of Transfer
Assistance
74Changes in Goal Clarity and Receipt of Peer
Advisement
75Limitations of Study
- Lack of random selection and assignment to
treatments - Self-selection bias
- Results are correlational not causal
- Data are an aggregate from the participating
colleges but there may be significant
differences among colleges procedures services
personnel etc.
76Thoughts
- The instruments are
- Inexpensive easy to complete and score
- Can help identify at risk students
- Can help formulate appropriate ways to assist
students - Gain scores derived from pre to post-test
assessments can be useful - Use as SLO assessment instruments that are good
matches to the services provided within Student
Services
77Recommendations
- Map your services to the constructs measured by
these instruments. - Develop new interventions where none currently
exist. - Create an assessment referral system.
78The Three Instruments Found Most Useful
- Academic and Career Goal Clarity (Tucker
Rudmann 2006) - Measures overall clarity and sub-components
of goal clarity - Academic Self-Efficacy (Chemers Hu Garcia
2001) - Measures confidence in reaching positive
academic outcomes - Efficacy for Self-Regulated Learning (Zimmerman
Bandura Marinez-Pons 1992) - Measures confidence in ones ability to manage
and regulate academic tasks students face in
college
79Goal Clarity Instrument Structure
80One Interpretation
81Converting to Adobe Acrobat Interactive Form
82Current Potential Services for Enhancing These
Important Student Learning Outcomes
83Research Team
- Jerry Rudmann PhD
- Professor of Psychology
- Irvine Valley College
- jrudmann_at_ivc.edu
- Kari Tucker PhD
- Professor of Psychology
- Department Chair
- ktucker_at_ivc.edu
- Shañon Gonzalez MA
- Coastline College
- Research Assistant III
- sgonzalez_at_coastline.edu
84Four Sources of Efficacy Beliefs
- Mastery experiences -- Outcomes interpreted as
successful raise efficacy those interpreted as
failures lower efficacy - Vicariously success or failure of models
- Verbal persuasions by others positive or
negative appraisals by others - Physiological states (e.g. anxiety stress
arousal fatigue mood) act as information about
efficacy beliefs and can raise or lower efficacy