A Longitudinal Survey Program to Meet Institutional and Academic Department Assessment Needs

1
A Longitudinal Survey Program to Meet
Institutional and Academic Department Assessment
Needs
AIRPO Annual Conference West Point, New York
June 16-18, 2004
Bruce P. Szelest bszelest_at_albany.edu
2
Albany History and Forward-looking Needs

• Long tradition in assessing general student
  outcomes
• W.K. Kellogg Foundation support
• Albany outcomes assessment model
• Ad hoc studies
• Albany assessment report series
• http//www.albany.edu/ir/reports.htm
• Future-oriented developments
• Middle States accreditation requirements
• Campus assessment plan
• Building a campus culture that embraces
  assessment at the unit level as well as at the
  institutional level

3
The Albany Outcomes Assessment Model
Personal Traits
Alumni Outcomes
Pre-College Characteristics
Educational Outcomes
Graduate Degrees Earned
Academic
Age, Ethnicity, Gender
College Experiences
Arts Letters Disciplinary Study Scientific
Method Academic Achievement Persistence/Graduation
Acad Integration
Occupational Status
Parents Education
Acad Conscientiousness Classroom
Experiences Faculty Contact Field of Study
Aptitude Test Scores
Income Level
High School Achievement
Satisfaction with Job and Career
Personal
Social Integration
College Preparation
Peer Relations Employment Extra-Curr.
Activities Residential Experience
Interpersonal Skills Openness
Tolerance Functioning Independently
Leadership Service
Parental Influence
Awards Recognition
Involvement Receptivity to Services Educational
Aspirations
Instl Integration
Alumni Giving
Instl Commitment Advisement Financial
Aid Affinity of Values
Acad Motivation Intrinsic,
Extrinsic Self-regulation
4
The Albany Longitudinal Survey Program
Year 1 Year 2 Year 3 Year 4 Entering
Student Senior Experience Survey
Survey Freshmen Student Experience Survey
Surveys administered every year
5

• Entering Student Survey
• Focus on student pre-college characteristics to
  identify students at risk prior to first semester
  of study
• Concentrate institutional resources on students
  in-need of attention - individual student
  profiles
• Communicate general findings back to new freshmen
  to raise awareness of successful student profiles
• One leg of the Advisement Services Centers
  tripartite assessment program
• http//web.albany.edu/ir/samplesurveys/ess.html

6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
Pre-college Scales
Parental Influence Composite of mother's and
father's education My parents encourage me to
do the best in whatever I do I can go to my
parents when I have a problem Educational
Aspirations (scale importance) To gain a
broad, well-rounded education (in the sciences,
humanities, and arts) To learn to think
creatively and analytically To learn more
about myself, my values, and my life's
goals To learn more about other cultures and
groups of people. Involvement (scale
degree of chance) Participate in
intercollegiate or intramural sports Belong to
a fraternity or sorority Join a student club
(academic, political, or cultural) Participate
in student government, student newspaper, and/or
campus radio station Receptivity to Services
(scale degree of chance) Visit professors
during their office hours Attend a study group
from one of your classes Contact my academic
advisor more than once per semester Visit the
Career Development Center College Preparation
(scale yes or no) Pre-calculus or
calculs Foreign language Science Advanced
placement (AP) courses
11
Pre-college Scales contd
Academic Motivation - Intrinsic (scale not
true to true) I enjoy the classes that
challenge my mind. I will feel successful if I
have the highest test scores. Thinking about my
future motivates me to work hard in
school. Solving difficult problems in my classes
gives me a sense of satisfaction. I feel
successful when I learn something
interesting. Learning new things gives me a
sense of accomplishment. Schoolwork is
interesting because you see yourself making
progress. I read books not assigned in
classes. Academic Motivation - Extrinsic
(scale not true to true) I study hard because
I want to prove myself as capable as anyone else
in class. I worry about not doing as well
academically as others in college. I don't feel
I can gain much in college other than obtaining a
degree. I feel that as long as my teachers are
satisfied with my academic performance, that is
enough. I don't really care how well I will do
in college, as long as I can pass. Getting a
high paying job is the main reason I study hard
in college. I feel ashamed if I don't do as well
as other students do in college. I worry about
failing exams and getting into academic
trouble. Self-Regulation (scale not true to
true) When I get something wrong in school, I
always stop and try to find out what went wrong
rather than simply asking my instructor the
correct answer. I take notes while reading
textbooks. When I run into difficulties in
schoolwork, I feel if I put in extra effort, I
will eventually figure them out. When I am
reading my textbooks, I stop and check whether I
really understand what I have just
read. If I study harder my academic abilities
will improve. I keep a regular study
schedule. When learning a new concept, I never
check if I really understand its meaning.
12

• Entering Student Survey
• Focus on student pre-college characteristics to
  identify students at risk prior to first semester
  of study
• Concentrate institutional resources on students
  in-need of attention -individual student profiles
• Communicate general findings back to new freshmen
  to raise awareness of successful student profiles
• One leg of the Advisement Services Centers
  tripartite assessment program
• http//web.albany.edu/ir/samplesurveys/ess.html

13

• Freshmen Student Experience Survey
• Examine changes in students psychological
  make-up after the first college year
• Gauge institutional engagement and relationships
  with educational outcomes
• First pass at gauging centralized student
  advisement
• Communicate general findings back to students and
  faculty to support the development of a campus
  culture of assessment
• http//web.albany.edu/ir/samplesurveys/froshses.ht
  ml

14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
College Experience Scales Items
Academic Integration
Academic Conscientiousness (scale various 5 pt
scales) How frequently have you worked harder
than you thought you could to meet an
instructor's expectations? Developed your
study skills. Utilized your study skills. I do
very little studying on weekends. I put a good
deal of effort into being well prepared for
examinations. My primary goal at UAlbany is to
take advantage of academic opportunities. Doing
homework and studying outside of
class. Classroom Experiences (scale never to
almost always) Been satisfied with your
academic experiences? Had out-of-class
assignments that were good learning
experiences? Worked with other students on class
assignments? Received feedback (written or oral)
from instructors on the quality of your
work? Had to critically appraise the value of
information, arguments, or methods in your
classes or class assignments? Faculty Contact
Items How frequently have you had discussions,
meetings, or conversations with instructors
outside of class? Visit professors during
their office hours? Seek out a faculty mentor?
26
College Experience Scales Items contd
Social Integration
Peer Relations (scale disagreement to
agreement) I have developed strong friendships
with other students. My interpersonal
relationships with other students have had a
positive influence on my personal growth. The
student friendships I have developed have been
personally satisfying. It has been difficult for
me to make friends with other students. I know
several UAlbany students who would be willing to
help me if I had a personal problem. My
interpersonal relationships with other students
have had a positive influence on my
intellectual growth. I am confident in my
ability to make friendships. Extra-curricular
activities (scale yes or no) Participate in
intercollegiate or intramural sports?. Belong to
a fraternity or sorority? Employment Working
on-campus (volunteer or paid). Working
off-campus (volunteer or paid).
27
College Experience Scales Items contd
Institutional Integration
Institutional Commitment (scale yes or
no) Visit the Writing Center? Visit the Career
Development Center? Contact your academic
advisor more than once per semester? Financial
Aid Advisement Items (scale disagreement to
agreement) Advisor was knowledgeable about
course requirements for the major(s). Advisor
was knowledgeable about course requirements for
the minor(s). Advisor was helpful in assisting
with scheduling/ registration procedures.
Advisor showed genuine interest in my academic
progress. Advisor went over my degree audit with
me to inform me of course requirements I have
not yet met. Advisor was knowledgeable about
academic policies (e.g., graduation requirements,
GPA requirements, General Education
guidelines, residence requirements, etc.).
Advisor was available when I needed
him/her. Advisor showed concern for my personal
growth and development. Advisor helped me to
identify career areas (or graduate school
opportunities) which fit my skills,
abilities, and interests Advisor encouraged me
to talk about myself and my college
experience. Advisor was easy to talk with.
Advisor promptly returned my phone calls,
e-mails, and/or messages.
28
Educational Outcomes
Academic (scales none to very large college
contribution)
Arts Letters Writing effectively. Developing
intellectual curiosity. Developing the ability
to formulate creative ideas and/or
solutions. Appreciating artistic and creative
expression. Placing current problems in
historical perspective. Scientific
Method Evaluating ideas, materials, and methods
critically. Developing problem solving
skills. Thinking analytically and
logically. Understanding scientific
findings. Understanding mathematical
concepts. Disciplinary Study Gaining factual
knowledge. Learning how to learn. Synthesizing
a body of information. Understanding a
particular discipline's research
methods. Understanding a particular discipline's
various schools of thought. Understanding the
inter-relatedness of different fields of
study. Using information and/or computer
technology in your academic discipline
29
Educational Outcomes contd
Social (scales none to very large college
contribution)
Interpersonal Skills Speaking effectively.
Developing interpersonal and social
skills. Functioning effectively as a member of a
team. Preparing for active participation in a
democratic society. Coping with
conflict. Adapting to different social
situations. Openness Tolerance Understanding
cultural differences. Gaining exposure to a
variety of new intellectual areas. Developing an
openness to new ideas. Coping with moral and
ethical issues. Developing a better
understanding of myself (e.g., interests,
talents, values, limitations). Understanding
cultural differences. Relating well to people of
different races Functioning Independently Funct
ioning independently (self-reliance). Exercising
personal responsibility. Developing a better
understanding of myself.
30

• Freshmen Student Experience Survey
• Examine changes in students psychological
  make-up after the first college year
• Gauge institutional engagement and relationships
  with educational outcomes
• First pass at gauging centralized student
  advisement
• Communicate general findings back to students and
  faculty to support the development of a campus
  culture of assessment
• http//web.albany.edu/ir/samplesurveys/froshses.ht
  ml

31

• Senior Survey
• Examine changes in students psychological
  make-up over their UAlbany career
• Gauge institutional engagement and relationships
  with educational outcomes
• Connect engagement and educational outcomes with
  student experiences in the major
• Focus on academic advisement in the major
• Communicate general findings back to students,
  faculty, and academic units to support the
  development of a campus culture of assessment
• http//web.albany.edu/ir/samplesurveys/seniorses.h
  tml

32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
Entering Student Survey ID at-risk
students General profiles of successful students
Personal Traits
Alumni Outcomes
Pre-College Characteristics
Educational Outcomes
Graduate Degrees Earned
Academic
Age, Ethnicity, Gender
College Experiences
Arts Letters Disciplinary Study Scientific
Method Academic Achievement Persistence/Graduation
Acad Integration
Occupational Status
Parents Education
Acad Conscientiousness Classroom
Experiences Faculty Contact Field of Study
Aptitude Test Scores
Income Level
High School Achievement
Satisfaction with Job and Career
Personal
Social Integration
College Preparation
Peer Relations Employment Extra-Curr.
Activities Residential Experience
Interpersonal Skills Openness
Tolerance Functioning Independently
Leadership Service
Parental Influence
Awards Recognition
Involvement Receptivity to Services Educational
Aspirations
Instl Integration
Alumni Giving
Instl Commitment Advisement Financial
Aid Affinity of Values
Acad Motivation Intrinsic,
Extrinsic Self-regulation
Freshmen Senior Surveys Expansion of the Albany
Model Gauge institutional engagement Evaluate
advisement (centralized in the major) Build
culture of assessment (generally in acad units)
37

• Pros
• Longitudinal survey program tailored to
  institutional needs
• Demonstrates institutional commitment to
  self-examination
• Amenable to post-hoc analyses of special student
  populations
• Immediate impact in enhancing student retention
• Promotes academic unit use and participation
• Cost effective approach

• Cons
• No benchmarking ?
• Subject to validation and reliability criticisms
• Considerable start-up investment by IR staff and
  collaborators
• Requires IT infrastructure and support

38
END