Title: Models for Future Comparative Measurement of Higher Education Learning: Lessons from the Collegiate
1Models for Future Comparative Measurement of
Higher Education Learning Lessons from the
Collegiate Learning Assessment Longitudinal Study
in the U.S.
- Richard Arum
- New York University and
- Social Science Research Council
Josipa Roksa (University of Virginia) and
Melissa Velez (NYU) collaborated on research
findings presented here. We thank Ford and Lumina
Foundations for their generous financial support
and the Council for Aid to Education for
assistance with data collection.
2College Learning in the Spotlight (U.S. Policy
Context)
-
- As other nations rapidly improve their higher
education systems, we are disturbed by evidence
that the quality of student learning at U.S.
colleges and universities is inadequate, and in
some cases, declining. - A Test of Leadership
- U.S. Secretary of Educations Commission
- on the Future of Higher Education (2006)
3College Learning in the Spotlight (U.S. Policy
Context)
- These shortcomings have real-world
consequences. Employers report repeatedly that
many new graduates they hire are not prepared to
work, lacking the critical thinking, writing and
problem-solving skills needed in todays
workplaces. - A Test of Leadership
- U.S. Secretary of Educations Commission
- on the Future of Higher Education (2006)
4Measurement of Learning in U.S. Higher Education
- Dearth of direct measures of higher education
student learning that are comparable across
institutions and/or states - Measuring Up 2008 Assigned a grade of
Incomplete to all states in the area of measuring
learning All states receive an incomplete in
learning because there are not sufficient data to
allow meaningful state-by-state comparisons.
5Measurement Challenges
- Curriculum varies widely across fields of study
and institutions little consensus on what is to
be learned - Practitioner resistance to reductionist
approaches - Students are sorted by ability and other factors
into different institutions
6Collegiate Learning Assessment (CLA)
- Dimensions of learning assessed
- critical thinking, analytical reasoning, and
written communication - Distinguishing characteristics
- Direct measures (as opposed to student reports)
- NOT multiple choice
- Holistic assessment based on open-ended prompts
representing real-world scenarios
7Collegiate Learning Assessment (CLA)
- Components
- Performance task
- Make an argument
- Break an argument
8Performance Task (example)
- You are the assistant to Pat Williams, the
president of DynaTech, a company that makes
precision electronic instruments and navigational
equipment. Sally Evans, a member of DynaTechs
sales force, recommended that DynaTech buy a
small private plane (a SwiftAir 235) that she and
other members of the sales force could use to
visit customers. Pat was about to approve the
purchase when there was an accident involving a
SwiftAir 235.
9Performance Task (example, cont.)
- Students are provided with a set of materials
(e.g. newspaper articles, Federal Accident
Report, e-mail exchanges, description and
performance characteristics of AirSwift 235 and
another model, etc.) and asked to prepare a memo
that addresses several questions, including what
data support or refute the claim that the type of
wing on the SwiftAir 235 leads to more in-flight
breakups, what other factors may have contributed
to the accident and should be taken into account,
and their overall recommendation about whether or
not DynaTech should purchase the plane.
10Determinants of College Learning Dataset
- Longitudinal Design
- Fall 2005 and Spring 2007 (beginning of freshman
and end of sophomore years) - Large Scale
- 24 diverse four-year institutions 2,341 students
- Breath of Information
- Family background and high school information,
- college experiences and contexts, college
transcripts - Collegiate Learning Assessment (CLA)
11Sample Characteristics Who are These Students?
12Research Questions
- What individual, social and institutional factors
are associated with learning in higher education? - How do disadvantaged groups of students fare in
college in terms of measured learning? - To what extent do individual, social and
institutional factors account for variation
across disadvantaged groups?
13Overview of the Conceptual Model Employed in the
Study
14Analysis - Part I
- Individual, Social and Institutional Factors
Associated with Learning as Measured by
Improvement in CLA Performance
15High School Preparation
16College Engagement and Learning
Figure 2. Predicted 2007 Score by College
Engagement and Involvement Measures
17College Employment and Learning
Figure 3. Predicted 2007 Test Score by Employment
Measures
18Faculty Expectations and Learning
Figure 4. Predicted 2007 Test Score by Level of
Faculty Expectations
19Fields of Study and Learning
Figure 5. Predicted 2007 Test Score by College
Major
20Analysis - Part II
- Social Disadvantaged Group Differences in
Learning as Measured by Improvement in CLA
Performance
21CLA Performance by Race
Figure 6. 2005 and 2007 Test Scores by
Race Note average growth34.32 standard
deviation188 (Fall 05), 211 (Spring 07)
22CLA Performance by Parental Education
Figure 7. 2005 and 2007 Test Scores by Parental
Education Note average growth34.32 standard
deviation188 (Fall 05), 211 (Spring 07)
23CLA Performance by High School Student
Composition and Home Language
Figure 8. 2005 and 2007 Test Scores by Level of
High School Student Composition and Home
Language Note average growth34.32 standard
deviation188 (Fall 05), 211 (Spring 07)
24Analysis - Part III
- Accounting for Variation in CLA Performance by
Social Disadvantaged Groups
25Accounting for Group Differences H.S.-College
Experiences and Institutional Differences
Figure 11. Test score gaps in baseline and full
models with college institutional fixed effects.
Note Baseline regression model predicts the
2007 score, controlling for the 2005 score and a
range of background characteristics. Full model
also includes measures of high school academic
preparation and college experiences.
Non-significant differences are shaded.
26Conclusions and Implications
- Policy makers need to focus attention on
improving individual student learning in higher
education, not just access and retention. - Practitioners need to recognize the extent to
which both student experiences as well as
institutional differences are associated with
variation in learning. - Additional systematic longitudinal research is
necessary to improve understanding of these
processes. - Measurement of learning across fields and
institutions is possible with instruments such as
the CLA.