Models for Future Comparative Measurement of Higher Education Learning: Lessons from the Collegiate Learning Assessment Longitudinal Study in the U.S.*

1
Models for Future Comparative Measurement of
Higher Education Learning Lessons from the
Collegiate Learning Assessment Longitudinal Study
in the U.S.

• Richard Arum
• New York University and
• Social Science Research Council

Josipa Roksa (University of Virginia) and
Melissa Velez (NYU) collaborated on research
findings presented here. We thank Ford and Lumina
Foundations for their generous financial support
and the Council for Aid to Education for
assistance with data collection.
2
College Learning in the Spotlight (U.S. Policy
Context)

• As other nations rapidly improve their higher
  education systems, we are disturbed by evidence
  that the quality of student learning at U.S.
  colleges and universities is inadequate, and in
  some cases, declining.
• A Test of Leadership
• U.S. Secretary of Educations Commission
• on the Future of Higher Education (2006)

3
College Learning in the Spotlight (U.S. Policy
Context)

• These shortcomings have real-world
  consequences. Employers report repeatedly that
  many new graduates they hire are not prepared to
  work, lacking the critical thinking, writing and
  problem-solving skills needed in todays
  workplaces.
• A Test of Leadership
• U.S. Secretary of Educations Commission
• on the Future of Higher Education (2006)

4
Measurement of Learning in U.S. Higher Education

• Dearth of direct measures of higher education
  student learning that are comparable across
  institutions and/or states
• Measuring Up 2008 Assigned a grade of
  Incomplete to all states in the area of measuring
  learning All states receive an incomplete in
  learning because there are not sufficient data to
  allow meaningful state-by-state comparisons.

5
Measurement Challenges

• Curriculum varies widely across fields of study
  and institutions little consensus on what is to
  be learned
• Practitioner resistance to reductionist
  approaches
• Students are sorted by ability and other factors
  into different institutions

6
Collegiate Learning Assessment (CLA)

• Dimensions of learning assessed
• critical thinking, analytical reasoning, and
  written communication
• Distinguishing characteristics
• Direct measures (as opposed to student reports)
• NOT multiple choice
• Holistic assessment based on open-ended prompts
  representing real-world scenarios

7
Collegiate Learning Assessment (CLA)

• Components
• Performance task
• Make an argument
• Break an argument

8
Performance Task (example)

• You are the assistant to Pat Williams, the
  president of DynaTech, a company that makes
  precision electronic instruments and navigational
  equipment. Sally Evans, a member of DynaTechs
  sales force, recommended that DynaTech buy a
  small private plane (a SwiftAir 235) that she and
  other members of the sales force could use to
  visit customers. Pat was about to approve the
  purchase when there was an accident involving a
  SwiftAir 235.

9
Performance Task (example, cont.)

• Students are provided with a set of materials
  (e.g. newspaper articles, Federal Accident
  Report, e-mail exchanges, description and
  performance characteristics of AirSwift 235 and
  another model, etc.) and asked to prepare a memo
  that addresses several questions, including what
  data support or refute the claim that the type of
  wing on the SwiftAir 235 leads to more in-flight
  breakups, what other factors may have contributed
  to the accident and should be taken into account,
  and their overall recommendation about whether or
  not DynaTech should purchase the plane.

10
Determinants of College Learning Dataset

• Longitudinal Design
• Fall 2005 and Spring 2007 (beginning of freshman
  and end of sophomore years)
• Large Scale
• 24 diverse four-year institutions 2,341 students
• Breath of Information
• Family background and high school information,
• college experiences and contexts, college
  transcripts
• Collegiate Learning Assessment (CLA)

11
Sample Characteristics Who are These Students?
CLA Analysis Sample IPEDS CLA Schools IPEDS All Schools
Demographics
Male 0.37 0.46 0.45
White 0.59 0.61 0.59
African-American 0.19 0.14 0.13
Hispanic 0.05 0.08 0.13
Asian 0.11 0.10 0.06
Test Scores
SAT, 25th percentile 1052.83 995.15 993.14
SAT, 75th percentile 1212.83 1219.02 1219.23
ACT, 25th percentile 22.05 20.86 20.33
ACT, 75th percentile 26.29 25.77 25.31
12
Research Questions

• What individual, social and institutional factors
  are associated with learning in higher education?
• How do disadvantaged groups of students fare in
  college in terms of measured learning?
• To what extent do individual, social and
  institutional factors account for variation
  across disadvantaged groups?

13
Overview of the Conceptual Model Employed in the
Study
14
Analysis - Part I

• Individual, Social and Institutional Factors
  Associated with Learning as Measured by
  Improvement in CLA Performance

15
High School Preparation
16
College Engagement and Learning
Figure 2. Predicted 2007 Score by College
Engagement and Involvement Measures
17
College Employment and Learning
Figure 3. Predicted 2007 Test Score by Employment
Measures
18
Faculty Expectations and Learning
Figure 4. Predicted 2007 Test Score by Level of
Faculty Expectations
19
Fields of Study and Learning
Figure 5. Predicted 2007 Test Score by College
Major
20
Analysis - Part II

• Social Disadvantaged Group Differences in
  Learning as Measured by Improvement in CLA
  Performance

21
CLA Performance by Race
Figure 6. 2005 and 2007 Test Scores by
Race Note average growth34.32 standard
deviation188 (Fall 05), 211 (Spring 07)
22
CLA Performance by Parental Education
Figure 7. 2005 and 2007 Test Scores by Parental
Education Note average growth34.32 standard
deviation188 (Fall 05), 211 (Spring 07)
23
CLA Performance by High School Student
Composition and Home Language
Figure 8. 2005 and 2007 Test Scores by Level of
High School Student Composition and Home
Language Note average growth34.32 standard
deviation188 (Fall 05), 211 (Spring 07)
24
Analysis - Part III

• Accounting for Variation in CLA Performance by
  Social Disadvantaged Groups

25
Accounting for Group Differences H.S.-College
Experiences and Institutional Differences
Figure 11. Test score gaps in baseline and full
models with college institutional fixed effects.
Note Baseline regression model predicts the
2007 score, controlling for the 2005 score and a
range of background characteristics. Full model
also includes measures of high school academic
preparation and college experiences.
Non-significant differences are shaded.
26
Conclusions and Implications

• Policy makers need to focus attention on
  improving individual student learning in higher
  education, not just access and retention.
• Practitioners need to recognize the extent to
  which both student experiences as well as
  institutional differences are associated with
  variation in learning.
• Additional systematic longitudinal research is
  necessary to improve understanding of these
  processes.
• Measurement of learning across fields and
  institutions is possible with instruments such as
  the CLA.