Capturing the Learning That Matters

1
Capturing the Learning That Matters

• Terrel L. Rhodes
• Association of American Colleges and Universities
• Williamsburg Invitational
• Accountability Today and Tomorrow
• October 6-8, 2009

2
What is AACU?

• Founded in 1915,
• AACU is dedicated to
• making the aims of liberal
• learning a vigorous and
• constant influence on
• institutional planning and
• educational practice in college.
• It is a meeting ground for all
• sectors of higher education and
• brings together faculty, academic
• and student affairs leaders and
• presidents across sectors,
• divisions, and disciplines to
• explore the aims of education, the
• future of the academy, and
• strategies for institutional change
• and higher student achievement.

3
Whats Next on Accountability and Assessment?
4
Assessment and Accountability What are the
Connections?

• Higher education institutions are accountable to
  students, to local communities, to funderspublic
  and private, and international/policy makers.
• Higher education institutions assess their
  success in order to demonstrate that they are
  accountable and worth investment.
• But, assessing student learning has other
  purposes beyond just accountability.

5
Why Assess? The Multiple Purposes

• To learn what is working and what is not and to
  improve programs, courses, departmental outcomes.
• To enable students to see their own progress and
  learn how to improve over time.
• To provide information students can use to
  demonstrate what they know and can do to future
  employers, colleges and others.

6
Two Paradigms of Assessment
Ewell, Peter T. (2007). Assessment and
Accountability in America Today Background and
Context. In Assessing and Accounting for Student
Learning Beyond the Spellings Commission. Victor
M. H. Borden and Gary R. Pike, Eds. Jossey-Bass
San Francisco.
7
LEAP Essential Learning Outcomes

• Knowledge of Human Cultures and the Physical and
  Natural World
• Through study in the sciences and mathematics,
  social sciences, humanities, histories,
  languages, and the arts
• Focused by engagement with big questions, both
  contemporary and
• enduring
• Intellectual and Practical Skills, including
• Inquiry and analysis
• Critical and creative thinking
• Written and oral communication
• Quantitative literacy
• Information literacy
• Teamwork and problem solving
• Practiced extensively, across the curriculum, in
  the context of
• progressively more challenging problems,
  projects, and standards for
• performance

8
LEAP Essential Learning Outcomes

• Personal and Social Responsibility, Including
• Civic knowledge and engagementlocal and global
• Intercultural knowledge and competence
• Ethical reasoning and action
• Foundations and skills for lifelong learning
• Anchored through active involvement with diverse
  communities and real
• world challenges
• Integrative Learning, including
• Synthesis and advanced accomplishment across
  general and specialized
• Demonstrated through the application of
  knowledge, skills, and
• responsibilities to new settings and complex
  problems

9
How Should Colleges Assess Improve College
Learning?
Employers Views on the Accountability Challenge
Key findings from survey among 301
employersConducted November 8 December 12,
2007for
10
Employers Find College Transcripts Of Limited Use
In Evaluating Potential
How useful do you find the college transcript in
helping you evaluate job applicants potential to
succeed at your company?
Not sure
Very useful
Fairly useful
Not useful
Just somewhat useful
10
11
Assessments Effectiveness In Ensuring College
Graduates Have Skills/Knowledge
Supervised/evaluated internship/community-based
project where students apply college learning in
real-world setting
83
Advanced comprehensive senior project, such as
thesis, demonstrating students depth of
knowledge in major problem-solving, writing,
and analytic reasoning skills
79
Essay tests to evaluate level of problem-solving,
writing, and analytical-thinking skills
60
Electronic portfolio of students college work,
including accomplishments in key skill areas and
faculty assessments
56
Multiple-choice tests of general content knowledge
32
11
12
Assessments Usefulness In Helping Employers
Evaluate College Graduates Potential
Faculty supervisors assessment of applicants
student internship/ community-based project
applying college learning in real-world setting
67
Sample of applicants student senior project and
overview of faculty assessment of the project
61
Electronic portfolio of applicants college work,
including accomplishments in key skill areas and
faculty assessments
56
Applicants score on essay test to evaluate level
of problem-solving, writing, and
analytical-thinking skills
54
Applicant colleges score showing how the college
compares to others in advancing students
critical-thinking skills
36
Applicants score on multiple-choice test of
general content knowledge
29
12
13
Employer Perspective

• It really matters very little if students can
  perform well on multiple-choice tests. We need
  more of them to be able to communicate, analyze,
  think critically.
• Wayne Johnson, Hewlett-Packard

14
An obvious place to begin

• Tell The Students What They Are Expected to
  Accomplish

15
The Power of Rubrics for Assessing Learning

• Help guide students around learning expectations
• Help guide faculty
• Reinforces assessment for learning
• Places individual faculty judgment within
  national shared experience
• Can build up from course level to institutional
  reporting needs AND down from general to specific
  program/course context

rhodes_at_aacu.org
16
VALUE Advisory Board

• Randy Bass, Georgetown University
• Marcia Baxter Magolda, Miami University, Ohio
• Veronica Boix Mansilla, Harvard University
• Johnnella Butler, Spelman College
• Helen Chen, Stanford University
• Ariane Hoy, The Bonner Foundation
• George Kuh, Indiana University Bloomington
• Peggy Maki, Peggy Maki Associates
• Marcia Mentkowski, Alverno College
• Gloria Rogers, ABET, Inc.
• Robert Sternberg, Tufts University
• Kathleen Blake Yancey, Florida State University

17
ELOs become Rubrics

• Inquiry and analysis
• Critical thinking
• Creative thinking
• Written communication
• Oral communication
• Quantitative literacy
• Information literacy
• Teamwork
• Problem solving
• Reading
• Civic knowledge and engagementlocal and global
• Intercultural knowledge and competence
• Ethical reasoning
• Foundations and skills for lifelong learning
• Integrative learning

18
Rubric Process

• Team of faculty from around the country
• Subject matter experts, or
• Assessment experts
• Novices
• Review rubrics stored at openedpractices.org

19
Openedpractices.org

• A website open to everyone in higher education
  
• Free to join, easy to sign up
• openedpractices.org
• A library of sortsa site where a collection of
  rubrics (and other best practices) can be
• Stored
• Shared
• Tagged
• Built with open source software to share
  knowledge and best practices openly

20
opened

21
opened

22
Rubric Development Teams

• Agnes Scott College
• Alverno College
• Association of American College Universities
• Atlantic Cape Community College
• Augustana College
• Azusa Pacific University and Claremont Graduate
  University
• Bellevue Community College
• Bowling Green State University
• California State University, Sacramento
• Carleton College
• Christopher Newport University
• College of San Mateo
• College of Wooster
• Columbia University Teachers College
• Community College of Baltimore County
• Concordia University, St. Paul
• Council for Adult and Experiential Learning
• Eastern Michigan University
• Georgetown University

Kapi'olani Community College King's
College Longwood University Maryville
College Meredith College National Communication
Association North Carolina State University North
Hennepin Community College Portland State
University Prescott College Sacred Heart
University St. Olaf College The Bonner
Foundation United States Air Force
Academy University of Baltimore University of
Colorado, Denver University of Massachusetts
Boston University of NebraskaLincoln University
of North Dakota University of San
Francisco University of the Pacific University of
Vermont University of Wisconsin
Oshkosh Washington State University Wellesley
College Western Oregon University
23
VALUE Process

• Develop criteria from rubric collection
• Each rubric development team member wrote a row
  of performance descriptors for one criteria
• Draft rubric then discussed and debated, edited
  and transformed
• Testing and feedback e-Portfolios

24
VALUE Leadership Campuses

• Alverno College
• Bowling Green State University
• City University of New York LaGuardia Community
  College
• College of San Mateo
• George Mason University
• Kapi'olani Community College
• Portland State University
• Rose-Hulman Institute of Technology
• San Francisco State University
• Spelman College
• St. Olaf College
• University of Michigan

25
VALUE Partner Campuses

• Augustana College
• California State University, Long Beach
• Clemson University
• College of Wooster
• DePaul University
• Jackson State University
• James Madison University
• King's College
• Lorain County Community College
• Lourdes College
• Minnesota State Colleges Universities
• Morehead State University
• North Central College
• Northwest Technical College
• Oregon Institute of Technology
• Pepperdine University
• Prescott College
• Queensborough Community College

South Central College Southern Oregon
University Temple University University of
Central Oklahoma University of Central
Florida University of Delaware University of
Idaho University of Kansas University of Michigan
Flint University of Minnesota Duluth University
of North Carolina at Fayetteville University of
North Carolina at Wilmington University of the
Pacific Virginia Commonwealth University Virginia
Polytechnic Institute and State University W2eP
(Web 2.0 Eportfolios) West Liberty State
College Youngstown State University
26
VALUE Process

• Revision of rubrics based on feedback
• Repeated testing and revision
• National Panel

27
VALUE Project

• http//www.aacu.org/programs/VALUE

28
(No Transcript)
29
(No Transcript)
30
Why multiple measures?

• Good assessment practice
• We must hold ourselves accountable for assessing
  our students best work, not generic skills and
  not introductory levels of learning.
• College-level learning takes different forms for
  different majors, fields, levels.
• Diagnostic, milestone, and culminating
  assessments all play a role.

31
Is there a role for standardized tests?

• Yes, but
• Given that currently available tests only measure
  a limited set of capabilities,standardized tests
  play only a limited role in any comprehensive
  assessment or accountability approach.

32
The Time is Right

• Assessment practices are changing to include
• Qualitative not just quantitative
• Formative not just summative
• Performance over time not just one course, one
  test
• Sophisticated skills and responsibilities not
  just knowledge
• Problems with multiple solutions not just
  right answers

33
VALUE Project Results

• Brings forward a focus on student learning
  outcomes that are essential statements of
  expected learning for 15 essential outcomes
• Provides the kind of information institutions,
  faculty most importantly students need
• Reflects actual progress, growth and development
  of learning not a snapshot
• Results can be used through aggregation to
  demonstrate institutional level student learning
  for accountability purposes
• Establishes nationally shared expectations for
  learning

34
Next Steps for VALUE

• VALUE in the Disciplines being submitted for
  funding
• Open to feedback on the rubric
• Working with campuses to compare rubric
  outcomes with other measures
• VALUE Publications coming in January 2010
• Electronic portfolios Symposium in January 2010

35
Feedback

• Comments?
• Suggestions?
• Questions?

36
Alverno
37
Bowling Green
38
George Mason
39
Kapiolani CC
40
LaGuardia CC
41
Portland State
42
Rose Hulman
43
San Francisco State
44
San Mateo CC
45
Spelman
46
St. Olaf
47
University of Michigan