Building a Culture of Evidence: What Leaders Need to Know

1
Building a Culture of EvidenceWhat Leaders Need
to Know

• Margaret Miller
• NEAIR Meeting
• November 2008

2
External Interest in Academic AchievementWhere
Did It Come From?

• Increasing importance of higher education
• Budget pressures
• Higher education as a business
• Culture of accountability

3
State Interest (1980s)

• Assessment for Improvement
• Mid-80s mandates in a number of states to assess
  student learning
• Generally campus based and individualized
• But,

4
State Interest (1980s)

• Assessment for Accountability
• States as owner-operators of public higher
  education systems
• Human capital question

5
National Interest (1980s)

• Assessment for Improvement
• Enforced through accreditors
• Campus based and individualized

6
National Interest (1990)

• National Education Goal 6 B (v)
• The objectives for this goal are that . . . The
  proportion of college graduates who demonstrate
  an advanced ability to think critically,
  communicate effectively, and solve problems will
  increase substantially

7
National Interest (1980s on)Employers

• Overall prep for work 64.3 adequate, 23.9
  excellent
• Percentage of college graduates deemed excellent
  in
• English Language . . . . . . . . . . 26.2
• Reading Comprehension . . . . . 25.9
• Mathematics . . . . . . . . . . . . . . 18.3
• Writing in English . . . . . . . . . . 15.5
• Science . . . . . . . . . . . . . . . . . 14.6
• Humanities/Arts . . . . . . . . . . . 9.2
• Government/Economics . . . . . . 8.1
• Foreign Languages . . . . . . . . . 6.2
• History/Geography . . . . . . . . . 5.2
• Are They Ready to Work? (2006)

8
Employers (cont.)

• AACUs LEAP Employer Survey
• 63 Too many recent college graduates do not
  have the skills to be successful in todays
  economy.
• 79 All or most have the skills to succeed
  in entry-level positions.
• 46 All or most have the skills to advance
  or be promoted.

9
Adult LiteracyPercentage of proficient 4-year
college graduates, 1992 2003

• 1992 2003
• Prose 40 31
• Document 37 25
• Quantitative 39 36

• SOURCE U.S. Department of Education, Institute
  of Education Sciences, National Center for
  Education Statistics, 1992 National Adult
  Literacy Survey and 2003 National Assessment of
  Adult Literacy.

10
International Comparisons Literacy

• Adults with some college (2001)
• Country Prose Document Quantitative
• Sweden 341 339 333
• Finland 336 341 323
• Germany 329 345 344
• Norway 327 341 331
• Czech Republic 325 348 354
• Netherlands 322 321 322
• Belgium 320 323 332
• Portugal 315 295 305
• Australia 313 312 308
• New Zealand 313 311 302
• United States 313 312 310
• Alan Wagner, Measuring Up Internationally
  Developing Skills and Knowledge for the Global
  Knowledge Economy  (2006)

11
International Comparisons Degrees

• First qualification (of at least three years
  duration) awarded to graduates (2003 )
• Country Ratio of graduates to population at
  typical age of graduation
• Australia 49
• Finland (2002) 49
• Poland 44
• Iceland 43
• Denmark (2002) 42
• Norway 40
• United Kingdom 38
• Ireland 37
• Sweden 35
• Japan 34
• United States 33
• Country average 32

12
National Interest (2004-6)Assessment for
Accountability

• Measuring Up (2000, 2002, 2004, 2006, 2008)

13
National Interest (2004-6)Assessment for
Accountability

• Recent National Reports on Higher Education
• Business-Higher Education Forum (2004) Public
  Accountability for Student Learning in Higher
  Education
• National Commission on Accountability in Higher
  Education (2005) Accountability for Better
  Results
• National Governors Association (2006) Innovation
  America
• National Conference of State Legislatures
  Transforming Higher Education

14
National Interest (2006)

• (Spellings) Commission on the Future of Higher
  Education
• Assessment results should be
• Aggregated at the institutional level
• Focused on core intellectual skills
• Standardized
• Publicly available
• Value added

15
Commission on the Future of Higher Education
(cont.)

• Accreditors should
• encourage innovation and continuous improvement,
  and
• require institutions and programs to report
  measurable progress toward world-class quality

16
Commission on the Future of Higher Education
(cont.)

• Lack of useful data and accountability hinders
  policymakers and the public from making informed
  decisions and prevents higher education from
  demonstrating its contribution to the public
  good.

17
Congress Steps In

• College Opportunity and Affordability Act (2008)
• Part H (Program Integrity), Section 496
• Nothing in this section shall be construed to
  permit the Secretary to establish any criteria
  that specifies, defines, or prescribes the
  standards that accrediting agencies or
  associations shall use to assess any
  institutions success with respect to student
  achievement.
• Nothing in subsection (a)(5) shall be construed
  to restrict the ability of. . . an institution to
  develop and use institutional standards to show
  its success with respect to student achievement,
  which achievement may be considered as part of
  any accreditation review.

18
Institutional Associations Responses

• NSSE and other SSEs
• CIC Collegiate Learning Assessment Consortium
• CHEA Template for reporting outcomes and New
  Leadership for Student Learning and
  Accountability (with AACU)
• NAICU U-CAN (University and College
  Accountability Network)
• AASCU NASULGC Voluntary System of
  Accountability (College Portrait)

19
College Portrait
20
College Portrait Goals

• Provide consistent, comparable, transparent
  information for higher education stakeholders
• Aid students and families in college choice
  process
• Demonstrate greater institutional accountability
  for student learning and development

21
College Portrait Information

• Student and family information
• Student experiences and perceptions
• Student learning outcomes

22
College Portrait Learning

• Average freshman and senior scores
• plus
• Learning gains between freshman and senior years
• Critical Thinking
• The increase in learning for the critical
  thinking module is ltwhat would be expectedgt at an
  institution with students of similar academic
  abilities.
• Writing
• The increase in learning for the writing essay
  is ltwhat would be expectedgt at an institution
  with students of similar academic abilities.
• (www.voluntarysystem.org)

23
College Portrait Learning

• Fundamental Premise
• Meaning is a matter of compared to what?
• standardized testing

24
Standardized Testing Challenges for the
Researcher

• Logistics Student participation motivation
• Sample size
• Displacement of campus-based instruments
• Faculty turned off to all assessment
• Challenge to existing practices and policies

25
Creating a Culture of Evidence on Campus

• Technically possible
• External pressures
• Improvement/best practices
• Strategic use of increasingly scarce college
  resources

26
External Pressures Public Reporting

• Progression/graduation/transfer rates
• Student experiences
• Student learning

27
Improvement/Best Practices

• Get past not invented here syndrome
• Get past compliance mentality
• Identifying practices that create results
• Communication among campuses

28
Strategic Use of Evidence

• Pitch to appropriate level of control
• Start with user goals
• Collect actionable information
• Build trust

29
Reporting Challenges

• Being clear, comprehensible, and compelling
• Communication (external and internal)
• Using the results
• Sustaining the momentum

30
Challenges for the Researcher on the Campus that
Doesnt Want to Hear

• The Cassandra effect

31
Challenges on the Campus that Does

• Fear of public exposure

32
(No Transcript)