Lessons Learned from Industry: Achieving Diversity & Efficacy in College Success

1
Lessons Learned from Industry Achieving
Diversity Efficacy in College Success
ETS - College Board Invitational
Conference Washington, DC Wayne Camara Krista
Mattern September 8, 2008
2
Job Analysis

• Organizations use job analysis to determine what
  work outcomes are desired.
• Sample individual outcomes (productivity, job
  performance, retention) and organizational
  outcomes (efficiency, quality, innovation, team
  work)
• Identify performance components (pc)
• pc Declarative knowledge x Procedural
  knowledge x Motivation
• Knowledge x Cognitive skills x
  Level of effort Goals Interpersonal
  Persistence
• Ability, Interests, Education, Experience
  Importance Prob. Of Outcomes
• Predictors

3
Job Analysis College Success

• Identify desired performance outcomes for
  individuals and organizations (college success)
  (GPA, return, graduate, life after college grad
  school, certification)
• Each outcome likely has different predictors
• Identify performance tasks associated with
  outcomes (persistence, academic ability, health,
  engagement)
• Identify or develop performance measures (GPA,
  advisory ratings, self report data, dB of student
  engagement)

4
Predicting Performance
Goldstein, Zedeck, Schneider (1993)
5
Group differences are not unique to tests They
are present across most educational measures
6
Large Mean Differences Persist on Cognitive
Ability Tests by Race/Ethnicity Remain
SAT CRM
7
College-Going Rates of High School Graduates Aged
18 to 24 by Ethnic Group, 1999-2006
Source U.S. Census Bureau
8
(No Transcript)
9
Disparities Exist in HS Graduation, HS Drop Out
and College Ready
Source Manhattan Institute, Public HS Graduation
and College-Readiness Rates 1991-2002,
http//Manhattan-institute.org/html/ewp_08.htm
Condition of Education, 2007 Table 23-2
10
Graduation Rates in 2004 by ethnicity
Published 3/7/2007 Title Awards conferred by
Title IV institutions, by race/ethnicity, level
of award, and gender United States, academic
year 200405 (recalculated to eliminate students
who with other or no ethnicity reported).
http//nces.ed.gov/ipeds/factsheets/pdf/fct_awards
_conferred_03072007_5.pdf Public HS graduation
rates WICHE 3/2008, http//www.wiche.edu/policy/
knocking/1992-2022/index.asp
11
Rationale for looking beyond Grades and Tests

• What is college success? Is it more than grades
  and GPA? (Camara Kimmel)
• Develop measures that predict your goal or
  desired outcome.
• Employers test multiple measures
• openness, conscientiousness, extraversion,
  agreeableness, neuroticism
• Military use today (GED).
• Can do does not equal will do.

12
Predictors of College Success
Not Collected in Standardized form
Colleges Collect in some form (applications,
transcripts)
Tests Measure
13
Research collaboration with Michigan State
University

• Identify a broader domain of college student
  performance
• Review university mission statements and
  department objectives
• Interview with university staff responsible for
  student life
• Review of the education literature on student
  outcomes
• Our systematic search (A JOB ANALYSIS OF
  UNDERGRADUATE STUDENTS) resulted in 12 dimensions
  of student performance
• Validate items with successful juniors they are
  the experts.

14
12 Dimensions of Student Performance

• Broadening the Performance Domain in the
  Prediction of Academic Success (Schmitt, Oswald,
  Gillespie, 2004)
• Knowledge, learning, mastery of general
  principles
• Continuous learning, intellectual interest and
  curiosity
• Artistic and cultural appreciation
• Multicultural appreciation
• Leadership
• Interpersonal skills
• Social responsibility, citizenship and
  involvement
• Physical and psychological health
• Career orientation
• Adaptability and life skills
• Perseverance
• Ethics and integrity

15
Two Noncognitive Measures

• Situational judgment inventory
• A situation is presented along with several
  alternative courses of action.
• The respondent is asked to indicate what she/he
  would be most likely and least likely to do.
• Biodata
• Short, multiple choice reports of past
  experience/background and interests/preferences.

16
Sample SJI Item for Leadership

• You are assigned to a group to work on a
  particular project. When you sit down together
  as a group, no one says anything.
• -1 Look at them until someone eventually says
  something
• Start the conversation yourself by introducing
  yourself
• 1 Get to know everyone first and see what they
  are thinking about the project to make sure the
  projects goals are clear to everyone
• Try to start working on the project by asking
  everyones opinion about the nature of the
  project
• You would take the leadership role by assigning
  people to do things or ask questions to get
  things rolling

17
Sample Biodata Items for Leadership

• The number of high school clubs and organized
  activities (such as band, sports, newspapers,
  etc.) in which I took a leadership role was
• 4 or more
• 3
• 2
• 1
• I did not take a leadership role
• How often do you talk your friends into doing
  what you want to do during the evening?
• most of the time
• sometimes (about half the time)
• occasionally (about as often as others in my
  group
• seldom or infrequently
• never

18
Study 1 Develop and refine the measures

• 644 MSU freshmen completed one of the two
  parallel forms of the biodata and SJI instruments
  at the beginning of the academic year.
• Results indicated significant incremental
  validity for some of the scales above and beyond
  the validity of SAT/ACT scores and existing
  measures of personality in predicting college
  GPA.
• The biodata and SJI demonstrated the greatest
  incremental validity when absenteeism, students
  self ratings, and peer-ratings of performance
  were examined ( .19, .22, and .14, respectively).
  

19
Study 2 Examine Validity Subgroup Differences
10 Participating Institutions 2,700 Freshmen

• HBCU N
• Winston-Salem (public) 229
• Spelman College (private) 254
• Big Ten (public) N
• University of Iowa 335
• Michigan State University 546
• Ohio State University 304
• University of Michigan 297
• Indiana University 170
• Other Institutions N
• University of Chicago (private) 168
• Cal State Fullerton (public) 223
• Virginia Tech (public) 237

20
Predicting FYGPA Total Sample across 10
Institutions (N 2443)
Non cognitive measures contribute little beyond
tests and grades in predicting academic outcomes
21
Predicting Class Absenteeism Total Sample across
10 Institutions (N 899)
However, non cognitive measures will predict non
cognitive outcomes better than tests or grades
(graduation, attendance, leadership, engagement)
22
Percent of Students SelectedTwo Composites and
Three Selection Strategies
Moderately selective

• Top 85 Top 50
  Top 15
• Group AB AB AB AB
  AB AB
• Hispanic 4.4 ?
  4.6 4.1 ? 4.9 3.9 ? 5.5
• (.2) (.8) (1.6)
• Asian 7.6 ? 7.7 9.9 ?
  9.5 17.5 ? 12.9
• (.1) (-.4) (-4.6)
• African-American 17.9 ? 19.8
  9.6 ? 13.6 1.3 ? 7.2
  (1.9) (4.0) (5.9)
• White 70.2 ? 67.9 76.4 ?
  71.9 77.2 ? 74.4
• (-2.3) (-4.5) (-2.8)
• AB equally weighted composite of HSGPA and
  SAT/ACT.
• AB equally weighted composite of HSGPA,
  SAT/ACT, Biodata, and SJI.

Less selective
Very selective
23
Correlations of Non-cognitive Measures with
Cumulative GPA and Graduation
Note. Bold values are significant at plt .01. N
ranges from 1560 to 1798 across variables.
Graduation is dichotomously scored (1, 0).
24
Study 3 Purpose Research Questions

• 15 institutions (n 4,164 for SJI and 7,645 for
  biodata)
• Purpose evaluating the utility of the biodata
  and situational judgment measures in as close to
  a real admissions situation as is possible
• Administer new measures to college applicants
  rather than college freshmen.
• On an annual basis, collect class absenteeism,
  self rated performance of the noncognitve
  dimensions, and commitment to the university from
  enrolled students institutions will provide
  course grades and retention.

25
Incremental Validity of Biodata Measures

• To preserve N in these regressions, the SJI was
  not included because of a relatively low response
  rate to this measure.
• It is worth noting that small sample sizes, such
  as those observed in these analyses, can
  seriously limit the ability to detect significant
  relationships due to decreased statistical power.

26
Next Steps

• In need of a demonstration project Implement
  with Research across a few colleges!
• Encourage applicants to complete on-line as part
  of admissions and only use data as a plus
  factor.
• Provide incentives for applicants to complete the
  new measures and institutions to track student
  success over time.
• Likely outcomes will be more diversity, broader
  talent, greater retention, and standardized
  defensible measures to evaluate applicants fairly
  and objectively.
• Increased efficiency and judgmental decisions
  based on data and comparability.
• For more information, go to http//www/iopsych.msu
  /cbstudy