Title: Lessons Learned from Industry: Achieving Diversity & Efficacy in College Success
1Lessons Learned from Industry Achieving
Diversity Efficacy in College Success
ETS - College Board Invitational
Conference Washington, DC Wayne Camara Krista
Mattern September 8, 2008
2Job Analysis
- Organizations use job analysis to determine what
work outcomes are desired. - Sample individual outcomes (productivity, job
performance, retention) and organizational
outcomes (efficiency, quality, innovation, team
work) - Identify performance components (pc)
- pc Declarative knowledge x Procedural
knowledge x Motivation - Knowledge x Cognitive skills x
Level of effort Goals Interpersonal
Persistence - Ability, Interests, Education, Experience
Importance Prob. Of Outcomes - Predictors
3Job Analysis College Success
- Identify desired performance outcomes for
individuals and organizations (college success)
(GPA, return, graduate, life after college grad
school, certification) - Each outcome likely has different predictors
- Identify performance tasks associated with
outcomes (persistence, academic ability, health,
engagement) - Identify or develop performance measures (GPA,
advisory ratings, self report data, dB of student
engagement)
4Predicting Performance
Goldstein, Zedeck, Schneider (1993)
5Group differences are not unique to tests They
are present across most educational measures
6Large Mean Differences Persist on Cognitive
Ability Tests by Race/Ethnicity Remain
SAT CRM
7College-Going Rates of High School Graduates Aged
18 to 24 by Ethnic Group, 1999-2006
Source U.S. Census Bureau
8(No Transcript)
9Disparities Exist in HS Graduation, HS Drop Out
and College Ready
Source Manhattan Institute, Public HS Graduation
and College-Readiness Rates 1991-2002,
http//Manhattan-institute.org/html/ewp_08.htm
Condition of Education, 2007 Table 23-2
10Graduation Rates in 2004 by ethnicity
Published 3/7/2007 Title Awards conferred by
Title IV institutions, by race/ethnicity, level
of award, and gender United States, academic
year 200405 (recalculated to eliminate students
who with other or no ethnicity reported).
http//nces.ed.gov/ipeds/factsheets/pdf/fct_awards
_conferred_03072007_5.pdf Public HS graduation
rates WICHE 3/2008, http//www.wiche.edu/policy/
knocking/1992-2022/index.asp
11Rationale for looking beyond Grades and Tests
- What is college success? Is it more than grades
and GPA? (Camara Kimmel) - Develop measures that predict your goal or
desired outcome. - Employers test multiple measures
- openness, conscientiousness, extraversion,
agreeableness, neuroticism - Military use today (GED).
- Can do does not equal will do.
12Predictors of College Success
Not Collected in Standardized form
Colleges Collect in some form (applications,
transcripts)
Tests Measure
13Research collaboration with Michigan State
University
- Identify a broader domain of college student
performance - Review university mission statements and
department objectives - Interview with university staff responsible for
student life - Review of the education literature on student
outcomes - Our systematic search (A JOB ANALYSIS OF
UNDERGRADUATE STUDENTS) resulted in 12 dimensions
of student performance - Validate items with successful juniors they are
the experts.
1412 Dimensions of Student Performance
- Broadening the Performance Domain in the
Prediction of Academic Success (Schmitt, Oswald,
Gillespie, 2004) - Knowledge, learning, mastery of general
principles - Continuous learning, intellectual interest and
curiosity - Artistic and cultural appreciation
- Multicultural appreciation
- Leadership
- Interpersonal skills
- Social responsibility, citizenship and
involvement - Physical and psychological health
- Career orientation
- Adaptability and life skills
- Perseverance
- Ethics and integrity
15Two Noncognitive Measures
- Situational judgment inventory
- A situation is presented along with several
alternative courses of action. - The respondent is asked to indicate what she/he
would be most likely and least likely to do. - Biodata
- Short, multiple choice reports of past
experience/background and interests/preferences.
16Sample SJI Item for Leadership
- You are assigned to a group to work on a
particular project. When you sit down together
as a group, no one says anything. - -1 Look at them until someone eventually says
something - Start the conversation yourself by introducing
yourself - 1 Get to know everyone first and see what they
are thinking about the project to make sure the
projects goals are clear to everyone - Try to start working on the project by asking
everyones opinion about the nature of the
project - You would take the leadership role by assigning
people to do things or ask questions to get
things rolling
17Sample Biodata Items for Leadership
- The number of high school clubs and organized
activities (such as band, sports, newspapers,
etc.) in which I took a leadership role was - 4 or more
- 3
- 2
- 1
- I did not take a leadership role
- How often do you talk your friends into doing
what you want to do during the evening? - most of the time
- sometimes (about half the time)
- occasionally (about as often as others in my
group - seldom or infrequently
- never
18Study 1 Develop and refine the measures
- 644 MSU freshmen completed one of the two
parallel forms of the biodata and SJI instruments
at the beginning of the academic year. - Results indicated significant incremental
validity for some of the scales above and beyond
the validity of SAT/ACT scores and existing
measures of personality in predicting college
GPA. - The biodata and SJI demonstrated the greatest
incremental validity when absenteeism, students
self ratings, and peer-ratings of performance
were examined ( .19, .22, and .14, respectively).
19Study 2 Examine Validity Subgroup Differences
10 Participating Institutions 2,700 Freshmen
- HBCU N
- Winston-Salem (public) 229
- Spelman College (private) 254
- Big Ten (public) N
- University of Iowa 335
- Michigan State University 546
- Ohio State University 304
- University of Michigan 297
- Indiana University 170
- Other Institutions N
- University of Chicago (private) 168
- Cal State Fullerton (public) 223
- Virginia Tech (public) 237
20Predicting FYGPA Total Sample across 10
Institutions (N 2443)
Non cognitive measures contribute little beyond
tests and grades in predicting academic outcomes
21Predicting Class Absenteeism Total Sample across
10 Institutions (N 899)
However, non cognitive measures will predict non
cognitive outcomes better than tests or grades
(graduation, attendance, leadership, engagement)
22Percent of Students SelectedTwo Composites and
Three Selection Strategies
Moderately selective
-
-
- Top 85 Top 50
Top 15 - Group AB AB AB AB
AB AB - Hispanic 4.4 ?
4.6 4.1 ? 4.9 3.9 ? 5.5 - (.2) (.8) (1.6)
- Asian 7.6 ? 7.7 9.9 ?
9.5 17.5 ? 12.9 - (.1) (-.4) (-4.6)
- African-American 17.9 ? 19.8
9.6 ? 13.6 1.3 ? 7.2
(1.9) (4.0) (5.9) - White 70.2 ? 67.9 76.4 ?
71.9 77.2 ? 74.4 - (-2.3) (-4.5) (-2.8)
- AB equally weighted composite of HSGPA and
SAT/ACT. - AB equally weighted composite of HSGPA,
SAT/ACT, Biodata, and SJI.
Less selective
Very selective
23Correlations of Non-cognitive Measures with
Cumulative GPA and Graduation
Note. Bold values are significant at plt .01. N
ranges from 1560 to 1798 across variables.
Graduation is dichotomously scored (1, 0).
24Study 3 Purpose Research Questions
- 15 institutions (n 4,164 for SJI and 7,645 for
biodata) - Purpose evaluating the utility of the biodata
and situational judgment measures in as close to
a real admissions situation as is possible - Administer new measures to college applicants
rather than college freshmen. - On an annual basis, collect class absenteeism,
self rated performance of the noncognitve
dimensions, and commitment to the university from
enrolled students institutions will provide
course grades and retention.
25Incremental Validity of Biodata Measures
- To preserve N in these regressions, the SJI was
not included because of a relatively low response
rate to this measure. - It is worth noting that small sample sizes, such
as those observed in these analyses, can
seriously limit the ability to detect significant
relationships due to decreased statistical power.
26Next Steps
- In need of a demonstration project Implement
with Research across a few colleges! - Encourage applicants to complete on-line as part
of admissions and only use data as a plus
factor. - Provide incentives for applicants to complete the
new measures and institutions to track student
success over time. - Likely outcomes will be more diversity, broader
talent, greater retention, and standardized
defensible measures to evaluate applicants fairly
and objectively. - Increased efficiency and judgmental decisions
based on data and comparability. - For more information, go to http//www/iopsych.msu
/cbstudy -