Ford PAS Employment Skills Assessment Pilot: Teamwork and Collaboration

1
Ford PAS Employment Skills Assessment Pilot
Teamwork and Collaboration
Richard D. Roberts, Lijuan Wang, Xiaohua Zhuang,
Lydia Liu, Carolyn MacCann Center for New
Constructs, ETS
Interested in Learning about Ford PAS Evaluation
Efforts? 5th Annual Ford PAS National Networking
Conference Making Connections June 18-21,
2007, Louisville, KY
2
Background

• The Workforce Readiness Report Card from the
  Partnership for 21st Century Skills, The
  Conference Board, Corporate Voices for Working
  Families, and the Society for Human Resource
  Management found the nation's new workforce
  entrants woefully ill-prepared for the demands
  of todaysand tomorrowsworkplace.
• Teams are prevalent in the workforce as a
  consequence of globalization and the rapidly
  changing face of knowledge
• Teamwork has been identified as a major
  educational skill for the 21st century

3
Pillars of Ford PAS
4
Learning Pillar Teamwork
Ford PAS expects and encourages students to work
together in teams to carry out investigations,
synthesize data, and communicate results. The
curriculum introduces students to a variety of
skillsincluding giving and receiving feedback,
negotiating, and facilitating meetingsthat build
their capacity to collaborate effectively with
their teammates.
5
Teamwork Components
6
Scientific and Research Traditions

• Personality factors
• Conscientiousness
• Extraversion
• Agreeableness
• Emotional Stability
• Social-emotional factors (emotional intelligence
  components)
• Cooperation
• Persuasion
• Negotiation
• Managing self and others
• Knowledge
• How to function in a team setting

7
Study Rationale

• Develop an assessment system
• Outcomes (e.g., evaluation studies)
• Self-insight
• Formative
• Reliable and valid
• Assessments need to be based on multiple methods
• Self-report
• Teacher-report
• Situational Judgment Test

8
Three Measurement Methods
Teamwork
Teacher Evaluation
STJ
Student Self-Report
9
Measures

• Teamwork Scale --- Student Survey
• Student self-report rating scale
• 57 items, 6-point Likert scale
• Situational Judgment Test (SJT)
• 8 scenario questions, 4 choices each
• Teamwork Scale --- Teacher Survey
• 10 items, 5-point Likert scale

10
Sample Characteristics

• Sample size 159 (lost N3 due to zero variance
  on 2 or more constructs)
• High school students (9th through 12th grades)
• 51.6 female
• Median age of respondents was 16 years old
• 19 White, 64.6 African-American, 3.2 Hispanic,
  1.9 Asian, 1.9 Native American, and 9.5 other
• Reading (65 A or B), math (52 A or B), science
  (58 A or B), social studies (73 A or B)

11
Teamwork Scale --- Student Self-Report Scale
12
Student self-report rating scale

• 57 items
• A sample item
• I enjoy bringing team members together
• Never
• Rarely
• Sometimes
• Often
• Usually
• Always

13
Simpler and Clearer Factor Structure

• Eliminate some problematic items
• 57 items -gt 30 items

14
Factor Loading matrix
15
First-order Confirmatory Factor Analysis
.55
.64
.80
Advocate Influence
Cooperation
Negotiation
Chi-square655 DF402 CFI 0.84 NNFI 0.83 RMSEA
0.06
16
Teamwork Components
17
Reliability

• Negotiation (11 items) .80
• Advocate and Influence (8 items) .76
• Cooperation (11 items) .88

18
Correlation with External Variables
Note . is significant at the .01 level. . is
significant at the .05 level.
No significant mean differences among different
ethnicity groups
19
Correlation with Personality Scales
Note . is significant at the .01 level. . is
significant at the .05 level.
20
Teamwork Scale --- Situational Judgment Test
21
Scale

• Eight scenarios, four options each
• You are part of a study group that has been
  assigned a large presentation for class. As you
  are all dividing up the workload, it becomes
  clear that both you and another member of the
  group are interested in researching the same
  aspect of the topic. Your colleague already has a
  great deal of experience in this area, but you
  have been extremely excited about working on this
  part of the project for several months.
• Which of the following is the best approach
  to dealing with this situation?

22

• Rate each response option as below
• (A) Flip a coin to determine who gets to work on
  that particular aspect of the project.
• (B) Insist that, for the good of the group, you
  should work on that aspect of the project because
  your interest in the area means you will do a
  particularly good job.
• (C) Compromise your preferences for the good of
  the group and allow the other person to work on
  that aspect of the project.
• (D) Choose a different group member to work on
  that aspect of the project so that no one person
  is privileged over another.
• 1.Very Ineffective
• 3. Neither ineffective or effective
• 5. Very Effective

23
Scoring the Teamwork SJT

• There are two ways to score the teamwork SJT
• By expert judgment
• The test developer decides that one option is the
  best. Reduce the number of items to 8.
• E.g., for scenario one, the best option is option
  C, so participants ratings (1 to 5) on option C
  are used.
• By group consensus
• The proportion of the sample selecting each point
  is the weight awarded to that group
• E.g., If 26 of the sample pick 1 for option A,
  this group gets .26 for option A.

24
Reliability and Scoring Method

• Expert scoring
• 8 items (key answers), reliability 0.71
• Consensus scoring
• 32 items, reliability 0.73
• Expert and Consensus scoring
• 8 items, reliability 0.55
• Correlation between expert scoring and consensus
  scoring is 0.73 on the basis of 8 items.

25
Correlation with External Variables
A one-factor solution was detected from EFA.
Note . is significant at the .01 level. . is
significant at the .05 level.
No significant mean differences among different
ethnicity groups No significant correlations
between teamwork skills evaluated by SJT method
and big five personality scales
26
Teamwork Scale --- Teacher Survey
27
Scale

• Ten items
• Sample item When working on a group goal or
  project, this student

28
Analysis Results

• One factor from exploratory factor analysis with
  83 explained variance
• Reliability (Cronbachs Alpha) 0.98

29
Correlation with External Variables
. Correlation is significant at the 0.01 level
No significant mean differences among different
ethnicity groups No significant correlations
between teamwork skills evaluated by teachers and
big five personality scales.
30
Correlation between the Three Methods
. is significant at the 0.01 level.
31
Discussion

• Validity evidence is encouraging
• Reliability is good for relatively brief
  assessments
• Need to develop framework that embraces theory
  and practice (symbiotic)
• Should teacher ratings be part of the assessment?
  Or independent criteria?
• Part of a larger picture, where noncognitives are
  being valued in their own right

32
Future Steps

• Resolve scoring issues with the SJT
• Further analyses of the three measurement methods
  (composite)
• Feedback and action plans
• Seems viable as an evaluation tool
• Report/peer-review scientific article

33
Comments?
Questions?

• Thanks to Mary Lucas, Linda DeLauro, Gerard
  Mannarino, Sara Kirsch, Jonathan Steinberg, staff
  and students, and our various Ford PAS contacts

34
Predictive Validity on Course Grades
R2
Teacher Evaluation
Reading
6
Math
6
Advocate
Science
18
Cooperation
Social
13
Negotiation
Music
16
SJT
Only significant paths were displayed.
35
Predictive Validity on Big Five
R2
Teacher Evaluation
Extraversion
16
Advocate
Agreeableness
22
Cooperation
Conscientiousness
19
Negotiation
Neuroticism
24
Openness
SJT
22
Only significant paths were displayed.
Red negative