Developed by

1
Conceptual Framework for Evaluation Assessment
of theNational Science Foundation (NSF)Alliance
for Graduate Education the Professoriate (AGEP)
Program

• Developed by
• Yolanda S. George, AAAS Education Human
  Resources Programs and
• Patricia Campbell, Campbell-Kibler Associates,
  Inc.
• With input from the AGEP Community

2
Goal of AGEP

• The goal of the AGEP program is to increase the
  number of underrepresented minority students
  pursuing advanced study, obtaining doctoral
  degrees, and entering the professorate in STEM
  disciplines (including social sciences).
• Alliances participating in this program are
  expected to engage in comprehensive institutional
  cultural changes that will lead to sustained
  increases in the conferral of STEM doctoral
  degrees, significantly exceeding historic levels
  of performance.

3
Specific objectives of AGEP are

• (1) to develop and implement innovative models
  for recruiting, mentoring, and advancing minority
  students in STEM doctoral programs, and
• (2) to develop effective strategies for
  identifying and supporting underrepresented
  minorities who want to pursue academic

4
The primary goal of the AGEP Program evaluation
capacity building project is to

• Help AGEP awardees to build a comprehensive
  evaluation and assessment infrastructure to
  examine their graduate education enterprise in
  terms of broadening participation in the STEM PhD
  workforce

5
The evaluation framework and model for change are
focused on graduate student affairs, including

• Recruitment and talent spotting strategies.
• Admissions and selection processes.
• Financial aid, particularly as it affects new
  graduate student enrollment and retention.
• Academic preparation support programs,
  particularly during the early course-taking
  years.
• Advising, mentoring, and retention.
• Workforce preparation.
• PhD career counseling and employment placement.

6
Key Administrators Using Evaluation Assessment
to Make informed Decisions

• Graduate School and Academic Deans Provosts and
  Other Administrators
• SE Department Chairs
• SE Departmental Admissions and Selection
  Committees
• SE Departmental Graduate Program Directors or
  Advisors
• SE Faculty

7
Collection And Use Of Disaggregated Data

• Race/Ethnicity
• Gender
• Citizenship
• SE Departments

8
Quantitative data should be used to examine
patterns and trends in number and percentage of

• UM applicants admits
• New or first time UM enrollees
• Overall UM enrollment
• UM students advancing to doctoral candidacy
• UM students completing degrees (Masters or PhD).
• Post PhD Employment

9
Indicators of Infrastructure Changes in Graduate
Programs

• Data collection practices at the graduate school
  level, particularly use of disaggregated data in
  decision-making.
• SE departmental practices related to graduate
  student admissions and selection.
• SE departmental practices related to academic
  preparation, early year course taking, advising
  and workforce mentoring
• SE departmental practices related to faculty
  evaluation and incentives.
• Family leave policies and practices within the
  graduate school or departments, as related to
  graduate students.

10
Points of Intervention for Retention

• Departmental environment and PhD completion
• Mentoring and PhD completion
• Faculty attitudes about retention in PhD programs
  vs government investment (Dispelling myths about
  retention in PhD programs)
• Chairs leadership on PhD completion and mentoring
• Information flow to students about milestone to
  the PhD and feedback about progress towards the
  PhD
• Graduate student integration into the
  intellectual community
• Attention to graduate student research
  productivity (impact on postdoctoral fellowship
  opportunities or academic employment)
• What can we do about students who are thinking
  about leaving? How can we spot them?

11
Also, data and information need to be collected
on

• The quality of the implementation including
  participants and/or external reviewers
  perceptions about the intended objectives of a
  workshop or program component.
• Time-on-task
• The effectiveness of the interventions or to what
  extent did the intervention achieve its intended
  objectives.

12
Quantitative and qualitative studies about
policies, practices, and programs in the Graduate
School SE departments should be used to
determine

• What is working and not working?
• What works for whom?

13
In Summary

• This is a community effort.
• The framework is evolving
• This is not an easy journey, neither is it one
  that can be completed quickly