Interaction with and Reflection on the Noyce Program Evaluation Data

1
Interaction with and Reflection on the Noyce
Program Evaluation Data

• Frances Lawrenz
• Pey-Yan Liou
• Christina Madsen
• Christopher Desjardins
• Allison Kirchhoff
• University of Minnesota

2
Noyce Program Evaluation Group

• Our evaluation project has four major components
• Preparation of an extensive literature review
  pertaining to recruitment and retention
• Thematic synthesis through content analysis of
  project information
• Statistical analyses to produce quantitative
  models of the program development
• Execution of an overall program evaluation plan
  through collaboration (participatory approach)
  with existing projects.

3
(No Transcript)
4
Methodology Data sources

• Surveys
• PI online survey (N66)
• Scholar online survey (N555)
• Disciplinary faculty online survey (N80)
• Interviews
• Scholars in progress (N12)
• Districts in progress (N 17)
• Monitoring data collected for NSF
• Self-response bias

5
Methodology Analyses

• Frequencies, cross tabs, means
• Open-ended responses categorization
• Factor and cluster analyses
• Regression and HLM analyses using 3 outcome
  variables
• Factor score Commitment to teaching in high
  needs school (Influence of Noyce)
• Would you have become a teacher if you hadnt
  received the Noyce scholarship?
• Would you have decided to teach in a high need
  school if you hadnt participated in the Noyce
  scholarship program?

6
PI Survey
7
Components of the PI survey

• Project overview
• Noyce money
• Program characteristics and organization
• Contacts and partners

8
Project overview

• 75 active Noyce projects with 88 of the projects
  responding
• Of the 9 non-responding PIs, 7 were in their
  first or second year and subsequently had no data
  to report
• Projects in 29 states responded, with 12
  responding from Texas
• Total of 141 strands reported with an average 1
  or 2 teacher education strands per responding PI
• 8 strands was the maximum
• Strand breakdown
• 49 Undergraduate program leading to a bachelors
  degree (34.8)
• 17 Teaching credential (no degree) (12.1)
• 27 Post-bac or graduate program (no masters
  awarded) (19.1)
• 27 Graduate program (19.1)
• 21 Other (14.9)

9
Noyce money

• 55 indicated that Noyce scholarship paid for
  over 75 of scholars tuition (n 66) (PI Sec II
  Q4)
• 60 indicated that their teaching preparation
  program would continue mostly unchanged without
  the Noyce funding (n 65) (PI Sec II Q7)
• In contrast, 62 reported that the funding made
  their teacher certification program different (n
  65) (PI Sec II Q8)
• In general, PIs indicated that Noyce funding
• Greatly increased their ability to recruit a
  variety of students
• Had a lesser effect on perceptions about teaching
  careers
• Had a lesser effect on relationships with
  community, districts, STEM faculty, and industry.
  (PI Sec II Q9)

10
Funded activities
Miscellaneous
Meetings
Mentoring
PI Sec II Q 1, number of respondents for each
item ranged from 65 to 56.
11
Program characteristics

• Required activities (PI Sec III Q2)
• Most Teaching methods classes specific to
  candidates subject area student teaching
  experience
• Least Mentoring during the candidates second
  year of teaching
• Required field experience (PI Sec III Q3)
• Most Supervised actual classroom teaching in
  high needs schools (mode of hrs gt 121).
• Least Education field experience working outside
  of schools with young people like those who
  attend high needs schools in your area (mode of
  hrs 1-40)
• Collaborations with school districts (PI Sec III
  Q5)
• Most School districts providing practicum sites
  for scholars
• Least The school districts agreeing to hire all
  scholar who successfully complete the program.

12
Screening criteria for selecting Noyce scholars
Criteria n responding yes
GPA 65 98.5
Personal statement 64 97.0
Letters of recommendation 60 90.9
Undergraduate status 66 81.8
Structured interviews 63 73.0
Bachelors degree 65 72.3
Previous experience 64 68.8
Standardized tests 63 36.5
STEM courses beyond required for bachelors 63 36.5
Subject matter test 63 30.2
Attitude tests or measures 63 17.7
Observation of a teaching event 64 14.1
PI Sec III Q 3
13
Scholar Survey
14
Components of the scholar survey

• Project overview
• Program characteristics and organization
• Teaching environment and experience
• The decision to become a teacher
• Background and experience
• Overall experience

15
Project overview

• Of the 555 scholars responding to the survey
• 46 were teaching full-time/part-time
• 31 were still in their certification program but
  not yet a full-time teacher
• 13 were still in their program but also teaching
  full-time
• 8 completed a program but never taught
• 1 left their program without completing
  certification
• 1 taught after being certified and were working
  in education but not as teachers
• 1 taught after being certified but were no
  longing working in education

16
Program characteristics

• Features of teacher certification program
  (Scholar Sec II Q1)
• Most Classes in teaching methods specific to
  their subject area.
• Least Guaranteed job at a participating school
  district.
• Developing specific strategies for teaching
  students from diverse racial and ethnic
  backgrounds was the most extensively explored
  opportunity across the programs (Scholar Sec II
  Q4)

17
Teaching environment

• 83 of the 320 responding scholars are currently
  teaching in a high needs school (Scholar Sec III
  Q1)
• 68 of the 324 responding scholars are in schools
  where over 50 of students receive free or
  reduced lunch (Scholar Sec III Q3)
• 41 of the 322 responding scholars are in schools
  where there has been 15 teacher attrition over
  the last 3 years (Scholar Sec III Q3)
• 78 of the 318 responding scholars have held
  leadership positions within the last 3 years
• (Scholar Sec III Q5)

18
Decision to become a teacher

• 53 of scholars decided to become a STEM teacher
  during adulthood (23 or older) (n 540) (Scholar
  Sec IV Q1)
• 87 of scholars learned about the Noyce
  scholarship after they decided to become a
  teacher (n 543) (Scholar Sec IV Q4)
• 61 of scholars learned about the Noyce
  scholarship from their advisor (n 586) (Scholar
  Sec IV Q5)

19
Factors influencing STEM teacher decision
Scholar Sec IV Q3, respondents may have multiple
selections (respondents vary from 516 to 535)
20
Effect of Noyce scholarship on teaching

• Would you have become a teacher if you had not
  received the Noyce Scholarship?

• Would you have taught in a high needs school if
  you hadnt participated in the Noyce program?

Yes Possibly No
I will not teach in a high needs school

Scholar Sec IV Q6 (n 543) Q7 (n 542)
21
Faculty Survey
22
Faculty survey results

• 91 were familiar with the Noyce program (n 79)
  (Fac Q1)
• 44 interact with STEM education faculty daily
  and 25 interact weekly (n 79) (Fac Q2)
• Half of those interacting daily were in
  mathematics
• Faculty encouraged students to pursue teaching
  because of the students interest or because of
  the students personalities (Fac Q6a)
• Most suggested response from faculty to students
  interested in teaching was that they should get
  actual experience teaching and that teaching is a
  noble profession (Fac Q6b)
• 57 indicated that availability of scholarships
  affects advice they give to STEM majors (n 69)
  (Fac Q6c)

23
Ability of STEM majors who intend to teach
compared to those that do not
Fac Q4 (n 80)
24
Perceived effect of Noyce on STEM teacher
certification program
Fac Q1 (n 71)
25
Group Work 1
26
Group work 1 Data analysis

• Work in small groups or individually for 10
  minutes to develop a question that you would like
  us to explore
• At the end of the 10 minutes hand in your
  question and well select a suggestion(s) to
  analyze
• Well discuss these results and implications
  after our next group work activity

27
PI Suggestions

1. Career/non-career changers
2. Timing of teaching decision
3. Unique program characteristics

28
Career/non career changers

• Overview
• Demographics
• High needs teaching

29
Overview

• Career changers
• ½ of all responding scholars indicated they were
  career changers (n532)
• Of the scholars indicating being career changers
• All scholars at seven institutions indicated they
  were career changers
• 25 scholars at one institution indicated they
  were career changers
• Of the scholars indicating not being career
  changers
• All scholars at four institutions indicated they
  were non-career changers
• 20 scholars at one institution indicated they
  were non-career changers

30
Demographics

• Of the 267 career changing scholars
• Program type
• 67 are/were in graduate programs (masters)
• 14 are/were in teaching credential programs (no
  degree)
• Teaching status
• 47 are/were teaching full-time
• 5 completed a teacher program, but never taught
• Of the 265 non career changing scholars
• Program type
• 37 are/were in undergraduate programs
• 9 are/were in teaching credential programs (no
  degree)
• Teaching status
• 43 are/were teaching full-time
• 10 completed a teacher program, but never taught

31
High needs teaching
Career changers
Non-career changers
Teaching high needs school
Teaching another type of school
Scholar Sec III Q1 Sec V Q4, career changers
(n267), non-career changers (n265)
32
Timing of teaching decision

• Scholars were asked, Did you first learn about
  the Noyce scholarship before or after you decided
  to become a teacher?
• Program type
• Recruitment strategies

33
Program type
Scholar Sec I Q1 Sec IV Q 4, Before (n74),
After (n469)
34
Recruitment strategies
Scholar Sec IV Q4 Q5, Before (n74), After
(n469)
35
Unique program characteristics

• Experiences in teacher certification program
• Required field experiences

36
Experiences

• Scholars responded to 14 experiences in their
  teacher certification program (Scholar Sec II Q1)
• Of the 14 experiences, 2 were unique
• Guaranteed job at a participating school district
• 27 of scholars indicated they were guaranteed a
  job (n544)
• Of the scholars indicating yes, 38 were from
  graduate programs (masters) whereas 9 were from
  teaching credential programs (n144)
• Mentoring provided by certification program
  during the second year
• 32 of scholars indicated they received second
  year mentoring by their program (n524)
• Of the scholars indicating yes, 38 were from
  graduate programs (masters) whereas 16 were from
  teaching credential programs (n165)

37
Required field experiences

• Scholars responded to 5 required field
  experiences in their teacher certification
  program (Scholar Sec II Q 2)
• Of the 5 field experiences, 2 were unique
• Education field experience working outside of
  schools (e.g. summer camp) with young people like
  those who attend high needs schools in the area
• 11 of scholars indicated they received education
  field experience outside of schools (n541)
• 39 of the scholars were in undergraduate program
  leading to a bachelors degree whereas 12 were
  from teaching credential programs (n57)
• Research field experience in your subject area
• 27 of scholars indicated they received research
  field experience (n539)
• 44 of scholars were in graduate programs
  (masters) whereas 11 were in teaching credential
  programs (n143)

38
Group Work 2
39
Group Work 2 Discussion

• Spend a few minutes thinking about the discussion
  questions included in your packet.
• Then in a small group, youll discuss either the
  first or second question.
• Well then reconvene and discuss as a large group
  ideas that were raised during the small group
  discussion.

40
Inferential Statistics
41
Factor analysis

• Factor analysis was utilized to discover
  underlying factors and as a psychometric
  procedure for the development and refinement of
  the questionnaire
• One main goal is to combine many items into a
  construct, and use this construct to do more
  analyses

42
Factor analysis results of Noyce scholar survey
Constructed Factor Cronbachs Alpha Number of Items
Commitment to teaching in a high needs school (influence of Noyce) 0.893 6
Preparation for high needs school 0.730 13
Path to teaching 0.722 7
District/school high needs environment 0.716 5
Personal beliefs towards teaching 0.611 8
School teaching environment 0.775 4
Mentoring experience 0.724 6
43
Commitment to teaching in high needs school
(influence of Noyce)

• Become a teacher (Scholar Sec IV Q8a)
• Complete the certification program (Scholar Sec
  IV Q8b)
• Take a teaching job (Scholar Sec IV Q8c)
• Teach in a high needs school (Scholar Sec IV Q8d)
• Remain teaching in a high needs school for
• the full term of your commitment (Scholar Sec
  IV Q8e)
• Remain teaching in a high needs school
• beyond the full term of your commitment
  (Scholar Sec IV Q8f)

44
Hierarchical cluster analysis of commitment to
teaching in a high needs school
1 2 3 4 5 6
1. teach in a high needs school (Scholar Sec IV
Q8d) 2. remain teaching in a high needs school
for the full term of your commitment (Scholar Sec
IV Q8e) 3. remain teaching in a high needs school
beyond the full term of your commitment (Scholar
Sec IV Q8f) 4. become a teacher (Scholar Sec IV
Q8a) 5. take a teaching job (Scholar Sec IV
Q8c) 6. complete the certification program
(Scholar Sec IV Q8b)
45
Influence of Noyce vs. timing
Commitment to teaching in a high needs school (Influence of Noyce scholarship) Did you first learn about the Noyce scholarship before or after you decided to become a teacher?
Commitment to teaching in a high needs school (Influence of Noyce scholarship) Spearmans rho N
Did you first learn about the Noyce scholarship before or after you decided to become a teacher? Spearmans rho N -.250 530
1 before 2 after
(Scholar Sec IV Q4)
. Correlation is significant at the 0.01 level
(2-tailed).
46
Influence of Noyce vs. timing
High
Commitment to high needs teaching
Spearmans rho -0.250
Low
After
Before
Did you first learn about the Noyce scholarship
before or after you decided to become a teacher?
(Scholar Sec IV Q4)
Not the actual relationship, just a visual
representation
47
Influence of funding on timing of teaching
decision
(Scholar Sec IVQ6)
Would you have become a teacher if you had not
received the Noyce scholarship?
(Scholar Sec IVQ4)
(2, N542) 83.296, plt0.001
48
Hierarchical Linear Models (HLM)

• Also known as multi-level analysis, is a more
  advanced form of simple linear regression and
  multiple linear regression
• Why is HLM used in analyzing Noyce data?
• The program-level variables help to explain
  between-program variance rather than
  within-program variance
• Scholars in the same program tend to be more
  alike than in other projects

49
Three outcome variables in HLM

• Factor score Commitment to teaching in a high
  needs school (influence of Noyce)
• Would you have become a teacher if you hadnt
  received the Noyce scholarship? (Scholar Sec IV
  Q6)
• Would you have decided to teach in a high needs
  school if you hadnt participated in the Noyce
  scholarship program? (Scholar Sec IV Q7)

50
Group Work 3
51
Group Work 3 Searching for HLM PI predictors

• Brief overview of predictor analyses
• Spend a few minutes reading through the PI level
  variables handout
• Work in small groups or individually for 10
  minutes to discuss the PI level variables that
  you think they would influence the outcome
  variable Commitment to teaching in a high needs
  school (influence of Noyce)

52
1 Commitment to teaching in a high needs school

• What proportion of the total tuition scholars
  need to pay is provided by the Noyce scholarship
  funding? (0.10, p0.013) (PI Sec II Q4)
• Your ability to recruit participants from
  ethnicities under-represented in STEM into your
  certification/licensure program (0.10, p0.047)
  (PI Sec II Q9f)
• Your relationships with businesses or industry
  (0.13, p0.049) (PI Sec II Q9l)
• Interactions with children from different
  cultures (-0.33, p0.01) (PI Sec III Q2b)
• Teaching methods class(es) specific to the
  candidates subject area (0.36, p0.032) (PI Sec
  III Q2d)
• Student teaching experience (-0.29, p0.015) (PI
  Sec III Q2g)

53
1 Commitment to teaching in a high needs school

• What proportion of the total tuition scholars
  need to pay is provided by the Noyce scholarship
  funding? (0.10, p0.013) ? (0.12, p0.001) (PI
  Sec II Q4)
• Your ability to recruit participants from
  ethnicities under-represented in STEM into your
  certification/licensure program (0.10, p0.047) ?
  (0.11, p0.001) (PI Sec II Q9f)
• Your relationships with businesses or industry
  (0.13, p0.049) (PI Sec II Q9I)
• Interactions with children from different
  cultures (-0.33, p0.01) (PI Sec III Q2b)
• Teaching methods class(es) specific to the
  candidates subject area (0.36, p0.032) (PI Sec
  III Q2d)
• Student teaching experience (-0.29, p0.015) (PI
  Sec III Q2g)

54
We would like to thank NSF and all
those involved with the Noyce project!!Thank
you for your participation and valuable ideas!