Title: Building an Effective Evaluation of Your Flexible Learning Program (FLP) Presenter: Sue Hawks Foster, FLP Education Program Specialist Georgia Department of Education sfoster@doe.k12.ga.us (404) 656-2636
1Building an Effective Evaluation of Your Flexible
Learning Program (FLP) PresenterSue Hawks
Foster, FLP Education Program SpecialistGeorgia
Department of Educationsfoster_at_doe.k12.ga.us(404
) 656-2636
2Agenda FLP Evaluation
- Federal Regulation Requires Evaluation of the
Title I Programs - FLP Evaluation Requirements-
- Federal, State, and District
- Building an Effective Program Evaluation
- LEA Plan for District FLP Evaluation
- Performance Based Observation of FLP
- Most Common Findings FY13
- Questions and Answers
- References
3- Federal requirements for evaluating title I
programs
4Federal Regulation for Title I Programs
- Federal regulations requires that a school
operating a Title I program annually evaluate the
implementation of, and results achieved by, the
Title I program - ESEA Title I program compliance requires written
program evaluation procedure and program
evaluation summary
5Federal Requirements for Evaluation
- The Title I program evaluation is an annual
review of Title I activities to determine if they
are contributing to the desired outcomes - Improved student achievement
- Effective family engagement
- High quality professional development
6Federal Requirements for Evaluation
- Evaluation Components
- Measures efficacy and impact of the Title I FLP
program - Documents impact of Title I FLP services on
student learning - Informs school and district planning (cycle of
continuous improvement) - Is an advocacy tool
- Is conducted at end of program year and used for
the next years planning
7Federal Requirements for Evaluation
- Description of
- Data used
- Constituents consulted
- Process used to arrive at evaluation findings
- Previous data
- Surveys
- How findings are utilized for planning and
improvement
8Federal Requirements for Evaluation
- Summary of
- Program strengths
- Program weaknesses
- Subsequent program changes
- Reallocation of funds
9Needs Assessment Program Evaluation
- What are the differences and how do they relate
to each other? - Program evaluation
- Evaluates Title I program-FLP specifically
- Completed after the fact (end of year)
- Helps to plan Title I program for next year
- including FLP
10Needs Assessment Program Evaluation
- Needs assessment
- Looks at all school data, including Title I
- Schoolwide data in all subject areas
- Targeted Assistance data in all subject areas
- FLP data
- Completed before beginning of school year
- Assesses priority needs in district/schools and
determines where Title I FLP fits in, as well as,
other resources
11- FLP evaluation requirements
12FLP Evaluation Requirements
- State Evaluation
- Participant Feedback
- Evaluation Question What is the overall
experience of participants with program? - Data Source Student and Parent Surveys
- Service Delivery
- Evaluation Question Is the LEAs and programs in
compliance with laws and regulations? - Data Source Annual monitoring data, Program
documentation, Federal reporting, Public
reporting, Technical Assistance, etc.
13State Evaluation contd
- State Evaluation
- Effectiveness
- Evaluation Question Are programs contributing to
increased student academic achievement and
performance on state education standards? - Data Source Student performance on state tests,
Pre-post assessment measures of state standards
and academic skills targeting by programs,
Performance Flag data, and student growth in
schools offering FLP - Evaluation results would be shared with
stakeholders and the public and used to inform
ongoing program improvement
14State Evaluation contd
- LEA Evaluation
- LEAs will be required to evaluate outcomes of FLP
interventions implemented at schools within their
district that have been identified - LEAs will submit their evaluation plan as part of
their overall FLP plan for approval by GaDOE
Title I Specialists
15State Evaluation contd
- LEA Evaluation
- The LEA must submit the following components
of an - evaluation plan with their FLP plan in the
CLIP for each - Priority, Focus, or Alert school within
their district - required to implement FLP
- Effectiveness target or overall quantifiable goal
of the program - Assessment instrument(s) that will be used to
measure this target/goal - Plan for collecting and maintaining data
- Data analysis plan
16State Evaluation contd
- What do the LEA evaluation requirements mean?
- Focused, purposeful intervention
- Tailor evaluation needs of program to match
program goals and student needs - Expand on broadband state assessments to quantify
benefits of FLP participation - Collect meaningful information for decision
making and continuous program improvement
17- Building an effective program evaluation plan
18Building Effective Program Evaluation
- Determining Purpose of Evaluation
- Why should we evaluate our program?
- What is our program trying to do?
- What information do our funders expect?
- What questions do we want to answer?
19Building Effective Program Evaluation
- Performance Measures
- Measures of effort
- Measures of effect
- Strong performance measures should
- Have strong ties to program goals, inputs, and
outputs - Be compatible with age and stage of your program
- Consider if the data you need are
available/accessible - Yield useful information to the program
20Building Effective Program Evaluation
- Assessing Capacity for Evaluation
- Who should be involved in the process?
- What resources must be in place?
- Who will conduct our evaluation?
21Building Effective Program Evaluation
- Selecting Evaluation Design
- What type of evaluation should we conduct?
- Formative to understand the changes that have
occurred - Summative demonstrate they have achieved the
standard required - What type of data should we collect?
- Qualitative
- Quantitative
22Building Effective Program Evaluation
- Selecting Evaluation Design (cont.)
- What is our time frame for collecting data?
- Causal
- Experimental
- Quasi-experimental
- Non-experimental
- Descriptive
- Pre-experimental
23Building Effective Program Evaluation
- Choosing evaluation instruments
- Construct
- Audience
- Format
- Reliability
- Validity
24Building Effective Program Evaluation
- Analyzing data
- How will data be analyzed and shared?
- Quantify central evaluation question (and
supporting questions) - Communicating results
- How?
- Where?
- Who?
25LEA Plan for District flexible learning program
(FLP) evaluation
26LEA Plan for District FLP Evaluation
- List the effectiveness target(s) or overall
quantifiable goal(s) of your programs - What are the measureable outcomes that your
intervention is designed to improve? Must include
at least one academic achievement indicator
27LEA Plan for District FLP Evaluation
- List the assessment instrument(s) that will be
used to measure each listed effectiveness
target/goal - Include a plan for administering assessment
instruments and how you will collect and maintain
data
28LEA Plan for District FLP Evaluation
- When will you administer pre- and post-
assessments? - How will you maintain student assessment
information to ensure confidentiality? - How will you determine program effectiveness
based on program goals and measurement
instruments listed above?
29LEA Plan for District FLP Evaluation
- FLP Program Evaluation Includes Academic Data
and Narrative - FLP Program Summary Chart
- Parent, Staff and Student Summary Results
30True Test for FLP
- Are graduation rates improving?
- Are CRCT scores increasing?
- Are EOCT pass rates increasing?
- Is our CCRPI increasing?
31Performance based observation (PBO)
32Performance Based Observation
- FY13 PBO in Review
- 12 Districts were selected for PBO as a pilot
- 16 schools were observed
- FY14 Review
- 26 Districts have been selected for review
- 26 School will be observed
33Performance Based Observation
- Components of Performance Based Observation(PBO)
- Review of Priority, Focus, and Alert School List
- Select number of districts to be reviewed based
on the ESEA Flexible Waiver - Selection of schools in district to be monitored
34Performance Based Observation
- Components of PBO contd
- Notifications Sent to LEAs
- Call to Title I Director
- Follow up Letter to LEA Superintendent, Title I
Director and Principal of selected schools with
brief outline of expectations
35Performance Based Observation
- Components of PBO contd
- Agenda Developed by Title I Office
- Follow up call to LEA by PBO Team Lead before
visit - Addresses and list of staff reviewed
- Selection of Teachers/Tutors for Interview
36Performance Based Observation
- Actual Visit to LEA
- Interview with LEA Staff
- Superintendent
- Title I Director
- School Principal
- School FLP Coordinator
- School FLP Instructors/Tutors (at least 2)
- Other vital staff as needed
37Performance Based Observation
- Actual Visit to LEA
- Classroom Observation -20 minutes per classroom
- Talk with students and staff in classroom
- Observe student and FLP staff interactions
- Depart Building- Inform building staff we are
leaving the building
38Performance Based Observation
- Follow-up Letter
- Review of the PBO findings for the district
- Non-Compliance
- If a LEA is found out of compliance the district
will be notified immediately by Program Manger - If corrective action is required it will be
provided in writing and followed up with the FLP
Specialist and/or the Program Director for FLP
39- FY13 FLP Most Common Findings
-
40FLPs Most Common Findings
- Models used for FLP
- After school programs were used by Middle and
High Schools - Elementary and Middle schools extended the day or
rotated the scheduled day - High schools used Saturday sessions
- Summer Sessions were held when extra funds were
available to extend the learning year
41FLPs Most Common Findings contd
- Technology Usage Increased
- Increase Professional Learning
- Increased Student Instruction Time
- More contact with parents
- Reported Increase in Test Scores
42FLPs Most Common Findings contd
- Frustration of operating the program by some
districts because it was so new - Large amounts of carryover because of FLP funds
were not used last year - Special Education Departments not connecting to
FLP in various districts - Lack of staff in some districts to serve students
43FLPs Most Common Findings in Plans
- Lack of understanding of FLP purpose
- Did not provide performance goals for plans
- Determining the needs assessment was not fully
understood to determine subject area - Many did not identify all subgroups within the
schools - Schools did not understand providing data for
more than one year in order to provide a trend
44FLPs Most Common Findings in Plans
- Special Education Departments were not really
aware of what was going on with Students with
Disabilities (SWD) in FLP. - Student ratio needs to be smaller
- Individual Learning Plans needed to be developed
and followed up - Documentation of Multiple Selection Criteria was
lacking in some districts - Lack of understanding Professional Learning
45Questions and Answers
46References
- Afterschool Evaluation 101 How to Evaluate an
Expanded Learning Program, Erin Harris, Harvard
Family Research Project, 2011 http//www.hfrp.org/
- Doing What Works Clearinghouse,
USED -http//ies.ed.gov/ncee/wwc/ - Preschool for California's Children
- Hfrp_pubs_at_gse.harvard.edu
- Harvard Family Research Project, 2013
- http//www.hfrp.org/evaluation/overview
- Georgia Department of Education, FLP Report
- http//www.gadoe.org/School-Improvement/Federal-Pr
ograms/Pages/default.aspx
47Contact Information
- Sue Hawks Foster
- FLP Education Specialist
- sfoster_at_doe.k12.ga.us
- 404-656-2636