Welcome to the Summative Evaluation Presentation

1
Welcome to the Summative Evaluation Presentation

• Moderator Pat Gonzalez, OSEP
• Speakers Tom Fiore, Westat Michele Rovins,
  TACC and Joanne Cashman, IDEA Partnership

2
Rationale for the Center to Improve Project
Performance (CIPP)

• OSEP seeks a way of improving the rigor and
  objectivity of project evaluations so that it can
  reliably use these results for strategic
  planning.

3
CIPP Description

• CIPP does not provide an external summative
  evaluation. CIPP provides technical assistance
  to projects to improve the quality of their
  summative or formative evaluations, and for
  selected projects, performs some summative
  evaluation tasks to enhance the objectivity of
  the findings.

4
Broader Benefits

• Opportunities for grantee professional
  development through collaborative work and
  technical assistance in evaluation planning,
  methods, analysis and reporting.
• The potential to glean best practices in
  project design or implementation by examining
  outcome data across projects in areas such as
  scale-up, TA or dissemination strategies.

5
Broader Benefits - continued

• Improved project stability during the grant
  performance period (i.e., less divergence over
  time from the approved goals/activities in the
  application).
• The development of evaluation models or methods
  for similar projects.

6
Cross-Project Evaluation Questions for OMB

• As a result of the activities of selected
  OSEP-funded projects, how and to what extent have
  targeted professionals changed their knowledge,
  skills, or behaviors in ways that are consistent
  with scientifically based evidence?
• As a result of the activities of selected
  OSEP-funded projects, what changes have occurred
  in State and local policy, practice, and
  implementation strategies that are consistent
  with scientifically based evidence?

7
Cross-Project Evaluation Questions for
OMB-continued

• As a result of the activities of selected
  OSEP-funded projects, what policy, program,
  management, and organizational changes have
  targeted organizations made that support the use
  of evidence-based interventions and practices?

8
Cross-Project Evaluation Questions for
OMB-continued

• With reference to their logic models, to what
  extent did the selected OSEP-funded projects
  ultimately (a) achieve short-term and
  intermediate-term outcomes (intended or
  unintended), (b) reach targeted customers, (c)
  meet the identified needs of the targeted
  customers, and (d) distribute the benefits of the
  products and services fairly across the targeted
  customers?

9
Center to Improve Project Performance
Presentation by Thomas Fiore, Ph.D. Project
Director Westat
Presentation to OSEP TAD Network
Conference March 24, 2009
10
Presentation Outline

• Presentation outline
• CIPPs goals
• CIPPs role with selected projects
• CIPPs role providing general TA
• Elements of the evaluation process
• Logic model
• Evaluation plan
• CIPP contact information

11
CIPPs Goals

• CIPPs fundamental goal is to improve the
  summative evaluations of OSEP-funded projects,
  thus ensuring that OSEPs investments are
  supported by evaluations that reach the level of
  objectivity required by the Office of Management
  and Budget (OMB).
• Additionally, CIPP will support projects in
  conducting high-quality evaluations that will be
  sufficiently rigorous and objective for OSEP to
  use, not only for program performance
  measurement, but also for strategic planning.

12
CIPPs Role

• CIPP role is two-fold
• To guide, coordinate, and oversee the summative
  evaluations of 12 large grant-funded projects
  selected by OSEP.
• To provide technical assistance to OSEP grantees
  in conducting formative and summative evaluations
  of their projects funded through the following
  programs Parent Information Centers, Technical
  Assistance and Dissemination, Personnel
  Development, and (as of April) Technology.

13
CIPPs Role with Selected Projects

• With selected projects, work with project staff
  and the OSEP project officer to . . .
• Refine or clarify the purposes, objectives, and
  outcomes of the project by refining the projects
  logic model.
• Identify sources of evaluation information,
  determine samples, and develop data collection
  instruments and protocols.
• Develop a summative evaluation design.

14
CIPPs Role with Selected Projects

• With selected projects, work with project staff
  and the OSEP project officer to . . . cont.
• Develop the specific summative evaluation plan
  based on the design, to include
• What at the service settings of interest?
• Who are the targeted recipients?
• What will be evaluated?
• What are the evaluation questions?
• What data are required to demonstrate project
  effectiveness?
• What data collection activities are needed?
• How will the data be analyzed?
• What are the necessary timelines, staff
  assignments, and cost allocations across years?
• How will this be documented and reported?

15
CIPPs Role with Selected Projects

• With selected projects, CIPP staff will also . .
  .
• Monitor the implementation of the plan.
• Conduct reliability checks on the data collection
  activities.
• Set up a database to store the raw summative data
  (w/o personally identifiable information
  regarding children or families).
• Analyze project-collected data and interpret the
  findings.
• Provide project staff with an outline for their
  summative evaluation report.
• Provide a substantive and methodological review
  of the summative evaluation report, and provide
  other edits as needed.

16
CIPPs Role Providing General TA

• CIPPs second role is to provide broad, general
  TA to OSEP-funded projects
• Provide TA in writing or by telephone to all
  OSEP-funded TA D, Personnel Development, Parent
  Center, and (as of April) Technology projects
  that submit evaluation questions.
• Conduct one annual evaluation workshop/presentatio
  n at a conference to be determined by OSEP.

17
CIPPs Role Providing General TA

• Vehicles for accessing TA
• CIPP website (www.cippsite.org) provides an
  opportunity for OSEP-funded projects to pose
  questions or ask for assistance.
• Grantees who visit the website are asked to
  request a user account by providing their name,
  OSEP grant number, and email address. This
  request triggers an automated process that
  provides a login and password.
• Once logged in, projects can ask questions
  concerning evaluation design, methods, data
  analysis, and reporting using the Submit a
  Request link.
• CIPP website contains information and reference
  materials on formative and summative evaluation.

18
CIPPs Role Providing General TA

• Vehicles for accessing TA cont.
• CIPPs website also includes discussion boards
  where grantees can share information and engage
  in dialogue regarding evaluation topics
• Evaluation Design
• Instrument Design
• Sampling
• Data Collection and Processing
• Analysis
• Report Preparation

19
CIPPs Role Providing General TA

• Vehicles for accessing TA cont.
• Toll-free telephone line (1-888-843-4101) also
  enables certain OSEP-funded projects to pose
  questions or ask for assistance.
• Evaluation Briefs that discuss information
  requested with some frequency and that can be
  sent to grantees or downloaded from the website.

20
CIPPs Role Providing General TA

• What CIPP staff will do under general TA
• Answer questions from certain OSEP-funded
  grantees concerning project-specific evaluation
  design or measurement.
• Provide examples of data collection methods and
  instruments, analysis techniques, relevant
  literature, and report formats.
• Review project-specific documents related to
  formative or summative evaluations.

21
CIPPs Role Providing General TA

• Examples of what CIPP staff will NOT be able do
  under general TA
• Design instruments
• Design sampling plans
• Collect data
• Analyze data
• Provide support to an applicant for grant funding

22
Elements of the Evaluation Process

• Steps necessary for effective evaluation
  planning
• Logic model
• project plan
• program theory
• Evaluation plan
• need
• design
• data collection
• sampling
• analysis
• reporting

23
Elements of the Evaluation Process

• Logic model . . .
• Portrays a projects overall plan.
• Clarifies the relationships among a projects
  goals, activities, and outputs.
• Displays the connections between those defining
  features of a project and its expected outcomes.
• Evaluation can then be viewed as a test of the
  logic models implied hypotheses.
• Logic models can be used to refine and guide data
  collection and analysis for assessing both
  project processes (formative) and project
  performance (summative).

24
Elements of the Evaluation Process

• Logic model components used by CIPP when working
  with the selected projects
• Goals/Objectives overarching purposes of the
  project anticipated impact on systems or
  individuals.
• Strategies/Activities strategies are the broad
  approaches to addressing the goals activities
  are specific actions.
• Outputs direct results of the activities,
  including project products and programs most
  will be quantifiable, such as tallies of the
  number of products and programs or counts of the
  customer contacts.

25
Elements of the Evaluation Process

• Logic model components used by CIPP when working
  with the selected projects cont.
• Direct/Intermediate Outcomes direct outcomes
  are what customers do or become as a result of
  outputs, usually changes in the customers
  actions or behaviors based on knowledge or skills
  acquired through project outputs intermediate
  outcomes result either directly from outputs or
  indirectly through direct outcomes and often
  represent a step between direct outcomes and
  long-term outcomes.
• Long-term Outcomes broadest project outcomes,
  follow logically from the direct and intermediate
  outcomes the results that fulfill the projects
  goals.

26
Elements of the Evaluation Process

• Logic model graphic

27
Elements of the Evaluation Process

• Evaluation plan CIPP is using with the selected
  projects
• Statement of need project is designed to address
• Description of goals and activities
• Projects program theory (logic model)
• Evaluation design
• Evaluation questions
• Approach

28
Elements of the Evaluation Process

• Evaluation plan CIPP is using with the selected
  projects cont.
• Data collection
• Data sources
• Instruments
• Schedule
• Sampling
• Analysis
• Reporting

29
CIPP Contact Information

• Website www.cippsite.org
• Toll-free telephone [phone]