Evaluation Planning; forms of Evaluation; evaluation Stages

1
Evaluation Planning forms of Evaluation
evaluation StagesChen and Holden-Zimmerman

• Mario A. Rivera
• PA 522
• School of Public Administration

2
Practical Program Evaluation Assessing and
Improving Planning, Implementation, and
Effectivenessby Huey-Tsyh Chen

• Chens approach (Chapter 3) Chen proposes a
  taxonomy of program evaluation, one built around
  the program stage that is the desired focus of
  the evaluation, as well as around the desired
  purpose of the evaluation (for either program
  improvement or program impact assessment).
• Program Planning Stage. The first of the four
  stages is the program planning stage. This is the
  very beginning of program evaluation.
  Stakeholders at this stagefor example, program
  designers and managers, other program principals,
  and clients or beneficiariesare developing a
  plan that will serve as a foundation for
  specifying, organizing, and implementing a
  program at a future date.

3
ChenSecond Stage, Implementation

• Implementation Stage. Program evaluation has, for
  much of its history, focused principally, and
  narrowly, on outcomes. Evaluation practice,
  however, suggests that program failures are often
  essentially implementation failures.
  Consequently, the practical scope of program
  evaluation has gradually broadened to include
  process evaluation, i.e., evaluation of
  implementation process, or processes. Focus on
  process is necessary when looking for
  explanations for shortfalls in program results.
• The current view is that a much of implementation
  failure can be traced back to poor program
  planning, and to poor program design and
  development. Evaluators can make important
  contributions to programs by assessing these
  developmental steps. Consequently, there needs to
  be concern with the entire programmatic arc in
  evaluation program planning, articulating
  program theory (theory of change), assessing
  implementation, and outcomes assessment.

4
ChenThird Stage, Mature Implementation

• Mature Implementation Stage. This stage follows
  initial implementation at a point when the
  program has settled into fairly routine
  activities and tried-and-true ways of doing
  things. Rules and procedures for conducting
  program activities are now well established.
  Stakeholders are likely to be interested in the
  following determination of the sources of
  immediate problems, accountability (generation of
  data reassuring to those to whom stakeholders are
  accountable), and continuous program improvement.
  
• Even in maturity, a program is likely to have
  problems such as client dissatisfaction with
  services. Identifying and resolving these
  problems is key to improving a program. And, as a
  program matures, stakeholders may think more
  about accountability, requiring concerted effort
  the direction of performance monitoring and
  reporting.

5
ChenFourth Stage, Program Outcome

• Outcome Stage. A fourth stage of program growth
  is known as the outcome stage. Following a period
  of program maturity, stakeholders inside and
  outside the program want to know more or less
  definitively whether the program is achieving its
  goals.
• An evaluation at this point can serve any of
  several evaluation needs, including merit
  assessment and fidelity assessment (how well the
  program has functioned, whether it was
  implemented as planned, and how closely it has
  come to projected outcomes). However, Chen
  reminds us in his writings that there needs to be
  an adaptation of fidelity assessment to program
  evolutionthe fidelity-adaptation approach. This
  even pertains to mature, well-settled programs.

6
Evaluation Phases, Purposes Types
Design-phase or Developmental Evaluation Helps
ensure that programs are well conceived, well
designed (e.g., in reference to well-established
best practices)
Program Development Phase
Formative and/or Process Evaluation Formative
evaluation helps improve the program
implementation and management. Process evaluation
focuses on assessment of program operational and
management process(es). Evaluations can be both
at once.
Program Implementation Phase
Summative or Outcome or Impact
Evaluation Helps determine whether and to what
extent a program has worked, by gauging its
demonstrable effects (results, outcomes)
Program Outcome Phase
7
Further Defining Evaluation Phases and Types

• Design-phase/Developmental Evaluation Conducted
  before or early in program implementation,
  testing evaluation plans, change models
  (rationalse), action models (implementation
  plans), etc.
• Formative Evaluation designed to determine
  (especially during later developmental phases of
  an intervention) (1) the feasibility of program
  implementation (2) the aptness of change and
  action models and (3) the short-term social,
  behavioral, or other impacts of an intervention.
  Focused on program improvement.
• Process Evaluation Designed to ascertain the
  degree to which replicable program procedures
  were implemented with fidelity by trained staff
  according to an articulated program plan black
  box, systems-based evaluation. Assesses program
  process(es). If also meant to inform and improve
  these, it may properly be called a formative
  evaluation as well. However, formative and
  process evaluation are too often used
  interchangeably, blurred.
• Outcome or Impact (Summative) Evaluation
  Intended to assess the feasibility, efficacy, and
  cost-effectiveness of a program intervention in
  producing significant, long-term benefits for a
  well-defined population. Results-oriented
  evaluation for accountability.

8
Chens Evaluation Strategies classification
Holden Zimmerman on evaluation planning

• Chen proposes four evaluation strategies that
  correspond to program phases as just discussed
  (1) assessment strategies (judging the results
  or performance of an intervention effort) (2)
  developmental strategies (judging the planning
  and early implementation of the intervention)
  (3) theory-elucidation or enlightenment
  strategies, which aim to make explicit the
  underlying assumptions and change models and
  action models of an intervention (often at early
  program stages) and (4) partnership strategies
  (ways of involving stakeholders, as well as other
  organizations in strategic and operational
  collaboration, and ways of evaluating such
  engagement).
• The distinction is based on the purpose or
  objectives of the evaluation, and what aspect of
  a program is under scrutiny. More than one of
  these efforts could be undertaken in one
  evaluation, probably at different stages of the
  evaluation.
• Both Chen and Holden stress evaluation planning,
  a projective process that occurs prior to
  carrying out an evaluation.

9
Holden Zimmerman

• Planning for evaluation involves
• Stating the purpose of the evaluation
• Understanding a programs organizational and
  political context
• Determining the uses of the prospective
  evaluation
• Working with stakeholders to identify primary and
  secondary evaluation questions
• Ensuring stakeholders buy-in for the evaluation
• Holden and Zimmerman developed a model called
  Evaluation Planning Incorporating Context, or
  EPIC, which aims to engage stakeholders, describe
  the program, and focus the evaluation plan. It
  provides a way to address issues in the
  pre-implementation phase of program evaluation.
  There are five steps in the model, namely
  assessing context, understanding the
  organizational and political environment,
  defining relationships, determining level of
  evaluation, gathering reconnaissance, specifying
  evaluation uses, and validating evaluative
  perspectives.

10
Evaluation Planning Incorporating Context
(EPIC)Model Overview Review

• The EPIC model provides a heuristic (or set of
  rules, or rules of thumb) for evaluation planning
  rather than a specified set of steps that are
  required for all evaluations. Some parts of the
  model may be more or less applicable depending on
  such issues as the type of evaluation, the
  setting of the evaluation, the outcomes of
  interest, and the sponsor's interests. Therefore,
  the EPIC model can be used as a kind of
  instruction guide to prepare for a program
  evaluation.
• However, the EPIC model as such is not in
  particularly wide use. Evaluation practitioners
  do ordinarily undertake evaluation planning
  (distinct from but often connected to program
  planning), following similar steps, however.

11
Holden Importance of Evaluation Planning

• For Holden Zimmerman, planning the evaluation
  is key to building evaluation capacity. Planning
  an evaluation involves anticipating what will be
  required to collect information, organize it,
  analyze it, and report it, in short what will be
  involved in administering the evaluation
• Everything from the articulation of evaluation
  questions to data-collection strategies should
  (to the extent feasible) be undertaken
  collaboratively with stakeholders and program
  partners. Examples of questions Have critical
  program activities occurred on time and within
  budget? Why is site A performing better than site
  B despite identical programs?
• A cogent evaluation plan presupposes a strong
  program plan
• Things evaluator would need to know
• Which activities were viewed as critical?
• Program time frames, budget by activity
• When each activity began/ended
• Total cost of each critical activity

12
EPIC Model Sequence
13
The Holden text incorporates the CDC Program
Evaluation Framework, which stresses the
continuous nature of evaluation
Framework for Program Evaluation in Public
Health MMWR, 1999
14
Holden Zimmerman, Chen, Role-sharing

• As stated in the reading Role Sharing Between
  Evaluators and Stakeholders in Practice, and as
  Chen stresses, program evaluation has moved away
  from traditional objective observation and now
  strives to engage stake-holders more fully in the
  evaluative process. The Vancouver case suggests
  that sharing roles between evaluators and project
  leaders, peer educators, and others was the norm
  among study participants but varied by their
  orientation and role. There was some tension and
  confusion due to this role-sharing, the kind of
  cross-functionality which is likely to obtain
  most markedly early on in a collaborative process
  (and often in Community-based Participatory
  Research). Role sharing requires strong
  communications skills on the part of evaluators.
  When these skills are absent, role-confusion
  prevails. There needs to be role clarification
  then.
• How did role-sharing characterize the education
  evaluation case study in Holden Zimmerman?

15
Holden Zimmermaneducation evaluation

• The INEPs initial phase (1999-2002) involved
  adapting and testing the curriculum with
  community stakeholders. INEP was housed in the
  Rocky Mountain Prevention Research Center
  (RMPRC), where a resource teacher and staff were
  available during parent events and evaluation
  activities. After 2002, funding by the USDA
  allowed for continued support for INEP teachers,
  specifically staff assistance to shop for food,
  perform food preparation, organize teaching
  materials, and clean up after lessons.
• The INEP Public School Organization engaged
  teachers, principals, the district
  superintendent, the school board, and the State
  Board of Education. A district health coordinator
  knowledgeable about the program provided a
  critical link among agencies to provide the
  program with greater visibility. All of these
  actors became involved in the evaluation in
  varying degrees, so that role-sharing clearly
  obtained in both program implementation and
  evaluation.

16
The education evaluations challenges

• Evaluators had to determine whether desired
  curriculum adaptations had taken place, whether
  unanticipated changes in program and context
  could be expected to alter evaluation findings,
  and, in general, whether the program was
  effective.
• It is difficult to measure changes in eating
  behavior. And there are many different barriers
  to healthy eating, particularly socio-economic
  status. You can teach children good eating habits
  and good food choices but some families are
  unable to afford healthy foods, or there may not
  be readily substitutable healthy foods for
  cultural staples (such as tortillas and other
  starch-heavy staples), or such changes may not be
  culturally sensitive or desirable. This was not a
  culturally-responsive evaluationculturally
  unaware.
• Another problem is the burden placed on the
  teachers to carry out these programs even though
  they are overextended already.
• The evaluators also noticed that there were many
  barriers to deal with in order to obtain
  necessary permissions to conduct research in
  public schools. It is essential to understand the
  political hierarchy of the system and the
  legal-regulatory requirements involved, in order
  to gain required approvals.

17
Planning a Service Program Evaluation

• Mari Millery was called in by the Leukemia and
  Lymphoma Society (LLS) to plan and conduct a
  pilot study for its Information Resource Center
  (IRC). In Planning for a Service Program
  Evaluation, Millery discusses the steps she took
  in the planning stages of the program evaluation.
  The goal of the pilot study was to enhance
  patient navigation through the IRC by way of an
  intervention program. Both management and the
  evaluation team wanted to see the short-term
  results of this pilot study before fully
  launching the intervention program. One
  short-term goal was to ensure the feasibility of
  the program before implementation. Ultimately,
  the evaluation team and LLS wanted to produce
  positive impacts on the patients level of care
  and quality of life. Millerys was an instance of
  a developmental (or design-phase) program
  evaluation.

18
Planning a Service Program Evaluation

• Chen stresses the importance of developing a
  program rationale concurrently with the
  development of a program plan. The program
  rationale can be to correspond closely to the
  change model for the evaluation, while the
  program plan is a key element of its action
  model. The main purposes of the program rationale
  are to define a target group as well as to
  specifically explain why those in the target
  group were selectedfor instance, in reference to
  a needs assessment. Program rationales provide
  support for three main tactics necessary for the
  proper planning of a program evaluation (1)
  establishing a foundation for planning, (2)
  effective communication, and (3) adequately
  providing for the evaluation of outcomes. A
  program rationale will serve as a guide that
  evaluators can follow throughout the planning
  process it will also support effective
  communications between evaluators and
  stakeholders.

19
Planning a Service Program Evaluation

• Chen also discusses strategies and approaches for
  articulating program rationales. He begins with a
  background information provision strategy whereby
  evaluators gather relevant information on things
  such as the characteristics of the target group,
  community needs, previous evaluative studies, and
  so on.
• In this context, Chen points out the value of
  both needs assessment and formative research. A
  needs assessment serves to identify, measure, and
  prioritize community needs. This in turn can aid
  in the process of goal-setting and target group
  selection, as well as in the subsequent steps of
  engaging stakeholders, specifying the programs
  change and action models (or program theory), and
  focusing the evaluation. In all, needs assessment
  provides the basis for program rationale.

20
Planning a Service Program Evaluation

• Chens approach parallels the EPIC model for
  organizing program evaluation efforts. Both
  entail 1) assessing context 2) assessing the
  need for the programits rationale, and how the
  evaluation is to be used 3) engaging
  stakeholders 4) describing the program and 5)
  focusing the evaluation.
• Under either approach, it is important to begin
  with an understanding of the organizational and
  political context of the given program, so as to
  understand in turn why the program is deemed
  necessary, and how it is to be implemented.
  Gathering reconnaissance is a critical step in
  the EPIC model which specifies how the evaluation
  will be used, and which should be consistent with
  prior needs assessments and with evaluation
  planning in general. The community service
  evaluation particularly stressed the third step
  in the EPIC Model, engaging stakeholders.

21
Chens approach

• The conceptualization facilitation approach is
  a key topic in Chens discussion of program
  rationales. Subtopics include whether to use a
  facilitative working group and whether to rely on
  an intensive interview format in assessing
  stakeholders backgrounds and preferences. Noting
  the frequent tendency of stakeholders to set
  high, unachievable goals, Chen stresses the need
  to set realistic short- to long-term program
  aims. Short-term objectives serve as a kind of
  grading sheet by which program staff and
  evaluators can note and measure tangible
  successes.
• A pitfall to be avoided, according to Chen, is
  confusing objectives with action steps. Here he
  is distinguishing between the program rationale
  (or change model) and program implementation plan
  (action model). Clarification and consistency of
  program goals is necessary to avoid creating
  incompatibility among goals and objectives.

22
Chens approach

• Chen describes program plans as blueprints for
  the actions prescribed by program rationales. In
  chapter five, How Evaluators Assist Stakeholders
  In Developing Program Plans, Chen moves from the
  why to the how part of helping clients with
  program planning. Concerns range from recruiting
  and training implementers to formulating research
  questions.
• Chen emphasizes the importance of a simple,
  clear, and realistic program rationale in order
  to develop an effective program plan and then an
  evaluation plan. Unnecessarily complex and
  over-detailed program rationales expose
  evaluators to complications in the evaluation
  planning stage. Checking program rationales
  against best practices among similar
  organizations and programs is one way to
  streamline and validate these. One should fully
  explore how these successful efforts may serve as
  templates for the focal program.

23
Chen the Service Program Evaluation case

• An action model framework can help the evaluator
  in facilitating the development of a new program
  plan, as seen in Millerys work. An action model
  is a means to ensure that there are no gaps or
  inconsistencies in the action plan over against
  implementation. In other words, it serves as a
  kind of proofreading tool for evaluators during
  planning stages. It is also a check on how
  various program activities work together in
  actual implementation.
• Chen describes a formative research approach in
  the context of a background-information provision
  strategy. The two main purposes of this approach
  are (a) to formulate research questions and (b)
  to gather data to answer these questions. The
  action model can help evaluators determine which
  questions should be researched, gaining insight
  into the program in order to develop a cogent
  evaluation plan.

24
Chen the Service Program Evaluation case

• Chen specifies six steps for planning an
  evaluation, which one can compare to
  Millerys/EPIC approach to evaluation planning
• 1. Assess, enhance, and ensure implementing
  organizations capacity. Parallels Preskill and
  Boyles Evaluation Capacity-building (ECB) Model
  as well as the EPIC model.
• 2. Delineate service content and delivery
  procedures
• 3. Recruit, train, and maintain the competency
  and commitment of, program implementers
• 4. Establish collaboration with community
  partners
• 5. Ecological context seek external support
• 6. Identify, recruit, screen, and serve the
  target population
• With regard to the implementing organization,
  Chen indicates that technical expertise, cultural
  competence, and manpower need to be considered.
  Technical expertise can determine the readiness
  of the implementers to carry out the necessary
  interventions and to help with the evaluation.
  Cultural competence provides them with effective
  methods of communication with clients.

25
Chen the Service Program Evaluation case

• The program plan should be specific about
  services provided (the intervention protocol). A
  clear explanation of program services and how
  they are to be provided is also necessary (the
  service delivery protocol). According to Chen,
  the best method of providing for apt intervention
  and service delivery protocols is one-to-one
  interaction with program principals.
• Consistent with Chens approach, in conducting
  the Service Program Evaluation Millery was able
  to work one-on-one with colleagues in an agency
  that greatly valued evaluation and was willing
  and able to provide her with essential
  information about organizational and program
  staff capacity, recruitment and training
  protocols, and incentives for program
  participants.
• Chen stresses the importance of collaborative
  networks for both the implementers and evaluators
  of programs. In Millerys case, she was
  evaluating the work of a large health services
  provider that was well connected with similar
  organizations and had experience with them.
  Millery built on and amplified these
  collaborative relationships in carrying out the
  evaluation.

26
Chen the Service Program Evaluation case

• If one uses Chens approach as a checklist,
  Millery was a very thorough evaluator. To begin
  with, she sought to provide a program rationale
  in addition to the program plan. Initially, when
  she responded to the request for proposals (RFP),
  LLS outlined for her the idea of patient
  navigation and desired outcomes. However,
  according to Millery, LLS did not explain where
  the concept of patient navigation originated or
  whether there was a theoretical or conceptual
  framework behind it, so Millery proceeded to
  fill these gaps in further discussion with LLS
  principals.
• Chen stresses the importance of gathering
  background information on both the agency and the
  program staff. Millery did this in several ways.
  At the outset, she researched the history of the
  agency, its code of conduct and mission
  statement, and so on. She studied the
  organizational structure of LLS and consulted LLS
  staff. Also, she was able to become familiar with
  the politics surrounding the relevant advocacy
  issues. She also found previous evaluative
  studies on which to build her own.

27
Planning a Service Program Evaluation

• In assessing the context of the program, Millery
  studied the organizational infrastructure of LLS.
  Specifically, she focused on how the IRC is
  structured as a department within LLS. She also
  familiarized herself with LLSs management model,
  which turned out to be much like that of a
  private sector organization. This information was
  vital in order to for her to be able to
  understand how evaluation findings were to be
  developed and used.
• Through background research, Millery was able to
  draw up a preliminary list of questions that
  would guide her initial conversations with LLS
  management. She categorized these questions by
  their relevancy and priority. An example of a
  question that would be asked early on From where
  does the funding for this evaluation study come?
  Another was Why specifically target clients in
  Wyoming?

28
Planning a Service Program Evaluation

• Millery requested that LLS explain the specific
  services provided by IRC (the intervention
  protocol) and also how they were to be provided
  (the service delivery protocol). According to
  Millery, it is much easier to start planning for
  a study when the evaluator has a clear picture of
  how the service is provided. Consistent with
  Chen, Millery wanted to be certain that service
  delivery processes were laid out clearly in order
  to avoid having any gaps in either program plan
  or action model.
• Millery had the advantage of working with
  well-trained professionals who supported the
  evaluation. She had little to do when it came to
  Chens third step Recruiting implementers and
  training them was not an issue. However, Millery
  did have to define the relationship she had with
  IRC staff as well as with LLS supervisors who
  were in charge of overseeing the program
  evaluation, i.e., role clarification. She had a
  great deal of one-on-one interaction with program
  staff and management, an important part of
  evaluation planning.

29
Planning a Service Program Evaluation

• Consequently, Millery and LLS supervisors agreed
  on a single liaison between herself and the LLS
  board of directors. This simple relationship
  helped Millery avoid any complicationsspecificall
  y, role abmiguity or confusionthat might arise
  from multiple and complex relationships with LLS
  principals. Millery also considered the
  importance of maintaining staff confidence. She
  made it a point to individually interview each
  program staff member and make it clear that the
  evaluation strictly concerned the program rather
  than the staff. It was to be a program-performance
  evaluation, not individual performance
  evaluation.
• Like Chen, Millery valued collaborative
  networking and partnership. She sought out
  organizations that provided services similar to
  those provided by LLS. In fact, she was able to
  find information on previous survey studies
  performed by both LLS and these other
  organizations. This information not only helped
  her formulate research questions, but it also
  helped her specify program determinants
  (mediators and moderators) that were germane to
  program implementation and evaluation processes.

30
Planning a Service Program Evaluation

• Why Wyoming residents only? LLS managers
  explained that Wyoming did not have a local
  chapter and therefore the residents of that state
  could benefit most from the enhanced IRC (phone)
  services, so that these would be evaluated in
  Wyoming in lieu of direct-contact chapter
  services.
• In focusing the evaluation, Millerys methods
  mirror those of Chen. After her initial research
  of background information on LLS, formulation of
  research questions, examination of collaborative
  networking, and clarification of roles and
  relationships with program managers and staff,
  Millery was able to better gauge the programs
  (strategic and operational) goals and objectives
  and, correspondingly, establish the evaluations
  goals and objectives. From there, she was able to
  determine the feasibility of the evaluation and
  the evaluability (evaluation readiness) of the
  program.

31
Chen the Service Program Evaluation case

• Chen stresses the importance of establishing a
  program rationale (program theory, or change and
  action models) in delineating the rationale of an
  evaluation. Millery helped the LLS evaluation
  team members clearly define the rationale, or
  change model and intended purposes, of their
  partnered evaluation.
• Chen likewise stresses the differences between a
  change model and an action model. Millery was
  able to articulate the why and the how of the
  evaluation in the planning stage in relation to
  program theory, both the programs theory of
  change and its implementation model. This effort
  in turn allowed her to gauge implementation
  fidelity and success.
• Consistent with Chen, Millery engaged
  stakeholders in the program evaluation to the
  maximum extent possiblestakeholder involvement
  was central to the evaluation. It also helped
  build evaluation capacity in the organization.

32
Chen the Media Evaluation case

• This case involves an effort to bring about
  behavioral change through social marketing. The
  Truth campaign media evaluation demonstrates how
  fidelity evaluation can be effective in assessing
  social marketing initiatives.
• Chen describes the Fidelity Evaluation Approach
  as a major evaluation method well-fitted to a
  mature implementation stage. Fidelity evaluation
  is principally a process evaluation approach that
  gauges the degree of congruence between program
  change and action models (program theory), on the
  one hand, and the program intervention as
  implemented, on the other. Target population
  fidelity evaluations assess whatever element of
  the change and action models are of special
  interest to stakeholders. Since outcomes are of
  vital interest to stakeholders funding or
  otherwise supporting the program, fidelity
  evaluation is also concerned with program
  impacts, and specifically impacts on intended
  populations.

33
Planning for a Media Evaluation

• Health communication may be defined as a complex
  of techniques and initiatives intended to inform,
  influence, and motivate individual,
  institutional, and public audiences about
  important health issues.
• Social marketing is a vehicle for health
  communication that seeks to influence social
  behaviors, not to benefit the marketer but to
  benefit the target audience and society as a
  whole (Kotler et al., 1971, in Holden p. 124).
  Social marketing is the systematic application of
  commercial marketing concepts and techniques so
  as to achieve specific behavioral goals for a
  social good. It attempts to prompt healthy
  behaviors in a population by using some of the
  proven marketing techniques used to promote
  commercial products (Kotler et al., 1996, in
  Holden p. 124).

34
Media Evaluation case

• Media campaigns, and media evaluations, are based
  on social marketing theory and behavioral theory,
  including theories of exposure, messaging,
  communication, and behavior change (Hornik, 2002,
  in Holden p. 124).
• Media evaluation may be divided into process and
  outcome evaluation methods, as follows
• Process evaluation helps to assess whether the
  target audience has been exposed to a campaigns
  messages and whether the target audience reacts
  favorably to the messages as delivered in
  real-world circumstances.
• Outcome evaluation helps to determine the effects
  of messages on health behavior and determinants
  of behavior, such as health knowledge, attitudes,
  and beliefs. Media evaluations often capture
  process and outcome data simultaneously to offer
  the immediate formative feedback that can enhance
  the campaign effort. (Evans et al., in Holden p.
  124)

35
Media Evaluation case

• When (1) immediate reactions to media messages,
  (2) longer-term recollections of these, and (3)
  associated health outcomes are correlated,
  process and outcome evaluation efforts are
  brought together.
• As a result of the Master Settlement Agreement
  between tobacco companies and 46 states, the
  American Legacy Foundation initiated the national
  truth campaign in February 2000. From 2000 to
  2002, annual funding for the campaign averaged
  100 million per year. National media purchase
  was employed by the campaign, as opposed to a
  randomized exposure design, for two primary
  reasons. First, it was considered that the
  campaign could not ethically assign some media
  markets to low or zero exposure, given the
  documented successes of the predecessor Florida
  truth campaign. Second, a national media
  purchase was roughly 40 cheaper than a
  market-to-market purchase, which would have been
  necessary to randomize exposure.

36
Media Evaluation case

• The truth campaign evaluation used data from the
  19972002 Monitoring the Future annual spring
  surveys, which were designed to monitor alcohol,
  tobacco, and illicit drug use among youths in the
  United States. The survey, funded primarily by
  the National Institute on Drug Abuse and
  conducted by the University of Michigans
  Institute for Social Research, was a
  self-administered questionnaire, involving about
  18,000, 17,000, and 16,000 8th-, 10th-, and
  12th-grade students a year, respectively.
• In-school surveys such as the National Youth
  Tobacco Survey (NYTS) and Monitoring the Future
  (MTF) are more appropriate for measuring
  substance use because they are self-administered
  without the presence of parents or others who
  could motivate youth to provide socially
  desirable responses to substance questions. With
  its large national sample and coverage of major
  media markets where the truth campaign was
  advertised, MTF became the cornerstone of
  evaluation planning efforts to assess the
  campaigns impact on youth smoking behaviors.
  (Evans et al., in Holden p.129)

37
Chens target population fidelity evaluation

• Chens target population fidelity evaluation
  looks at programs contact with their target
  populations. Chen writes, Programs must reach
  sufficient numbers of clients from the specified
  target population in order to be effective (Chen
  p.169).
• To conduct a target population fidelity
  evaluation, evaluators need to ask three main
  questions. First, how many clients were served by
  the program during a specific period? Second, how
  many of the clients served come from the target
  population? And third, upon determining how many
  clients served come from the target population,
  based on that number, the evaluators next
  question elicits a judgment call about a
  programs performance Does the number of clients
  served justify the programs existence?

38
Chen and the truth campaign evaluation

• Chen indicates that the evaluator must remain
  aware of the distinction between clients
  recruited and clients served. This was the case
  with the truth campaign evaluators. In its first
  year, the campaign reached three fourths of
  American youths and was associated with
  campaign-related changes in youth attitudes
  toward tobacco and the tobacco industry. (Siegel,
  2002, in Farrelly p.431)
• All survey-based analyses have limitations.
  Measures of youth smoking prevalence are
  self-reported and may be subject to social
  desirability bias so that youths are less likely
  to report smoking in areas with high exposure to
  the campaign than in areas with lower exposure.
  This would lead to an overstatement of the
  campaigns effects. However, some studies have
  found that underreporting of smoking by youths is
  actually minimal.

39
Chen and the truth campaign evaluation

• Results also rely on repeated cross-sectional
  surveys, not repeated measures on the same
  students, which weaken the strength of causal
  inference (Messeri et al., 2002, in Farrelly
  p.430). Evaluators included youths surveyed
  before 2000 as well, so that students in the
  19971999 surveys served as an unexposed
  cross-sectional control group.
• For the purpose of the truth campaign, the
  second component of target population fidelity
  evaluation that must be addressed as, Is it
  possible that the estimated truth campaign
  effects may have been due to other unmeasured
  youth-focused prevention activities (e.g.,
  in-school substance abuseprevention programs,
  the national antidrug campaign by the Office of
  National Drug Control Policy, other media-borne
  messages, secular trends in social behavior) that
  were correlated by chance with the truth
  campaign exposure? This is the attribution
  question in causal analysis (Chen).

40
Chen and the truth campaign evaluation

• Following a socio-ecological model (Glanz et al.,
  1997, in Farrelly p.431) that recognizes multiple
  levels and types of influence on health behaviors
  (in particular, intrapersonal, interpersonal,
  community, media, policy, economic), evaluators
  controlled for a wide array of potential
  confounding influences.
• Considering the Media Market Level Low-exposure
  markets tended to be more rural, White, and less
  educated, and have lower incomesall factors
  associated with smokingthan markets with high
  campaign exposure. Failing to control for these
  factors (high pre-treatment smoking rates coupled
  with low exposure to truth campaign messages)
  could lead to a spurious negative correlation
  between campaign exposure and smoking prevalence.
  Evaluators statistically modeled possible
  correlations between preexisting media market
  smoking rates and the subsequent campaign dose.
  (Heckman et al., 1989, in Farrelly p.431). This
  controlled for average market-level smoking
  rates, effectively making each market its own
  control group. Evaluators also included direct,
  local media marketlevel measures of potential
  confounders.

41
Chen and the truth campaign evaluation

• Chens third and ultimate component of target
  population fidelity evaluation asks, Does the
  number of clients and distribution of clients
  served justify the programs existence? Findings
  suggest that the truth campaign may have had the
  largest impact among 8th-grade students, which is
  consistent with evidence from Florida that
  indicates the Florida truth campaign led to
  declines in smoking rates and that smoking rate
  declines were greatest among middle school
  students (grades 6 through 8) from 1998 to 2002.
  (Farrelly et al., p.427)
• In addition to being consistent with previous
  findings, this study improves on previous
  research by reaching generalized conclusions
  about the effects of antismoking campaigns for
  youths across the U.S. and by implementing a
  pre/post quasi-experimental design that
  controlled for potential threats to validity,
  such as secular trends in smoking prevalence, the
  influence of cigarette prices, state tobacco
  control programs, and other factors.

42
Chen and the truth campaign evaluation

• This result was confirmed in multivariate
  analyses that controlled for confounding
  influences and indicated a dose-response
  relationship between truth campaign exposure and
  current youth smoking prevalence.
• The evaluators found that by 2002, smoking rates
  overall were 1.5 percentage points lower than
  they would have been in the absence of the
  campaign, which translates to roughly 300,000
  fewer youth smokers based on 2002 US census
  population statistics. (Farrelly et al., p.428).
  That was the actual impact attributed to the
  campaign.
• In sum, the truth campaign was effective,
  demonstrably associated with significant declines
  in youth smoking prevalence.

43
Truth Campaign impact (marginal impact equals
projected smoking rates without program versus
rates with program)
44
Chen and the truth campaign evaluation

• The evaluators found that implementers were
  consistenti.e., faithful to the socio-ecological
  change/action model underlying the programin
  their execution of the program, and this model
  therefore became the efficacy test in the
  evaluators assessment of the campaign. The
  program also made consistent use of social
  marketing theory (involving vectors of exposure,
  messaging, communication, and behavior change).
• Therefore, program fidelity in the sense of close
  theory- implementation correspondence was high,
  as was target population fidelity. Consistent
  with the Results-mapping approach (Reed, et al.),
  the truth campaign was impactful in both the
  quantity (extent) and quality of results
  attained.
• 1. Evans, W. D., Davis, K. C., Farrelly, M. C.
  (2009). Planning for a Media Evaluation. In
  Holden, D. J., Zimmerman, M. A., A Practical
  Guide to Program Evaluation Planning, pp.
  123-142.
• 2. Farrelly, M. C., Davis, K. C., Haviland, M.
  L., Messeri, P., Healton, C. G. (2005).
  Evidence of a DoseResponse Relationship Between
  truth Antismoking Ads and Youth Smoking
  Prevalence. American Journal of Public Health,
  Vol. 95, No. 3, pp. 425-431.

45
The contribution/attribution challenge

• Attribution for outcomes always a challenge
• A credible performance story needs to address
  attribution
• Sensible accountability needs to address
  attribution
• Complexity significantly complicates the issue
• Attribution is based on the theory of change
  (change model) of the program, and it is
  buttressed by evidence validating the theory of
  change,
• Attribution is einforced by examination of other
  influencing factors,
• Contribution analysis builds a reasonably
  credible case about the difference the program is
  making. Attribution determinations are based on
  analyses of net program impact (program
  contribution).

46
Attribution

• Outcomes not controlled there are always other
  factors at play
• Conclusive causal links dont exist
• You are trying to understand better the influence
  you are having on intended outcomes
• Need to understand the theory of the program
  (program theory), to establish plausible
  association
• Something like contribution analysis can help
• Measuring outcomes
• Linking outcomes to actions (activities and
  outputs), i.e. attribution Are we making a
  difference with our interventions?

47
Accountability for outcomes

• In order to be accountable, we need to credibly
  demonstrate
• The extent to which the expected results were
  achieved
• The contribution made by activities and outputs
  of the program to the outcomes
• The learning or other behavioral/social changes
  that have resulted, and, therefore
• The soundness and propriety of the intervention
  means used.

48
Contribution analysis

• There is a postulated theory of change
• The activities of the program were implemented
• The theory of change is supported by evidence
• Other influencing factors have been assessed
  accounted for
• Therefore
• The program very likely made a net contribution,
  to be gauged against the counterfactual What
  would have occurred, plausibly, in the absence of
  the program?

49
Theory of change Truth Campaign Change Model

• A results chain with embedded assumptions and
  risks is identified
• An explanation of why the results chain is
  expected to work what has to happen
• These two elements comprise the Change Model

Assumptions target is reached (national media
coverage), messages are heard, messages are
convincing, commercial-campaign techniques are
effective, non-smoking as rebellion concept
works, other major influences are identified and
their impact considered and measured. Risks
target not reached, poor message in some
contexts, lack of randomization introduces
validity issues, attribution difficulties
50
Other influencing factors

• Literature and knowledgeable others can identify
  the possible other factors (direct and indirect
  influences)
• Reflecting on the theory of change may provide
  some insight on their plausibility
• Prior evaluation/research may provide insight
• Relative size compared to the program
  intervention can be examined
• Knowledgeable others will have views on the
  relative importance of other factors

51
Chen Program Theory and Fidelity

• Theory-driven program evaluation
• All programs have implicit theories
• Program modeling (e.g., via logic models) helps
  make implicit (or tacit) theory more explicit and
  therefore subject to scrutiny
• Implementation fidelity
• Preserving causal mechanisms in implementation
• Scaling up
• Staying close to projected, intended outcomes
  (What of positive unintended outcomes? Or
  negative unintended consequences?)

52
Chen Program Implementation and Fidelity

• Intended modelImplemented model (is program
  implemented as intendedfocused on program action
  model)
• Normative theory (induced positive
  behavioral/social change that is intendede.g.,
  changing smoking behaviors)
• Causative theory (theory of change, change model)
• However, models too often substitute for reality
  (they should nota kind of formalism). Dangers
  of reification
• Models can support
• Assessment of evaluability (is the program
  ready to be evaluated, or how to ready a program
  for evaluationbased on the work of Joseph
  Wholey)
• Client needs and resource assessments
• Program development, refinement,
  capacity-building
• Monitoring and evaluation

53
Chen Program Implementation and Fidelity

• Formative and process forms of evaluation are
  undertaken to assess whether the program is
  proceeding as planned, the fidelity of
  implementation to program design (Chen), and the
  degree to which changes need to be made.
• Summative evaluation is conducted to asses
  whether planned outcomes have been achieved
  (fidelity of outcomes) and what impacts (intended
  and unattended) have occurred.
• Context for evaluating fidelityit may become
  evident that the program has strayed from its
  design but for good reasons, making for better
  outcomes if so, make all of that explicit.
• Considerations for conceptualizing fidelity
• Multilevel nature of many interventions
• Level and intensity of measurement aligned with
  need
• Capacity for monitoring fidelity
• Burden of monitoring fidelity
• Alignment with desired outcomes

54

• Program theory can be either descriptive or
  prescriptive
• Descriptive theory specifies what impacts are
  generated and how this occurs. It suggests a
  causal mechanism, including intervening factors,
  and the necessary context for program efficacy.
  Descriptive theories are generally
  empirically-based, relying on best practices in
  the practitioner and academic literatures.
  Description here includes causative sequences.
• Prescriptive theory indicates what ought to be
  done. It specifies program design and
  implementation, what outcomes should be expected,
  and how performance should be judged.
• Comparison of the programs descriptive and
  prescriptive theories can help to identify,
  diagnose, and explain implementation
  difficultiesthe two should be consistent.
• Logic modeling is largely limited to normative
  theorywhat is expected to happen. However, we
  need both normative and causative forms of
  theory. Both are required to explain how project
  outputs are expected to lead to a chain of
  intermediate outcomes and, in turn, eventual
  impacts, based on program observations. Causal
  logic models incorporate both kinds of
  theorydepicting both actual and expected program
  elements.

55
Causal logic models
A causal logic model clarifies the theory of how
interventions produce outcomes.
Multiple methods and techniques establish the
relative importance of causes of changes in
outcomes
56
Over time, the relative influence of a program
decreases over against exogenous factors actors
57
Determinants of Success (Mediating Moderating
Variables, or Mediators and Moderators)
Mediators are intervening variables (intervening
between the intervention effort and program
outcomes), while moderators are contextual
factors that constrain or enable those
outcomesChen, page 91.


Mediator (following from the intervention)

Outcome
Intervention




Mediator (exogenous)


Moderator
Moderator

58
Chen pp.240-241 Action Model for HIV/AIDS
education
Action Model (which along with the Change
ModelProgramTheory)
Implementation (intervention?determinants?program
outcomes)
Mediating Variables
Moderating Variables
Instrumental variables inherent in program
design. E.g., openness to learning and change
regarding sexual behaviors may well be either
presumed or actively fostered by the program,
since this cognitive factor would be considered a
key variable intervening between program
intervention(s) and behavioral change(s) among
program subjects
Often, less than positive e.g., lack of partner
support, social and economic variables such as
poverty, education, prejudice. However, may be
positive e.g., the incidence of help from
supportive networkssupport groups, family and
friends, reinforcing messages, social and
institutional and cultural supports
Impacts on individual subject(s) of the
intervention, with impacts defined as the
aggregate of comparative net outcomes
59
Logic Model

• A graphic representation that
• clearly identifies and lays out the logical
  relationships among program conditions (needs),
  resources/inputs, activities, outputs, and
  outcomes or impacts.

60
Welfare-To-Work Logic Model
Inputs Activities/Outputs Short-term
to Intermediate Outcomes Impacts
Goal Increase Self-Sufficiency in the Community
through Increased Employment Measures Decrease
in Welfare Ratio of TANF funds to wages paid to
clients Decrease Unemployment unemployment
rate total unemployment rate for
clients Increase Self-Sufficiency of community
achieving a self-sufficient wage of clients
achieving self-sufficient wage
FTE FTE FTE FTE FTE FTE

Outputs for Strategy 1 of clients trained for
standard employment of clients trained or
completing degree in high-wage employment
area Activities for Strategy 1 of training
courses held training methodologies developed
employer surveys completed career counseling
sessions provided employers offering continuing
education assistance
Strategy 1 Improve Hard Skills of Clients to
Fit Hiring Needs of the Current Economy Increase
of clients with adequate hard skills for
standard employment Increase of clients
completing continuing education coursework for
high-wage career advancement Strategy 2 Improve
the Soft Skills of Clients to Aid in Job
Placement and Retention Increase of clients
with appropriate soft skills Strategy 3 Enhance
Day Care Access Decrease clients w/out day care
accessStrategy 4 Enhance Access to
Transportation Decrease of clients w/out
transport Strategy 5 Decrease Barriers Presented
by Physical Disability Increase of employers
offering integrative workplace for people with
disabilities


61
Logic Model implicit/explicit program theory

• A good logic model clearly identifies Program
  Goals, Objectives, Inputs, Activities, Outputs,
  Desired Outcomes, and Eventual Impacts, in their
  sequential interrelation.
• Program theory specifies the relationship between
  program efforts and expected results (cf.
  theory-driven and utilization-focused
  evaluationChen). Causal logic models specify the
  connections among program elements with reference
  to a specific theory or theories of change and of
  action in some instances, they may just provide
  if-then linkages.
• A logic model helps specify what to measure in an
  evaluation, guides assessment of underlying
  assumptions, and allows for stakeholder
  consultation and corrective action, for telling a
  programs performance story.
• Partnered, collaborative programs involving a
  number of agencies or organizations have more
  complex causal chains it is a challenge to
  capture and assess these in evaluation, as
  indicated in the following two slides.

62
Multi-agency Monitoring Evaluation Logic Model
Agency 2 Inputs
Agency 3 Inputs
Adapted from Milstein Kreuter. A Summary
Outline of Logic Models What are They and What
Can They Do for Planning and Evaluation? CDC 2000
63
Complex effects chain in partnered programs
Partners 1, 2, 3, etc.
Attribution difficulties transparency
accountability challenges
Shared Common Outcomes
64
Identifying Design and Data Collection Methods in
Evaluation Planning

• Involve client and stakeholders in deciding
  necessary information to best answer each key
  evaluation question
• Evaluation designs specify the organization and
  structure and resources needed for data
  collection and analysis
• Causal designs (quasi)experimental designs
• Multiple regression, ANOVA, t-tests, or other
  statistical methods are applied in order to
  answer evaluation questions.
• Descriptive designs describe (case study),
  analyze the program, show a trend (time series),
  assess public opinions (cross-sectional),
  illustrate a process (thick description)
• Commonly used in needs assessment and process
  evaluation research

65

• Evaluator and stakeholders examine each question
  carefully to identify any important research
  design issues
• Most evaluations involve multiple research
  designs or combination of methodshybrid
  evaluation designs. This is also called
  mixed-method evaluation, involving the
  triangulation of both methods and data
• Important to discuss early to see if
• Focus groups are available, appropriateness of
  random assignment, time available for collecting
  data, access to data sources such as program
  files, training needs that may be indicated for
  staff, cost, etc.
• Intensive interviewing, semi-structured
  interviews, or other methods are feasible.
• Is the design doable?

66
Identifying Appropriate Information Sources

• Once information requirements are agreed upon,
  the sources of the information must be specified
  the following questions are key
• Who will have information or access to it?
• Who will be able to collect those data?
• Using existing data as information source
• Does necessary information already exist in a
  readily available form? Preferable to use it
• Commonly used information sources
• Program recipients, deliverers, persons who have
  knowledge of the program recipients, public
  documents/databases, file data, reports, position
  papers, grant proposals
• Policies that restrict information sources
• Are there policies about collecting data from
  clients or program files?
• Confidentiality, anonymity, privacy, IRB
  protocols

67
Identifying Appropriate Information Sources

• Using existing data as information sources
• Does necessary information already exist in a
  readily available form? Preferable to use it
• Commonly used information sources
• Program recipients, deliverers, persons who have
  knowledge of the program recipients, public
  documents/databases, file data, reports, position
  papers, grant proposals
• Policies that restrict information sources
• Do policies exist concerning collecting data from
  clients or existing files?
• Confidentiality, anonymity, privacy, IRB
  protocols

68
Identifying Appropriate Information Sources

• Client and stakeholder involvement in identifying
  sources
• The evaluator, by training and experience, often
  can identify key sources of information
• Client groups will be able to identify sources of
  information that may be missed by the evaluator
• This is one area where evaluator-client and
  evaluator-stakeholder collaboration yields
  helpful answers and makes for a sense of shared
  ownership of the evaluation process

69
Identifying Data Collection Methods, Instruments

• Data collected directly from individuals
  identified as sources of information
• Self-reports
• interviews, surveys, rating scales, focus groups,
  logs/journals
• Personal Products
• Tests, narratives, survey responses
• Data collected by independent observer
• Narrative accounts
• Observation forms (rating scales, checklists)
• Unobtrusive measures participant observation

70

• Data collected from existing information
• Public documents
• federal, state, local, databases, Census data,
  etc.
• Review of organizational documents
• client files, notes of employees/directors,
  audits, minutes, publications, reports, proposals
• Program files
• Original grant proposal
• Position papers
• Program planning documents
• Correspondence, e-mails, etc.

71

• After identifying for methods, it is important to
  review adequacy of techniques
• Will the information collected provide a
  comprehensive picture?
• Are the methods both legal and ethical?
• Is the cost of data collection worthwhile?
• Can data be collected without undue disruption?
• Can data be collected within time constraints?
• Will the information be reliable and valid for
  the purposes of the evaluation?

72
Determining Appropriate Conditions for Collecting
Information

• Examples of issues around data collection
• Will sampling be used?
• How will data actually be collected?
• When will data be collected?
• Specifying sampling procedures to be employed
• Sampling helps researcher draw inferences about
  the population in the study
• Sampling is useful when it will not diminish the
  confidence of results
• Sample size must be appropriate too small a
  sample is of limited value, and over-large,
  unfeasible

73

• Specifying how/when information will be collected
• Who will collect data?
• For interviews, focus groups Might
  characteristics of the evaluator or evaluators
  influence data collection? For instance, cultural
  distance.
• What training should be given to people
  collecting the data? Striving for consistency
  across applications.
• In what setting should data collection take
  place?
• Is confidentiality protected?
• Are special equipment, materials needed?
• When will the information be needed? Available?
• When can the information conveniently be
  collected?

74
Determining Appropriate Methods to Organize,
Analyze, and Interpret Information

• Develop a system to code, organize, store, and
  retrieve data
• For each evaluation question, specify how
  collected information will be analyzed
• Identify statistical and other analytical
  techniques
• Designate some means for conduc