PPA786: Urban Policy

1
PPA786 Urban Policy

• Class 3
• Evaluating Social Programs

2
PPA786, Class 3 Evaluating Social Programs

• Class Outline
• Positive vs. normative analysis
• The role of program evaluation
• Basic principles of program evaluation
• Program evaluation and decision making

3
PPA786, Class 3 Evaluating Social Programs

• Positive vs. Normative Analysis
• Positive analysis How do people behave?
• How are prices and quantities determined in a
  particular market?
• What is the impact of a particular government
  program and peoples behavior?
• If the necessary data are available, positive
  statements can be tested .

4
PPA786, Class 3 Evaluating Social Programs

• Positive vs. Normative Analysis
• Normative analysis What is a good outcome?
• What are the appropriate objectives for
  government intervention in a given market?
• Which objectives are the most important?
• Normative statements cannot be tested, but they
  certainly can be debated!

5
PPA786, Class 3 Evaluating Social Programs

• The Role of Program Evaluation
• Government programs change behavior.
• You cannot determine whether the outcome of a
  government program meets your own objectives
  without determining how it changes behavior.
• Program evaluation is necessary to identify the
  programs that best meet you own objectives!

6
PPA786, Class 3 Evaluating Social Programs

• What is the basic problem facing someone wanting
  to evaluate any public program?
• What you want is to know how one place differs
  with and without the program.
• What you observe is either (a) what the world is
  like after and before the program or (b) what one
  place is like with the program and another is
  without it.
• Thus, you cannot be sure that the effects you
  observe are not due to non-program differences
  over time or across places.

7
PPA786, Class 3 Evaluating Social Programs

• What is the basic problem facing someone wanting
  to evaluate any public program?
• Another way to put this is that the great
  challenge of any program evaluation is to
  identify the counterfactual, that is, to
  identify what would have happened if the program
  had not been implemented.
• The counterfactual cannot be observed directly,
  so all evaluation methods are attempts to
  estimate the counterfactual in an unbiased way.

8
PPA786, Class 3 Evaluating Social Programs

• Example (from Hollister)
• Consider a program that provides training and
  counseling to improve participants
  employability.
• Suppose a high share of previously unemployed
  participants become employed after leaving the
  program.
• Does this evidence indicate that the program is
  effective?

9
PPA786, Class 3 Evaluating Social Programs

• Example (continued)
• Answer No!
• People tend to move from unemployment to
  employment over time, and programs tend to select
  people who are unemployed.
• So the increase in employment may reflect the
  natural process of moving to employment, not
  program impact.
• This is called regression to the mean.
• Local labor market conditions might have improved
  at the time the program was implemented.
• So the increase in employment might reflect
  factors other than the impact of the program.
• This is called omitted variable bias.

10
PPA786, Class 3 Evaluating Social Programs

• The two ways to estimate program impacts are
• random assignment
• statistical control.
• Random assignment ensures that differences across
  time and place are not correlated with program
  participation.
• Statistical controls can account for observable
  differences across place or time and for certain
  kinds of unobservable factors.

11
PPA786, Class 3 Evaluating Social Programs

• Random assignment is the preferred method in most
  cases.
• It provides results that are intuitively
  compelling and scientifically sound.
• If you want to know a programs impacts, become
  an advocate for evaluation using random
  assignment!

12
PPA786, Class 3 Evaluating Social Programs

• Random assignment can be applied at many
  different scales.
• Some evaluations randomly assign treatment to
  individuals.
• Others randomly assign treatment to organizations
  (firms, schools, etc.)
• Still others randomly assign treatment to
  communities.

13
PPA786, Class 3 Evaluating Social Programs

• Random assignment has been used to study
• Welfare-to-work programs
• Unemployment insurance
• Job training
• Income maintenance
• Housing assistance
• Electricity pricing
• Education
• Early childhood development
• Criminal justice policy
• Child health and nutrition

14
PPA786, Class 3 Evaluating Social Programs

• Random assignment is not always feasible.
• A huge literature indicates that the best
  statistical studies
• must have extensive data to ensure that
  differences arent due to unobservable factors.
• must have comparable experimental and control
  groups based on observable factors.

15
PPA786, Class 3 Evaluating Social Programs

• For the case of community economic development
  programs, Hollister discusses several evaluation
  that do not use random assignment.
• This discussion gives you a sense of what to look
  for in statistical studies.
• You may want to return to it when we discuss
  community economic development!

16
PPA786, Class 3 Evaluating Social Programs

• Formal evaluation of programs or management
  reforms are often not available.
• Thus, it is appropriate for you (when you become
  public officials!) to use your own judgment
• to select programs and reforms that appear to
  have worked in other places
• to design new programs and reforms

17
PPA786, Class 3 Evaluating Social Programs

• Evaluations of intermediate results can also be
  helpful.
• Here is the figure in Hollister
• Your Planned Work Your Intended Results

Impact
Resources/ Inputs
Activities
Outputs
Outcomes
1
2
4
3
5
18
PPA786, Class 3 Evaluating Social Programs

• But evaluation should always be in the back of
  your mind.
• Search for evaluations of the programs or reforms
  you are interested in.
• Make an honest judgment about the quality of
  existing evaluations.
• Informally apply basic evaluation principles to
  programs and reforms you are considering.
• Implement formal evaluations whenever possible!

19
PPA786, Class 3 Evaluating Social Programs

• Informal Evaluations
• Informal evaluations can be very helpful.
• Learn about the market in which the program will
  operatethat is about the economic and social
  factors that influence the behavior of market
  participants.
• Think about how various government programs
  change the incentives of people in this market.
• Use your understanding from other cases to make
  an educated guess about the impact of these
  changes in incentives on behaviorand hence on
  your objectives.

20
PPA786, Class 3 Evaluating Social Programs

• The Punchline (from Hollister)
• People who want to know whether programs have an
  impact on the outcomes they care about
• should not be asking programs to prove this when
  there is not an available method that meets a
  high proof standard.
• At the same, they should not be persuaded by
  flawed estimates that are offered as proof.