The PROGRESAOportunidades program of Mexico and its Impact Evaluation

1
The PROGRESA/Oportunidades program of Mexico and
its Impact Evaluation

• Emmanuel Skoufias
• The World Bank
• PRMPR
• May 2007

2
What is a CCT?

• A Conditional Cash Transfer (CCT) program is a
  targeted transfer program whereby cash is
  directly provided to beneficiary families
  (usually mothers) on the condition that children
  attend school regularly and family members visit
  health centers regularly.

3
Dual Objectives of CCT programs

• Long-run poverty alleviation through investment
  in human capital (i.e., education, health and
  nutrition)
• Early Interventions have much higher returns over
  life-cycle
• Short-run effect on poverty through cash
  transfers

4
Why are CCT programs attractive?-1

• CCT can provide the foundation of a comprehensive
  Poverty Alleviation and Social Protection Policy
• Induce investments by poor on human capital
• Can mitigate short-run macroeconomic shocks
• Can facilitate the phase-out of price subsidies
  and/or other less effective programs
• Can serve as a basic social safety net system
  available to ALL households (complementary to the
  safety net system accessible through employment
  in formal sector)

5
Why are CCT programs attractive?-2

• CCT can achieve a significant redistribution of
  income in favor of the poor under tight fiscal
  conditions
• PROGRESA was initiated in the context of a
  short-run economic crisis (1994-95) and was
  designed as medium-term response to the crisis
• PROGRESA gradually replaced generalized food
  subsidies with direct monetary transfers

6
Why are CCT programs attractive? -3

• CCT exploit the complementarities among health,
  education, and nutrition.
• Coordination Promote coordination of poverty
  alleviation efforts among Govt ministries
  (education, health, nutrition)
• Synergy simultaneous provision of health,
  education and nutrition benefits to all the
  beneficiaries.

7
Why are CCT programs attractive?-4

• Co-responsibility The beneficiaries need to take
  specific actions or else they do not receive
  benefit
• CCT have potential of leading to lasting
  improvements in the well being of the poor
• Evidence from rigorous impact evaluation of
  Oportunidades in Mexico Familias en Accion in
  Colombias and other countries

8
How CCT try to achieve their objectives?-1

• Targeting (geographic/household-level)
• Oportunidades combines geographic/village-level
  with household level targeting within villages
• Simultaneous intervention in 3 key sectors
  (synergy)

9
How CCT try to achieve their objectives?-2

• Conditioning cash transfers to regular school
  attendance and visits to health centers
• Cash transfers given to mothers
• Parallel support on Supply Side (schools health
  centers)

10
Controversial aspects of Oportunidades-1

• Why grant direct monetary transfers instead of
  food in-kind, vouchers, or improving supply side
  of services.
• Distribution of large volumes of food free of
  charge can inhibit the development of private
  commercial channels and create unfair competition
  with marginal producers in the area
• Why target on the extreme/structurally poor and
  not include all?
• Setting new selection criteria Why not than
  start from beneficiary lists of existing programs
  or obtaining the roster of beneficiaries from
  community proposals

11
Controversial aspects of Oportunidades-2

• Creating a single national roster of
  beneficiaries
• Giving transfers directly to individuals rather
  than to communities
• Having unique, non-discretionary rules for the
  whole country rather than allowing flexibility
  for local initiatives and conditions in each
  state

12
Controversial aspects of Oportunidades-3

• Granting benefits to women, given potential
  family conflicts
• Having possible impact on fertility (since
  benefits are linked to family demographics)
• Size of cash transfer
• the definition of family co-responsibilities and
  their certification (might generate additional
  workload for teachers and medical personnel)

13
How the controversial aspects of Oportunidades
were managed

• Piloting
• Expansion of the program in phases
• Independent and rigorous evaluation (targeting,
  impact of the program on health, education,
  nutrition, social relations, womens status etc.)
• Monitoring
• Operational evaluation of the program
• Cost analysis

14
Why Evaluation?

• Economic Reasons
• Improve design and effectiveness of the program
• Comparing program impacts allows G to reallocate
  funds from less to more effective programs and
  thus to an increase in Social Welfare
• Social Reasons (increases transparency
  accountability)
• Political Reasons
• Credibility/break with bad practices of past

15
Key elements of a successful and rigorous
evaluation

• Evaluation built-in as a component of the program
  early in the program design stages
• Evaluation has clear objectives
• Impact?
• Program Design? e.g. PROGRESA package
• Evaluation has political support
• Evaluation Design that yields credible estimates
  of Impact

16
Key elements of a successful and rigorous
evaluation

• Log frame helps identify subject to the budget
  constraint available (agreed) upon
• objectives
• setting indicators of impact
• data needs (quantitative and qualitative)
• the threshold value of the CHANGE in impact
  indicator if a program HAS an effect
• Survey sizes needed
• Budget allocated to the evaluation

17
Key elements of a successful and rigorous
evaluation

• Pre-existing household surveys administrative
  data
• Can be used for an ex-ante evaluation of the
  expected program impact and determining size of
  benefits (design)
• Can be used to evaluate program impact (using
  before after estimator)

18
CCT programs (like Oportunidades) Expanding

• Brazil Bolsa FamiliaBolsa Escola, Bolsa
  Alimentacao Programa de Erradicaçao do Trabalho
  Infantil (PETI)
• Colombia Familias en Acción
• Honduras Programa de Asignación Familiar (PRAF)
  
• Jamaica Program of Advancement through Health
  and Education (PATH)
• Nicaragua Red de Protección Social (RPS)
• Turkey
• Ecuador Bono Solidario
• Argentina
• Bangladesh Food for Education

19
OPORTUNIDADES (previously called PROGRESA)

• Large program covering rural and marginal urban
  areas
• In 2004 5 million families or 25 million
  individuals
• In 2004 budget of US 2.5 billion or 0.3 of GDP

20
Program Description Benefits

• Education component
• A system of educational grants (details below)
• Monetary support or the acquisition of school
  materials/supplies
• (The above benefits are tied to enrollment and
  regular (85) school attendance)
• Improved schools and quality of educations
  (teacher salaries)

21
Program Description Benefits

• Health and Nutrition Component
• basic package of primary heath-care services
• Food support (cash)
• nutritional supplements 6 packs/child/mo 20 of
  caloric requirements and 100 of necessary
  micronutrients)
• (The above benefits are tied to regular visits to
  health-service centers).
• Information and training
• Improved supply and quality of health services
  (medicine availability etc.)

22
Program Description Benefits

• Average benefit received by beneficiary
  households or 20 of the value of consumption
  expenditure before program
• About ½ of transfer is the cash transfer for food
  and the rest from the school-related cash
  transfer

23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
Targeting

• Step 1 geographical targeting
• Identify localities of highest marginality used
  census data
• Step 2 Household-level targeting
• Within the localities identified in step 1,
  conduct a household census and use that census to
  collect socio-demographic data and information on
  housing characteristics that is consistent and
  standard nationwide.
• Within Village household-level targeting (village
  household census)
• Used hh income, assets, and demographic
  composition to estimate the probability of being
  poor (Inc per capltStandard Food basket).
• Discriminant analysis applied separately by
  region
• Discriminant score (DS) of each household
  compared to a threshold value (high
  DSNoneligible, low DSEligible)

28
(No Transcript)
29
PROGRESA/OPORTUNIDADES Evaluation Design

• EXPERIMENTAL DESIGN Program randomized at the
  locality level (Pipeline experimental design)
• IFPRI not present at time of selection of T and C
  localities
• Report examined differences between T and C for
  more than 650 variables at the locality level
  (comparison of locality means) and at the
  household level (comparison of household means)
• Sample of 506 localities
• 186 control (no program)
• 320 treatment (receive program)
• 24, 077 Households (hh)
• 78 beneficiaries
• Differences between eligible hh and actual
  beneficiaries receiving benefits
• Densification (initially 52 of hh classified as
  eligible)

30
Table A Decomposition of the Sample of All
Households in Treatment and Control Villages
31
(No Transcript)
32
Using regressions to get 2DIF estimates
Limit sample to eligible households in treatment
and control and run regression

• Y(i,t) denotes the value of the outcome indicator
  in household (or individual) i in period t,
• alpha, beta and theta are fixed parameters to be
  estimated,
• T(i) is an binary variable taking the value of 1
  if the household belongs in a treatment community
  and 0 otherwise (i.e., for control communities),
• R2 is a binary variable equal to 1 for the second
  round of the panel (or the round after the
  initiation of the program) and equal to 0 for the
  first round (the round before the initiation of
  the program),
• X is a vector of household (and possibly village)
  characteristics
• last term is an error term summarizing the
  influence random disturbances.

33
(No Transcript)
34
(No Transcript)
35
Evaluation Tools

• Formal surveys
• (Semi)-structured observations and interviews
• Focus groups with stakeholders (beneficiaries,
  local leaders, local PROGRESA officials, doctors,
  nurses, school teachers, promotoras)

36
PROGRESA Evaluation Surveys/Data
37
PROGRESA Evaluation Surveys

• Additional Info Sources
• School clinic survey
• School and clinic administrative data
• Nutrition survey conducted independently by Min.
  of Health and INSP
• Student achievement test scores by Min of
  Education
• Record of payments distributed to beneficiary
  households

38
Topics of PROGRESAs Evaluation

• Targeting accuracy and impact on poverty
• School enrollment, attendance, child labor,
  achievement scores
• Health and utilization of health facilities
• Child Nutrition
• Household Consumption Nutrition

39
Topics of PROGRESAs Evaluation contd

• Operation of the program and perceptions of
  stakeholders
• Cost-Analysis and Cost Effectiveness
• Status of women, community relations
• Adult labor supply, leisure
• Impact on short-run poverty
• intrahousehold transfers

40
Evaluation Results-Targeting

• Geographic targeting of the program in rural
  areas is good
• Method of selecting poor households within
  localities is generally accurate (undercoverage
  of 7 )
• PROGRESAs targeting decreases the poverty gap
  P(1) by 30 and the severity of poverty P(2) by
  45

41
NOTE Program Operation Impact

• Linkage between program operation
  implementation estimated program impact
• Delays in Benefit Distribution
• Mistakes in lists of Beneficiaries
• Intention to Treat Effect vs Treatment Effect on
  those who actually received treatment

42
(No Transcript)
43
(No Transcript)
44
Education

• Are more children attending school because of
  PROGRESA?
• How much can schooling be expected to increase?
• Are there more cost effective ways of bringing
  children to school?
• Does PROGRESA have more impact in certain grades?
• Any effects on drop-out rates, grade progression,
  repetition, reentry?

45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
Evaluation Results Education

• PROGRESA has a positive effect the school
  attendance of both boys and girls in primary and
  secondary school
• Boys in secondary increase by 8
• Girls in secondary increase by 14
• Negative impact on childrens labor market
  participation (especially boys)
• No observed increase in the attendance rate
  (frequency) of children in school.
• PROGRESA increases overall educational attainment
  by 10 (and 8 higher earnings)

50
Evaluation Results Education

• Program effective in keeping children in school
  especially during the critical transition from
  primary to secondary
• Less effective in bringing back to and keeping in
  school children who were out.
• Earlier entry ages, less grade repetition, better
  grade progression
• PROGRESA more cost-effective than increasing
  access to junior secondary education

51
Health

• Does it increase visits to public health clinics?
• Does PROGRESA have an effect on child health?
• On the health of adults?

52
(No Transcript)
53
(No Transcript)
54
Evaluation results Health

• Significant increase in visit rates in PROGRESA
  communities
• Increasing in nutrition monitoring visits,
  immunization rates and prenatal care in 1st
  trimester (8 increase)
• No substitution between private and public
  facilities

55
Evaluation results Health

• 12 lower incidence of illness in children
  between ages 0-5.
• Significantly positive effects on adult health

56
Nutrition

• Does PROGRESA impact of child growth?
• Household consumption and food diet?

57
Evaluation results Nutrition

• Significant effect at increasing child growth
  (1cm higher growth) and reducing the probability
  of stunting among children 12-36 mo.
• Household total consumption increases
• PROGRESA households eat better (higher
  expenditures on fruits, vegetables, meats
  animal products)

58
Impact of PROGRESA on Poverty (1)

• Results so far program has no adverse effects on
  labor income
• Effects on total hh income and thus poverty
  depends on the direct and indirect costs
  associated with participation in PROGRESA.
• Participation in PROGRESA?
• (a) income losses form childrens work
• (b) give up benefits from other programs (DIF,
  Ninos de Solidaridad, Abasto Social de Leche) in
  additional to the elimination of the Tortilla
  subsidy

59
Impact of PROGRESA on Poverty (2)

• Figures 1 2 the effects of PROGRESA on hh
  income and poverty may not be adequately
  summarized by the size and incidence of the cash
  transfers
• Econometric analysis based on individual and hh
  income in each round from a variety of sources
  labor income, income from self-employment, other
  income (pensions, rent, and community profits)
  and government transfers (Ninos de Solidaridad,
  ININ, PROBECAT, PET, PROCAMPO) PROGRESA Cash
  Transfers (from program admin records)

60
Impact on Poverty
where

61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
Table 4 results (1)

• PROGRESA had a significant impact in reducing
  poverty between November 1997 and November 1999.
• E.g. using the 50th percentile of the value of
  consumption per capita as a poverty line,
  suggests that the headcount poverty rate declined
  by around 4.88 between November 1997 and
  November 1998 and by 18.11 in the November 1999
  in treatment areas (using as base the 55.44
  headcount poverty rate in treatment localities in
  November 1997).
• Over the same period, and using as base the
  corresponding value of the poverty gap and
  squared poverty gap indices in treatment areas in
  November 1997, the poverty gap measure declined
  by 37.40, and the severity of poverty measure
  (squared poverty gap) declined by 47.42.
• The higher impacts of the program in reducing
  poverty over time are consistent with the
  findings of Gertler et al. (2006), who
  demonstrated that rural households increased
  their investments in microenterprises and
  agricultural activities which, improved the
  ability of households to generate income.

65
Table 4 results (2)

• Estimates are remarkably in line with the
  estimates obtained using ex-ante simulations.
• E.g. simulations based on the predicted
  consumption of each household in the evaluation
  sample in November 1997 and adding the maximum
  amount of PROGRESA cash transfers an eligible
  household could receive assuming full compliance
  with the programs requirements (see Skoufias et
  al., 2001).

66
Table 4 results (3)

• The poverty reduction effects are stronger for
  the poverty gap and severity of poverty measures,
  which put greater weight on the poorest of the
  poor, and our evidence suggests that these
  estimated poverty effects are robust to the
  choice of different poverty lines.
• Figure 3 and Appendix A

67
Cost Analysis

• Are the administrative costs of PROGRESA high?
• What are the private costs associated with
  participation in the program?
• What might be the indirect effects of the program
  on the national economy? (e.g. financing of the
  program)?

68
Evaluation Results Cost Analysis

• For every 100 pesos allocated to the program, 8.2
  pesos are administration/program costs.
• Very low compared to LICONSA (40 pesos per 100
  pesos) and TORTIVALES (14 pesos per 100)
• Targeting and conditioning of the program makeup
  56 of program costs (4.6 pesos out of 8.2 pesos)
• Private costs (3.8 pesos out of 8.2 pesos)

69
Evaluation Results Cost Analysis

• Eliminating distortionary food subsidies and
  using funds to finance a program like PROGRESA
  leads to substantial welfare gains.

70
The Contributions of Program Evaluation-1

• Program continued and improved
• Fox administration (begun in 2001) kept and
  expanded the program
• Early operations reports in PROGRESA identified
  implementation issues to be analyzed further
  (food supplements, intra-household conflict,
  targeting views)
• Decision to maintain household targeting in
  PROGRESA expansion, but to add self-selection
  to administrative selection in urban areas

71
The Contributions of Program Evaluation-2

• Program Design improved Program expanded to
  urban areas
• Benefits extended to Preparatoria Secondary level
• Jovenes con Oportunidades- aims to create income
  generating opportunities for poor households
  through preferential access to microcredit,
  housing improvements, adult education and access
  to social/health insurance.

72
Critical Issues to be Resolved on CCT

• Do CCT programs break the intergenerational
  transmission of poverty? Need long time-series
• What is the minimal CCT that may be paid?
• Oportunidades size of transfer based on
  opportunity cost of children (child wage/value of
  childrens contribution to family)

73
Critical Issues to be Resolved

• Impact on Childrens Achievement Learning?
• Do CCT increase achievement or induce teachers to
  lower grade-passing standards?
• Teacher health worker incentives
• Quality of Services

74
Critical Issues to be Resolved

• Do CCT generate Program/Welfare Dependency?
• So far no negative incentive effects on
  adult work
• Transfers Income generation
• Exit Rule?
• Lack built-in flexibility to expand coverage to
  households falling below poverty during crisis

75
Final Issue

• Long-Run Sustainability of Program Budget
  Political Economy of Program Support

76
Thank you
77
(No Transcript)
78
(No Transcript)
79
Concluding Remarks

• Our analysis reveals that in the PROGRESA sample,
  the RDD performs very well (i.e. yields program
  impacts close to the ideal experimental impact
  estimates).
• Critical to be aware of some of the limitations
  of the RDD approach
• Estimates treatment effects at the point of
  discontinuity (eligibility threshold). Impact on
  this group of households may be of less interest
  than impact of the program on the poorer
  households

80
Concluding Remarks

• The integrity/quality of the control/comparison
  group is of vital importance.
• Spillover effects do not necessarily lead to a
  violation of the RDD approach. As long as the
  local continuity assumption continues to hold
  even though there are spillover effects the
  presence of spillover effects would only affect
  the interpretation of the RD effect It is the
  effect of being eligible for program
  participation in treatment villages net of
  spillover effects.
• Social programs at the national scale may be very
  difficult to evaluate ex-post because of the
  difficulty in finding an adequate comparison group