The Impact Evaluation of the PROGRESA/Oportunidades program of Mexico

1
The Impact Evaluation of the PROGRESA/Oportunidade
s program of Mexico

• Emmanuel Skoufias
• The World Bank
• PRMPR
• April 2007

2
Why are CCT programs attractive?-1

• CCT can provide the foundation of a comprehensive
  Poverty Alleviation and Social Protection Policy
• Induce investments by poor on human capital
• Can mitigate short-run macroeconomic shocks
• Can facilitate the phase-out of price subsidies
  and/or other less effective programs
• Can serve as a basic social safety net system
  available to ALL households (complementary to the
  safety net system accessible through employment
  in formal sector)

3
Why are CCT programs attractive?-2

• CCT can achieve a significant redistribution of
  income in favor of the poor under tight fiscal
  conditions
• Oportunidades was initiated in the context of a
  short-run economic crisis (1994-95) and was
  designed as medium-term response to the crisis
• Oportunidades gradually replaced generalized food
  subsidies with direct monetary transfers

4
Why are CCT programs attractive? -3

• CCT exploit the complementarities among health,
  education, and nutrition.
• Coordination Promote coordination of poverty
  alleviation efforts among Govt ministries
  (education, health, nutrition)
• Synergy simultaneous provision of health,
  education and nutrition benefits to all the
  beneficiaries.

5
Why are CCT programs attractive?-4

• Co-responsibility The beneficiaries need to take
  specific actions or else they do not receive
  benefit
• CCT have potential of leading to lasting
  improvements in the well being of the poor
• Evidence from rigorous impact evaluation of
  Oportunidades in Mexico Familias en Accion in
  Colombias and other countries

6
Dual Objectives of CCT

• Long-run poverty alleviation through investment
  in human capital (i.e., education, health and
  nutrition)
• Early Interventions have much higher returns over
  life-cycle
• Short-run effect on poverty through cash
  transfers

7
How CCT try to achieve their objectives?-1

• Targeting (geographic/household-level)
• Oportunidades combines geographic/village-level
  with household level targeting within villages
• Simultaneous intervention in 3 key sectors
  (synergy)

8
How CCT try to achieve their objectives?-2

• Conditioning cash transfers to regular school
  attendance and visits to health centers
• Cash transfers given to mothers
• Parallel support on Supply Side (schools health
  centers)

9
Controversial aspects of Oportunidades-1

• Why grant direct monetary transfers instead of
  food in-kind, vouchers, or improving supply side
  of services.
• Distribution of large volumes of food free of
  charge can inhibit the development of private
  commercial channels and create unfair competition
  with marginal producers in the area
• Why target on the extreme/structurally poor and
  not include all?
• Setting new selection criteria Why not than
  start from beneficiary lists of existing programs
  or obtaining the roster of beneficiaries from
  community proposals

10
Controversial aspects of Oportunidades-2

• Creating a single national roster of
  beneficiaries
• Giving transfers directly to individuals rather
  than to communities
• Having unique, non-discretionary rules for the
  whole country rather than allowing flexibility
  for local initiatives and conditions in each
  state

11
Controversial aspects of Oportunidades-3

• Granting benefits to women, given potential
  family conflicts
• Having possible impact on fertility (since
  benefits are linked to family demographics)
• Size of cash transfer
• the definition of family co-responsibilities and
  their certification (might generate additional
  workload for teachers and medical personnel)

12
How the controversial aspects of Oportunidades
were managed

• Piloting
• Expansion of the program in phases
• Independent and rigorous evaluation (targeting,
  impact of the program on health, education,
  nutrition, social relations, womens status etc.)
• Monitoring
• Operational evaluation of the program
• Cost analysis

13
Why Evaluation?

• Economic Reasons
• Improve design and effectiveness of the program
• Comparing program impacts allows G to reallocate
  funds from less to more effective programs and
  thus to an increase in Social Welfare
• Social Reasons (increases transparency
  accountability)
• Political Reasons
• Credibility/break with bad practices of past

14
CCT programs (like Oportunidades) Expanding

• Brazil Bolsa FamiliaBolsa Escola, Bolsa
  Alimentacao Programa de Erradicaçao do Trabalho
  Infantil (PETI)
• Colombia Familias en Acción
• Honduras Programa de Asignación Familiar (PRAF)
  
• Jamaica Program of Advancement through Health
  and Education (PATH)
• Nicaragua Red de Protección Social (RPS)
• Turkey
• Ecuador Bono Solidario
• Argentina
• Bangladesh Food for Education

15
OPORTUNIDADES (previously called PROGRESA)

• Large program covering rural and marginal urban
  areas
• In 2004 5 million families or 25 million
  individuals
• In 2004 budget of US 2.5 billion or 0.3 of GDP

16
Program Description Benefits

• Education component
• A system of educational grants (details below)
• Monetary support or the acquisition of school
  materials/supplies
• (The above benefits are tied to enrollment and
  regular (85) school attendance)
• Improved schools and quality of educations
  (teacher salaries)

17
Program Description Benefits

• Health and Nutrition Component
• basic package of primary heath-care services
• Food support (cash)
• nutritional supplements 6 packs/child/mo 20 of
  caloric requirements and 100 of necessary
  micronutrients)
• (The above benefits are tied to regular visits to
  health-service centers).
• Information and training
• Improved supply and quality of health services
  (medicine availability etc.)

18
Program Description Benefits

• Average benefit received by beneficiary
  households or 20 of the value of consumption
  expenditure before program
• About half transfer is the cash transfer for food
  and the rest from the school-related cash
  transfer

19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
PROGRESA/OPORTUNIDADES Evaluation Design

• EXPERIMENTAL DESIGN Program randomized at the
  locality level (Pipeline experimental design)
• IFPRI not present at time of selection of T and C
  localities
• Report examined differences between T and C for
  more than 650 variables at the locality level
  (comparison of locality means) and at the
  household level (comparison of household means)
• Sample of 506 localities
• 186 control (no program)
• 320 treatment (receive program)
• 24, 077 Households (hh)
• 78 beneficiaries
• Differences between eligible hh and actual
  beneficiaries receiving benefits
• Densification (initially 52 of hh classified as
  eligible)

24
Targeting

• Step 1 geographical targeting
• Identify localities of highest marginality
• Step 2 Household-level targeting
• Within the localities identified in step 1,
  conduct a household census and use that census to
  collect socio-demographic data and information on
  housing characteristics that is consistent and
  standard nationwide.
• Use discriminant analysis on census data to
  classify household as poor or nonpoor

25
Table A Decomposition of the Sample of All
Households in Treatment and Control Villages
26
(No Transcript)
27
Using regressions to get 2DIF estimates
Limit sample to eligible households in treatment
and control and run regression

• Y(i,t) denotes the value of the outcome indicator
  in household (or individual) i in period t,
• alpha, beta and theta are fixed parameters to be
  estimated,
• T(i) is an binary variable taking the value of 1
  if the household belongs in a treatment community
  and 0 otherwise (i.e., for control communities),
• R2 is a binary variable equal to 1 for the second
  round of the panel (or the round after the
  initiation of the program) and equal to 0 for the
  first round (the round before the initiation of
  the program),
• X is a vector of household (and possibly village)
  characteristics
• last term is an error term summarizing the
  influence random disturbances.

28
(No Transcript)
29
(No Transcript)
30
Evaluation Tools

• Formal surveys
• (Semi)-structured observations and interviews
• Focus groups with stakeholders (beneficiaries,
  local leaders, local PROGRESA officials, doctors,
  nurses, school teachers, promotoras)

31
PROGRESA Evaluation Surveys/Data
BEFORE initiation of program Oct/Nov 97 Household census to select beneficiaries March 98 consumption, school attendance, health AFTER initiation of program Nov 98 June 99 Nov/Dec 99 Included survey of beneficiary households regarding operations
32
PROGRESA Evaluation Surveys

• Additional Info Sources
• School clinic survey
• School and clinic administrative data
• Nutrition survey conducted independently by Min.
  of Health and INSP
• Student achievement test scores by Min of
  Education
• Record of payments distributed to beneficiary
  households

33
Topics of PROGRESAs Evaluation

• Targeting accuracy and impact on poverty
• School enrollment, attendance, child labor,
  achievement scores
• Health and utilization of health facilities
• Child Nutrition
• Household Consumption Nutrition

34
Topics of PROGRESAs Evaluation contd

• Operation of the program and perceptions of
  stakeholders
• Cost-Analysis and Cost Effectiveness
• Status of women, community relations
• Adult labor supply, leisure
• Impact on short-run poverty
• intrahousehold transfers

35
Conditional Cash Transfers, Adult Work
Incentives, and Poverty

• Emmanuel Skoufias
• The World Bank
• eskoufias_at_worldbank.org
• Vincenzo di Maro
• University College London
• v.dimaro_at_ucl.ac.uk
• November 3, 2006
• LACEA meetings Mexico City

36
Introduction-1

• Means-tested CCT programs increasingly popular
• Their success at reducing current poverty depends
  on whether, and the extent to which, cash
  transfers reduce adult work incentives.
• In a CCT a cash transfer can be considered as
  having offsetting income and substitution
  effects.
• Income effect The cash transfer increases
  household income which in turn increases both
  Consumption and Leisure (decreases adult labor
  supply).
• Substitution effects associated with meeting the
  conditionalities of the program. Direct and
  indirect time costs that depend on the
  substitution possibilities in the time of family
  members used in household production.

37
Introduction-2

• Thus, the response of adult labor supply to CCTs
  can only be determined empirically.
• Most welfare programs in the US, have explicit
  disincentives to work.
• In most CCT programs the level of benefits
  received is not affected by the work decisions of
  hh members or the income level of the hh (once
  the hh is determined as eligible for the
  program).
• However, there are other sources of negative
  incentives on adult labor supply for both
  eligible and non-eligible households.
• Eligible hh may work/earn less so as not to get
  an income level above the poverty threshold used
  by the program.
• Non-eligible hh may work/earn less so as to
  qualify for the program

38
Introduction-3

• Empirical evidence on how labor supply responds
  to transfers is scarce. Sahn Alderman (1996)
  the labor supply effect of a rice subsidy in Sri
  Lanka is significantly large.

39
Objectives-1

• Use data from the PROGRESA 1997-99 evaluation
  sample to investigate
• whether eligibility for program affect adult
  labor force participation (in the treatment
  villages)
• whether the presence of the program affects the
  labor force participation of adults from
  non-eligible households (in the treatment
  villages)
• The effect of eligibility on adult leisure time
• The impact of the program on poverty measures
  based on household income.

40
Participation in Labor Force
where

Leisure (from time allocation survey in June
1999
41
Table 1 Impact (2DIF estimates) on the
probability of working among eligible (E1)
adults
42
Table 2 Impact (2DIF estimates) on the
probability of working among NON-Eligible (E0)
adults
43
Tables 1 2 results 2DIF (1)

• PROGRESA has no significant impact on the labor
  force participation of adult males and/or females
• There are no significant/lasting impacts on the
  allocation of time between salaried and
  self-employment activities,
• though the program appears to be associated with
  a negative, albeit non-significant, effect on
  self-employment among males

44
Tables 1 2 results 2DIF (2)

• The presence of PROGRESA has no significant
  impact on the labor force participation of adult
  males and/or females from noneligible households
  (in the treatment villages)

45
Table 3 Impact (CSDIF estimates) on Leisure of
Eligible (E1) adults
46
Table 3 CSDIF (1)

• PROGRESA has no significant impact on the leisure
  time of adult males and/or females

47
Impact of PROGRESA on Poverty (1)

• Results so far program has no adverse effects on
  labor income
• Effects on total hh income and thus poverty
  depends on the direct and indirect costs
  associated with participation in PROGRESA.
• Participation in PROGRESA?
• (a) income losses form childrens work
• (b) give up benefits from other programs (DIF,
  Ninos de Solidaridad, Abasto Social de Leche) in
  additional to the elimination of the Tortilla
  subsidy

48
Impact of PROGRESA on Poverty (2)

• Figures 1 2 the effects of PROGRESA on hh
  income and poverty may not be adequately
  summarized by the size and incidence of the cash
  transfers
• Econometric analysis based on individual and hh
  income in each round from a variety of sources
  labor income, income from self-employment, other
  income (pensions, rent, and community profits)
  and government transfers (Ninos de Solidaridad,
  ININ, PROBECAT, PET, PROCAMPO) PROGRESA Cash
  Transfers (from program admin records)

49
Figure 1 Mean household income from
children Among beneficiary households with
children aged 8-17 Panel a Mean household TOTAL
income from Children (excluding PROGRESA cash
transfer)
50
Figure 1 Mean household income from
children Among beneficiary households with
children aged 8-17 Panel b Mean household LABOR
income from Children
51
Figure 1 Mean household income from
children Among beneficiary households with
children aged 8-17 Panel c Mean household Other
Income from Children
52
Figure 2
53
Impact on Poverty
where

54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
Table 4 results (1)

• PROGRESA had a significant impact in reducing
  poverty between November 1997 and November 1999.
• E.g. using the 50th percentile of the value of
  consumption per capita as a poverty line,
  suggests that the headcount poverty rate declined
  by around 4.88 between November 1997 and
  November 1998 and by 18.11 in the November 1999
  in treatment areas (using as base the 55.44
  headcount poverty rate in treatment localities in
  November 1997).
• Over the same period, and using as base the
  corresponding value of the poverty gap and
  squared poverty gap indices in treatment areas in
  November 1997, the poverty gap measure declined
  by 37.40, and the severity of poverty measure
  (squared poverty gap) declined by 47.42.
• The higher impacts of the program in reducing
  poverty over time are consistent with the
  findings of Gertler et al. (2006), who
  demonstrated that rural households increased
  their investments in microenterprises and
  agricultural activities which, improved the
  ability of households to generate income.

58
Table 4 results (2)

• Estimates are remarkably in line with the
  estimates obtained using ex-ante simulations.
• E.g. simulations based on the predicted
  consumption of each household in the evaluation
  sample in November 1997 and adding the maximum
  amount of PROGRESA cash transfers an eligible
  household could receive assuming full compliance
  with the programs requirements (see Skoufias et
  al., 2001).

59
Table 4 results (3)

• The poverty reduction effects are stronger for
  the poverty gap and severity of poverty measures,
  which put greater weight on the poorest of the
  poor, and our evidence suggests that these
  estimated poverty effects are robust to the
  choice of different poverty lines.
• Figure 3 and Appendix A

60
Concluding Remarks

• PROGRESA does not have any significant effect on
  adult labor supply choices.
• There is not much evidence to support the
  hypothesis that PROGRESA beneficiaries use their
  transfers to buy more leisure.
• Results are reinforced further by the result that
  PROGRESA leads to a substantial reduction in
  current poverty.

61
The Contributions of Program Evaluation-1

• Program continued and improved
• Fox administration (begun in 2001) kept and
  expanded the program
• Early operations reports in PROGRESA identified
  implementation issues to be analyzed further
  (food supplements, intra-household conflict,
  targeting views)
• Decision to maintain household targeting in
  PROGRESA expansion, but to add self-selection
  to administrative selection in urban areas

62
The Contributions of Program Evaluation-2

• Program Design improved Program expanded to
  urban areas
• Benefits extended to Preparatoria Secondary level
• Jovenes con Oportunidades- aims to create income
  generating opportunities for poor households
  through preferential access to microcredit,
  housing improvements, adult education and access
  to social/health insurance.

63
Critical Issues to be Resolved on CCT

• Do CCT programs break the intergenerational
  transmission of poverty? Need long time-series
• What is the minimal CCT that may be paid?
• Oportunidades size of transfer based on
  opportunity cost of children (child wage/value of
  childrens contribution to family)

64
Critical Issues to be Resolved

• Impact on Childrens Achievement Learning?
• Do CCT increase achievement or induce teachers to
  lower grade-passing standards?
• Teacher health worker incentives
• Quality of Services

65
Critical Issues to be Resolved

• Do CCT generate Program/Welfare Dependency?
• So far no negative incentive effects on
  adult work
• Transfers Income generation
• Exit Rule?
• Lack built-in flexibility to expand coverage to
  households falling below poverty during crisis

66
Final Issue

• Long-Run Sustainability of Program Budget
  Political Economy of Program Support

67
Thank you
68
Evaluation Results-Targeting

• Geographic targeting of the program in rural
  areas is good
• Method of selecting poor households within
  localities is generally accurate (undercoverage
  of 7 )
• PROGRESAs targeting decreases the poverty gap
  P(1) by 30 and the severity of poverty P(2) by
  45

69
Education

• Are more children attending school because of
  PROGRESA?
• How much can schooling be expected to increase?
• Are there more cost effective ways of bringing
  children to school?
• Does PROGRESA have more impact in certain grades?
• Any effects on drop-out rates, grade progression,
  repetition, reentry?

70
(No Transcript)
71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
Evaluation Results Education

• PROGRESA has a positive effect the school
  attendance of both boys and girls in primary and
  secondary school
• Boys in secondary increase by 8
• Girls in secondary increase by 14
• Negative impact on childrens labor market
  participation (especially boys)
• No observed increase in the attendance rate
  (frequency) of children in school.
• PROGRESA increases overall educational attainment
  by 10 (and 8 higher earnings)

75
Evaluation Results Education

• Program effective in keeping children in school
  especially during the critical transition from
  primary to secondary
• Less effective in bringing back to and keeping in
  school children who were out.
• Earlier entry ages, less grade repetition, better
  grade progression
• PROGRESA more cost-effective than increasing
  access to junior secondary education

76
Health

• Does it increase visits to public health clinics?
• Does PROGRESA have an effect on child health?
• On the health of adults?

77
(No Transcript)
78
(No Transcript)
79
Evaluation results Health

• Significant increase in visit rates in PROGRESA
  communities
• Increasing in nutrition monitoring visits,
  immunization rates and prenatal care in 1st
  trimester (8 increase)
• No substitution between private and public
  facilities

80
Evaluation results Health

• 12 lower incidence of illness in children
  between ages 0-5.
• Significantly positive effects on adult health

81
Nutrition

• Does PROGRESA impact of child growth?
• Household consumption and food diet?

82
Evaluation results Nutrition

• Significant effect at increasing child growth
  (1cm higher growth) and reducing the probability
  of stunting among children 12-36 mo.
• Household total consumption increases
• PROGRESA households eat better (higher
  expenditures on fruits, vegetables, meats
  animal products)

83
Cost Analysis

• Are the administrative costs of PROGRESA high?
• What are the private costs associated with
  participation in the program?
• What might be the indirect effects of the program
  on the national economy? (e.g. financing of the
  program)?

84
Evaluation Results Cost Analysis

• For every 100 pesos allocated to the program, 8.2
  pesos are administration/program costs.
• Very low compared to LICONSA (40 pesos per 100
  pesos) and TORTIVALES (14 pesos per 100)
• Targeting and conditioning of the program makeup
  56 of program costs (4.6 pesos out of 8.2 pesos)
• Private costs (3.8 pesos out of 8.2 pesos)

85
Evaluation Results Cost Analysis

• Eliminating distortionary food subsidies and
  using funds to finance a program like PROGRESA
  leads to substantial welfare gains.