Using Impact Evaluations to Identify What Works to Improve Agriculture Productivity

1
Using Impact Evaluations to Identify What Works
to Improve Agriculture Productivity
Ximena V. Del Carpio
The World Bank Mywish Maredia
Michigan State University

• March 31st, 2009
• NONIE conference
• Cairo, Egypt

2
Why do this study?

• Renewed attention to the role of agriculture
  investments in developing countries
• Direct and indirect linkages to sustained rural
  development and poverty reduction
• Genuine interest in institutional learning
• Opportunity to take stock of a growing body of
  evidence using various impact evaluation methods
• Understand what works (and why) to improve
  productivity

3
Study objectives

• Establish a structured way of aggregating the
  evidence
• Gather recent evaluations
• Apply quality filter (methodological rigor) to
  identify impact evaluations
• Provide a synthesis of the interventions and
  their impact channels (in the paper)
• Analyze statistically to identify relationships,
  gaps in knowledge and derive evidence-based
  conclusions
• What does the evidence in agriculture,
  derived from impact evaluations, reveal about
  what increases farmer productivity?

4
Meta-analysis

• MA allows for a statistical analysis of a
  collection of individual pieces (different from
  meta-review)
• Asses studies (1-by-1), dissect them from all
  angles, structure them in a cohesive form, and
  code it in a transparent and replicable manner
• Overcomes sample size limits, allows for
  understanding the effect of an intervention on
  productivity
• Minimizes risk of selection bias
• Allows for other intervention and/or evaluation
  factors to be accounted for (biases)

5
Challenges

• Large heterogeneity in rigor appliedevaluation
  designs rely heavily on techniques of
  association, qualitative, participatory, small
  samples
• Increasing but small availability of evaluations
  with/without/before/after data in a developing
  country context
• Difficulty in targeting policy interventions,
  release of broad technologies or keeping
  intervention to a well-defined group
• Types of interventions vary (different constructs
  and relationships), one at a time analyzed
• Overcoming challenges
• Break interventions into its components and
  categorize them into types based on their impact
  channel
• Statistical analysis is restricted to the
  direction of impact, not magnitude
• Analysis only uses evaluations with comparable
  units (household)

6
Study structure

• Methodology
• Definitions, conceptual framework and quality
  filter
• Description
• Descriptive information Study pool and
  Intervention types
• Meta-analysis
• Quantitative method of approach, analysis
• Conclusions

7
Definitions

• Intervention (project, program, policy, shock)
• Clear actions whose impacts are evaluated.
• Each has its own theory of change guiding its
  evaluation.
• Impact evaluation
• Changes in indicators of programs that can be
  attributed to the intervention, the causal link
  between action and productivity outcomes
• Establishes a credible counterfactual
  (experimentally and/or through econometric
  techniques) and deals with potential biases

8
Definitions (2)

• Productivity
• Direct Output produced per unit of input (land.
  Labor, capital)
• Example yields per unit of input or proxy
  indicators that measure outputs affecting change
  of farm household economic wellbeing (e.g.
  income)
• Indirect Various intermediate channels
  ?inclusiveness and flexible
• Examples farmer knowledge, input use,
  diversification of crops, infrastructure

9
Conceptual framework

• Impact pathway to productivity
• Clear intervention (expects) positive change and
  measures its effect
• Asses thematic fit and causal mechanism direct
  or indirect transmission channel

10
Study selection filter
11
Profile of studies
12
Interventions tracking productivity

• Eight broad categories
• -Titling 10, soil and water conservation 9
• -Rural financial services (non-credit) 2,
  cooperatives 3
• Overlapping themes and components
• Discern actions and link them to outcomes
• -Irrigation impact channels through improvements
  in water management systems (institutional),
  techniques (knowledge), systems to access water (
  pumps, dams, canals)
• Reconcile stated objectives with actual actions,
  measurement before statistical analysis

13
Counting exercise

• Most evaluations deal with more than one
  indicator and range in components
• A casual look at the numbers shows

14
Quantitative analysis

• Did the intervention component examined by paper
  X affect productivity or a indirect proxy for
  economic welfare?
• Eight broad categories? assigned to six groups
  that cut across intervention classification into
  the causes of impact
• Link studies with similar impact channels,
  theoretical constructs (98 observations)
• 61 of components look at direct impact
  indicators, 39 indirect
• Weight of the evidence
• Implementation characteristics
• Evaluation characteristics
• Parsimonious and cohesive models (clarity and
  interpretation)
• General model and by intervention type
• Weighted OLS (size of sample),
• Conditional (average or marginal) or
  unconditional effects

15
Interventions by impact channel

• Input enhancement adoption of improved
  technologies
• Farmer enhancement participation in farmer
  education
• Land enhancing Access to irrigation,
• Policy enhancing Increases in tenure security,
• Output enhancing Participation in contracts for
  export
• Financial services Loans

16
Results on impact channels

• Simple average effects analysis finds
• Success is highest for output related
  interventions (82)
• Lowest for land interventions (62)
• All interventions included finds
• Most keep positive relation but average effects
  reduce for all except policy interventions 77 to
  63. Credit interventions go from 66 to 7 and
  loses significance
• By intervention (complementarities)
• Input enhancing and farmer, land and output are
  all (complements)
• Marketing benefit from farmer knowledge, credit
  and policy
• Policy programs (reforms) show effectiveness on
  their own

17
Implementation characteristics
18
Results on implementation characteristics

• Neither national scale of program or government
  institutional capacity help determine success
• Cash crop focus is less likely to result in
  productivity impacts in all except soil related
  interventions (this is opposite for food crops or
  no crop focus)
• The role of private sector positively affects all
  except output related interventions

19
Evaluation characteristics
20
Results on evaluation characteristics

• Evaluation paper (date and venue) and other
  (author, method)
• Published studies show more positive results
• There is a general trend of obtaining positive
  results in recent years
• Donor as an author land interventions positive,
  input activities negative
• Experimental and quasi-experimental evaluations
  show few/no results

21
Conclusions

• Where are the gaps in the evidence?
• The evidence is slowly growing but it is
  generally thin (only 81 studies passed quality
  filter).
• Three weakest areas in need of evidence input
  enhancement (technology non-NRM), land
  enhancement (irrigation), and micro-finance
  (farmer loans, non-credit services) interventions
• Output enhancing (value chain) is also thin but
  there seems to be a growing trend (participation
  in marketing oriented groups)

22
Conclusions (2)

• What works to improve agriculture productivity?
• Overall, agriculture interventions have a high
  rate of success on increasing productivity
• Land enhancement is the most robust intervention
  in the sense that it has the highest rate of
  success whether accompanied by other components
  or not
• irrigation, conservation, soil management
• Policy reforms also have high success rates on
  their own (titling, deregulations, fomenting
  commerce/rural banking)
• Generally successful but depend on several other
  components are
• Input enhancement (seeds, fertilizer, technology
  adoption)
• Farmer enhancement (extension services)
• output enhancement (marketing, value chain)
• Micro-finance has some success (limited) but
  interventions may be too indirectmore evidence
  needed