Impact evaluation in the absence of baseline surveys

1
Impact evaluation in the absence of baseline
surveys

• By Fabrizio Felloni, Office of Evaluation, IFAD
• International Workshop on Development Impact
  Evaluation, Paris, November 15, 2006

2
The context of IFAD

• Relatively small projects 2005 median of IFAD
  loans US 15.5 m, project costs US 26 m
• Focus on rural poverty reduction
• Traditionally limited field presence of IFAD (15
  countries on a pilot basis), IFAD not executing
  or supervising projects, limited self- evaluation
• This scenario is evolving with new Action Plan

3
Field-based evaluation at IFAD - OE

• Necessary to make up for distance of headquarters
  from the field and information gap
• Several types project, country programme and
  corporate evaluations
• All include field visit and some form of primary
  data collection
• Project evaluations conducted just before or soon
  after project closure

4
Methodological requirements

• Standardised methodology for project and country
  programme evaluations requires assessing impact
  (standardised categories)
• No standardised data collection methods to be
  identified at approach paper phase
• Impact is but one of the analytical domains (also
  relevance, effectiveness, efficiency,
  sustainability innovation, performance of
  partners)
• So no dedicated instrument for impact assessment

5
Shoe string evaluation in action

• Considerations from personal experience

6
A case of shoe string impact evaluation

• See Bamberger et alii AJE, 25 (1), 2004
• A number of constraints
• 1. Time and budget (impact is one of the
  evaluation domains)
• 2. Poor performance of ME function at project
  level
• 3. Absence or limited usefulness of baseline
  data (now changing baseline survey with
  anthropometric and hh asset indicators for all
  new projects)

7
Logical steps for impact assessment
1
2
3
Multi-disciplinary field visit (mainly
qualitative direct observations)
Preliminary quantitative mini-survey
Impact assessment
Formulate first impact hypotheses, collect
evidence on selected basic indicators
Triangulation of mini-survey, focus groups and
individual interviews key informants
Validate hypotheses, probe on a set of narrower
questions
8
A pragmatic approach

• Within this context, impact assessment based on
  triangulation, still important qualitative
  component
• Still place for theory-based approach
• Quantitative survey used to test and generate new
  hypotheses, better focus questions during main
  mission
• Small sample size 200 350 respondents
  including project and control. Size determined
  by practical issues (represent project
  activities, time, transportation, budget)

9
Ideal scenario for the survey
1. Best case scenario quasi-experimental design
T0 programme group
T1 programme group
Baseline
Follow-up
C0 control group
C1 control group

• Ti and Ci measurable characteristic of the
  population, i time of observation (0,1).
• Unfortunately, this scenario is almost never
  found

10
Typical scenarios
1. Programme group only
T0 programme group
T1 programme group
Baseline
Follow-up
2. No baseline at all most frequent case
???
Evaluation
11
Other common issues

• Classical problems with control samples
  (selection bias, spill-over effects,
  non-compliance)
• Ex ante (i) visit similar communities or hh in
  administrative areas outside project, (ii) select
  new entries
• Ex post Mostly dealt with qualitatively at
  mission phase (triangulation)
• Main constraint to use of econometric techniques
  availability of trained specialists, time (impact
  is one of the evaluation domains)

12
Dealing with lack of baseline data

• Several options (not mutually exclusive)
• 1. Reconstructing baseline data ex post recall
  method (more later)
• 2. Use key informants and triangulate (mostly
  qualitative)
• 3. Reconstruct a baseline scenario with
  secondary data (not always practical given
  absence and quality of baseline studies)
• 4. Single difference with econometric techniques
  some practical obstacles (workload, time
  constraints, availability of trained specialists)

13
Recall methods

• Ask about current situation (e.g. cropping
  practices) now and at programme start-up

recall
T1 programme group
T0
C1 control group
C0
recall
14
Typical problems with recall methods

• Telescoping of major events / expenditures
• Under-estimation of small and routine events /
  expenditures
• Recall time line (events that are 3 -7 years old)
• Unintended misidentification of project start-up
• Strategic behaviour of respondents (to please
  the interviewer or express complaints)
• Some indicators are more complex to identify and
  remember with precision (income)

15
Some techniques to control problems

• Concentrate on few impact variables that are
  easier to visualise and recall. Some examples
  
• household appliances, livestock size (depending
  on the context),
• cropping patterns, agricultural and grazing
  practices, community initiatives)
• Help identify baseline point by helping recollect
  key facts and events
• Do not simply ask what, ask why, i.e.
  respondents to state causal linkages. E.g. the
  number of goats increased why? and how? Also
  useful for attribution.
• Pre-test the instruments

16
Practical examples
17
Ex 1.The Gambia Rural Finance Project (2004)

• Preliminary survey
• - Project and control group
• - Recall income and assets at hh and kafo
  level
• Data analysis
• - Descriptive statistics and significance tests
  principal component analysis
• - Generated two hypotheses
• (i) limited overall impact on hh income
• (ii) biases against relatively poorer hh in
  villages

18
Gambia, Rural Finance (contd)

• Field mission focus groups, individual
  interviews key informants
• - Confirmed limited effects on income
  generation opportunities
• - Credit collateral discouraged participation
  from poorer hh, ineffective in establishing
  credit discipline
• Main observations
• - Some validity threats in recall data on
  income and monetary assets
• - Consistency with qualitative findings
• - Help focus the scope of field mission

19
Ex. 2 Ghana Upper East Region

• Similar to the Gambia case (project control,
  recall)
• Multi-component agricultural project main
  intervention, small dams
• Recall on household productive and other durable
  assets
• Main findings seemed to show larger effects for
  project group
• Some methodological shortcomings
• - difficult to find matched observation for
  control group (given multi-component nature)
• - small sample size of control group may
  have affected significance tests

20
Example 3. Morocco, Southern Oasis

• Again, project and control, with recall method
• Many interventions, very heterogeneous, difficult
  to standardise questionnaires
• Focus on perceptions of trends (e.g. income
  generating opportunities, irrigation / potable
  water availability, feed for livestock)
• Hypothesis the project was effective as a buffer
  measure during years of drought. Supported by
  qualitative analysis in field mission

21
Concluding remarks

• Preliminary survey and recall methods never a
  stand alone measure but rather propaedeutic to
  (mainly) qualitative mission
• Triangulation to validate reliability of
  reconstructed baseline survey data, with field
  observations, focus group, individual interviews
  and key informants
• By and large, trends suggested by preliminary
  survey found to be consistent with qualitative
  data
• Some legitimate concerns on accuracy of estimated
  means for certain indicators (income, monetary
  assets)

22
Concluding remarks (contd)

• Evolution towards focus on perceived trends on a
  narrower set of key indicators
• Cost effective to conduct preliminary work with
  local specialists and students as enumerators
• Project teams consulted in planning and sampling
  phase. Results and database made available
• Valuable experience for local students
  enumerators
• In principle, replicable model for public
  authorities in charge of programme implementation